Simulation sound effect generation method and device, electronic equipment and storage medium

Abstract

The invention provides a simulated sound effect generation method and device, electronic equipment and a storage medium, and the method comprises the steps: obtaining a target animation element in a to-be-processed video frame and interaction relation information of the target animation element, generating posture description information of the target animation element according to the target animation element and the interaction relation information, and generating a simulated sound effect according to the posture description information. And according to the attitude description information, generating a mimicking sound effect of the target animation element, and according to the mimicking sound effect and the to-be-processed video frame, performing time sequence alignment so as to synchronously play the mimicking sound effect when the to-be-processed video frame is played. According to the scheme, the real-time generation and synchronization of the sound effect are realized by identifying the content of the video frame, generating the mimicking sound effect and automatically performing time sequence alignment according to the mimicking sound effect and the to-be-processed video frame, the labor cost is reduced, the method can adapt to different types of animations and diversified scene changes, and the method has good universality and expansibility.

Description

Technical Field

[0001] The present application relates to the technical field of audio processing, and in particular to a method, device, electronic device and storage medium for generating emulated sound effects. Background Art

[0002] In the process of game production, such as role-playing games (RPG), there are a large number of realistic non-player character (NPC) cutscenes that require sound effects. Among them, sound effects can add emotion and realism to the animation.

[0003] In the related art, a script is used to automatically generate the sound effects of video actions, and the sound effects are matched and adjusted manually. However, manually matching and adjusting the sound effects is not only time-consuming, but also requires a high level of professional knowledge and experience, and the labor cost is high. Summary of the invention

[0004] In view of this, the embodiments of the present application provide a method, device, electronic device and storage medium for generating foley sound effects to solve the problem of high labor costs caused by using scripts to automatically generate sound effects for video actions and manually matching and adjusting them.

[0005] In a first aspect, an embodiment of the present application provides a method for generating a mime sound effect, comprising:

[0006] Obtain target animation elements in the video frame to be processed and information on interactive relationships between the target animation elements;

[0007] Generating posture description information of the target animation element according to the target animation element and the interactive relationship information;

[0008] Generating a sound effect of the target animation element according to the posture description information;

[0009] Timing alignment is performed based on the Foley sound effect and the video frame to be processed, so that the Foley sound effect is played synchronously when the video frame to be processed is played.

[0010] In a second aspect, the embodiment of the present application further provides a device for generating a mime sound effect, including:

[0011] An acquisition module, used to acquire target animation elements in a video frame to be processed and information on interactive relationships between the target animation elements;

[0012] A generating module, used for generating posture description information of the target animation element according to the target animation element and the interactive relationship information;

[0013] The generating module is further used to generate the foley sound effect of the target animation element according to the posture description information;

[0014] A processing module is used to perform time alignment according to the foley sound effect and the video frame to be processed, so as to synchronously play the foley sound effect when playing the video frame to be processed.

[0015] In a third aspect, an embodiment of the present application further provides an electronic device, comprising: a processor, a memory and a bus, wherein the memory stores machine-readable instructions executable by the processor, and when the electronic device is running, the processor communicates with the memory through the bus, and the processor executes the machine-readable instructions to execute any one of the methods described in the first aspect.

[0016] In a fourth aspect, an embodiment of the present application further provides a computer-readable storage medium, on which a computer program is stored, and when the computer program is executed by a processor, the method described in any one of the first aspects is executed.

[0017] The present application provides a method, device, electronic device and storage medium for generating foley sound effects, wherein the method includes: obtaining target animation elements in a video frame to be processed and information about the interactive relationship between the target animation elements, generating posture description information of the target animation elements according to the target animation elements and the interactive relationship information, generating foley sound effects of the target animation elements according to the posture description information, and performing time alignment according to the foley sound effects and the video frame to be processed, so as to synchronously play the foley sound effects when playing the video frame to be processed. This solution generates foley sound effects by identifying the content of video frames, and automatically performs time alignment according to the foley sound effects and the video frame to be processed, thereby achieving real-time generation and synchronization of sound effects, reducing labor costs, being able to adapt to different types of animations and diverse scene changes, and having good versatility and extensibility. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings required for use in the embodiments will be briefly introduced below. It should be understood that the following drawings only show certain embodiments of the present application and therefore should not be regarded as limiting the scope. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without paying creative work.

[0019] Figure 1 Schematic diagram of the process of generating the foley sound effect provided in the embodiment of the present application Figure 1 ;

[0020] Figure 2 Schematic diagram of the process of generating the foley sound effect provided in the embodiment of the present application Figure 2 ;

[0021] Figure 3 Schematic diagram of the process of generating the foley sound effect provided in the embodiment of the present application Figure 3 ;

[0022] Figure 4 Schematic diagram of the process of generating the foley sound effect provided in the embodiment of the present application Figure 4 ;

[0023] Figure 5 Schematic diagram of the process of generating the foley sound effect provided in the embodiment of the present application Figure 5 ;

[0024] Figure 6 Schematic diagram of the process of generating the foley sound effect provided in the embodiment of the present application Figure 6 ;

[0025] Figure 7 Schematic diagram of the process of generating the foley sound effect provided in the embodiment of the present application Figure 7 ;

[0026] Figure 8 Schematic diagram of the process of generating the foley sound effect provided in the embodiment of the present application Figure 8 ;

[0027] Fig. 9 A schematic diagram of the structure of a device for generating emoticon sound effects provided in an embodiment of the present application;

[0028] Fig.10 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0029] To make the purpose, technical scheme and advantages of the embodiments of the present application clearer, the technical scheme in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, rather than all of the embodiments. The components of the embodiments of the present application generally described and shown in the drawings here can be arranged and designed in various different configurations. Therefore, the following detailed description of the embodiments of the present application provided in the drawings is not intended to limit the scope of the application claimed for protection, but merely represents the selected embodiments of the present application. Based on the embodiments of the present application, all other embodiments obtained by those skilled in the art without making creative work belong to the scope of protection of the present application.

[0030] The production of Foley sound effects for games traditionally requires sound artists to manually select or create sound effects suitable for animation. However, traditional sound effect production is not only time-consuming but also requires a high level of expertise and experience. Manual operations are cumbersome and involve a large amount of manpower and computing resources, which significantly increases production costs, especially for complex and long-lasting animation content.

[0031] With the development of technology, it has become possible to automatically generate sound effects that match animations. Usually, automated scripts or existing semi-automatic or auxiliary tools are used to help generate foley sound effects. However, the automatic generation of sound effects for video actions using scripts requires manual matching and adjustment. For example, if there is a change in action or time misalignment, it is necessary to manually match the points and edit the sample length again, which is time-consuming and labor-intensive, and has high labor costs. In addition, although some existing semi-automatic or auxiliary tools can help generate sound effects, these tools often cannot accurately match the subtle movements and details in the animation, resulting in the sound effects and visual content not being fully synchronized or not natural enough, and the match is not accurate. In addition, some sound effect generation tools may only be applicable to specific types of animations or specific sound effect libraries, lacking versatility and extensibility.

[0032] Most of the above-mentioned sound effect generation solutions are based on preset libraries or simple rule matching, lack intelligent artificial intelligence (AI) algorithms, and are difficult to dynamically adapt to different types of animations and diverse scene changes. In addition, sound effects cannot be generated in real time and require multiple adjustments and post-processing. They are not suitable for real-time application scenarios, such as live animation or interactive applications, and have a long cycle. If the animation rhythm is temporarily changed, it is easy to be caught off guard.

[0033] Based on this, the present application generates onomatopoeia sound effects by identifying the content of video frames, and automatically performs timing alignment based on the onomatopoeia sound effects and the video frames to be processed, thereby realizing real-time generation and synchronization of sound effects, reducing labor costs, and being able to adapt to different types of animations and diverse scene changes, with good versatility and scalability.

[0034] Figure 1 Schematic diagram of the process of generating the foley sound effect provided in the embodiment of the present application Figure 1 The execution subject of this embodiment can be an electronic device, such as a mobile phone, a computer, etc.

[0035] like Figure 1 As shown, the method may include:

[0036] S101, obtaining target animation elements in a video frame to be processed and interactive feature information of the target animation elements.

[0037] The video frame to be processed may be any video frame among multiple video frames of the target video, and the target video may be any video, such as a game video.

[0038] Computer vision technology is used to parse the video frame to be processed to obtain the target animation element and the interactive relationship information of the target animation element, wherein the target animation element is any animation element in the video frame to be processed, such as people, animals, vehicles, etc. in the video frame to be processed, and the interactive relationship information of the target animation element is used to indicate the interactive relationship between the target animation element and other animation elements in the video frame to be processed. For example, the target animation elements are "cup" and "vehicle", and the other animation elements are the ground and the bridge, and the interactive relationship information is "broken on the ground" and "driving on the bridge".

[0039] S102: Generate posture description information of the target animation element according to the target animation element and the interactive relationship information.

[0040] According to the target animation element and the interactive relationship information, the posture description information of the target animation element is generated, wherein the posture description information is used to describe the posture of the target animation element. For example, if the target animation element is a "cup" and the interactive feature information is "broken on the ground", then the posture description information may be "the cup is broken on the ground"; for another example, if the target animation element is a "vehicle" and the interactive feature information is "driving on a bridge", then the posture description information may be "the vehicle drives over the bridge".

[0041] As an example, a predefined template is used to generate the posture description information of the target animation element according to the target animation element and the interactive relationship information. The reference code is as follows:

[0042] def generate_description(predictions):

[0043] objects=[label for imagenet_id,label,score in predictions[0]]

[0044] if'person'in objects and'cup'in objects:

[0045] description="The person is holding a cup."

[0046] In this example, the gesture description information is “someone picked up the cup”.

[0047] elif'dog'in objects and'ball'in objects:

[0048] description="The dog is playing with a ball."

[0049] In this example, the gesture description information is "a puppy is playing with a ball."

[0050] S103: Generate a mime sound effect of the target animation element according to the posture description information.

[0051] Among them, the posture description information can reflect the posture changes of the target animation element in the video frame to be processed. According to the posture description information, the onomatopoeia sound effect of the target animation element can be generated, so that the onomatopoeia sound effect of the target animation element generated in this way can match the posture description information of the target animation element.

[0052] Taking the posture description information as "a cup is broken on the ground" as an example, the onomatopoeia sound effect is the sound effect of the cup breaking on the ground, taking the posture description information as "a vehicle passes over a bridge" as an example, the onomatopoeia sound effect is the sound effect of the vehicle passes over a bridge, taking the posture description information as "picking up the cup" as an example, the onomatopoeia sound effect is the sound effect of the cup being picked up from the table, taking the posture description information as "a person is drinking water" as an example, the onomatopoeia sound effect is the sound effect of a person drinking water, such as the "gulp gulp" sound effect.

[0053] S104: Perform time alignment according to the Foley sound effect and the video frame to be processed, so as to synchronously play the Foley sound effect when playing the video frame to be processed.

[0054] The Foley sound effect and the video frame to be processed are time-aligned so that the Foley sound effect is played synchronously when the video frame to be processed is played. This ensures that the generated Foley sound effect accurately matches the picture of the video frame to be processed on the timeline.

[0055] It is worth mentioning that the video frames to be processed can be imported into the preset audio editor, and video frame parsing, sound effect generation and timing alignment can be performed in the preset audio editor.

[0056] In this embodiment, by accurately parsing the video frames to be processed, the interactive relationship information between the target animation elements and the target animation elements in the video frames to be processed is obtained, and then the posture description information of the target animation elements is generated, and then the matching foley sound effects are automatically generated, and the timing alignment is automatically performed according to the foley sound effects and the video to be processed, so as to realize the real-time generation and synchronization of the sound effects, ensure that the sound effects are perfectly matched with the posture description information of the target animation elements, and the generated foley sound effects are of high quality and naturalness, especially when processing complex actions and details, and can be applied to different types of animations and diversified scenes, with good scalability and versatility.

[0057] Figure 2Schematic diagram of the process of generating the foley sound effect provided in the embodiment of the present application Figure 2 ,like Figure 2 As shown, in an optional implementation, the above step S104, performing timing alignment according to the Foley sound effect and the video frame to be processed, so as to synchronously play the Foley sound effect when playing the video frame to be processed, includes:

[0058] S201, obtaining a video frame rate of a target video corresponding to a to-be-processed video frame, and a frame index of the to-be-processed video frame in the target video.

[0059] The video frame rate of the target video is used to indicate the number of frames displayed per second of the target video, for example, 50 frames per second (50 FPS).

[0060] The frame index is used to indicate the position of the video frame to be processed in multiple video frames of the target video. The frame index can be an integer starting from 0. For example, the frame index of the video frame to be processed in the target video is 50, indicating that the video frame to be processed is the 50th frame of the target video.

[0061] S202: Determine the starting time point of the to-be-processed video frame in the target video according to the video frame rate and the frame index.

[0062] According to the video frame rate and frame index, the frame index of the video frame to be processed is converted into a time point to obtain the starting time point of the video frame to be processed in the target video. The starting time point is the time point in the target video that triggers the playback of the video frame to be processed, which can be expressed as:

[0063] Starting time = frame index / video frame rate

[0064] The unit of the starting time point may be seconds. For example, the video frame rate is 50 FPS, the frame index is the 50th frame, and the starting time point is 1 second. That is, the playback of the video frame to be processed is triggered at the 1st second of the target video.

[0065] S203, according to the starting time point of the video frame to be processed, timing aligning the Foley sound effect and the video frame to be processed, so as to synchronously play the video frame to be processed and the Foley sound effect from the starting time point of the video frame to be processed.

[0066] According to the starting time point of the video frame to be processed, the Foley sound effect and the video frame to be processed are time-aligned on the timeline, so that the video frame to be processed and the Foley sound effect are played synchronously from the starting time point of the video frame to be processed, so as to ensure that the generated Foley sound effect accurately matches the picture of the video frame to be processed on the timeline.

[0067] In this embodiment, by reading the video frame rate and the frame index of the video frame to be processed, the frame index is converted into a time point to automatically synchronize the video frame to be processed and the simulated sound effect, thereby realizing real-time generation and synchronization of the sound effect and reducing labor costs.

[0068] Figure 3 Schematic diagram of the process of generating the foley sound effect provided in the embodiment of the present application Figure 2 ,like Figure 3 As shown, in an optional embodiment, the method further includes:

[0069] S301, according to the sound effect peak value of the Foley sound effect, intercept the waveform of the Foley sound effect to obtain the intercepted sound effect.

[0070] Among them, the Foley sound effect is in waveform form, and the sound effect peak value of the Foley sound effect is found. The sound effect peak value can be understood as the maximum volume.

[0071] Starting from the time point at which the sound effect peak value of the Foley sound effect is located, the waveform of the Foley sound effect is intercepted to obtain an intercepted sound effect, and the starting time point of the intercepted sound effect is the time point at which the sound effect peak value is located.

[0072] The above step S203, according to the starting time point of the video frame to be processed, performs time alignment on the Foley sound effect and the video frame to be processed, so as to synchronously play the video frame to be processed and the Foley sound effect from the starting time point of the video frame to be processed, includes:

[0073] S302: According to the starting time point of the video frame to be processed, the captured sound effect and the video frame to be processed are time-aligned to synchronously play the video frame to be processed and the captured sound effect starting from the starting time point of the video frame to be processed.

[0074] According to the starting time point of the video frame to be processed, the intercepted sound effect and the video frame to be processed are time-aligned on the timeline, so that the video frame to be processed and the intercepted sound effect are played synchronously from the starting time point of the video frame to be processed, that is, the playing of the video frame to be processed is the moment when the sound effect is triggered. This triggering moment is the peak value of the sound effect, and it is automatically aligned with the video frame to be processed on the timeline to ensure that the generated intercepted sound effect accurately matches the picture of the video frame to be processed on the timeline.

[0075] In this embodiment, by automatically aligning the sound effect peak and the video frame to be processed on the timeline, when the video frame to be processed is triggered to play, the sound effect matching the animation element in the video frame to be processed can be clearly perceived, thereby improving the user experience.

[0076] Figure 4 Schematic diagram of the process of generating the foley sound effect provided in the embodiment of the present application Figure 4 ,like Figure 4 As shown, in an optional embodiment, the method further includes:

[0077] S401, extracting features of the Foley sound effect to obtain sound effect features of the Foley sound effect.

[0078] S402: Acquire a candidate sound effect that matches the Foley sound effect according to the sound effect feature of the Foley sound effect.

[0079] Feature extraction is performed on the mimeograph sound effect to obtain the sound effect features of the mimeograph sound effect. The sound effect features may include, for example, volume features, pitch features, and timbre features. Based on the sound effect features of the mimeograph sound effect, a candidate sound effect matching the mimeograph sound effect may be searched from a preset sound effect library. Alternatively, based on the sound effect features of the mimeograph sound effect, a candidate sound effect matching the mimeograph sound effect may be generated, wherein the similarity between the candidate sound effect and the mimeograph sound effect exceeds a preset similarity threshold.

[0080] It is worth noting that, through the compatibility of the AI ​​plug-in and the preset audio editor, the AI ​​plug-in is started in the preset audio editor to use the AI ​​plug-in to obtain candidate sound effects that match the Foley sound effect based on the sound effect characteristics of the Foley sound effect.

[0081] The above step S104, performing time alignment according to the Foley sound effect and the video frame to be processed, so as to synchronously play the Foley sound effect when playing the video frame to be processed, includes:

[0082] S403: Determine a Foley sound effect to be played from the Foley sound effects and the candidate sound effects.

[0083] S404: timing-align the Foley sound effect to be played and the video frame to be processed, so that the Foley sound effect to be played is played synchronously with the video frame to be processed.

[0084] According to the input selection operation for the foley sound effect and the candidate sound effects, the foley sound effect to be played is determined from the foley sound effects and the candidate sound effects, and the foley sound effect to be played and the video frame to be processed are time-aligned. The foley sound effect to be played and the video frame to be processed are time-aligned on the timeline to synchronously play the video frame to be processed and the foley sound effect to be played starting from the starting time point of the video frame to be processed, so as to ensure that the foley sound effect to be played accurately matches the picture of the video frame to be processed on the timeline.

[0085] In an optional embodiment, the waveform of the simulated sound effect to be played can be captured to obtain the captured sound effect, and the captured sound effect and the video frame to be processed can be time-aligned according to the starting time point of the video frame to be processed, so that the video frame to be processed and the captured sound effect can be played synchronously from the starting time point of the video frame to be processed.

[0086] In this embodiment, by matching candidate sound effects of the Foley sound effect and determining the Foley sound effect to be played from the Foley sound effect and the candidate sound effects, more sound effects can be generated, thereby improving the diversity of the generated sound effects.

[0087] In an optional implementation, the above step S101, obtaining the target animation element in the video frame to be processed and the interactive relationship information of the target animation element, may include:

[0088] A pre-trained deep learning model is used to perform target detection and posture detection on the video frames to be processed, and to determine the target animation elements and interactive relationship information.

[0089] Among them, the pre-trained deep learning model can be a target detection model such as a regional convolutional neural network (Region-CNN, R-CNN) model and a You Only Look Once (YOLO) model.

[0090] A pre-trained deep learning model is used to perform target monitoring on the video frames to be processed, obtain the target animation elements in the video frames to be processed, and perform posture detection on the video frames to be processed to obtain the interactive relationship information of the target animation elements.

[0091] It is worth noting that a deep learning model is used to perform target detection on the video frames to be processed, and the prediction probabilities of multiple candidate animation elements, i.e., confidence levels, are obtained, and the candidate animation element with the highest prediction probability among the multiple candidate animation elements is determined as the target animation element, that is, the candidate animation element with the highest prediction probability is determined as the target animation element.

[0092] It is worth noting that the output of the pre-trained deep learning model can be a two-dimensional tensor with a shape of (batch_size, num_classes), where batch_size is the batch size, that is, the number of video frames input at a time, and num_classes is the number of categories that the model can predict (that is, the number of candidate animation elements). For example, the ImageNet dataset can have 1000 categories, that is, the output is a tensor with a shape of (1, 1000).

[0093] Through the decoding function, such as `decode_predictions`, the output tensor is converted into a specific category label and probability or confidence. The parsed result contains three kinds of information: ImageNet ID (category identifier in the dataset, that is, the identifier of the candidate animation element in the dataset), Label (category name, that is, the name of the candidate animation element) and Score (confidence of the candidate animation element).

[0094] The decoding function can be used to decode the model output into readable categories and confidence levels. Through this decoding process, the original output of the model can be converted into easy-to-understand prediction results, which is convenient for subsequent processing and use.

[0095] For example, the prediction result of the video frame to be processed can be expressed as:

[0096] 1:coffee_mug(0.75)

[0097] 2: cup (0.10)

[0098] 3: espresso (0.05)

[0099] 4:wine_bottle(0.03)

[0100] 5:carton(0.02)

[0101] Among them, serial number 1 represents the first prediction result, and the others are similar. Category label (label): represents the category name (animation element name) predicted by the model, such as `coffee_mug` (coffee cup), `cup` (cup), etc. Confidence (score): represents the model's prediction probability for this category.

[0102] In the above example, the confidence of `coffee_mug` is 0.75 (i.e. 75% probability), followed by `cup` with a confidence of 0.10 (i.e. 10% probability), and so on. Therefore, it can be determined that `coffee_mug` (coffee cup) is the target animation element.

[0103] In an optional implementation, before the step S101 of obtaining the target animation element in the video frame to be processed and the interactive relationship information of the target animation element, the video frame to be processed may be preprocessed.

[0104] The preprocessing of the video frame to be processed includes at least one of adjusting the image size, normalizing the pixel values, and adding a batch dimension.

[0105] The input of the pre-trained deep learning model is usually a tensor of shape (batch_size, height, width, channels), where batch_size is the batch size, that is, the number of video frames input at a time, height and width are the height and width of the video frame, such as 224x224, and channels is the number of channels of the video frame, which is 3 for RGB video frames.

[0106] Among them, since the pre-trained deep learning model may require the input image to have a specific size, such as 224x224 pixels, you can use OpenCV's `resize` function to adjust the image size of the video frame to be processed. The reference code is as follows:

[0107] input_frame=cv2.resize(frame,(224,224)).

[0108] To reduce the processing complexity, the pixel values ​​of the video frames to be processed can be normalized to scale the pixel values ​​from the range of [0,255] to the range expected by the pre-trained deep learning model, such as [0,1] or [-1,1]. The reference code is as follows:

[0109] frame_normalized=frame_resized / 255.0

[0110] You can also convert pixel values ​​to a specified data type, such as the floating-point type of `float32`. The reference code is as follows:

[0111] frame_normalized=frame_normalized.astype(np.float32)

[0112] Among them, since the pre-trained deep learning model may expect the input data to have a batch dimension, even a single video frame needs to be converted into a tensor with a shape of (1, height, width, channels). The reference code is as follows:

[0113] frame_batch=np.expand_dims(frame_normalized,axis=0)

[0114] Among them, you can use the `np.expand_dims` function to add a batch dimension in the 0th dimension to convert the image from shape (224,224,3) to (1,224,224,3). In this way, even if you are processing a single video frame, the pre-trained deep learning model will think it is a batch containing one video frame.

[0115] Figure 5 Schematic diagram of the process of generating the foley sound effect provided in the embodiment of the present application Figure 5 ,like Figure 5 As shown, in an optional implementation, a pre-trained deep learning model is used to perform target detection and posture detection on the video frame to be processed, and target animation elements and interactive relationship information are determined, including:

[0116] S501. Use the feature extraction module in the deep learning model to extract visual features from the video frame to be processed to obtain target visual features.

[0117] S502: Use the first recognition module in the deep learning model to perform target detection on the target visual features to obtain the target animation element.

[0118] S503: Using the second recognition module in the deep learning model, perform posture detection on the target visual features according to the target animation elements to obtain interactive relationship information.

[0119] A feature extraction module in the deep learning model is used to extract visual features of the video frames to be processed to obtain target visual features, wherein the target visual features are visual features of the video frames to be processed. A first recognition module in the deep learning model is used to perform target detection on the target visual features to obtain target animation elements. A second recognition module in the deep learning model is used to perform posture detection on the target visual features according to the target animation elements to obtain interactive relationship information of the target animation elements.

[0120] It uses intelligent AI algorithms to identify and analyze the video frames to be processed, and can dynamically adapt to different types of animations and diverse scene changes with high recognition accuracy.

[0121] In an optional implementation, the above step S503 uses the second recognition module in the deep learning model to perform posture detection on the target visual features according to the target animation elements to obtain the interactive relationship information, including:

[0122] The second recognition module is used to perform posture detection on the target visual features and the visual features of at least one video frame continuous with the video frame to be processed according to the target animation element to obtain interactive relationship information.

[0123] Among them, the visual features of at least one video frame continuous with the video frame to be processed are obtained by extracting visual features of at least one video frame using a feature extraction module in a deep learning model, and the visual features of at least one video frame are visual features of the corresponding video frame.

[0124] A second recognition module is used to perform posture detection on the target visual features and at least one video frame continuous with the video frame to be processed according to the target animation element, so as to obtain the posture features of the target animation element in the video frame to be processed and the posture features in at least one video frame. The posture features are static features, and the posture trajectory of the target animation element is determined according to the posture features, and then the interactive relationship information of the target animation element is determined according to the posture trajectory.

[0125] Taking the target animation element as a "cup" and at least one video frame continuous with the video frame to be processed as the previous video frame of the video frame to be processed as an example, the posture feature of the target animation element in the video frame to be processed is that the cup is on the ground and broken, and the posture feature of the target animation element in the video frame previous to the video frame to be processed is that the cup is in a person's hand, then the interactive relationship information of the cup is determined to be "falling from the hand and being broken onto the ground."

[0126] Continuing with the example of the target animation element being a "cup", the posture feature of the target animation element in the video frame to be processed is that the cup is picked up by a person, and the posture feature of the target animation element in the previous video frame of the video frame to be processed is that the cup is on the table, then the interactive relationship information of the cup is determined to be "a person picks it up from the table".

[0127] In this embodiment, an intelligent AI algorithm is used to analyze and identify the video frame to be processed and at least one video frame continuous with the video frame to be processed to obtain the interactive relationship information of the target animation element. The accuracy of the interactive relationship information is improved to ensure that the sound effect and the posture description information of the target animation element are perfectly matched. The generated sound effect needs to be of high quality and naturalness, especially when dealing with complex actions and details, and can be applied to different types of animations and diversified scenes, with good scalability and versatility.

[0128] Figure 6 Schematic diagram of the process of generating the foley sound effect provided in the embodiment of the present application Figure 6 ,like Figure 6 As shown, in an optional implementation, the above step S102 generates the posture description information of the target animation element according to the target animation element and the interactive relationship information, including:

[0129] S601: Generate a first prompt word according to the target animation element and the interactive relationship information.

[0130] S602: Generate gesture description information according to the first prompt word using a preset natural language model.

[0131] According to the target animation element and the interactive relationship information, a first prompt word (Prompts) is generated, and the first prompt word is used to prompt the generation of posture description information under natural language description based on the target animation element and the interactive relationship information. The first prompt word is used as an input of a preset natural language model, and the first prompt word is semantically analyzed to generate the posture description information of the target animation element.

[0132] The preset natural language model may be, for example, a Transformer model, which is used to convert the first prompt word into a natural language description, such as “the person is drinking water”.

[0133] In this embodiment, the target animation element and the interactive relationship information are converted into a first prompt word and then converted into gesture description information, so that a subsequent model can generate a mime sound effect based on the gesture description information.

[0134] Figure 7 Schematic diagram of the process of generating the foley sound effect provided in the embodiment of the present application Figure 7 ,like Figure 7 As shown, in an optional implementation, the above step S103, generating the mime sound effect of the target animation element according to the posture description information, includes:

[0135] S701. Generate a simulated sound effect using a pre-trained sound effect generation model according to the posture description information.

[0136] Among them, the pre-trained sound effect generation model is a sound effect generation model obtained by pre-training, which uses the posture description information as the input of the sound effect generation model to generate the imitation sound effect of the target animation element.

[0137] or,

[0138] S702: Generate a second prompt word according to the posture change description information.

[0139] S703: Generate a simulated sound effect using a preset large model according to the second prompt word.

[0140] A second prompt word is generated according to the posture change description information. The second prompt word is used to generate the onomatopoeia sound effect of the target animation element by using the preset large model prompt. The second prompt word is used as the input of the preset large model to generate the onomatopoeia sound effect of the target animation element.

[0141] Among them, the preset large model refers to a machine learning model with ultra-large-scale parameters and super-powerful computing resources, which can process massive data and complete various complex tasks, such as natural language processing and image recognition.

[0142] In this embodiment, the pre-trained sound effect generation model or the preset large model can understand different types of sound effects, such as environmental sounds, object collision sounds, or sounds produced by character actions. Therefore, the pre-trained sound effect generation model or the preset large model is used to generate high-quality imitation sound effects, which can match the element features and interaction features of the animation elements in the video frame to be processed, thereby realizing AI algorithm recognition and adaptive generation of sound effects.

[0143] In an optional implementation, the above step S103, generating the mime sound effect of the target animation element according to the posture description information, includes:

[0144] Generate foley sound effects based on the input sound effect configuration information and posture description information.

[0145] The sound effect configuration information is input by the user through the interface. According to the input sound effect configuration information and the posture description information, the sound effect of the target animation element is generated, wherein the sound effect configuration information includes at least one of sound length, pitch, tone, and volume. In this way, the configuration information of the generated sound effect matches the input sound effect configuration information. For example, if the input sound length is 3 seconds, the duration of the generated sound effect is 3 seconds.

[0146] In an optional implementation, a pre-trained sound effect generation model is used to generate the simulated sound effect based on the input sound effect configuration information and posture description information.

[0147] In another optional implementation, a second prompt word is generated based on the input sound effect configuration information and posture description information, and based on the second prompt word, a preset large model is used to generate a simulated sound effect.

[0148] In this embodiment, the video frames to be processed are parsed by an AI algorithm, and the Foley sound effects of corresponding duration are automatically generated, thereby achieving precise control of the Foley sound effects.

[0149] In an optional embodiment, the method may further include:

[0150] The foley sound effects of the target animation element are stored in the preset sound effect library.

[0151] The generated foley sound effects of the target animation elements are stored in a preset sound effect library, and a correspondence between the target animation elements and the interactive relationship information and the foley sound effects is established. In subsequent use, the corresponding foley sound effects can be automatically matched from the preset sound effect library based on the parsed animation elements and interactive relationship information.

[0152] The reference code is as follows:

[0153] sound_effects = {

[0154] 'cup':'path / to / cup_sound.wav',

[0155] 'water':'path / to / water_pour_sound.wav',

[0156] 'clothing':'path / to / clothing_rustle_sound.wav'

[0157] }

[0158] def match_sound_effect(features):

[0159] matched_sounds = []

[0160] for feature in features:

[0161] object_class,_=feature

[0162] if object_class in sound_effects:

[0163] matched_sounds.append(sound_effects[object_class])

[0164] return matched_sounds

[0165] It is worth noting that the electronic device may have a built-in cross-modal search platform tool, which may be connected to a pre-trained sound effect generation model and a preset large model. The sound effect configuration information and posture description information may be input through the cross-modal search platform tool to generate onomatopoeia sound effects. In addition, the number of onomatopoeia sound effects to be generated and reverse prompt words may be input, and the corresponding number of onomatopoeia sound effects may be generated in combination with the number of generation and the reverse prompt words, and the onomatopoeia sound effects may be exported to the preset sound effect library.

[0166] It is understandable that the cross-modal search platform tool can also display the identification of the generated foley sound effect, set page turning controls, and when a foley sound effect is selected, display the waveform of the selected foley sound effect. The specific display style of the cross-modal search platform tool can be set according to the situation and is not particularly limited here.

[0167] Figure 8 Schematic diagram of the process of generating the foley sound effect provided in the embodiment of the present application Figure 8 ,like Figure 8 As shown, in an optional implementation, the method may further include:

[0168] S801. Create a target audio track in a preset audio editor, and load the Foley sound effect into the target audio track.

[0169] S802: Edit the Foley sound effect of the video frame to be processed according to the input sound effect editing operation to generate a new Foley sound effect.

[0170] A target audio track is created in a preset audio editor, and the Foley sound effect of the video frame to be processed is loaded into the target audio track. The user can input an audio effect editing operation to edit the Foley sound effect of the video frame to be processed and generate a new Foley sound effect. The audio effect editing operation is used to edit at least one of the length, pitch, tone, and volume of the Foley sound effect of the video frame to be processed.

[0171] In an optional implementation, the Foley sound effect of the video frame to be processed is saved as a ReaScript script file (such as auto_sound_effect_generator.py), the script is loaded and run, the digital audio workstation is opened, and a target audio track is created, the script file of the Foley sound effect of the video frame to be processed is loaded into the target audio track in the preset audio editor, the sound effect editing operation is input, and a new Foley sound effect is generated. The new Foley sound effect can also be saved in the preset sound effect library. In the digital audio workstation, the Foley sound effect can also be selected and played.

[0172] The above step S104, performing time alignment according to the Foley sound effect and the video frame to be processed, so as to synchronously play the Foley sound effect when playing the video frame to be processed, includes:

[0173] S803: Perform time alignment based on the new Foley sound effect and the video frame to be processed, so as to synchronously play the new Foley sound effect when playing the video frame to be processed.

[0174] The new Foley sound effect and the video frame to be processed are time-aligned so that the new Foley sound effect is played synchronously when the video frame to be processed is played. This ensures that the generated new Foley sound effect accurately matches the picture of the video frame to be processed on the timeline.

[0175] In this embodiment, after generating the foto sound effects of the video frame to be processed, the sound effects can also be edited, realizing a one-stop full process of sound effect matching and editing, making the sound effect processing more flexible and diverse, realizing real-time generation and synchronization of sound effects, and reducing labor costs.

[0176] Fig. 9 This is a schematic diagram of the structure of a device for generating imitation sound effects provided in an embodiment of the present application, which can be integrated into an electronic device.

[0177] like Fig. 9 As shown, the device may include:

[0178] An acquisition module 901 is used to acquire target animation elements in a video frame to be processed and information on interactive relationships between target animation elements;

[0179] A generating module 902, for generating posture description information of a target animation element according to the target animation element and the interactive relationship information;

[0180] The generating module 902 is further used to generate the foley sound effect of the target animation element according to the posture description information;

[0181] The processing module 903 is used to perform time alignment according to the Foley sound effect and the video frame to be processed, so as to synchronously play the Foley sound effect when playing the video frame to be processed.

[0182] In an optional implementation manner, the processing module 703 is specifically configured to:

[0183] Obtain the video frame rate of the target video corresponding to the video frame to be processed, and the frame index of the video frame to be processed in the target video;

[0184] According to the video frame rate and frame index, determine the starting time point of the video frame to be processed in the target video;

[0185] According to the starting time point of the video frame to be processed, the Foley sound effect and the video frame to be processed are time-aligned so that the video frame to be processed and the Foley sound effect are played synchronously starting from the starting time point of the video frame to be processed.

[0186] In an optional implementation, the processing module 903 is further configured to:

[0187] According to the sound effect peak of the Foley sound effect, the waveform of the Foley sound effect is intercepted to obtain an intercepted sound effect, and the starting time point of the intercepted sound effect is the time point of the sound effect peak;

[0188] The processing module 903 is specifically used to perform time alignment on the captured sound effect and the video frame to be processed according to the starting time point of the video frame to be processed, so as to synchronously play the video frame to be processed and the captured sound effect from the starting time point of the video frame to be processed.

[0189] In an optional implementation, the processing module 903 is further configured to:

[0190] Extracting features of the Foley sound effect to obtain sound effect features of the Foley sound effect;

[0191] An acquisition module 901 is used to acquire a candidate sound effect matching the Foley sound effect according to the sound effect feature of the Foley sound effect;

[0192] The processing module 903 is specifically used for:

[0193] Determine a Foley sound effect to be played from the Foley sound effects and the candidate sound effects;

[0194] The Foley sound effect to be played and the video frame to be processed are time-aligned so that the Foley sound effect to be played is played synchronously when the video frame to be processed is played.

[0195] In an optional implementation, the acquisition module 901 is specifically configured to:

[0196] A pre-trained deep learning model is used to perform target detection and posture detection on the video frames to be processed, and to determine the target animation elements and interactive relationship information.

[0197] In an optional implementation, the acquisition module 901 is specifically configured to:

[0198] The feature extraction module in the deep learning model is used to extract visual features of the video frame to be processed to obtain the target visual features;

[0199] Using the first recognition module in the deep learning model, target detection is performed on the target visual features to obtain the target animation element;

[0200] The second recognition module in the deep learning model is used to perform posture detection on the target visual features according to the target animation elements to obtain the interactive relationship information.

[0201] In an optional implementation, the acquisition module 901 is specifically configured to:

[0202] The second recognition module is used to perform posture detection on the target visual features and the visual features of at least one video frame continuous with the video frame to be processed according to the target animation element to obtain interactive relationship information.

[0203] In an optional implementation, the generating module 902 is specifically configured to:

[0204] Generate a first prompt word according to the target animation element and the interactive relationship information;

[0205] According to the first prompt word, a preset natural language model is used to generate posture description information.

[0206] In an optional implementation, the generating module 902 is specifically configured to:

[0207] Based on the posture description information, a pre-trained sound effect generation model is used to generate foley sound effects;

[0208] or,

[0209] Generate a second prompt word according to the posture change description information;

[0210] According to the second prompt word, a preset large model is used to generate a sound effect.

[0211] In an optional implementation, the generating module 902 is specifically configured to:

[0212] Generate a mime sound effect of a target animation element according to the input sound effect configuration information and posture description information, wherein the sound effect configuration information includes at least one of sound length, pitch, tone, and volume.

[0213] In an optional implementation, the processing module 903 is further configured to:

[0214] Create a target audio track in the preset audio editor and load the Foley sound effect into the target audio track;

[0215] Edit the Foley sound effect according to the input sound effect editing operation to generate a new Foley sound effect;

[0216] The processing module 903 is specifically used for:

[0217] Timing alignment is performed according to the new Foley sound effect and the video frame to be processed, so that the new Foley sound effect is played synchronously when the video frame to be processed is played.

[0218] In an optional embodiment, the device further comprises:

[0219] The saving module 904 is used to store the foley sound effect in a preset sound effect library.

[0220] In this embodiment, the acquisition module is used to acquire the target animation element in the video frame to be processed and the interactive relationship information of the target animation element. The generation module is used to generate the posture description information of the target animation element based on the target animation element and the interactive relationship information. The generation module is also used to generate the foley sound effect of the target animation element based on the posture description information. The processing module is used to perform timing alignment according to the foley sound effect and the video frame to be processed, so as to synchronously play the foley sound effect when playing the video frame to be processed. This solution generates foley sound effects by identifying the content of the video frame, and automatically performs timing alignment according to the foley sound effects and the video frame to be processed, thereby realizing real-time generation and synchronization of sound effects, reducing labor costs, being able to adapt to different types of animations and diverse scene changes, and having good versatility and extensibility.

[0221] Fig.10 A schematic diagram of the structure of an electronic device provided by an embodiment of the present application, as shown in the figure, the device may include: a processor 1001, a memory 1002 and a bus 1003, the memory 1002 stores machine-readable instructions executable by the processor 1001, when the electronic device is running, the processor 1001 communicates with the memory 1002 through the bus 1003, and the processor 1001 executes the machine-readable instructions to perform the following steps:

[0222] Obtain target animation elements in the video frame to be processed and information about the interactive relationship between the target animation elements;

[0223] Generate posture description information of the target animation element according to the target animation element and the interactive relationship information;

[0224] Generate the Foley sound effect of the target animation element according to the posture description information;

[0225] The Foley sound effect and the to-be-processed video frame are time-aligned so that the Foley sound effect is played synchronously when the to-be-processed video frame is played.

[0226] In an optional implementation, timing alignment is performed according to the Foley sound effect and the video frame to be processed so as to synchronously play the Foley sound effect when playing the video frame to be processed, including:

[0227] Obtain the video frame rate of the target video corresponding to the video frame to be processed, and the frame index of the video frame to be processed in the target video;

[0228] According to the video frame rate and frame index, determine the starting time point of the video frame to be processed in the target video;

[0229] According to the starting time point of the video frame to be processed, the Foley sound effect and the video frame to be processed are time-aligned so that the video frame to be processed and the Foley sound effect are played synchronously starting from the starting time point of the video frame to be processed.

[0230] In an optional embodiment, the method further comprises:

[0231] According to the sound effect peak of the Foley sound effect, the waveform of the Foley sound effect is intercepted to obtain an intercepted sound effect, and the starting time point of the intercepted sound effect is the time point of the sound effect peak;

[0232] According to the starting time point of the video frame to be processed, the Foley sound effect and the video frame to be processed are time-aligned to synchronously play the video frame to be processed and the Foley sound effect from the starting time point of the video frame to be processed, including:

[0233] According to the starting time point of the video frame to be processed, the intercepted sound effect and the video frame to be processed are time-aligned so that the video frame to be processed and the intercepted sound effect are played synchronously starting from the starting time point of the video frame to be processed.

[0234] In an optional embodiment, the method further comprises:

[0235] Extracting features of the Foley sound effect to obtain sound effect features of the Foley sound effect;

[0236] According to the sound effect characteristics of the Foley sound effect, a candidate sound effect matching the Foley sound effect is obtained;

[0237] Perform timing alignment based on the Foley sound effect and the video frame to be processed, so as to synchronously play the Foley sound effect when playing the video frame to be processed, including:

[0238] Determine a Foley sound effect to be played from the Foley sound effects and the candidate sound effects;

[0239] The Foley sound effect to be played and the video frame to be processed are time-aligned so that the Foley sound effect to be played is played synchronously when the video frame to be processed is played.

[0240] In an optional implementation, obtaining target animation elements in a video frame to be processed and information about interactive relationships between target animation elements includes:

[0241] A pre-trained deep learning model is used to perform target detection and posture detection on the video frames to be processed, and to determine the target animation elements and interactive relationship information.

[0242] In an optional implementation, a pre-trained deep learning model is used to perform target detection and posture detection on the video frame to be processed, and target animation elements and interactive relationship information are determined, including:

[0243] The feature extraction module in the deep learning model is used to extract visual features of the video frame to be processed to obtain the target visual features;

[0244] Using the first recognition module in the deep learning model, target detection is performed on the target visual features to obtain the target animation element;

[0245] The second recognition module in the deep learning model is used to perform posture detection on the target visual features according to the target animation elements to obtain the interactive relationship information.

[0246] In an optional implementation, the second recognition module in the deep learning model is used to perform posture detection on the target visual features according to the target animation elements to obtain interactive relationship information, including:

[0247] The second recognition module is used to perform posture detection on the target visual features and the visual features of at least one video frame continuous with the video frame to be processed according to the target animation element to obtain interactive relationship information.

[0248] In an optional implementation, the posture description information of the target animation element is generated according to the target animation element and the interactive relationship information, including:

[0249] Generate a first prompt word according to the target animation element and the interactive relationship information;

[0250] According to the first prompt word, a preset natural language model is used to generate posture description information.

[0251] In an optional implementation, generating a sound effect of a target animation element according to the posture description information includes:

[0252] Based on the posture description information, a pre-trained sound effect generation model is used to generate foley sound effects;

[0253] or,

[0254] Generate a second prompt word according to the posture change description information;

[0255] According to the second prompt word, a preset large model is used to generate a sound effect.

[0256] In an optional implementation, generating a sound effect of a target animation element according to the posture description information includes:

[0257] Generate a mime sound effect of a target animation element according to the input sound effect configuration information and posture description information, wherein the sound effect configuration information includes at least one of sound length, pitch, tone, and volume.

[0258] In an optional implementation, before performing timing alignment on the Foley sound effect and the video frame to be processed so as to synchronously play the Foley sound effect when playing the video frame to be processed, the method further includes:

[0259] Create a target audio track in the preset audio editor and load the Foley sound effect into the target audio track;

[0260] Edit the Foley sound effect according to the input sound effect editing operation to generate a new Foley sound effect;

[0261] Perform timing alignment based on the Foley sound effect and the video frame to be processed, so as to synchronously play the Foley sound effect when playing the video frame to be processed, including:

[0262] Timing alignment is performed according to the new Foley sound effect and the video frame to be processed, so that the new Foley sound effect is played synchronously when the video frame to be processed is played.

[0263] In an optional embodiment, the method further comprises:

[0264] Save Foley sound effects to your preset sound library.

[0265] In this embodiment, the processor executes machine-readable instructions to obtain target animation elements and interactive relationship information of the target animation elements in the video frame to be processed, generate posture description information of the target animation elements according to the target animation elements and the interactive relationship information, generate foley sound effects of the target animation elements according to the posture description information, and perform timing alignment according to the foley sound effects and the video frame to be processed, so as to play the foley sound effects synchronously when playing the video frame to be processed. This solution generates foley sound effects by identifying the content of the video frame, and automatically performs timing alignment according to the foley sound effects and the video frame to be processed, thereby achieving real-time generation and synchronization of sound effects, reducing labor costs, being able to adapt to different types of animations and diverse scene changes, and having good versatility and extensibility.

[0266] The embodiment of the present application further provides a computer-readable storage medium, on which a computer program is stored, and the computer program is executed by a processor to perform the following steps:

[0267] Obtain target animation elements in the video frame to be processed and information about the interactive relationship between the target animation elements;

[0268] Generate posture description information of the target animation element according to the target animation element and the interactive relationship information;

[0269] Generate the Foley sound effect of the target animation element according to the posture description information;

[0270] The Foley sound effect and the to-be-processed video frame are time-aligned so that the Foley sound effect is played synchronously when the to-be-processed video frame is played.

[0271] In an optional implementation, timing alignment is performed according to the Foley sound effect and the video frame to be processed so as to synchronously play the Foley sound effect when playing the video frame to be processed, including:

[0272] Obtain the video frame rate of the target video corresponding to the video frame to be processed, and the frame index of the video frame to be processed in the target video;

[0273] According to the video frame rate and frame index, determine the starting time point of the video frame to be processed in the target video;

[0274] According to the starting time point of the video frame to be processed, the Foley sound effect and the video frame to be processed are time-aligned so that the video frame to be processed and the Foley sound effect are played synchronously starting from the starting time point of the video frame to be processed.

[0275] In an optional embodiment, the method further comprises:

[0276] According to the sound effect peak of the Foley sound effect, the waveform of the Foley sound effect is intercepted to obtain an intercepted sound effect, and the starting time point of the intercepted sound effect is the time point of the sound effect peak;

[0277] According to the starting time point of the video frame to be processed, the Foley sound effect and the video frame to be processed are time-aligned to synchronously play the video frame to be processed and the Foley sound effect from the starting time point of the video frame to be processed, including:

[0278] According to the starting time point of the video frame to be processed, the intercepted sound effect and the video frame to be processed are time-aligned so that the video frame to be processed and the intercepted sound effect are played synchronously starting from the starting time point of the video frame to be processed.

[0279] In an optional embodiment, the method further comprises:

[0280] Extracting features of the Foley sound effect to obtain sound effect features of the Foley sound effect;

[0281] According to the sound effect characteristics of the Foley sound effect, a candidate sound effect matching the Foley sound effect is obtained;

[0282] Perform timing alignment based on the Foley sound effect and the video frame to be processed, so as to synchronously play the Foley sound effect when playing the video frame to be processed, including:

[0283] Determine a Foley sound effect to be played from the Foley sound effects and the candidate sound effects;

[0284] The Foley sound effect to be played and the video frame to be processed are time-aligned so that the Foley sound effect to be played is played synchronously when the video frame to be processed is played.

[0285] In an optional implementation, obtaining target animation elements in a video frame to be processed and information about interactive relationships between target animation elements includes:

[0286] A pre-trained deep learning model is used to perform target detection and posture detection on the video frames to be processed, and to determine the target animation elements and interactive relationship information.

[0287] In an optional implementation, a pre-trained deep learning model is used to perform target detection and posture detection on the video frame to be processed, and target animation elements and interactive relationship information are determined, including:

[0288] The feature extraction module in the deep learning model is used to extract visual features of the video frame to be processed to obtain the target visual features;

[0289] Using the first recognition module in the deep learning model, target detection is performed on the target visual features to obtain the target animation element;

[0290] The second recognition module in the deep learning model is used to perform posture detection on the target visual features according to the target animation elements to obtain the interactive relationship information.

[0291] In an optional implementation, the second recognition module in the deep learning model is used to perform posture detection on the target visual features according to the target animation elements to obtain interactive relationship information, including:

[0292] The second recognition module is used to perform posture detection on the target visual features and the visual features of at least one video frame continuous with the video frame to be processed according to the target animation element to obtain interactive relationship information.

[0293] In an optional implementation, the posture description information of the target animation element is generated according to the target animation element and the interactive relationship information, including:

[0294] Generate a first prompt word according to the target animation element and the interactive relationship information;

[0295] According to the first prompt word, a preset natural language model is used to generate posture description information.

[0296] In an optional implementation, generating a sound effect of a target animation element according to the posture description information includes:

[0297] Based on the posture description information, a pre-trained sound effect generation model is used to generate foley sound effects;

[0298] or,

[0299] Generate a second prompt word according to the posture change description information;

[0300] According to the second prompt word, a preset large model is used to generate a sound effect.

[0301] In an optional implementation, generating a sound effect of a target animation element according to the posture description information includes:

[0302] Generate a mime sound effect of a target animation element according to the input sound effect configuration information and posture description information, wherein the sound effect configuration information includes at least one of sound length, pitch, tone, and volume.

[0303] In an optional implementation, before performing timing alignment on the Foley sound effect and the video frame to be processed so as to synchronously play the Foley sound effect when playing the video frame to be processed, the method further includes:

[0304] Create a target audio track in the preset audio editor and load the Foley sound effect into the target audio track;

[0305] Edit the Foley sound effect according to the input sound effect editing operation to generate a new Foley sound effect;

[0306] Perform timing alignment based on the Foley sound effect and the video frame to be processed, so as to synchronously play the Foley sound effect when playing the video frame to be processed, including:

[0307] Timing alignment is performed according to the new Foley sound effect and the video frame to be processed, so that the new Foley sound effect is played synchronously when the video frame to be processed is played.

[0308] In an optional embodiment, the method further comprises:

[0309] Save Foley sound effects to your preset sound library.

[0310] In this embodiment, the computer program is executed by the processor when it is running to obtain the target animation element in the video frame to be processed and the interactive relationship information of the target animation element, generate the posture description information of the target animation element according to the target animation element and the interactive relationship information, generate the foley sound effect of the target animation element according to the posture description information, and perform time alignment according to the foley sound effect and the video frame to be processed, so as to play the foley sound effect synchronously when playing the video frame to be processed. This solution generates foley sound effects by identifying the content of the video frame, and automatically performs time alignment according to the foley sound effects and the video frame to be processed, thereby realizing real-time generation and synchronization of sound effects, reducing labor costs, being able to adapt to different types of animations and diverse scene changes, and having good versatility and extensibility.

[0311] In the embodiment of the present application, the computer program can also execute other machine-readable instructions when run by the processor to execute other methods described in the embodiment. For the specific execution method steps and principles, please refer to the description of the embodiment, which will not be repeated here.

[0312] In the embodiments provided in the present application, it should be understood that the disclosed devices and methods can be implemented in other ways. The device embodiments described above are merely schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some communication interfaces, and the indirect coupling or communication connection of the devices or units can be electrical, mechanical or other forms.

[0313] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0314] In addition, each functional unit in the embodiments provided in the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit.

[0315] If the functions are implemented in the form of software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art or the part of the technical solution, can be embodied in the form of a software product, which is stored in a storage medium and includes several instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the methods described in each embodiment of the present application. The aforementioned storage medium includes: various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk.

[0316] It should be noted that similar numbers and letters represent similar items in the following figures. Therefore, once an item is defined in one figure, it does not need to be further defined and explained in subsequent figures. In addition, the terms "first", "second", "third", etc. are only used to distinguish the description and are not to be understood as indicating or implying relative importance.

[0317] Finally, it should be noted that the above-described embodiments are only specific implementation methods of the present application, which are used to illustrate the technical solutions of the present application, rather than to limit them. The protection scope of the present application is not limited thereto. Although the present application is described in detail with reference to the above-mentioned embodiments, ordinary technicians in the field should understand that any technician familiar with the technical field can still modify the technical solutions recorded in the above-mentioned embodiments within the technical scope disclosed in the present application, or can easily think of changes, or make equivalent replacements for some of the technical features therein; and these modifications, changes or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present application. They should all be included in the protection scope of the present application. Therefore, the protection scope of the present application shall be based on the protection scope of the claims.

Claims

1. A method for generating a sound effect, characterized in that: include: Obtain target animation elements in the video frame to be processed and information on interactive relationships between the target animation elements; Generating posture description information of the target animation element according to the target animation element and the interactive relationship information; Generating a sound effect of the target animation element according to the posture description information; Timing alignment is performed based on the Foley sound effect and the video frame to be processed, so that the Foley sound effect is played synchronously when the video frame to be processed is played.

2. The method according to claim 1, characterized in that The step of performing time alignment according to the Foley sound effect and the to-be-processed video frame so as to synchronously play the Foley sound effect when playing the to-be-processed video frame includes: Obtaining a video frame rate of a target video corresponding to the to-be-processed video frame, and a frame index of the to-be-processed video frame in the target video; Determine the starting time point of the to-be-processed video frame in the target video according to the video frame rate and the frame index; According to the starting time point of the video frame to be processed, the foley sound effect and the video frame to be processed are time-aligned so that the video frame to be processed and the foley sound effect are synchronously played starting from the starting time point of the video frame to be processed.

3. The method according to claim 2, characterized in that The method further comprises: According to the sound effect peak value of the Foley sound effect, the waveform of the Foley sound effect is intercepted to obtain an intercepted sound effect, wherein the starting time point of the intercepted sound effect is the time point at which the sound effect peak value is located; The step of performing time alignment on the Foley sound effect and the video frame to be processed according to the starting time point of the video frame to be processed, so as to synchronously play the video frame to be processed and the Foley sound effect from the starting time point of the video frame to be processed, comprises: According to the starting time point of the video frame to be processed, the intercepted sound effect and the video frame to be processed are time-aligned to synchronously play the video frame to be processed and the intercepted sound effect starting from the starting time point of the video frame to be processed.

4. The method according to claim 1, characterized in that The method further comprises: Extracting features of the Foley sound effect to obtain sound effect features of the Foley sound effect; Acquire a candidate sound effect that matches the Foley sound effect according to the sound effect feature of the Foley sound effect; The step of performing time alignment according to the Foley sound effect and the to-be-processed video frame so as to synchronously play the Foley sound effect when playing the to-be-processed video frame includes: Determine a Foley sound effect to be played from the Foley sound effects and the candidate sound effects; The Foley sound effect to be played and the video frame to be processed are time-aligned so that the Foley sound effect to be played is played synchronously when the video frame to be processed is played.

5. The method according to claim 1, characterized in that The step of obtaining target animation elements in the video frame to be processed and information on the interactive relationship between the target animation elements includes: A pre-trained deep learning model is used to perform target detection and posture detection on the video frame to be processed to determine the target animation element and the interactive relationship information.

6. The method according to claim 5, characterized in that The pre-trained deep learning model is used to perform target detection and posture detection on the video frame to be processed, and the target animation element and the interactive relationship information are determined, including: Using the feature extraction module in the deep learning model, extracting visual features from the video frame to be processed to obtain target visual features; Using the first recognition module in the deep learning model, perform target detection on the target visual features to obtain the target animation element; The second recognition module in the deep learning model is used to perform posture detection on the target visual features according to the target animation elements to obtain the interactive relationship information.

7. The method according to claim 6, characterized in that The adopting the second recognition module in the deep learning model to perform posture detection on the target visual feature according to the target animation element to obtain the interactive relationship information includes: The second recognition module is used to perform posture detection on the target visual features and the visual features of at least one video frame continuous with the video frame to be processed according to the target animation element to obtain the interactive relationship information.

8. The method according to claim 1, characterized in that The step of generating the posture description information of the target animation element according to the target animation element and the interactive relationship information includes: Generate a first prompt word according to the target animation element and the interactive relationship information; The posture description information is generated according to the first prompt word using a preset natural language model.

9. The method according to claim 1, characterized in that: Generating the mime sound effect of the target animation element according to the posture description information includes: According to the posture description information, a pre-trained sound effect generation model is used to generate the foley sound effect; or, generating a second prompt word according to the posture change description information; According to the second prompt word, the imitation sound effect is generated using a preset large model.

10. The method according to claim 1, characterized in that Generating the mime sound effect of the target animation element according to the posture description information includes: The mime sound effect of the target animation element is generated according to the input sound effect configuration information and the posture description information, wherein the sound effect configuration information includes at least one of sound length, pitch, tone, and volume.

11. The method according to any one of claims 1 to 10, characterized in that: Before performing time alignment according to the Foley sound effect and the to-be-processed video frame so as to synchronously play the Foley sound effect when playing the to-be-processed video frame, the method further comprises: Creating a target audio track in a preset audio editor, and loading the Foley sound effect into the target audio track; Editing the Foley sound effect according to the input sound effect editing operation to generate a new Foley sound effect; The step of performing time alignment according to the Foley sound effect and the to-be-processed video frame so as to synchronously play the Foley sound effect when playing the to-be-processed video frame includes: Timing alignment is performed based on the new foley sound effect and the video frame to be processed, so that the new foley sound effect is played synchronously when the video frame to be processed is played.

12. The method according to claim 1, characterized in that The method further comprises: The foley sound effect is stored in a preset sound effect library.

13. A device for generating a sound effect of imitation sound, characterized in that: include: An acquisition module, used to acquire target animation elements in a video frame to be processed and information on interactive relationships between the target animation elements; A generating module, used for generating posture description information of the target animation element according to the target animation element and the interactive relationship information; The generating module is further used to generate the foley sound effect of the target animation element according to the posture description information; A processing module is used to perform time alignment according to the foley sound effect and the video frame to be processed, so as to synchronously play the foley sound effect when playing the video frame to be processed.

14. An electronic device, characterized in that: include: A processor, a memory and a bus, wherein the memory stores machine-readable instructions executable by the processor, and when the electronic device is running, the processor and the memory communicate via the bus, and the processor executes the machine-readable instructions to perform the method described in any one of claims 1 to 12.

15. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the method according to any one of claims 1 to 12 is executed.

Images