Audio processing method, device, processing equipment and storage medium

Abstract

The present invention provides an audio processing method, device, processing equipment and storage medium, and relates to the field of computer technology. The method comprises: performing picture recognition on multiple video frames in a video to be processed to obtain picture contents of multiple video frames, wherein the picture contents of each video frame include: picture elements of each video frame; obtaining audio corresponding to the picture elements from a preset material library as target audio corresponding to each video frame according to the picture elements of each video frame; and adding target audio corresponding to multiple video frames to multiple video frames. Multiple video frames are recognized to obtain the picture elements of each video frame, and the audio corresponding to the picture elements in the preset material library is used as the target audio corresponding to each video frame, wherein the target audio matches the picture elements of the video frame, and the target audio corresponding to multiple video frames is added to multiple video frames, so that the added target audio matches the multiple video frames better, and the target audio added to the multiple video frames is also better.

Description

Technical Field

[0001] The present invention relates to the field of computer technology, and in particular to an audio processing method, apparatus, processing device and storage medium. Background Art

[0002] With the development of science and technology, videos can be captured through various smart devices, such as cameras, mobile phones, etc. The audio corresponding to some captured videos is not noisy or there is sound missing. Therefore, adding audio to video has also become a research hotspot.

[0003] In the related art, when it is necessary to add audio to a video, template music is manually added to the video in a video editing application.

[0004] However, in the related art, the manually added template music has a weak correlation with the video, the audio effect added to the video is not good, and unnecessary human resources are wasted. Summary of the invention

[0005] The purpose of the present invention is to provide an audio processing method, apparatus, processing device and storage medium to address the deficiencies in the above-mentioned prior art, so as to solve the problem in the related art that the manually added template music has a weak correlation with the video, the audio effect added to the video is poor, and unnecessary human resources are wasted.

[0006] To achieve the above purpose, the technical solution adopted by the embodiment of the present invention is as follows:

[0007] In a first aspect, an embodiment of the present invention provides an audio processing method, the method comprising:

[0008] Performing picture recognition on multiple video frames in the video to be processed to obtain picture contents of the multiple video frames, wherein the picture contents of each video frame include: picture elements of each video frame;

[0009] According to the picture elements of each video frame, the audio corresponding to the picture elements is acquired from a preset material library as the target audio corresponding to each video frame;

[0010] Target audio corresponding to the multiple video frames is added to the multiple video frames.

[0011] Optionally, the picture content of each video frame further includes: position information of the picture element, and before adding the target audio corresponding to the multiple video frames to the multiple video frames, the method further includes:

[0012] According to the position information of the picture elements in at least two consecutive video frames, the sound playback parameters of the audio corresponding to the at least two consecutive video frames are adjusted to obtain the target audio corresponding to the at least two consecutive video frames.

[0013] Optionally, adjusting the sound playback parameters of the audio corresponding to the at least two consecutive video frames according to the position information of the picture elements in the at least two consecutive video frames to obtain the target audio corresponding to the at least two consecutive video frames includes:

[0014] Determining a moving direction of the picture element according to position information of the picture element in the at least two consecutive video frames;

[0015] According to the movement direction, sound playback parameters of the audio corresponding to the at least two consecutive video frames are adjusted to obtain target audio corresponding to the at least two consecutive video frames.

[0016] Optionally, before performing picture recognition on a plurality of video frames in the video to be processed, the method further includes:

[0017] Dividing the video to be processed into a plurality of time periods, each time period having a corresponding plurality of initial video frames;

[0018] A video frame is determined from the multiple initial video frames in each time period to obtain the multiple video frames.

[0019] Optionally, the method further includes:

[0020] Calculating the moving speed of the picture element in each time period according to the multiple initial video frames in each time period;

[0021] The step of adjusting the sound playback parameters of the audio corresponding to the at least two consecutive video frames according to the motion direction to obtain the target audio corresponding to the at least two consecutive video frames includes:

[0022] According to the motion direction and the moving speed of the picture element, the sound playback parameters of the audio corresponding to the at least two consecutive video frames are adjusted to obtain the target audio corresponding to the at least two consecutive video frames.

[0023] Optionally, adjusting the sound playback parameters of the audio corresponding to the at least two consecutive video frames according to the motion direction and the moving speed of the picture element to obtain the target audio corresponding to the at least two consecutive video frames includes:

[0024] If the movement direction is: a first preset direction, the audio channel of the audio corresponding to the at least two consecutive video frames is adjusted according to the moving speed of the picture element and the first preset direction to obtain the target audio corresponding to the at least two consecutive video frames.

[0025] Optionally, adjusting the sound playback parameters of the audio corresponding to the at least two consecutive video frames according to the motion direction and the moving speed of the picture element to obtain the target audio corresponding to the at least two consecutive video frames includes:

[0026] If the movement direction is: a second preset direction, the volume of the audio corresponding to the at least two consecutive video frames is adjusted according to the moving speed of the picture element and the second preset direction to obtain the target audio corresponding to the at least two consecutive video frames.

[0027] Optionally, if there are multiple screen elements of the target video frame in the multiple video frames, the screen content of the target video frame further includes: position information of the multiple elements in the target video frame;

[0028] Before adding the target audio corresponding to the multiple video frames to the multiple video frames, the method further includes:

[0029] Determining, according to the position information of the multiple elements, the screen proportions of the multiple elements in the target video frame;

[0030] According to the screen ratio, the volume of the audio corresponding to the target video frame is adjusted to obtain the target audio corresponding to the target video frame.

[0031] In a second aspect, an embodiment of the present invention further provides a sound effect processing device, the device comprising:

[0032] A picture recognition module is used to perform picture recognition on multiple video frames in the video to be processed to obtain picture contents of the multiple video frames, wherein the picture contents of each video frame include: picture elements of each video frame;

[0033] An acquisition module, used for acquiring, according to the picture elements of each video frame, the audio corresponding to the picture elements from a preset material library as the target audio corresponding to each video frame;

[0034] An adding module is used to add target audio corresponding to the multiple video frames to the multiple video frames.

[0035] Optionally, the picture content of each video frame further includes: position information of the picture element, and the device further includes:

[0036] The adjustment module is used to adjust the sound playback parameters of the audio corresponding to the at least two consecutive video frames according to the position information of the picture elements in the at least two consecutive video frames to obtain the target audio corresponding to the at least two consecutive video frames.

[0037] Optionally, the adjustment module is also used to determine the movement direction of the picture element based on the position information of the picture element in the at least two consecutive video frames; according to the movement direction, adjust the sound playback parameters of the audio corresponding to the at least two consecutive video frames to obtain the target audio corresponding to the at least two consecutive video frames.

[0038] Optionally, the device further comprises:

[0039] A division module, used for dividing the video to be processed into a plurality of time periods, each time period having a corresponding plurality of initial video frames;

[0040] The first determination module is used to determine a video frame from the multiple initial video frames in each time period to obtain the multiple video frames.

[0041] Optionally, the device further comprises:

[0042] A calculation module, used for calculating the moving speed of the picture element in each time period according to the multiple initial video frames in each time period;

[0043] The adjustment module is also used to adjust the sound playback parameters of the audio corresponding to the at least two consecutive video frames according to the motion direction and the moving speed of the picture element to obtain the target audio corresponding to the at least two consecutive video frames.

[0044] Optionally, the adjustment module is also used to adjust the audio channel corresponding to the at least two consecutive video frames according to the moving speed of the picture element and the first preset direction if the movement direction is: a first preset direction, so as to obtain the target audio corresponding to the at least two consecutive video frames.

[0045] Optionally, the adjustment module is also used to adjust the volume of the audio corresponding to the at least two consecutive video frames according to the moving speed of the picture element and the second preset direction if the movement direction is: a second preset direction, so as to obtain the target audio corresponding to the at least two consecutive video frames.

[0046] Optionally, if there are multiple screen elements of the target video frame in the multiple video frames, the screen content of the target video frame further includes: position information of the multiple elements in the target video frame;

[0047] The device also includes:

[0048] A second determination module, configured to determine, according to the position information of the multiple elements, the screen proportions of the multiple elements in the target video frame;

[0049] The first adjustment module is used to adjust the volume of the audio corresponding to the target video frame according to the screen ratio to obtain the target audio corresponding to the target video frame.

[0050] In a third aspect, an embodiment of the present invention further provides a processing device, comprising: a memory and a processor, wherein the memory stores a computer program executable by the processor, and when the processor executes the computer program, the method described in any one of the first aspects is implemented.

[0051] In a fourth aspect, an embodiment of the present invention further provides a computer-readable storage medium, on which a computer program is stored. When the computer program is read and executed, the method described in any one of the first aspects above is implemented.

[0052] The beneficial effects of the present invention are as follows: an embodiment of the present invention provides an audio processing method, the method comprising: performing screen recognition on multiple video frames in a video to be processed to obtain screen contents of multiple video frames, the screen contents of each video frame comprising: screen elements of each video frame; obtaining audio corresponding to the screen elements from a preset material library as target audio corresponding to each video frame according to the screen elements of each video frame; adding target audio corresponding to multiple video frames to multiple video frames. Multiple video frames of the video to be processed are recognized to obtain the screen elements of each video frame, and the audio corresponding to the screen elements in the preset material library is used as the target audio corresponding to each video frame. The target audio matches the screen elements of the video frame, and the two are highly correlated. Adding target audio corresponding to multiple video frames to multiple video frames makes the added target audio more matched with the multiple video frames, and the target audio added to the multiple video frames is also better. BRIEF DESCRIPTION OF THE DRAWINGS

[0053] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings required for use in the embodiments are briefly introduced below. It should be understood that the following drawings only show certain embodiments of the present invention and therefore should not be regarded as limiting the scope. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without creative work.

[0054] Figure 1 A flowchart of an audio processing method provided by an embodiment of the present invention;

[0055] Figure 2 A flowchart of an audio processing method provided by an embodiment of the present invention;

[0056] Figure 3 A flowchart of an audio processing method provided by an embodiment of the present invention;

[0057] Figure 4 A flowchart of an audio processing method provided by an embodiment of the present invention;

[0058] Figure 5 A flowchart of an audio processing method provided by an embodiment of the present invention;

[0059] Figure 6 A schematic diagram of the structure of an audio processing device provided by an embodiment of the present invention;

[0060] Figure 7 A schematic diagram of the structure of a processing device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0061] In order to make the purpose, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments.

[0062] Therefore, the following detailed description of the embodiments of the present application provided in the accompanying drawings is not intended to limit the scope of the present application for which protection is sought, but merely represents selected embodiments of the present application. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in the field without creative work are within the scope of protection of the present application.

[0063] In the description of the present application, it should be noted that if the terms "upper", "lower", etc. appear, the orientation or position relationship indicated is based on the orientation or position relationship shown in the drawings, or is the orientation or position relationship in which the product of the application is usually placed when used. It is only for the convenience of describing the present application and simplifying the description, and does not indicate or imply that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation, and therefore cannot be understood as a limitation on the present application.

[0064] In addition, the terms "first", "second", etc. in the specification and claims of the present invention and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments of the present invention described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units that are clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0065] It should be noted that, in the absence of conflict, the features in the embodiments of the present application may be combined with each other.

[0066] In the related art, when audio needs to be added to a video, template music is manually added to the video in a video editing application. However, in the related art, the manually added template music has a weak correlation with the video, the audio effect added to the video is not good, and unnecessary human resources are wasted.

[0067] An embodiment of the present application provides an audio processing method, which includes: identifying multiple video frames of a video to be processed to obtain picture elements of each video frame, using the audio corresponding to the picture elements in a preset material library as the target audio corresponding to each video frame, the target audio matches the picture elements of the video frame, and the two are highly correlated, adding target audio corresponding to multiple video frames to multiple video frames, so that the added target audio is more matched with the multiple video frames, and the target audio added to the multiple video frames is also better.

[0068] The following is an explanation of an audio processing method provided in an embodiment of the present application.

[0069] Figure 1 A flowchart of an audio processing method provided by an embodiment of the present invention is shown in FIG. Figure 1 As shown, the method may include:

[0070] S101 , performing picture recognition on a plurality of video frames in a video to be processed to obtain picture contents of the plurality of video frames.

[0071] The picture content of each video frame may include: picture elements of each video frame.

[0072] In some implementations, the processing device may perform picture recognition on each of the multiple video frames to obtain picture content of each video frame, including picture elements of each video frame, where the picture elements may be objects in the picture.

[0073] For example, the picture elements may be birds, flowers, people, cats, fans, etc. in the picture. Different videos contain different picture elements, and the embodiments of the present application do not impose specific limitations on this.

[0074] It should be noted that the processing device can perform image recognition on each video frame simultaneously, or can perform image recognition on each video frame in sequence, or can perform image recognition on each video frame in other orders, and the embodiments of the present application do not impose specific restrictions on this.

[0075] S102: According to the picture elements of each video frame, the audio corresponding to the picture elements is acquired from a preset material library as the target audio corresponding to each video frame.

[0076] The preset material library includes a plurality of preset elements and preset audio corresponding to each preset element.

[0077] In some implementations, the processing device may search for a target preset element that matches the picture element of each video frame from a preset material library based on the picture element of each video frame, and use the preset audio corresponding to the target preset element as the target audio corresponding to each video frame.

[0078] In addition, if the picture elements of each video frame are the same, the target audio corresponding to each video frame is also the same, that is, multiple video frames have one audio. If the picture elements of some video frames are different, the target audio corresponding to the video frames are also different, that is, multiple video frames have multiple audios.

[0079] S103: Add target audio corresponding to the multiple video frames to the multiple video frames.

[0080] In an embodiment of the present application, target audio corresponding to multiple video frames is added to multiple video frames, and the multiple video frames are associated with the target audio corresponding to the multiple video frames. When the multiple video frames are played through a display device, the corresponding target audio can be played through a speaker.

[0081] It should be noted that the multiple video frames include multiple first video frames and multiple second video frames in a chronological order. If the picture elements of the multiple first video frames are different from the picture elements of the multiple second video frames, the target audio played when playing the multiple first video frames is different from the target audio played when playing the multiple first video frames.

[0082] In summary, an embodiment of the present invention provides an audio processing method, which includes: performing screen recognition on multiple video frames in a video to be processed to obtain screen contents of multiple video frames, wherein the screen contents of each video frame include: screen elements of each video frame; obtaining audio corresponding to the screen elements from a preset material library as target audio corresponding to each video frame according to the screen elements of each video frame; and adding target audio corresponding to multiple video frames to multiple video frames. Multiple video frames of the video to be processed are recognized to obtain the screen elements of each video frame, and the audio corresponding to the screen elements in the preset material library is used as the target audio corresponding to each video frame. The target audio matches the screen elements of the video frame, and the two are highly correlated. Adding target audio corresponding to multiple video frames to multiple video frames makes the added target audio better match the multiple video frames, and the target audio added to the multiple video frames is also better.

[0083] Before the process of performing picture recognition on the multiple video frames in the to-be-processed video in S101, the method may further include: the processing device may obtain the to-be-processed video, and the to-be-processed video may have or have the original audio;

[0084] In some embodiments, if the video to be processed has original audio, the processing can determine whether the sound volume of the original audio is less than a preset sound threshold. If so, it is determined that the original audio is unavailable; if the sound volume of the original audio is greater than or equal to the preset sound threshold, it is determined whether the original audio is clear. If the judgment result indicates that the original audio is not clear, it is determined that the original audio is unavailable. When the original audio is unavailable, the above-mentioned process S101 to S103 can be used to add audio to multiple video frames in the video to be processed.

[0085] In addition, if the video to be processed does not have the original audio or the original audio effect is not good, the above-mentioned process S101 to S103 can also be used to add audio to multiple video frames in the video to be processed. The embodiment of the present application does not impose specific restrictions on this.

[0086] It should be noted that the preset sound threshold can be 15 decibels or 18 decibels, and can also be set according to empirical values ​​or actual needs. The embodiments of the present application do not impose specific restrictions on this.

[0087] Optionally, the picture content of each video frame may further include: position information of picture elements.

[0088] Before the process of adding target audio corresponding to the multiple video frames to the multiple video frames in S103, the method may further include:

[0089] According to the position information of the picture elements in at least two consecutive video frames, the sound playing parameters of the audio corresponding to the at least two consecutive video frames are adjusted to obtain the target audio corresponding to the at least two consecutive video frames.

[0090] The position information of the picture element may be coordinate information. At least two video frames may include the same picture element.

[0091] In some implementations, the processing device establishes a standard coordinate axis for each video frame, determines the position of the element center of the picture element in the standard coordinate axis, and uses the position of the element center of the picture element in the standard coordinate axis as the position information of the picture element.

[0092] In an embodiment of the present application, a processing device adjusts the audio channels and / or audio volumes corresponding to at least two consecutive video frames according to position information of picture elements in at least two video frames to obtain target audio corresponding to at least two consecutive video frames.

[0093] Optional, Figure 2 A flowchart of an audio processing method provided by an embodiment of the present invention is shown in FIG. Figure 2 As shown, the process of adjusting the sound playback parameters of the audio corresponding to at least two consecutive video frames according to the position information of the picture elements in at least two consecutive video frames to obtain the target audio corresponding to at least two consecutive video frames may include:

[0094] S201: Determine a moving direction of a picture element according to position information of the picture element in at least two consecutive video frames.

[0095] The at least two consecutive video frames have a sequence, and the moving direction of the picture element can be determined according to the change information of the position information of the picture element in the at least two consecutive video frames.

[0096] Optionally, the movement direction of the screen element may be: a first preset direction and / or a second preset direction, specifically, the screen element moves only in the first preset direction, or the screen element moves only in the second preset direction, or the screen element moves in both the first preset direction and the second direction. The first preset direction and the second preset direction are different movement directions.

[0097] S202: Adjust the sound playback parameters of the audio corresponding to at least two consecutive video frames according to the movement direction to obtain the target audio corresponding to the at least two consecutive video frames.

[0098] In some embodiments, the sound playback parameters include multiple types of parameters. According to the direction of movement, at least one target playback parameter to be adjusted is determined from the multiple types of parameters, and then at least one target playback parameter of the audio corresponding to at least two consecutive video frames is adjusted to obtain the target audio corresponding to at least two consecutive video frames.

[0099] To sum up, according to the movement direction of the picture elements, the sound playback parameters of the audio corresponding to at least two consecutive video frames are adjusted to obtain the target audio corresponding to at least two consecutive video frames, so that the target audio is more consistent with the movement direction of the picture elements, thereby improving the playback effect of the target audio.

[0100] Optional, Figure 3 A flowchart of an audio processing method provided by an embodiment of the present invention is shown in FIG. Figure 3 As shown, before the process of performing picture recognition on the multiple video frames in the to-be-processed video in the above S101, the method may further include:

[0101] S301: Divide the video to be processed into multiple time periods.

[0102] Each time period has a corresponding plurality of initial video frames.

[0103] In some embodiments, the video to be processed includes multiple initial video frames, and the video to be processed also has a corresponding duration. The video to be processed is divided into multiple time periods according to a preset duration, and the duration of each time period is the preset duration. Then, each time period has corresponding multiple initial video frames, and the multiple initial video frames have a chronological order.

[0104] Optionally, each time period may be 1 second or 1.5 seconds, and may be set according to actual needs. The embodiments of the present application do not impose any specific restrictions on this.

[0105] S302: Determine a video frame from a plurality of initial video frames in each time period to obtain a plurality of video frames.

[0106] The multiple initial video frames in each time period have a chronological order.

[0107] In some implementations, the processing device may determine the nth initial video frame among the multiple initial videos in each time period, and use the nth initial video frame as a video frame, so that multiple time periods correspond to multiple video frames.

[0108] For example, n can be 1, 2, or other values, and the setting of the value cannot exceed the total number of initial video frames in a time period.

[0109] Optional, Figure 4A flowchart of an audio processing method provided by an embodiment of the present invention is shown in FIG. Figure 4 As shown, the method may also include:

[0110] S401. Calculate the moving speed of the screen elements in each time period according to a plurality of initial video frames in each time period.

[0111] In some embodiments, the picture elements in multiple initial video frames of each time period and the positions of the picture elements are identified, the distance moved by the picture elements in each time period is determined based on the positions of the picture elements, and the ratio of the distance to the time period is used as the movement speed of the picture elements in each time period.

[0112] For example, a time period may be one second, and the moving speed of the screen element may be recorded from xx hours and xx seconds to xx hours and xx seconds, and the average speed of the screen element per second may be aa, bb, cc, dd, ..., nn, respectively.

[0113] The process of adjusting the sound playback parameters of the audio corresponding to at least two consecutive video frames according to the motion direction in S202 to obtain the target audio corresponding to at least two consecutive video frames may include:

[0114] S402: adjusting sound playback parameters of audio corresponding to at least two consecutive video frames according to the motion direction and the moving speed of the picture elements to obtain target audio corresponding to at least two consecutive video frames.

[0115] The adjustment of the sound playing parameters may include: adjusting the parameter values ​​of the sound playing parameters, and adjusting the changing speed of the sound playing parameters.

[0116] In an embodiment of the present application, the parameter values ​​of the sound playback parameters of the audio corresponding to at least two consecutive video frames can be adjusted according to the direction of movement, and the changing speed of the parameter values ​​of the sound playback parameters of the audio corresponding to at least two consecutive video frames can be adjusted according to the moving speed of the picture elements to obtain the target audio corresponding to at least two consecutive video frames.

[0117] To sum up, according to the movement direction and moving speed of the picture elements, the sound playback parameters of the audio corresponding to at least two consecutive video frames are adjusted to obtain the target audio corresponding to at least two consecutive video frames, so that the target audio is more consistent with the movement direction and movement speed of the picture elements, thereby improving the playback effect of the target audio.

[0118] The process of adjusting the sound playback parameters of the audio corresponding to at least two consecutive video frames according to the motion direction and the moving speed of the picture element in S402 to obtain the target audio corresponding to at least two consecutive video frames may include:

[0119] If the motion direction is: a first preset direction, the audio channels of the audio corresponding to at least two consecutive video frames are adjusted according to the moving speed of the picture element and the first preset direction to obtain the target audio corresponding to the at least two consecutive video frames.

[0120] Among them, the first preset direction can be a horizontal movement direction, for example, the screen element can move from left to right, or from right to left, or from the upper left to the lower right, or from the upper right to the lower left, etc.

[0121] In some embodiments, parameter values ​​of the left channel and the right channel of the audio corresponding to at least two consecutive video frames are adjusted according to a first preset direction, and the changing speed of the parameter values ​​of the left channel and the right channel is adjusted according to the moving speed of the picture elements to obtain the target audio corresponding to at least two consecutive video frames.

[0122] For example, the screen element may be a bird, and the motion direction and the moving speed of the screen element may be expressed as: from xx hours xx seconds to xx hours xx seconds, the bird moves from left to right, and the average speeds per second are aa, bb, cc, dd...nn respectively.

[0123] Optionally, the process of adjusting the sound playback parameters of the audio corresponding to at least two consecutive video frames according to the motion direction and the moving speed of the picture element in the above S402 to obtain the target audio corresponding to at least two consecutive video frames may include:

[0124] If the motion direction is: the second preset direction, the volume of the audio corresponding to at least two consecutive video frames is adjusted according to the moving speed of the picture element to obtain the target audio corresponding to the at least two consecutive video frames.

[0125] The second preset direction may be moving from near to far, or moving from far to near.

[0126] It should be noted that if the area of ​​the screen element is large, it means it is near, and if the area of ​​the screen element is small, it means it is far; if the area of ​​the screen element in at least two video frames changes from large to small, it moves from near to far; if the area of ​​the screen element in at least two video frames changes from small to large, it moves from far to far. The area of ​​the screen element can be determined according to the position of the screen element in the coordinates.

[0127] In some embodiments, the parameter value of the volume of the audio corresponding to at least two consecutive video frames is adjusted according to a second preset direction, and the changing speed of the parameter value of the volume is adjusted according to the moving speed of the picture elements to obtain the target audio corresponding to at least two consecutive video frames.

[0128] For example, the picture element may be a bird, and the motion direction and the moving speed of the picture element may be expressed as: from xx hours xx seconds to xx hours xx seconds, the bird moves from near to far, and the average speeds per second are aa, bb, cc, dd...nn respectively.

[0129] Optionally, if there are multiple screen elements of the target video frame in the multiple video frames, the screen content of the target video frame further includes: position information of the multiple elements in the target video frame.

[0130] Figure 5 A flowchart of an audio processing method provided by an embodiment of the present invention is shown in FIG. Figure 5 As shown, before the process of adding target audio corresponding to the multiple video frames to the multiple video frames in the above S104, the method may further include:

[0131] S501: Determine, according to position information of multiple elements, screen proportions of the multiple elements in a target video frame.

[0132] In some implementations, the position information of each element is used to calculate the area of ​​each element in the target video frame, and the ratio of each area to the total area of ​​the target video frame is calculated to obtain the screen ratio of each element in the target video frame.

[0133] S502: adjusting the volume of the audio corresponding to the target video frame according to the screen ratio to obtain the target audio corresponding to the target video frame.

[0134] Among them, the screen ratio of each element in the target video frame can be called the first screen ratio corresponding to each element.

[0135] In some implementations, the first screen ratio corresponding to each element is sorted to obtain a sorting result; the element with the largest first screen ratio is taken as the target element, and the screen ratio of each other element in the target video frame to the target element is calculated to obtain the second screen ratio corresponding to each other element. The other elements are elements other than the target element in the target video frame.

[0136] In an embodiment of the present application, the volume of the audio corresponding to the target element is adjusted to: the maximum volume multiplied by 100%, and the volume of the audio corresponding to other elements is adjusted according to the second screen ratio corresponding to other elements. For example, the volume of the audio corresponding to each other element is adjusted to: the maximum volume multiplied by the second screen ratio; the target audio corresponding to the target video frame may include: the superposition of the audio corresponding to each element, and the audio is the audio after the volume is adjusted.

[0137] It should be noted that the way of sorting the proportion of each element in the target video frame can be from small to large or from large to small, and the embodiment of the present application does not impose any specific limitation on this.

[0138] In summary, an embodiment of the present invention provides an audio processing method, which includes: performing screen recognition on multiple video frames in a video to be processed to obtain screen contents of multiple video frames, wherein the screen contents of each video frame include: screen elements of each video frame; obtaining audio corresponding to the screen elements from a preset material library as target audio corresponding to each video frame according to the screen elements of each video frame; and adding target audio corresponding to multiple video frames to multiple video frames. Multiple video frames of the video to be processed are recognized to obtain the screen elements of each video frame, and the audio corresponding to the screen elements in the preset material library is used as the target audio corresponding to each video frame. The target audio matches the screen elements of the video frame, and the two are highly correlated. Adding target audio corresponding to multiple video frames to multiple video frames makes the added target audio better match the multiple video frames, and the target audio added to the multiple video frames is also better.

[0139] Furthermore, by adjusting the sound playback parameters of the audio according to the direction of motion and the speed of movement of the screen elements, the sound effects added to the video frames can be enhanced, thus increasing the user's sense of immersion.

[0140] The following describes an audio processing device, a processing equipment, a storage medium, etc. used to execute the audio processing method provided in this application. The specific implementation process and technical effects thereof can be found in the relevant contents of the above method, which will not be described in detail below.

[0141] Figure 6 A schematic diagram of the structure of an audio processing device provided by an embodiment of the present invention is shown in FIG. Figure 6 As shown, the device may include:

[0142] The picture recognition module 601 is used to perform picture recognition on multiple video frames in the video to be processed to obtain picture contents of the multiple video frames, where the picture contents of each video frame include: picture elements of each video frame;

[0143] The acquisition module 602 is used to acquire, according to the picture elements of each video frame, the audio corresponding to the picture element from a preset material library as the target audio corresponding to each video frame;

[0144] The adding module 603 is configured to add target audio corresponding to the multiple video frames to the multiple video frames.

[0145] Optionally, the picture content of each video frame further includes: position information of the picture element, and the device further includes:

[0146] The adjustment module is used to adjust the sound playback parameters of the audio corresponding to the at least two consecutive video frames according to the position information of the picture elements in the at least two consecutive video frames to obtain the target audio corresponding to the at least two consecutive video frames.

[0147] Optionally, the adjustment module is also used to determine the movement direction of the picture element based on the position information of the picture element in the at least two consecutive video frames; according to the movement direction, adjust the sound playback parameters of the audio corresponding to the at least two consecutive video frames to obtain the target audio corresponding to the at least two consecutive video frames.

[0148] Optionally, the device further comprises:

[0149] A division module, used for dividing the video to be processed into a plurality of time periods, each time period having a corresponding plurality of initial video frames;

[0150] The first determination module is used to determine a video frame from the multiple initial video frames in each time period to obtain the multiple video frames.

[0151] Optionally, the device further comprises:

[0152] A calculation module, used for calculating the moving speed of the picture element in each time period according to the multiple initial video frames in each time period;

[0153] The adjustment module is also used to adjust the sound playback parameters of the audio corresponding to the at least two consecutive video frames according to the motion direction and the moving speed of the picture element to obtain the target audio corresponding to the at least two consecutive video frames.

[0154] Optionally, the adjustment module is also used to adjust the audio channel corresponding to the at least two consecutive video frames according to the moving speed of the picture element and the first preset direction if the movement direction is: a first preset direction, so as to obtain the target audio corresponding to the at least two consecutive video frames.

[0155] Optionally, the adjustment module is also used to adjust the volume of the audio corresponding to the at least two consecutive video frames according to the moving speed of the picture element and the second preset direction if the movement direction is: a second preset direction, so as to obtain the target audio corresponding to the at least two consecutive video frames.

[0156] Optionally, if there are multiple screen elements of the target video frame in the multiple video frames, the screen content of the target video frame further includes: position information of the multiple elements in the target video frame;

[0157] The device also includes:

[0158] A second determination module, configured to determine, according to the position information of the multiple elements, the screen proportions of the multiple elements in the target video frame;

[0159] The first adjustment module is used to adjust the volume of the audio corresponding to the target video frame according to the screen ratio to obtain the target audio corresponding to the target video frame.

[0160] The above-mentioned device is used to execute the method provided by the aforementioned embodiment, and its implementation principle and technical effect are similar, which will not be repeated here.

[0161] The above modules may be one or more integrated circuits configured to implement the above methods, such as one or more application specific integrated circuits (ASIC), or one or more digital singnal processors (DSP), or one or more field programmable gate arrays (FPGA). For another example, when a module is implemented in the form of a processing element scheduling program code, the processing element may be a general-purpose processor, such as a central processing unit (CPU) or other processor that can call program code. For another example, these modules may be integrated together and implemented in the form of a system-on-a-chip (SOC).

[0162] Figure 7 A schematic diagram of a processing device provided by an embodiment of the present invention is shown in FIG. Figure 7 As shown, the processing device includes: a processor 701 and a memory 702 .

[0163] The memory 702 is used to store programs, and the processor 701 calls the programs stored in the memory 702 to execute the above method embodiment. The specific implementation method and technical effect are similar and will not be repeated here.

[0164] Optionally, the present invention also provides a program product, such as a computer-readable storage medium, comprising a program, which is used to execute the above method embodiment when executed by a processor.

[0165] In the several embodiments provided by the present invention, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0166] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0167] In addition, each functional unit in each embodiment of the present invention may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of hardware plus software functional units.

[0168] The above-mentioned integrated unit implemented in the form of a software functional unit can be stored in a computer-readable storage medium. The above-mentioned software functional unit is stored in a storage medium, including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) or a processor (English: processor) to perform some steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (English: Read-Only Memory, abbreviated: ROM), random access memory (English: Random Access Memory, abbreviated: RAM), disk or optical disk and other media that can store program codes.

[0169] The above are only preferred embodiments of the present invention and are not intended to limit the present invention. For those skilled in the art, the present invention may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.

Claims

1. An audio processing method, characterized in that: The method comprises: Performing picture recognition on multiple video frames in the video to be processed to obtain picture contents of the multiple video frames, wherein the picture contents of each video frame include: picture elements of each video frame; According to the picture elements of each video frame, the audio corresponding to the picture elements is acquired from a preset material library as the target audio corresponding to each video frame; Adding target audio corresponding to the multiple video frames to the multiple video frames; If there are multiple screen elements of the target video frame in the multiple video frames, the screen content of the target video frame further includes: position information of the multiple elements in the target video frame; before adding the target audio corresponding to the multiple video frames to the multiple video frames, the method further includes: Determining, according to the position information of the multiple elements, the screen proportions of the multiple elements in the target video frame; According to the screen ratio, the volume of the audio corresponding to the target video frame is adjusted to obtain the target audio corresponding to the target video frame.

2. The method according to claim 1, characterized in that The picture content of each video frame further includes: position information of the picture element. Before adding the target audio corresponding to the multiple video frames to the multiple video frames, the method further includes: According to the position information of the picture elements in at least two consecutive video frames, the sound playback parameters of the audio corresponding to the at least two consecutive video frames are adjusted to obtain the target audio corresponding to the at least two consecutive video frames.

3. The method according to claim 2, characterized in that The step of adjusting the sound playback parameters of the audio corresponding to the at least two consecutive video frames according to the position information of the picture elements in the at least two consecutive video frames to obtain the target audio corresponding to the at least two consecutive video frames includes: Determining a moving direction of the picture element according to position information of the picture element in the at least two consecutive video frames; According to the movement direction, sound playback parameters of the audio corresponding to the at least two consecutive video frames are adjusted to obtain target audio corresponding to the at least two consecutive video frames.

4. The method according to claim 3, characterized in that Before performing picture recognition on a plurality of video frames in the video to be processed, the method further includes: Dividing the video to be processed into a plurality of time periods, each time period having a corresponding plurality of initial video frames; A video frame is determined from the multiple initial video frames in each time period to obtain the multiple video frames.

5. The method according to claim 4, characterized in that The method further comprises: Calculating the moving speed of the picture element in each time period according to the multiple initial video frames in each time period; The step of adjusting the sound playback parameters of the audio corresponding to the at least two consecutive video frames according to the motion direction to obtain the target audio corresponding to the at least two consecutive video frames includes: According to the motion direction and the moving speed of the picture element, the sound playback parameters of the audio corresponding to the at least two consecutive video frames are adjusted to obtain the target audio corresponding to the at least two consecutive video frames.

6. The method according to claim 5, characterized in that The step of adjusting the sound playback parameters of the audio corresponding to the at least two consecutive video frames according to the motion direction and the moving speed of the picture element to obtain the target audio corresponding to the at least two consecutive video frames includes: If the movement direction is: a first preset direction, the audio channel of the audio corresponding to the at least two consecutive video frames is adjusted according to the moving speed of the picture element and the first preset direction to obtain the target audio corresponding to the at least two consecutive video frames.

7. The method according to claim 5, characterized in that The step of adjusting the sound playback parameters of the audio corresponding to the at least two consecutive video frames according to the motion direction and the moving speed of the picture element to obtain the target audio corresponding to the at least two consecutive video frames includes: If the movement direction is: a second preset direction, the volume of the audio corresponding to the at least two consecutive video frames is adjusted according to the moving speed of the picture element and the second preset direction to obtain the target audio corresponding to the at least two consecutive video frames.

8. A sound effect processing device, characterized in that: The device comprises: A picture recognition module is used to perform picture recognition on multiple video frames in the video to be processed to obtain picture contents of the multiple video frames, wherein the picture contents of each video frame include: picture elements of each video frame; An acquisition module, used for acquiring, according to the picture elements of each video frame, the audio corresponding to the picture elements from a preset material library as the target audio corresponding to each video frame; An adding module, used for adding target audio corresponding to the multiple video frames to the multiple video frames; If there are multiple screen elements of the target video frame in the multiple video frames, the screen content of the target video frame further includes: position information of the multiple elements in the target video frame; The device also includes: A second determination module, configured to determine, according to the position information of the multiple elements, the screen proportions of the multiple elements in the target video frame; The first adjustment module is used to adjust the volume of the audio corresponding to the target video frame according to the screen ratio to obtain the target audio corresponding to the target video frame.

9. A processing device, characterized in that: include: A memory and a processor, wherein the memory stores a computer program executable by the processor, and the processor implements the method described in any one of claims 1 to 7 when executing the computer program.

10. A computer-readable storage medium, characterized in that: The storage medium stores a computer program, and when the computer program is read and executed, the method described in any one of claims 1 to 7 is implemented.

Images