Data processing method and related equipment

Abstract

The invention provides a data processing method and related equipment. The method comprises the following steps: determining a to-be-processed image and audio data corresponding to the to-be-processed image; based on the audio data, determining a target frequency band corresponding to the to-be-processed image in spectrum data corresponding to the audio data; performing smoothing processing on the target frequency band to obtain target data; and based on the target data, adding a picture effect to the to-be-processed image to obtain a target image.

Description

Technical Field

[0001] The present disclosure relates to the field of computer technologies, and in particular, to a data processing method and related devices. Background Art

[0002] In the field of multimedia data processing, images can be processed based on audio data.

[0003] However, the inventors of the present disclosure have found that in related technologies, after converting audio data into spectral data, images can be processed based on spectral information. However, when the spectral information is not further processed, the processing effect on images is poor. Summary of the Invention

[0004] The present disclosure provides a data processing method and related devices to solve or partially solve the above problems.

[0005] In a first aspect of the present disclosure, a data processing method is provided, including:

[0006] Determining a to-be-processed image and audio data corresponding to the to-be-processed image;

[0007] Based on the audio data, determining a target frequency band corresponding to the to-be-processed image in spectral data corresponding to the audio data;

[0008] Performing smoothing processing on the target frequency band to obtain target data;

[0009] Based on the target data, adding a picture effect to the to-be-processed image to obtain a target image.

[0010] In a second aspect of the present disclosure, a data processing device is provided, including:

[0011] A first determination module configured to: determine a to-be-processed image and audio data corresponding to the to-be-processed image;

[0012] A second determination module configured to: based on the audio data, determine a target frequency band corresponding to the to-be-processed image in spectral data corresponding to the audio data;

[0013] A smoothing module configured to: perform smoothing processing on the target frequency band to obtain target data;

[0014] An adding module configured to: based on the target data, add a picture effect to the to-be-processed image to obtain a target image.

[0015] In a third aspect of the present disclosure, a computer device is provided, including one or more processors, a memory; and one or more programs, wherein the one or more programs are stored in the memory and executed by the one or more processors, and the programs include instructions for executing the method according to the first aspect.

[0016] In a fourth aspect of the present disclosure, a non-volatile computer-readable storage medium containing a computer program is provided. When the computer program is executed by one or more processors, the processors are caused to execute the method according to the first aspect.

[0017] In a fifth aspect of the present disclosure, a computer program product is provided, including computer program instructions. When the computer program instructions run on a computer, the computer is caused to execute the method according to the first aspect.

[0018] Before adding a picture effect to the image to be processed based on the target frequency band, the data processing method and related devices provided in the embodiments of the present disclosure first smooth the target frequency band to obtain target data, and then add a picture effect to the image to be processed based on the target data to obtain a target image, so that the picture processing effect of the target image is better. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] In order to more clearly illustrate the technical solutions in the present disclosure or related technologies, the following will briefly introduce the drawings required for use in the embodiments or related technology descriptions. Obviously, the drawings in the following description are only the embodiments of the present disclosure. For those of ordinary skill in the art, other drawings can be obtained based on these drawings without creative efforts.

[0020] Figure 1 FIG. shows a schematic diagram of an exemplary system provided by the embodiments of the present disclosure.

[0021] Figure 2 FIG. shows a schematic diagram of an exemplary method provided by the embodiments of the present disclosure.

[0022] Figure 3A FIG. shows a schematic diagram of an exemplary page according to an embodiment of the present disclosure.

[0023] Figure 3B FIG. shows a schematic diagram of an exemplary target frequency band according to an embodiment of the present disclosure.

[0024] Figure 3C FIG. shows a schematic diagram of an exemplary smoothed spectrum according to an embodiment of the present disclosure.

[0025] Figure 3D FIG. shows a schematic diagram of an exemplary target image according to an embodiment of the present disclosure.

[0026] Figure 4 It shows a schematic flowchart of an exemplary method provided by an embodiment of the present disclosure.

[0027] Figure 5 It shows a schematic hardware structure diagram of an exemplary computer device provided by an embodiment of the present disclosure.

[0028] Figure 6 It shows a schematic diagram of an exemplary device provided by an embodiment of the present disclosure. Detailed implementation manners

[0029] To make the objectives, technical solutions, and advantages of the present disclosure more clear and understandable, the present disclosure will be further described in detail below in conjunction with specific embodiments and with reference to the accompanying drawings.

[0030] It should be noted that unless otherwise defined, the technical terms or scientific terms used in the embodiments of the present disclosure should have the ordinary meanings understood by those of ordinary skill in the art to which the present disclosure belongs. The "first", "second", and similar terms used in the embodiments of the present disclosure do not indicate any order, quantity, or importance, but are only used to distinguish different components. The terms such as "including" or "comprising" mean that the elements or objects appearing before this word cover the elements or objects listed after this word and their equivalents, without excluding other elements or objects. The terms such as "connected" or "coupled" are not limited to physical or mechanical connections, but may include electrical connections, whether direct or indirect. The terms such as "upper", "lower", "left", and "right" are only used to represent relative positional relationships, and when the absolute position of the object being described changes, the relative positional relationship may also change accordingly.

[0031] It can be understood that before using the technical solutions of the various embodiments of the present disclosure, the types, usage scopes, usage scenarios, etc. of the personal information involved will be informed to the user in an appropriate manner and the user's authorization will be obtained.

[0032] For example, when responding to a user's active request, a prompt message is sent to the user to clearly prompt the user that the operation requested by the user will require obtaining and using the user's personal information. Thus, the user can autonomously choose whether to provide personal information to software or hardware such as an electronic device, an application program, a server, or a storage medium that executes the operations of the technical solutions of the present disclosure according to the prompt message.

[0033] As an optional but non-limiting implementation, in response to receiving an active request from a user, the manner of sending a prompt message to the user can be, for example, in the form of a pop-up window, and the prompt message can be presented in text in the pop-up window. In addition, the pop-up window can also carry selection controls for the user to choose "agree" or "disagree" to provide personal information to the electronic device.

[0034] It can be understood that the above notification and user authorization acquisition process is only illustrative and does not limit the implementation of the present disclosure. Other methods that comply with relevant laws and regulations can also be applied to the implementation of the present disclosure.

[0035] Figure 1 FIG. shows a schematic diagram of an exemplary system 100 provided by an embodiment of the present disclosure.

[0036] As Figure 1 shown, the system 100 may include a terminal device 102, a server 106, and a database server 108. A medium (e.g., a network) for providing a communication link may be included between the terminal device 102, the server 106, and the database server 108. The network may include various connection types, such as wired, wireless communication links, or fiber optic cables, etc.

[0037] Exemplarily, various applications (APPs) or software may be installed on the terminal device 102, such as, for example, multimedia data editing applications or software, knowledge Q&A applications or software, life service applications or software, collaborative office applications or software, video conferencing applications or software, reading applications or software, video applications or software, social applications or software, payment applications or software, web browsers, and instant messaging tools, etc.

[0038] The terminal device 102 here may be hardware or software. When the terminal device 102 is hardware, it may be various electronic devices with a display screen, including but not limited to smart phones, tablet computers, e-book readers, MP3 players, laptop computers, and desktop computers (PCs), etc. When the terminal device 102 is software, it may be installed in the above-listed electronic devices. It may be implemented as multiple software or software modules (e.g., for providing distributed services), or it may be implemented as a single software or software module. No specific limitation is made here.

[0039] Server 106 can be a server that provides various services, such as a background server that supports various applications or software displayed on the terminal device 102. The database server 108 can also be a database server that provides various services. It can be understood that when the relevant functions of the database server 108 can be implemented by the server 106, the database server 108 may not be provided in the system 100.

[0040] Here, the server 106 and the database server 108 can be either hardware or software. When they are hardware, they can be implemented as a distributed server cluster composed of multiple servers, or as a single server. When they are software, they can be implemented as multiple software or software modules (e.g., for providing distributed services), or as a single software or software module. Specific limitations are not made here.

[0041] It should be understood that Figure 1 the numbers of the terminal devices, users, servers, and database servers in

[0042] As an exemplary scenario, the user 104 can use the multimedia data editing application or software of the terminal device 102 to edit and process multimedia data such as videos, audios, or images. For example, clip videos and audios, generate videos based on images, add special effects to videos and images, and so on. Optionally, the terminal device 102 can obtain materials for editing and processing from the server 106.

[0043] As mentioned above, in the field of multimedia data processing, images can be processed based on audio data. For example, by analyzing the audio data to obtain audio attribute information, and then processing the image based on this attribute information. As an implementable way, the short-time Fourier transform can be used to convert the audio data from the loudness sequence into a spectral distribution sequence (i.e., spectral data), and then process the image based on the spectral information in the spectral distribution sequence.

[0044] However, the inventors of the present disclosure have found that in the related art, usually, the spectral distribution information obtained directly by audio analysis is directly used to process images without data processing, resulting in poor image processing effects.

[0045] In view of this, the embodiments of the present disclosure provide a data processing method to at least partially solve the above problems.

[0046] Figure 2A schematic diagram showing an exemplary data processing method 200 provided by an embodiment of the present disclosure is shown. The method 200 can be used to process data. Optionally, the method 200 can be applied to Figure 1 a terminal device 102, or, applied to Figure 1 a system 100 and executed interactively among devices in the system 100.

[0047] Exemplarily, when processing data, an image to be processed and audio data corresponding to the image to be processed can be determined first.

[0048] Optionally, the image to be processed can be an image added by a user 104, or a video frame extracted from video data added by the user 104.

[0049] As an alternative embodiment, when the image to be processed is a video frame extracted from an image added by the user 104, the audio data corresponding to the image to be processed can be audio data extracted from the video data. In this way, when processing the video frame based on the audio data in the video data, it can be more in line with the scenario of the video data and the audio visualization effect is better.

[0050] Optionally, the audio data can also be audio data added by the user 104, for example, used as the background music of a live video.

[0051] Figure 3A A schematic diagram showing an exemplary page 300 according to an embodiment of the present disclosure is shown.

[0052] As Figure 3A shown, exemplarily, the page 300 can be a page of a multimedia data editing software. The user 104 can import multimedia data through an import button 302, such as videos, audios, pictures, etc. And it can support scanning a code to upload multimedia data stored in a mobile terminal. The imported data can be local data or cloud data obtained from a server 106.

[0053] Exemplarily, as Figure 3AAs shown, user 104 can import video data 304. The terminal device 102 can extract audio data 306A from the video data 304. Moreover, user 104 can also import audio data 306B. Therefore, user 104 can select at least a part from the audio data 306A and the audio data 306B as the audio data corresponding to the image to be processed in this embodiment. For example, if it is desired to process the image based on the audio data of the video data itself, the audio data 306A can be determined as the audio data corresponding to the image to be processed. If it is desired to process the image based on the audio data added according to preferences, the audio data 306B can be determined as the audio data corresponding to the image to be processed. Alternatively, both the audio data 306A and 306B can be determined as the audio data corresponding to the image to be processed, and then the spectrum data corresponding to the two audio data can be fused to process the image. Of course, it can be understood that multiple audio data can also be determined as the audio data corresponding to the image to be processed, and then the spectrum data corresponding to the multiple audio data can be fused to process the image. Specifically, it can be selected by user 104 himself / herself (for example, double-click on the audio data to select).

[0054] Generally, the original audio data is a loudness time series, which is not convenient for analysis. Therefore, the audio data can be transformed into frequency domain information for analysis. Therefore, after determining the audio data, the target frequency band in the spectrum data corresponding to the audio data for subsequent processing of the image to be processed can be further determined.

[0055] Optionally, user 104 can convert the selected entire audio data into spectrum data based on the short-time Fourier transform (STFT), and then intercept the spectrum associated with the image to be processed from the spectrum data as the target frequency band. Alternatively, user 104 can intercept a part of the audio data associated with the image to be processed from the audio data 306A or the audio data 306B, and then convert this part of the audio data into the target frequency band based on the short-time Fourier transform.

[0056] Exemplarily, the short-time Fourier transform can divide the audio into small segments with a window, perform a fast Fourier transform (FFT) on each small segment to obtain the frequency distribution of each small segment, that is, the spectrum information of this time period. Then, the spectrum information of all small segments is composed into a spectrum distribution time series.

[0057] As mentioned above, the target frequency band without data processing has poor effects when processing images. Therefore, in some embodiments, the target frequency band can be smoothed to obtain target data, and then based on the target data, a picture effect is added to the image to be processed to obtain a target image.

[0058] Thus, in the data processing method provided by the embodiments of the present disclosure, before adding a picture effect to the image to be processed based on the target frequency band, the target frequency band is first smoothed to obtain target data, and then a picture effect is added to the image to be processed based on the target data to obtain a target image, so that the picture processing effect of the target image is better.

[0059] In some embodiments, the picture effect added to the image to be processed may be to superimpose and display the relevant information of the target frequency band corresponding to the image to be processed on the image to be processed, thereby enhancing the image effect. As an alternative embodiment, when the relevant information of the superimposed target frequency band is for a video frame in video data and is superimposed based on the target frequency band of the audio data corresponding to the video frame, in this way, dynamic changing spectrum information can be displayed during video playback, thereby enhancing the interestingness.

[0060] However, the inventors of the present disclosure have found that if the target frequency band is not processed, when superimposing and displaying based on the unprocessed spectrum, there is a problem that the connection lines are sharp and not smooth.

[0061] Therefore, as an alternative embodiment, for a spectrum drawing type processing scheme, the target frequency band can be smoothed and then superimposed on the image to be processed, thereby at least to a certain extent solving the problem that the connection lines are sharp and not smooth.

[0062] In some embodiments, smoothing the target frequency band includes: determining at least two first reference points in the target frequency band; based on the at least two first reference points, drawing a smooth curve, and then the target frequency band can be processed based on the smooth curve to obtain a smooth spectrum. The selection of the first reference point can be arbitrary, as long as the spectrum with irregular edges can be processed into a spectrum with smooth edges based on the first reference point, the effect of more beautiful picture display can be achieved.

[0063] As an alternative embodiment, the at least two first reference points include at least one peak point in the target frequency band and at least one valley point adjacent to the peak point, so that the smooth curve can be closer to the edge shape of the target frequency band, and a better spectrum drawing effect can be obtained.

[0064] Figure 3B FIG. shows a schematic diagram of an exemplary target frequency band 310 according to an embodiment of the present disclosure.

[0065] As Figure 3BAs shown, optionally, peak points PB, PD, PG and valley points PA, PC, PE, PF adjacent to each peak point can be determined from the target frequency band 310. Among them, a peak point can be judged by the values on both sides of it being smaller than the value of this point, and then the lowest point between two adjacent peak points can be determined as the valley point. When there are multiple valley points, the valley point where there is no valley point between it and the closest peak point can be determined as the finally selected valley point. In other words, the valley point closest to the peak point is taken as the finally selected valley point. Optionally, the peak point and the valley point can also be manually marked by the user 104 in the spectrum, or after automatically identifying the peak point and the valley point, the user 104 can manually adjust the positions of the peak point and the valley point.

[0066] Then, two adjacent peak points and valley points can be selected to draw a smooth curve. For example, the first peak point among the peak points PB, PD, PG can be determined as the peak point PB, and the valley point PA adjacent to the first peak point can be determined as the first valley point. It can be understood that the valley point PC can also be selected as the first valley point. In fact, depending on the selected points, the drawn curve segment is also in different sections. Ultimately, it is necessary to ensure that the entire target frequency band is drawn as a continuous curve. Therefore, selecting the left or right valley point does not affect the final result.

[0067] As an optional embodiment, according to the different distances between the selected peak points and valley points, different curve relationships can be adopted to draw a smooth curve in order to obtain a better drawing effect.

[0068] Optionally, in response to the distance between the first peak point and the first valley point being less than the distance threshold, the smooth curve can be drawn based on the first peak point and the first valley point according to the first curve relationship. Or, in response to the distance between the first peak point and the first valley point being greater than or equal to the distance threshold, the smooth curve can be drawn based on the first peak point and the first valley point according to the second curve relationship. In this way, when the distance between the first peak point and the first valley point is small, the smooth curve is drawn according to the first curve relationship, and when the distance between the first peak point and the first valley point is large, the smooth curve is drawn according to the second curve relationship, so as to obtain a better smoothing effect. Optionally, the distance can refer to the difference in abscissas, or the difference in ordinates, or the straight-line distance between two points, and can be specifically set according to needs. As an optional embodiment, the distance can refer to the difference in abscissas, so as to ensure the final drawing effect when processing in combination with the distance threshold regarding the target width.

[0069] In some embodiments, the distance threshold is related to the target width of the smoothing curve corresponding to the target frequency band. This target width can be the width of the smoothed spectrum that is finally superimposed and displayed on the image to be processed. Thus, by using a distance threshold related to the target width, better image processing effects can also be obtained. Optionally, the distance threshold is 5% - 10% of the target width. Since there is a large amount of frequency information in the target frequency band, although the spectrum shown in Figure 3B is relatively simple, the actual situation is more complex. Setting the distance threshold to be smaller can ensure that the smoothing effect is closer to the original spectrum.

[0070] In some embodiments, the first curve relationship includes a cubic function. For example, the ordinate is a function of the cube of the abscissa; the second curve relationship includes a quartic function. For example, the ordinate is a function of the fourth power of the abscissa. This smoothing method has good performance and can meet the real-time drawing requirements of video editing scenarios. The obtained smoothed spectrum 320 is as shown in Figure 3C shown.

[0071] In some scenarios, when adding a picture effect to video data, if the target frequency band is not processed, there is still a certain possibility that the spectra superimposed on the front and back video frames will jump when superimposed and displayed based on the unprocessed spectrum.

[0072] Therefore, in some embodiments, the image to be processed includes the target video frame in the video data, and further smoothing processing can be performed based on the time sequence.

[0073] To perform smoothing processing on the target frequency band, the historical smoothing curve corresponding to the previous video frame of the target video frame can be determined first, and then based on the target frequency band, the current smoothing curve corresponding to the target video frame (for example, the smoothing curve shown in Figure 3B ) can be determined. Then, the historical smoothing curve and the current smoothing curve are weighted and fused to obtain the target smoothing curve. Among them, the weight corresponding to the historical smoothing curve is less than the weight corresponding to the current smoothing curve. In this way, the target smoothing curve is obtained by weighted fusion of the historical data in the time sequence in combination with the weight and the current smoothing curve, thereby improving the jump problem.

[0074] It can be understood that multiple historical smoothing curves can be selected, that is, the previous video frame, the previous previous video frame, the previous previous previous video frame, and so on, of the target video frame.

[0075] Therefore, in some embodiments, the target data can be the smoothing curve 320 or the smoothed spectrum corresponding to the target smoothing curve. Then, the smoothed spectrum can be superimposed and displayed on the image to be processed to obtain the target image, thereby realizing adding a picture effect to the image to be processed and enhancing the interestingness, as shown in Figure 3D shown.

[0076] In some scenarios, the special effect intensity can also be determined based on audio data, so as to adjust the special effect according to the audio data. However, the inventors of the present disclosure have found that in the related art, directly selecting the frequency data in the spectrum as the special effect intensity value to adjust the special effect also has the problem of jump between the previous and the current frames, which is not convenient for realizing harmonious and beautiful video aftereffects.

[0077] Therefore, in some embodiments, when the image to be processed includes a target video frame in video data, in order to obtain target data by smoothing the target frequency band, the historical target data corresponding to the previous video frame of the target video frame can be determined first, then based on the target frequency band, the current target data corresponding to the target video frame can be determined, and finally, the historical target data and the current target data are smoothed to obtain the target data. In this way, by performing temporal smoothing on the historical target data and the current target data to obtain a smoothing parameter as the target data, the jump problem can be better improved.

[0078] In some embodiments, determining the current target data corresponding to the target video frame based on the target frequency band includes: sampling a plurality of second reference points in the target frequency band, and determining the current target data based on the plurality of second reference points. It can be understood that the specific sampling method is determined according to the desired effect. As long as the sampling standard is the same, similar processing effects can be obtained to ensure consistency.

[0079] Optionally, the plurality of second reference points include at least one peak point (for example, peak points PB, PD, PG) in the target frequency band. In this way, using the peak point to set the special effect intensity, so that the special effect intensity is associated with the audio intensity, better processing effects can be obtained.

[0080] In some embodiments, determining the current target data based on the plurality of second reference points includes: solving the average value of the ordinates corresponding to the plurality of second reference points to obtain the current target data, so that the processing effect is not likely to jump.

[0081] In some embodiments, smoothing the historical target data and the current target data to obtain the target data includes:

[0082] determining a first weight corresponding to the historical target data and a second weight corresponding to the current target data, the first weight being less than the second weight;

[0083] Based on the historical target data, the current target data, the first weight, and the second weight, the target data is obtained by weighted averaging.

[0084] In this way, the target data is obtained through weighted average, which can better improve the jump problem.

[0085] In some embodiments, the current target data serves as the historical target data corresponding to the next video frame of the target video frame. For example, each time the target data is obtained, it is stored as historical data, so that the next video frame can be directly called during processing, improving the processing efficiency. For example, if there is historical data, the nearest n historical data are obtained, and weighted average is performed with the current data according to weights 1, 2, …, n + 1, and the obtained value is used as the current parameter value to control the special effect intensity related to the post-processing of the picture.

[0086] In some embodiments, based on the target data, adding a picture effect to the image to be processed to obtain a target image includes:

[0087] Determining a target special effect, for example, a blur special effect (i.e., blurring the picture);

[0088] Based on the target data, determining the special effect intensity corresponding to the target special effect, for example, the level of blurring;

[0089] Based on the target special effect and the special effect intensity, adding the special effect to the image to be processed to obtain the target image.

[0090] In this way, the special effect intensity is associated with the audio data, realizing the audio visualization effect.

[0091] In some embodiments, when the image to be processed is the target video frame in the video data, the target image can also be used to replace the target video frame in the video data, thereby obtaining a target video with a better matching effect with the audio. In this way, when watching the target video, at the time node of viewing the target video frame, a target image matching the audio can be viewed, thereby realizing the audio visualization effect in the video and improving the user experience.

[0092] As can be seen from the above embodiments, the embodiments of the present disclosure provide an audio visualization solution applicable to the video editing scenario, which associates audio information with the picture and is relatively beautiful. This solution first analyzes the spectral distribution of the audio, then performs data processing by smoothing the shape or time series of the spectrum, and then uses the obtained data for post-processing, and can support real-time audio visualization effects that take into account both beauty and relevance.

[0093] It should be noted that the method of the embodiments of the present disclosure can be executed by a single device, such as a computer or a server. The method of this embodiment can also be applied to a distributed scenario and completed by multiple devices cooperating with each other. In such a distributed scenario, one of the multiple devices can only execute one or more steps of the method of the embodiments of the present disclosure, and these multiple devices will interact with each other to complete the described method.

[0094] It should be noted that some embodiments of the present disclosure have been described above. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recited in the claims can be executed in a different order from that in the above embodiments and still achieve the desired result. Additionally, the processes depicted in the drawings do not necessarily require the specific order or sequential order shown to achieve the desired result. In certain embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0095] The embodiments of the present disclosure also provide a data processing method. Figure 4 The flowchart of an exemplary method 400 provided by the embodiments of the present disclosure is shown. This method 400 can be implemented Figure 1 by the terminal device 102 or jointly implemented by the devices of the system 100. As Figure 4 shown, this method 400 may include the following steps.

[0096] In step 402, determine the image to be processed and the audio data corresponding to the image to be processed;

[0097] In step 404, based on the audio data, determine the target frequency band corresponding to the image to be processed in the spectral data corresponding to the audio data;

[0098] In step 406, perform smoothing processing on the target frequency band to obtain target data;

[0099] In step 408, based on the target data, add a picture effect to the image to be processed to obtain a target image.

[0100] In the data processing method provided by the embodiments of the present disclosure, before adding a picture effect to the image to be processed based on the target frequency band, first perform smoothing processing on the target frequency band to obtain target data, and then add a picture effect to the image to be processed based on the target data to obtain a target image, so that the picture processing effect of the target image is better.

[0101] In some embodiments, performing smoothing processing on the target frequency band includes:

[0102] Determine at least two first reference points in the target frequency band;

[0103] Draw a smooth curve based on the at least two first reference points.

[0104] In this way, drawing a smooth curve based on the reference points in the target frequency band can make the smooth curve closer to the spectral shape and achieve a better drawing effect.

[0105] In some embodiments, the at least two first reference points include at least one peak point in the target frequency band and at least one valley point adjacent to the peak point;

[0106] Drawing a smooth curve based on the at least two first reference points as the target data includes:

[0107] Determine a first peak point among the at least one peak point and a first valley point adjacent to the first peak point;

[0108] In response to the distance between the first peak point and the first valley point being less than the distance threshold, draw the smooth curve based on the first peak point and the first valley point according to a first curve relationship;

[0109] In response to the distance between the first peak point and the first valley point being greater than or equal to the distance threshold, draw the smooth curve based on the first peak point and the first valley point according to a second curve relationship.

[0110] In this way, when the distance between the first peak point and the first valley point is small, draw the smooth curve according to the first curve relationship, and when the distance between the first peak point and the first valley point is large, draw the smooth curve according to the second curve relationship, so as to obtain a better smoothing effect.

[0111] In some embodiments, the distance threshold is related to the target width of the smooth curve corresponding to the target frequency band. This target width can be the width of the smooth spectrum finally superimposed and displayed on the image to be processed. Therefore, by using a distance threshold related to the target width, a better image processing effect can also be obtained.

[0112] Optionally, the distance threshold is 5% - 10% of the target width. Since there is a lot of frequency information in the target frequency band, although Figure 3B the spectrum shown is relatively simple, the actual situation is more complex. Setting the distance threshold to be smaller can ensure that the smoothing effect is closer to the original spectrum.

[0113] In some embodiments, the first curve relationship includes a cubic function, and the second curve relationship includes a quartic function. This smoothing method has good performance and can meet the real-time drawing requirements of video editing scenarios.

[0114] In some embodiments, the image to be processed includes a target video frame in video data;

[0115] Performing smoothing processing on the target frequency band further includes:

[0116] Determining a historical smoothing curve corresponding to the previous video frame of the target video frame;

[0117] Based on the target frequency band, determining a current smoothing curve corresponding to the target video frame;

[0118] Performing weighted fusion on the historical smoothing curve and the current smoothing curve to obtain a target smoothing curve.

[0119] In this way, based on historical data in time series, combining weights with the current smoothing curve for weighted fusion to obtain a target smoothing curve, thereby improving the jump problem.

[0120] In some embodiments, the target data includes the smoothing curve or the smoothing spectrum corresponding to the target smoothing curve; based on the target data, adding a picture effect to the image to be processed to obtain a target image, including: superimposing and displaying the smoothing spectrum on the image to be processed to obtain the target image, thereby realizing adding a picture effect to the image to be processed and enhancing the interest.

[0121] In some embodiments, the image to be processed includes a target video frame in video data;

[0122] Performing smoothing processing on the target frequency band to obtain target data includes:

[0123] Determining historical target data corresponding to the previous video frame of the target video frame;

[0124] Based on the target frequency band, determining current target data corresponding to the target video frame;

[0125] Performing smoothing processing on the historical target data and the current target data to obtain the target data.

[0126] In this way, performing temporal smoothing processing based on the historical target data and the current target data to obtain smoothing parameters as the target data can better improve the jump problem.

[0127] In some embodiments, based on the target frequency band, determining current target data corresponding to the target video frame includes:

[0128] Sampling a plurality of second reference points in the target frequency band;

[0129] Based on the plurality of second reference points, determining the current target data.

[0130] In this way, determining the current target data based on the reference points in the target frequency band can associate the image processing effect with the audio information and enhance the audio visualization effect.

[0131] In some embodiments, the multiple second reference points include at least one peak point in the target frequency band. In this way, using the peak point information to process the image can associate the image processing effect with the audio intensity and obtain a better processing effect.

[0132] In some embodiments, determining the current target data based on the multiple second reference points includes: calculating the average value of the ordinates corresponding to the multiple second reference points to obtain the current target data, so that the processing effect is not likely to jump.

[0133] In some embodiments, smoothing the historical target data and the current target data to obtain the target data includes:

[0134] determining a first weight corresponding to the historical target data and a second weight corresponding to the current target data, where the first weight is less than the second weight;

[0135] obtaining the target data through weighted averaging based on the historical target data, the current target data, the first weight, and the second weight.

[0136] In this way, obtaining the target data through weighted averaging can better improve the jump problem.

[0137] In some embodiments, the current target data serves as the historical target data corresponding to the next video frame of the target video frame. For example, each time the target data is obtained, it is stored as historical data, so that it can be directly called when processing the next video frame, improving the processing efficiency.

[0138] In some embodiments, adding a picture effect to the image to be processed based on the target data to obtain a target image includes:

[0139] determining a target special effect;

[0140] determining the special effect intensity corresponding to the target special effect based on the target data;

[0141] adding the special effect to the image to be processed based on the target special effect and the special effect intensity to obtain the target image.

[0142] In this way, the special effect intensity is associated with the audio data to achieve the audio visualization effect.

[0143] In some embodiments, the method further includes: replacing the target video frame in the video data with the target image to generate a target video.

[0144] By adopting such a method, a target video with better cooperation effect with the audio can be obtained. In this way, when watching the target video, at the time node of viewing the target video frame, a target image matching the audio can be viewed, thereby realizing the visualization effect of the audio in the video and improving the user experience.

[0145] In some embodiments, the audio data includes the audio data in the video data and / or the audio data selected by the user, so that the selection of the audio data is more diversified and has a wider range.

[0146] It should be noted that the method of the embodiments of the present disclosure can be executed by a single device, such as a computer or a server. The method of this embodiment can also be applied to a distributed scenario and completed by multiple devices cooperating with each other. In such a distributed scenario, one of the multiple devices can only execute one or more steps of the method of the embodiments of the present disclosure, and these multiple devices will interact with each other to complete the described method.

[0147] It should be noted that some embodiments of the present disclosure are described above. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recited in the claims can be performed in a different order from those in the above embodiments and still achieve the desired result. Additionally, the processes depicted in the drawings do not necessarily require the specific order or sequential order shown to achieve the desired result. In certain embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0148] The embodiments of the present disclosure also provide a computer device for implementing the above method 400. Figure 5 The hardware structure diagram of an exemplary computer device 500 provided by the embodiments of the present disclosure is shown. The computer device 500 can be used to implement Figure 1 the server 106, and can also be used to implement Figure 1 the terminal devices 102, 104. In some scenarios, this computer device 500 can also be used to implement Figure 1 the database server 108.

[0149] As Figure 5 shown, the computer device 500 may include: a processor 502, a memory 504, a network module 506, a peripheral interface 508, and a bus 510. Among them, the processor 502, the memory 504, the network module 506, and the peripheral interface 508 are communicatively connected to each other inside the computer device 500 through the bus 510.

[0150] The processor 502 may be a Central Processing Unit (CPU), an image processor, a Neural Network Processor (NPU), a microcontroller (MCU), a programmable logic device, a Digital Signal Processor (DSP), an Application Specific Integrated Circuit (ASIC), or one or more integrated circuits. The processor 502 may be used to execute functions related to the technologies described in this disclosure. In some embodiments, the processor 502 may also include multiple processors integrated as a single logic component. For example, as Figure 5 shown, the processor 502 may include multiple processors 502a, 502b, and 502c.

[0151] The memory 504 may be configured to store data (e.g., instructions, computer code, etc.). As Figure 5 shown, the data stored in the memory 504 may include program instructions (e.g., program instructions for implementing the method 400 of the embodiments of this disclosure) and data to be processed (e.g., the memory may store configuration files of other modules, etc.). The processor 502 may also access the program instructions and data stored in the memory 504 and execute the program instructions to operate on the data to be processed. The memory 504 may include a volatile storage device or a non-volatile storage device. In some embodiments, the memory 504 may include a Random Access Memory (RAM), a Read Only Memory (ROM), an optical disc, a magnetic disk, a hard disk, a Solid State Drive (SSD), a flash memory, a memory stick, etc.

[0152] The network interface 506 may be configured to provide communication with other external devices to the computer device 500 via a network. The network may be any wired or wireless network capable of transmitting and receiving data. For example, the network may be a wired network, a local wireless network (e.g., Bluetooth, WiFi, Near Field Communication (NFC), etc.), a cellular network, the Internet, or a combination of the above. It can be understood that the type of the network is not limited to the above specific examples.

[0153] The peripheral interface 508 may be configured to connect the computer device 500 to one or more peripheral devices to achieve information input and output. For example, the peripheral devices may include input devices such as a keyboard, a mouse, a touchpad, a touch screen, a microphone, various sensors, etc. and output devices such as a display, a speaker, a vibrator, an indicator light, etc.

[0154] The bus 510 can be configured to transfer information between various components of the computer device 500 (such as the processor 502, the memory 504, the network interface 506, and the peripheral interface 508), such as internal buses (e.g., the processor-memory bus), external buses (USB ports, PCI-E buses), etc.

[0155] It should be noted that although the architecture of the above computer device 500 only shows the processor 502, the memory 504, the network interface 506, the peripheral interface 508, and the bus 510, in the specific implementation process, the architecture of the computer device 500 may further include other components necessary for normal operation. In addition, those skilled in the art can understand that the architecture of the above computer device 500 may also only include the components necessary to implement the solution of the embodiments of the present disclosure, and does not necessarily include all the components shown in the figure.

[0156] The embodiments of the present disclosure also provide a data processing device 600. Figure 6 The schematic diagram of the exemplary device 600 provided by the embodiments of the present disclosure is shown. As Figure 6 shown, the device 600 can be used to implement the method 400, and may further include the following modules.

[0157] The first determination module 602 is configured to: determine the image to be processed and the audio data corresponding to the image to be processed;

[0158] The second determination module 604 is configured to: based on the audio data, determine the target frequency band corresponding to the image to be processed in the spectral data corresponding to the audio data;

[0159] The smoothing module 606 is configured to: perform smoothing processing on the target frequency band to obtain target data;

[0160] The adding module 608 is configured to: based on the target data, add a picture effect to the image to be processed to obtain a target image.

[0161] In some embodiments, the smoothing module 606 is configured to:

[0162] determine at least two first reference points in the target frequency band;

[0163] Based on the at least two first reference points, draw a smoothing curve.

[0164] In some embodiments, the at least two first reference points include at least one peak point in the target frequency band and at least one valley point adjacent to the peak point;

[0165] The smoothing module 606 is configured to:

[0166] Determine a first peak point among the at least one peak point and a first valley point adjacent to the first peak point;

[0167] In response to the distance between the first peak point and the first valley point being less than the distance threshold, draw the smooth curve based on the first peak point and the first valley point according to a first curve relationship;

[0168] In response to the distance between the first peak point and the first valley point being greater than or equal to the distance threshold, draw the smooth curve based on the first peak point and the first valley point according to a second curve relationship.

[0169] In some embodiments, the distance threshold is related to a target width of the smooth curve corresponding to the target frequency band.

[0170] In some embodiments, the distance threshold is 5% - 10% of the target width.

[0171] In some embodiments, the first curve relationship includes a cubic function, and the second curve relationship includes a quartic function.

[0172] In some embodiments, the image to be processed includes a target video frame in video data;

[0173] A smoothing module 606, configured to:

[0174] Determine a historical smooth curve corresponding to the previous video frame of the target video frame;

[0175] Based on the target frequency band, determine a current smooth curve corresponding to the target video frame;

[0176] Perform weighted fusion on the historical smooth curve and the current smooth curve to obtain a target smooth curve.

[0177] In some embodiments, the target data includes the smooth curve or a smooth spectrum corresponding to the target smooth curve;

[0178] An adding module 608, configured to: superimpose and display the smooth spectrum on the image to be processed to obtain the target image.

[0179] In some embodiments, the image to be processed includes a target video frame in video data;

[0180] A smoothing module 606, configured to:

[0181] Determine historical target data corresponding to the previous video frame of the target video frame;

[0182] Based on the target frequency band, determine current target data corresponding to the target video frame;

[0183] Smooth the historical target data and the current target data to obtain the target data.

[0184] In some embodiments, the smoothing module 606 is configured to:

[0185] Sample a plurality of second reference points in the target frequency band;

[0186] Determine the current target data based on the plurality of second reference points.

[0187] In some embodiments, the plurality of second reference points include at least one peak point in the target frequency band.

[0188] In some embodiments, the smoothing module 606 is configured to:

[0189] Solve the average value of the ordinates corresponding to the plurality of second reference points to obtain the current target data.

[0190] In some embodiments, the smoothing module 606 is configured to:

[0191] Determine a first weight corresponding to the historical target data and a second weight corresponding to the current target data, the first weight being less than the second weight;

[0192] Based on the historical target data, the current target data, the first weight, and the second weight, obtain the target data through weighted averaging.

[0193] In some embodiments, the current target data serves as the historical target data corresponding to the next video frame of the target video frame.

[0194] In some embodiments, the adding module 608 is configured to:

[0195] Determine a target special effect;

[0196] Based on the target data, determine the special effect intensity corresponding to the target special effect;

[0197] Add a special effect to the image to be processed based on the target special effect and the special effect intensity to obtain the target image.

[0198] In some embodiments, the adding module 608 is configured to:

[0199] Replace the target video frame in the video data with the target image to generate a target video.

[0200] In some embodiments, the audio data includes the audio data in the video data and / or the audio data selected by the user.

[0201] For the convenience of description, when describing the above device, various modules are described separately according to their functions. Of course, when implementing the present disclosure, the functions of each module can be implemented in the same or multiple software and / or hardware.

[0202] The device in the above embodiment is used to implement the corresponding method 400 in any of the foregoing embodiments, and has the beneficial effects of the corresponding method embodiments, which will not be elaborated here.

[0203] Based on the same inventive concept, corresponding to the method in any of the above embodiments, the present disclosure also provides a non-transitory computer-readable storage medium storing computer instructions for causing the computer to execute the method 400 described in any of the foregoing embodiments.

[0204] The computer-readable medium of this embodiment includes permanent and non-permanent, removable and non-removable media, and information storage can be implemented by any method or technology. The information can be computer-readable instructions, data structures, program modules, or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, compact disc read-only memory (CD-ROM), digital versatile disc (DVD) or other optical storage, magnetic cassette tapes, magnetic disk storage or other magnetic storage devices, or any other non-transmission medium that can be used to store information accessible by a computing device.

[0205] The computer instructions stored in the storage medium of the above embodiment are used to cause the computer to execute the method 400 described in any of the foregoing embodiments, and have the beneficial effects of the corresponding method embodiments, which will not be elaborated here.

[0206] Based on the same inventive concept, corresponding to the method 400 in any of the above embodiments, the present disclosure also provides a computer program product including a computer program. In some embodiments, the computer program is executable by one or more processors to cause the processors to execute the method 400. Corresponding to the execution subject of each step in the method 400, the processor executing the corresponding step can belong to the corresponding execution subject.

[0207] The computer program product of the above embodiment is used to cause the processor to execute the method 400 described in any of the foregoing embodiments, and has the beneficial effects of the corresponding method embodiments, which will not be elaborated here.

[0208] Those of ordinary skill in the art should understand that: the discussion of any of the above embodiments is only exemplary and is not intended to imply that the scope of the present disclosure (including the claims) is limited to these examples; under the concept of the present disclosure, the technical features in the above embodiments or different embodiments can also be combined, the steps can be implemented in any order, and there are many other variations in different aspects of the embodiments of the present disclosure as described above, and they are not provided in detail for the sake of brevity.

[0209] In addition, for the sake of simplicity of explanation and discussion, and in order not to make the embodiments of the present disclosure difficult to understand, the well-known power / ground connections to integrated circuit (IC) chips and other components may or may not be shown in the provided drawings. Further, the devices may be shown in block diagram form in order to avoid making the embodiments of the present disclosure difficult to understand, and this also takes into account the fact that the details of the implementation of these block diagram devices are highly dependent on the platform on which the embodiments of the present disclosure are to be implemented (i.e., these details should be entirely within the understanding of those skilled in the art). In cases where specific details (such as circuits) are set forth to describe exemplary embodiments of the present disclosure, it will be apparent to those skilled in the art that the embodiments of the present disclosure may be implemented without these specific details or with variations of these specific details. Accordingly, these descriptions should be considered illustrative rather than restrictive.

[0210] Although the present disclosure has been described in connection with specific embodiments of the present disclosure, many alternatives, modifications, and variations of these embodiments will be apparent to those of ordinary skill in the art based on the foregoing description. For example, other memory architectures (such as dynamic RAM (DRAM)) may be used with the embodiments discussed.

[0211] The embodiments of the present disclosure are intended to cover all such alternatives, modifications, and variations that fall within the broad scope of the appended claims. Therefore, any omissions, modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the embodiments of the present disclosure shall be included within the protection scope of the present disclosure.

Claims

1. A data processing method, comprising: Determine an image to be processed and audio data corresponding to the image to be processed; Based on the audio data, determining a target frequency band corresponding to the image to be processed in the frequency spectrum data corresponding to the audio data; Performing smoothing processing on the target frequency band to obtain target data; Based on the target data, a picture effect is added to the image to be processed to obtain a target image.

2. The method of claim 1, wherein: The target frequency band is smoothed, including: Determining at least two first reference points in the target frequency band; Based on the at least two first reference points, a smooth curve is drawn.

3. The method of claim 2, wherein: The at least two first reference points include at least one peak point in the target frequency band and at least one valley point adjacent to the peak point; Based on the at least two first reference points, drawing a smooth curve as the target data comprises: Determine a first peak point among the at least one peak point and a first valley point adjacent to the first peak point; In response to the distance between the first peak point and the first valley point being less than the distance threshold, drawing the smooth curve based on the first peak point and the first valley point according to a first curve relationship; In response to the distance between the first peak point and the first valley point being greater than or equal to the distance threshold, the smooth curve is drawn based on the first peak point and the first valley point according to a second curve relationship.

4. The method of claim 3, wherein: The distance threshold is related to the target width of the smooth curve corresponding to the target frequency band; and / or, the distance threshold is 5% to 10% of the target width; and / or, the first curve relationship includes a cubic function, and the second curve relationship includes a quartic function.

5. The method of claim 2, wherein: The image to be processed includes a target video frame in the video data; The target frequency band is smoothed, further comprising: Determine a historical smooth curve corresponding to a previous video frame of the target video frame; Based on the target frequency band, determining a current smooth curve corresponding to the target video frame; The historical smooth curve and the current smooth curve are weightedly fused to obtain a target smooth curve.

6. The method according to any one of claims 2 to 5, wherein: The target data includes the smooth curve or a smooth spectrum corresponding to the target smooth curve; Based on the target data, adding a picture effect to the image to be processed to obtain a target image includes: The smoothed frequency spectrum is superimposed and displayed on the image to be processed to obtain the target image.

7. The method of claim 1, wherein: The image to be processed includes a target video frame in the video data; Smoothing the target frequency band to obtain target data includes: Determining historical target data corresponding to a previous video frame of the target video frame; Based on the target frequency band, determining current target data corresponding to the target video frame; The historical target data and the current target data are smoothed to obtain the target data.

8. The method of claim 7, wherein: Determining current target data corresponding to the target video frame based on the target frequency band includes: Sampling in the target frequency band to obtain a plurality of second reference points; Determining the current target data based on the multiple second reference points; The multiple second reference points include at least one peak point in the target frequency band.

9. The method of claim 8, wherein: Determining the current target data based on the plurality of second reference points includes: The average value of the vertical coordinates corresponding to the multiple second reference points is solved to obtain the current target data.

10. The method of claim 7, wherein: Smoothing the historical target data and the current target data to obtain the target data includes: Determine a first weight corresponding to the historical target data and a second weight corresponding to the current target data, wherein the first weight is less than the second weight; Obtaining the target data by weighted average based on the historical target data, the current target data, the first weight, and the second weight; The current target data is used as the historical target data corresponding to the next video frame of the target video frame.

11. The method of claim 7, wherein: Based on the target data, adding a picture effect to the image to be processed to obtain a target image includes: Determine the target special effects; Based on the target data, determining a special effect intensity corresponding to the target special effect; Based on the target special effect and the special effect strength, a special effect is added to the image to be processed to obtain the target image.

12. The method according to any one of claims 5, 7 to 11, wherein: The method further comprises: Replacing the target video frame in the video data with the target image to generate a target video; The audio data includes audio data in the video data and / or audio data selected by a user.

13. A data processing device, comprising: A first determining module is configured to: determine an image to be processed and audio data corresponding to the image to be processed; A second determination module is configured to: determine, based on the audio data, a target frequency band corresponding to the image to be processed in the frequency spectrum data corresponding to the audio data; A smoothing module is configured to: perform smoothing processing on the target frequency band to obtain target data; The adding module is configured to: add picture effects to the image to be processed based on the target data to obtain a target image.

14. A computer device comprising one or more processors, a memory; and one or more programs, wherein the one or more programs are stored in the memory and executed by the one or more processors, and the programs include instructions for executing the method according to any one of claims 1 to 12.

15. A non-volatile computer-readable storage medium containing a computer program, which, when executed by one or more processors, causes the processors to perform the method of any one of claims 1 to 12.

16. A computer program product comprising computer program instructions, which, when executed on a computer, cause the computer to execute the method according to any one of claims 1 to 12.

Images