Visual effect enhancement method and device, electronic equipment and readable medium

Abstract

The invention discloses a visual effect enhancement method and device, electronic equipment and a readable medium. The method comprises the following steps: acquiring audio data containing a plurality of notes; analyzing the audio intensity and the audio rhythm corresponding to any note in the audio data; when a multimedia video containing the audio data is rendered, executing a visual effect enhancement display task on a display interface of the multimedia video according to the audio rhythm and the audio intensity; wherein the visual effect enhancement display task comprises a vibration screen enhancement display task and / or a light enhancement display task. According to the scheme, note content in the audio data does not need to be accurately recognized, the whole note content is directly processed as the audio data, the visual enhancement display effect (such as vibration screen and scene light) is controlled according to the audio strength and the audio rhythm, and a user can obtain audio-visual experience with good consistency.

Description

Technical Field

[0001] Embodiments of the present invention relate to the field of Internet technology, and in particular, to a visual effect enhancement method, device, electronic device, and readable medium. Background Art

[0002] In a virtual interactive scene, users can watch the various actions of virtual targets in the scene, and some users can also control virtual characters to participate in it.

[0003] In order to give users a better viewing experience, various sound effects and dynamic display effects are generally added to the scene. However, sometimes the added sound effects (for example, background music added by a user himself) cannot match the dynamic display effects, or the dynamic display cannot be displayed dynamically according to the sound effects. Sometimes, even if the dynamic display can be coordinated with the sound effects, due to the large number of sound sources in the virtual scene and the noisy sound, the consistency between the two is poor. The immersive experience obtained by the user is limited. Therefore, a solution is needed to enhance the user's sensory experience. Summary of the invention

[0004] The embodiments of the present invention provide a visual effect enhancement method, device, electronic device and readable medium to improve the sensory experience effect of users in virtual interaction.

[0005] In a first aspect, an embodiment of the present invention provides a visual effect enhancement method, the method comprising: obtaining audio data containing multiple notes; parsing the audio intensity and audio rhythm corresponding to any note in the audio data; when rendering a multimedia video containing the audio data, performing a visual effect enhancement display task on the display interface of the multimedia video according to the audio rhythm and the audio intensity; wherein the visual effect enhancement display task comprises: a screen shaking enhancement display task and / or a lighting enhancement display task.

[0006] Optionally, the visual effect enhancement display task is a screen shaking display enhancement display task, and when rendering a multimedia video containing the audio data, the visual effect enhancement display task is performed on the display interface according to the audio rhythm and the audio intensity, including: determining the screen shaking parameters according to the audio rhythm and the audio intensity; planning the movement trajectory and movement period of the virtual camera according to the screen shaking parameters and the target object in the display content; controlling the movement of the virtual camera relative to the target object based on the movement trajectory and the movement period, and performing the screen shaking display task on the display interface.

[0007] Optionally, the movement trajectory includes: a starting trajectory and a recovery trajectory; when executing the screen shaking enhanced display task, if the target object does not move, the starting trajectory is the same as the recovery trajectory; if the target object moves, the center of the current display interface is used as the end point of the recovery trajectory and the recovery trajectory is generated.

[0008] Optionally, the movement trajectory includes: a starting trajectory and a recovery trajectory; when executing the screen shaking enhanced display task, if the display viewing angle of the display content is adjusted, the center of the current display interface is used as the end point of the recovery trajectory and the recovery trajectory is generated, or, the center of the current interface is used as the end point of the starting trajectory and the starting trajectory is generated.

[0009] Optionally, the movement trajectory includes: a linear reciprocating trajectory, a viewing angle adjustment trajectory centered on the target object, and a rotation trajectory centered on the target object.

[0010] Optionally, the visual effect enhancement is a light enhancement display effect; when rendering a multimedia video containing the audio data, according to the audio rhythm and the audio intensity, a visual effect enhancement display task is performed on the display interface of the multimedia video, including: grouping scene lights; according to the audio rhythm and the audio intensity, determining scene lighting parameters of each group; wherein the scene lighting parameters include: at least one of scene light color, scene light duty cycle and scene light direction; based on the scene light parameters, performing a light enhancement display task on the display interface.

[0011] Optionally, the grouping of scene lights includes: grouping the scene lights according to the types of notes contained in the audio data.

[0012] Optionally, performing a light enhancement display task on the display interface based on the scene light parameters includes: configuring a directional light corresponding to each of the scene lights according to the scene light parameters, the parameters of the directional light corresponding to the scene light parameters; and performing a light enhancement display task on the display interface based on the parameters of the directional light.

[0013] Optionally, the parsing of the audio intensity and audio rhythm corresponding to any note in the audio data includes: parsing the frequency spectrum data of any note in the audio data corresponding to each frame of the multimedia video to be presented to obtain sub-audio intensities and sub-audio rhythms of multiple notes; and generating the audio intensity and audio rhythm of the audio data based on the sub-audio intensities and sub-audio rhythms of the multiple notes.

[0014] In a second aspect, an embodiment of the present invention provides a visual effect enhancement device, which includes: an acquisition module for acquiring audio data containing multiple notes; a parsing module for parsing the audio intensity and audio rhythm corresponding to any note in the audio data; an execution module for performing a visual effect enhancement display task on the display interface of the multimedia video when rendering the multimedia video containing the audio data according to the audio rhythm and the audio intensity; wherein the visual effect enhancement display task includes: a screen shaking enhancement display task and / or a lighting enhancement display task.

[0015] In a third aspect, an embodiment of the present invention further provides an electronic device, comprising a processor and a memory, wherein the memory stores at least one instruction, at least one program, a code set or an instruction set, and the at least one instruction, at least one program, a code set or an instruction set is loaded and executed by the processor to implement the visual effect enhancement method according to the first aspect.

[0016] In a fourth aspect, an embodiment of the present invention further provides a computer-readable medium on which is stored at least one instruction, at least one program, a code set or an instruction set, and the at least one instruction, at least one program, a code set or an instruction set is loaded and executed by a processor to implement the visual effect enhancement method according to the first aspect.

[0017] In an embodiment of the present invention, audio data containing multiple musical notes is obtained; the audio intensity and audio rhythm corresponding to any musical note in the audio data are parsed; when rendering a multimedia video containing the audio data, a visual enhancement display task is performed on the display interface of the multimedia video according to the audio rhythm and the audio intensity. With this solution, there is no need to accurately identify the musical note content in the audio data, and the entire audio data can be directly processed as audio data, that is, the visual enhancement display effect (such as screen shaking and scene lighting) is controlled according to the audio intensity and audio rhythm, so that the user can obtain a more consistent audio-visual experience. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0019] Figure 1 A flowchart of a visual effect enhancement method provided in an embodiment of the present application;

[0020] Figure 2a , 2b2c is a schematic diagram of the mobile trajectory planning effect illustrated in the embodiment of the present application;

[0021] Figure 3 A schematic diagram of scene light grouping for illustrating an embodiment of the present application;

[0022] Figure 4 A schematic diagram of the structure of a visual effect enhancement device provided in an embodiment of the present application;

[0023] Figure 5 For Figure 4 A schematic diagram of the structure of an electronic device corresponding to the visual effect enhancement device provided in the embodiment. DETAILED DESCRIPTION

[0024] In order to make the purpose, technical solution and advantages of the embodiments of the present invention clearer, the technical solution in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0025] The terms used in the embodiments of the present invention are only for the purpose of describing specific embodiments, and are not intended to limit the present invention. The singular forms "a", "said" and "the" used in the embodiments of the present invention and the appended claims are also intended to include plural forms, unless the context clearly indicates other meanings, and "multiple" generally includes at least two.

[0026] As used herein, the words "if" and "if" may be interpreted as "at the time of" or "when" or "in response to determining" or "in response to detecting", depending on the context. Similarly, the phrases "if it is determined" or "if (stated condition or event) is detected" may be interpreted as "when it is determined" or "in response to determining" or "when detecting (stated condition or event)" or "in response to detecting (stated condition or event)", depending on the context.

[0027] In addition, the step sequence in the following method embodiments is only an example and not a strict limitation.

[0028] In a virtual interactive scene (for example, a metaverse virtual world interactive scene), users can watch various actions of virtual targets in the scene, and some users can also control virtual characters to participate in it. In some live scenes of virtual entertainment programs, various sound effects and dynamic display effects are added to enable users to obtain better viewing effects. However, sometimes, the added sound effects (for example, background music added by a user himself) cannot match the dynamic display effects, or the dynamic display cannot be displayed dynamically according to the sound effects. Sometimes, even if the dynamic display can match the rhythm of the sound effects, due to the large number of sound sources in the virtual scene and the noisy sound, the consistency between the two is poor. The immersive experience obtained by users is limited. Therefore, a solution is needed that can enhance the visual enhancement display effect based on audio intensity and audio rhythm.

[0029] To facilitate understanding, specific embodiments will be used as examples below.

[0030] like Figure 1 The flowchart of the visual effect enhancement method provided in the embodiment of the present application is shown in FIG. Figure 1 As you can see, the specific steps include the following:

[0031] Step 101: Acquire audio data containing multiple notes.

[0032] Step 102: Analyze the audio intensity and audio rhythm corresponding to any note in the audio data.

[0033] Step 103: When rendering a multimedia video containing the audio data, a visual effect enhancement display task is performed on the display interface of the multimedia video according to the audio rhythm and the audio intensity; wherein the visual effect enhancement display task includes: a screen shaking enhancement display task and / or a lighting enhancement display task.

[0034] The audio data of multiple notes mentioned here can be understood as containing multiple notes at the same time in the original multimedia data. For example, quarter notes, eighth notes, etc. In some audio data, in addition to multiple notes, there are also singing, various music, etc. Therefore, when performing visual effect enhancement tasks, it is difficult to accurately identify the situation where multiple notes, singing, and music are contained at the same time, or the recognition result is inaccurate. This solution uses the audio intensity and audio rhythm in the audio data to identify and perform the corresponding visual enhancement display task, without the need to identify a single note, which can effectively improve the accuracy. In addition, since the sound heard by the user during use is also a mixed sound containing multiple singing and music playing, it is not easy for the user to distinguish which audio data is contained therein, but the user can directly hear the audio intensity and audio rhythm. Therefore, when performing visual effect enhancement display tasks based on audio intensity and audio rhythm, the user obtains a better visual and auditory synchronization effect.

[0035] When parsing audio data, the audio intensity and audio rhythm contained in the audio data can be parsed. Specifically, it can be based on beat parsing (for example, one beat one enhanced display mode) or frame rate parsing (parsing once for each video frame). Taking the screen shaking in the perspective enhancement effect as an example, when parsing according to the beat, the screen shakes once for each beat, the screen shaking curve is half of the complete sine cycle, and the screen shaking duration is the first half of each beat. In addition, the screen shaking intensity is related to the audio intensity. For example, the greater the audio intensity, the greater the screen shaking amplitude, and the smaller the audio intensity, the smaller the screen shaking amplitude. Of course, in practical applications, the intensity can be designed as needed, for example, setting the upper and lower limits of the screen shaking amplitude. The time domain data of the audio can be obtained through the AudioClip.GetData function, and the time domain data of each beat of the audio is analyzed by spectrum analysis and converted into frequency domain data. This step requires the use of Fourier transform, that is, decomposing the waveform data of the audio time domain into the superposition of sine waves of different frequencies and intensities. For this purpose, the C# library of DSPLib FFT fast Fourier transform is introduced. The intensity value of the audio is calculated based on the frequency domain data of the audio. Fast Fourier Transform (FFT) is a discrete Fourier transform (DFT) algorithm that can be completed in O(nlogn) time. It is triggered once per beat, and the current operation time for each beat of data is 3ms-4ms (editor environment test, mono). In actual implementation, further optimization can be made according to actual needs. For example, the waveform data of 1 / 4 audio can be sampled, so that the performance consumption can be reduced to 1 / 4 of the original; of course, the sampling frequency or sampling ratio can be selected according to actual application needs.

[0036] When frame rate-based analysis is used, the current music intensity value is obtained by calculating each frame in the real-time rendered and presented video. The screen shaking trajectory is obtained from the current virtual lens position to the end position after rendering based on the music intensity value. When performing trajectory planning, for example, the linear interpolation function Linear Interpolation can be used for trajectory planning, so that a continuous and stable virtual camera movement trajectory can be obtained. From the user's viewing experience, the screen shaking effect seen is more coherent, and there will be no trajectory deviation or offset.

[0037] It should be noted that when the multimedia data in the above embodiment is presented, it is not simply played directly, but presented in a real-time rendering manner, which means that when different audio data are matched with the picture, the visual effects presented are different (different screen shaking effects, different lighting effects).

[0038] Among them, audio intensity can be reflected in the visual effect enhancement display in many ways, for example: the stronger the audio intensity, the brighter the light brightness, the weaker the audio intensity, the dimmer the light brightness; the stronger the audio intensity, the longer or thicker the light column distance, the weaker the audio intensity, the more segmented or thin the light column distance; the stronger the audio intensity, the warmer the light color, the weaker the audio intensity, the colder the light color, and so on. The light brightness, length, thickness and other attributes are synchronized with the strength of the audio intensity. The faster the audio rhythm, the faster the light changes, the slower the audio rhythm, the slower the light changes, and the speed of light changes is synchronized with the audio rhythm.

[0039] In addition, since the presentation of the picture is adjusted by the parameters of the virtual camera in real-time rendering, the stronger the audio intensity, the greater the screen shaking amplitude, which means the longer the distance the virtual camera moves, and the weaker the audio intensity, the smaller the screen shaking amplitude, which means the shorter the distance the virtual camera moves. If you add perspective rotation or twisting to the screen shaking effect; the stronger the audio intensity, the greater the twisting or rotation amplitude, and the weaker the audio intensity, the smaller the twisting or rotation amplitude. The faster the audio rhythm, the faster the screen shaking changes, and the slower the audio rhythm, the slower the screen shaking changes. The speed of the screen shaking changes is synchronized with the audio rhythm.

[0040] In one or more embodiments of the present application, the visual effect enhancement display task is a screen shaking display enhancement display task, and when rendering a multimedia video containing the audio data, the visual effect enhancement display task is performed on the display interface according to the audio rhythm and the audio intensity, including: determining the screen shaking parameters according to the audio rhythm and the audio intensity; planning the movement trajectory and movement period of the virtual camera according to the screen shaking parameters and the target object in the display content; controlling the movement of the virtual camera relative to the target object based on the movement trajectory and the movement period, and performing the screen shaking display task on the display interface.

[0041] In practical applications, before each frame is presented, it is necessary to determine the screen shaking parameters based on the audio rhythm and audio intensity, including the screen shaking amplitude, screen shaking cycle, screen shaking angle, etc. These screen shaking parameters are used as part of the video frame rendering parameters. Furthermore, the movement trajectory and movement cycle of the virtual camera are planned based on these screen shaking parameters and the target object in the display content. When planning, the movement trajectory will limit the target object in the display content from moving out of the display range. The movement trajectory is a round-trip trajectory, that is, the starting trajectory and the recovery trajectory. The movement trajectories of these two trajectories may not be exactly the same, but the movement cycle is determined based on the audio rhythm, etc., and is generally the same. Generally speaking, within a movement cycle range, it may be necessary to render multiple frames continuously. During rendering, the target object is rendered in real time according to the continuous changes in the movement trajectory and the relative position relationship between the virtual camera and the target object, including changes in the pitch angle, distance, etc. that cause changes in the light of the target object (such as facial brightness).

[0042] By rationally planning the movement trajectory and movement cycle of the virtual camera, the rendered video display content is matched with the audio rhythm and audio intensity. It should be noted that when determining the screen shaking parameters based on the audio intensity and audio rhythm, and planning the movement trajectory of the virtual camera based on the screen shaking parameters, the display position and display effect of the target object should be fully considered (for example, the target object cannot exceed the display interface range).

[0043] In one or more embodiments of the present application, the movement trajectory includes: a starting trajectory and a recovery trajectory; when executing the screen shaking enhanced display task, if the target object does not move, the starting trajectory is the same as the recovery trajectory; if the target object moves, the center of the current display interface is used as the end point of the recovery trajectory and the recovery trajectory is generated.

[0044] Those skilled in the art should understand that the determination of whether the target object moves in the above embodiment is not absolute, that is, when the target object moves but the moving distance is less than the threshold, it can also be determined as not moving.

[0045] As mentioned above, when generating the movement trajectory of the virtual camera, the position and state of the target object will be taken into consideration. Because in some scenes, the target object or other virtual characters can be controlled by the user (for example, a virtual dance interactive scene). When planning the movement trajectory, the movement state of the target object must be considered. If the target object does not move in the current frame, the starting trajectory and the recovery trajectory can be exactly the same, that is, the starting point of the starting trajectory is the same as the end point of the recovery trajectory, the end point of the starting trajectory is the same as the starting point of the recovery trajectory, and the points on the trajectory completely overlap.

[0046] Of course, if the user controls the target object to move during the display of the content, in order to ensure that the target object can be fully displayed in the display content, it is necessary to adjust the starting trajectory or the recovery trajectory according to the adjustment time. For example, if the target object moves at the start of the screen shaking, the starting trajectory will be planned according to the position after the movement; if the target object moves at the end of the screen shaking, the recovery trajectory will be planned according to the position after the movement.

[0047] It should be noted that when planning the starting point and end point of the initial trajectory, as well as the starting point and end point of the recovery trajectory, the center of the display interface can be used as the starting point and end point. When the display interface is adjusted by the user, the positions of the starting point and end point remain unchanged, but the trajectory trend may change. In addition, the target object can also be used as the starting point and end point of the initial trajectory, as well as the starting point and end point of the recovery trajectory.

[0048] In one or more embodiments of the present application, the movement trajectory includes: a starting trajectory and a recovery trajectory; when executing the screen shaking enhanced display task, if the display viewing angle of the displayed content is adjusted, the center of the current display interface is used as the end point of the recovery trajectory and the recovery trajectory is generated, or, the center of the current interface is used as the end point of the starting trajectory and the starting trajectory is generated.

[0049] As mentioned above, when generating the movement trajectory of the virtual camera, the position and state of the target object will be taken into consideration. Because in some scenes, the display content can be controlled by the viewing user (for example, the interactive dance scene of the virtual live broadcast), for example, user A drags the display interface to the left, and user B rotates the display interface to the right. When planning the movement trajectory, it is necessary to combine the movement state of the display content. If the display content has not been moved in the current frame, the starting trajectory and the recovery trajectory can be exactly the same, that is, the starting point of the starting trajectory is the same as the end point of the recovery trajectory, and the end point of the starting trajectory is the same as the starting point of the recovery trajectory, and the points on the trajectory completely overlap.

[0050] Of course, if the user controls the movement of the displayed content during the presentation of the displayed content, in order to ensure that the target object can be fully displayed in the displayed content, the starting trajectory or the recovery trajectory needs to be adjusted according to the adjustment time. For example, if the displayed content is moved at the start of the screen shaking, the starting trajectory will be planned according to the position after the movement; if the displayed content is moved at the end of the screen shaking, the recovery trajectory will be planned according to the position after the movement.

[0051] In actual applications, the same multimedia video can be played on multiple different clients at the same time for different users to watch, which means that the viewing habits of different users are not exactly the same. If these users adjust the display content according to their own viewing habits or needs, it means that the multimedia video effects presented by the clients of different users are not exactly the same. For example, some are presented relative to the left side of the target object, while others are presented relative to the right side of the target object. Each client will perform real-time rendering according to its actual display needs, thereby achieving a rendering effect that is different for each person and meeting the user's needs for synchronized auditory and visual enhanced display.

[0052] It should be noted that when planning the starting point and end point of the initial trajectory, as well as the starting point and end point of the recovery trajectory, the center of the display interface can be used as the starting point and end point. When the display interface is adjusted by the user, the positions of the starting point and end point remain unchanged, but the trajectory trend may change. In addition, the target object can also be used as the starting point and end point of the initial trajectory, as well as the starting point and end point of the recovery trajectory.

[0053] In one or more embodiments of the present application, the movement trajectory includes: a linear reciprocating trajectory, a viewing angle adjustment trajectory centered on the target object, and a rotation trajectory centered on the target object.

[0054] like Figure 2a , 2b 2c are schematic diagrams of the mobile trajectory planning effect illustrated in the embodiment of the present application. Figure 2a As can be seen in the figure, the movement trajectory of the virtual camera is a straight-line reciprocating motion relative to the target object. Generally, it is necessary to ensure that the target object does not move during the rendering of the movement trajectory.

[0055] like Figure 2b As shown, when planning the movement trajectory, the viewing angle adjustment is planned with the target object as the center. That is, when the target object moves, the starting point and end point of the movement trajectory of the virtual camera are set to the position of the target object. Depending on the movement mode of the target object, the generated movement trajectory is different, and a non-linear movement trajectory may be obtained.

[0056] like Figure 2c As shown, the target object can be used as the center to plan the rotation trajectory. When planning the rotation trajectory, the starting trajectory and the recovery trajectory can be the same or different. When they are different, the starting trajectory and the recovery trajectory can form a closed loop to form a complete ring.

[0057] It should be noted that the above-mentioned movement trajectory planning method is only used as an example. In actual application, it can be adjusted as needed, or multiple planning methods can be integrated together.

[0058] In one or more embodiments of the present application, the visual effect enhancement is a lighting enhancement display effect; when rendering a multimedia video containing the audio data, according to the audio rhythm and the audio intensity, a visual effect enhancement display task is performed on the display interface of the multimedia video, including: grouping scene lights; according to the audio rhythm and the audio intensity, determining scene lighting parameters of each of the groups; wherein the scene lighting parameters include: at least one of scene light color, scene light duty cycle and scene light direction; based on the scene light parameters, performing a lighting enhancement display task on the display interface.

[0059] When grouping lights, you can group them according to the distance and position relationship with the target object. For example, the two sides of the target object are adjacent to each other in one group, and the two sides are next to each other in another group, etc. You can also group them according to the color of the lights or the length of the light beams. You can also group them according to the matching notes, that is, different groups of lights correspond to different notes in the audio data, and there may be a staggered display effect. The scene lights mentioned here refer to the original existing parts of the display content, which are part of the original display content and are rendered and displayed in real time according to the lighting parameters.

[0060] When determining the scene lighting parameters, it is determined based on the audio intensity and audio rhythm. For example, the stronger the audio intensity, the brighter the light brightness, the weaker the audio intensity, the dimmer the light brightness; the stronger the audio intensity, the longer or thicker the light column distance, the weaker the audio intensity, the more segmented or thin the light column distance; the stronger the audio intensity, the warmer the light color, the weaker the audio intensity, the colder the light color, and so on. The light brightness, length, thickness and other attributes are synchronized with the strength of the audio intensity. The faster the audio rhythm, the faster the light changes, the slower the audio rhythm, the slower the light changes, and the speed of light changes is synchronized with the audio rhythm.

[0061] In addition, the lighting parameters also include the scene lighting direction, for example, different lights face different directions. Another example is that the lighting direction can be changed, and the lighting rotation effect can be added according to the actual situation.

[0062] By rationally planning the scene light color, scene light duty cycle and scene light direction, the rendered video display content is matched with the scene light color, scene light duty cycle and scene light direction. It should be noted that when determining the lighting parameters based on the audio intensity and audio rhythm, and planning the lighting effects based on the lighting parameters, the display position and display effects of the target object should be fully considered (for example, the rendering of the target object reflects the influence of the lighting effect on the target object, including the occlusion relationship).

[0063] like Figure 3 This is a schematic diagram of scene light grouping for example in the present application. Figure 3 As you can see, the scene lights are divided into Group A and Group B. The light directions of scene lights in different groups are different, and the length and thickness of the light beams can be set as needed. In addition, the directions of scene lights are diverse, and some may affect the light effects on the faces of the characters in the center of the stage. At this time, you can set appropriate directional light according to the scene light parameters to illuminate the characters, showing the scene light change effect.

[0064] In one or more embodiments of the present application, the grouping of scene lights includes: grouping the scene lights according to the type of note contained in the audio data.

[0065] As mentioned above, the audio data contains a variety of notes. If the note types can be accurately identified, the scene lights can be grouped according to the note types, so that different groups have different light display effects.

[0066] When creating groups according to notes, you can group them according to the length of the beats. For example, short beats correspond to the outer groups, or to the groups with short light beams; long beats correspond to the inner groups, or to the groups with long light beams. Of course, you can also establish a grouping correspondence between different beats and different light colors. This is only an example and does not constitute a limitation to the technical solution of this application. In actual applications, users can choose according to their own needs.

[0067] It should be noted that although the lights are grouped, the frame is still the basic rendering unit, that is, lights in different groups in the same frame are rendered at the same time.

[0068] In one or more embodiments of the present application, the performing of a lighting enhancement display task on the display interface based on the scene lighting parameters includes: configuring a directional light corresponding to each of the scene lights according to the scene lighting parameters, the parameters of the directional light corresponding to the scene lighting parameters; and performing a lighting enhancement display task on the display interface based on the parameters of the directional light.

[0069] In actual applications, there are multiple scene light sources, and some scene lights will affect the target object in the scene content. For example, a light and dark effect will be produced on the face of the target object. If the scene light flickers or changes color, the face of the target object will also have a dynamic change effect of flickering or the facial color will change accordingly. Since the scene light parameters are generated based on the audio intensity and audio rhythm, when there are multiple scene lights, the effects of each scene light may be different. In order to make the rendering effect of the target object more realistic, it is necessary to calculate the corresponding directional light parameters for each scene light parameter, and then, based on the directional light rendering, a more realistic display effect of the target object is obtained. When calculating the directional light parameters, the calculation is based on the light parameters of the scene light obtained in real time, and rendering is based on the scene light and directional light.

[0070] The directional light mentioned here can be, for example, DirectionalLight. By introducing DirectionalLight, the scene light is simulated to produce light and dark effects on the character. Because when the user watches the video, the focus is mainly on the target object, and because the scene light is rendered in real time following the audio intensity and audio rhythm, the impact of the scene light on the target object cannot be reflected only through the scene light. Therefore, after determining the lighting parameters, the corresponding directional light parameters (such as direction, brightness, color, etc.) should be determined according to the lighting parameters. Specifically, according to the scene light color, scene light working cycle, scene light direction, brightness, etc. in the lighting parameters, the impact on the target object and other scene objects is judged, such as the brightness and color changes of the target object, so as to achieve these effects through directional light rendering. For example, DirectionalLight corresponds to each group of lights in the rhythm group, including color and direction, and then renders the current frame.

[0071] For example, after determining the scene lighting parameters, the parameters of the directional light are further determined. Including determining the direction of the directional light: First, the direction of the directional light needs to be determined, which is usually defined as the direction of the sunlight. The position and direction of the sun can be calculated using a daylight model or a sky model. This can estimate the position of the sun based on parameters such as geographic location, date and time, and convert it into a direction vector in the scene. In the scenario of the present application, the direction of the directional light can be determined based on the various scene lighting parameters (for example, the direction of the scene light).

[0072] In addition, the intensity of the directional light needs to be estimated: Once the direction of the directional light is determined, the intensity of the directional light needs to be estimated. The intensity can be estimated based on the position of the scene lights and the scene lighting conditions. For example, if the scene light source is at a high position (such as above the target object's head), the intensity of the directional light may be strong; if there is smoke in the air above to enhance the atmosphere, the intensity of the directional light may be weakened.

[0073] In addition, color and other properties need to be considered: in addition to direction and intensity, the color and other properties of directional light can also be considered. For example, the color of scene lights may be affected by scattering and present different tones. You can also consider the properties of directional light such as soft and hard shadows, projection quality, etc. to enhance the rendering effect. Through the above scheme, directional light can solve the problem that the scene light display process cannot reflect the effect of the scene light on the display effect of the target object. As a result, when rendering video frames, not only the screen shaking effect and scene light dynamic effect are rendered, but also the directional light can be used to render the effect of each scene light on the target object.

[0074] In one or more embodiments of the present application, the parsing of the audio intensity and audio rhythm corresponding to any note in the audio data includes: parsing the frequency spectrum data of any note in the audio data corresponding to each frame of the multimedia video to be presented to obtain sub-audio intensities and sub-audio rhythms of multiple notes; and generating the audio intensity and audio rhythm of the audio data based on the sub-audio intensities and sub-audio rhythms of the multiple notes.

[0075] As mentioned above, when parsing, it is necessary to calculate the parsing in real time according to the frame rate. There are many ways to parse. For example, one way is to use the AudioSource.spectrumdata function to obtain audio data, and convert the audio data into time domain audio, wherein the time domain audio data contains multiple sub-audio intensities, and then sum the multiple sub-audio intensities to obtain the audio intensity of the audio data. The audio rhythm is determined in the same way. Another way is to use the AudioClip.getdata function to obtain audio data, and process the audio data to obtain time domain audio, and Fourier transform the audio time domain data to obtain frequency domain data, wherein the frequency domain data contains multiple sub-audio intensities, and then sum the multiple sub-audio intensities to obtain the audio intensity of the audio data. When the AudioClip.getdata function is used to obtain audio data and calculate the audio intensity, it will not be affected by the volume. In other words, regardless of the volume, the audio intensity obtained by using the AudioClip.getdata function to obtain audio data is real, and the audio intensity remains unchanged when the volume is turned up or down.

[0076] Based on the same idea, the embodiment of the present application also provides a visual effect enhancement device. Figure 4 A schematic diagram of the structure of a visual effect enhancement device provided in an embodiment of the present application. Figure 4 As can be seen, the device includes:

[0077] The acquisition module 41 is used to acquire audio data containing multiple notes.

[0078] The parsing module 42 is used to parse the audio intensity and audio rhythm corresponding to any note in the audio data.

[0079] The execution module 43 is used to perform a visual effect enhancement display task on the display interface of the multimedia video when rendering the multimedia video containing the audio data according to the audio rhythm and the audio intensity; wherein the visual effect enhancement display task includes: a screen shaking enhancement display task and / or a lighting enhancement display task.

[0080] Optionally, the execution module 43 is used to determine the screen shaking parameters according to the audio rhythm and the audio intensity; plan the movement trajectory and movement period of the virtual camera according to the screen shaking parameters and the target object in the display content; and control the movement of the virtual camera relative to the target object based on the movement trajectory and the movement period, and perform the screen shaking enhanced display task on the display interface.

[0081] Optionally, the movement trajectory includes: a starting trajectory and a recovery trajectory;

[0082] The execution module 43 is used to, when executing the screen shaking enhanced display task, if the target object does not move, make the starting trajectory the same as the recovery trajectory; if the target object moves, take the center of the current display interface as the end point of the recovery trajectory and generate the recovery trajectory.

[0083] Optionally, the movement trajectory includes: a starting trajectory and a recovery trajectory;

[0084] The execution module 43 is used to, when executing the screen shaking enhanced display task, if the display viewing angle of the display content is adjusted, take the center of the current display interface as the end point of the recovery trajectory and generate the recovery trajectory, or take the center of the current interface as the end point of the starting trajectory and generate the starting trajectory.

[0085] Optionally, the movement trajectory includes: a linear reciprocating trajectory, a viewing angle adjustment trajectory centered on the target object, and a rotation trajectory centered on the target object.

[0086] Optionally, the visual effect enhancement is a light enhancement display effect;

[0087] The execution module 43 is used to group the scene lights; determine the scene light parameters of each group according to the audio rhythm and the audio intensity; wherein the scene light parameters include: at least one of the scene light color, the scene light duty cycle and the scene light direction; and based on the scene light parameters, perform a light enhancement display task on the display interface.

[0088] Optionally, the grouping of scene lights includes: grouping the scene lights according to the types of notes contained in the audio data.

[0089] Optionally, the execution module 43 is used to configure the directional light corresponding to each of the scene lights according to the scene light parameters, and the parameters of the directional light correspond to the scene light parameters; and based on the parameters of the directional light, perform a light enhancement display task on the display interface.

[0090] Optionally, the analysis module 42 is used to analyze the spectrum data of any note in the audio data corresponding to each frame of the multimedia video to be presented to obtain sub-audio intensities and sub-audio rhythms of multiple notes; and generate audio intensity and audio rhythm of the audio data based on the sub-audio intensities and sub-audio rhythms of the multiple notes.

[0091] Based on the above embodiment, audio data containing multiple notes is obtained; the audio intensity and audio rhythm corresponding to any note in the audio data are analyzed; when rendering a multimedia video containing the audio data, a visual effect enhancement display task is performed on the display interface of the multimedia video according to the audio rhythm and the audio intensity. With this solution, there is no need to accurately identify the note content in the audio data, and the entire audio data can be directly processed as audio data, that is, the visual enhancement display effect (such as screen shaking and scene lighting) is controlled according to the audio intensity and audio rhythm, so that the user can obtain a more consistent audio-visual experience.

[0092] In one possible design, the above Figure 4 The structure of the visual effect enhancement device shown can be implemented as an electronic device. Figure 5 As shown, the electronic device may include: a processor 51 and a memory 52. ​​The memory 52 stores executable code, and when the executable code is executed by the processor 51, the processor 51 can at least implement the visual effect enhancement method provided in the above embodiment. The structure of the electronic device may also include a communication interface 53 for communicating with other devices or communication networks.

[0093] In addition, an embodiment of the present invention provides a non-transitory machine-readable storage medium, on which executable code is stored. When the executable code is executed by a processor in a server or a computer, the processor executes the operations provided in the above embodiments. Figure 1 Corresponding visual effect enhancement method.

[0094] The device embodiments described above are merely illustrative, wherein the modules described as separate components may or may not be physically separated. Some or all of the modules may be selected according to actual needs to achieve the purpose of the present embodiment. Those of ordinary skill in the art may understand and implement the present invention without creative effort.

[0095] Through the description of the above embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by adding a necessary general hardware platform, and of course can also be implemented by combining hardware and software. Based on such an understanding, the above technical solution can essentially or contribute to the part in the form of a computer product, and the present invention can be implemented in the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program codes.

[0096] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for enhancing visual effects, characterized in that: The method comprises: Get audio data containing multiple notes; Analyze the audio intensity and audio rhythm corresponding to any note in the audio data; When rendering a multimedia video containing the audio data, a visual effect enhancement display task is performed on the display interface of the multimedia video according to the audio rhythm and the audio intensity; wherein the visual effect enhancement display task includes: a screen shaking enhancement display task and / or a lighting enhancement display task.

2. The method according to claim 1, characterized in that The visual effect enhancement display task is a screen shaking display enhancement display task, and when rendering the multimedia video containing the audio data, the visual effect enhancement display task is performed on the display interface according to the audio rhythm and the audio intensity, including: Determining a screen shaking parameter according to the audio rhythm and the audio intensity; Planning the movement trajectory and movement cycle of the virtual camera according to the screen shaking parameters and the target object in the display content; The virtual camera is controlled to move relative to the target object based on the movement trajectory and the movement period, and a screen shaking enhanced display task is performed on the display interface.

3. The method according to claim 2, characterized in that The moving trajectory includes: a starting trajectory and a recovery trajectory; In the case of executing the screen shaking enhanced display task, if the target object does not move, the starting trajectory is the same as the recovery trajectory; If the target object moves, the center of the current display interface is used as the end point of the restoration trajectory and the restoration trajectory is generated.

4. The method according to claim 2, characterized in that: The moving trajectory includes: a starting trajectory and a recovery trajectory; When executing the screen shaking enhanced display task, if the display viewing angle of the display content is adjusted, the center of the current display interface is used as the end point of the recovery trajectory and the recovery trajectory is generated, or the center of the current interface is used as the end point of the starting trajectory and the starting trajectory is generated.

5. The method according to claim 3 or 4, characterized in that: The moving trajectory includes: a linear reciprocating trajectory, a viewing angle adjustment trajectory centered on the target object, and a rotation trajectory centered on the target object.

6. The method according to claim 1, characterized in that The visual effect enhancement is light enhancement display effect; The method of performing a visual effect enhancement display task on a display interface of the multimedia video according to the audio rhythm and the audio intensity when rendering the multimedia video containing the audio data comprises: Group scene lights; Determine the scene light parameters of each of the groups according to the audio rhythm and the audio intensity; wherein the scene light parameters include: at least one of scene light color, scene light duty cycle and scene light direction; Based on the scene lighting parameters, a lighting enhancement display task is performed on the display interface.

7. The method according to claim 6, characterized in that The grouping of scene lights includes: The scene lights are grouped according to the types of the notes contained in the audio data.

8. The method according to claim 6, characterized in that The performing the lighting enhancement display task on the display interface based on the scene lighting parameter includes: Configuring directional lights corresponding to the scene lights according to the scene light parameters, wherein the parameters of the directional lights correspond to the scene light parameters; Based on the parameters of the directional light, a lighting enhancement display task is performed on the display interface.

9. The method according to claim 1, characterized in that: The step of analyzing the audio intensity and audio rhythm corresponding to any note in the audio data includes: Parsing the frequency spectrum data of any note in the audio data corresponding to each frame of the multimedia video to be presented to obtain sub-audio intensities and sub-audio rhythms of multiple notes; Based on the sub-audio intensities and sub-audio rhythms of the plurality of notes, an audio intensity and an audio rhythm of the audio data are generated.

10. A visual effect enhancement device, characterized in that: The device comprises: An acquisition module, used to acquire audio data containing multiple notes; An analysis module, used for analyzing the audio intensity and audio rhythm corresponding to any note in the audio data; An execution module is used to perform a visual effect enhancement display task on the display interface of the multimedia video when rendering the multimedia video containing the audio data according to the audio rhythm and the audio intensity; wherein the visual effect enhancement display task includes: a screen shaking enhancement display task and / or a lighting enhancement display task.

11. An electronic device comprising a processor and a memory, wherein the memory stores at least one instruction, at least one program, a code set or an instruction set, and the at least one instruction, at least one program, a code set or an instruction set is loaded and executed by the processor to implement the method according to any one of claims 1 to 9.

12. A computer-readable medium having stored thereon at least one instruction, at least one program, code set or instruction set, wherein the at least one instruction, at least one program, code set or instruction set is loaded and executed by a processor to implement the method according to any one of claims 1 to 9.

Images