Light dynamic display method in virtual scene, computer device and product

Abstract

The application relates to the computer technical field, and relates to a light dynamic display method in a virtual scene, a computer device and a computer program product, which can effectively consider the design efficiency of light actions and the adaptability of the light actions and song audio. The method comprises the following steps: acquiring the pitch of each audio frame in a plurality of audio frames in audio; determining a plurality of light control coefficients according to the pitch of each audio frame, wherein the light control coefficients are used for controlling the change of light in a virtual scene; the plurality of light control coefficients form a light control sequence, and the light control sequence is associated with the plurality of audio frames in the audio; and adjusting the change of pre-set light in the virtual scene according to the light control sequence.

Description

Technical Field

[0001] This application relates to the field of computer technology, and in particular to a method for dynamic display of lighting in a virtual scene, a computer device, and a computer program product. Background Technology

[0002] With the development of computer technology, lighting effects can be added to relevant scenes during audio playback to enrich the overall audio display effect, such as displaying lighting effects during online audio / video live streaming or offline audio playback.

[0003] In related technologies, designers can design different lighting actions for specified audio, or pre-stored lighting actions can be directly applied to relevant scenes when playing audio.

[0004] However, of the above methods, the former requires designers to have professional lighting design experience and combine it with the characteristics of different audio to design targeted lighting actions, making it difficult to design lighting actions for a large number of audio files. While the latter can be set up quickly, the lighting actions are often difficult to adapt to the audio being played. It is evident that the relevant technologies, when controlling the display of lighting actions, struggle to balance the efficiency of lighting action design with the compatibility of lighting actions with the playing audio. Summary of the Invention

[0005] Therefore, it is necessary to provide a method, computer equipment, and computer program product for dynamic lighting display in a virtual scene to address the aforementioned technical problems.

[0006] In a first aspect, this application provides a method for dynamic lighting display in a virtual scene, the method comprising:

[0007] Get the pitch of each audio frame in multiple audio frames;

[0008] Based on the pitch of each audio frame, multiple lighting control coefficients are determined, which are used to control the changes in lighting in the virtual scene; the multiple lighting control coefficients form a lighting control sequence, which is associated with multiple audio frames in the audio.

[0009] According to the lighting control sequence, the pre-set lighting changes in the virtual scene are adjusted.

[0010] In one embodiment, adjusting the pre-set changes in the lighting in the virtual scene according to the lighting control sequence includes:

[0011] Multiple video frames are acquired, each of which is used to display the static representation of pre-set lights in a virtual scene, and each video frame is associated with the light control sequence.

[0012] The target video is generated based on the lighting control sequence and the static display of the lights in the virtual scene.

[0013] In one embodiment, generating the target video based on the lighting control sequence and the static display of the lights in the virtual scene includes:

[0014] Based on the lighting control sequence, the duration of the static display of the lights in the virtual scene is adjusted to obtain the duration of each video frame;

[0015] Based on the duration of each video frame, an adjusted target video frame is obtained, and based on multiple target video frames, the target video is generated, wherein the target video is the adjusted lighting in the virtual scene.

[0016] In one embodiment, adjusting the duration of the static display of the light in the virtual scene according to the light control sequence to obtain the duration of each video frame includes:

[0017] Based on the association between each video frame and the lighting control sequence, obtain the lighting control coefficient associated with each video frame from the lighting control sequence;

[0018] The increment of the light change progress in each video frame is determined by multiplying the associated light control coefficient with the predetermined video frame duration.

[0019] Based on the change progress increment, the pre-set change progress of the lights in each video frame is adjusted to obtain the target change progress of each video frame, which is then used as the duration of each video frame.

[0020] In one embodiment, adjusting the pre-set light change rate in each video frame according to the change rate increment to obtain the target change rate for each video frame includes:

[0021] Obtain the display cycle of the light; the display cycle is the time taken for the light to display once at a preset change speed;

[0022] The summation of the change progress increment and the pre-set light change progress in each video frame is determined, and the ratio of the summation result to the display period is determined as the target change progress for each video frame.

[0023] In one embodiment, the virtual scene includes at least one virtual light source, and obtaining the adjusted target video frame based on the duration of each video frame includes:

[0024] The position parameters, lighting rendering parameters, and on / off status of each virtual light source are determined for the duration of each video frame; the on / off status indicates whether the corresponding virtual light source is in an on or off state.

[0025] Based on the position parameters, lighting rendering parameters, and on / off status of each virtual light source in each video frame, the video frame is rendered to obtain the adjusted target video frame.

[0026] In one embodiment, before adjusting the preset lighting changes in the virtual scene according to the lighting control sequence, the method further includes:

[0027] Determine the song style of the audio;

[0028] At least one light source is selected from the set of lights associated with the song style as a preset light source.

[0029] In one embodiment, adjusting the pre-set changes in the lighting in the virtual scene according to the lighting control sequence includes:

[0030] According to the pre-set display order of various lights, various lights are displayed sequentially in the virtual scene. During the light display process, the changes of the currently displayed light in the virtual scene are adjusted according to the light control sequence until all the light control coefficients in the light control sequence have been used to adjust the changes of the lights in the virtual scene.

[0031] In one embodiment, determining the lighting control coefficient based on the pitch of the audio frame includes:

[0032] The target power values ​​of each audio frame are summed, and the lighting control coefficient of the audio frame is obtained based on the summation result; the target power value is the power value of the audio frame within the human hearing frequency range.

[0033] Secondly, this application also provides a computer device. The computer device includes a memory and a processor, the memory storing a computer program, and the processor executing the computer program to perform the following steps:

[0034] Get the pitch of each audio frame in multiple audio frames;

[0035] Based on the pitch of each audio frame, multiple lighting control coefficients are determined, which are used to control the changes in lighting in the virtual scene; the multiple lighting control coefficients form a lighting control sequence, which is associated with multiple audio frames in the audio.

[0036] According to the lighting control sequence, the pre-set lighting changes in the virtual scene are adjusted.

[0037] Thirdly, this application also provides a computer program product. The computer program product includes a computer program that, when executed by a processor, performs the following steps:

[0038] Get the pitch of each audio frame in multiple audio frames;

[0039] Based on the pitch of each audio frame, multiple lighting control coefficients are determined, which are used to control the changes in lighting in the virtual scene; the multiple lighting control coefficients form a lighting control sequence, which is associated with multiple audio frames in the audio.

[0040] According to the lighting control sequence, the pre-set lighting changes in the virtual scene are adjusted.

[0041] The aforementioned method, computer device, and computer program product for dynamic lighting display in a virtual scene can acquire the pitch of each audio frame in multiple audio frames and determine multiple lighting control coefficients based on the pitch of each audio frame. These lighting control coefficients are used to control changes in the lighting within the virtual scene. The multiple lighting control coefficients form a lighting control sequence, which is associated with the multiple audio frames in the audio. Furthermore, the changes in pre-set lighting within the virtual scene can be adjusted according to the lighting control sequence. In this application, by determining lighting control coefficients based on the pitch of multiple audio frames, forming a lighting control sequence based on these coefficients, and using this action control sequence to adjust changes in the lighting within the virtual scene, the lighting in the virtual scene can change in tandem with changes in audio pitch. This matches visual lighting changes with auditory audio pitch changes, quickly obtaining lighting that is compatible with the audio, effectively balancing lighting design efficiency and the compatibility between lighting and audio. Attached Figure Description

[0042] Figure 1 This is a flowchart illustrating a method for dynamically displaying lights in a virtual scene in one embodiment;

[0043] Figure 2 This is a schematic diagram of lighting in a virtual scene in one embodiment;

[0044] Figure 3 This is a flowchart illustrating one step in generating a target video in one embodiment;

[0045] Figure 4 This is a flowchart illustrating a method for dynamically displaying lights in a virtual scene, as described in another embodiment.

[0046] Figure 5 This is a schematic diagram of a process for adjusting the pace of light changes in another embodiment;

[0047] Figure 6 This is an internal structural diagram of a computer device in one embodiment;

[0048] Figure 7 This is an internal structural diagram of another computer device in one embodiment. Detailed Implementation

[0049] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0050] In one embodiment, such as Figure 1 As shown, a method for dynamic lighting display in a virtual scene is provided. This embodiment illustrates the application of this method to a server. It is understood that this method can also be applied to a terminal, or to a system including both a terminal and a server, and is implemented through interaction between the terminal and the server. In this embodiment, the method includes the following steps:

[0051] S101, obtain the pitch of each audio frame in multiple audio frames in the audio.

[0052] For example, the audio can be audio with a corresponding melody, such as instrumental music or song audio with lyrics.

[0053] In practice, lighting can be configured in a virtual scene for audio. This virtual scene can be a two-dimensional image or a three-dimensional space constructed by a computer application. The virtual scene may include lighting, or it may include one or more virtual objects. For example, audio or video applications can display the corresponding virtual scene and the lighting within that virtual scene when playing audio. Alternatively, it could be a game scene or a virtual space for live video and / or audio streaming (such as an online virtual concert).

[0054] In this step, after determining the audio of the virtual scene lights to be configured, the audio can be processed into frames. Based on the result of the frame processing, multiple audio frames are obtained, and then the pitch of each audio frame can be obtained.

[0055] For example, real-time audio can be segmented into frames, and corresponding audio frames can be extracted according to a preset frame interval or time interval, then the pitch of the extracted audio frames can be determined. Alternatively, upon detecting a trigger event associated with virtual scene lighting rendering, the pitch of the current audio frame can be obtained. Or, the pitch of each individual audio frame can be obtained; that is, for each audio frame obtained, its pitch is acquired accordingly. It can be understood that the pitches of multiple audio frames in the audio can be obtained in real-time or pre-acquired.

[0056] S102, based on the pitch of each audio frame, determine multiple lighting control coefficients, which are used to control the changes in lighting in the virtual scene; multiple lighting control coefficients form a lighting control sequence, which is associated with multiple audio frames in the audio.

[0057] As an example, changes in lighting in a virtual scene can refer to the dynamic changes in lighting within a two-dimensional plane or three-dimensional space. Lighting in a virtual scene can be achieved using virtual light sources. Figure 2 The diagram illustrates the lighting in a virtual scene at a certain moment; the changes in the lighting may include, but are not limited to, changes in lighting parameters such as brightness, color, or illumination angle.

[0058] The lighting control coefficient can be used to control changes in lighting. In one embodiment, the lighting control coefficient can be used to control the speed of changes in lighting. For example, the speed of change can be the speed of change of at least one of the following lighting parameters: brightness, color, illumination angle, and switching speed of virtual light sources.

[0059] In practical applications, after obtaining the pitch of each audio frame in multiple audio frames, the corresponding light control coefficient can be determined for each audio frame pitch. Then, a light control sequence can be formed based on the light control coefficients corresponding to the pitches of multiple audio frames.

[0060] It is understandable that the lighting control sequence is associated with multiple audio frames in the audio. When the content of an audio frame changes, the pitch of the audio frame and the corresponding lighting control coefficient can also change. For example, the lighting control coefficient can be positively correlated with the pitch of the audio frame or negatively correlated with the pitch of the audio frame; alternatively, multiple pitch ranges can be set, and each audio frame pitch range can include multiple pitches. A corresponding lighting control coefficient can be configured for each pitch range, wherein the lighting control coefficient is positively or negatively correlated with the size of the pitch range.

[0061] S103, adjust the changes of pre-set lights in the virtual scene according to the lighting control sequence.

[0062] Specifically, the lights to be displayed in the virtual scene can be preset. For example, the light type, light color and light change speed of the lights to be displayed in the virtual scene can be set.

[0063] After determining the lighting control sequence associated with the pitch of multiple audio frames, the lighting in the virtual scene can be adjusted sequentially using multiple lighting control coefficients based on the order of the lighting control coefficients in the lighting control sequence. This allows the lighting in the virtual scene to change accordingly as the lighting control coefficients in the lighting control sequence change.

[0064] Since the lighting control coefficients are determined based on the pitch of the audio frames, in other words, changes in the lighting in the virtual scene can change accordingly with changes in the pitch of the audio melody. Specifically, as the audio playback progress changes, the pitch of the audio frame at the current playback progress can also change. Correspondingly, the lighting control coefficients associated with the pitch of the audio frame will also change, ultimately affecting the changes in the lighting in the virtual scene. Thus, as the audio playback progress changes, the brightness of the lights in the virtual scene also changes in tandem with the change in the audio pitch, matching the visual changes in lighting with the auditory changes in audio pitch, thereby quickly obtaining lighting actions that are adapted to the audio.

[0065] In this embodiment, the pitch of each audio frame in multiple audio frames can be obtained, and multiple lighting control coefficients can be determined based on the pitch of each audio frame. These lighting control coefficients are used to control changes in the lighting in the virtual scene. Multiple lighting control coefficients form a lighting control sequence, which is associated with the multiple audio frames in the audio. Furthermore, the changes in the pre-set lighting in the virtual scene can be adjusted according to the lighting control sequence. In this application, by determining lighting control coefficients based on the pitch of multiple audio frames, forming a lighting control sequence based on these coefficients, and using this action control sequence to adjust changes in the lighting in the virtual scene, the lighting in the virtual scene can change in tandem with changes in audio pitch. This matches the visual changes in lighting with changes in audio pitch, quickly obtaining lighting that is compatible with the audio, effectively balancing lighting design efficiency and the compatibility between lighting and audio.

[0066] In one embodiment, such as Figure 3 As shown, adjusting the pre-set lights in a virtual scene according to the lighting control sequence can include the following steps:

[0067] S301, acquire multiple video frames. Each video frame is used to display the static representation of pre-set lights in the virtual scene. Each video frame is associated with a light control sequence.

[0068] In practice, lighting changes in a virtual scene can be displayed through lighting videos. Specifically, when displaying lights in a virtual scene, multiple video frames can be played continuously. Each video frame is used to display the static display of pre-set lights in the virtual scene. Dynamic changes in lights in the virtual scene can be achieved by continuously displaying multiple video frames. Correspondingly, adjusting the changes of pre-set lights in the virtual scene can also be achieved by adjusting the lights displayed in multiple video frames in the lighting video.

[0069] For example, if 24 video frames are played continuously per second, with each frame representing 1 / 24 of the total frame, the video will finish playing after one second. However, if the video frame content is adjusted so that each frame represents 1 / 48 of the total frame, the video will finish playing after two seconds. The latter method results in a slower playback speed compared to the former.

[0070] In this step, multiple video frames can be acquired. Each video frame displays the static representation of pre-set lights in a virtual scene. The content displayed in different video frames can be different, such as different light displays. Of course, there can also be the same video frames, such as displaying one or more lights in a loop.

[0071] In this system, each of the multiple video frames is associated with a lighting control coefficient in a lighting control sequence. In one embodiment, the multiple video frames can be sequentially associated with multiple lighting control coefficients in the lighting control sequence. The association can be sequential, for example, consecutive video frames a, b, and c can be associated with consecutive lighting control coefficients k1, k2, and k3 in the lighting control sequence. Alternatively, the multiple video frames can be sequentially associated with multiple spaced lighting control coefficients in the lighting control sequence.

[0072] S302, Generate the target video based on the lighting control sequence and the static display of the lights in the virtual scene.

[0073] After determining the multiple video frames corresponding to the pre-set lights in the virtual scene, the static display content of the lights in the virtual scene in the multiple video frames can be determined based on the multiple light control coefficients in the light control sequence, and the target video can be generated.

[0074] In this embodiment, a target video can be generated based on the lighting control sequence and the static display of lights in the virtual scene in multiple video frames, thereby enabling rapid adjustment of lighting changes in the virtual scene.

[0075] In one embodiment, generating a target video based on a lighting control sequence and the static representation of lights in a virtual scene may include the following steps:

[0076] Based on the lighting control sequence, the duration of the static display of the lights in the virtual scene is adjusted to obtain the duration of each video frame; based on the duration of each video frame, the adjusted target video frame is obtained, and based on multiple target video frames, a target video is generated, which is the adjusted lighting in the virtual scene.

[0077] The duration of a light's static display in a virtual scene can be understood as the rate of change of the light within that scene, while the duration of a video frame can be understood as the rate of change of the light within that frame. For example, if the duration of a light's static display in a virtual scene is extended (e.g., if the rate of change of the light within the virtual scene is reduced), then at the same playback speed, the rate of change of the light displayed in a video frame will decrease.

[0078] In this embodiment, the lighting control coefficient in the lighting control sequence can be used as a parameter to adjust the speed of light change in the virtual scene. When displaying the changes of light in the virtual scene through multiple consecutive video frames, the duration of the static display of light in the virtual scene can be adjusted according to the lighting control sequence to obtain the duration of each video frame.

[0079] After obtaining the duration of each video frame, the adjusted static display of the virtual scene in each video frame can be determined based on the duration of each video frame. Then, video rendering can be performed to obtain the corresponding target video frame, and a target video can be generated based on multiple target video frames. The static display of the lights in each target video frame is the content adjusted based on the light control sequence. It can be understood that the target video generated from multiple target video frames can display the changed and adjusted lights in the virtual scene.

[0080] In this embodiment, the duration of each video frame is determined according to the lighting control sequence, and the adjusted target video frame is obtained based on the duration of each video frame to generate the target video, thereby realizing the fine adjustment of the lighting changes in the virtual scene and adapting the lighting changes in the virtual scene to the frame-level pitch changes.

[0081] In one embodiment, adjusting the duration of static display of lights in a virtual scene according to a lighting control sequence to obtain the duration of each video frame may include the following steps:

[0082] S401, based on the association between each video frame and the lighting control sequence, obtain the lighting control coefficient associated with each video frame from the lighting control sequence.

[0083] In this step, after determining multiple video frames and lighting control sequences, for each video frame, the lighting control coefficient associated with that video frame can be obtained from the lighting control sequence based on the association between the video frame and the lighting control sequence.

[0084] In one embodiment, an audio frame associated with a video frame can be determined, and the light control coefficient corresponding to that audio frame in the light control sequence can be determined as the light control coefficient associated with the video frame. When determining the audio frame associated with a video frame, the start rendering time (or display time) of the video frame can be determined, and the audio frame whose corresponding time matches the start rendering time (or display time) of the video frame can be the audio frame associated with the video frame.

[0085] S402, determine the increment of the light change progress in each video frame based on the product of the associated light control coefficient and the predetermined video frame duration.

[0086] The video frame duration can be understood as the display duration of a video frame. It can be a preset value or determined based on the time required to render the previous video frame.

[0087] In practical applications, when displaying lighting changes in a virtual scene through multiple video frames, if the duration of a video frame is deltaTime, and the lighting change method is not adjusted using a lighting control coefficient, then after the video frame is displayed, the progress of the lighting change can be considered to have increased by deltaTime.

[0088] After determining the lighting control coefficients associated with video frames, the corresponding lighting change rate can be adjusted using these coefficients to alter the statically displayed lighting in the video frames to be rendered (or displayed). Specifically, the product of the lighting control coefficients and the video frame duration can be obtained, and the increment of the lighting change rate in each video frame can be determined based on this product.

[0089] For example, the duration of multiple video frames can be a preset value. Without using a light control coefficient to intervene in the rate of change, the rate of change of the lights in the virtual scene across multiple video frames can increase uniformly according to the predetermined video frame duration. For example, if the rate of change in the first video frame is 0.04s, the rate of change in the second video frame is also 0.04s, and so on. However, when using a light control coefficient to adjust the rate of change, if the light control coefficient is 0.5, then the rate of change of the lights in the second video frame to be rendered can be determined to be 0.02s (0.04s * 0.5).

[0090] S403, adjust the preset light change progress in each video frame according to the change progress increment, obtain the target change progress of each video frame, and use it as the duration of each video frame.

[0091] After obtaining the change progress increment, the pre-set change progress of the lights in each video frame can be adjusted using the change progress increment to obtain the target change progress, and this change progress is used as the duration of the video frame.

[0092] For example, when adjusting the changes in lighting in a virtual scene using the lighting control coefficient, if the lighting control coefficient associated with the second video frame is 0.5, the current change progress, i.e., the change progress of the lights after the first video frame is displayed, which is 0.04s, can be updated using the change progress increment of 0.02, and the resulting 0.06s can be determined as the target change progress of the second video frame.

[0093] In this embodiment, the increment of the light change progress in the video frame can be determined according to the light control coefficient associated with the video frame, and the pre-set light change progress in each video frame can be adjusted according to the increment, so as to achieve fine adjustment of light change at the video frame level.

[0094] In one embodiment, adjusting the pre-set light change rate in each video frame according to the change rate increment to obtain the target change rate for each video frame may include:

[0095] Obtain the display cycle of the light; determine the sum of the change progress increment and the pre-set change progress of the light in each video frame; determine the ratio of the sum to the display cycle as the target motion progress of the reference light action in the video frame to be rendered.

[0096] The display cycle is the time it takes for the lights to change once according to a preset speed.

[0097] In practice, the time taken for the light to complete one full display at a preset change rate can be obtained to determine the display cycle. Furthermore, for each video frame, the increment of the change rate in that video frame and the change rate of the light in that video frame can be summed. The increment of the change rate and the change rate of the light in that video frame can be duration values. For example, if the increment of the change rate is 0.08s, and the light's change rate before the current video frame is displayed is 0.12s, it can be understood that the light has been displayed up to the state at 0.12s. Therefore, the current video frame is intended to display the state at the 0.20s (0.12s + 0.08s) progress point.

[0098] After obtaining the summation result, normalization processing can be performed, that is, the ratio of the summation result to the display period is obtained, and the ratio is determined as the target change progress of the lights in the current video frame. The target change progress obtained in this way can also be called the process parameter, which can quickly locate the position of the light change progress in the video frame to be rendered relative to the overall change, providing a basis for quickly determining the light state under the corresponding progress.

[0099] In one embodiment, one or more virtual light sources can be set in the virtual scene, each of which can emit light. Based on the duration of each video frame, the adjusted target video frame can be obtained, which may include the following steps:

[0100] Determine the position parameters, rendering parameters, and on / off status of each virtual light source for the duration of each video frame; based on the position parameters, rendering parameters, and on / off status of each virtual light source in each video frame, render the video frame to obtain the adjusted target video frame.

[0101] The switch status indicates whether the corresponding virtual light source is on or off. When it is on, the virtual light source can emit light; when it is off, the virtual light source does not emit light. The initial light state of the virtual light source can be off.

[0102] Position parameters can be used to indicate the change in the position of the virtual light source at the current change stage relative to the position of the virtual light source at the previous change stage. For example, position parameters may include rotation parameters and / or position parameters. The rotation parameter can characterize the angular change of the virtual light source's own rotation angle, and the position parameter can characterize the spatial position change of the virtual light source.

[0103] Lighting rendering parameters can be used to configure the form of light emitted by a virtual light source. For example, lighting rendering parameters may include at least one of the following: light color, light length, light brightness, and the size of the fan angle corresponding to the light illumination range.

[0104] In practice, the lighting status information of the light at different stages of change can be stored in advance. The lighting status information may include at least one of the virtual light source's position parameters, lighting rendering parameters, and on / off status.

[0105] After determining the target change progress of the lights in the video frame to be rendered, the position parameters, light rendering parameters, and on / off status of each virtual light source under the target change progress can be read as the light state information under the target change progress. Then, for each video frame, the state of the statically displayed lights in the video frame can be determined according to the position parameters, light rendering parameters, and on / off status of each virtual light source in the video frame, and the video frame can be rendered to obtain the adjusted target video frame.

[0106] In some optional embodiments, if multiple virtual light sources exist, they can be grouped. Each group can have a parent node, and each virtual light source within a group can have a corresponding group node. For example, virtual light sources A, B, C, D, E, and F can be divided into two groups: A, B, C and D, E, and F. Virtual light sources A, B, and C can share a common parent node 1 and their respective group nodes a, b, and c. The light status information of the parent node can be used to uniformly control the light status information of each virtual light source within the group. When the light status information of the parent node changes, the light status information of each virtual light source within the group also changes. The light status information of the group nodes is used to control a corresponding virtual light source. Through the light status information of the parent node and group nodes, various combinations of virtual light sources can be achieved.

[0107] In this embodiment, by rendering video frames and obtaining target video frames based on the position parameters, lighting rendering parameters, and on / off states of the virtual light source in each video frame, the state changes of the light in each video frame can be accurately controlled, thereby achieving rich lighting effects.

[0108] In one embodiment, prior to S103, the method may further include the following steps:

[0109] Determine the song style of the audio; obtain at least one light from the set of lights associated with the song style as a pre-set light.

[0110] As an example, a song style, also known as a genre, can refer to the distinctive and representative appearance of an audio track as a whole. For instance, the song style of an audio track can be at least one of the following: rock, pop, classical, hip-hop, country, folk, metal, electronic, rhythm and blues, or world music.

[0111] In practice, different types of lighting and different music styles can create different atmospheres. For example, a spotlight that swings back and forth quickly can create a rhythmic atmosphere, while a floodlight can emit a uniform and soft light to create a warm and comfortable atmosphere.

[0112] Based on this, multiple lights can be pre-set, each of which can be called a light action. A light action can be a process in which the light's state information changes relatively. In addition, the song style that each light is adapted to can be marked, and one or more lights that are adapted to each song style can be determined to obtain a set of lights.

[0113] After obtaining the audio of the light to be configured, the song style of the audio can be determined. For example, the audio identifier can be obtained, and the corresponding song style can be queried in a preset database based on the audio identifier. Then, one or more lights can be selected from the set of lights corresponding to that song style as preset lights.

[0114] In this embodiment, pre-set lights can be obtained from a set of lights associated with the song style, so that the atmosphere created by the lights can be adapted to the song style of the audio, and lights that match the song audio can be obtained quickly.

[0115] In one embodiment, adjusting the pre-set changes of lights in a virtual scene according to a lighting control sequence may include:

[0116] According to the pre-set display order of various lights, various lights are displayed sequentially in the virtual scene. During the display of lights, the changes of the currently displayed light in the virtual scene are adjusted according to the light control sequence until all the light control coefficients in the light control sequence have been used to adjust the changes of the lights in the virtual scene.

[0117] In practical applications, after selecting at least one light from the light set, the display order of each light can be determined. In one example, if multiple lights are selected, they can be randomly arranged, and the display order can be determined based on the arrangement. After determining the display order, since each light has a corresponding display cycle, a time series and action sequence of the lights can also be generated accordingly, as shown in Table 1 below.

[0118] Table 1

[0119] Action display sequence i 0 1 …… N-1 Time Series tList 4 9 …… 120 Action sequence mList Motion1 Motion15 …… Motion2

[0120] Once the display order is determined, various pre-set lights can be displayed sequentially in the virtual scene. For the currently displayed pre-set light, the changes in its movement within the virtual scene can be adjusted using a light control sequence.

[0121] In practical implementation, after adjusting the currently displayed pre-set light using the light control coefficients in the light control sequence, several unused light control coefficients may still remain in the action control sequence after the current light display ends. In other words, audio playback may not have finished after the current light display ends. At this point, the pre-set light to be displayed can be determined again according to the display order, and the light control sequence can continue to adjust the changes of the pre-set lights to be displayed in the virtual scene until all the light control coefficients in the light control sequence have been used to adjust the changes of the lights in the virtual scene. In one example, the various pre-set lights can be displayed in a loop. For example, after the last type of light is displayed, the first light can be determined again as the currently pre-set light according to the display order.

[0122] In this embodiment, various lighting changes can be adjusted using lighting control coefficients until the audio ends, allowing for appropriate lighting changes to be set throughout the entire audio playback process.

[0123] In one embodiment, S101 obtaining the pitch of each audio frame in a plurality of audio frames may include the following steps:

[0124] Perform time-domain and frequency-domain transformation on each of the multiple audio frames in the audio to obtain the power value of each audio frame; based on the power value of each audio frame, determine the audio frame pitch of each audio frame.

[0125] In practical applications, multiple audio frames can be acquired, and after performing time-frequency transformation on each audio frame and obtaining the corresponding spectrogram, the power value of each audio frame can be determined.

[0126] Specifically, the audio signal in the time domain can be obtained. The audio signal in the time domain can reflect the mapping relationship between the power value of the audio signal and time. Then, the audio signal in the time domain can be processed into frames to obtain multiple audio frames. For example, if the audio sampling frequency is 44.1 kHz and the preset number of sampling points for each audio frame is 1024, then audio frames can be obtained at time intervals of 23.2 milliseconds.

[0127] After obtaining the audio frame in the time domain, a Fourier transform can be performed on the audio signal in the audio frame to convert the audio signal in the time domain into an audio signal in the frequency domain, thereby obtaining the spectrum of the audio frame. The spectrum can reflect the mapping relationship between the power value of the audio signal and the signal frequency.

[0128] Since different pitches can correspond to different frequencies, after obtaining the spectrum of the audio frame, the power value of each audio frame at different signal frequencies can be determined, and the pitch of the audio frame can be determined based on the power value of each audio frame.

[0129] In one embodiment, S102 determines multiple lighting control coefficients based on the pitch of each audio frame, which may include the following steps:

[0130] The target power values ​​of each audio frame are summed, and the lighting control coefficient of the audio frame is obtained based on the summation result; the target power value is the power value of the audio frame within the human hearing frequency range.

[0131] In practical applications, for each audio frame, a target power value can be determined and summed. Specifically, the pitch that humans can perceive has a corresponding frequency range, namely the human hearing frequency range. After acquiring the audio frame, the power values ​​within the human hearing frequency range can be obtained from the audio frame's spectrogram as the target power values ​​and summed to obtain the summation result.

[0132] For example, the human ear's hearing frequency range is 20 to 20,000 Hz. 15,804 Hz is the frequency value corresponding to the highest pitch in the International Standard Pitch. With a sampling frequency of 44,100 Hz and a spectrum diagram including 1,024 frequencies, the power values ​​corresponding to the first 367 (15,804 / 44,100*1024≈367) frequencies can be summed.

[0133] The lighting control coefficient for the audio frame can then be determined based on the summation result. For example, the value range of this lighting control coefficient can be (0,1). The lighting control coefficient determined in this way is positively correlated with the pitch of the audio frame; that is, the higher the pitch of the audio frame, the higher the lighting control coefficient, and the faster the lighting changes.

[0134] In this embodiment, the lighting control coefficient of the audio frame can be obtained based on the summation of the target power values ​​of the audio frame, so that within the range of pitch perceived by the human ear, the higher the audio pitch, the faster the lighting changes, so that the visual effect of the lighting matches the atmosphere rendered by the audio.

[0135] To enable those skilled in the art to better understand the above steps, the following example illustrates the embodiments of this application, but it should be understood that the embodiments of this application are not limited thereto.

[0136] like Figure 4 As shown, after receiving the light configuration command for audio, an audio file containing the time-domain audio signal can be obtained.

[0137] Then, on the one hand, the song style of the audio can be determined by querying a preset database (such as a song genre library). Then, multiple lights, i.e. multiple light actions, can be obtained from the light action library (i.e., light set or light action library) corresponding to the song style, and the actions can be arranged to obtain the time series tList and action sequence mList as shown in Table 1. Among them, tList is the time range in which various lights are displayed according to the preset change speed (i.e. without the intervention of the light control coefficient). For example, the first Motion1 is displayed from 0 to 4 seconds, and the second Motion15 is displayed from 5 to 9 seconds.

[0138] On the other hand, based on the audio file corresponding to the audio, multiple audio frames and the spectrogram of each audio frame can be obtained in real time. The pitch of each audio frame can be determined according to each spectrogram, and the lighting control coefficient can be determined according to the pitch of the audio frame. This lighting control coefficient can also be called the acceleration coefficient α.

[0139] Then, the acceleration factor α, as well as the pre-acquired time sequence tList and action sequence mList, can be input into the time control system, and the static display of lights in the virtual scene in multiple video frames can be adjusted through the time control system.

[0140] During the adjustment process, such as Figure 5 As shown, parameters t and i can be initialized first. t is the total progress corresponding to the currently displayed light. The total progress can be understood as the total duration of the light display at the preset change speed. i is the display sequence number of multiple lights. The display sequence number i of the first light is 0. N represents the number of elements in the time sequence tList. The value of N is the same as the number of lights appearing in the action sequence, and tList[-1] is preset to 0.

[0141] Then, the increment of the change progress can be determined based on the product of the light control coefficient and the video frame duration deltaTime(△t). The current total progress t is updated using the increment of the change progress to obtain the updated t. Then, the tList[i-1] of the previous light action is subtracted from the updated t to obtain the change progress for the current light (the duration progress of the current light when the light is displayed according to the preset change speed). Then, this change progress and the light name mList[i] corresponding to the light can be sent to the light control system (also known as the motion control system).

[0142] After receiving the parameters "t-tList[i-1]" and "mList[i]", the lighting control system can normalize the progress of the lighting action based on the display period T corresponding to "t-tList[i-1]" and the lighting action "mList[i]", determine the calculation result of (t-tList[i-1]) / T, obtain the target progress of the action, and acquire the lighting state information of each virtual light source under the target progress. Based on this lighting state information, a video frame is rendered to obtain the target video frame. For example, when rendering the video frame, Unity's Update function can be used.

[0143] After rendering the corresponding target video frame, it can be determined whether the current type of light has been displayed. Specifically, it can be determined whether the total progress t (the total duration of the light being displayed according to the preset change speed) of the current light exceeds the time range tList[i] of the light.

[0144] If not, then when rendering the next video frame, the changes of the current light in the virtual scene continue to be adjusted according to the light control sequence; if yes, then the next light can be acquired, and the above steps are repeated until the audio playback is complete. The method of this application can adaptively adjust the changes of light in the virtual scene based on the audio characteristics (pitch), effectively balancing the adaptability of light and audio in the virtual scene with the efficiency of light configuration.

[0145] It should be understood that although the steps in the flowcharts of the embodiments described above are shown sequentially according to the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the flowcharts of the embodiments described above may include multiple steps or multiple stages. These steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these steps or stages is not necessarily sequential, but can be performed alternately or in turn with other steps or at least some of the steps or stages of other steps.

[0146] In one embodiment, a computer device is provided, which may be a server, and its internal structure diagram may be as follows: Figure 6As shown, the computer device includes a processor, memory, and a network interface connected via a system bus. The processor provides computing and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system, computer programs, and a database. The internal memory provides an environment for the operation of the operating system and computer programs in the non-volatile storage media. The database stores lighting motion data. The network interface communicates with external terminals via a network connection. When the computer program is executed by the processor, it implements a method for dynamically displaying lights in a virtual scene.

[0147] In one embodiment, a computer device is provided, which may be a terminal, and its internal structure diagram may be as follows: Figure 7 As shown, the computer device includes a processor, memory, communication interface, display screen, and input devices connected via a system bus. The processor provides computing and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system and computer programs. The internal memory provides an environment for the operation of the operating system and computer programs stored in the non-volatile storage media. The communication interface is used for wired or wireless communication with external terminals; wireless communication can be achieved through Wi-Fi, mobile cellular networks, NFC (Near Field Communication), or other technologies. When the computer program is executed by the processor, it implements a method for dynamically displaying lights in a virtual scene. The display screen can be an LCD screen or an e-ink screen. The input devices can be a touch layer covering the display screen, buttons, a trackball, or a touchpad on the computer device's casing, or an external keyboard, touchpad, or mouse.

[0148] Those skilled in the art will understand that Figure 6 and Figure 7 The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the computer device to which the present application is applied. Specific computer devices may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0149] In one embodiment, a computer device is provided, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to perform the following steps:

[0150] Get the pitch of each audio frame in multiple audio frames;

[0151] Based on the pitch of each audio frame, multiple lighting control coefficients are determined, which are used to control the changes in lighting in the virtual scene; the multiple lighting control coefficients form a lighting control sequence, which is associated with multiple audio frames in the audio.

[0152] According to the lighting control sequence, the pre-set lighting changes in the virtual scene are adjusted.

[0153] In one embodiment, the processor also performs the steps described in the other embodiments when executing the computer program.

[0154] In one embodiment, a computer program product is provided, including a computer program that, when executed by a processor, performs the following steps:

[0155] Get the pitch of each audio frame in multiple audio frames;

[0156] Based on the pitch of each audio frame, multiple lighting control coefficients are determined, which are used to control the changes in lighting in the virtual scene; the multiple lighting control coefficients form a lighting control sequence, which is associated with multiple audio frames in the audio.

[0157] According to the lighting control sequence, the pre-set lighting changes in the virtual scene are adjusted.

[0158] In one embodiment, the computer program, when executed by a processor, also implements the steps described in the other embodiments above.

[0159] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, data stored, data displayed, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties.

[0160] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium, and when executed, it can include the processes of the embodiments of the above methods. Any references to memory, databases, or other media used in the embodiments provided in this application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM can take many forms, such as Static Random Access Memory (SRAM) or Dynamic Random Access Memory (DRAM). The databases involved in the embodiments provided in this application may include at least one type of relational database and non-relational database. Non-relational databases may include, but are not limited to, blockchain-based distributed databases. The processors involved in the embodiments provided in this application may be general-purpose processors, central processing units, graphics processing units, digital signal processors, programmable logic devices, quantum computing-based data processing logic devices, etc., and are not limited to these.

[0161] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0162] The embodiments described above are merely illustrative of several implementation methods of this application, and while the descriptions are specific and detailed, they should not be construed as limiting the scope of this patent application. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this application should be determined by the appended claims.

Claims

1. A method for dynamically displaying lights in a virtual scene, characterized in that, The method includes: Get the pitch of each audio frame in multiple audio frames; Based on the pitch of each audio frame, multiple lighting control coefficients are determined, which are used to control the changes in lighting in the virtual scene; the multiple lighting control coefficients form a lighting control sequence, which is associated with multiple audio frames in the audio. Adjusting the changes of pre-set lights in the virtual scene according to the lighting control sequence includes: acquiring multiple video frames, each video frame being used to display the static display of the pre-set lights in the virtual scene; associating each video frame with the lighting control sequence, obtaining the lighting control coefficient associated with each video frame from the lighting control sequence; determining the change progress increment of the lights in each video frame according to the associated lighting control coefficient; adjusting the change progress of the pre-set lights in each video frame according to the change progress increment to obtain the target change progress of each video frame, which is used as the duration of each video frame; generating a target video based on the duration of each video frame, the target video being the adjusted lights in the virtual scene.

2. The method according to claim 1, characterized in that, The generation of the target video based on the duration of each video frame includes: Based on the duration of each video frame, an adjusted target video frame is obtained, and based on multiple target video frames, the target video is generated.

3. The method according to claim 1, characterized in that, The step of determining the increment of the light change progress in each video frame based on the associated light control coefficient includes: The increment of the light change rate in each video frame is determined by multiplying the associated light control coefficient with the predetermined video frame duration.

4. The method according to claim 3, characterized in that, The step of adjusting the pre-set light change rate in each video frame according to the change rate increment to obtain the target change rate for each video frame includes: Obtain the display cycle of the light; the display cycle is the time taken for the light to display once at a preset change speed; The summation of the change progress increment and the pre-set light change progress in each video frame is determined, and the ratio of the summation result to the display period is determined as the target change progress for each video frame.

5. The method according to claim 2, characterized in that, The virtual scene includes at least one virtual light source, and obtaining the adjusted target video frame based on the duration of each video frame includes: The position parameters, lighting rendering parameters, and on / off status of each virtual light source are determined for the duration of each video frame; the on / off status indicates whether the corresponding virtual light source is in an on or off state. Based on the position parameters, lighting rendering parameters, and on / off status of each virtual light source in each video frame, the video frame is rendered to obtain the adjusted target video frame.

6. The method according to claim 1, characterized in that, Before adjusting the preset lighting changes in the virtual scene according to the lighting control sequence, the method further includes: Determine the song style of the audio; At least one light source is selected from the set of lights associated with the song style as a preset light source.

7. The method according to claim 6, characterized in that, The step of adjusting the preset lighting changes in the virtual scene according to the lighting control sequence includes: According to the pre-set display order of various lights, various lights are displayed sequentially in the virtual scene. During the light display process, the changes of the currently displayed light in the virtual scene are adjusted according to the light control sequence until all the light control coefficients in the light control sequence have been used to adjust the changes of the lights in the virtual scene.

8. The method according to any one of claims 1-7, characterized in that, The determination of multiple lighting control coefficients based on the pitch of each audio frame includes: The target power values ​​of each audio frame are summed, and the lighting control coefficient of each audio frame is obtained based on the summation result; the target power value is the power value of the audio frame within the human hearing frequency range.

9. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that, When the processor executes the computer program, it implements the steps of the method according to any one of claims 1 to 8.

10. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 8.

Images