Music playing method and device, equipment and medium

By matching background and animation materials according to music sentences and word characteristics in 3D virtual space, dynamically adjusting the virtual space, the problem of single and boring virtual space is solved, and the display effect and user experience of music playback are improved.

CN118939814AInactive Publication Date: 2024-11-12SHENYANG MXNAVI CO LTD
View PDF 3 Cites 0 Cited by

Patent Information

Application Number
CN202310537598.7
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2023-05-12
Publication Date
2024-11-12
Estimated Expiration
Not applicable · inactive patent

AI Technical Summary

Technical Problem

When playing music in a 3D virtual space, the existing technology makes the virtual space single and boring, affecting the user's viewing experience, and failing to effectively match the dynamic changes of music with virtual avatars and backgrounds.

Method used

By determining the environmental characteristics and emotional characteristics of different sentences and words in the music to be played, matching the corresponding background image materials and animation effect materials, dynamically adjusting the background and foreground virtual images of the virtual space during music playback to improve the display effect and Fitness.

Benefits of technology

It realizes adaptive adjustment of the background and foreground virtual images in the virtual space, improves the display effect and user experience of music playback, and enhances the fit and relevance of the virtual space and music.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118939814A_ABST
    Figure CN118939814A_ABST
Patent Text Reader

Abstract

The invention discloses a music playing method and device, equipment and a medium. The music playing method comprises the following steps: determining environment characteristics corresponding to different statements in to-be-played music and emotion characteristics corresponding to different words; background image materials matched with the environment features under different statements and animation effect materials matched with the emotion features under different words are determined; and in the process of playing the to-be-played music, taking the background image material corresponding to the currently played fragment as a display background, and displaying a foreground virtual image according to the animation effect material corresponding to the currently played fragment. According to the technical scheme, the display background and the foreground virtual image in the virtual space can be adaptively adjusted along with the difference of the currently played segments of the played music, the display effect of the foreground virtual image and the display background in the virtual space is improved, and the integrating degree and correlation of the virtual space and the to-be-played music are improved.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence technology, and in particular to a music playing method, device, equipment and medium. Background Art

[0002] When playing music in a 3D (Three Dimensions) virtual space, in order to make the virtual space more vivid, the virtual image in the space can have corresponding accompanying movements or expressions, and the virtual space can also switch backgrounds.

[0003] In the prior art, some body movements, facial expressions and background images are usually preset for the virtual image during the song playing process, and the virtual image can interact with the user during the music playing, or some dynamic effects are added to the virtual image considering the environment the user is in. However, the above scheme has the problem that the virtual space is monotonous and boring, which affects the user's viewing experience. Summary of the invention

[0004] The present invention provides a music playing method, device, equipment and medium to improve the playing effect of music in a virtual space.

[0005] According to one aspect of the present invention, there is provided a music playing method, comprising:

[0006] Determine the environmental features corresponding to different sentences in the music to be played, and the emotional features corresponding to different words;

[0007] Determine background image materials matching the environmental features under different sentences, and animation effect materials matching the emotional features under different words;

[0008] During the process of playing the music to be played, the background image material corresponding to the currently playing segment is used as the display background, and the foreground virtual image is displayed according to the animation effect material corresponding to the currently playing segment.

[0009] According to another aspect of the present invention, there is provided a music playing device, comprising:

[0010] A feature determination module, used to determine the environmental features corresponding to different sentences in the music to be played, and the emotional features corresponding to different words;

[0011] A material determination module, used to determine background image materials matching the environmental features under different sentences, and animation effect materials matching the emotional features under different words;

[0012] The playing and displaying module is used to display a foreground virtual image based on the background image material corresponding to the currently playing segment during the process of playing the music to be played, and according to the animation effect material corresponding to the currently playing segment.

[0013] According to another aspect of the present invention, an electronic device is provided, the electronic device comprising:

[0014] at least one processor; and

[0015] a memory communicatively connected to the at least one processor; wherein,

[0016] The memory stores a computer program that can be executed by the at least one processor, and the computer program is executed by the at least one processor so that the at least one processor can execute any one of the music playing methods provided in the embodiments of the present invention.

[0017] According to another aspect of the present invention, a computer-readable storage medium is provided, wherein the computer-readable storage medium stores computer instructions, and the computer instructions are used to enable a processor to implement any one of the music playing methods provided in the embodiments of the present invention when executed.

[0018] The technical solution of the embodiment of the present invention determines the environmental features corresponding to different sentences in the music to be played, and the emotional features corresponding to different words, and then determines the background image materials matching the environmental features, and the animation effect materials matching the emotional features. During the playback of the music to be played, the background image materials corresponding to the currently playing segment are displayed as the display background of the virtual space, and the foreground virtual image with the animation effect materials corresponding to the currently playing segment are displayed. This achieves that the display background and foreground virtual image in the virtual space will be adaptively adjusted according to the currently playing segment of the music to be played, thereby improving the display effect of the foreground virtual image and the display background in the virtual space, as well as the fit and relevance of the virtual space to the music to be played.

[0019] It should be understood that the contents described in this section are not intended to identify the key or important features of the embodiments of the present invention, nor are they intended to limit the scope of the present invention. Other features of the present invention will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0021] Figure 1 is a flowchart of a music playing method provided according to Embodiment 1 of the present invention;

[0022] Figure 2 is a flowchart of a music playing method provided according to Embodiment 2 of the present invention;

[0023] Figure 3 is a schematic diagram of a data structure storage content provided according to Embodiment 2 of the present invention;

[0024] Figure 4 is a structural schematic diagram of a music playing device provided according to Embodiment 3 of the present invention;

[0025] Figure 5 It is a structural schematic diagram of an electronic device for implementing the music playing method according to an embodiment of the present invention. DETAILED DESCRIPTION

[0026] In order to enable those skilled in the art to better understand the scheme of the present invention, the technical scheme in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present invention.

[0027] It should be noted that the terms "first", "second", etc. in the specification and claims of the present invention and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments of the present invention described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units that are clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0028] In addition, it should be noted that the collection, storage, use, processing, transmission, provision and disclosure of the data to be processed involved in the technical solution of the present invention are in compliance with the provisions of relevant laws and regulations and do not violate public order and good morals.

[0029] Embodiment 1

[0030] Figure 1A flowchart of a music playing method is provided for the first embodiment of the present invention. This embodiment is applicable to the case where a foreground virtual image is displayed according to the played music when playing music in a 3D virtual space. The method can be executed by a music playing device, which can be implemented in the form of hardware and / or software, and can be configured in various general-purpose computing devices. Figure 1 As shown, the method includes:

[0031] S110, determining environmental features corresponding to different sentences in the music to be played, and emotional features corresponding to different words.

[0032] Among them, environmental features can refer to words that describe the characteristics of the meteorological environment or the natural environment; for example, sunny, rainy, grassland, and river. Emotional features can refer to words that describe the emotions of characters; for example, happy, pleasant, sad, sad, and worried. Among them, the music to be played includes at least one sentence, and different sentences are composed of at least one word. Among them, sentences can be divided by pauses in music playback, or set punctuation marks in lyrics, etc. The present invention does not impose any limitation on the way of dividing sentences for playing music. For each word in the sentence, it can be implemented by word segmentation, etc., and the present invention does not impose any limitation on the way of dividing words.

[0033] In an embodiment of the present invention, the lyrics of the music to be played can be identified and analyzed based on natural language processing technology (NLP) to determine the environmental features corresponding to different sentences in the lyrics of the music to be played, as well as the emotional features corresponding to different words in the sentences. For example, for a sentence in the lyrics of a certain music to be played that contains words such as "under the sun" and "bubbles", after the NLP technology is used to identify and analyze it, the corresponding environmental features can be obtained as "sunshine" and "bubbles"; for a sentence in the lyrics of a certain music to be played that contains words such as "tears in the corners of the eyes", after the NLP technology is used to identify and analyze it, the corresponding emotional feature can be obtained as "sadness".

[0034] Optionally, in an embodiment of the present invention, the lyrics of the music to be played can be identified and analyzed based on natural language processing technology, the action features corresponding to different words in the sentence of the music to be played can be determined, and the action materials matching the action features can be determined. The action features can include at least one of expression features and body features. The expression features are used to limit the facial expressions of the foreground virtual image, so as to output facial emotions; the body features are used to limit the body movements of the foreground virtual image.

[0035] S120, determining background image materials that match environmental features under different sentences, and animation effect materials that match emotional features under different words.

[0036] Among them, the background image material may include at least one of a background video and a background picture collection, which can be used to display as the display background of the virtual space during the playback of the music to be played; the animation effect material may include at least one of a body movement material and a facial expression material, which can be used to display at least one of the body movement and facial expression movements of the foreground virtual image in the virtual space during the playback of the song to be played.

[0037] Optionally, the background image material can be obtained by searching an online search engine or in a preset database. The present invention does not impose any limitation on the method for obtaining the background image material. Optionally, the facial expression material can be a file in a preset format that includes facial expression action information and facial model information and other information exported by a setting tool. Among them, the setting tool and the preset format can be defined by a technician according to actual needs. The present invention does not impose any limitation on the setting tool and the preset format used. In a specific implementation, the setting tool can be 3DMAX three-dimensional animation rendering and production software, and the preset format can be FBX format.

[0038] In an embodiment of the present invention, background image materials matching each sentence are determined based on environmental features corresponding to different sentences in the lyrics of the music to be played, and animation effect materials matching each word are determined based on emotional features corresponding to different words in each sentence.

[0039] For example, the correspondence between different environmental features and background image materials can be preset, and the search and matching of different sentences corresponding to the background image materials can be performed based on the correspondence. Also, the correspondence between different emotional features and animation effect materials can be preset, and the search and matching of different words corresponding to the animation effect materials can be performed based on the correspondence.

[0040] It should be noted that not all sentences in the music to be played have environmental characteristics or emotional characteristics. In order to improve the display effect of the virtual space during music playback, for sentences with empty environmental characteristics, the basic background environment material can be preset to avoid the poor visual experience caused by playing with an empty background; for sentences with empty emotional characteristics, the basic facial expression material and the basic body movement material can be preset to avoid the poor visual experience caused by the stagnation of the foreground virtual image.

[0041] The basic background environment material, basic facial expression material, and basic body movement material can be set or adjusted by the technician or the playback user according to actual needs, and the present invention does not impose any limitation on this. For example, the basic background environment can be a street or natural scenery, the basic facial expression material can be a smile, and the basic body movement material can be walking.

[0042] S130. During the process of playing the music to be played, the background image material corresponding to the currently playing segment is used as the display background, and the foreground virtual image is displayed according to the animation effect material corresponding to the currently playing segment.

[0043] Among them, different sentences in the music to be played correspond to different playing segments, and different playing segments can correspond to a background image material respectively; there can be at least one word in the sentence corresponding to the same playing segment, and different words can correspond to an animation effect material respectively. It should be noted that the background image materials corresponding to different playing segments can be the same or different; the animation effect materials corresponding to different words in the same playing segment can be the same or different.

[0044] Optionally, if there are multiple facial expression materials corresponding to an emotional feature, one facial expression material can be randomly selected as the facial expression material of the foreground virtual image; optionally, if the current playing segment corresponds to a background image material, the background image material can be directly determined according to the corresponding relationship; if the current playing segment corresponds to multiple background image materials, the multiple background image materials can be combined, synthesized or cut to determine the background image material. Exemplarily, if the environmental features corresponding to the current playing segment are "sunshine" and "bubbles", the background image material finally determined can be a background image material that includes both "sunshine" and "bubbles" features.

[0045] Optionally, in an embodiment of the present invention, the limb motion material of the foreground virtual image can be pre-made, and the simulation of the limb motion material can be completed by controlling the bone transformation matrix parameters (such as displacement or rotation angle, etc.) of the bones required for the limb motion material to be applied to achieve the control of the foreground virtual image; Optionally, the limb motion material can include at least one of thinking, jumping, and running. Exemplarily, if the limb motion material to be displayed by the foreground virtual image is running, it is necessary to apply the movement of the arms and / or legs, therefore, a combined transformation of the arm and leg bone transformation matrices is required to complete the movement of simulating the running of the foreground virtual image.

[0046] Optionally, in an embodiment of the present invention, the facial expression material of the foreground virtual image can switch the facial expression of the foreground virtual image by modifying the weight value of the virtual facial expression (at least one of the mouth shape, eyebrows and eyes, etc.) channel corresponding to the facial expression material.

[0047] In an embodiment of the present invention, when the music to be played is being played, the background image material corresponding to the playing segment of the music to be played is used as the display background of the virtual space; the foreground virtual image in the virtual space is displayed according to the animation effect material corresponding to the playing segment of the music to be played.

[0048] The technical solution of the embodiment of the present invention determines the environmental features corresponding to different sentences in the music to be played, and the emotional features corresponding to different words, and then determines the background image materials matching the environmental features, and the animation effect materials matching the emotional features. During the playback of the music to be played, the background image materials corresponding to the currently playing segment are displayed as the display background of the virtual space, and the foreground virtual image with the animation effect materials corresponding to the currently playing segment are displayed. This achieves that the display background and foreground virtual image in the virtual space will be adaptively adjusted according to the currently playing segment of the music to be played, thereby improving the display effect of the foreground virtual image and the display background in the virtual space, as well as the fit and correlation between the virtual space and the music.

[0049] Embodiment 2

[0050] Figure 2 This is a flowchart of a music playing method provided in Embodiment 2 of the present invention. This embodiment is optimized and improved on the basis of the technical solutions of the above embodiments.

[0051] Furthermore, before “using the background image material corresponding to the currently playing segment as the display background, and displaying the foreground virtual image according to the animation effect material corresponding to the currently playing segment”, add the operation of “determining the audio characteristics of the music to be played under different sentences; for each sentence, updating the background image material under the sentence and the animation effect material corresponding to different words under the sentence according to the audio characteristics of the sentence” to achieve adaptive adjustment of the background image material and the animation effect material. It should be noted that, for the part not described in detail in the embodiments of the present invention, please refer to the relevant descriptions of other embodiments, which will not be repeated here. Figure 2 As shown, the method includes:

[0052] S210: Determine the environmental features corresponding to different sentences in the music to be played, and the emotional features corresponding to different words.

[0053] S220, determining background image materials that match environmental features under different sentences, and animation effect materials that match emotional features under different words.

[0054] S230: Determine the audio characteristics of the music to be played in different sentences.

[0055] The audio characteristics may include at least one of sentence duration information, word duration information, and word tone information. The sentence duration information is used to represent the playback duration of the corresponding sentence; the word duration information is used to represent the playback duration of the corresponding word; and the word tone information is used to represent the pitch when the corresponding word is played.

[0056] Optionally, determining the audio characteristics of the music to be played in different sentences may include: identifying rhythm information of the music to be played; and determining the audio characteristics of the music to be played in different sentences according to the rhythm information.

[0057] The rhythm information is used to characterize the rhythm of the music to be played, and may include at least one of the beat information of the sentence in the music to be played and the pitch information of the words in the sentence. The beat information, as a unit for measuring the rhythm of the music, can be used as a basis for determining the duration-related information (such as the sentence duration information and the word time information, etc.); the pitch information, as a parameter for measuring the sound frequency, can be used as a basis for determining the word pitch information.

[0058] In an optional embodiment, the simple musical notation of the music to be played (or other musical notation rules corresponding to the music score) can be analyzed based on image recognition technology to identify the beat information and pitch information of the music to be played.

[0059] In an embodiment of the present invention, by identifying the rhythmic information of the music to be played and determining the audio features of different sentences in the music to be played based on the rhythmic information, the matching degree between the determined background image material and animation effect material and the rhythmic information of the music to be played is improved.

[0060] Optionally, determining the audio characteristics of the music to be played in different sentences based on the rhythmic information may include: determining sentence duration information of different sentences and word duration information of different words based on the beat information; determining word tone information of different words based on the tone information; and generating audio characteristics including at least one of sentence duration information, word duration information and word tone information.

[0061] For example, if it is identified that the beat of a certain music to be played is 4 / 4 (i.e., one quarter note is one beat, and each measure takes up 4 beats), and a certain sentence in the music to be played takes up a total of 3 beats; if the playing time of the sentence is calculated to be 3 seconds based on the total duration of the music to be played and the number of beats occupied by the sentence. If the words "gently", "I will" and "leave you" each take up one beat in the sentence, it indicates that the word duration information of the corresponding words is 1 second each. If it is identified that the first "light" character in the beat of "gently" takes up 1 / 4 beat, the word duration information of the word is 0.25 seconds; the second "light" character takes up 1 / 2 beat, the word duration information of the word is 0.5 seconds; the word "ground" character takes up 1 / 4 beat, the word duration information of the word is 0.25 seconds. Similarly, the sentence duration information of different sentences in the music to be played and the word duration information of each word in different sentences can be analyzed.

[0062] Exemplarily, image recognition technology can also be used to identify bass words and treble words in a musical score to be played, and the word audio information corresponding to the bass words can be marked as bass, and the word tone information corresponding to the treble words can be marked as treble.

[0063] In another possible implementation, based on speech recognition technology, audio recognition can be directly performed on the music to be played to obtain beat information and pitch information, and then determine sentence duration information of different sentences and word duration information of different words.

[0064] In an embodiment of the present invention, sentence duration information of different sentences and word duration information of different words are determined through beat information; word tone information of different words is determined through pitch information, and audio characteristics including at least one of sentence duration information, word duration information and word tone information are generated, thereby improving the richness and accuracy of the audio characteristics, and further helping to improve the accuracy of background image materials and animation effect materials determined based on the audio characteristics.

[0065] S240. For each sentence, according to the audio characteristics of the sentence, respectively update the background image material under the sentence and the animation effect materials corresponding to different words under the sentence.

[0066] In an optional embodiment, if the audio characteristics include word pitch information, the animation amplitude parameters of the corresponding animation effect material can be adjusted according to the word pitch information corresponding to different words in the sentence.

[0067] Wherein, the animation amplitude parameter refers to a parameter for controlling the facial expression amplitude and / or body movement amplitude of the foreground virtual image. Exemplarily, if the tone information of the words in the current playing sentence of the music to be played is a low tone, the animation amplitude parameter of the foreground virtual image at this moment is controlled to reduce the first preset ratio on the basis of the preset amplitude, thereby reducing the facial expression amplitude and / or body movement amplitude; if the tone information of the words in the current playing sentence of the music to be played is a high tone, the animation amplitude parameter of the foreground virtual image at this moment is controlled to enlarge the second preset ratio on the basis of the preset amplitude, thereby enlarging the facial expression amplitude and / or body movement amplitude. Wherein, the animation amplitude parameter may include amplitude parameters for controlling different parts. Wherein, the parts may include at least one of the mouth shape, eyebrow shape, eyes, arms, legs, etc. Wherein, the preset amplitude, the first preset ratio, the second preset ratio, etc. may be set or adjusted by the technicians and the playing users according to actual needs or experience, and the preset amplitudes, the first preset ratios, and the second preset ratios corresponding to different parts may correspond to the same or different, and the present invention does not make any limitation to this.

[0068] Since the animation amplitude parameters of the preset animation effect materials are relatively fixed, directly driving the foreground virtual image according to the preset animation effect materials cannot well present the degree of emotional change of the foreground virtual image. The above technical solution adjusts the animation amplitude parameters of the corresponding animation effect materials through the pitch information corresponding to different words in the sentence of the music to be played, so that the display amplitude of the animation effect materials is better matched with the pitch information of the words in the music to be played, and the facial expressions and body movements of the foreground virtual image are more consistent with the emotions of the current playing clip, thereby improving the user experience.

[0069] In another optional embodiment, if the audio characteristics include word duration information, the animation duration parameters of the corresponding animation effect material are adjusted according to the word duration information corresponding to different words in the sentence; if the audio characteristics include sentence duration information, the background duration parameters of the corresponding background image material are adjusted according to the sentence duration information of the sentence.

[0070] Among them, the animation duration parameter may refer to the actual playback duration of the animation effect material; and the background duration parameter may refer to the actual playback duration of the background image material.

[0071] Since the playback duration of preset animation effect materials and background image materials is relatively fixed, directly playing the matched animation effect materials or background image materials in conjunction with the music to be played may cause audio and video asynchrony, or virtual space display freezes, affecting the user's viewing experience.

[0072] The above technical solution adjusts the playback duration of animation effect materials by introducing animation duration parameters, and adjusts the playback duration of background image materials by introducing background duration parameters, so that the animation duration parameters and background image materials are more consistent with the music duration of the music to be played, ensuring the time matching between the music to be played and the content displayed in the virtual space, avoiding the occurrence of audio and video asynchrony or playback jams, thereby improving the user experience.

[0073] When adjusting the animation duration parameters of the animation effect material according to the word duration information, you can directly randomly extract material segments of corresponding duration from the animation effect material according to the word duration information for subsequent display.

[0074] However, the above method has the problem that the animation display is relatively mechanical and has poor fluency, which affects the user's viewing experience. In order to overcome the above problem, in an optional embodiment, for each word in the sentence, at least one animation frame that matches the word duration information corresponding to the word can be extracted from the animation effect material corresponding to the word; the extracted animation frames are sequentially combined to generate a new animation effect material for the word to adjust the animation duration parameters of the corresponding animation effect material. The extraction can be uniformly extracted according to a preset frame extraction frequency. The preset frame extraction frequency can be set or adjusted by the technician according to needs or experience.

[0075] For example, if there is a word in the sentence of music to be played, and the body movement material of the foreground virtual image is raising an arm, and the body movement needs to be completed within the word duration information corresponding to the word (for example, three seconds), and the duration of the body movement material is five seconds, then at least one animation frame of the body movement material can be extracted, and the at least one extracted animation frame can be combined in sequence to generate new body movement material corresponding to the word, so that the animation duration of the new body movement material matches the word duration information of the word. The final presentation effect through the above-mentioned frame extraction method can be: when a certain word is played in the music to be played, it is necessary to display the animation of the foreground virtual image raising its arm within the word duration information (three seconds) of the word, and the duration of the body movement material corresponding to the word (that is, the arm raising material) is five seconds (corresponding to the arm rotation angle of 0° at the 0 second position, the arm rotation angle of 60° at 2.5 seconds, and the arm rotation angle of 30° relative to the 0 second position at 5 seconds). The new body movement material generated can be the arm rotation angle of 0° at the 0 second position within 3 seconds, the arm rotation angle of 60° at 1.5 seconds, and the arm rotation angle of 30° relative to the 0 second position at 3 seconds. Similarly, facial expression material can also generate new facial expression material according to the above-mentioned frame extraction method, which will not be repeated here.

[0076] The embodiment of the present invention generates new animation effects material corresponding to the word duration information of the words in the music to be played by extracting at least one animation frame of the animation effects material. Compared with directly intercepting the animation effects material to generate new animation effects material corresponding to the word duration information of the words in the music to be played, the fluency and completeness of the new animation effects material generated according to the word duration information are guaranteed to a greater extent, thereby improving the display effect of the animation effects material.

[0077] S250. During the process of playing the music to be played, the background image material corresponding to the currently playing segment is used as the display background, and the foreground virtual image is displayed according to the animation effect material corresponding to the currently playing segment.

[0078] Optionally, based on the above embodiments of the present invention, a data structure can be established. Figure 3 As shown, two mapping relationships are stored in the data structure, including the mapping relationship between the sentences in the music to be played and the background image material and the sentence duration information, and the mapping relationship between the words in the sentences in the music to be played and the animation effect material, animation amplitude parameters and word duration information. Through the above two mapping relationships, the facial expression changes and body movement changes of the foreground virtual image can be accurately controlled during the playback of the music to be played, and the display background of the virtual space can be dynamically switched.

[0079] In a specific example, the background image material corresponding to a sentence in the music to be played is night, and the sentence duration information is five seconds. The facial expression material corresponding to word one in the sentence is basic facial expression, the body movement material is basic movement, the animation amplitude parameter is empty, and the word duration information is two seconds; the facial expression material corresponding to word two in the sentence is tears, the body movement material is wiping tears, the animation amplitude parameter is empty, and the word duration information is two seconds; the facial expression material corresponding to word three in the sentence is basic facial expression, body movement material is basic movement, the animation amplitude parameter is low, and the word duration information is one second. Correspondingly, when the sentence in the music to be played is played, the display background of the virtual space is night, and the duration of the night background is five seconds; during the playback of word one in the sentence, the facial expression of the foreground virtual image is smiling, and the body movement of the foreground virtual image is walking, and the process lasts for two seconds; during the playback of word two in the sentence, the facial expression of the foreground virtual image is tears, and the body movement of the foreground virtual image is wiping tears, and the process lasts for two seconds; during the playback of word three in the sentence, the facial expression of the foreground virtual image is smiling, the body movement of the foreground virtual image is walking, and the amplitude of the change of the mouth shape of the foreground virtual image when smiling is less than the normal amplitude value set for a normal smile, and the process lasts for one second.

[0080] In an embodiment of the present invention, the background image material corresponding to each sentence and the animation effect material corresponding to different words under each sentence are updated respectively according to the audio characteristics of different sentences of the music to be played, thereby improving the matching degree between the background image material and the animation effect material and the music to be played, so that the virtual space can accurately describe the rhythm corresponding to the music to be played and the emotion expressed, thereby improving the user experience.

[0081] Embodiment 3

[0082] Figure 4 This is a structural diagram of a music playing device provided in Embodiment 3 of the present invention. The device is suitable for displaying a foreground virtual image according to the played music when playing music in a 3D virtual space. The device includes:

[0083] The feature determination module 310 is used to determine the environmental features corresponding to different sentences in the music to be played, and the emotional features corresponding to different words;

[0084] The material determination module 320 is used to determine the background image material matching the environmental characteristics under different sentences, and the animation effect material matching the emotional characteristics under different words;

[0085] The playing and displaying module 330 is used to display a foreground virtual image based on the background image material corresponding to the currently playing segment as the display background during the process of playing the music to be played, and based on the animation effect material corresponding to the currently playing segment.

[0086] The technical solution of the embodiment of the present invention determines the environmental features corresponding to different sentences in the music to be played, and the emotional features corresponding to different words, and then determines the background image materials matching the environmental features, and the animation effect materials matching the emotional features. During the playback of the music to be played, the background image materials corresponding to the currently playing segment are displayed as the display background of the virtual space, and the foreground virtual image with the animation effect materials corresponding to the currently playing segment are displayed. This achieves that the display background and foreground virtual image in the virtual space will be adaptively adjusted according to the currently playing segment of the music to be played, thereby improving the display effect of the foreground virtual image and the display background in the virtual space, as well as the fit and correlation between the virtual space and the music to be played.

[0087] Optionally, the device further comprises:

[0088] The audio characteristic determination module is used to determine the audio characteristics of the music to be played under different sentences.

[0089] The material updating module is used to update the background image material under each sentence and the animation effect material corresponding to different words under the sentence according to the audio characteristics of the sentence.

[0090] In an optional embodiment, the material update module includes:

[0091] The animation amplitude updating unit is used to adjust the animation amplitude parameters of the corresponding animation effect material according to the word pitch information corresponding to different words in the sentence if the audio characteristics include word pitch information.

[0092] The animation duration updating unit is used to adjust the animation duration parameters of the corresponding animation effect material according to the word duration information corresponding to different words in the sentence if the audio characteristics include word duration information.

[0093] The background duration updating unit is used to adjust the background duration parameters of the corresponding background image material according to the sentence duration information of the sentence if the audio characteristics include sentence duration information.

[0094] In an optional embodiment, the animation duration updating unit includes:

[0095] The animation frame extraction subunit is used to extract, for each word in the sentence, at least one animation frame that matches the word duration information corresponding to the word from the animation effect material corresponding to the word.

[0096] The animation frame synthesis subunit is used to sequentially combine the extracted animation frames to generate new animation effect material for the word, so as to adjust the animation duration parameters of the corresponding animation effect material.

[0097] In an optional embodiment, the audio characteristic determination module includes:

[0098] The rhythm information recognition unit is used to recognize the rhythm information of the music to be played.

[0099] The audio characteristic determination unit is used to determine the audio characteristics of the music to be played in different sentences according to the rhythm information; wherein the rhythm information includes beat information and pitch information.

[0100] In an optional embodiment, the audio characteristic determination unit includes:

[0101] The duration information determination subunit is used to determine the sentence duration information of different sentences and the word duration information of different words according to the beat information.

[0102] The word tone information determination subunit is used to determine the word tone information of different words according to the tone information.

[0103] The audio characteristic generating subunit is used to generate audio characteristics including at least one of sentence duration information, word duration information and word tone information.

[0104] The music playing device provided in the embodiment of the present invention can execute any one of the music playing methods provided in the embodiment of the present invention, and has the corresponding functional modules and beneficial effects for executing each music playing method.

[0105] Embodiment 4

[0106] Figure 5A schematic diagram of an electronic device 410 that can be used to implement an embodiment of the present invention is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device may also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices (such as helmets, glasses, watches, etc.) and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present invention described and / or required herein.

[0107] like Figure 5 As shown, the electronic device 410 includes at least one processor 411, and a memory connected to the at least one processor 411 in communication, such as a read-only memory (ROM) 412, a random access memory (RAM) 413, etc., wherein the memory stores a computer program that can be executed by at least one processor, and the processor 411 can perform various appropriate actions and processes according to the computer program stored in the read-only memory (ROM) 412 or the computer program loaded from the storage unit 418 to the random access memory (RAM) 413. In the RAM 413, various programs and data required for the operation of the electronic device 410 can also be stored. The processor 411, the ROM 412, and the RAM 413 are connected to each other via a bus 414. An input / output (I / O) interface 415 is also connected to the bus 414.

[0108] Multiple components in the electronic device 410 are connected to the I / O interface 415, including: an input unit 416, such as a keyboard, a mouse, etc.; an output unit 417, such as various types of displays, speakers, etc.; a storage unit 418, such as a disk, an optical disk, etc.; and a communication unit 419, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 419 allows the electronic device 410 to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks.

[0109] The processor 411 may be a variety of general and / or special processing components with processing and computing capabilities. Some examples of the processor 411 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various dedicated artificial intelligence (AI) computing chips, various processors running machine learning model algorithms, a digital signal processor (DSP), and any appropriate processor, controller, microcontroller, etc. The processor 411 executes the various methods and processes described above, such as a music playing method.

[0110] In some embodiments, the music playing method may be implemented as a computer program, which is tangibly contained in a computer-readable storage medium, such as a storage unit 418. In some embodiments, part or all of the computer program may be loaded and / or installed on the electronic device 410 via the ROM 412 and / or the communication unit 419. When the computer program is loaded into the RAM 413 and executed by the processor 411, one or more steps of the music playing method described above may be performed. Alternatively, in other embodiments, the processor 411 may be configured to perform the music playing method in any other appropriate manner (e.g., by means of firmware).

[0111] Various implementations of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), load programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various implementations can include: being implemented in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0112] Computer programs for implementing the methods of the present invention may be written in any combination of one or more programming languages. These computer programs may be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device, so that when the computer program is executed by the processor, the functions / operations specified in the flow chart and / or block diagram are implemented. The computer program may be executed entirely on the machine, partially on the machine, partially on the machine and partially on a remote machine as a stand-alone software package, or entirely on a remote machine or server.

[0113] In the context of the present invention, a computer-readable storage medium may be a tangible medium that may contain or store a computer program for use by or in combination with an instruction execution system, device or equipment. A computer-readable storage medium may include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, devices or equipment, or any suitable combination of the foregoing. Alternatively, a computer-readable storage medium may be a machine-readable signal medium. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0114] To provide interaction with a user, the systems and techniques described herein may be implemented on an electronic device having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and a pointing device (e.g., a mouse or trackball) through which the user can provide input to the electronic device. Other types of devices may also be used to provide interaction with the user; for example, the feedback provided to the user may be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user may be received in any form (including acoustic input, voice input, or tactile input).

[0115] The systems and techniques described herein may be implemented in a computing system that includes backend components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes frontend components (e.g., a user computer with a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such backend components, middleware components, or frontend components. The components of the system may be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: a local area network (LAN), a wide area network (WAN), a blockchain network, and the Internet.

[0116] A computing system may include a client and a server. The client and the server are generally remote from each other and usually interact through a communication network. The client and server relationship is generated by computer programs running on the corresponding computers and having a client-server relationship with each other. The server may be a cloud server, also known as a cloud computing server or cloud host, which is a host product in the cloud computing service system to solve the defects of difficult management and weak business scalability in traditional physical hosts and VPS services.

[0117] It should be understood that the various forms of processes shown above can be used to reorder, add or delete steps. For example, the steps described in the present invention can be executed in parallel, sequentially or in different orders, as long as the desired results of the technical solution of the present invention can be achieved, and this document does not limit this.

[0118] The above specific implementations do not constitute a limitation on the protection scope of the present invention. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions can be made according to design requirements and other factors. Any modification, equivalent substitution and improvement made within the spirit and principle of the present invention should be included in the protection scope of the present invention.

Claims

1. A music playing method, characterized in that: include: Determine the environmental features corresponding to different sentences in the music to be played, and the emotional features corresponding to different words; Determine background image materials matching the environmental features under different sentences, and animation effect materials matching the emotional features under different words; During the process of playing the music to be played, the background image material corresponding to the currently playing segment is used as the display background, and the foreground virtual image is displayed according to the animation effect material corresponding to the currently playing segment.

2. The method according to claim 1, characterized in that Before displaying the foreground virtual image using the background image material corresponding to the currently played segment as the display background and displaying the foreground virtual image according to the animation effect material corresponding to the currently played segment, the method further includes: Determine the audio characteristics of the music to be played under different sentences; For each sentence, the background image material under the sentence and the animation effect materials corresponding to different words under the sentence are updated respectively according to the audio characteristics of the sentence.

3. The method according to claim 2, characterized in that The audio characteristics include word tone information; accordingly, the background image material under the sentence and the animation effect materials corresponding to different words under the sentence are updated according to the audio characteristics of the sentence, including: According to the word tone information corresponding to different words in the sentence, the animation amplitude parameters of the corresponding animation effect material are adjusted.

4. The method according to claim 2, characterized in that: If the audio characteristics include word duration information, then updating the background image material under the sentence and the animation effect materials corresponding to different words under the sentence respectively according to the audio characteristics of the sentence includes: According to the word duration information corresponding to different words in the sentence, adjust the animation duration parameters of the corresponding animation effect material; If the audio characteristics include sentence duration information, then updating the background image material under the sentence and the animation effect materials corresponding to different words under the sentence respectively according to the audio characteristics of the sentence includes: According to the sentence duration information of the sentence, the background duration parameters of the corresponding background image material are adjusted.

5. The method according to claim 4, characterized in that The step of adjusting the animation duration parameters of the corresponding animation effect material according to the word duration information corresponding to different words in the sentence includes: For each word in the sentence, extract at least one animation frame that matches the word duration information corresponding to the word from the animation effect material corresponding to the word; The extracted animation frames are sequentially combined to generate new animation effect material for the word, so as to adjust the animation duration parameters of the corresponding animation effect material.

6. The method according to claim 2, characterized in that The determining of the audio characteristics of the music to be played under different sentences includes: Identifying rhythm information of the music to be played; The audio characteristics of the music to be played in different sentences are determined according to the rhythm information.

7. The method according to claim 6, characterized in that The rhythm information includes beat information and pitch information; accordingly, determining the audio characteristics of the music to be played in different sentences according to the rhythm information includes: Determining sentence duration information of different sentences and word duration information of different words according to the beat information; Determining word tone information of different words according to the tone information; An audio characteristic is generated including at least one of sentence duration information, word duration information, and word pitch information.

8. A music playing device, characterized in that: include: A feature determination module, used to determine the environmental features corresponding to different sentences in the music to be played, and the emotional features corresponding to different words; A material determination module, used to determine background image materials matching the environmental features under different sentences, and animation effect materials matching the emotional features under different words; The playing and displaying module is used to display a foreground virtual image based on the background image material corresponding to the currently playing segment during the process of playing the music to be played, and according to the animation effect material corresponding to the currently playing segment.

9. An electronic device, characterized in that: include: one or more processors; A memory for storing one or more programs; When the one or more programs are executed by the one or more processors, the one or more processors implement a music playing method as described in any one of claims 1-7.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, a music playing method as described in any one of claims 1 to 7 is implemented.

Citation Information

Patent Citations

  • Virtual reality technology-based music playing method and device

    CN107977184A

  • Text-based virtual object animation generation method and device, storage medium and terminal

    CN112184858A

  • Audio playing method and device, electronic equipment and readable storage medium

    CN113851106A