Method, apparatus and device for determining beat information, and storage medium

Abstract

The application discloses a beat information determination method and device, equipment and a storage medium, and belongs to the technical field of audio. The method comprises the following steps: acquiring an audio file corresponding to a first song; acquiring auxiliary information corresponding to the first song, wherein the auxiliary information comprises a text related to the first song; determining target beat information of the first song according to the audio file corresponding to the first song and the auxiliary information corresponding to the first song, wherein the target beat information is used for indicating the beat of the first song. Since the audio features of the first song are referred to in the beat recognition process, and the semantic features of the text related to the first song are also referred to, the accuracy of beat recognition can be improved.

Description

Technical Field

[0001] This application relates to the field of audio technology, and in particular to a method, apparatus, device and storage medium for determining beat information. Background Technology

[0002] Beat recognition is one of the core tasks of song information retrieval, and its accuracy directly affects the user experience of applications such as automatic accompaniment, song synchronization, dynamic sound effect triggering, and intelligent music editing.

[0003] In related technologies, beat recognition relies heavily on acoustic features extracted from the audio signal of a song. For example, in beat recognition methods based on the audio signal of a song, the core lies in detecting periodically occurring transient pulses from the time spectrum of the song to identify the beat.

[0004] The above method for identifying the beat of a song only refers to the audio features of the song, resulting in low accuracy in beat recognition. Summary of the Invention

[0005] This application provides a method, apparatus, device, and storage medium for determining beat information, which can improve the accuracy of beat recognition. The technical solution is as follows: According to one aspect of this application, a method for determining beat information is provided, the method comprising: Get the audio file corresponding to the first song; Obtain auxiliary information corresponding to the first song, wherein the auxiliary information includes text related to the first song; Based on the audio file corresponding to the first song and the auxiliary information corresponding to the first song, the target beat information of the first song is determined, and the target beat information is used to indicate the beat of the first song.

[0006] According to another aspect of this application, a beat information determining device is provided, the device comprising: The acquisition module is used to acquire the audio file corresponding to the first song; The acquisition module is further configured to acquire auxiliary information corresponding to the first song, the auxiliary information including text related to the first song; The determining module is used to determine the target beat information of the first song based on the audio file corresponding to the first song and the auxiliary information corresponding to the first song, wherein the target beat information is used to indicate the beat of the first song.

[0007] In an optional design, the target beat information includes the beat type of n beats in the first song, and the timestamps of the n beats; The timestamps of the n beats are determined according to the sequence number of the time window in which the n beats are located. Different time windows have the same length, and n is a positive integer.

[0008] In an optional design, the determining module is used for: Based on the audio file, the auxiliary information, and the prompt information, candidate beat information for the first song is generated using a multimodal large model; The target beat information is determined based on the candidate beat information; The multimodal large model includes a generative model that supports generating matching output information based on multimodal input information. The prompting information is used to prompt the multimodal large model to identify the sequence number of the time window where the beat in the first song is located according to the audio file and the auxiliary information, based on a time window of a preset length.

[0009] In an optional design, the prompting information includes a first prompting information and a second prompting information, and the multimodal large model includes a first multimodal large model and a second multimodal large model; the determining module is used for: Based on the audio file, the auxiliary information, and the first prompt information, the first candidate beat information of the first song is generated through the first multimodal large model; Based on the audio file, the auxiliary information, and the second prompt information, the second candidate beat information of the first song is generated through the second multimodal large model; The first prompt information is used to prompt the multimodal large model to identify the sequence number of the time window where all types of beats in the first song are located according to the audio file and the auxiliary information, and according to the preset length of the time window; the second prompt information is used to prompt the multimodal large model to identify the sequence number of the time window where the stressed beat is located in the first song according to the audio file and the auxiliary information, and according to the preset length of the time window.

[0010] In an optional design, the determining module is used for: Based on the first candidate beat information, determine the first timestamp corresponding to all types of beats in the first song, and based on the second candidate beat information, determine the second timestamp corresponding to the accented beats in the first song; The first timestamp that is closest to the second timestamp is replaced by the second timestamp to obtain the replaced first timestamp; The target beat information is determined based on the first timestamp that was replaced and the first timestamp that was not replaced.

[0011] In an optional design, the determining module is used for: The beat type corresponding to the first timestamp that was replaced is determined to be the rebeat; The beat type corresponding to the first timestamp that was not replaced is determined to be a beat type other than the stressed beat; The target beat information is determined based on the replaced first timestamp, the beat type corresponding to the replaced first timestamp, the unreplaced first timestamp, and the beat type corresponding to the unreplaced first timestamp.

[0012] In an optional design, the auxiliary information includes at least one of the style information of the first song, the structural information of the first song, and the lyrics of the first song.

[0013] In an optional design, the acquisition module is used for: Obtain the audio feature map corresponding to the first song, wherein the audio feature map includes an image reflecting the audio features of the first song; The determining module is used for: The target beat information of the first song is determined based on the audio file, the auxiliary information, and the audio feature map.

[0014] According to another aspect of this application, a computer device is provided, the computer device including a processor and a memory, the memory storing at least one program, the at least one program being loaded and executed by the processor to implement the method for determining beat information as described above.

[0015] According to another aspect of this application, a computer-readable storage medium is provided, wherein at least one program is stored therein, the at least one program being loaded and executed by a processor to implement the method for determining beat information as described above.

[0016] According to another aspect of this application, a computer program product or computer program is provided, comprising computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the computer device to perform the method for determining clock information provided in various alternative implementations of the above aspects.

[0017] The beneficial effects of the technical solution provided in this application include at least the following: By using the audio file corresponding to the first song and its auxiliary information, the target beat information of the first song is determined, thereby achieving beat recognition. Since the beat recognition process considers not only the audio features of the first song but also the semantic features of the text related to the song, the accuracy of beat recognition can be improved. Attached Figure Description

[0018] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0019] Figure 1 This is a structural block diagram of a computer system provided in an exemplary embodiment of this application; Figure 2 This is a schematic diagram illustrating the process of determining the beat of a song according to an exemplary embodiment of this application; Figure 3 This is a flowchart illustrating a method for determining beat information provided in an exemplary embodiment of this application; Figure 4 This is a flowchart illustrating a method for determining beat information provided in an exemplary embodiment of this application; Figure 5 This is a schematic diagram of the converter structure provided in an exemplary embodiment of this application; Figure 6 This is a schematic diagram of a user interface provided in an exemplary embodiment of this application; Figure 7 This is a schematic diagram of a data flow process provided in an exemplary embodiment of this application; Figure 8 This is a schematic diagram of the structure of a beat information determination device provided in an exemplary embodiment of this application; Figure 9 This is a schematic diagram of the structure of a terminal provided in an exemplary embodiment of this application; Figure 10 This is a schematic diagram of the structure of a computer device provided in an exemplary embodiment of this application.

[0020] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application. Detailed Implementation

[0021] To make the objectives, technical solutions, and advantages of this application clearer, the embodiments of this application will be described in further detail below with reference to the accompanying drawings.

[0022] First, let me introduce the names involved in this application: Rhythm: The rhythm of a song is a periodic unit formed by the regular alternation of strong and weak beats. It constitutes the rhythmic skeleton of the song, giving it a stable rhythm and structure. Simply put, the rhythm of a song is like its "heartbeat".

[0023] Multimodal large models: Multimodal large models are artificial intelligence (AI) models capable of simultaneously processing and understanding multiple data types such as text, images, audio, and video. They break the limitations of single-modality models, making AI closer to human perception. These models achieve cross-modal understanding and generation by fusing information from different modalities. For example, they can not only "describe images," but also generate images based on text descriptions, understand voice commands, and make decisions based on the visual environment. Currently, mainstream multimodal large model architectures typically use a Large Language Model (LLM) as the core, integrating components such as visual encoders and audio encoders. The input multimodal data is first encoded into a unified semantic representation, then inferred and generated by the LLM, with the final output being text, images, or control commands.

[0024] Figure 1 This is a structural block diagram of a computer system provided in an exemplary embodiment of this application. The computer system 100 includes: a terminal 110 and a server 120.

[0025] Terminal 110 has an application 111 installed and running that supports music playback. This application 111 includes any one of the following: music listening applications, song applications, singing applications, live streaming applications, social networking applications, office applications, game applications, food delivery applications, online shopping applications, video-on-demand applications, short video applications, financial applications, lifestyle service applications, navigation applications, medical applications, learning applications, and mini-programs. In some embodiments, application 111 is a client. Terminal 110 is the terminal used by user 112, and user account 112 can be logged into application 111. Terminal 110 can refer to one of multiple terminals. Optionally, the device type of terminal 110 includes at least one of the following: smartphone, tablet, smartwatch, e-book reader, MP3 player, MP4 player, laptop, and desktop computer.

[0026] Terminal 110 is connected to server 120 via wireless or wired network.

[0027] Server 120 includes at least one of a single server, multiple servers, a cloud computing platform, and a virtualization center. Server 120 provides background services for application 111 that supports song playback. Optionally, server 120 undertakes the primary computing task, and terminal 110 undertakes the secondary computing task; or, server 120 undertakes the secondary computing task, and terminal 110 undertakes the primary computing task; or, server 120 and terminal 110 collaborate on computing using a distributed computing architecture.

[0028] In an illustrative example, server 120 includes processor 121, user account database 122, song management module 123, and user-facing input / output interface (I / O interface) 124. Processor 121 loads instructions stored in server 120 and processes data in user account database 122 and song management module 123. User account database 122 stores user account data used by terminal 110 and other terminals, such as user account avatars, nicknames, and group memberships. Song management module 123 provides services related to song playback functions of application 111. User-facing I / O interface 124 establishes communication and exchanges data with terminal 110 via wireless or wired network.

[0029] Based on the above description of the implementation environment involved in this application, the method provided in the embodiments of this application will be described below.

[0030] Figure 2 This is a schematic diagram illustrating the process of determining the beat of a song according to an exemplary embodiment of this application. Figure 2 As shown, the computer device acquires the audio file 2011 corresponding to the first song 201 and the auxiliary information 2012 corresponding to the first song 201. The audio file 2011 corresponding to the first song 201 can be a complete audio file of the first song 201 or an audio file corresponding to a segment of the first song 201. The auxiliary information 2012 corresponding to the first song 201 includes text related to the first song 201, such as at least one of the style information, structural information, and lyrics of the first song 201. Furthermore, the auxiliary information 2012 may also include at least one of the beats per minute (BPM) and time signature information of the first song 201.

[0031] The computer device determines the target beat information 203 of the first song 201 based on the audio file 2011 corresponding to the first song 201 and the auxiliary information 2012 corresponding to the first song 201. The target beat information 203 indicates the beat of the first song 201. In some embodiments, the target beat information 203 includes the beat types of n beats in the first song 201 and the timestamps of the n beats in the first song 201, where n is a positive integer. The timestamps of the n beats are determined according to the sequence number of the time window in which the n beats are located, and different time windows have the same length. Optionally, the computer device generates candidate beat information of the first song 201 using a multimodal large model 202 based on the audio file 2011, the auxiliary information 2012, and the prompt information, and determines the target beat information 203 based on the candidate beat information. The multimodal large model 202 includes a generative model that supports generating matching output information based on multimodal input information. In this embodiment, the multimodal large model 202 supports input in both audio file and text modal modes. This embodiment does not limit the type of the multimodal large model 202. The prompt information is used to prompt the multimodal large model 202 to identify the sequence number of the time window containing the beat in the first song 201 according to the audio file 2011 and auxiliary information 2012, based on a preset length of time window. The candidate beat information includes the sequence number of the time window containing the beat in the first song 201 identified by the multimodal large model 202. The computer device can determine the timestamp corresponding to the beat in the first song 201 based on the sequence number of the time window containing the beat in the first song 201, thereby determining the target beat information 203 of the first song 201. Optionally, the computer device can obtain the candidate beat information of the first song 201 by identifying different multimodal large models 202. For example, the computer device generates first candidate beat information for the first song 201 using a first multimodal large model 202, and generates second candidate beat information for the first song 201 using a second multimodal large model 202. The first multimodal large model 202 is used to identify the time window numbers of all types of beats in the first song 201, and the second multimodal large model 202 is used to identify the time window numbers of accented beats in the first song 201.

[0032] By using the audio file corresponding to the first song and its auxiliary information, the target beat information of the first song is determined, thereby achieving beat recognition. Since the beat recognition process considers not only the audio features of the first song but also the semantic features of the text related to the song, the accuracy of beat recognition can be improved.

[0033] Figure 3 This is a flowchart illustrating a method for determining beat information according to an exemplary embodiment of this application. The method can be used in a computer device, for example, for... Figure 1 The client in the terminal shown, or the server in the server. For example... Figure 3 As shown, the method includes: Step 302: Obtain the audio file corresponding to the first song.

[0034] The type of the first song includes at least one of the following: a complete song, a song fragment, a medley, instrumental music, a musical accompaniment, a song with vocals, and a song without vocals. This application embodiment does not limit the type of the first song. The audio file corresponding to the first song can be an audio file corresponding to the complete first song, or an audio file corresponding to a fragment of the first song; this application embodiment does not limit this.

[0035] In some embodiments, the computer device obtains the audio file corresponding to the first song through locally stored data, or through uploaded data, or through other computer devices. This application embodiment does not limit the method of obtaining the audio file corresponding to the first song.

[0036] Step 304: Obtain the auxiliary information corresponding to the first song.

[0037] The auxiliary information corresponding to the first song includes text related to the first song. Optionally, the auxiliary information corresponding to the first song includes at least one of the following: style information, structural information, and lyrics. The style information describes the style of the first song, such as jazz, classical, or pop. The structural information describes the composition of the first song, such as the position of the verse, chorus, intro, and outro. In some embodiments, in addition to the above, the auxiliary information corresponding to the first song may further include at least one of the following: beats per minute (BPM) and time signature information. The time signature information indicates the time signature of the first song, such as 4 / 4 or 6 / 8 time.

[0038] In some embodiments, the computer device obtains the auxiliary information corresponding to the first song through locally stored data, or through uploaded data, or through other computer devices. This application embodiment does not limit the method of obtaining the auxiliary information corresponding to the first song.

[0039] Step 306: Determine the target beat information of the first song based on the audio file corresponding to the first song and the auxiliary information corresponding to the first song.

[0040] The target beat information for the first song is used to indicate the beat of the first song. The beat of a song is a periodic unit formed by the regular alternation of strong and weak beats. It constitutes the rhythmic skeleton of the song, giving it a stable rhythm and structure. Simply put, the beat of a song is like the "heartbeat" of the song.

[0041] In some embodiments, the target beat information of the first song includes the beat types of n beats in the first song and the timestamps of the n beats in the first song. For example, it includes the beat type of each of the n beats and the timestamp of each of the n beats, where n is a positive integer. The n beats in the first song can be all the beats in the first song or a portion of the beats in the first song. Beat types include accented beats and beat types other than accented beats, for example, "1" represents an accented beat, and "2", "3", and "4" represent beat types other than accented beats.

[0042] Optionally, the process of determining the target beat information of the first song described above can be implemented using a multimodal large model, which includes a generative model that supports generating matching output information based on multimodal input information. In this embodiment, the multimodal large model supports input in both audio file modality and text modality, and the type of multimodal large model is not limited in this embodiment.

[0043] It should be noted that when the audio file corresponding to the first song is a complete audio file, the target beat information of the first song determined by the computer device is the complete beat information of the first song. When the audio file corresponding to the first song is an audio file corresponding to a segment of the first song, the computer device splices the target beat information corresponding to different segments in chronological order to obtain the complete beat information of the first song.

[0044] In some embodiments, when the method provided in this application is applied to a client, after determining the target beat information of the first song, the client can display or use the target beat information of the first song. When the method provided in this application is applied to a server, after determining the target beat information of the first song, the server can use or send the target beat information of the first song to the client.

[0045] It should be noted that the computer device may execute step 302 first and then step 304; or it may execute step 304 first and then step 302; or it may execute steps 302 and 304 simultaneously. This application embodiment does not limit this.

[0046] In summary, the method provided in this embodiment determines the target beat information of a first song based on the audio file corresponding to the first song and the auxiliary information corresponding to the first song, thereby achieving beat recognition of the first song. Since the beat recognition process considers not only the audio features of the first song but also the semantic features of the text related to the first song, it can improve the accuracy of beat recognition.

[0047] Figure 4 This is a flowchart illustrating a method for determining beat information according to an exemplary embodiment of this application. The method can be used in a computer device, for example, for... Figure 1 The client in the terminal shown, or the server in the server. For example... Figure 4 As shown, the method includes: Step 402: Obtain the audio file corresponding to the first song.

[0048] The type of the first song includes at least one of the following: a complete song, a song fragment, a medley, instrumental music, a musical accompaniment, a song with vocals, and a song without vocals. This application embodiment does not limit the type of the first song. The audio file corresponding to the first song can be an audio file corresponding to the complete first song, or an audio file corresponding to a fragment of the first song; this application embodiment does not limit this.

[0049] Step 404: Obtain the auxiliary information corresponding to the first song.

[0050] The auxiliary information corresponding to the first song includes text related to the first song. Optionally, the auxiliary information corresponding to the first song includes at least one of the style information, structural information, and lyrics of the first song. The style information describes the style of the first song, and the structural information describes its composition. In some embodiments, in addition to the above, the auxiliary information corresponding to the first song may further include at least one of the BPM and time signature information of the first song.

[0051] It should be noted that the computer device may execute step 402 first and then step 404; or it may execute step 404 first and then step 402; or it may execute steps 402 and 404 simultaneously. This application embodiment does not limit this.

[0052] Step 406: Based on the audio file, auxiliary information, and prompts, generate candidate beat information for the first song using a multimodal large model.

[0053] A multimodal large model includes a generative model that supports generating matching output information based on multimodal input information. Multimodality refers to multiple modalities, including text, image, video, and audio modalities. In this embodiment, the multimodal large model supports audio file modal input as well as text modal input. This embodiment does not limit the type of multimodal large model.

[0054] The prompt information guides the multimodal large model to identify the time window number of the beat in the first song based on the audio file and auxiliary information, according to a preset time window length. Correspondingly, the multimodal large model generates candidate beat information for the first song, including the time window number of the beat in the first song. Different time windows have the same length, all of which are preset lengths. The time window numbers can be sequentially increasing or sequentially decreasing.

[0055] Optionally, the multimodal large model in this embodiment is implemented based on a converter architecture. For example, Figure 5 This is a schematic diagram of the converter structure provided in an exemplary embodiment of this application. Figure 5 As shown, the converter 501 includes an encoding network 502 and a decoding network 503. By inputting input information into the encoding network 502, the feature extraction result of the encoding network 502 on the input information can be obtained, i.e., encoded information. By inputting the encoded information into the decoding network 503, the output information of the decoding network 503 can be obtained, i.e., predicted information that matches the input information. The encoding network 502 and decoding network 503 are N-layer structures; the encoding network 502 is a cascaded structure of N encoders, and the decoding network 503 is a cascaded structure of N decoders. The structure of each layer in the encoding network 502 is consistent, and the structure of each layer in the decoding network 503 is similar to that in the decoding network 503.

[0056] Continue to refer to Figure 5 Each layer (encoder) of the 502 encoding network typically includes a multi-head self-attention module, i.e. Figure 5 The left-hand encoder structure contains "self-attention" and a fully connected feedforward network (also known as a feedforward network, FFN)... Figure 5 The "feedforward full connection" in the encoder structure on the left side of the middle.

[0057] Continue to refer to Figure 5 Each layer (decoder) of the 503 decoding network typically includes a Mask Multi-Head Self-Attention Module (which can be considered a type of multi-head self-attention module). Figure 5 The "self-attention" module at the bottom of the decoder structure on the right side. This self-attention module of a cross encoder and decoder (also known as a cross self-attention module, which can be considered a multi-head self-attention module) is... Figure 5 The self-attention mechanism in the middle of the right-hand decoder structure, along with a feedforward fully connected module. Figure 5 The upper part of the decoder structure on the right side is a "feedforward fully connected".

[0058] In this network, the multi-head self-attention module of the encoding network 502 is used to obtain the weight relationship of each word in the input text relative to other words in the input text. The feedforward fully connected module of the encoding network 502 is used to perform non-linear transformation on the input features. The masked multi-head self-attention module of the decoding network 503 has a similar function to the multi-head self-attention module of the encoding network 502, except that it is also used to prevent the decoding network 503 from obtaining the prediction results corresponding to the words following that word in the input text when generating a prediction result that matches a certain word in the input text (during training, this is done). Figure 5 The lower position in the decoder structure on the right side represents the prediction result corresponding to the input text. The cross-self-attention module of the decoding network 503 functions similarly to the multi-head self-attention module of the encoding network 502, the difference being that its input consists of the output information of the previous module in the decoding network 503 and the output information of the last layer in the encoding network 502. The feedforward fully connected module of the decoding network 503 functions similarly to the feedforward fully connected module of the encoding network 502.

[0059] In addition, continue to refer to Figure 5 Each of the modules (multi-head self-attention module, feedforward fully connected module) in the encoding network 502 and decoding network 503 of the converter 501 is provided with a residual connection and a layer normalization (LayerNorm) layer (i.e. Figure 5 The model employs residual and normalization (Add & Norm) layers. Residual connections can be viewed as structures that allow the output of a module to serve as the input to a subsequent, non-adjacent module, reducing model complexity and preventing gradient vanishing. Normalization layers are used to normalize the input information, such as through standardization. Both residual connections and normalization layers stabilize the model's training. During the training of transformer 501, the computer acquires a large amount of textual data for pre-training, enabling transformer 501 to achieve good generalization across different domains of text.

[0060] Generative models are generally classified into three types: autoregressive models, autoencoder models, and sequence-to-sequence models.

[0061] The autoregressive model is pre-trained using a classic language modeling task, i.e., given the preceding text, it predicts the following text. Its structure corresponds to... Figure 5 The decoding network part of the transformer. Because the decoding network can only see the preceding text and not the following text, it is generally used for text generation tasks.

[0062] The Auto Encoder Model is pre-trained using a sentence reconstruction task. This involves pre-destructing the sentence in some way, such as by masking or shuffling the order, with the aim of restoring the destroyed parts to the model. Its structure corresponds to... Figure 5 The encoding network part of the transformer in the model. Unlike autoregressive models, the model can see both preceding and following information. Due to this characteristic, autoencoders are often used for natural language understanding tasks, such as text classification and reading comprehension.

[0063] Sequence-to-sequence model uses both methods simultaneously. Figure 5 The model consists of the encoder and decoder parts of the transducer. The most natural application of this model is in tasks such as text summarization and machine translation; in fact, almost all Natural Language Processing (NLP) tasks can be solved using sequence-to-sequence methods.

[0064] Optionally, the computer device identifies all types of beats and accented beats in the first song using different multimodal large models, thereby obtaining candidate beat information for the first song. In this case, the cue information includes first cue information and second cue information, and the multimodal large model includes a first multimodal large model and a second multimodal large model. The computer device generates first candidate beat information for the first song using the first multimodal large model based on the audio file, auxiliary information, and the first cue information; and generates second candidate beat information for the first song using the second multimodal large model based on the audio file, auxiliary information, and the second cue information. The aforementioned first and second candidate beat information constitute the candidate beat information for the first song.

[0065] The first prompt message instructs the multimodal large model to identify the time window numbers of all types of beats in the first song based on the audio file and auxiliary information, according to a preset time window length. For example, it might say, "Please identify the window number of each beat, using a 3ms window length." Accordingly, the first candidate beat information generated by the first multimodal large model includes the time window numbers of all types of beats in the first song. The second prompt message instructs the multimodal large model to identify the time window numbers of stressed beats in the first song based on the audio file and auxiliary information, according to a preset time window length. For example, it might say, "Please identify the window number of each stressed beat, using a 3ms window length." Accordingly, the second candidate beat information generated by the second multimodal large model includes the time window numbers of stressed beats in the first song.

[0066] Step 408: Determine the target beat information based on the candidate beat information.

[0067] The target beat information of the first song is used to indicate the beat of the first song. In some embodiments, the target beat information of the first song includes the beat types of n beats in the first song, and the timestamps of the n beats. For example, it includes the beat type of each of the n beats, and the timestamp of each of the n beats, where n is a positive integer. The timestamps of the n beats are determined according to the sequence number of the time window in which the n beats belong; that is, the timestamp of each of the n beats is determined according to the sequence number of the time window in which each beat belongs, and different time windows have the same length.

[0068] Optionally, the computer device determines the first timestamps corresponding to all types of beats in the first song based on the first candidate beat information. For example, the computer device determines the first timestamp corresponding to each beat in all types of beats based on the sequence number of the time window in which each type of beat in the first song belongs, as shown in the first candidate beat information. The first timestamp can be determined based on the start position of the time window corresponding to the sequence number of each beat, the end position of the time window corresponding to the sequence number of each beat, or the position between the start and end positions of the time window corresponding to the sequence number of each beat. The computer device also determines the second timestamps corresponding to the stressed beats in the first song based on the second candidate beat information. The process of determining the second timestamp can be referred to the description of the process of determining the first timestamp, which will not be repeated in this embodiment. Then, the computer device replaces the first timestamp that is closest to the second timestamp with the second timestamp to obtain the replaced first timestamp. The first timestamps other than the replaced first timestamp are the unreplaced first timestamps. In this process, the beat corresponding to the first timestamp that was replaced is a repeat, while the beat corresponding to the first timestamp that was not replaced is of a type other than a repeat. The computer then determines the target beat information based on both the replaced and unreplaced first timestamps.

[0069] Optionally, in the process of determining the target beat information based on the replaced first timestamp and the unreplaced first timestamp, the computer device determines the beat type of the beat corresponding to the replaced first timestamp as a repeat, and determines the beat type of the beat corresponding to the unreplaced first timestamp as a beat type other than a repeat. Then, the target beat information is determined based on the replaced first timestamp, the beat type corresponding to the replaced first timestamp, the unreplaced first timestamp, and the beat type corresponding to the unreplaced first timestamp.

[0070] In some embodiments, the computer device uses beat type numbers to represent the beat type of the first song. For example, "1" represents an accented beat, and "2", "3", and "4" represent beat types other than accented beats. After determining that the beat type of the beat corresponding to the first timestamp to be replaced is an accented beat, that is, after determining the beat type number of the beat corresponding to the first timestamp to be replaced as the beat type number of an accented beat, the computer device will also determine whether the first beat in the first song is an accented beat. If the first beat in the first song is an accented beat, the computer device will increment the beat type numbers of the beats between two adjacent accented beats sequentially, that is, sequentially, to obtain the beat type number of each beat in the first song (e.g., 1, 2, 3, 4, 1, 2, 3, 4), that is, the beat type of each beat. If the first beat in the first song is not an accented beat, the computer will cause the beat type number of the beats before the first accented beat in the first song to decrease sequentially (i.e., decrease in reverse order), and the beat type number of the beats between two adjacent accented beats to increase sequentially (i.e., increase in order), thus obtaining the beat type number of each beat in the first song (e.g., 2, 3, 4, 1, 2, 3, 4, 1), that is, the beat type of each beat.

[0071] In some embodiments, when the method provided in this application is applied to a client, after determining the target beat information of the first song, the client can display or use the target beat information of the first song. When the method provided in this application is applied to a server, after determining the target beat information of the first song, the server can use or send the target beat information of the first song to the client. For example, Figure 6 This is a schematic diagram of a user interface provided in an exemplary embodiment of this application. Figure 6 As shown, after the client determines or receives the target beat information of the first song, it can display the target beat information of the first song in the user interface 601, including the beat type 602 of the beat in the first song and the timestamp 603 of the beat in the first song.

[0072] It should be noted that, in addition to the aforementioned audio files and auxiliary information, the computer device can also combine information from other modalities to determine the target beat information of the first song. For example, the computer device acquires an audio feature map corresponding to the first song, which includes an image reflecting the audio features of the first song, such as a Mel-spectrum graph of the first song. Then, the computer device determines the target beat information of the first song based on the audio file, auxiliary information, and audio feature map. The process of determining the target beat information of the first song can be referred to the relevant content above, and will not be repeated here. In this case, the multimodal large model in this application embodiment supports audio file modal input, text modal input, and image modal input; this application embodiment does not limit the type of multimodal large model.

[0073] Optionally, after obtaining the target beat information of the first song, the computer device determines the target song from the reference songs included in the reference library, and obtains the audio file and beat information of the target song through the reference library. The reference library includes multiple reference songs and their corresponding audio files and beat information. The target song is a song among the reference songs whose audio features have a similarity greater than a similarity threshold with the first song. The computer device then adjusts the target beat information of the first song based on the beat information of the target song using a multimodal large model, thereby obtaining the output beat information of the first song. For example, the computer device inputs the audio file and beat information of the target song, the audio file and target beat information of the first song, and adjustment prompts into the multimodal large model, thereby obtaining the output beat information of the first song from the multimodal large model. The adjustment prompts are used to prompt the multimodal large model to optimize the beat information of the first song based on the beat information of the target song, which has similar audio features to the first song.

[0074] In summary, the method provided in this embodiment determines the target beat information of a first song based on the audio file corresponding to the first song and the auxiliary information corresponding to the first song, thereby achieving beat recognition of the first song. Since the beat recognition process considers not only the audio features of the first song but also the semantic features of the text related to the first song, it can improve the accuracy of beat recognition.

[0075] The method provided in this embodiment further improves the accuracy of beat timestamp determination by determining the beat timestamp based on the sequence number of the time window in which the beat belongs. This transforms the beat timestamp recognition task into a time window classification task. Using a multimodal large model to determine the target beat information improves beat recognition accuracy because it possesses strong understanding capabilities and can align features from different modalities. Using different multimodal large models to identify all types of beats and accents separately avoids interference between different types of beats during the beat recognition process, thus improving beat recognition accuracy. Using the timestamp obtained from accent recognition as the final accent timestamp improves accent recognition accuracy because it is determined by a model specifically for accent recognition. Determining the target beat information of the first song by combining it with the audio feature map of the first song improves beat recognition accuracy by additionally referencing features from modalities other than audio and text.

[0076] This application provides a beat recognition method based on multimodal data, which can achieve cross-modal information complementarity and semantic alignment, thereby fundamentally solving the problems of representation ambiguity and lack of context in single-modal methods. For example, at the point of change in song structure (such as the transition from verse to chorus), single-modal methods are prone to beat prediction errors due to abrupt changes in audio features, while multimodal methods, by predicting "this is the chorus" through text modality, can activate the expectation of a stronger and more stable rhythm in advance, achieving a smooth beat phase transition and improving overall tracking stability. In addition, in scenarios with low signal-to-noise ratio or missing key beat signals, text modality can serve as a reliable auxiliary signal source, forming cross-validation with audio modality. At the same time, the multimodal large model can link audio features of different styles with their corresponding text descriptions (such as "funk" and "reggae"), achieving efficient zero-shot or few-shot generalization.

[0077] This application's embodiments are based on an open-source model for model design and optimization. First, since the output of this application only includes text output, the module for audio output in the open-source model is not needed; only the module for text output is required. Furthermore, if the module for text output supports input from image, audio, and text modalities, but this application does not have visual modal information input, the visual encoder module in the text output module is frozen to save training memory consumption.

[0078] For example, Figure 7 This is a schematic diagram illustrating a data flow process provided in an exemplary embodiment of this application. For example... Figure 7 As shown, the audio segment 701 and text input 702 to be input will be converted into token representations by the audio encoder and text encoder of the multimodal large model 703, respectively. Then, the tokens are concatenated and sent into the model to generate text output 704 through autoregression.

[0079] The following describes the process involved in the method provided in the embodiments of this application.

[0080] (1) Data processing.

[0081] Data Processing: Regarding data, this application collected publicly available beat tracking datasets and datasets annotated by internal annotators. The annotation method was audio-text pairs of audio and beat data. For example, continue to refer to... Figure 6 The first column is the timestamp corresponding to the beat (beat), and the second column is the beat type. Type 1 indicates an accented beat, and other types indicate normal beats.

[0082] In addition, this application uses an expert small model to extract information such as the style, structure, and lyrics of the song. These information will be referred to as auxiliary information in the following text.

[0083] Large model input data construction: The large model input includes raw audio and text information, where the text information is also called a prompt. Since the large model calculates the loss based on the accuracy of the token output, for example, if the large model's token dictionary only contains numbers from 0 to 9, and the output timestamp is 2.45, it represents 4 token outputs. When creating ground truth samples, if the output is presented as a timestamp string, the loss will be too small. Specifically, if the ground truth is 3.24 and the model output is 3.54, the token accuracy is 3 / 4 = 0.75. However, the difference between the model output value and the ground truth is already greater than the beat recognition threshold of 0.07, meaning the recognition result is incorrect. Yet, for this example, the model output loss value will be very small, causing the model to converge too slowly.

[0084] Therefore, this application transforms the timestamp regression task into a bucket ID classification task. Simultaneously, a new batch of numeric tokens is added to the model's token dictionary for training; that is, the original word segmentation module also participates in the training. Each numeric token is represented by <|number|>, so the output bucket ID can be directly represented as a token. Specifically, the output is converted to use a fixed time interval as the bucket interval, forcing the model to output the bucket ID (serial number) corresponding to each beat. For example, if the bucket interval is represented by 3ms, then the original 3.24 and 3.54 correspond to bucket IDs <|1080|> and <|1180|>, respectively. Since the bucket ID is already represented as a token, the model's output token accuracy for this example is 0, resulting in a large loss and accelerating model convergence. Finally, the prompt is constructed in the following way: ① For regular rackets: Auxiliary information + "Please use 3ms as the window length to identify the window number of each racket", where the racket number includes the repetition number.

[0085] ② For repetitive beats: Auxiliary information + "Please use 3ms as the window length to identify the window number of each repetitive beat".

[0086] (2) Short audio pre-training.

[0087] Since the large model is primarily trained using Automatic Speech Recognition (ASR) and audio description data during pre-training and fine-tuning, it has very little knowledge of beat. Therefore, this application employs full-scale fine-tuning for the entire model. Considering the training difficulty, this application uses a two-stage training approach. For example, in Stage 1, long audio clips are trimmed into 15-second segments with redundant cuts at 5-second intervals, and short audio clips are used for rapid pre-training. Stage 2, based on Stage 1, loads the model from Stage 1 and extends the audio segments to 30 seconds to address the issue of insufficient contextual information in the 15-second segments, resulting in poor performance on audio data with beat transitions. This stage involves adjusting parameters and training for 5 epochs with a batch size of 192.

[0088] During inference, the test audio is not redundantly cut; instead, 15 seconds are directly trimmed for prediction. After prediction, all results are concatenated to obtain the timestamps of the beat and downbeat, with the beat timestamp containing the downbeat timestamp. For the beat type output, this application uses the following processing logic: Initialize a beat type list T1 with a length equal to the number of beat timestamps. Find the timestamp index closest to the downbeat timestamp from the beat timestamps, denoted as L1. Assign the value 1 to the value in T1 based on the value in L1, indicating the downbeat type. Replace all L1 timestamps with the downbeat timestamps output by the model. For ordinary beat types, assign incrementing indices based on the downbeat type's ID. For example, if the 5th and 9th positions in list T1 are 1, then the beat types for positions 6 to 8 are 2, 3, and 4, respectively. Considering that some songs may not have a downbeat on the first beat (e.g., the first downbeat might be at position 3), this approach is not strictly followed. Based on the overall T1 list, the overall time signature of the song is calculated, and then the time signatures are reduced accordingly. For example, if the overall time signature is 4 / 4, then the time signatures of positions 1 and 2 should be 3 and 4. If it is 6 / 8, then the time signatures of positions 1 and 2 should be 5 and 6, and so on.

[0089] (3) Fine-tuning of long audio.

[0090] To fully utilize the large model's ability to understand the context of long texts, this application extends the cropping length to 30 seconds and performs redundant cutting at 10-second intervals. This stage involves adjusting parameters and training for three epochs with a batch size of 192.

[0091] (4) Beat-Reply Large Model.

[0092] During experiments, it was found that training both beat and repeat types simultaneously, and having a large model output beat and repeat timestamps, resulted in a significantly faster convergence rate for beat timestamp recognition, while the repeat timestamp recognition rate converged much more slowly. Therefore, considering that training beat and repeat recognition tasks together in the same model could cause interference, this application uses two separate models: a large beat model and a large repeat model. The large beat model outputs the beat timestamp, while the large repeat model only outputs the repeat timestamp.

[0093] (5) Indicator effect.

[0094] The recognition accuracy rates for the 50 test songs (37 non-time-lapse and 13 time-lapse) are shown in Table 1.

[0095] Here, beat represents the accuracy of all beat recognition, downbeat represents the accuracy of downbeat recognition, and avg represents the overall beat recognition accuracy.

[0096] It should be noted that this application may display prompt interfaces, pop-ups, or output voice prompts before and during the collection of user data. These prompt interfaces, pop-ups, or voice prompts are used to inform the user that their data is being collected. This ensures that the application only begins the steps for collecting user data after receiving confirmation from the user regarding the prompt interface or pop-up; otherwise (i.e., without user confirmation), the steps for collecting user data end, meaning no user data is collected. In other words, all user data collected in this application is collected with the user's consent and authorization, and the collection, use, and processing of related user data must comply with the relevant laws, regulations, and standards of the relevant countries and regions.

[0097] It should be noted that the order of the method steps provided in the embodiments of this application can be appropriately adjusted, and the steps can also be added or removed as appropriate. Any method variations that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the protection scope of this application, and therefore will not be elaborated further.

[0098] Figure 8 This is a schematic diagram of the structure of a beat information determination device provided in an exemplary embodiment of this application. Figure 8 As shown, the device includes: Module 801 is used to obtain the audio file corresponding to the first song; The acquisition module 801 is further configured to acquire auxiliary information corresponding to the first song, the auxiliary information including text related to the first song; The determining module 802 is used to determine the target beat information of the first song based on the audio file corresponding to the first song and the auxiliary information corresponding to the first song, wherein the target beat information is used to indicate the beat of the first song.

[0099] In an optional design, the target beat information includes the beat type of n beats in the first song, and the timestamps of the n beats; The timestamps of the n beats are determined according to the sequence number of the time window in which the n beats are located. Different time windows have the same length, and n is a positive integer.

[0100] In an optional design, the determining module 802 is used to: Based on the audio file, the auxiliary information, and the prompt information, candidate beat information for the first song is generated using a multimodal large model; The target beat information is determined based on the candidate beat information; The multimodal large model includes a generative model that supports generating matching output information based on multimodal input information. The prompting information is used to prompt the multimodal large model to identify the sequence number of the time window where the beat in the first song is located according to the audio file and the auxiliary information, based on a time window of a preset length.

[0101] In an optional design, the prompting information includes a first prompting information and a second prompting information, and the multimodal large model includes a first multimodal large model and a second multimodal large model; the determining module 802 is used for: Based on the audio file, the auxiliary information, and the first prompt information, the first candidate beat information of the first song is generated through the first multimodal large model; Based on the audio file, the auxiliary information, and the second prompt information, the second candidate beat information of the first song is generated through the second multimodal large model; The first prompt information is used to prompt the multimodal large model to identify the sequence number of the time window where all types of beats in the first song are located according to the audio file and the auxiliary information, and according to the preset length of the time window; the second prompt information is used to prompt the multimodal large model to identify the sequence number of the time window where the stressed beat is located in the first song according to the audio file and the auxiliary information, and according to the preset length of the time window.

[0102] In an optional design, the determining module 802 is used to: Based on the first candidate beat information, determine the first timestamp corresponding to all types of beats in the first song, and based on the second candidate beat information, determine the second timestamp corresponding to the accented beats in the first song; The first timestamp that is closest to the second timestamp is replaced by the second timestamp to obtain the replaced first timestamp; The target beat information is determined based on the first timestamp that was replaced and the first timestamp that was not replaced.

[0103] In an optional design, the determining module 802 is used to: The beat type corresponding to the first timestamp that was replaced is determined to be the rebeat; The beat type corresponding to the first timestamp that was not replaced is determined to be a beat type other than the stressed beat; The target beat information is determined based on the replaced first timestamp, the beat type corresponding to the replaced first timestamp, the unreplaced first timestamp, and the beat type corresponding to the unreplaced first timestamp.

[0104] In an optional design, the auxiliary information includes at least one of the style information of the first song, the structural information of the first song, and the lyrics of the first song.

[0105] In an optional design, the acquisition module 801 is used for: Obtain the audio feature map corresponding to the first song, wherein the audio feature map includes an image reflecting the audio features of the first song; The determining module 802 is used for: The target beat information of the first song is determined based on the audio file, the auxiliary information, and the audio feature map.

[0106] It should be noted that the beat information determination device provided in the above embodiments is only an example of the division of the above functional modules. In practical applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the device can be divided into different functional modules to complete all or part of the functions described above. In addition, the beat information determination device and the beat information determination method embodiments provided in the above embodiments belong to the same concept, and their specific implementation process can be found in the method embodiments, which will not be repeated here.

[0107] Embodiments of this application also provide a computer device, which includes a processor and a memory. The memory stores at least one instruction, at least one program, code set, or instruction set. The at least one instruction, at least one program, code set, or instruction set is loaded and executed by the processor to implement the method for determining the clock information provided in the above-described method embodiments.

[0108] In some embodiments, the computer device described above is a terminal. For example, Figure 9 This is a schematic diagram of the structure of a terminal provided in an exemplary embodiment of this application.

[0109] Typically, terminal 900 includes a processor 901 and a memory 902.

[0110] Processor 901 may include one or more processing cores, such as a quad-core processor, an octa-core processor, etc. Processor 901 may be implemented using at least one hardware form selected from DSP (Digital Signal Processing), FPGA (Field-Programmable Gate Array), and PLA (Programmable Logic Array). Processor 901 may also include a main processor and a coprocessor. The main processor, also known as a CPU (Central Processing Unit), is used to process data in the wake-up state; the coprocessor is a low-power processor used to process data in the standby state. In some embodiments, processor 901 may integrate a GPU (Graphics Processing Unit), which is responsible for rendering and drawing the content to be displayed on the screen. In some embodiments, processor 901 may also include an AI (Artificial Intelligence) processor, which is used to handle computational operations related to machine learning.

[0111] The memory 902 may include one or more computer-readable storage media, which may be non-transitory. The memory 902 may also include high-speed random access memory and non-volatile memory, such as one or more disk storage devices or flash memory devices. In some embodiments, the non-transitory computer-readable storage media in the memory 902 are used to store at least one instruction, which is executed by the processor 901 to implement the method for determining clock information provided in the method embodiments of this application.

[0112] In some embodiments, the terminal 900 may also optionally include a peripheral device interface 903 and at least one peripheral device. The processor 901, memory 902, and peripheral device interface 903 can be connected via a bus or signal line. Each peripheral device can be connected to the peripheral device interface 903 via a bus, signal line, or circuit board. Specifically, the peripheral device includes at least one of the following: a radio frequency circuit 904, a display screen 905, a camera assembly 906, an audio circuit 907, and a power supply 908.

[0113] Peripheral device interface 903 can be used to connect at least one I / O (Input / Output) related peripheral device to processor 901 and memory 902. In some embodiments, processor 901, memory 902 and peripheral device interface 903 are integrated on the same chip or circuit board; in some other embodiments, any one or two of processor 901, memory 902 and peripheral device interface 903 can be implemented on separate chips or circuit boards, and this application embodiment does not limit this.

[0114] The radio frequency (RF) circuit 904 is used to receive and transmit RF (Radio Frequency) signals, also known as electromagnetic signals. The RF circuit 904 communicates with communication networks and other communication devices via electromagnetic signals. The RF circuit 904 converts electrical signals into electromagnetic signals for transmission, or converts received electromagnetic signals back into electrical signals. Optionally, the RF circuit 904 includes: an antenna system, an RF transceiver, one or more amplifiers, a tuner, an oscillator, a digital signal processor, a codec chipset, a user identity module card, etc. The RF circuit 904 can communicate with other terminals through at least one wireless communication protocol. This wireless communication protocol includes, but is not limited to: the World Wide Web, metropolitan area networks, intranets, various generations of mobile communication networks (2G, 3G, 4G, and 5G), wireless local area networks, and / or WiFi (Wireless Fidelity) networks. In some embodiments, the RF circuit 904 may also include circuitry related to NFC (Near Field Communication), which is not limited in this application.

[0115] Display screen 905 is used to display a UI (User Interface). This UI may include graphics, text, icons, videos, and any combination thereof. When display screen 905 is a touch display screen, it also has the ability to collect touch signals on or above its surface. These touch signals can be input as control signals to processor 901 for processing. In this case, display screen 905 can also be used to provide virtual buttons and / or a virtual keyboard, also known as soft buttons and / or a soft keyboard. In some embodiments, there may be one display screen 905, which serves as the front panel of terminal 900; in other embodiments, there may be at least two display screens 905, respectively disposed on different surfaces of terminal 900 or in a folded design; in still other embodiments, display screen 905 may be a flexible display screen, disposed on a curved or folded surface of terminal 900. Furthermore, display screen 905 may be configured as a non-rectangular, irregular shape, i.e., a non-rectangular screen. Display screen 905 may be made of materials such as LCD (Liquid Crystal Display) or OLED (Organic Light-Emitting Diode).

[0116] The camera assembly 906 is used to acquire images or videos. Optionally, the camera assembly 906 includes a front-facing camera and a rear-facing camera. Typically, the front-facing camera is located on the front panel of the terminal 900, and the rear-facing camera is located on the back of the terminal. In some embodiments, there are at least two rear-facing cameras, which are any one of a main camera, a depth-sensing camera, a wide-angle camera, and a telephoto camera, to achieve background blurring by fusion of the main camera and the depth-sensing camera, panoramic shooting by fusion of the main camera and the wide-angle camera, VR (Virtual Reality) shooting, or other fusion shooting functions. In some embodiments, the camera assembly 906 may also include a flash. The flash can be a single-color temperature flash or a dual-color temperature flash. A dual-color temperature flash refers to a combination of a warm-light flash and a cool-light flash, which can be used for light compensation at different color temperatures.

[0117] The audio circuit 907 may include a microphone and a speaker. The microphone is used to collect sound waves from the user and the environment, converting them into electrical signals that are input to the processor 901 for processing, or to the radio frequency circuit 904 for voice communication. For stereo sound acquisition or noise reduction purposes, multiple microphones may be used, each positioned at a different location on the terminal 900. The microphone may also be an array microphone or an omnidirectional microphone. The speaker is used to convert electrical signals from the processor 901 or the radio frequency circuit 904 into sound waves. The speaker may be a conventional diaphragm speaker or a piezoelectric ceramic speaker. When the speaker is a piezoelectric ceramic speaker, it can convert electrical signals not only into audible sound waves but also into inaudible sound waves for purposes such as distance measurement. In some embodiments, the audio circuit 907 may also include a headphone jack.

[0118] Power supply 908 is used to power the various components in terminal 900. Power supply 908 can be AC ​​power, DC power, a disposable battery, or a rechargeable battery. When power supply 908 includes a rechargeable battery, the rechargeable battery can be a wired rechargeable battery or a wireless rechargeable battery. A wired rechargeable battery is a battery that is charged via a wired line, while a wireless rechargeable battery is a battery that is charged via a wireless coil. The rechargeable battery can also be used to support fast charging technology.

[0119] In some embodiments, the terminal 900 further includes one or more sensors 909. The one or more sensors 909 include, but are not limited to, an accelerometer 910, a gyroscope 911, a pressure sensor 912, an optical sensor 913, and a proximity sensor 914.

[0120] Accelerometer 910 can detect the magnitude of acceleration along the three coordinate axes of a coordinate system established by terminal 900. For example, accelerometer 910 can be used to detect the components of gravitational acceleration along the three coordinate axes. Processor 901 can control touchscreen 905 to display the user interface in landscape or portrait view based on the gravitational acceleration signal acquired by accelerometer 910. Accelerometer 910 can also be used for games or for acquiring user motion data.

[0121] The gyroscope sensor 911 can detect the orientation and rotation angle of the terminal 900. The gyroscope sensor 911, in conjunction with the accelerometer sensor 910, can collect 3D motion data from the user on the terminal 900. Based on the data collected by the gyroscope sensor 911, the processor 901 can perform the following functions: motion sensing (e.g., changing the UI based on the user's tilt), image stabilization during shooting, game control, and inertial navigation.

[0122] The pressure sensor 912 can be disposed on the side bezel of the terminal 900 and / or on the lower layer of the touch display screen 905. When the pressure sensor 912 is disposed on the side bezel of the terminal 900, it can detect the user's grip signal on the terminal 900, and the processor 901 can perform left / right hand recognition or quick operation based on the grip signal collected by the pressure sensor 912. When the pressure sensor 912 is disposed on the lower layer of the touch display screen 905, the processor 901 can control the operable controls on the UI interface based on the user's pressure operation on the touch display screen 905. The operable controls include at least one of button controls, scroll bar controls, icon controls, and menu controls.

[0123] An optical sensor 913 is used to collect ambient light intensity. In one embodiment, the processor 901 can control the display brightness of the touch screen 905 based on the ambient light intensity collected by the optical sensor 913. Specifically, when the ambient light intensity is high, the display brightness of the touch screen 905 is increased; when the ambient light intensity is low, the display brightness of the touch screen 905 is decreased. In another embodiment, the processor 901 can also dynamically adjust the shooting parameters of the camera assembly 906 based on the ambient light intensity collected by the optical sensor 913.

[0124] The proximity sensor 914, also known as a distance sensor, is typically located on the front panel of the terminal 900. The proximity sensor 914 is used to detect the distance between the user and the front of the terminal 900. In one embodiment, when the proximity sensor 914 detects that the distance between the user and the front of the terminal 900 is gradually decreasing, the processor 901 controls the touchscreen display 905 to switch from a screen-on state to a screen-off state; when the proximity sensor 914 detects that the distance between the user and the front of the terminal 900 is gradually increasing, the processor 901 controls the touchscreen display 905 to switch from a screen-off state to a screen-on state.

[0125] Those skilled in the art will understand that Figure 9 The structure shown does not constitute a limitation on terminal 900, and may include more or fewer components than shown, or combine certain components, or use different component arrangements.

[0126] In some embodiments, the computer device described above is a server. For example, Figure 10 This is a schematic diagram of the structure of a computer device provided in an exemplary embodiment of this application, which can be implemented as a server.

[0127] The computer device 1000 includes a central processing unit (CPU) 1001, a system memory 1004 including random access memory (RAM) 1002 and read-only memory (ROM) 1003, and a system bus 1005 connecting the system memory 1004 and the CPU 1001. The computer device 1000 also includes a basic input / output system (I / O system) 1006 to facilitate information transfer between various devices within the computer device, and a mass storage device 1007 for storing the operating system 1013, application programs 1014, and other program modules 1015.

[0128] The basic input / output system 1006 includes a display 1008 for displaying information and an input device 1009 for user input, such as a mouse or keyboard. Both the display 1008 and the input device 1009 are connected to the central processing unit 1001 via an input / output controller 1010 connected to the system bus 1005. The basic input / output system 1006 may also include the input / output controller 1010 for receiving and processing input from multiple other devices such as a keyboard, mouse, or electronic stylus. Similarly, the input / output controller 1010 also provides output to a display screen, printer, or other types of output devices.

[0129] The mass storage device 1007 is connected to the central processing unit 1001 via a mass storage controller (not shown) connected to the system bus 1005. The mass storage device 1007 and its associated computer-readable storage media provide non-volatile storage for the computer device 1000. That is, the mass storage device 1007 may include computer-readable storage media (not shown) such as a hard disk or a compact disc read-only memory (CD-ROM) drive.

[0130] Without loss of generality, the computer-readable storage medium may include computer storage media and communication media. Computer storage media include volatile and non-volatile, removable and non-removable media implemented using any method or technology for storing information such as computer-readable storage instructions, data structures, program modules, or other data. Computer storage media include RAM, ROM, erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other solid-state storage devices, CD-ROM, digital versatile disc (DVD) or other optical storage, magnetic tape cassettes, magnetic tape, disk storage, or other magnetic storage devices. Of course, those skilled in the art will recognize that the computer storage medium is not limited to the above-mentioned types. The system memory 1004 and mass storage device 1007 described above can be collectively referred to as memory.

[0131] The memory stores one or more programs, which are configured to be executed by one or more central processing units 1001. The one or more programs contain instructions for implementing the above method embodiments, and the central processing unit 1001 executes the one or more programs to implement the methods provided by the various method embodiments described above.

[0132] According to various embodiments of this application, the computer device 1000 can also be connected to a remote computer device on a network, such as the Internet. That is, the computer device 1000 can be connected to the network 1012 via the network interface unit 1011 connected to the system bus 1005, or the network interface unit 1011 can be used to connect to other types of networks or remote computer device systems (not shown).

[0133] The memory further includes one or more programs stored in the memory, and the one or more programs include steps performed by a computer device in the methods provided in the embodiments of this application.

[0134] This application also provides a computer-readable storage medium storing at least one instruction, at least one program, code set, or instruction set. When the at least one instruction, at least one program, code set, or instruction set is loaded and executed by the processor of a computer device, the method for determining the clock information provided in the above-described method embodiments is implemented.

[0135] This application also provides a computer program product or computer program that includes computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the computer device to perform the method for determining clock information provided in the above-described method embodiments.

[0136] Those skilled in the art will understand that all or part of the steps of the above embodiments can be implemented by hardware or by a program instructing related hardware. The program can be stored in a computer-readable storage medium, such as a read-only memory, a disk, or an optical disk.

[0137] The above description is merely an optional embodiment of this application and is not intended to limit this application. Any modifications, equivalent switching, improvements, etc., made within the spirit and principles of this application should be included within the protection scope of this application.

Claims

1. A method for determining beat information, characterized in that, The method includes: Get the audio file corresponding to the first song; Obtain auxiliary information corresponding to the first song, wherein the auxiliary information includes text related to the first song; Based on the audio file corresponding to the first song and the auxiliary information corresponding to the first song, the target beat information of the first song is determined, and the target beat information is used to indicate the beat of the first song.

2. The method according to claim 1, characterized in that, The target beat information includes the beat type of n beats in the first song, and the timestamp of the n beats; The timestamps of the n beats are determined according to the sequence number of the time window in which the n beats are located. Different time windows have the same length, and n is a positive integer.

3. The method according to claim 2, characterized in that, The step of determining the target beat information of the first song based on the audio file corresponding to the first song and the auxiliary information corresponding to the first song includes: Based on the audio file, the auxiliary information, and the prompt information, candidate beat information for the first song is generated using a multimodal large model; The target beat information is determined based on the candidate beat information; The multimodal large model includes a generative model that supports generating matching output information based on multimodal input information. The prompting information is used to prompt the multimodal large model to identify the sequence number of the time window where the beat in the first song is located according to the audio file and the auxiliary information, based on a time window of a preset length.

4. The method according to claim 3, characterized in that, The prompt information includes first prompt information and second prompt information; the multimodal large model includes a first multimodal large model and a second multimodal large model; generating candidate beat information for the first song using the multimodal large model based on the audio file, the auxiliary information, and the prompt information includes: Based on the audio file, the auxiliary information, and the first prompt information, the first candidate beat information of the first song is generated through the first multimodal large model; Based on the audio file, the auxiliary information, and the second prompt information, the second candidate beat information of the first song is generated through the second multimodal large model; The first prompt information is used to prompt the multimodal large model to identify the sequence number of the time window where all types of beats in the first song are located according to the audio file and the auxiliary information, and according to the preset length of the time window; the second prompt information is used to prompt the multimodal large model to identify the sequence number of the time window where the stressed beat is located in the first song according to the audio file and the auxiliary information, and according to the preset length of the time window.

5. The method according to claim 4, characterized in that, Determining the target beat information based on the candidate beat information includes: Based on the first candidate beat information, determine the first timestamp corresponding to all types of beats in the first song, and based on the second candidate beat information, determine the second timestamp corresponding to the accented beats in the first song; The first timestamp that is closest to the second timestamp is replaced by the second timestamp to obtain the replaced first timestamp; The target beat information is determined based on the first timestamp that was replaced and the first timestamp that was not replaced.

6. The method according to claim 5, characterized in that, Determining the target beat information based on the replaced first timestamp and the unreplaced first timestamp includes: The beat type corresponding to the first timestamp that was replaced is determined to be the rebeat; The beat type corresponding to the first timestamp that was not replaced is determined to be a beat type other than the stressed beat; The target beat information is determined based on the replaced first timestamp, the beat type corresponding to the replaced first timestamp, the unreplaced first timestamp, and the beat type corresponding to the unreplaced first timestamp.

7. The method according to any one of claims 1 to 6, characterized in that, The auxiliary information includes at least one of the following: the style information of the first song, the structural information of the first song, and the lyrics of the first song.

8. The method according to any one of claims 1 to 6, characterized in that, The method further includes: Obtain the audio feature map corresponding to the first song, wherein the audio feature map includes an image reflecting the audio features of the first song; The step of determining the target beat information of the first song based on the audio file corresponding to the first song and the auxiliary information corresponding to the first song includes: The target beat information of the first song is determined based on the audio file, the auxiliary information, and the audio feature map.

9. A device for determining beat information, characterized in that, The device includes: The acquisition module is used to acquire the audio file corresponding to the first song; The acquisition module is further configured to acquire auxiliary information corresponding to the first song, the auxiliary information including text related to the first song; The determining module is used to determine the target beat information of the first song based on the audio file corresponding to the first song and the auxiliary information corresponding to the first song, wherein the target beat information is used to indicate the beat of the first song.

10. A computer device, characterized in that, The computer device includes a processor and a memory, the memory storing at least one program, which is loaded and executed by the processor to implement the method for determining beat information as described in any one of claims 1 to 9.

11. A computer-readable storage medium, characterized in that, The readable storage medium stores at least one program, which is loaded and executed by a processor to implement the method for determining beat information as described in any one of claims 1 to 9.

12. A computer program product, characterized in that, The computer program product includes computer instructions stored in a computer-readable storage medium, a processor of a computer device reading the computer instructions from the computer-readable storage medium, and the processor executing the computer instructions to cause the computer device to perform the method for determining beat information as described in any one of claims 1 to 9.

Images