Lyric and rhythm recognition printing method based on audio fingerprints and related equipment

Abstract

The invention discloses a lyric and rhythm printing method based on an audio fingerprint and related equipment, and the method comprises the steps: carrying out the preprocessing operation of a to-be-processed audio, and obtaining the preprocessed audio data; dividing the preprocessed audio data into a plurality of segments; respectively extracting key features of each fragment; converting each key feature into a corresponding audio fingerprint through a Hash algorithm; searching audio fingerprints in a preset song library and judging whether matching succeeds or not; if the matching is successful, first primary printing information is generated, the first primary printing information comprises lyric information and rhythm information arranged according to a preset format, and the lyric information and the rhythm information are lyric information and rhythm information corresponding to the matched audio fingerprint; the first primary printing information is rendered according to a preset style, and a final printing picture is obtained; and sending the final printing picture to the printing equipment to enable the printing equipment to print the final printing picture. The lyrics and rhythms can be quickly and accurately recognized and printed, and the accuracy and efficiency are improved.

Description

Technical Field

[0001] The present application belongs to the field of audio recognition printing, and in particular, relates to a lyrics and rhythm printing method based on audio fingerprint and related equipment. Background Art

[0002] With the popularization of digital music and the widespread use of smart devices, users' requirements for music experience are constantly improving. Among them, accurate lyrics display and rhythm recognition have become key factors in improving user interactive experience.

[0003] At present, lyrics display and rhythm recognition mainly rely on timestamp-based lyrics synchronization, audio wave analysis, song recognition technology, and speech recognition technology. However, timestamp-based lyrics synchronization requires the preparation of lyrics files with timestamps in advance, and the corresponding lyrics are displayed by matching the audio playback time. However, this method requires a lot of manual annotation work and is difficult to adapt to the needs of massive music content. Audio wave analysis is easily affected by background noise and has low accuracy in complex music environments. Song recognition technology can usually only provide song information and cannot achieve real-time lyrics synchronization and rhythm recognition. Speech recognition technology has low recognition accuracy because the human voice in music is often mixed with the sound of instruments. In summary, the relevant technologies still have the problems of low printing accuracy and low efficiency of lyrics and rhythm recognition. Summary of the invention

[0004] The present application provides a lyrics and rhythm printing method and related equipment based on audio fingerprints, which are used to quickly and accurately identify and print lyrics and rhythms, thereby improving accuracy and efficiency.

[0005] In a first aspect, the present application provides a lyrics and rhythm printing method based on audio fingerprint, which performs a preprocessing operation on the audio to be processed to obtain preprocessed audio data; Divide the preprocessed audio data into a number of segments, each segment containing an equal amount of audio data; Extract the key features of each fragment separately; Each key feature is converted into a corresponding audio fingerprint through a hash algorithm; Search the audio fingerprint in the preset song library and determine whether it is a successful match; If the match is successful, first primary print information is generated, the first primary print information includes lyrics information and rhythm information arranged in a preset format, the lyrics information and rhythm information are the lyrics information and rhythm information corresponding to the matched audio fingerprint; Rendering the first primary printing information according to a preset style to obtain a final printed image; The final print image is sent to the printing device so that the printing device prints the final print image.

[0006] By adopting the above technical solution, a series of steps are performed to pre-process the audio to be processed, segment, extract key features, generate audio fingerprints, match the preset song library, generate printing information, render print pictures and finally print, thus realizing a fully automated lyrics and rhythm printing method based on audio fingerprint technology. The shortcomings of traditional manual printing of lyrics, such as low efficiency and easy errors, are overcome. By introducing audio fingerprint technology, the song corresponding to the audio can be quickly and accurately identified, and print pictures containing lyrics and rhythm information can be automatically generated, which greatly improves the efficiency and accuracy of lyrics and rhythm printing. At the same time, the invention adopts a standardized audio fingerprint algorithm to convert audio key features into unique fingerprint information through hash operations, which can effectively avoid copyright disputes. In addition, the lyrics and rhythm information are typeset and rendered in a preset format and style to ensure the readability of the printout. It is used to quickly and accurately identify and print lyrics and rhythms, improving accuracy and efficiency.

[0007] In combination with some embodiments of the first aspect, in some embodiments, performing a preprocessing operation on the audio to be processed to obtain preprocessed audio data specifically includes: Filtering the audio to be processed to obtain filtered audio; Perform noise reduction processing on the filtered audio to obtain a denoised audio frequency; Adjust the volume of the noise-removed audio to the preset volume to obtain the audio with the same volume; Perform spectrum splitting operation on audio with the same volume to obtain preprocessed audio data.

[0008] By adopting the above technical solution, filtering the original audio can remove low-frequency and high-frequency noise in the audio and improve the signal-to-noise ratio; then further noise reduction processing is performed on the filtered audio to eliminate the influence of residual noise; then the volume is adjusted to the preset value to unify the volume differences of different sound sources; finally, the spectrum is split on the audio with the same volume to obtain preprocessed audio data that is easy to extract key features. This series of preprocessing operations can retain the effective information of the audio to the greatest extent, remove various noise interferences, and lay a good foundation for subsequent audio fingerprint extraction. At the same time, the preprocessing process is implemented using a standard algorithm, with a clear and definite process, strong versatility, and can adapt to different sound sources, reducing the difficulty of algorithm implementation. In addition, the preprocessed audio data is small in scale and simple in structure, which can speed up the subsequent fingerprint extraction and comparison.

[0009] In combination with some embodiments of the first aspect, in some embodiments, performing a spectrum splitting operation on audio with the same volume to obtain preprocessed audio data specifically includes: Convert preprocessed audio data into spectral data; The spectrum data is segmented using a preset algorithm to obtain preprocessed audio data.

[0010] By adopting the above technical solution, the data is first converted into spectrum data and then segmented using a preset algorithm. Converting the time domain audio signal into a frequency domain signal through mathematical methods such as Fourier transform can reveal the frequency composition of the audio, which is conducive to capturing key information such as pitch and timbre that reflect the characteristics of the song. This provides a more essential and stable data basis for the extraction and comparison of audio fingerprints. The segmentation processing based on spectrum data avoids the randomness and uncertainty of the time domain waveform. The use of a preset algorithm can simplify the amount of calculation while ensuring the segmentation accuracy, making it easy to implement in engineering. In addition, the segmented spectrum data presents a regular time-frequency distribution, which makes it easier to extract key features that reflect the essential attributes of the song.

[0011] In combination with some embodiments of the first aspect, in some embodiments, after sending the final printed image to a printing device so that the printing device prints the final printed image, the method further includes: If the match fails, the key features are searched in the preset search engine to obtain several search results; Extracting search result feature information of each search result respectively; Calculate the similarity between the key features and the search result feature information respectively to obtain the similarity of several songs; Determine the lyrics information and rhythm information corresponding to the song with the greatest similarity; generating second primary printing information, wherein the second primary printing information includes lyrics information and rhythm information arranged in a preset format; Rendering the second primary printing information according to a preset style to obtain a final printed image; The step of sending the final print image to the printing device is executed so that the printing device prints the final print image.

[0012] By adopting the above technical solution, when the audio fingerprint fails to match in the preset song library, the related song results can be obtained by retrieving the key features in the search engine. By extracting features from these results and calculating the similarity with the key features of the audio to be processed, the closest song can be found. Using the lyrics and rhythm information of the best matching song, a suboptimal printout can be generated as a remedial measure when the song cannot be directly identified. This approximate matching strategy based on similarity comparison can effectively expand the adaptation range of the algorithm, so that it is not limited to the preset song library, and enhances robustness. At the same time, both retrieval and similarity calculation can be implemented using mature information retrieval technology, which will not significantly increase the complexity of the system. Providing an approximate printout for difficult-to-identify songs is better than being unable to process them at all, and can reduce errors and omissions caused by recognition failures.

[0013] In combination with some embodiments of the first aspect, in some embodiments, after respectively calculating the similarity between the key feature and the search result feature information to obtain the similarities of several songs, the method further includes: The search result feature information in the search result corresponding to the greatest song similarity is converted into the corresponding added audio fingerprint through a hash algorithm; Add audio fingerprints to the preset song library.

[0014] By adopting the above technical solution, when the audio fingerprint matching fails, the system retrieves similar songs through the search engine and calculates the similarity between the key features and the search results. For the results with the highest similarity, its feature information is extracted, converted into a new audio fingerprint through the hash algorithm, and added to the preset song library. This mechanism of dynamically expanding the fingerprint library according to the actual recognition situation can continuously accumulate new song fingerprints, making the algorithm smarter and smarter. With the enrichment of recognition history, the song library will become more and more comprehensive, and its adaptability to unfamiliar audio will become stronger and stronger. At the same time, the new fingerprints are stored in hash to avoid repeated additions and maintain efficient retrieval of the song library. In addition, this strategy makes full use of the powerful information acquisition capabilities of the search engine, realizes cross-library association through similarity comparison, eliminates the need for manual maintenance of the song library, and reduces the system operation and maintenance costs.

[0015] In combination with some embodiments of the first aspect, in some embodiments, rendering the first primary printing information according to a preset style to obtain a final printed image specifically includes: Preset styles include font, font size and font color; Rendering the lyrics information into the pre-printed image according to a preset style to obtain a secondary printed image; Determine the visual elements of the rhythmic information; Rotate the visual element into the secondary print image to obtain the backup print image; Add the time to the prepared print image to obtain the final print image.

[0016] By adopting the above technical solution, the preset style includes font, size, color and other attributes to ensure that the lyrics are neatly laid out, with clear layers, and easy for users to read. Then, based on the preset style, a secondary print image containing lyrics information is generated as a master for subsequent visual processing. Next, unique visual elements are designed for the rhythmic information, such as patterns such as notes and dance movements, to enhance the visual impact. These visual elements are then integrated with the secondary print image to obtain a pre-print image with both content and form beauty. Finally, time information is added to the pre-print image to form the final printed product.

[0017] In combination with some embodiments of the first aspect, in some embodiments, adding time to the prepared print picture to obtain the final print picture specifically includes: Determine the time type to be added, which includes the current playing time, the total song time and the remaining time; Format the time type to get a unified time type; Use the image processing library to render the unified time type into the print-ready image to obtain the final print image.

[0018] By adopting the above technical solutions, the system can decide which time elements to display in the printed picture according to the actual application scenario. The playing time can remind the user of the current progress, the total time can show the scale of the song, and the remaining time is convenient for controlling the listening schedule. This customizable time display scheme takes into account both comprehensive information and practicality. Then, to ensure the consistency of different time formats, the system standardizes the selected time types and uniformly converts them into minutes: seconds, eliminating format differences. Finally, using a mature image processing library, with the print picture as the canvas, the formatted time information is rendered to the appropriate position of the picture, perfectly integrated with other elements, and the final print picture is generated. The rendering process is highly automated, does not require human intervention, and the time information is clear and beautiful.

[0019] An embodiment of the present application provides a device related to lyrics and rhythm printing based on audio fingerprints, and the device includes a system, a storage medium and a computer program product.

[0020] In a second aspect, an embodiment of the present application provides a lyrics and rhythm printing system based on audio fingerprints, and the lyrics and rhythm printing system based on audio fingerprints includes: one or more processors and a memory; the memory is coupled to the one or more processors, the memory is used to store computer program code, the computer program code includes computer instructions, and one or more processors call the computer instructions to enable the system to execute the method described in the first aspect and any possible implementation method of the first aspect.

[0021] In a third aspect, an embodiment of the present application provides a computer-readable storage medium, comprising instructions, which, when executed on a system, causes the system to execute the method described in the first aspect and any possible implementation of the first aspect.

[0022] In a fourth aspect, an embodiment of the present application provides a computer program product, characterized in that when the computer program product runs on a system, the system executes the method described in any possible implementation manner in the first aspect.

[0023] One or more technical solutions provided in the embodiments of the present application have at least the following technical effects or advantages: 1. The present application provides a method for printing lyrics and rhythms based on audio fingerprints, which includes a series of steps of preprocessing, segmenting, extracting key features, generating audio fingerprints, matching a preset song library, generating printing information, rendering and printing pictures, and finally printing the processed audio, thereby realizing a fully automated method for printing lyrics and rhythms based on audio fingerprint technology. The shortcomings of traditional manual printing of lyrics, such as low efficiency and prone to errors, are overcome. By introducing audio fingerprint technology, the song corresponding to the audio can be quickly and accurately identified, and printed pictures containing lyrics and rhythm information can be automatically generated, which greatly improves the efficiency and accuracy of lyrics and rhythm printing. At the same time, the invention adopts a standardized audio fingerprint algorithm, which converts audio key features into unique fingerprint information through hash operations, which can effectively avoid copyright disputes. In addition, the lyrics and rhythm information are typeset and rendered in preset formats and styles to ensure the readability of the printout. It is used to quickly and accurately identify and print lyrics and rhythms, improving accuracy and efficiency.

[0024] 2. The present application provides a method for printing lyrics and rhythms based on audio fingerprints. When the audio fingerprint fails to match in the preset song library, the related song results can be obtained by retrieving key features in the search engine. By extracting features from these results and calculating the similarity with the key features of the audio to be processed, the closest song can be found. Using the lyrics and rhythm information of the best matching song, a suboptimal printout can be generated as a remedial measure when the song cannot be directly identified. This approximate matching strategy based on similarity comparison can effectively expand the adaptation range of the algorithm, so that it is not limited to the preset song library, and enhances robustness. At the same time, both retrieval and similarity calculation can be implemented using mature information retrieval technology, which will not significantly increase the complexity of the system. Providing approximate printouts for difficult-to-identify songs is better than being unable to process them at all, and can reduce errors and omissions caused by recognition failures.

[0025] 3. The present application provides a method for printing lyrics and rhythms based on audio fingerprints. When the audio fingerprint match fails, the system retrieves similar songs through a search engine and calculates the similarity between the key features and the search results. For the results with the highest similarity, its feature information is extracted, converted into a new audio fingerprint through a hash algorithm, and added to the preset song library. This mechanism of dynamically expanding the fingerprint library according to the actual recognition situation can continuously accumulate new song fingerprints, making the algorithm smarter and smarter. With the enrichment of the recognition history, the song library will become more and more comprehensive, and its adaptability to unfamiliar audio will become stronger and stronger. At the same time, the new fingerprints are stored in hash format to avoid repeated additions and maintain efficient retrieval of the song library. In addition, this strategy makes full use of the powerful information acquisition capabilities of the search engine, realizes cross-library association through similarity comparison, eliminates the need for manual maintenance of the song library, and reduces the system operation and maintenance costs. BRIEF DESCRIPTION OF THE DRAWINGS

[0026] Figure 1 It is a flow chart of a method for printing lyrics and rhythms based on audio fingerprints in an embodiment of the present application.

[0027] Figure 2 This is another flowchart of a method for printing lyrics and rhythms based on audio fingerprints in an embodiment of the present application.

[0028] Figure 3 It is a schematic diagram of the physical device structure of a lyrics and rhythm printing system based on audio fingerprint provided in an embodiment of the present application. DETAILED DESCRIPTION

[0029] The terms used in the following embodiments of the present application are only for the purpose of describing specific embodiments, and are not intended to be used as limitations to the present application. As used in the specification and appended claims of the present application, the singular expressions "one", "a kind of", "said", "above", "the" and "this" are intended to also include plural expressions, unless there is a clear indication to the contrary in the context. It should also be understood that the term "and / or" used in the present application refers to any or all possible combinations comprising one or more listed items.

[0030] In the following, the terms "first" and "second" are used for descriptive purposes only and are not to be understood as suggesting or implying relative importance or implicitly indicating the number of the indicated technical features. Thus, a feature defined as "first" or "second" may explicitly or implicitly include one or more of the features, and in the description of the embodiments of the present application, unless otherwise specified, "plurality" means two or more.

[0031] An application scenario of an embodiment of the present application is described below: In today's fast-paced life, music has become an indispensable part of people's daily lives. Whether it is during fitness, parties or being alone, music can bring wonderful enjoyment and relaxation. With the popularity of smartphones and music players, people can listen to a vast amount of music resources anytime and anywhere. However, simply playing music can no longer meet people's growing demand for music interaction. Accurate lyrics display and rhythm recognition functions have become standard features of music software, bringing users an immersive experience.

[0032] Taking karaoke applications as an example, real-time and accurate lyrics display is one of its core functions. When singing, users need to master the singing rhythm and emotional expression according to the rhythm and content of the lyrics. If the lyrics are displayed inaccurately or delayed, it will seriously affect the user's singing experience and effect. In addition, karaoke applications can also provide functions such as pitch scoring and pitch guidance, which all rely on the precise synchronization of lyrics. Therefore, karaoke applications urgently need an efficient and accurate lyrics recognition and synchronization technology to meet users' high-quality singing needs.

[0033] Taking music rhythm games as an example, players need to perform body movements according to the rhythm and melody of the music to interact with the music. The dance movements, special effects switching, and score calculations in the game all rely on the accurate recognition of the music rhythm. If the rhythm recognition is inaccurate or delayed, the player's movements will be inconsistent with the music, affecting the gaming experience. In addition, music rhythm games also need to generate different rhythm scores according to different difficulty levels, which requires the rhythm recognition algorithm to be able to comprehensively and deeply analyze the music content and extract key rhythm features. In order to solve the above technical problems, the present application provides a lyrics and rhythm printing method based on audio fingerprints, which is used to quickly and accurately identify and print lyrics and rhythms, and improve accuracy and efficiency.

[0034] Combine the following Figure 1 , a method for printing lyrics and rhythm based on audio fingerprint in an embodiment of the present application is described: See also Figure 1 , which is a flow chart of a method for printing lyrics and rhythms based on audio fingerprints in an embodiment of the present application.

[0035] S101, performing a preprocessing operation on the audio to be processed to obtain preprocessed audio data; The system performs a preprocessing operation on the audio to be processed to obtain preprocessed audio data. Specifically, the audio to be processed is filtered to obtain filtered audio. Perform noise reduction processing on the filtered audio to obtain a denoised audio frequency; Adjust the volume of the noise-removed audio to the preset volume to obtain the audio with the same volume; Perform spectrum splitting operation on audio with the same volume to obtain pre-processed audio data, including: Convert preprocessed audio data into spectral data; The spectrum data is segmented using a preset algorithm to obtain preprocessed audio data.

[0036] In this step, the system first performs a series of preprocessing operations on the audio to be processed to improve the accuracy of subsequent feature extraction and fingerprint generation. Preprocessing operations usually include filtering, noise reduction, volume normalization, etc.

[0037] Specifically, the system first filters the original audio to remove high-frequency noise and low-frequency interference in the audio and extract effective audio signal components. Common filtering methods include low-pass filtering, high-pass filtering, and band-pass filtering. For example, the system can use a low-pass filter with a cutoff frequency of 8kHz to filter out noise components above 8kHz in the audio.

[0038] Next, the system performs noise reduction on the filtered audio to further eliminate interference such as background noise and current noise, and improve the audio signal-to-noise ratio. Common noise reduction algorithms include spectral subtraction, wavelet transform, and adaptive filtering. For example, the system can use a noise reduction method based on wavelet packet decomposition to perform multi-scale and multi-resolution analysis on audio signals, effectively distinguish between noise and valid signals, and achieve adaptive noise reduction.

[0039] Then, the system adjusts the volume of the denoised audio to a preset value, eliminating the volume differences between different audios and providing a normalized input for subsequent processing. For example, the system can normalize the volume peaks of all audios to -3dB, which not only ensures the consistency of the volume but also avoids clipping distortion.

[0040] Finally, the system segments and spectrally converts the normalized audio to obtain preprocessed audio data. Specifically, the system first segments the audio into several frames of equal length, each frame corresponding to a short-time analysis window, such as 20ms. Then, a fast Fourier transform (FFT) is performed on each frame of audio to obtain the spectrum data corresponding to the frame. The spectrum data reflects the energy distribution of the audio at different frequencies and is the basis for further extracting audio features. In order to highlight the rhythm of music, the system can focus on the spectrum characteristics of the low-frequency band. For example, the system can extract the energy distribution of the 20Hz~200Hz frequency band in each frame of spectrum data as a feature representation of the rhythm.

[0041] S102, dividing the pre-processed audio data into several segments; The system divides the preprocessed audio data into several segments, each segment containing an equal amount of audio data.

[0042] In this step, the system divides the preprocessed audio data into several segments, each containing an equal amount of audio data. Audio segmentation is a common method of audio analysis and processing, which can convert complex audio signals into multiple short-term analysis units that are easy to process.

[0043] The main parameters of audio segmentation include segment length and segment overlap. The segment length determines the time span of each segment and is usually selected according to the rhythm of the music. For example, for a 4 / 4 pop song, 1 second (corresponding to 4 beats) can be selected as the segment length. Short segments are conducive to fine-grained audio analysis, while long segments can capture more global music structures. At the same time, in order to avoid information discontinuity between adjacent segments, segment overlap is usually adopted, that is, there is a certain time overlap between adjacent segments, and the overlapping duration can be 1 / 2 or 1 / 4 of the segment length.

[0044] S103, extracting key features of each segment respectively; In this step, the system extracts key features from each audio clip. Audio key features are data representations that reflect the essential attributes of audio content and are the basis for realizing audio fingerprints. By characterizing audio characteristics from different angles such as time domain, frequency domain, and time-frequency domain, key features can fully capture multi-dimensional information such as audio pitch, rhythm, and timbre.

[0045] Commonly used audio key features can be divided into low-level features and high-level features. Low-level features are directly extracted from audio signals, such as short-time energy and zero-crossing rate in the time domain, and Mel-frequency cepstral coefficients (MFCC) in the frequency domain. Taking MFCC as an example, it simulates the auditory characteristics of the human ear, maps the linear spectrum to the Mel spectrum, and then performs spectrum analysis to obtain a compact feature vector that reflects the timbre. For example, the system can extract 13-dimensional MFCC features for each audio clip, and then calculate the first-order difference and second-order difference to obtain a 39-dimensional MFCC feature vector.

[0046] High-level features perform semantic extraction based on low-level features to explore the deep structure of music content, such as pitch, chord, rhythm, etc. For example, the system can use a fundamental frequency extraction algorithm based on the harmonic noise model (HNM) to estimate the fundamental frequency trajectory from the spectrum of each segment, and then obtain the pitch feature through pitch category mapping. Rhythm features can extract the beat structure of music by calculating the beat pulse diagram of each segment.

[0047] In addition, the system can also combine technologies such as instrument recognition and sentiment analysis to further enrich the connotation of key features. For example, through spectrum envelope analysis and support vector machine classification, the main instrument in each segment can be identified and the instrument distribution characteristics can be generated. Based on the arousal-valence sentiment model, the emotional attributes of each segment can be analyzed to construct the musical sentiment characteristics.

[0048] The core of key feature extraction lies in the distinguishability and compactness of feature representation. Features with strong distinguishability can maximize the differences between different audio contents, which is beneficial for subsequent fingerprint matching. Compact features help reduce storage and computing overhead and improve retrieval efficiency. In order to balance the two, when designing key features, the system usually uses dimensionality reduction methods, such as principal component analysis (PCA), to remove information redundancy in the feature set and extract the most discriminative feature subset.

[0049] S104, converting each key feature into a corresponding audio fingerprint through a hash algorithm; In this step, the system uses a hashing algorithm to convert the key features of each audio clip into a compact audio fingerprint. An audio fingerprint is a digital summary of the audio content. It maps high-dimensional, continuous feature data to low-dimensional, discrete hash values ​​to achieve compact representation and fast matching of audio.

[0050] Common audio fingerprint hash algorithms can be divided into two categories: algorithms based on locality sensitive hashing (LSH) and algorithms based on fingerprint models. LSH maps the original feature space to the hash space through random projection, so that the hash values ​​of similar features have a higher probability of collision. For example, the system can use the p-stable distributed LSH algorithm to perform hash projection on the MFCC feature vector of each audio clip to obtain a set of binary hash values ​​as the audio fingerprint.

[0051] The fingerprint model builds a compact and discriminative fingerprint representation by learning the intrinsic distribution of audio features. Commonly used fingerprint models include the phoneme n-gram model and the GMM-UBM model. Taking the phoneme n-gram model as an example, the system first performs HMM-GMM phoneme recognition on each audio clip to obtain a phoneme sequence, and then extracts the n-gram phoneme combination as a fingerprint feature. Compared with LSH, the fingerprint model can better model the temporal structure and semantic information of the audio content.

[0052] S105, searching the audio fingerprint in the preset song library and determining whether a match is successful; In this step, the system searches the preset song fingerprint library for the fingerprint of the current audio clip to determine whether there is a matching song. The song fingerprint library is a pre-built song fingerprint data set that contains fingerprint information of a large number of known songs. By calculating the similarity between the current fingerprint and the fingerprint in the library and setting the similarity threshold, the system can quickly determine the song to which the current audio belongs.

[0053] The construction of fingerprint database usually adopts distributed computing frameworks such as MapReduce to extract fingerprint features from massive songs in parallel, organize indexes according to keywords such as song ID and timestamp, and store them in distributed database or NoSQL database. For example, the system can use MongoDB to build a shard cluster to store fingerprint database of 400 million songs, with an average of 1MB of fingerprint data per song, occupying a total storage space of 400TB.

[0054] In the actual retrieval process, the system uses an inverted index to speed up the query. The inverted index uses a single fingerprint hash value as the keyword (Key), and the song ID and timestamp list containing the fingerprint as the value (Value) to build a Key-Value index structure. The system first extracts the hash value of the current audio fingerprint, queries the index for the list of <song ID, timestamp> where the hash value appears, and then counts the number of timestamps for each song ID. Songs that exceed the threshold are considered to be matched successfully. To further improve retrieval efficiency and accuracy, the system usually adopts a multi-stage matching strategy. In the first stage, the global fingerprint is used for coarse screening to quickly lock the candidate song subset; in the second stage, the local fingerprint is used for fine screening to accurately locate the matching fragments from the candidate songs. The multi-stage strategy can effectively balance the recall rate and precision, significantly saving retrieval time and computing resources.

[0055] At the same time, when matching fingerprints, the system also needs to consider the robustness of the fingerprint, that is, the ability to resist noise and distortion. In reality, audio recording is often accompanied by interference such as environmental noise and transmission distortion, which leads to certain differences between the extracted fingerprint and the original fingerprint. In order to improve the robustness of matching, the system usually introduces fault-tolerant mechanisms, such as allowing a certain number of bit errors and supporting missing fragment matching. For example, the system can use the Hamming distance to measure the similarity of fingerprints. If the Hamming distance of two fingerprints is less than 5, the match is considered successful.

[0056] Let's take a specific example. Assume that the local fingerprint hash value extracted from the current audio clip is "1101001", the system queries the hash value in the index and obtains the following <song ID, timestamp> list: <S1, 0:35> ,<S1, 0:37> ,<S1, 0:39> ,<S2, 1:12> ,<S2, 1:15> ,<S3, 2:01> Count the timestamps of each song ID and get: S1: 3 timestamps, S2: 2 timestamps, S3: 1 timestamp If the matching threshold is set to 3, the number of timestamps of song S1 reaches the threshold, and the current segment is considered to be successfully matched to song S1. Further analysis of the timestamp difference shows that the current segment is located between 0:35 and 0:39 of song S1. Combined with global fingerprint verification, if the match is consistent, it is finally determined that the current audio segment comes from song S1.

[0057] S106, generating first primary printing information; If the match is successful, the system generates first primary print information, which includes lyrics information and rhythm information arranged in a preset format, and the lyrics information and rhythm information are the lyrics information and rhythm information corresponding to the matched audio fingerprint.

[0058] In this step, if the fingerprint match is successful, the system generates the first primary printing information, which contains the lyrics information and rhythm information of the matching song. This is the first step in the lyrics and rhythm printing process, which aims to extract the lyrics text and beat data corresponding to the matching audio clip, in preparation for subsequent visual printing.

[0059] In specific implementation, the system first locates the song ID and timestamp based on the fingerprint matching results, and then accesses the pre-built song metadata database to obtain the entire lyric text and timeline information of the song. By aligning the timestamp of the matching segment with the timeline of the lyrics, the system can accurately extract the lyric clause corresponding to the current segment.

[0060] While extracting lyrics, the system also needs to analyze the rhythm structure of the music and generate rhythm information synchronized with the lyrics. Usually, beat detection algorithms, such as BPM estimation based on autocorrelation functions, are used to extract the basic beat units of music from the audio signal. Based on the beat information, higher-level rhythm patterns, such as beats and measures, are extracted. By using the mapping relationship between the rhythm pattern and the lyrics timeline, the system can generate synchronized rhythm information for each word in the lyrics.

[0061] S107, rendering the first primary printing information according to a preset style to obtain a final printed image; In this step, the system typeset and renders the first primary printing information according to the preset style template, and finally generates a printable lyrics rhythm picture. This is the key link in the visualization of lyrics rhythm, and its core lies in designing a beautiful, eye-catching, rhythmic visual layout to fully display the lyrics content and music rhythm.

[0062] The style rendering of the lyrics rhythm needs to consider many factors, such as fonts, colors, arrangement methods, dynamic effects, etc. The system usually presets multiple sets of style templates, each of which defines a set of configuration parameters for visual elements. Users can choose the appropriate template according to the song style, personal preferences, etc., or achieve personalized customization by adjusting parameters. The design of style templates usually adopts Web front-end technologies, such as CSS, JavaScript, etc., which is convenient for flexible control and dynamic update.

[0063] During the rendering process, the system reads the lyrics text, timeline, beat information, etc. in the primary print information, arranges and draws them sentence by sentence and word by word according to the guidance of the style template, and inserts beat animation to generate lyrics images frame by frame. Combined with algorithms such as smooth transition and keyframe interpolation, the system can generate interpolation animation between lyrics image sequences, so that the lyrics move smoothly between different beat points, forming a coherent rhythmic effect.

[0064] For example, for the aforementioned primary print information, the system first arranges the words "Let's rock together" in order, sets the font, color, line spacing, etc., and generates a static lyrics image. Then, at each beat point on the timeline, the system repeatedly generates a scaled version of the current lyrics, superimposed on the corresponding position of the original image, forming a zoomed animation sequence. Using the interpolation algorithm, the system renders the zooming process of the lyrics between two adjacent beat points to produce a smooth zooming animation. Multiple animation sequences are connected and combined to finally synthesize a complete lyrics rhythm video.

[0065] The resolution and frame rate of the rendered output can be adjusted according to the specific application scenario and device performance. In order to balance clarity and transmission efficiency, the system generally uses high-compression video encoding formats such as H.264 to minimize storage overhead while ensuring image quality. At the same time, the system can also use technical means such as GPU acceleration and parallel rendering to improve the speed and real-time performance of visual rendering.

[0066] S108, sending the final printed image to a printing device so that the printing device prints the final printed image; In this step, the system sends the rendered lyrics rhythm image to the printing device to perform the actual print output. This is the last step in the lyrics rhythm printing process, which aims to convert the digital visual content into physical print media for users to appreciate and collect intuitively.

[0067] To achieve seamless printing, the system needs to establish a reliable communication connection with the printing device. Usually, wireless network transmission protocols such as Wi-Fi and Bluetooth are used to communicate with the printer within the local area network or personal area network. For networked printers, the system can also send print tasks remotely through the Internet. Transmission protocols generally follow industry standard specifications, such as AirPrint, Mopria, etc., to ensure interoperability and compatibility between devices.

[0068] In terms of transmission content, the system intelligently selects the optimal data format based on the type and capabilities of the printing device. For traditional two-dimensional printers, the system transcodes the lyrics rhythm video frame by frame into static images, such as JPEG, PNG, etc., and transmits them in the form of continuous images; for new three-dimensional printers, the system can directly send parameterized data such as lyrics text and rhythm information to the printer, and the printer's internal algorithm will render the three-dimensional lyrics model in real time. Optimization methods such as data compression and packet transmission can effectively reduce transmission delays and increase printing speed.

[0069] S109, searching for key features in a preset search engine to obtain a number of search results; If the match fails, the key features are searched in the preset search engine to obtain several search results.

[0070] When the audio fingerprint fails to match in the preset song library, it means that the audio clip has not been included. In order to provide a more comprehensive lyrics rhythm printing service, the system will start the content-based music retrieval mechanism and try to find similar songs from a large number of online music resources.

[0071] Specifically, the system uses preset search engines, such as Google and Baidu, to retrieve the key features of each audio clip. The key features here can be data that can characterize the audio content, such as pitch sequences and timbre parameters. The system converts these features into query statement formats supported by search engines, such as keywords, vectors, etc., and then submits a search request.

[0072] Based on the query, the search engine uses text matching, multimedia indexing and other technologies to conduct a comprehensive scan in its music database to find a batch of songs that are most similar to the target audio in content and return them as candidate search results. These search results usually contain metadata information such as the song title, singer, album, audio link, lyrics, etc.

[0073] After obtaining the preliminary search results, the system needs to further analyze and screen them to improve the matching accuracy. On the one hand, the system semantically understands the text information of the search results, extracts key words in the song titles and lyrics, and determines their relevance to the original audio; on the other hand, the system compares the acoustic features of the search results with the differences in the original audio, and eliminates invalid results with too large differences in sound quality and timbre.

[0074] After post-processing optimization, the system finally obtains several high-confidence search results, preparing for the subsequent similarity calculation. The generation method based on content retrieval has greatly expanded the adaptation scope of lyrics rhythm printing, making it no longer limited to a limited preset music library, but can provide real-time rhythm support for a large number of new songs.

[0075] S110, extracting search result feature information of each search result respectively; After obtaining a batch of search results for related songs, the system needs to extract effective feature information from them for subsequent similarity calculation and matching. This is a process of deeply mining the music content and building a compact song representation.

[0076] The first task of extracting search result features is to obtain the audio data of candidate songs. The system parses the audio links in the search results and downloads the audio clips or complete tracks of each song. In order to save transmission time and improve retrieval efficiency, a reasonable data length and sampling rate can be set when downloading.

[0077] After obtaining the audio data, the system performs comprehensive acoustic feature extraction. On the one hand, it extracts low-order features that reflect basic musical attributes such as pitch, rhythm, and timbre, such as fundamental frequency, spectral entropy, and MFCC; on the other hand, it extracts high-order features that reflect musical structures such as melody direction and chord progression, such as Chroma vectors and pitch probability distribution. These two types of features respectively depict the local details and overall outline of the music clip, and are important bases for judging the similarity of songs.

[0078] In the feature extraction process, the system makes full use of various algorithms in the field of music signal processing, such as Fourier transform, wavelet analysis, Kalman filtering, etc., to deeply analyze the audio data from the perspective of time and frequency domain. At the same time, the system also draws on the feature representation methods of speech recognition, speaker recognition and other technologies to further enhance the expressiveness and differentiation of the features.

[0079] The massive amount of feature data extracted is compressed into a compact feature vector by removing noise and redundant components through data dimensionality reduction methods such as principal component analysis (PCA) and independent component analysis (ICA). A song is composed of several such feature vectors, each of which corresponds to a segment in the song. To facilitate subsequent processing, the system aggregates the feature vectors of all candidate songs into a feature matrix, with rows representing different songs and columns representing different segments.

[0080] Finally, the system needs to further enrich the semantic representation of features based on the text information of the search results, such as lyrics, tags, etc. Using natural language processing technologies such as word segmentation and named entity recognition, the system extracts key words and topic tags in the lyrics and maps them to predefined semantic concepts. Each semantic concept corresponds to a feature dimension, and its feature value reflects the importance of the concept in the song. By fusing these semantic features with acoustic features, a complete feature representation of a song is obtained.

[0081] S111, respectively calculating the similarity between the key features and the search result feature information to obtain a number of song similarities; After obtaining the key features of the original audio clip and the search results features of the candidate songs, the system needs to measure the similarity between the two and find the song that best matches the original audio. This is a feature matching process of finding the nearest neighbor. After that, the system converts the search result feature information in the search result corresponding to the greatest song similarity into the corresponding added audio fingerprint through a hash algorithm; Add the audio fingerprint to the preset song library.

[0082] The core of calculating song similarity is to design a reasonable feature similarity measurement function. The system uses different measurement methods for different types of features. For numerical feature vectors, classic metrics such as Euclidean distance and cosine similarity can be used. Euclidean distance measures the straight-line distance between two feature vectors in a high-dimensional space. The smaller the distance, the higher the similarity; cosine similarity calculates the cosine value of the angle between two vectors. The smaller the angle, the higher the similarity.

[0083] For concept semantic features, semantic similarity calculation based on knowledge base can be used. For example, using concept dictionaries such as WordNet, the shortest path length between two semantic labels in the concept hierarchy, or the amount of information of the common parent concept, can be calculated as a measure of semantic similarity. The shorter the path, or the greater the amount of information, the closer the semantic association between concepts.

[0084] For feature similarity calculation based on matching, the system searches for the nearest neighbors of the key features of the original audio in a given feature space. Common nearest neighbor search algorithms include KD tree, locality sensitive hashing (LSH), etc. They greatly reduce the number of comparisons in similar feature searches by building an efficient index structure, making the matching process among massive candidates lightweight and efficient.

[0085] When multiple types of features coexist, the system performs weighted fusion of the similarity scores of different features to form a comprehensive song similarity metric. The setting of weights needs to take into account factors such as the distinctiveness, stability, and interpretability of each feature, and is continuously optimized through data verification and cross-evaluation. The adaptive learning of feature weights is a machine learning process for similarity quantification.

[0086] It is worth mentioning that the differences in the duration and rhythm of the original audio and the song in terms of dynamic time will bring challenges to similarity calculation. To this end, the system introduces sequence matching technologies such as dynamic time warping (DTW). It uses dynamic programming to find an optimal nonlinear mapping between two time series, making time series of different lengths and rates comparable. This greatly improves the matching accuracy of cross-time domain audio.

[0087] In terms of computing efficiency, the system uses a variety of parallel acceleration methods, such as multi-threading and GPU, to fully utilize the concurrency advantages of modern computing architectures. The calculation of feature similarity is decomposed into multiple independent subtasks and distributed to different processing units, realizing distributed computing of similarity search. At the same time, for applications with high real-time requirements, the system also provides approximate nearest neighbor search based on local sensitive hashing, which greatly improves the retrieval speed at the expense of a small amount of accuracy.

[0088] Through the above method, the system finally obtains a similarity list containing several matching songs. This list is sorted by similarity score and records the degree of match between each candidate song and the original audio. The top few highly similar songs are taken as matching results, providing data support for the generation of lyrics rhythm. At the same time, the discovery of similar songs also provides a basis for applications such as music recommendation and playlist generation.

[0089] S112, determining the lyrics information and rhythm information corresponding to the song with the greatest similarity; After multiple rounds of similarity comparisons with candidate songs, the system finally locked in the target song that best matched the key features of the original audio. The similarity score of this song topped all search results, marking the successful completion of the retrieval process. Next, the system will extract lyrics and rhythm information from this song for the generation of lyrics and rhythm print content.

[0090] The process of extracting lyrics information is essentially a process of lyric-audio alignment. Modern lyrics data is usually stored in LRC format, which contains the starting timestamp of each line of lyrics. By parsing the LRC file, the system can obtain the lyrics text that should appear at different time points. The audio data of the target song is divided according to the time axis, and then the lyrics with timestamps falling within each audio segment interval are extracted, thus completing the coarse-grained lyrics-audio alignment.

[0091] On this basis, the system needs to further accurately locate the lyrics to each syllable of the audio. This requires the use of speech recognition and acoustic model analysis technology, through short-term analysis of audio energy, fundamental frequency, etc., to find the pronunciation duration and start and end boundaries of each word in the lyrics. By combining the syllable-level time tags with the lyrics text, the system can obtain an accurate syllable-level lyrics timeline. This fine-grained lyrics positioning is the key to accurate synchronization of lyrics rhythm.

[0092] While determining the lyrics, the system also needs to analyze and extract the rhythm information of the song. The rhythm of music comes from its inherent rhythmic structure, such as beats and measures. The system determines the basic rhythmic unit of the song through beat detection algorithms, such as beat point analysis based on autocorrelation functions. At the measure level, the measure structure of the song can be divided by identifying the periodic changes in music data, such as the periodic peaks of the spectrum. The extraction of rhythmic elements such as beats and measures is an important reference for the visual presentation of the rhythm of lyrics.

[0093] It is worth noting that neat lyrics arrangement and regular rhythm organization do not always cover all songs. Some music forms such as improvisation and rap have relatively free lyrics and rhythms, which are difficult to describe with general music rules. In this regard, the system introduces a statistical learning-based method in rhythm analysis, which automatically discovers the inherent rhythm patterns from a large amount of irregular song data through clustering, Markov models, etc. This allows the lyrics rhythm technology to adapt to a wider range of music styles.

[0094] S113, generating second primary printing information; The system generates second primary printing information, which includes lyrics information and rhythm information arranged in a preset format.

[0095] After obtaining the lyrics and rhythm information of the target song, the system needs to convert it into a standardized data format suitable for print output, namely the second primary print information. This is a key step in the visual presentation of music information, and its core is the optimization of the arrangement of lyrics text and rhythm timing.

[0096] First, the system automatically divides the lyrics into lines and sentences according to the preset typesetting rules. While ensuring the semantic integrity, it also takes into account visual aesthetic factors such as the uniform number of words in each line and left-right alignment. Special symbols in the lyrics, such as English brackets and instrument names, are also identified and formatted to highlight the main body of the lyrics.

[0097] Secondly, the system marks the pronunciation duration of each word in the lyrics to form a syllable-level timestamp sequence. A syllable corresponds to a word or several consecutive words in the lyrics, and its duration is determined by the duration of the pronunciation. The system uses this to build a syllable duration table to record the start time and duration of each syllable. This is the time reference for the rhythm synchronization control of the lyrics.

[0098] On this basis, the system refers to the rhythm information of the song, such as beats and measures, and makes rhythmic adjustments to the syllable duration table. Specifically, the stressed syllables of the lyrics are aligned with the strong beats of the beat to achieve a visual enhancement of the sense of rhythm; reasonable pauses are set between measures to highlight the beginning, development, transition and ending of the phrases. After rhythmic processing, the rhythm of the lyrics can better reflect the musicality of the song.

[0099] Then, the system applies special rhythmic effects to some key lyrics based on the semantic information of the lyrics. For example, color change and amplification are used to highlight emotions in the climax; repetition and flashing are used to enhance recognition in the chorus. In addition, different fonts and layouts can be selected according to the theme of the lyrics to create an atmosphere. Semantic-driven rhythmic design is an important means to create a sense of rhythmic art in lyrics.

[0100] Finally, the system integrates information such as lyrics, syllable duration, rhythm, and rhythmic effects to generate a complete primary printing information data package. When storing, it uses common data exchange formats such as JSON and XML, and organizes data according to standardized field names and hierarchical structures to ensure the standardization of the parsing and rendering process.

[0101] S114: Rendering the second primary printing information according to a preset style to obtain a final printed image.

[0102] The system renders the second primary printing information according to a preset style to obtain a final printed image. The specific implementation of this step will be described below and will not be elaborated here.

[0103] Finally, step S108 is executed to send the final printed image to the printing device so that the printing device prints the final printed image.

[0104] The above embodiments have the following beneficial effects: A series of steps are performed to pre-process the audio to be processed, segment it, extract key features, generate audio fingerprints, match the preset song library, generate printing information, render print pictures and finally print it, realizing a fully automated lyrics and rhythm printing method based on audio fingerprint technology. The shortcomings of traditional manual printing of lyrics, such as low efficiency and easy errors, are overcome. By introducing audio fingerprint technology, the song corresponding to the audio can be quickly and accurately identified, and print pictures containing lyrics and rhythm information can be automatically generated, which greatly improves the efficiency and accuracy of lyrics and rhythm printing. At the same time, the invention adopts a standardized audio fingerprint algorithm, which converts the key features of the audio into unique fingerprint information through hash operations, which can effectively avoid copyright disputes. In addition, the lyrics and rhythm information are typeset and rendered in a preset format and style to ensure the readability of the printout. It is used to quickly and accurately identify and print lyrics and rhythms, improving accuracy and efficiency.

[0105] Filtering the original audio can remove low-frequency and high-frequency noise in the audio and improve the signal-to-noise ratio; then further noise reduction processing is performed on the filtered audio to eliminate the influence of residual noise; then the volume is adjusted to the preset value to unify the volume differences of different sound sources; finally, the spectrum is split on the audio with the same volume to obtain preprocessed audio data that is easy to extract key features. This series of preprocessing operations can retain the effective information of the audio to the greatest extent, remove various noise interferences, and lay a good foundation for subsequent audio fingerprint extraction. At the same time, the preprocessing process is implemented using a standard algorithm, with a clear and definite process and strong versatility. It can adapt to different sound sources and reduce the difficulty of algorithm implementation. In addition, the preprocessed audio data is small in scale and simple in structure, which can speed up the subsequent fingerprint extraction and comparison.

[0106] First convert it into spectrum data, then use the preset algorithm to segment it. Converting the time domain audio signal into a frequency domain signal through mathematical methods such as Fourier transform can reveal the frequency composition of the audio, which is conducive to capturing key information such as pitch and timbre that reflect the characteristics of the song. This provides a more essential and stable data basis for the extraction and comparison of audio fingerprints. The segmentation processing based on spectrum data avoids the randomness and uncertainty of the time domain waveform. The use of preset algorithms can simplify the amount of calculation while ensuring the segmentation accuracy, making it easy to implement in engineering. In addition, the segmented spectrum data presents a regular time-frequency distribution, which makes it easier to extract key features that reflect the essential attributes of the song.

[0107] When the audio fingerprint fails to match in the preset song library, the related song results can be obtained by retrieving key features in the search engine. By extracting features from these results and calculating the similarity with the key features of the audio to be processed, the closest song can be found. Using the lyrics and rhythm information of the best matching song, a suboptimal printout can be generated as a remedy when the song cannot be directly identified. This approximate matching strategy based on similarity comparison can effectively expand the algorithm's adaptation range, so that it is not limited to the preset song library, and enhances robustness. At the same time, both retrieval and similarity calculation can be implemented using mature information retrieval technology, which will not significantly increase the complexity of the system. Providing an approximate printout for difficult-to-identify songs is better than being unable to process them at all, and can reduce errors and omissions caused by recognition failures.

[0108] When the audio fingerprint match fails, the system retrieves similar songs through the search engine and calculates the similarity between the key features and the search results. For the results with the highest similarity, its feature information is extracted, converted into a new audio fingerprint through a hash algorithm, and added to the preset song library. This mechanism of dynamically expanding the fingerprint library according to the actual recognition situation can continuously accumulate new song fingerprints, making the algorithm smarter and smarter. With the enrichment of recognition history, the song library will become more and more comprehensive, and its adaptability to unfamiliar audio will become stronger and stronger. At the same time, the new fingerprints are stored in hash to avoid repeated additions and maintain efficient retrieval of the song library. In addition, this strategy makes full use of the powerful information acquisition capabilities of the search engine, realizes cross-library association through similarity comparison, eliminates the need to manually maintain the song library, and reduces the system operation and maintenance costs.

[0109] In the following, another embodiment is combined to describe the step of rendering the first primary printing information according to a preset style to obtain a final printed image in the above embodiment. Figure 2 , another method for printing lyrics and rhythm based on audio fingerprint in an embodiment of the present application is described: See also Figure 2 , is another flow chart of a lyrics and rhythm printing method based on audio fingerprint in an embodiment of the present application.

[0110] S201, rendering lyrics information into a pre-printed image according to a preset style to obtain a secondary printed image; Preset styles include font, font size and font color.

[0111] This step is to map the extracted lyrics text onto a background image according to a predefined visual style to form a preliminary version of the lyrics printing. The style includes attributes such as font, size, color, etc., which determines the aesthetics of the lyrics on the image.

[0112] Specifically, the system first loads a preset background image as the canvas for printing lyrics. This image can be a solid color background or a designed artistic pattern, which has an important impact on the rendering effect. Suppose a light blue background is selected, the color value is #e0ffff, and the resolution is 1920*1080.

[0113] Then, the system reads the lyrics text and splits it into each line to get a lyrics line array. For example, part of the lyrics of the song "Night Piano Piece 5": "Let me become a song / Replace all languages / Let me become a lamp / Take you to find the direction" is split into 4 lines.

[0114] Next, the system traverses each line of the lyrics array and renders it onto the background image in a preset style. Take the first line of "Let Me Become a Song" as an example, assuming that the specified style is: font "Kaiti", font size 36px, color #8b0000. The system first creates a transparent canvas of the same size as the background image, and then draws this line of lyrics text with the specified font at (100, 200) as the starting coordinate on the canvas, and sets the opacity to 0.8. When drawing, it is necessary to consider the centering, line breaks, spaces and other layout factors of the lyrics to ensure visual beauty.

[0115] After the drawing is completed, the system will synthesize the lyrics canvas with the background image, and use the alpha blending mode to make the lyrics naturally appear on the background. To highlight the sense of layering, special effects such as projection and glow can be added between the lyrics and the background. The system follows this process to render each line of lyrics line by line until all the lyrics lines are processed.

[0116] S202, determining visual elements of rhythmic information; After completing the static rendering of the lyrics, the system needs to further determine the visual elements that match the rhythm of the music. They are like jumping notes, giving the lyrics a dynamic beauty. Through clever design, the visual expression of the music rhythm can be achieved.

[0117] The selection of visual elements needs to be combined with the style and emotion of the song. Taking the above-mentioned "Night Piano Piece 5" as an example, it has a soothing melody and a tranquil mood, which is suitable for elegant and subtle rhythmic presentation. The system can start with lines, graphics, textures and other aspects for design.

[0118] First, the line element. The system uses a smooth and elastic curve to complement the soft melody of the piano music. Here, for example, a NURBS spline curve is used, and the control points are [(0, 0), (200, 200), (400, 0), (800, 250)]. The curve thickness is set to 2px, the color is #ffa07a that matches the lyrics, and the endpoint style is set to rounded corners.

[0119] Next is the graphic element. The system uses an ellipse as the basic graphic, with a circumscribed rectangle size of (100, 50) and a corner radius of 20. At the same time, a built-in gradient fill is used, with a center color of #ff6eb4, an edge color of #ffc0cb, and a radiation radius of 80%. This set of parameters creates a soft, dreamy visual image, setting off the warm and romantic tone of the song.

[0120] As for the texture element, the system uses a soft cloud pattern, scales it to 1 / 4 of the lyrics area, rotates it 25 degrees, and reduces its transparency to 40%. The texture element and the lyrics complement each other, creating an ethereal and elegant mood, reminiscent of the lyrics image of "Let me turn into a lamp to take you to find the direction".

[0121] S203, rotating the visual element into the secondary print image to obtain a standby print image; After determining the rhythmic visual elements, the system needs to merge them with the generated lyrics images to finally form a printable image containing dynamic information. The synthesis of visual elements and animation drive are the key.

[0122] First, the system needs to embed the visual elements into the appropriate position of the lyrics image. For the lines, graphics, and textures in the above example, the rendering area can be selected between the lyrics, around the lyrics, and at the bottom of the lyrics. Taking the line element as an example, the system can render a streamline 250 pixels above each line of lyrics. From left to right, the line passes through the anchor point of each word in the lyrics in turn, connecting the lyrics into an organic whole.

[0123] Then, the system controls the animation presentation of visual elements based on the music rhythm data. The rhythm data contains information such as the duration and pitch of the notes, which determines the dynamic changes of the visual elements. Taking the piano accompaniment of a song as an example, its MIDI data is parsed into a set of event sequences like this: [(0.5s, NoteOn, C4, 100), (0.8s, NoteOff, C4), (1.3s, NoteOn, E4, 120)...].

[0124] Each event consists of elements such as the time of occurrence, type, note, and intensity.

[0125] The system maps these note events to the animation parameters of visual elements. For example, the intensity of the note corresponds to the thickness of the line, and the pitch corresponds to the amplitude of the line. The process of a note from pressing (NoteOn) to releasing (NoteOff) corresponds to a periodic change of the line from thin to thick and then to thin again. The peak value corresponding to the high note is higher than that of the low note, and the transition between two adjacent notes is achieved through parameter interpolation.

[0126] For example, at 1.3s, an E4 note with a velocity of 120 is pressed. The system first locates the animation frame at 1.3s and sets its serial number to 33. Then, from frame 33 to frame 58 (assuming the note length is 0.5s), the thickness of the line is gradually increased from 1px to 3px (the thickness of the velocity mapping of 120), and the amplitude is increased from 50px to 80px (the amplitude of the E4 pitch mapping). The line color gradually changes from the note color of 1.1s #ffa07a to #ff6eb4, and then continues until the next note is triggered. In this way, the rhythm of the piano sound is given a perceptible form one by one.

[0127] For visual elements with periodic motion, such as ellipses, the focus of animation driving is on phase control. The system sets the first beat of each measure to 0°, 90°, 180°, and 270°, representing the starting phase of the elliptical motion. The greater the note intensity, the faster the angular velocity of the corresponding elliptical motion. When each note is released, the ellipse moves to a new phase, and then slowly rebounds, creating a visual effect of the fading of the stepping sound.

[0128] From static to dynamic, from plane to three-dimensional, from regular to rhythmic, the system uses a series of algorithms such as graphics drawing, parameter mapping, animation interpolation, etc., and finally interweaves lyrics, notes, and visual primitives into a series of musical paintings that come alive on paper. Taking the above song as an example, the dynamic lyrics seem to dance in the colorful rhythmic elements, exuding an elegant, ethereal, and romantic temperament, leading people into a dreamlike musical realm.

[0129] S204: Add the time to the prepared print image to obtain a final print image.

[0130] The system adds the time to the prepared print picture to obtain the final print picture. Specifically: determine the type of time to be added, which includes the current play time, the total song time and the remaining time; Format the time type to get a unified time type; Use the image processing library to render the unified time type into the print-ready image to obtain the final print image.

[0131] On the basis of integrating lyrics and rhythm, the system also needs to add time information synchronized with the playback progress in the picture to provide users with an intuitive playback reference. This is an important part of creating an immersive music experience.

[0132] Time information includes the current playing time, the total song time, the remaining time, etc. They reflect the progress of music playing from different perspectives. Taking the current playing time as an example, it indicates the time point of the song in the format of "minutes: seconds", such as "03:25" means the song is played to the 3rd minute and 25 seconds.

[0133] The system first determines which types of time information to display in the printable image. In consideration of the simplicity of the image and user needs, this example selects the current playing time and the total song time as the default display items. Other types, such as the remaining time, can be turned on by the user as optional options.

[0134] Next, the system needs to unify the formats of various types of time. For example, the total duration and remaining time are stored in milliseconds internally, but they need to be converted to "minutes: seconds" when displayed. At the same time, "0" should be added to the single-digit seconds, such as "3:05" instead of "3:5". In addition, for songs longer than 1 hour, it should be expanded to "hours: minutes: seconds". For example, 4230000ms is formatted as "01:10:30".

[0135] The above embodiments have the following beneficial effects: The preset style includes font, size, color and other attributes to ensure that the lyrics are neatly laid out, with clear layers and easy for users to read. Then, based on the preset style, a secondary print image containing the lyrics information is generated as a master for subsequent visual processing. Next, unique visual elements are designed for the rhythmic information, such as musical notes, dance movements and other patterns to enhance the visual impact. These visual elements are then integrated with the secondary print image to obtain a pre-print image with both content and form beauty. Finally, time information is added to the pre-print image to form the final printed product.

[0136] The system can decide which time elements to display in the printed image based on the actual application scenario. The playing time can remind the user of the current progress, the total time can show the scale of the song, and the remaining time is convenient for controlling the listening schedule. This customizable time display scheme takes into account both comprehensive information and practicality. Then, to ensure the consistency of different time formats, the system standardizes the selected time types and converts them into minutes: seconds, eliminating format differences. Finally, using a mature image processing library, the print image is used as a canvas, and the formatted time information is rendered to the appropriate position of the image, perfectly integrated with other elements, and the final printed image is generated. The rendering process is highly automated, does not require human intervention, and the time information is clear and beautiful.

[0137] The following describes the system in the embodiment of the present invention from the perspective of hardware processing. Figure 3 , which is a schematic diagram of the physical device structure of a lyrics and rhythm printing system based on audio fingerprint provided in an embodiment of the present application.

[0138] It should be noted that Figure 3 The structure of the system shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present invention.

[0139] like Figure 3As shown, the system includes a central processing unit (CPU) 301, which can perform various appropriate actions and processes according to the program stored in the read-only memory (ROM) 302 or the program loaded from the storage part 308 to the random access memory (RAM) 303, such as executing the method in the above embodiment. In RAM 303, various programs and data required for system operation are also stored. CPU 301, ROM 302 and RAM 303 are connected to each other through a bus 304. Input / output (I / O) interface 305 is also connected to bus 304.

[0140] The following components are connected to the I / O interface 305: an input section 306 including a camera, an infrared sensor, etc.; an output section 307 including a liquid crystal display (LCD) and a speaker, etc.; a storage section 308 including a hard disk, etc.; and a communication section 309 including a network interface card such as a LAN (Local Area Network) card, a modem, etc. The communication section 309 performs communication processing via a network such as the Internet. A drive 310 is also connected to the I / O interface 305 as needed. A removable medium 311, such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, etc., is installed on the drive 310 as needed so that a computer program read therefrom is installed into the storage section 308 as needed.

[0141] In particular, according to an embodiment of the present invention, the process described above with reference to the flowchart can be implemented as a computer software program. For example, an embodiment of the present invention includes a computer program product, which includes a computer program carried on a computer-readable medium, and the computer program includes a computer program for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from a network through the communication part 309, and / or installed from a removable medium 311. When the computer program is executed by the central processing unit (CPU) 301, various functions defined in the present invention are performed.

[0142] It should be noted that the computer-readable medium shown in the embodiment of the present invention may be a computer-readable signal medium or a computer-readable storage medium or any combination of the above two. The computer-readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM), a flash memory, an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the present invention, a computer-readable storage medium may be any tangible medium containing or storing a program, which may be used by or in combination with an instruction execution system, device or device. In the present invention, a computer-readable signal medium may include a data signal propagated in a baseband or as part of a carrier wave, which carries a computer-readable computer program. Such a propagated data signal may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the foregoing.

[0143] The flowcharts and block diagrams in the accompanying drawings illustrate the possible architecture, functions and operations of the systems, methods and computer program products according to various embodiments of the present invention. Among them, each box in the flowchart or block diagram can represent a module, a program segment, or a part of the code, and the above-mentioned module, program segment, or a part of the code contains one or more executable instructions for implementing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a different order from the order marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram or flowchart, and the combination of boxes in the block diagram or flowchart, can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0144] As another aspect, the present invention further provides a computer-readable storage medium, which may be included in the system described in the above embodiment; or may exist independently without being assembled into the system. The above storage medium carries one or more computer programs, and when the above one or more computer programs are executed by a processor of a system, the system implements the method provided in the above embodiment.

[0145] As described above, the above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit them. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present application.

[0146] As used in the above embodiments, the term "when..." may be interpreted to mean "if..." or "after..." or "in response to determining..." or "in response to detecting...", depending on the context. Similarly, the phrases "upon determining..." or "if (the stated condition or event) is detected" may be interpreted to mean "if determining..." or "in response to determining..." or "upon detecting (the stated condition or event)" or "in response to detecting (the stated condition or event)", depending on the context.

[0147] In the above embodiments, it can be implemented in whole or in part by software, hardware, firmware or any combination thereof. When implemented using software, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the process or function described in the embodiment of the present application is generated in whole or in part. The computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions may be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions may be transmitted from a website site, computer, server or data center to another website site, computer, server or data center by wired (e.g., coaxial cable, optical fiber, digital subscriber line) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium may be any available medium that a computer can access or a data storage device such as a server or data center that includes one or more available media integrated. The available medium may be a magnetic medium (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a DVD), or a semiconductor medium (e.g., a solid-state hard disk), etc.

[0148] Those skilled in the art can understand that to implement all or part of the processes in the above-mentioned embodiments, the processes can be completed by computer programs to instruct related hardware, and the programs can be stored in computer-readable storage media. When the programs are executed, they can include the processes of the above-mentioned method embodiments. The aforementioned storage media include: ROM or random access memory RAM, magnetic disk or optical disk and other media that can store program codes.

Claims

1. A method for printing lyrics and rhythm based on audio fingerprint, characterized in that: include: Perform a preprocessing operation on the audio to be processed to obtain preprocessed audio data; Dividing the preprocessed audio data into a plurality of segments, each segment comprising an equal amount of audio data; Extracting key features of each of the fragments respectively; Convert each of the key features into a corresponding audio fingerprint through a hash algorithm; Searching for the audio fingerprint in a preset song library and determining whether a match is successful; If the match is successful, generating first primary print information, wherein the first primary print information includes lyrics information and rhythm information arranged in a preset format, wherein the lyrics information and rhythm information are the lyrics information and rhythm information corresponding to the matched audio fingerprint; Rendering the first primary printing information according to a preset style to obtain a final printed image; The final printed image is sent to a printing device so that the printing device prints the final printed image.

2. The method according to claim 1, characterized in that The preprocessing operation of the audio to be processed to obtain preprocessed audio data specifically includes: Filtering the audio to be processed to obtain filtered audio; Performing noise reduction processing on the filtered audio to obtain a denoised audio frequency; Adjusting the volume of the noise-removed audio to a preset volume to obtain audio of the same volume; A spectrum splitting operation is performed on the audio with the same volume to obtain the preprocessed audio data.

3. The method according to claim 2, characterized in that The performing a spectrum splitting operation on the audio with the same volume to obtain the pre-processed audio data specifically includes: Converting the preprocessed audio data into frequency spectrum data; The frequency spectrum data is segmented using a preset algorithm to obtain the preprocessed audio data.

4. The method according to claim 1, characterized in that: After sending the final printed image to a printing device so that the printing device prints the final printed image, the method further includes: If the match fails, the key feature is searched in a preset search engine to obtain a number of search results; Extracting search result feature information of each search result respectively; Calculating the similarity between the key feature and the search result feature information respectively to obtain a number of song similarities; Determine the lyrics information and rhythm information corresponding to the greatest song similarity; generating second primary printing information, wherein the second primary printing information includes lyrics information and rhythm information arranged in the preset format; Rendering the second primary printing information according to a preset style to obtain a final printed image; Execute the step of sending the final printed image to a printing device so that the printing device prints the final printed image.

5. The method according to claim 4, characterized in that After respectively calculating the similarities between the key features and the search result feature information to obtain a number of song similarities, the method further includes: The search result feature information in the search result corresponding to the greatest song similarity is converted into a corresponding added audio fingerprint through a hash algorithm; The added audio fingerprint is added to the preset song library.

6. The method according to claim 1, characterized in that The rendering operation of the first primary printing information according to a preset style to obtain a final printed image specifically includes: The preset style includes font, font size and font color; Rendering the lyrics information into the pre-printed image according to the preset style to obtain a secondary printed image; Determining visual elements of the rhythmic information; Rotating the visual element into the secondary printed image to obtain a standby printed image; The time is added to the prepared print picture to obtain the final print picture.

7. The method according to claim 6, characterized in that The adding of time to the prepared print picture to obtain the final print picture specifically includes: Determine the time type to be added, the time type includes the current playing time, the total song time and the remaining time; Formatting the time type to obtain a unified time type; The uniform time type is rendered into the prepared print picture using an image processing library to obtain the final print picture.

8. A lyrics and rhythm printing system based on audio fingerprint, characterized in that: The system comprises: One or more processors and a memory; the memory is coupled to the one or more processors, the memory is used to store computer program code, the computer program code includes computer instructions, and the one or more processors call the computer instructions to cause the system to execute the method as described in any one of claims 1-7.

9. A computer-readable storage medium comprising instructions, characterized in that: When the instructions are executed on a system, the system is caused to execute the method according to any one of claims 1 to 7.

10. A computer program product, characterized in that When the computer program product is run on a system, the system is caused to execute the method according to any one of claims 1 to 7.

Images