Lyrics processing method and related product

Abstract

Embodiments of the present application disclose a song lyrics processing method and related products. In order to detect the wrong artificial annotation timestamp, a plurality of timestamps such as a first timestamp and a second timestamp are introduced. The higher the alignment confidence calculated by comparing the timestamp deviation is, on the one hand, the higher the accuracy of the second timestamp is, and on the other hand, it is more reasonable to indirectly show that if there is a large deviation between the artificial annotation timestamp and the second timestamp, the artificial annotation timestamp is more likely to be a wrong timestamp. Embodiments of the present application adopt a twice threshold filtering mode, that is, the alignment confidence, the unit time deviation and the respective threshold are automatically compared, which can reduce the comparison error caused by completely relying on one comparison result (i.e. once filtering), test whether the artificial annotation timestamp is wrong from multiple aspects, make the detection result more reliable, and at the same time, can speed up the problem detection efficiency of a large number of artificial annotation timestamps.

Description

Technical Field

[0001] The embodiments of the present application relate to the field of audio technology, and in particular to a method for processing lyrics and related products. Background Art

[0002] When a music player plays a song, the interface usually scrolls to display the lyrics text corresponding to the current song playing progress, and to achieve this effect, a lyrics source file (qrc, Qt Recources) is usually required. As a word-by-word timestamp file of the song, the qrc file contains the lyrics corresponding to the song, as well as the start playing time and the continuous playing time (usually in milliseconds ms) of each word in the lyrics corresponding to the song in this song. As expressed in the following partial content of the qrc file, the word "跑" will start playing at the 21042ms and end after 513ms:

[0003] 跑(21042,513)啊(21555,544)啊(22099,672)啊(22771,208)的(22979,240)山(23219,943)上(24162,872)一(25298,473)朵(25771,487)啊(26258,528)啊(26786,241)的(27027,288)云(27315,517)哟(27832,1218)啊(29394,520)啊(29914,497)啊(30411,473)啊(30884,294)的(31178,288)照(31466,1001)在(32467,808)

[0004] The above qrc file is usually made by manual annotation or machine automatic alignment annotation. Although the existing machine automatic alignment annotation technology (or lyrics timestamp technology) can achieve more accuracy than manual annotation for most of the lyrics, there are still some lyrics that will have a large deviation, and this kind of large deviation rarely occurs in manual annotation, which results in the current lyrics timestamp technology being unable to directly replace the manual annotation method. It can be seen that the manual annotation method still has certain advantages, but it is still necessary to detect the problematic manual qrcs made in this way to avoid words with inaccurate timing annotation, resulting in the out-of-sync of the content sung by the singer and the words scrolled and displayed in the lyrics, which affects the user experience.

[0005] Currently, some users will feedback the lyrics with inaccurate timing to the background, and the background professionals will conduct manual review on the feedback manual qrc timestamps. However, the number of songs in the music library is increasing day by day. Relying on user feedback to detect each problematic lyrics in this way is inefficient and not conducive to effectively handling a large number of manual problem qrcs. Therefore, an effective solution needs to be provided. Summary of the Invention

[0006] Embodiments of the present application provide a lyrics processing method and related products for efficiently detecting problems qrc of time stamps labeled by human errors.

[0007] The first aspect of the embodiments of the present application provides a lyrics processing method, comprising:

[0008] obtaining, in a pure lyrics text of a target song, a first timestamp, a second timestamp and a human-labeled timestamp respectively corresponding to each text unit; wherein the first timestamp, the second timestamp and the human-labeled timestamp of the text unit respectively label the starting playing time and the continuous playing time of the text unit in the target song in different ways; the first timestamp is obtained by identifying the audio features of the target song; the second timestamp is the timestamp corresponding to the text unit after aligning the text unit to the vocal lyrics of the target song;

[0009] calculating the alignment confidence between the first timestamp and the corresponding second timestamp of each text unit based on the timestamp deviation between the first timestamp and the corresponding second timestamp; the reference timestamp is any one of the first timestamp and the second timestamp, or a third timestamp calculated based on the two; the higher the alignment confidence, the more consistent the first timestamp and the corresponding second timestamp are;

[0010] calculating the unit time deviation between the second timestamp and the corresponding human-labeled timestamp of each text unit;

[0011] determining the labeling anomaly detection result for the human-labeled timestamp based on the alignment confidence and the unit time deviation.

[0012] Optionally, after determining the labeling anomaly detection result for the human-labeled timestamp, the method further comprises:

[0013] if the labeling anomaly detection result indicates that the human-labeled timestamp is a wrongly labeled timestamp, replacing the wrongly labeled timestamp with another timestamp; the other timestamp is determined based on the first timestamp and the second timestamp of the text unit corresponding to the unit time deviation.

[0014] Optionally, the calculating the alignment confidence between the first timestamp and the corresponding second timestamp of each text unit based on the timestamp deviation between the first timestamp and the corresponding second timestamp comprises:

[0015] If the number of lyrics identified from the audio features of the target song is consistent with the number of lyrics in the pure lyrics text, the alignment confidence between the first timestamp and the corresponding second timestamp is calculated based on the ratio between the alternative value of each text unit and the first timestamp.

[0016] The alternative value includes one of the timestamp deviation of the text unit or the corresponding first timestamp.

[0017] Optionally, the alignment confidence between the first timestamp and the corresponding second timestamp is calculated based on the timestamp deviation between the first timestamp and the corresponding second timestamp of each text unit.

[0018] If the number of lyrics identified from the audio features of the target song is inconsistent with the number of lyrics in the pure lyrics text, at least one target sentence is selected from the identified lyrics text; the target sentence refers to a sentence with a preset number of words that also appears continuously in the pure lyrics text.

[0019] For each text unit in the target sentence, the alignment confidence of the target sentence is calculated based on the ratio between the alternative value of each text unit and the first timestamp of each text unit.

[0020] The alternative value includes one of the timestamp deviation of the text unit or the corresponding first timestamp.

[0021] Optionally, the selection of at least one target sentence from the identified lyrics text includes:

[0022] According to a preset sliding step, at least one target sentence is selected from the identified lyrics text from front to back.

[0023] Optionally, if there is no target sentence but a short sentence in the identified lyrics text within a preset length, the short sentence also appears continuously in the pure lyrics text but has fewer words than the target sentence, the method further includes:

[0024] The alignment confidence of the target sentence adjacent to the short sentence is used as the alignment confidence of the short sentence; the alignment confidence of the short sentence is used to participate in the calculation of the alignment confidence of the entire pure lyrics text.

[0025] Optionally, if there is no target sentence adjacent to the short sentence, the method further includes:

[0026] The alignment confidence of the short sentence is set to be less than a preset threshold.

[0027] Optionally, the selecting at least one target lyric from the recognized lyric text comprises:

[0028] calculating a change distance between the recognized lyric text and the pure lyric text by using a distance calculation function; the change distance is used to represent a cost of a lyric text with a preset number of words being changed into another lyric text with a preset number of words;

[0029] taking the change distance smaller than the preset distance as a target distance, and taking a lyric from the recognized lyric text pointed by the target distance as the target lyric.

[0030] Optionally, the determining of the ratio result comprises:

[0031] taking a timestamp deviation about a starting playing time between the first timestamp and the corresponding second timestamp of each text unit as the alternative value;

[0032] determining a ratio between a sum of the alternative values of the text units and a first duration sum as the ratio result; the first duration sum is a sum of the first timestamps of the text units with a duration playing time.

[0033] Optionally, the determining of the labeling anomaly detection result for the artificial labeling timestamp based on the alignment confidence and the unit time deviation comprises:

[0034] comparing the alignment confidence and the unit time deviation with a confidence threshold and a unit time threshold, respectively;

[0035] if the alignment confidence is greater than the confidence threshold and the unit time deviation is greater than the unit time threshold, determining that the labeling anomaly detection result is that the artificial labeling timestamp is a wrongly labeled timestamp.

[0036] The second aspect of the embodiments of the present application provides an electronic device, comprising: a processor and a memory;

[0037] The processor is configured to communicate with the memory and execute instructions in the memory to implement the method described in the first aspect of the embodiments of the present application or any specific implementation manner of the first aspect.

[0038] The third aspect of the embodiments of the present application provides a computer readable storage medium, the readable storage medium stores computer instructions, and the computer instructions are executed by a processor to implement the method described in the first aspect of the embodiments of the present application or any specific implementation manner of the first aspect.

[0039] The fourth aspect of the embodiments of the present application provides a computer program product, which comprises computer instructions, and the computer instructions, when executed by a processor, implement the method described in the first aspect of the embodiments of the present application or any specific implementation manner of the first aspect.

[0040] From the above technical solutions, the embodiments of the present application have at least the following advantages:

[0041] To detect the wrong artificial annotation timestamp, the embodiments of the present application introduce multiple timestamps such as the first timestamp and the second timestamp. The higher the alignment confidence calculated by comparing the timestamp deviation is, on the one hand, it indicates that the accuracy of the second timestamp is higher, and on the other hand, it also has reason to show from the side that if there is a large deviation between the artificial annotation timestamp and the second timestamp, the artificial annotation timestamp is more likely to be a wrong timestamp. Through the alignment confidence and the unit time deviation, the embodiments of the present application can reduce the comparison errors caused by completely relying on one result, and test whether the artificial annotation timestamp is wrong from multiple aspects, so that the detection result is more reliable. BRIEF DESCRIPTION OF DRAWINGS

[0042] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the following will briefly introduce the drawings needed in the embodiment description. Obviously, the drawings in the following description are only some embodiments described in the present application, and other drawings can also be obtained by those skilled in the art according to these drawings.

[0043] It should be noted that although each step in the flowchart (if any) involved in each embodiment is drawn in sequence according to the arrow, unless otherwise stated in this paper, the execution of these steps has no strict order limit, and these steps can be executed in other order. Moreover, at least one part of the steps in the flowchart involved in each embodiment can include multiple steps or multiple stages, which do not necessarily be executed at the same time, but can be executed at different times, and the execution order of these steps or stages is not necessarily sequential, but can be executed in rotation or alternation with other steps or steps or stages in other steps.

[0044] Figure 1 The system architecture diagram of the lyrics processing method of the embodiments of the present application;

[0045] Figure 2 、 Figure 3 The flowchart of the lyrics processing method of the embodiments of the present application;

[0046] Figure 4 The structure diagram of the electronic device of the embodiments of the present application. DETAILED DESCRIPTION

[0047] In order to make the purposes, technical solutions and advantages of the present application clearer, the present application will be further described in detail below with the accompanying drawings. The described embodiments should not be regarded as limitations to the present application. All other embodiments obtained by those of ordinary skill in the art without creative work under the premise that the described embodiments should fall within the scope of protection of the present application.

[0048] The terms "first", "second", "third", "fourth" and the like used in the description and the claims of the present application and the drawings (if any) are used to distinguish similar objects, and do not necessarily indicate a specific order or sequence. It should be understood that the data thus used can be interchanged under appropriate circumstances, so that the embodiments described herein can be implemented in an order other than that illustrated or described herein. In addition, the terms "comprise" and "have" and any variations thereof are intended to cover non-exclusive inclusion, for example, a process, method, system, product or device comprising a series of steps or units does not necessarily have to be limited to those steps or units clearly listed, but can include other steps or units not clearly listed or inherent to these processes, methods, products or devices.

[0049] In the following description, references are made to "one embodiment" or "one specific example" and the like, which describe a subset of all possible embodiments, but it can be understood that "one embodiment" or "one specific example" can be the same subset or different subsets of all possible embodiments, and can be combined with each other without conflict. In the following description, the term "a plurality of" refers to at least two. If a certain numerical value reaches a threshold value (if any) according to the present application, it can include the case where the former is greater than the threshold value in some specific examples. If "any" or "at least one" and the like are mentioned, it can specifically refer to any one of the listed examples or any combination between these examples.

[0050] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which the present application belongs. The terms used herein are only for the purpose of describing the embodiments of the present application and are not intended to limit the present application.

[0051] Please refer to Figure 1 , Figure 1 is a system architecture diagram provided by the embodiments of the present application. As Figure 1As shown, the system architecture may include a business server 100 and a terminal cluster. The terminal cluster may include terminal devices 200a, 200b, 200c, ..., 200n. The business server 100 may be an independent physical server, a server cluster composed of multiple physical servers, or a distributed system. It may also be a cloud server providing basic cloud computing services such as cloud databases, cloud services, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, CDN, and big data and artificial intelligence platforms. The terminal devices (including terminal devices 200a, 200b, 200c, ..., 200n) may be smartphones, tablets, laptops, desktop computers, PDAs, mobile internet devices (MIDs), wearable devices (such as smartwatches and smart bracelets), smart computers, smart in-vehicle devices, and other smart terminals. The business server 100 can establish communication connections with each terminal device in the terminal cluster, and the terminal devices in the terminal cluster can also establish communication connections with each other. In other words, the service server 100 can establish communication connections with each of the terminal devices 200a, 200b, 200c, ..., 200n. For example, a communication connection can be established between terminal device 200a and the service server 100. A communication connection can be established between terminal device 200a and terminal device 200b, and also between terminal device 200a and terminal device 200c. The above communication connections are not limited to any particular method; they can be established directly or indirectly via wired communication or wireless communication, etc. The specific method can be determined according to the actual application scenario, and this application does not impose any restrictions on it.

[0052] It should be understood that, such as Figure 1 Each terminal device in the terminal cluster shown can have an application client installed. When the application client runs on each terminal device, it can interact with the aforementioned... Figure 1The business server 100 shown exchanges data with each other, so that the business server 100 can receive service data (such as a song uploaded by a user through a terminal device) from each terminal device. The application client can be a browser application, a social application, an instant messaging application, a live broadcast application, a game application, a short video application, a video application, a music application, a karaoke application, or any other application client that has the function of displaying data information such as text, images, audio, and video. The specific application client can be determined according to the actual application scenario, and is not limited herein. The application client can be a standalone client or an embedded sub-client integrated in a certain client (such as an instant messaging client or a social client), and the specific application client can be determined according to the actual application scenario, and is not limited herein.

[0053] Taking the music application as an example, in the process of using the music application through the terminal device, the user can not only sing favorite songs, but also upload the song audio and artificial annotation qrc produced through the terminal device to the business server 100. The singer and the uploader of the artificial annotation qrc can not be the same user. Each terminal device (terminal device 200a, terminal device 200b, terminal device 200c,..., terminal device 200n) that establishes a communication connection with the business server 100 can play the song audio and artificial annotation qrc sent by the user through the music application. In addition, after the business server 100 receives the song audio and artificial annotation qrc sent by the user through the music application, the business server 100 can process the artificial annotation qrc of the song. The processing process can include: calculating the alignment confidence between the first timestamp and the corresponding second timestamp of the lyrics of the song; calculating the unit time deviation between the second timestamp and the corresponding artificial annotation timestamp of each text unit; and automatically comparing the alignment confidence, the unit time deviation, and the respective thresholds to detect the timestamps (i.e., the problematic qrc) that are incorrectly annotated in the artificial annotation qrc. Then, the incorrectly annotated timestamps can be repaired to improve the quality of the song qrc and enhance the visual and auditory experience of the user for the song information.

[0054] The method provided by the embodiments of the present application can be executed by the business server 100 as shown in Figure 1 The method provided by the embodiments of the present application can also be executed by any one of the terminal devices 200a, 200b,..., 200n as shown in Figure 1 The method provided by the embodiments of the present application can also be executed by the terminal device and the business server together, and the specific execution can be determined according to the actual application scenario, and is not limited herein.

[0055] The method provided by the embodiments of the present application can also be executed by the terminal device and the business server together, and the specific execution can be determined according to the actual application scenario, and is not limited herein.

[0056] Please refer to Figure 2The first aspect of the present application provides a specific embodiment of a lyrics processing method, which comprises the following operation steps:

[0057] In step S21, the first timestamp, the second timestamp and the manually annotated timestamp corresponding to each text unit in the pure lyrics text of the target song are obtained.

[0058] The first timestamp, the second timestamp and the manually annotated timestamp of the text unit are annotated in different ways to indicate the starting playback time and the continuous playback time of the text unit in the target song. The first timestamp is obtained by recognizing the audio features of the target song. The second timestamp is the timestamp corresponding to the text unit after aligning the text unit to the vocal lyrics of the target song.

[0059] In some examples, the first timestamp and the second timestamp can be obtained by a model output, which will be described below, and will not be described here. The difference is that the first timestamp is obtained by the model recognizing the audio features autonomously, such as an automatic speech recognition (ASR) model, which does not rely on or use the known pure lyrics text. The second timestamp is output by using the known pure lyrics text, which can be referred to as an automatic alignment algorithm. The generation of the two is different, so the results are generally different, and there is a need to compare the two. Specifically, the automatic alignment algorithm analyzes the acoustic features of the audio signal in the context of existing lyrics and audio alignment, and matches them with the word pronunciation or syllable pronunciation in the lyrics text to determine the starting and ending time of each lyrics element in the audio. It can be simply understood that the playback time of the current pronunciation in the audio is found word by word. The ASR model mainly receives an audio speech file as input, and does not input the pure lyrics text. After the ASR model recognizes the speech content, it further decomposes the recognition result into words or phrases, that is, it focuses on automatically recognizing the lyrics text from the audio and trying to assign timestamps to them.

[0060] It is further explained that the above-mentioned pure lyrics text refers to pure text content without lyrics timestamp information. A text unit includes a single character or a single phoneme. A phoneme can refer to the smallest unit of speech divided according to the natural properties of a language. One phonation action forms one phoneme, for example, the pronunciation of "ah", "bo", and "de". Each pronunciation can correspond to a phoneme. There are various ways to represent phonemes, for example, the phonemes corresponding to the pronunciation of Mandarin can be represented by pinyin, such as "p", "u", "t", "o", "ng", "h", "u", and "a". Specifically, it can be understood as an initial consonant and a vowel. The phonemes corresponding to the pronunciation of English can be represented by phonetic symbols, such as / i / , / I / , / e / , etc.

[0061] The artificial annotation timestamp can refer to a timestamp divided by a relevant person according to his / her experience, etc. for a text unit, such as annotation of when a character starts playing and how long it lasts. Compared with the first timestamp and the second timestamp obtained by a machine, the artificial annotation timestamp can have subjective judgment errors, and thus needs to be verified and corrected.

[0062] In step S22, alignment confidence between the first timestamp and the corresponding second timestamp is calculated based on the timestamp deviation between the first timestamp and the corresponding second timestamp of each text unit.

[0063] It can be understood that the alignment confidence is determined by dynamically aligning the first timestamp and the second timestamp. The higher the alignment confidence (or simply confidence) is, the closer and more consistent the second timestamp is with the first timestamp. It can be understood that the more consistent the results or ideas given by multiple parties (i.e., the smaller the deviation), the more reliable the results or ideas are, that is, the more reliable the evaluation timestamp is. In this case, if a third-party timestamp is compared with the evaluation timestamp, it can be initially determined that the third-party timestamp has a problem to some extent. Therefore, to further accurately verify whether the artificial annotation timestamp is incorrect or the evaluation timestamp compared with the artificial annotation timestamp is incorrect, the reliability of the evaluation timestamp can be verified to enhance the detection reliability of the artificial annotation timestamp.

[0064] In step S23, unit time deviation between the second timestamp and the corresponding artificial annotation timestamp of each text unit is calculated.

[0065] The timestamp of the lyrics includes two time concepts of start playing time and duration playing time. Taking a single character as an example, the unit time deviation S can be compared character by character, and the calculation formula is as follows:

[0066] S = |time_auto_begin-time_man_begin| - |time_auto_end-time_man_end|

[0067] Wherein, time_auto_begin and time_auto_end are the start time (i.e., start playing time) and end time of a character in the evaluation timestamp, and time_man_begin and time_man_end are the start time and end time of a character in the artificial annotation qrc. The end time of the character here can refer to the time point indicated by the "duration playing time" or "start playing time + duration playing time". Of course, when calculating the unit time deviation S, only the start time of the character can be considered without considering the end time of the character, because in some examples, as long as the start deviation of the character is aligned, the end deviation of the character can also be naturally aligned. Of course, when calculating the unit time deviation S, only the end time of the character can be considered without considering the start time of the character.

[0068] Similar to the process of calculating the unit time deviation, the timestamp deviation of each word can be calculated word by word by combining the two time concepts of the starting playing time and the duration playing time according to the above formula S. Of course, either of the two time concepts can be selected to calculate the timestamp deviation, and the specific selection is not limited. Since the alignment confidence is calculated based on the timestamp deviation, it can be understood that the alignment confidence can also be calculated based on at least one of the two time concepts, and the specific selection can be determined according to the actual situation.

[0069] It is additionally explained that the artificial qrc of the embodiments of the present application mainly refers to the timestamp of the lyrics rather than the lyrics themselves, and accordingly, the detection problem qrc mainly refers to the timestamp to be detected which is marked by the artificial misalignment rather than the lyrics themselves.

[0070] In step S24, based on the alignment confidence and the unit time deviation, a marking abnormality detection result for the artificial marking timestamp is determined.

[0071] The marking abnormality detection result can be used to represent whether the artificial marking timestamp is a wrongly marked timestamp. For example, according to a threshold (such as a confidence threshold, a unit time threshold) comparison logic, it can be determined whether the artificial marking timestamp of the current text unit is a wrongly marked timestamp, and the specific implementation can be seen below.

[0072] If the alignment confidence and the confidence threshold are compared, and the unit time deviation and the unit time threshold are compared in step S24, the execution order between the two comparison processes can not be limited, and the two comparison processes can be executed simultaneously, and the specific implementation can be determined according to the actual situation.

[0073] In summary, in order to detect the wrongly marked artificial marking timestamp, the embodiments of the present application introduce multiple timestamps such as the first timestamp and the second timestamp. The higher the alignment confidence calculated by the compared timestamp deviation is, on the one hand, it can be explained that the accuracy of the second timestamp is higher, and on the other hand, it can also be explained that if there is a large deviation between the artificial marking timestamp and the second timestamp, the artificial marking timestamp is more likely to be a wrongly marked timestamp. It can be seen that the embodiments of the present application adopt a two-time threshold filtering method, that is, the alignment confidence, the unit time deviation and the respective thresholds are automatically compared, which can reduce the comparison errors caused by completely relying on one comparison result (i.e. one-time filtering), and can test the artificial marking timestamp from multiple aspects to make the detection result more reliable, and at the same time, the problem detection efficiency of a large number of artificial marking timestamps can be accelerated.

[0074] The execution order between the above-mentioned steps S22 and S23 can not be limited, and can also be executed simultaneously. On the basis of the above-mentioned example explanation, some specific possible implementation examples will be provided below. In actual application, the implementation content between these examples can be combined or implemented individually according to the corresponding functional principle and application logic. If combined and implemented, the execution order between the combined examples can be determined according to the respective processing logic, and the specific implementation can be determined according to the actual scene.

[0075] Based on the example content of Figure 2 , the present application provides another specific embodiment of a lyrics processing method, which includes the following operation steps:

[0076] Step S21, obtaining the first timestamp, the second timestamp, and the manually annotated timestamp corresponding to each text unit in the pure lyrics text of the target song.

[0077] As shown in Figure 3 , the audio of the target song and the corresponding pure lyrics text can be obtained, and then input in two ways to output the first timestamp and the second timestamp corresponding to each text unit. Specifically, the first way can be to input the audio of the target song into an ASR (Automatic Speech Recognition) model. The ASR model can automatically output the lyrics text of the target song (which can be referred to as ASR lyrics) and the first timestamp corresponding to the lyrics text by recognizing the audio features. That is, the ASR model can generate a qrc file by itself, and the timestamp information in the qrc file is generally different from the second timestamp and the manually annotated timestamp. The second way can be to input the audio of the target song and the pure lyrics text into an automatic alignment model to output the second timestamp corresponding to the pure lyrics text.

[0078] The difference between the above-mentioned automatic alignment model and the ASR model is that the input and output are different. Specifically, the text of the automatic alignment model is known, and the model only needs to align the vocals in the song audio with the text, such as synchronously shifting the known pure lyrics text with the time progress information of the vocals. The ASR model only inputs the song audio without text information, and the model needs to recognize the words sung in the song audio to output a version of lyrics and the corresponding timestamp. In actual application, in terms of effect, the timestamp output by the ASR model is often more reliable than the timestamp output by the automatic alignment model. If the automatic alignment timestamp and the ASR output timestamp are very close, it can be considered that the alignment confidence of the automatic alignment result will be very high, but the defect is that the lyrics recognized by the ASR may not be consistent with the original pure lyrics text. For the timestamp task, the inconsistency of the word content does not have much impact, but the inconsistency of the lyrics length can easily cause misalignment problems.

[0079] Step S22, calculating the alignment confidence between the first timestamp and the corresponding second timestamp based on the timestamp deviation between each text unit.

[0080] Please refer to Figure 3 The second timestamp can be used as a candidate timestamp compared with the manually annotated timestamp, but in some examples, the candidate timestamp can be determined based on the first timestamp and the second timestamp, for example, candidate timestamp = a*first timestamp + b*second timestamp, where a and b are weight values, and when one of the weights is 0, it means that the candidate timestamp is the other (for example, if b is 0, the candidate timestamp is the first timestamp); of course, a and b can also take non-zero values, and the candidate timestamp at this time can be the third timestamp, such as one of the time points between the first timestamp and the corresponding second timestamp. In short, the above-mentioned candidate timestamp can be determined by the actual situation, and for the convenience of understanding, the second timestamp is mainly used as a candidate timestamp in the examples of the present application.

[0081] The text unit can be a single character at the character level, and of course it can also be a single phoneme at the phoneme level, which is not limited. Because the lyrics recognized by the ASR and the original pure lyrics text length can be inconsistent, the following will be described in detail according to the situation.

[0082] a. As a possible implementation, the specific operation process of step S22 can include: if the number of lyrics recognized by the audio features of the target song is consistent with the number of lyrics in the pure lyrics text, calculating the alignment confidence between the first timestamp and the corresponding second timestamp based on the ratio result between the selected value of each text unit and each first timestamp; the selected value includes one of the timestamp deviation of the text unit or the corresponding first timestamp.

[0083] For example, the alignment confidence of each character can be determined word by word first, and the alignment confidence of the entire lyrics text is obtained by summing the alignment confidence of each character. The alignment confidence of the i-th character can be the ratio result between the timestamp deviation of the i-th character and the total duration of all characters (m). The timestamp deviation can be the difference between different model outputs in the timestamp concept about the starting playback time or the duration, or the difference after combining the two single concepts. The total duration can be the total duration of all characters of any of the above model outputs.

[0084] For example, if the number of lyrics recognized by the ASR and the number of lyrics aligned automatically (i.e. the number of original pure lyrics text) are consistent, the alignment confidence at this time can be:

[0085]

[0086] wherein, len_asr_word is the sum of the word durations output by the ASR model; di represents the timestamp deviation between the timestamp of the i-th word output by the ASR model and the timestamp of the i-th word output by the automatic alignment model, the calculation formula of the timestamp deviation di can be as follows: di = |time_auto_begin - time_ai_begin| - |time_auto_end - time_ai_end|; ai is the duration of the current word output by the ASR model. min(di, ai) is the above-mentioned alternative value, which means that if the timestamp deviation di is too large, it is larger than the word duration ai output by the ASR model, then the word duration ai is taken, which is to ensure that the value of the alignment confidence is in the range of 0 to 1. The m here can be the total number of words of the whole lyrics, of course, the parameter m can be the number of words of part of the lyrics in some cases, which can be determined according to the situation.

[0087] In some examples, the above-mentioned alignment confidence conf avg The formula is transformed to obtain other alignment confidence calculation formulas, for example, the above-mentioned alignment confidence conf avg In the denominator len_ASR_word, the sum of the start times between the 1st word and the mth word output by the ASR model can be tried to replace; or, conf avg In the numerator, the above-mentioned formula can be replaced by The denominator can also be replaced. In other words, in some examples, the deviation of the first timestamp in the continuous playing time is not used to calculate the alignment confidence, specifically, the determination process of the above-mentioned ratio result can include: taking the timestamp deviation about the start playing time between the first timestamp and the corresponding second timestamp of each text unit as the alternative value, for example, the timestamp deviation di = |time_auto_begin - time_ai_begin|; the ratio between the sum of the alternative values of each text unit and the first duration sum is determined as the ratio result; the first duration sum is the sum of the continuous playing time in the first timestamp of each text unit, as len_asr_word above.

[0088] Or, the above-mentioned alignment confidence conf avg formula can be replaced by the dynamic time warping formula (DTW, Dynamic Time Warping) because the DTW formula can essentially reflect the similarity between two time series (which can be seen in detail below).

[0089] As described above, if the number of words recognized by the ASR and the number of words in the pure lyrics text are consistent (a smooth case), the alignment confidence of the entire lyrics can be calculated, i.e. an alignment confidence is calculated, which can be used for subsequent rapid detection of artificial problem timestamps.

[0090] b. As another possible implementation, the specific operation process of step S22 can include: if the number of words recognized by the audio features of the target song and the number of words in the pure lyrics text are inconsistent, selecting at least one target sentence from the recognized lyrics text; the target sentence refers to a piece of lyrics with a preset number of words and appearing continuously in the pure lyrics text; for each text unit in the target sentence, calculating the alignment confidence of the target sentence based on the ratio result between the alternative value of each text unit and the first timestamp of each text unit; wherein the alignment confidence of each target sentence can be used as the alignment confidence of the entire pure lyrics text; the alternative value includes one of the timestamp deviation of the text unit or the corresponding first timestamp.

[0091] Generally, the actual situation is not as smooth as case a, and the ASR recognition may have more or fewer words, i.e. the number of words recognized by the ASR and the number of words of automatic alignment are not the same. At this time, the alignment confidence of a part of the lyrics instead of the entire lyrics can be calculated, and the calculation formula can be similar to the alignment confidence conf avg The formula; the difference is that the parameter m at this time is the number of words of each small part of the lyrics (such as 3 words), instead of the total number of words of the entire lyrics, and the alignment confidence conf avg obtained is the alignment confidence of each small part of the lyrics.

[0092] The above case b will be described in detail below.

[0093] b1. In some examples, the process of "selecting at least one target sentence from the recognized lyrics text" described above can specifically include: selecting at least one target sentence from the recognized lyrics text by sliding from front to back according to a preset sliding step length.

[0094] For example, the alignment confidence when the ASR output length and the actual lyrics length are inconsistent can be calculated by means of dynamic sliding window. Specifically, the sliding window length, i.e. the preset number of words, can be defined as 3, and the sliding step length is 1, and the target sentence satisfying the sliding window condition is searched from the ASR result, for example:

[0095] Actual lyrics (or original lyrics): "If you love me"

[0096] ASR lyrics: "If you love me"

[0097] The content of the first sliding window is "if oh", which is not consistent in the original lyrics. Therefore, the window is moved one character to the right to obtain the content "fruit oh you", which is still not consistent in the original lyrics. The window is moved one character to the right again until the three characters "you love me" are obtained, which satisfy the condition (appear in the original lyrics). The three characters can be used as the target sentence. Then, the alignment confidence of the current three characters "you love me" can be calculated according to the calculation method in case a. At this time, the alignment confidence conf avg The parameter m in the formula represents the preset number of characters 3. In this way, each target sentence corresponds to an alignment confidence. The alignment confidences of all target sentences can be combined to obtain the alignment confidence of the pure lyrics text as a whole.

[0098] In some examples, as in the above dynamic sliding window manner, there can be a case that no target sentence exists in the recognized lyrics text within the preset length, but a short sentence exists. The short sentence also appears continuously in the pure lyrics text, but the number of characters is less than that of the target sentence. In this case, the embodiment of the present application can further include the following operation: the alignment confidence of the target sentence adjacent to the short sentence is used as the alignment confidence of the short sentence; and the alignment confidence of the short sentence is used to participate in the calculation of the alignment confidence of the pure lyrics text as a whole.

[0099] The above-mentioned short sentence can refer to a sentence with other characters more than the corresponding sentence in the pure lyrics text, such as the exclamation "oh". For example, in the original lyrics and the ASR lyrics in the above example, the preset length is assumed to be 5 characters. If no target sentence satisfying the condition exists within the first 5 characters of the ASR lyrics, but a short sentence "if" exists, the alignment confidence of "if" can reuse the alignment confidence of the adjacent target sentence "you love me". This operation is to avoid that useful sentences are filtered out too much, resulting in a low alignment confidence and an increased false detection rate.

[0100] As described above, the character recognition results of the entire ASR lyrics can be traversed to obtain multiple alignment confidence values. For the sentences that do not satisfy the condition, the alignment confidence of the adjacent target sentence result can be taken as the alignment confidence of the current sentence.

[0101] In some examples, as in the above dynamic sliding window manner, there can be a case that no target sentence exists in the adjacent position of the short sentence. In this case, the embodiment of the present application can further include the following operation: setting the alignment confidence of the short sentence to be less than a preset threshold. The preset threshold can be a confidence threshold.

[0102] For example, it can be assumed that if there is no alignment confidence in the vicinity when traversing (i.e. distance sliding window) from the beginning of the identified lyrics text for more than 5 words, the alignment confidence of the short phrase can be determined to be less than a preset threshold, such as setting the alignment confidence to 0 to indicate that the result is not reliable. Of course, if the preset length is set to be long enough, such as close to or even the total number of words of the entire lyrics, at this time, if there is no nearest alignment confidence when the sliding window exceeds the preset length, it can be determined that the alignment confidence of the entire lyrics is less than the preset threshold, indicating that the result is not reliable. Therefore, it can be seen that in order to make the confidence result reasonable, the preset length should be set according to the actual situation (such as the total number of words of the original lyrics and / or the ASR lyrics phonemes), so as not to affect the confidence result due to the preset length being too long or too short.

[0103] b2. In some examples, the process of "selecting at least one target phrase from the identified lyrics text" described above can specifically include: using a distance calculation function to calculate the change distance between the identified lyrics text and the pure lyrics text; the change distance is used to represent the cost of changing a phrase of a preset number of words in a lyrics text into a phrase of a preset number of words in another lyrics text; the change distance less than a preset distance is taken as a target distance, and the phrase taken from the identified lyrics text pointed by the target distance is taken as a target phrase.

[0104] The distance calculation function described above can refer to a dynamic time warping function (DTW, Dynamic Time Warping). The DTW algorithm can find the optimal deformation path between two time series (which can not be equal in length) by dynamic programming method, so that the cumulative distance (usually based on a certain distance metric such as Euclidean distance) of the two sequences along this path is the minimum cost to achieve deformation. This cumulative distance is the output of the DTW algorithm, which represents the shortest distance between the two time series and reflects the similarity between the two time series. Therefore, it can be understood that in the lyrics field, the DTW algorithm can output the cost of changing a phrase of a lyrics text into a phrase of another lyrics text, and the smaller the deformation cost is, the more similar the two phrases are, so the ASR recognized phrase with small cost can be selected as the target phrase to calculate the alignment confidence. For example, the DTW value of the phrase formed by every 3 words can be calculated, and the phrase corresponding to the smaller DTW value can be taken as the target phrase. In some examples, the entire lyrics text can be divided into groups according to punctuation marks, and the DTW value of the phrase formed by every n words in each group can be calculated, so as to determine the target phrase with the minimum deformation distance in the group, and thus the target phrase of the entire lyrics text can be selected.

[0105] The DTW algorithm described above can be as follows:

[0106] D(i,j) = d(Ai, Bj) + min(D(i-1,j), D(i,j-1), D(i-1,j-1))

[0107] wherein D(i,j) represents the DTW distance from the 1st element to the i th element of time series A, and the 1st element to the j th element of time series B, the time series here can refer to a certain sentence, and the element here can refer to a text unit such as a single word in the embodiment of the application; d(Ai,Bj) represents the distance between the i th element of A and the j th element of B, which is usually a simple distance measure such as Euclidean distance, or the distance can be the time difference between the i th word recognized by ASR and the i th word automatically aligned, which can be calculated similarly to S in the above deviation formula; min(D(i-1,j),D(i,j-1),D(i-1,j-1)) represents the minimum DTW distance between three adjacent points. The above DTW algorithm allows nonlinear stretching or compression on the time series; the above DTW algorithm can not consider the end time of the text unit.

[0108] Step S23, calculating the unit time deviation between the second timestamp of each text unit and the corresponding manually labeled timestamp.

[0109] Step S24, determining the labeling anomaly detection result for the manually labeled timestamp based on the alignment confidence and the unit time deviation.

[0110] In some examples, the specific process of step S24 can include: comparing the alignment confidence and the unit time deviation with a confidence threshold and a unit time threshold, respectively; if the alignment confidence is greater than the confidence threshold, and the unit time deviation is greater than the unit time threshold, determining that the labeling anomaly detection result is that the manually labeled timestamp is a wrongly labeled timestamp.

[0111] As described above, while obtaining the alignment confidence value, the unit time deviation between the automatically aligned timestamp and the manual qrc can be compared. If the alignment confidence is greater than the confidence threshold, and one of the unit time deviations is greater than the unit time threshold, it can be believed with high alignment confidence that the timestamp of the manual qrc is indeed problematic, that is, the current problematic manually labeled timestamp is detected. The manually labeled timestamp without problems can be ignored and not corrected.

[0112] The confidence threshold and the unit time threshold can be set according to experience and historical data. Specifically, an initial confidence threshold can be determined according to alignment confidence generated by a large number of audio samples, and an initial unit time threshold can be determined according to unit time deviation generated by the audio samples. Then, the initial confidence threshold and the initial unit time threshold can be used to check whether the manual annotation timestamps (i.e., manual qrcs) of a batch of audio are incorrect. If the checking result does not meet the expectation, the initial confidence threshold and the initial unit time threshold can be continuously adjusted until the expectation is met, so as to set ideal confidence threshold and unit time threshold. The expectation can be that the ratio between the number of detected manual problem qrcs and the total number of actual problem manual qrcs is greater than a certain percentage value according to the current confidence threshold and the unit time threshold.

[0113] Generally, the higher the threshold is set, the fewer the detection is, and the more reliable it is, but some incorrect qrcs can be missed. The lower the threshold is set, the more the detection is, and the higher the false detection rate is. Therefore, the confidence threshold and the unit time threshold should be reasonably set according to the actual situation, and should not be too high or too low, so as to avoid affecting the detection result.

[0114] As Figure 3 described above, in addition to comparing the second timestamp with the manual annotation timestamp, the first timestamp and the second timestamp are also compared. This alignment confidence-based timing problem qrc detection strategy is based on automatic alignment technology, introduces song ASR technology, dynamically compares the word timestamps generated by automatic alignment and the word timestamps generated by song ASR technology, and obtains an alignment confidence of the timestamps. Compared with the existing manual detection method, the alignment confidence of the automatic alignment timestamp result is added in the embodiment of the application, which avoids the limitation of completely or only relying on the performance of the automatic alignment algorithm, and the detection result obtained by the two threshold filtering methods is more reliable.

[0115] Based on Figure 2 the example content, as a possible implementation manner, after step S24, the embodiment of the application can further include the following operation (error correction): if the annotation anomaly detection result indicates that the manual annotation timestamp is a wrong annotation timestamp, the wrong annotation timestamp is replaced by another timestamp; the other timestamp is determined based on the first timestamp and the second timestamp of the text unit corresponding to the unit time deviation.

[0116] In actual application, if the manually annotated time stamp of a certain text unit, such as a certain word, is incorrect, the first time stamp or the second time stamp of the text unit can be used to replace the manually annotated time stamp, so as to realize the timing error correction of the manual problem qrc. Since the time stamp output by the ASR model is more reliable than the time stamp output by the automatic alignment model, the first time stamp output by the ASR model can be preferably used to replace the incorrect manually annotated time stamp. Of course, as a possible implementation manner, a compromise time stamp can be calculated based on the first time stamp and the second time stamp, and used to replace the incorrect manually annotated time stamp, which is not limited here. In summary, the above other time stamps can be summarized as: other time stamp = a*first time stamp + b*second time stamp, wherein a and b are weight values, and when the weight of a certain term is 0, the other time stamp is another term (for example, when b is 0, the other time stamp is the first time stamp); of course, a and b can also take non-zero values, which can be determined according to actual situations, and are not limited here.

[0117] In summary, the embodiment of the present application proposes a timing problem qrc detection strategy based on alignment confidence. By introducing ASR technology, the alignment confidence of the time stamp generated by the current algorithm can be calculated, and the time deviation of the word of the manual qrc and the algorithm qrc is combined to determine whether the current word time stamp annotated by the manual is a problem, thereby reducing the false detection caused by the deviation of a single technology (such as relying only on the performance of the automatic alignment algorithm), increasing the probability of detecting the problem qrc, and at the same time, helping to reduce the labor cost, repairing more problems qrc quickly and accurately, and promoting the improvement of the quality of the library qrc.

[0118] Compared with Figure 2 As illustrated in the example, the above-mentioned additional or refined examples or possible implementation manners (such as error correction operations) do not necessarily have to be executed in specific implementation, for example, if more than two examples or possible implementation manners are added, these examples or possible implementation manners can be implemented in combination or separately, if implemented in combination, the execution order of the combined examples can be determined according to the respective processing logic, and the specific implementation can be determined according to the actual scene.

[0119] Please refer to Figure 4 The second aspect of the embodiment of the present application provides an electronic device 400, which can include one or more processors (such as central processing units CPU) 401 and a memory 405, and the memory 405 stores one or more application programs or data.

[0120] The memory 405 can be volatile memory or persistent storage. The programs stored in the memory 405 can include one or more modules, each of which can include a series of instruction operations in the electronic device. Further, the processor 401 can be configured to communicate with the memory 405 to execute the series of instruction operations in the memory 405 on the electronic device 400.

[0121] The electronic device 400 can further include one or more power supplies 402, one or more wired or wireless network interfaces 403, one or more input / output interfaces 404, and / or one or more operating systems, such as Windows ServerTM, Mac OS XTM, UnixTM, LinuxTM, FreeBSDTM, etc.

[0122] The processor 401 can perform the operations performed by the first aspect or any of the specific method embodiments of the first aspect, and details are not repeated.

[0123] The present application provides a computer-readable storage medium including instructions that, when executed on a computer, cause the computer to perform the method described in the first aspect or any of the specific implementation manners of the first aspect.

[0124] The present application provides a computer program product including instructions or a computer program, which, when executed on a computer, causes the computer to perform the method described in the first aspect or any of the specific implementation manners of the first aspect.

[0125] It can be understood that in various embodiments of the present application, the sequence number of each step does not mean the order of execution, and the execution order of each step should be determined according to its function and inherent logic, and should not constitute any limitation on the implementation process of the embodiments of the present application. The operation content added or refined by each example scheme of the above method, system or device (if any) does not necessarily have to be executed in specific implementation, such as adding more than two operations, which can be combined or implemented separately, and the specific implementation can be determined according to the actual scene.

[0126] Those skilled in the art can clearly understand that, for the convenience and brevity of the description, the specific working process of the above-described system (if any) and device can refer to the corresponding process in the foregoing method embodiments, which will not be repeated here.

[0127] In several embodiments provided in the present application, it should be understood that the disclosed apparatus and method can be implemented by other manners. For example, the apparatus embodiments described above are merely illustrative, for example, the division of the units is merely a logical function division, and actual implementation can have another division manner, for example, a plurality of units or components can be combined or integrated into another system or apparatus, or some features can be ignored or not executed. In addition, the coupling or direct coupling or communication connection between the units or components shown or discussed can be indirect coupling or communication connection through some interfaces, apparatuses or units, and can be electrical, mechanical or other forms.

[0128] The units described as separate components can or can not be physically separated, and the components shown as units can or can not be physical units, that is, they can be located in one place or distributed on a plurality of network units. Part or all of the units can be selected according to actual needs to achieve the purpose of the embodiment.

[0129] In addition, the functional units in each embodiment of the present application can be integrated in one processing unit, or each unit can be physically present separately, or two or more units can be integrated in one unit. The integrated unit can be realized in the form of hardware or in the form of a software functional unit.

[0130] The integrated unit, if realized in the form of a software functional unit and sold or used as an independent product, can be stored in a computer readable storage medium. Based on this understanding, the technical solutions of the present application essentially or the part of the prior art that contributes or the whole or part of the technical solutions can be embodied in the form of a software product. The computer software product (or computer program product) stored in a storage medium includes a plurality of instructions for causing a computer device (which can be a personal computer, a business server or a network device, etc.) to execute all or part of the steps of the method described in each embodiment of the present application. The foregoing storage medium includes a U disk, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk and various program code storage media.

Claims

1. A lyrics processing method characterized by comprising: The method comprises the following steps: obtaining a first timestamp, a second timestamp and an artificial annotation timestamp corresponding to each text unit in a pure song lyrics text of a target song; wherein the first timestamp, the second timestamp and the artificial annotation timestamp of the text unit respectively represent the starting playing time and the continuous playing time of the text unit in the target song in different ways; the first timestamp is obtained by identifying the audio features of the target song; the second timestamp is the timestamp corresponding to the text unit after aligning the text unit with the vocal lyrics of the target song; calculating the alignment confidence between the sequence of the first timestamp of each text unit and the sequence of the second timestamp of each text unit based on the timestamp deviation between the first timestamp and the corresponding second timestamp of each text unit; wherein the alignment confidence represents the similarity of the first timestamp and the second timestamp of the same text unit in the sequence of the first timestamp and the sequence of the second timestamp, and the higher the alignment confidence is, the more consistent the first timestamp and the corresponding second timestamp are; calculating the unit time deviation between the second timestamp and the corresponding artificial annotation timestamp of each text unit; determining the annotation anomaly detection result of the artificial annotation timestamp based on the alignment confidence and the unit time deviation; if the annotation anomaly detection result indicates that the artificial annotation timestamp of any text unit is a false annotation timestamp, replacing the false annotation timestamp of the text unit with other timestamp of the text unit; the other timestamp of the text unit is determined based on the first timestamp and the second timestamp of the text unit.

2. The lyrics processing method of claim 1, wherein, The method for calculating the alignment confidence between the sequence of the first timestamp of each text unit and the sequence of the second timestamp of each text unit based on the timestamp deviation between the first timestamp and the corresponding second timestamp of each text unit comprises: if the number of lyrics recognized by the audio features of the target song is consistent with the number of lyrics in the pure song lyrics text, calculating the alignment confidence between the sequence of the first timestamp of each text unit and the sequence of the second timestamp of each text unit based on the ratio result between the alternative value of each text unit and each first timestamp; The alternative value includes one of the timestamp deviation of the text unit or the corresponding first timestamp.

3. The lyrics processing method of claim 1, wherein, The method for calculating the alignment confidence between the sequence of the first timestamp of each text unit and the sequence of the second timestamp of each text unit based on the timestamp deviation between the first timestamp and the corresponding second timestamp of each text unit comprises: if the number of lyrics recognized by the audio features of the target song is inconsistent with the number of lyrics in the pure song lyrics text, selecting at least one target sentence from the recognized lyrics text; the target sentence refers to a sentence with a preset number of words that also appears continuously in the pure song lyrics text. For each text unit in the target lyric, based on a ratio result between the alternative value of the text unit and the first timestamp of the text unit, calculate an alignment confidence between a sequence of first timestamps of each text unit containing the target lyric and a sequence of second timestamps of each text unit containing the target lyric; Wherein, the alternative value includes one of the timestamp deviation of the text unit or the corresponding first timestamp.

4. The lyrics processing method of claim 3, wherein, The method further includes: Sliding the target lyric from the recognized lyric text in a preset sliding step from front to back to select at least one segment of the target lyric.

5. The lyrics processing method according to claim 4, characterized by, If there is no target lyric in the recognized lyric text within a preset length but there is a short lyric, the short lyric also continuously appears in the pure lyric text but the number of words is less than that of the target lyric, the method further includes: The alignment confidence of the target lyric adjacent to the short lyric is used as the alignment confidence of the short lyric; and the alignment confidence of the short lyric is used to participate in the calculation of the alignment confidence of the pure lyric text as a whole.

6. The lyrics processing method of claim 5, wherein, If there is no target lyric adjacent to the short lyric, the method further includes: Setting the alignment confidence of the short lyric to be less than a preset threshold.

7. The lyrics processing method of claim 3, wherein, The method further includes: Using a distance calculation function to calculate the change distance between the recognized lyric text and the pure lyric text; the change distance is used to represent the cost of changing a segment of lyrics with a preset number of words in a lyric text into a segment of lyrics with a preset number of words in another lyric text; The change distance less than a preset distance is used as a target distance, and the lyric from the recognized lyric text pointed by the target distance is used as the target lyric.

8. The lyrics processing method of claim 3, wherein, The determination process of the ratio result includes: The timestamp deviation of the start playback time between the first timestamp and the corresponding second timestamp of each text unit is used as the alternative value; The ratio between the sum of the alternative values of each text unit and the first duration sum is determined as the ratio result; the first duration sum is the sum of the duration of continuous playback in the first timestamp of each text unit.

9. The lyrics processing method of claim 1, wherein, The method further includes: The alignment confidence and the unit time deviation are compared with a confidence threshold and a unit time threshold, respectively; If the alignment confidence is greater than the confidence threshold and the unit time deviation is greater than the unit time threshold, it is determined that the annotation abnormality detection result is that the manually annotated timestamp is a wrongly annotated timestamp.

10. An electronic device, comprising: The method further includes: A processor and a memory; The processor is configured to communicate with the memory and execute instructions in the memory to implement the method of any one of claims 1 to 9.

11. A readable storage medium, characterized by, The readable storage medium stores computer instructions, and the computer instructions are executed by the processor to implement the method in any one of claims 1 to 9.

12. A computer program product, characterised in that, The computer program product comprises computer instructions, and the computer instructions are executed by the processor to implement the method in any one of claims 1 to 9.

Images