Synchronization method of song information table, computer device and readable storage medium

Abstract

The embodiment of the invention provides a song information table synchronization method, a computer device and a readable storage medium, which are used for realizing real-time synchronization between an offline song information table and cache data storage system and song data in a song information database. The method provided by the embodiment of the invention comprises the following steps: transferring original song information in a song information database to obtain an offline song information table; the original song information is in a data lake format, and the offline song information table is used for providing a data basis for offline analysis of a song processing task; when the song information in the song information database is updated, acquiring first changed song information in the song information database, and performing data updating on the offline song information table based on the first changed song information; the song data stream of the offline song information table is read to a cache data storage system by using the stream processing mode of the first stream processing task, and the song data in the cache data storage system provides a data basis for online analysis of the song processing task.

Description

Technical Field

[0001] The present invention relates to the field of data processing technology, and in particular to a method for synchronizing a song information table, a computer device, and a readable storage medium. Background Art

[0002] When it comes to music recommendations, side information (additional information) about songs, such as artist information, genre information, language information, style information, and release year, is crucial. The recommendation system performs additional calculations on this information to provide users with higher-quality, more accurate recommendations. Beyond recommendations, music app statistics pages, such as the "Artist Chart," also rely on song information to dynamically calculate the rankings in real time.

[0003] The above music recommendations and online business implementations are inseparable from offline and online analysis of songs, such as Figure 1 As shown, in existing solutions, offline analysis usually relies on the song information Hive table, and online analysis relies on the song information Redis. The song information Hive table is data transferred from the song information database once a day. The song information Redis is based on the data exported from the song information Hive table, and uses the real-time song change data of the song information database to update the exported data. During the day, as time goes by, the data difference between the song information Hive table that the offline analysis relies on and the song information Redis that the analysis relies on will become larger and larger, which will lead to poor accuracy in music recommendations and online business implementation. Summary of the Invention

[0004] An embodiment of the present invention provides a song information table synchronization method, a computer device and a readable storage medium, which utilizes an offline song information table to provide a data basis for offline analysis of song processing tasks, and provides a data basis for online analysis of song task processing through a cache data storage system. Because data is transmitted in real time between the offline song information table and the cache data storage system through the stream processing mode of the first stream processing task, the synchronization of song data in the offline song information table and the cache data storage system is guaranteed, and the consistency and accuracy of data between offline song task processing and online song task processing are correspondingly achieved.

[0005] A first aspect of an embodiment of the present application provides a method for synchronizing a song information table, comprising:

[0006] The original song information in the song information database is transferred to obtain an offline song information table. The original song information is in the data lake format. The offline song information table is used to provide a data basis for offline analysis of song processing tasks.

[0007] When song information in the song information database changes, first changed song information in the song information database is obtained, and data in the offline song information table is updated based on the first changed song information;

[0008] The stream processing mode of the first stream processing task is used to read the song data stream of the offline song information table to the cache data storage system. The song data in the cache data storage system provides a data basis for the online analysis of the song processing task.

[0009] As an optional embodiment, the original song information in the song information database is transferred to obtain an offline song information table, including:

[0010] Obtaining the original song information from the song information database using a message queue to transfer the original song information to the offline song information table;

[0011] or,

[0012] When a preset update cycle is reached, the original song information in the song information data is transferred to the intermediate offline song information table;

[0013] The original song information is inserted into the offline song information table from the intermediate offline song information table using the batch processing mode of the second stream processing task.

[0014] As an optional embodiment, when the song information in the song information database is updated, obtaining first changed song information in the song information database, and updating the offline song information table based on the first changed song information, includes:

[0015] Subscribing to the first changed song information in the song information database by using a message queue;

[0016] The first changed song information is updated to the offline song information table using the stream processing mode of the third stream processing task.

[0017] As an optional embodiment, the method further includes:

[0018] In response to the type setting instruction for the offline song information table, the type of the offline song information table is set to Merge on read type.

[0019] As an optional embodiment, the method further includes:

[0020] In response to a primary key setting instruction for the offline song information table, setting the primary key of the offline song information table to the data ID;

[0021] In response to a pre-aggregation key setting instruction for the offline song information table, setting the pre-aggregation key of the offline song information table to the update time of the data;

[0022] When updating the first changed song information, if the primary key of the first changed song information conflicts with the primary key of the original song information, the pre-aggregate key of the first changed song information is compared with the pre-aggregate key of the original song information, and the song information with the most recent update time is determined as the final updated data.

[0023] As an optional embodiment, the method further includes:

[0024] In response to the setting instruction for the indexing mode of the offline song information table, the indexing mode of the offline song information table is set to bucket index bucketindex mode.

[0025] As an optional embodiment, the method further includes:

[0026] In response to the setting instruction for the update time of the first changed song information, the trigger time for updating the first changed song information to the offline song information table is set to the checkpoint period of the stream processing task checking mechanism.

[0027] As an optional embodiment, the method further includes:

[0028] In response to the instruction for setting the checkpoint period of the third stream processing task checking mechanism, the checkpoint period of the third stream processing task checking mechanism is set to a custom preset duration.

[0029] As an optional embodiment, the method further includes:

[0030] In response to a setting instruction for the first stream processing task attribute, enabling a stream read configuration of the first stream processing task;

[0031] The stream reading interval of the first stream processing task is set to the checkpoint period of the stream processing task checking mechanism.

[0032] As an optional embodiment, the method further includes:

[0033] In response to a setting instruction for the first stream processing task attribute, the stream read attribute of the first stream processing task is set to skip compaction.

[0034] As an optional embodiment, the method further includes:

[0035] Song data is obtained from the cache data storage system using multiple stream processing tasks to perform song task processing on the obtained song data, wherein the song task processing includes filtering target songs, filling in song genre information, filling in song language information, filling in song style information, and filling in song tag information.

[0036] As an optional embodiment, the method further includes:

[0037] For a target stream processing task among the multiple stream processing tasks, the data acquired by the target stream processing task is cached using the state of the stream processing task.

[0038] As an optional embodiment, the method further includes:

[0039] At least one of song model training or song data statistical analysis is performed using the song data in the offline song information table.

[0040] As an optional embodiment, the database is a song database, the stream processing task includes a stream processing task under the Flink framework, the message queue includes a Kafka log system, and the offline song information table includes a Hudi data table.

[0041] A second aspect of an embodiment of the present application provides a computer device comprising a processor, which, when executing a computer program stored in a memory, is used to implement the song information table synchronization method provided in the first aspect of the embodiment of the present application.

[0042] A third aspect of an embodiment of the present application provides a computer-readable storage medium on which a computer program is stored. When the computer program is executed by a processor, it is used to implement the song information table synchronization method provided in the first aspect of the embodiment of the present application.

[0043] The fourth aspect of the embodiment of the present application provides a computer program product on which a computer program is stored. When the computer program is executed by a processor, it is used to implement the song information table synchronization method provided in the first aspect of the embodiment of the present application.

[0044] It can be seen from the above technical solutions that the embodiments of the present invention have the following advantages:

[0045] In the embodiment of the present application, an offline song information table is used to provide a data basis for offline analysis of song processing tasks, and the song data in the cache data storage system is used to provide a data basis for online analysis of song processing tasks. Because the offline song information table can be synchronized with the song data in the song information database, the song data in the cache data storage system is further synchronized with the song data in the offline song information table through the stream processing mode of the first stream processing task. Therefore, the embodiment of the present application realizes real-time synchronization between the offline song information table and the song information database on the one hand, and realizes real-time synchronization between the song data in the cache data storage system and the song data in the offline song information table on the other hand. That is, the offline song information table and the cache data storage system in the embodiment of the present application realize real-time synchronization and consistency with the song data in the song information database, and correspondingly realizes data consistency and accuracy between offline song task processing and online song task processing. BRIEF DESCRIPTION OF THE DRAWINGS

[0046] Figure 1 A schematic diagram of an embodiment of a method for calculating a song chart in the prior art;

[0047] Figure 2 This is a schematic diagram of the architecture of the song information table synchronization system in an embodiment of the present application;

[0048] Figure 3 A schematic diagram of an embodiment of a method for synchronizing song information in an embodiment of the present application;

[0049] Figure 4 Schematic diagram of the process of obtaining song data from the offline song information table and cache data storage system in an embodiment of the present application;

[0050] Figure 5 This is a schematic diagram of an embodiment of storing original song information and first changed song information in an offline song information table in an embodiment of the present application;

[0051] Figure 6 This is a schematic diagram of another embodiment of storing original song information and first changed song information in an offline song information table in an embodiment of the present application;

[0052] Figure 7 This is a schematic diagram of an embodiment of the process of updating the offline song information table in an embodiment of the present application;

[0053] Figure 8 This is a schematic diagram of an embodiment of performing song task processing on song data in a cache data storage system according to an embodiment of the present application;

[0054] Figure 9 This is a schematic diagram of an embodiment of a computer device in an embodiment of the present application. DETAILED DESCRIPTION

[0055] An embodiment of the present invention provides a song information table synchronization method, a computer device and a readable storage medium, which are used to use the offline song information table to provide a data basis for the offline analysis of the song processing task, and to provide a data basis for the online analysis of the song task processing through the cache data storage system. Because the offline song information table and the cache data storage system transmit data in real time through the stream processing mode of the first stream processing task, and the offline song information table is synchronized with the song data in the song information database in real time, the real-time synchronization of the song data in the offline song information table and the cache data storage system with the song data in the song information database is guaranteed, and the consistency and accuracy of the data between the offline song task processing and the online song task processing are correspondingly achieved.

[0056] In order to enable those skilled in the art to better understand the solutions of the present invention, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the embodiments described are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts should fall within the scope of protection of the present invention.

[0057] The terms "first," "second," "third," "fourth," and the like in the specification and claims of the present invention and in the accompanying drawings are used to distinguish similar objects and are not necessarily used to describe a particular order or precedence. It should be understood that the terms used in this manner are interchangeable where appropriate so that the embodiments described herein can be implemented in an order other than that illustrated or described herein. In addition, the terms "including" and "having," and any variations thereof, are intended to cover non-exclusive inclusions. For example, a process, method, system, product, or apparatus comprising a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0058] An embodiment of the present application provides a method for synchronizing a song information table. The general principle of the scheme is: the original song information in the song information database is transferred to obtain an offline song information table; the original song information is in a data lake format, and the offline song information table is used to provide a data basis for offline analysis of song processing tasks; when the song information in the song information database is updated, the first changed song information in the song information database is obtained, and the offline song information table is updated based on the first changed song information, and the song data stream of the offline song information table is read to the cache data storage system using the stream processing mode of the first stream processing task, and the song data in the cache data storage system provides a data basis for online analysis of the song processing task.

[0059] In order to better implement the synchronization method of the song information table, the embodiment of the present application provides a synchronization system for the song information table. Figure 2 , Figure 2 A schematic diagram of the architecture of a song information table synchronization system provided in an embodiment of the present application. The song information table synchronization system may include at least one terminal device 201, a server 202, and a cache data storage system 203; different types of applications may be installed on the terminal device 201, for example, a music playback application, a short video application, a conference communication application, etc. may be installed on the terminal device 201; the terminal device 201 may be a smart phone, a tablet computer, a laptop computer, a desktop computer, a smart car, etc. The server 202 may be used to store a database. The server 202 may be an independent physical server, or a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, content delivery networks (CDN), and basic cloud computing services such as big data and artificial intelligence platforms, etc., and the cache data storage system 203 may be Redis or a distributed cache system, etc.

[0060] For ease of understanding, the following describes the method for generating a real-time data table in the embodiment of the present application. Figure 3 An embodiment of a method for generating a real-time data table in an embodiment of the present application includes:

[0061] 301. Transferring the original song information in the song information database to obtain an offline song information table; the original song information is in a data lake format, and the offline song information table is used to provide a data basis for offline analysis of the song processing task;

[0062] Specifically, the song information in the embodiment of the present application is stored in a song information database. In order to obtain an offline song information table, when the original song information is in the song information database, the original song information in the song information database can be transferred to the offline song information table, wherein the offline song information table here is used to provide a data basis for offline analysis of song processing tasks.

[0063] In order to achieve reliable reading and writing of song information in the song information database, as an optional embodiment, the song information database in the embodiment of the present application can adopt HDFS (Hadoop Distributed File System). HDFS is a distributed file system of Hadoop, and its target usage scenario is that the data has the characteristics of "write once, read many times", and the data write operation is sequential write, that is, writing when the file is created or adding after the existing file.

[0064] The data lake format provides the ability to update and delete data and consume changed data on top of Hadoop's file system.

[0065] As another optional embodiment, in the embodiment of the present application, the data lake format may also be an iceberg table or a paimon table.

[0066] Furthermore, when transferring the original song information in the song information database to the offline song information table, different transfer methods can be used for transfer. The specific transfer method will be described in the following embodiments and will not be repeated here.

[0067] 302. When song information in the song information database is updated, obtain first changed song information in the song information database, and update data in the offline song information table based on the first changed song information;

[0068] Therefore, when the song information in the song information data changes based on the original song information (the change here can be addition, deletion or modification, etc.), the first changed song information in the song information database can be obtained, and the offline song table can be updated based on the first changed song information.

[0069] Because the offline song information table in the embodiment of the present application can achieve real-time synchronization with the song data in the song information database, the synchronization and consistency of the song data in the offline song information table and the song data in the song information database are achieved.

[0070] 303. Utilize the stream processing mode of the first stream processing task to read the song data stream of the offline song information table to the cache data storage system. The song data in the cache data storage system provides a data basis for the online analysis of the song processing task.

[0071] Because the offline song information table provides a data basis for offline song analysis, and in order to maintain the synchronization and consistency of online song analysis and offline song analysis data, the embodiment of the present application further utilizes the stream processing mode of the first stream processing task to read the song data stream of the offline song information table to the cache data storage system, so as to provide a data basis for the online analysis of the song processing task through the song data in the cache data storage system.

[0072] The stream processing application is designed to run continuously and minimize downtime. It can realize continuous processing of unbounded data through the streaming mode to achieve real-time acquisition of song data in the offline song information table. Therefore, the embodiment of the present application reads the song data from the offline song information stream to the cache data storage system through the streaming mode of the first stream processing task, maintaining synchronization and consistency with the song data in the offline song information table.

[0073] As an optional embodiment, when the stream processing task is a stream processing task under the Flink architecture, the stream processing mode of the stream processing task can be a Flink task in stream mode, and the data stream in the offline song information table is read to the cache data storage system through the Flink task in stream mode.

[0074] As another optional embodiment, the stream processing task in the embodiment of the present application can be a stream processing task under the Apache Flink architecture, Apache Kafka Streams architecture, Apache Storm architecture, etc.

[0075] For ease of understanding, Figure 4 A schematic diagram of the data acquisition process of the song information database, offline song information table and cache data system in the embodiment of the present application is given.

[0076] In the embodiment of the present application, an offline song information table is used to provide a data basis for offline analysis of song processing tasks, and the song data in the cache data storage system is used to provide a data basis for online analysis of song processing tasks. Because the offline song information table can be synchronized with the song data in the song information database, the song data in the cache data storage system is further synchronized with the song data in the offline song information table through the stream processing mode of the first stream processing task. Therefore, the embodiment of the present application realizes real-time synchronization between the offline song information table and the song information database on the one hand, and realizes real-time synchronization between the song data in the cache data storage system and the song data in the offline song information table on the other hand. That is, the offline song information table and the cache data storage system in the embodiment of the present application realize real-time synchronization and consistency with the song data in the song information database, and correspondingly realizes data consistency and accuracy between offline song task processing and online song task processing.

[0077] based on Figure 3 and Figure 4 The embodiment described above is an optional embodiment. In the process of synchronizing the song data in the song information database to the offline song information table, the present application can adopt the following method: Figure 5 The first method shown and Figure 6 The second way shown is to synchronize:

[0078] 1. The first method, such as Figure 5 As shown:

[0079] Use a message queue to obtain the original song information from the song information database to transfer the original song information to the offline song information table, use a message queue to subscribe to the first changed song information in the song information database; use the stream processing mode of the third stream processing task to update the first changed song information to the offline song information table.

[0080] Specifically, the song information database in the embodiment of the present application can be installed in the server, and the message queue is installed in the terminal, or installed in another server different from the storage database. In order to obtain the first changed song information in the song information database in real time, the embodiment of the present application can use the message queue to subscribe to the first changed song information in the song information database, and when there is updated data in the song information database (such as added song data, deleted song data or modified song data), directly use the message queue to obtain the first changed song information in the song information database, and use the third stream processing task of the stream processing mode to synchronize the first changed song information to the offline song information table.

[0081] For the original song information in the song information database, the message queue can also query the original song information in the song information database, and store the queried original song information in the message queue, and then further transfer the original song information obtained by the message queue to the offline song information table.

[0082] The embodiment of the present application obtains the original song information in the song information database through a message queue. However, when the amount of original song information is large, the efficiency of obtaining the original song information through a message queue is low.

[0083] Second, the second method, such as Figure 6 As shown:

[0084] When the preset update cycle is reached, the original song information in the song information database is transferred to the intermediate offline song information table, the batch processing mode of the second stream processing task is used to insert the original song information table from the intermediate offline song information table into the offline song information table, and the message queue is used to subscribe to the first changed song information in the song information database; the stream processing mode of the third stream processing task is used to update the first changed song information to the offline song information table.

[0085] It is easy to understand that the data in the song information database is updated gradually, such as a part is updated every day, so the data in the song information database includes the original song information and the first changed song information. When the execution subject of this application is a terminal or another server different from the storage song information database, in order to improve the synchronization speed of the original song information in the offline song information table, the embodiment of the present application can also store the original song information of the song information database in the intermediate offline song information table, and use the batch processing mode of the second stream processing task to insert the original song information in the intermediate offline song information table into the offline song information table. For example, when the second stream processing task is a stream processing task under the Flink architecture, a batch mode Flink task can be started, and the original song information of the intermediate offline song information table can be inserted into the offline song information table by inserting the Flink task in batch mode. Because the batch mode inserts the original song information into the offline song information in a batch manner, compared to Figure 5 In the embodiment, the speed of inserting the original song information in the offline song information table is improved, and for the first changed song information in the song information database, the original song information is inserted into the offline song information table. Figure 5In the same way, subscribe to the first changed song information in the song database through the message queue, and when there is the first changed song information in the song information database, obtain the first changed song information from the song information database, and update the first changed song information to the offline song information table in the stream processing mode of the third stream processing task. For example, when the third stream task is a stream processing task under the Flink architecture, you can start a Flink task in stream mode, and use the Flink task in stream mode to update the first changed song information to the offline song information table in upsert mode.

[0086] It can be seen from this that the process of updating the first changed song information in the offline song information table in the embodiment of the present application is faster than Figure 5 In terms of the method of updating the offline song information table in the embodiment, the speed of inserting original song information into the offline song information table is improved.

[0087] As an optional embodiment, in order to improve the efficiency of streaming reading from offline song information, the embodiment of the present application can also set the type of offline song information to Merge onread type. In this way, when the first changed song information is written in the offline song information, it is written in the form of an incremental file, and the incremental file and the basic file will be merged during streaming reading, thereby improving the speed and efficiency of streaming reading data from the offline song information table.

[0088] As an optional embodiment, in order to avoid a primary key conflict between the first changed song information and the original song information when writing the first changed song information into the offline song information table, the embodiment of the present application may further perform the following steps:

[0089] In response to a primary key setting instruction for the offline song information table, the primary key of the offline song information table is set to the data ID; in response to a pre-aggregate key setting instruction for the offline song information table, the pre-aggregate key of the offline song information table is set to the update time of the data; when updating the first changed song information, if the primary key of the first changed song information conflicts with the primary key of the original song information, the pre-aggregate key of the first changed song information is compared with the pre-aggregate key of the original song information, and the song information with the most recent update time is determined as the final updated song information.

[0090] In the offline song information table, in order to identify each data, a corresponding primary key is set for each data in the data table. The primary key here is equivalent to the identification code of the data, which plays an identification role for the data during the retrieval process. When writing the first changed song information in the offline song information table, in order to avoid the conflict between the written first changed song information and the original song information, the embodiment of the present application also sets a precombine key for each data in the offline song information table, and sets the precombine key of each data to the update time. In this way, when the first changed song information is updated in the offline song information table, if the primary key of the first changed song information conflicts with the primary key of the original song information, the precombine key of the first changed song information is compared with the precombine key of the original song information, and the song information with the most recent update time is determined as the final updated song information.

[0091] For ease of understanding, Figure 7 For example, the following example illustrates:

[0092] Assume that the data is song data. In the original song information of the offline song information table, if the genre of song A with the primary key of 01 is folk style, and in the first changed song information, the genre of song A with the primary key of 01 is rock style, then when updating the first changed song information, a data conflict will occur. In this case, the embodiment of the present application compares the precombine key of the song with the primary key of 01 in the original song information and the first changed song information, because the precombine key The key is the update time of the song, so that the update time of the song with the primary key of 01 in the original song information and the song with the primary key of 01 in the first changed song information can be compared. Assuming that the update time of song A with the primary key of 01 in the original song information is March 10, 2024, and the update time of song A with the primary key of 01 in the first changed song information is October 30, 2024, when updating the first changed song information, the data with the update time of October 30, 2024 is used as the final updated song information, that is, the genre of the song with the primary key of 01 is updated to rock style.

[0093] As an optional embodiment, in order to improve the convenience and efficiency of searching for data during the process of updating the first changed song information, the embodiment of the present application can also set the indexing method of the offline song information table to the bucket index bucketindex mode to improve the efficiency of searching for data in the offline song information table.

[0094] Specifically, in the offline song information table, there are generally two data indexing methods. For example, when the offline song information table is a Hudi table, the Flink task generally supports two indexing methods, one is the flink state index, and the second is the bucket index index. If the flink state index is used, you need to first set index.bootstrap.enabled = true, and then load the full index into the Flink task. This will increase the complexity of the task and increase the maintenance cost of the Flink task. In order to facilitate the simplicity and maintainability of the Flink task when the first changed song information is written in real time in the offline song information table, the embodiment of the present application adopts the bucket index bucket index mode to improve the convenience of indexing data in the offline song information table.

[0095] As an optional embodiment, in order to ensure the fault tolerance of the stream processing task, the embodiment of the present application also sets a checkpoint mechanism in the stream processing task. This mechanism saves state snapshots periodically. When a failure occurs, the Flink task can be restored to the most recent Checkpoint state to ensure the correctness of the data. In order to enable the stream processing task to synchronously restore the data in the offline song information table to the most recent Checkpoint state when an error occurs, the embodiment of the present application sets the trigger time for updating the first changed song information to the offline song information table to the checkpoint period of the stream processing task check mechanism. In this way, when an error occurs in the Flink task, if the data is restored to the most recent Checkpoint state, the data can be synchronously restored to the most recent Checkpoint state in the offline song information table, thereby ensuring the correctness of the data in the offline song information table.

[0096] Furthermore, as an optional embodiment, the embodiment of the present application may further perform the following steps:

[0097] In response to the instruction for setting the checkpoint period of the third stream processing task checking mechanism, the checkpoint period of the third stream processing task checking mechanism is set to a user-defined preset duration.

[0098] Specifically, because the flushing of the first changed song information in the offline song information table is set synchronously with the checkpoint period of the Flink task, and in order to avoid setting the checkpoint period very long, which causes the data in the message queue to be accumulated and not written into the offline song information table, the embodiment of the present application sets the checkpoint period of the third stream processing task inspection mechanism to a custom preset duration, such as 30 seconds, or 1 minute, so that when the message queue obtains the first changed song information in the song information database, the first changed song information can be quickly synchronized to the offline song information table.

[0099] As an optional embodiment, in order to further quickly process the data in the offline song information table, the embodiment of the present application may further perform the following steps:

[0100] In response to the setting instruction for the first stream processing task attribute, the stream reading configuration of the first stream processing task is turned on; and the stream reading interval of the first stream processing task is set to the checkpoint period of the stream processing task inspection mechanism.

[0101] Specifically, when the offline song information table is a Hudi table, the default way for the Flink task to read the Hudi table is snapshot reading, that is, the data in the Hudi table is read in its entirety only at a pre-set time. In order to continuously obtain streaming data from the offline song information table, it is necessary to enable the streaming reading configuration of the first stream processing task so that the Flink task can continuously obtain the first changed song information from the offline song information table. After enabling the streaming reading configuration, the streaming reading time also needs to be configured, that is, how often the first changed song information is streamed from the offline song information table. In order to ensure the timeliness of the streaming reading, the embodiment of the present application sets the streaming reading interval of the first stream processing task to the checkpoint period of the stream processing task inspection mechanism, that is, the streaming reading interval of the first stream processing task is set to the commit time of the checkpoint. In this way, when the Flink task executes a checkpoint each time, it can trigger the first stream processing task to perform the streaming reading operation, thereby improving the timeliness of the streaming reading operation.

[0102] Optionally, in order to avoid repeated execution of the stream reading operation caused by the compact file merging, that is, before the compaction, the first stream processing task has already stream-read the first changed song information once, and after the file performs the data merge, the first stream processing task performs the stream reading operation again. Therefore, the embodiment of the present application can also set the stream reading attribute of the first stream processing task to skip compaction to prevent repeated stream reading of the first changed song information before and after the compact file merge.

[0103] As an optional embodiment, Figure 8 As shown, after reading the song data stream in the offline song information table into the cache data storage system, the embodiment of the present application may further perform the following steps:

[0104] Use the data in the offline song information table to perform song model training or song statistical analysis, use multiple stream processing tasks to obtain song data from the cache data storage system, and perform song task processing on the obtained song data. For the target stream processing task among the multiple stream processing tasks, use the state of the stream processing task to cache the song data obtained by the target stream processing task.

[0105] Specifically, after synchronizing the song data in the song library to the offline song information table, the embodiment of the present application can use the song data in the offline song information table as training samples to train the song model or perform statistical analysis on the song data, that is, perform offline analysis using the data in the offline song information table.

[0106] Furthermore, after caching the data in the offline song information table to the cache data storage system, multiple stream processing tasks can be further created, such as creating multiple Flink tasks to perform further song task processing on the song data in the cache data storage system, such as using multiple Flink tasks to perform target song screening, fill in song genre information (such as folk style, rock style or pop song), fill in song sentence information (such as English songs, French songs, etc.), fill in song style information (such as rough style, graceful style, etc.) and fill in song tag information, etc.

[0107] In the process of multiple stream processing tasks performing task processing on song data, in order to avoid multiple stream tasks repeatedly reading data from the cache data storage system, thereby causing the cache data storage system to be overloaded, the embodiment of the present application can also use the state of the stream processing task to cache the song data obtained by the target stream processing task, so as to avoid the cache data storage system being overloaded due to repeated reading of data in the cache data storage system, thereby reducing the data reading speed.

[0108] It is understandable that, in various embodiments of the present invention, the size of the serial numbers of the above steps does not mean the order of execution. The order of execution of each step should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present invention.

[0109] An embodiment of the present application further provides a computer program product on which a computer program is stored. When the computer program is executed by a processor, it is used to implement the various steps in the above-mentioned method embodiment of the present application.

[0110] The following describes the computer device in the embodiment of the present invention from the perspective of hardware processing:

[0111] like Figure 9 As shown, an embodiment of a computer device 900 in an embodiment of the present invention includes: one or more central processing units (CPUs) 901 and a memory 905, wherein the memory 905 stores one or more application programs or data.

[0112] Memory 905 may be volatile or persistent storage. The program stored in memory 905 may include one or more modules, each of which may include a series of instruction operations on the server. Furthermore, the central processing unit 901 may be configured to communicate with memory 905 and execute the series of instruction operations in memory 905 on the server 900.

[0113] The computer device 900 may also include one or more power supplies 902, one or more wired or wireless network interfaces 903, one or more input and output interfaces 904, and / or one or more operating systems, such as Windows Server™, Mac OS X™, Unix™, Linux™, FreeBSD™, etc.

[0114] The central processing unit 901 can execute each step in the above-mentioned method embodiment of the present application, and the details will not be repeated here.

[0115] The computer device may be a computing device such as a desktop computer, laptop, PDA, or cloud server. The computer device may include, but is not limited to, a processor and memory. Those skilled in the art will appreciate that a processor and memory are merely examples of computer devices and do not constitute a limitation of the computer device. The computer device may include more or fewer components, or a combination of certain components, or different components. For example, the computer device may also include input and output devices, network access devices, buses, and the like.

[0116] The processor may be a central processing unit (CPU), other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or any conventional processor, etc. The processor is the control center of the computer device, connecting various parts of the entire computer device using various interfaces and lines.

[0117] The memory can be used to store the computer programs and / or modules, and the processor implements various functions of the computer device by running or executing the computer programs and / or modules stored in the memory, and calling the data stored in the memory. The memory can mainly include a program storage area and a data storage area, wherein the program storage area can store an operating system, at least one application required for a function, etc.; the data storage area can store data created based on the use of the terminal, etc. In addition, the memory can include a high-speed random access memory, and can also include a non-volatile memory, such as a hard disk, a memory, a plug-in hard disk, a smart memory card (Smart Media Card, SMC), a secure digital (Secure Digital, SD) card, a flash card (Flash Card), at least one disk storage device, a flash memory device, or other volatile solid-state storage device.

[0118] The present invention also provides a computer-readable storage medium on which a computer program is stored. When the computer program is executed by a processor, the processor can be used for each step in the above-mentioned method embodiment of the present application.

[0119] It is understood that if the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a corresponding computer-readable storage medium. Based on this understanding, the present invention implements all or part of the processes in the above-mentioned corresponding embodiment methods, and can also be completed by instructing the relevant hardware through a computer program. The computer program can be stored in a computer-readable storage medium. When the computer program is executed by a processor, it can implement the steps of the above-mentioned various method embodiments. Among them, the computer program includes computer program code, and the computer program code can be in source code form, object code form, executable file or some intermediate form. The computer-readable medium may include: any entity or device capable of carrying the computer program code, recording medium, USB flash drive, mobile hard disk, magnetic disk, optical disk, computer memory, read-only memory (ROM), random access memory (RAM), electric carrier signal, telecommunication signal and software distribution medium. It should be noted that the content contained in the computer-readable medium can be appropriately increased or decreased according to the requirements of legislation and patent practice in the jurisdiction. For example, in some jurisdictions, according to legislation and patent practice, computer-readable media do not include electric carrier signals and telecommunication signals.

[0120] Those skilled in the art will clearly understand that, for the convenience and brevity of description, the specific working processes of the systems, devices and units described above can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0121] In the several embodiments provided in this application, it should be understood that the disclosed systems, devices and methods can be implemented in other ways. For example, the device embodiments described above are merely schematic. For example, the division of the units is merely a logical function division. In actual implementation, there may be other division methods, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be an indirect coupling or communication connection through some interfaces, devices or units, which can be electrical, mechanical or other forms.

[0122] The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of these units may be selected to achieve the purpose of this embodiment according to actual needs.

[0123] In addition, the functional units in the various embodiments of the present invention may be integrated into a single processing unit, each unit may exist physically separately, or two or more units may be integrated into a single unit. The aforementioned integrated units may be implemented in the form of hardware or software functional units.

[0124] As described above, the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit the same. Although the present invention has been described in detail with reference to the above embodiments, those skilled in the art should understand that the technical solutions described in the above embodiments can still be modified, or some of the technical features thereof can be replaced by equivalents. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for synchronizing a song information table, characterized in that: include: Transferring the original song information in the song information database to obtain an offline song information table; The original song information is in a data lake format, and the offline song information table is used to provide a data basis for offline analysis of song processing tasks; When song information in the song information database is updated, obtaining first changed song information in the song information database, and updating data in the offline song information table based on the first changed song information; The song data stream of the offline song information table is read to the cache data storage system using the stream processing mode of the first stream processing task. The song data in the cache data storage system provides a data basis for the online analysis of the song processing task.

2. The method according to claim 1, characterized in that The original song information in the song information database is transferred to obtain an offline song information table, including: Obtaining the original song information from the song information database using a message queue to transfer the original song information to the offline song information table; or, When a preset update cycle is reached, the original song information in the song information data is transferred to the intermediate offline song information table; The original song information is inserted into the offline song information table from the intermediate offline song information table using the batch processing mode of the second stream processing task.

3. The method according to claim 1, characterized in that When the song information in the song information database is updated, obtaining first changed song information in the song information database, and updating data in the offline song information table based on the first changed song information, includes: Subscribing to the first changed song information in the song information database by using a message queue; The first changed song information is updated to the offline song information table using the stream processing mode of the third stream processing task.

4. The method according to claim 1, wherein The method further comprises: In response to the type setting instruction for the offline song information table, the type of the offline song information table is set to Merge on read type.

5. The method according to claim 1, wherein The method further comprises: In response to a primary key setting instruction for the offline song information table, setting the primary key of the offline song information table to the data ID; In response to a pre-aggregation key setting instruction for the offline song information table, setting the pre-aggregation key of the offline song information table to the update time of the data; When updating the first changed song information, if the primary key of the first changed song information conflicts with the primary key of the original song information, the pre-aggregate key of the first changed song information is compared with the pre-aggregate key of the original song information, and the song information with the most recent update time is determined as the final updated song information.

6. The method according to claim 1, characterized in that The method further comprises: In response to the setting instruction for the indexing mode of the offline song information table, the indexing mode of the offline song information table is set to bucket index mode.

7. The method according to claim 3, characterized in that The method further comprises: In response to the setting instruction for the update time of the first changed song information, the trigger time for updating the first changed song information to the offline song information table is set to the checkpoint period of the third stream processing task checking mechanism.

8. The method according to claim 7, characterized in that The method further comprises: In response to the instruction for setting the checkpoint period of the third stream processing task checking mechanism, the checkpoint period of the third stream processing task checking mechanism is set to a custom preset duration.

9. The method according to claim 1, characterized in that The method further comprises: In response to a setting instruction for the first stream processing task attribute, enabling a stream read configuration of the first stream processing task; The stream reading interval of the first stream processing task is set to the checkpoint period of the stream processing task checking mechanism.

10. The method according to claim 1, characterized in that The method further comprises: In response to a setting instruction for the first stream processing task attribute, the stream read attribute of the first stream processing task is set to skip compaction.

11. The method according to claim 1, wherein The method further comprises: Song data is obtained from the cache data storage system using multiple stream processing tasks to perform song task processing on the obtained song data, wherein the song task processing includes filtering target songs, filling in song genre information, filling in song language information, filling in song style information, and filling in song tag information.

12. The method according to claim 11, characterized in that The method further comprises: For a target stream processing task among the multiple stream processing tasks, the data acquired by the target stream processing task is cached using the state of the stream processing task.

13. The method according to claim 11, characterized in that The method further comprises: At least one of song model training or song data statistical analysis is performed using the song data in the offline song information table.

14. The method according to claim 2, characterized in that The first stream processing task includes a stream processing task under the Flink framework, and the message queue includes a Kafka log system.

15. A computer device comprising a processor, characterized in that: When executing the computer program stored in the memory, the processor is used to implement the synchronization method of the song information table as described in any one of claims 1 to 14.

16. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, it is used to implement the synchronization method of the song information table as described in any one of claims 1 to 14.

17. A computer program product having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, it is used to implement the synchronization method of the song information table as described in any one of claims 1 to 14.

Images