Music recommendation model training sample generation method and device, medium and product

Abstract

The invention discloses a music recommendation model training sample generation method and device, a medium and a product, relates to the field of big data, and is applied to a calculation engine connected with a preset data lake, and the calculation engine and the preset data lake support stream batch integration. Comprising the steps of obtaining real-time recommendation feature data of target music; if the first user behavior real-time data of the target music is monitored, generating a first data type label and registering a timer; acquiring non-first user behavior real-time data within the timing duration and generating a second data type label; and if the timing duration ends, splicing the real-time recommendation feature data, the first data type tag and the second data type tag to the same data sample and reporting the data sample to a preset data lake, so that the music recommendation model performs online and / or offline model training by using the data sample. According to the method, the spliced real-time samples are stored by adopting the stream-batch integrated data lake, stream real-time training and batch offline training can be supported at the same time, and stream-batch integration is realized.

Description

Technical Field

[0001] The present invention relates to the field of big data, and in particular to a method, device, medium and product for generating training samples of a music recommendation model. Background Art

[0002] In music recommendation, model training is often the most important part, and the training samples determine the effect of the trained model. Training samples generally refer to offline sample data. Offline samples are generated by Spark by large-scale offline splicing of user behavior offline data stored on HDFS (Hadoop Distributed File System) and feature offline data on HDFS and written to HDFS. User behavior data is reported by the client and stored on HDFS, and feature data is stored on HDFS when the recommendation service calculates recommended content for users. The streaming real-time training sample generation solution based on Flink and Kafka message queues forwards the user behavior data collected by the client to the Kafka message queue, and the recommendation service also sends real-time feature data to the Kafka message queue. Then the Flink computing engine consumes the user behavior and real-time features of the two Kafka message queues for real-time splicing, generates a real-time sample message stream and sends it to the real-time sample Kafka message queue for streaming real-time training.

[0003] However, the real-time sample generation method based on Flink and Kafka can only be used for streaming real-time training, because the final real-time samples will be stored in the Kafka message queue, so model training can only read sample data through real-time consumption. Therefore, if batch offline training is required, it is necessary to build another offline sample generation method based on Spark and HDFS, which will result in double machine costs and manual maintenance costs; and the cost of Kafka message queue is relatively high. Compared with the real-time samples on HDFS, there may be a 10-fold cost difference. Therefore, when it is necessary to retain historical sample data, it may be necessary to store an additional sample data with a longer life cycle on HDFS. Therefore, how to make samples meet the needs of both streaming and batch model training of the Internet real-time recommendation system is a problem to be solved in this field. Summary of the invention

[0004] In view of this, the purpose of the present invention is to provide a method, device, medium and product for generating training samples for a music recommendation model, which can use a data lake that meets the requirements of stream and batch integration to store spliced ​​real-time samples, support real-time streaming training like a message queue for real-time consumption, and can also be directly used as an offline table for batch offline training like HDFS, thus realizing the integration of stream and batch based on the generated sample model training and greatly saving costs. The specific scheme is as follows:

[0005] In a first aspect, the present application provides a method for generating training samples for a music recommendation model, which is applied to a preset computing engine connected to a preset data lake, wherein both the preset computing engine and the preset data lake support stream-batch integration; the method comprises:

[0006] Acquire the real-time recommendation feature data issued by the music recommendation service after the target music is recommended to the target client, and then monitor whether the real-time data of the first user behavior reported by the target client after the first user operation is performed on the recommended target music is obtained;

[0007] If the real-time data of the first user behavior is monitored, a corresponding first data type tag is generated based on the real-time data of the first user behavior, and an operation of registering the current timer is triggered;

[0008] Acquire the non-first-time user behavior real-time data reported by the target client after performing a non-first-time user operation on the target music within the timing duration corresponding to the current timer after registration, and generate corresponding second data type labels based on each non-first-time user behavior real-time data;

[0009] If it is monitored that the timing duration corresponding to the current timer has ended, the real-time recommendation feature data, the first data type label and each of the second data type labels are spliced ​​into the same data sample, and the data sample is reported to the preset data lake, so that the music recommendation model corresponding to the music recommendation service uses the data samples in the preset data lake to perform online and / or offline model training.

[0010] Optionally, before obtaining real-time recommendation feature data and various user behavior real-time data, the following is also included:

[0011] Create a recommendation feature data consumption task and create user behavior real-time data consumption tasks corresponding to various user behavior real-time data according to the data type of the user behavior real-time data;

[0012] Accordingly, the acquisition of real-time recommendation feature data and various user behavior real-time data includes:

[0013] By executing the recommendation feature data consumption task, the real-time recommendation feature data correspondingly sent by the music recommendation service after the music recommendation service recommends the target music to the target client is consumed, thereby obtaining the real-time recommendation feature data;

[0014] By executing each of the user behavior real-time data consumption tasks, the corresponding type of user behavior real-time data reported by the target client for the recommended target music is consumed to obtain various user behavior real-time data.

[0015] Optionally, the preset data lake includes an initial sample data lake and a training sample data lake;

[0016] Accordingly, the step of splicing the real-time recommendation feature data, the first data type label, and each of the second data type labels into the same data sample, and reporting the data sample to the preset data lake includes:

[0017] splicing the real-time recommendation feature data, the first data type label, and each of the second data type labels into the same data sample, and reporting the data sample to the initial sample data lake;

[0018] The data samples in the initial sample data lake are processed to obtain training samples that can be used to directly perform model training, and the training samples are reported to the training sample data lake.

[0019] Optionally, the method for generating training samples for a music recommendation model further includes:

[0020] The acquired real-time recommendation feature data and various user behavior real-time data are added to the end of the corresponding data list respectively, so as to obtain the current data list corresponding to the real-time recommendation feature data and various user behavior real-time data respectively;

[0021] Accordingly, the step of concatenating the real-time recommendation feature data, the first data type label, and each of the second data type labels into the same data sample, and reporting the data sample to the initial sample data lake includes:

[0022] Consume each of the current data lists by executing a data list consumption task, and splice each of the current data lists obtained after consumption into the same data sample, and report the original table of the spliced ​​data sample to the initial sample data lake;

[0023] Each of the current data lists is configured with a corresponding data type label of real-time recommendation feature data or user behavior real-time data.

[0024] Optionally, processing the data samples in the initial sample data lake to obtain training samples that can be used to directly perform model training, and reporting the training samples to the training sample data lake includes:

[0025] The data sample original tables in the initial sample data lake are consumed by executing the data sample original table consumption task, and feature conversion and feature extraction are performed on each of the data sample original tables obtained after consumption to obtain training samples that can be used directly for model training, and the training samples are reported to the training sample data lake.

[0026] Optionally, the method for generating training samples for a music recommendation model further includes:

[0027] Create a target general structure; the target general structure includes a data type field corresponding to the data type of the real-time recommendation feature data and data type fields corresponding to the data types of various user behavior real-time data;

[0028] Using the corresponding data type field in the target general structure to record the data type label corresponding to the real-time recommendation feature data, the first data type label and each second data type label;

[0029] Accordingly, the step of concatenating the real-time recommendation feature data, the first data type label, and each of the second data type labels into the same data sample includes:

[0030] The real-time recommendation feature data and the data type label corresponding to the real-time recommendation feature data recorded in the target general structure, the first data type label and each second data type label are all spliced ​​into the same data sample.

[0031] Optionally, reporting the data sample to the preset data lake includes:

[0032] Determine whether the preset sample reporting conditions are currently met;

[0033] If the preset sample reporting condition is currently met, the data sample is reported to the preset data lake;

[0034] If the preset sample reporting condition is not met at present, the data sample is cached, and during the caching period, if the real-time user behavior data reported by the target client for the target music is obtained again, the operation of registering the next timer is triggered to collect the data type labels corresponding to the real-time user behavior data of a new round, and the data type labels corresponding to the real-time user behavior data collected in this round are merged with the cached data sample to obtain a new data sample, and jump to the step of determining whether the preset sample reporting condition is met at present;

[0035] The preset sample reporting condition is a condition determined based on the relationship between the actual total cache duration of the data sample and a preset cache duration threshold.

[0036] In a second aspect, the present application provides a music recommendation method, comprising:

[0037] Acquire a data sample corresponding to the target music recommended by the music recommendation service from a preset data lake; the data sample is a data sample generated by the aforementioned method;

[0038] Using the data sample, performing online and / or offline model training on a music recommendation model corresponding to the music recommendation service to obtain a corresponding optimized music recommendation model;

[0039] The music recommendation service is used to perform music recommendation operations on the client using the optimized music recommendation model.

[0040] In a third aspect, the present application provides an electronic device, comprising a processor and a memory; wherein the memory is used to store a computer program, and the computer program is loaded and executed by the processor to implement the aforementioned method.

[0041] In a fourth aspect, the present application provides a computer-readable storage medium for storing a computer program, wherein the computer program implements the aforementioned method when executed by a processor.

[0042] In a fifth aspect, the present application provides a computer program product, comprising a computer program / instructions, which implement the aforementioned method when executed by a processor.

[0043] In the present application, after obtaining the real-time recommendation feature data issued by the music recommendation service after recommending the target music to the target client, it is monitored whether the real-time data of the first user behavior reported by the target client after the first user operation on the recommended target music is obtained; if the real-time data of the first user behavior is monitored, the corresponding first data type label is generated based on the real-time data of the first user behavior, and the operation of registering the current timer is triggered, and the real-time data of non-first user behavior reported by the target client after the non-first user operation on the target music is obtained within the timing duration corresponding to the registered current timer, and the corresponding second data type labels are generated based on each non-first user behavior real-time data; thereafter, if it is monitored that the timing duration corresponding to the current timer has ended, the real-time recommendation feature data, the first data type label and the second data type labels are all spliced ​​into the same data sample, and the data sample is reported to the preset data lake, so that the music recommendation model corresponding to the music recommendation service uses the data samples in the preset data lake for online and / or offline model training. It can be seen that the present application can use a data lake that meets the requirements of stream and batch integration to store spliced ​​real-time samples, and can support real-time consumption by streaming real-time training like a message queue. At the same time, it can also be used directly as an offline table for batch offline training like HDFS, avoiding the need to build two sets of samples at the same time when training the model. At the same time, the data collection link is transformed, and the real-time user behavior data reported by the client and the real-time feature data reported by the music recommendation service are put into the data lake, avoiding the high cost when using Kafka message queues. It can also ensure low training costs during offline training, and realize the stream-batch integration of training based on the generated sample model, which greatly saves costs. BRIEF DESCRIPTION OF THE DRAWINGS

[0044] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on the provided drawings without paying creative work.

[0045] Figure 1 A diagram of the system architecture for generating training samples for a music recommendation model disclosed in this application;

[0046] Figure 2 A flow chart of a method for generating training samples for a music recommendation model disclosed in this application;

[0047] Figure 3 This is a flow chart of a real-time data lake sample generation method based on Flink and Iceberg disclosed in this application;

[0048] Figure 4 A schematic diagram of a data lake sample link in a production environment disclosed in this application;

[0049] Figure 5 This is a schematic diagram of Flink's native interval join process;

[0050] Figure 6 A schematic diagram of an efficient multi-stream splicing process disclosed in this application;

[0051] Figure 7 A schematic diagram of a behavior within a delay window disclosed in the present application;

[0052] Figure 8 A schematic diagram of a timer registration mechanism disclosed in this application;

[0053] Fig. 9 A flowchart of an offline sample generation method based on SPARK and HDFS;

[0054] Fig.10 A flowchart of a real-time sample generation method based on Flink and Kafka;

[0055] Fig.11 A flow chart of a data sample reporting method disclosed in this application;

[0056] Fig.12 A sample splicing flow chart disclosed in this application;

[0057] Fig.13 A flow chart of a music recommendation method disclosed in this application;

[0058] Fig.14 A structural diagram of an electronic device provided for this application. DETAILED DESCRIPTION

[0059] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0060] Currently, the real-time sample generation method based on Flink and Kafka can only be used for streaming real-time training. Because the final real-time samples will be stored in the Kafka message queue, model training can only read sample data through real-time consumption. Therefore, if batch offline training is required, it is necessary to build a set of offline sample generation methods based on Spark and HDFS, which will result in double machine costs and manual maintenance costs. The present application can use a data lake that meets the requirements of streaming and batch integration to store spliced ​​real-time samples, which can support real-time consumption by streaming real-time training like a message queue, and can also be directly used as an offline table like HDFS for batch offline training, realizing streaming and batch integration based on the generated sample model training, which greatly saves costs.

[0061] The system framework used in the music recommendation model training sample generation method of this application can be specifically referred to in Figure 1 As shown, it may specifically include: a client 01, a computing engine 02, and a data lake 03 connected to the computing engine 02.

[0062] In this application, the computing engine 02 is used to obtain the real-time recommendation feature data issued by the music recommendation service after recommending the target music to the client 01, and to obtain the real-time user behavior data reported by the client 01 after the user performs the user operation on the recommended target music. Furthermore, the data lake 03 is used to store the data samples generated by the computing engine 02 by splicing the data type labels corresponding to the real-time recommendation feature data and the real-time user behavior data, so that the music recommendation model corresponding to the music recommendation service can use the data samples in the data lake 03 for online and / or offline model training.

[0063] See also Figure 2 As shown, an embodiment of the present invention discloses a method for generating training samples for a music recommendation model, which is applied to a preset computing engine connected to a preset data lake, wherein both the preset computing engine and the preset data lake support stream-batch integration; the method includes:

[0064] Step S11, obtaining the real-time recommendation feature data issued by the music recommendation service after recommending the target music to the target client, and then monitoring whether the real-time data of the first user behavior reported by the target client after the first user operation on the recommended target music is obtained.

[0065] like Figure 3As shown, in this embodiment, the data collection link is modified, and the user behavior reported by the client is put into the real-time user behavior data table of iceberg in real time. At the same time, the recommendation service is modified internally, and the real-time feature data reported in real time is put into the real-time feature data table of iceberg. That is to say, in this embodiment, the real-time recommendation feature data issued by the music recommendation service after the target music is recommended to the target client can be obtained first, and then it is monitored whether the first user behavior real-time data reported by the target client after the first user operation is performed on the recommended target music can be obtained, so as to continue to collect the non-first user behavior real-time data of the client after the first user behavior real-time data is detected. And it can be understood that the above-mentioned real-time recommendation feature data includes but is not limited to the relevant feature data of the target music and the feature data of the target client user, such as: the number of listeners of the target music at this time, the music listening information of the client user, etc.

[0066] And if Figure 4 As shown, in this embodiment, before obtaining real-time recommendation feature data and various user behavior real-time data, it is also possible to create a recommendation feature data consumption task and create various user behavior real-time data consumption tasks corresponding to various user behavior real-time data according to the data type of the user behavior real-time data, so as to use multiple Flink tasks to consume the client reports and recommended distribution. That is to say, when obtaining real-time recommendation feature data and various user behavior real-time data, it is possible to consume the real-time recommendation feature data corresponding to the music recommendation service recommending the target music to the target client by executing the recommendation feature data consumption task, and obtain the real-time recommendation feature data; and to consume the corresponding type of user behavior real-time data reported by the target client for the recommended target music by executing each user behavior real-time data consumption task, and obtain various user behavior real-time data. Among them, the above-mentioned real-time data of user behavior include but are not limited to real-time data corresponding to behaviors such as exposure within the point, click within the point, play within the point, global collection, global blacklisting, etc., among which the relevant operations within the above-mentioned points are the relevant operations performed by the user on a specific music recommendation page, and the above-mentioned global actions are the relevant operations performed by the user in the target client other than the above-mentioned music recommendation page, and the above-mentioned target client is not limited to a specific client, and the information of the same user logged in to each client can be used.

[0067] It should also be pointed out that in this embodiment, when multiple real-time recommendation feature data or real-time user behavior data are obtained, the obtained real-time recommendation feature data and various real-time user behavior data are added to the end of the corresponding data list in each Iceberg data lake table in real time by appending, so as to obtain the current data list corresponding to the real-time recommendation feature data and various real-time user behavior data.

[0068] Step S12: If the real-time data of the first user behavior is monitored, a corresponding first data type tag is generated based on the real-time data of the first user behavior, and an operation of registering the current timer is triggered.

[0069] Step S13: obtain the real-time data of non-first user behavior reported by the target client after performing a non-first user operation on the target music within the timing duration corresponding to the current timer after registration, and generate corresponding second data type labels based on each non-first user behavior real-time data.

[0070] Based on the above steps S12 and S13, in this embodiment, in order to avoid Figure 5 The native Flink intervaljoin technology shown in the figure relies on the interval join results of the previous two streams to perform multi-stream splicing. As the number of streams increases, the machine resources increase and the real-time performance of the data decreases. For example Figure 6 As shown, in this embodiment, a target general structure including data type fields corresponding to the data types of real-time recommendation feature data and data types corresponding to various data types of user behavior real-time data can be created, and then the corresponding data type fields in the target general structure are used to record the data type labels, first data type labels and second data type labels corresponding to the real-time recommendation feature data. In this way, this embodiment can perform stream fusion through unified conversion of the structure, pre-define a general structure, including the required fields of various streams, and there is a field that identifies the type of this stream, so that after converting various different streams, a stream of a unified structure can be obtained. After that, the timer in Flink's KeyedProcessFunction can be used to start the delay task, and after a certain delay, the samples are generated and sent and written into the data lake Iceberg. In this way, the real-time performance of the task will be greatly improved, and the increase in streams will not lead to an increase in data association dependencies. At the same time, the entire task has only one splicing operator, which greatly saves resources.

[0071] And if Figure 7 In the data situation of a music recommendation shown in FIG. 1 , many behaviors can be collected in the delay window of a timer. In order to avoid the sample splitting problem caused by generating a sample for each unified structure, the timer registration mechanism is used to register the timer when receiving the real-time data corresponding to each behavior. Figure 8As shown in the figure, only the first behavior, i.e. the first user behavior real-time data, is registered with the timer. In other words, when non-first user behavior real-time data continues to be received, if there is an old timer, the new behavior will no longer be registered. In this way, even if the user consumes a song multiple times, only one sample will be generated, and all the collected labels can be marked in one sample, which aligns the actual user behavior performance to avoid the user generating multiple samples with different labels for different behaviors of the same music, avoiding the risk of sample splitting and avoiding misleading the normal training of the model.

[0072] Step S14: If it is monitored that the timing duration corresponding to the current timer has ended, the real-time recommendation feature data, the first data type label and each of the second data type labels are spliced ​​into the same data sample, and the data sample is reported to the preset data lake, so that the music recommendation model corresponding to the music recommendation service uses the data samples in the preset data lake to perform online and / or offline model training.

[0073] In this embodiment, if it is monitored that the timing duration corresponding to the current timer has ended, the real-time recommendation feature data, the first data type label, and each second data type label are spliced ​​into the same data sample, and the data sample is reported to the preset data lake, so that the music recommendation model corresponding to the music recommendation service can use the data samples in the preset data lake for online and / or offline model training.

[0074] And, if Figure 4 As shown, when data samples are spliced, each current data list can be consumed by executing a data list consumption task, and each current data list obtained after consumption can be spliced ​​into the same data sample; wherein each of the above current data lists is configured with a corresponding real-time recommendation feature data or a data type label of user behavior real-time data. And it can be understood that the above preset data lake includes an initial sample data lake and a training sample data lake, so that the real-time recommendation feature data, the first data type label and each second data type label can be spliced ​​into the same data sample, and the data sample can be reported to the initial sample data lake, and then the data sample in the initial sample data lake is processed to obtain a training sample that can be used for direct model training, and the training sample is reported to the training sample data lake. It should be pointed out that, as Figure 4 As shown, the samples in the training sample data lake can be stored in the form of triples, and the specific form of the triples is not specifically limited in this embodiment.

[0075] Furthermore, when processing the data samples in the initial sample data lake, the data sample original tables in the initial sample data lake can be consumed by executing the data sample original table consumption task, and feature conversion and feature extraction processing can be performed on each data sample original table obtained after consumption to obtain training samples that can be used directly for model training. That is to say, in this embodiment, a Flink task can be used to consume each Iceberg table to perform multi-stream join splicing to generate an Iceberg sample original table, and finally a Flink task can be used to consume the Iceberg sample original table to perform feature conversion and feature extraction to generate an Iceberg sample table that can be directly used for model training. And accordingly, when performing data sample splicing, such as Figure 6 As shown, the real-time recommendation feature data and the data type label, the first data type label and each second data type label corresponding to the real-time recommendation feature data recorded in the target general structure can be spliced ​​into the same data sample.

[0076] In this way, in this embodiment, the two iceberg tables can be consumed in real time on the Flink real-time computing engine for sample splicing, and the spliced ​​real-time training samples can be put into the iceberg real-time sample table in real time. At this time, since Iceberg is a kind of data that supports batch data reading like offline HDFS data, and can also be consumed in real time like a message queue, given the stream-batch nature of the data, the real-time sample table can be used for batch offline training like HDFS offline samples; it can also be directly consumed by streaming real-time training to produce real-time training models. And it can be understood that the generation of real-time samples of the data lake in this embodiment is not limited to online splicing by the Flink computing engine. The stream-batch integrated sample generation scheme that is spliced ​​and output to the data lake by other computing engines should also be protected by this scheme. At the same time, the data lake is not limited to Iceberg. For example, Hudi, Paimon and other data lakes can also be used as alternatives. And because Iceberg is just a data format, its underlying storage can use HDFS, so the storage cost of the Iceberg data lake sample is consistent with the storage cost of the HDFS offline sample, which effectively reduces the cost of model training, so that when the cost does not increase, compared with Fig. 9 , Fig.10 The sample generation method shown in the figure migrates the offline training sample generation method based on Spark and HDFS to the real-time training sample generation method based on Flink and Iceberg, which improves the real-time performance of model training and significantly improves the effect of the music recommendation system.

[0077] Based on the previous embodiment, it can be seen that the present application can migrate the offline training sample generation method based on Spark and HDFS to the real-time training sample generation method based on Flink and Iceberg. Next, the process of sample splicing and sample reporting will be described in detail in this embodiment. Fig.11 As shown, the embodiment of the present application discloses a data sample reporting method, including:

[0078] Step S21: concatenate the real-time recommended feature data, the first data type label, and each second data type label into the same data sample, and determine whether the preset sample reporting condition is currently met.

[0079] In this embodiment, after the real-time recommendation feature data, the first data type label, and each second data type label are spliced ​​into the same data sample, it can be determined whether the preset sample reporting condition is currently met. The above-mentioned preset sample reporting condition is a condition determined based on the relationship between the actual total cache time of the data sample and the preset cache time threshold.

[0080] Step S22: If the preset sample reporting condition is currently met, the data sample is reported to the preset data lake.

[0081] In this embodiment, if the preset sample reporting conditions are currently met, the data sample is reported to the preset data lake. Fig.12 As shown, the preset cache duration threshold of the data sample can be set to 12 hours. Then, when the actual total cache duration of the data sample reaches 12 hours, the data sample obtained within this time is reported to the data lake.

[0082] Step S23: If the preset sample reporting condition is not met at present, the data sample is cached, and during the caching period, if the real-time user behavior data reported by the target client for the target music is obtained again, the operation of registering the next timer is triggered to collect the data type labels corresponding to the new round of each user behavior real-time data, and merge the data type labels corresponding to the real-time user behavior data collected in this round with the cached data samples to obtain new data samples, and jump again to the step of determining whether the preset sample reporting condition is met at present.

[0083] In this embodiment, if the preset sample reporting condition is not met at present, the data sample is cached, and during the cache period, if the real-time user behavior data reported by the target client for the target music is obtained again, the operation of registering the next timer is triggered to collect the data type labels corresponding to the new round of real-time user behavior data, and merge the data type labels corresponding to the real-time user behavior data collected in this round with the cached data samples to obtain new data samples, and jump again to the step of determining whether the preset sample reporting condition is met at present. In other words, if Fig.12 As shown, after the sample generation starts, if the user's first real-time data is received, a timer is registered for the data, and the real-time data of the user's behavior is obtained within the corresponding timing duration, and the corresponding data samples are generated by splicing after the corresponding labels are generated. If the 12h duration threshold is not reached at this time, the data sample generated at this time is reported, and the label corresponding to the data sample is cached, so that when the user's behavior real-time data is received again after the above timing duration ends, the next round of sample duration is started, and in the next round of sample generation process, the previously cached label is brought into the generation of the data sample to obtain a new data sample corresponding to the current recommended music. It should also be pointed out that if multiple samples of the same target music are received in the data lake within the preset cache duration threshold, the last reported sample is saved and other samples are deleted to avoid sample splitting.

[0084] Based on the above technical solution, the sample splitting problem is effectively avoided in this embodiment. Through the timer registration mechanism, the timer is not registered when the real-time data corresponding to each behavior is received. Only the first behavior, that is, the first user behavior real-time data is received, is registered. Even if the user consumes a song multiple times, only one sample will be generated, and all the collected labels can be marked in one sample, which aligns the actual user behavior performance to avoid the user generating multiple samples with different labels for different behaviors of the same music, avoiding the risk of sample splitting and avoiding misleading the normal training of the model.

[0085] See also Fig.13 As shown, the embodiment of the present application also discloses a music recommendation method, including:

[0086] Step S31: Acquire a data sample corresponding to the target music recommended by the music recommendation service from a preset data lake; the data sample is a data sample generated by the method described in the above embodiment.

[0087] Step S32: Use the data sample to perform online and / or offline model training on the music recommendation model corresponding to the music recommendation service to obtain a corresponding optimized music recommendation model.

[0088] Step S33: Perform music recommendation operations on the client through the music recommendation service and using the optimized music recommendation model.

[0089] In this embodiment, a data sample generated by the data sample generation method in the above embodiment and corresponding to the target music recommended by the music recommendation service can be obtained from the preset data lake, and the music recommendation model corresponding to the music recommendation service can be trained online and / or offline using the data sample to obtain a corresponding optimized music recommendation model, so as to perform music recommendation operations to the client through the music recommendation service and using the optimized music recommendation model, such as Figure 3 As shown in the figure, a data lake that supports streaming and batch integration is used to store spliced ​​real-time samples. It can support real-time consumption by streaming real-time training like a message queue, and can also be used directly as an offline table for batch offline training like HDFS. The generated data samples meet the training requirements of both models at the same time, realizing streaming and batch integration and greatly saving costs.

[0090] Furthermore, the present application also discloses an electronic device. Fig.14 This is a structural diagram of an electronic device 10 according to an exemplary embodiment. The content in the diagram cannot be regarded as any limitation on the scope of use of the present application.

[0091] Fig.14 The present invention provides a schematic diagram of the structure of an electronic device 10 provided in an embodiment of the present application. The electronic device 10 may specifically include: at least one processor 11, at least one memory 12, a power supply 13, a communication interface 14, an input / output interface 15, and a communication bus 16. The memory 12 is used to store a computer program, which is loaded and executed by the processor 11 to implement the relevant steps in the method disclosed in any of the aforementioned embodiments. In addition, the electronic device 10 in this embodiment may specifically be an electronic computer.

[0092] In this embodiment, the power supply 13 is used to provide working voltage for each hardware device on the electronic device 10; the communication interface 14 can create a data transmission channel between the electronic device 10 and the external device, and the communication protocol it follows is any communication protocol that can be applied to the technical solution of the present application, and is not specifically limited here; the input and output interface 15 is used to obtain external input data or output data to the outside world, and its specific interface type can be selected according to specific application needs and is not specifically limited here.

[0093] In addition, the memory 12, as a carrier for storing resources, can be a read-only memory, a random access memory, a disk or an optical disk, etc. The resources stored thereon can include an operating system 121, a computer program 122, etc., and the storage method can be temporary storage or permanent storage.

[0094] The operating system 121 is used to manage and control the hardware devices on the electronic device 10 and the computer program 122, which can be Windows Server, Netware, Unix, Linux, etc. In addition to including a computer program that can be used to complete the method performed by the electronic device 10 disclosed in any of the aforementioned embodiments, the computer program 122 can further include a computer program that can be used to complete other specific tasks.

[0095] Furthermore, the present application also discloses a computer-readable storage medium for storing a computer program; wherein the computer program, when executed by a processor, implements the aforementioned disclosed method. The specific steps of the method can refer to the corresponding contents disclosed in the aforementioned embodiments, and will not be repeated here.

[0096] Furthermore, the present application also discloses a computer program product, including a computer program / instruction, wherein the computer program / instruction implements the aforementioned disclosed method when executed by a processor. The specific steps of the method can refer to the corresponding contents disclosed in the aforementioned embodiment, and will not be repeated here.

[0097] In this specification, each embodiment is described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the embodiments can be referred to each other. For the device disclosed in the embodiment, since it corresponds to the method disclosed in the embodiment, the description is relatively simple, and the relevant parts can be referred to the method part.

[0098] Professionals may further appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of the two. In order to clearly illustrate the interchangeability of hardware and software, the composition and steps of each example have been generally described in the above description according to function. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professionals and technicians may use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.

[0099] The steps of the method or algorithm described in conjunction with the embodiments disclosed herein may be implemented directly using hardware, a software module executed by a processor, or a combination of the two. The software module may be placed in a random access memory (RAM), a memory, a read-only memory (ROM), an electrically programmable ROM, an electrically erasable programmable ROM, a register, a hard disk, a removable disk, a CD-ROM, or any other form of storage medium known in the art.

[0100] Finally, it should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the presence of other identical elements in the process, method, article or device including the elements.

[0101] The technical solution provided by the present application is introduced in detail above. Specific examples are used in this article to illustrate the principles and implementation methods of the present application. The description of the above embodiments is only used to help understand the method of the present application and its core idea. At the same time, for general technicians in this field, according to the idea of ​​the present application, there will be changes in the specific implementation methods and application scope. In summary, the content of this specification should not be understood as a limitation on the present application.

Claims

1. A method for generating training samples for a music recommendation model, characterized in that: A preset computing engine connected to a preset data lake is applied, wherein both the preset computing engine and the preset data lake support batch-stream integration; the method comprises: Acquire the real-time recommendation feature data issued by the music recommendation service after recommending the target music to the target client, and then monitor whether the real-time data of the first user behavior reported by the target client after the target client performs the first user operation on the recommended target music is obtained; If the real-time data of the first user behavior is monitored, a corresponding first data type tag is generated based on the real-time data of the first user behavior, and an operation of registering the current timer is triggered; Acquire the non-first-time user behavior real-time data reported by the target client after performing a non-first-time user operation on the target music within the timing duration corresponding to the current timer after registration, and generate corresponding second data type labels based on each non-first-time user behavior real-time data; If it is monitored that the timing duration corresponding to the current timer has ended, the real-time recommendation feature data, the first data type label and each of the second data type labels are spliced ​​into the same data sample, and the data sample is reported to the preset data lake, so that the music recommendation model corresponding to the music recommendation service uses the data samples in the preset data lake to perform online and / or offline model training.

2. The method for generating music recommendation model training samples according to claim 1, characterized in that: Before obtaining real-time recommendation feature data and various user behavior real-time data, it also includes: Create a recommendation feature data consumption task and create user behavior real-time data consumption tasks corresponding to various user behavior real-time data according to the data type of the user behavior real-time data; Accordingly, the acquisition of real-time recommendation feature data and various user behavior real-time data includes: By executing the recommendation feature data consumption task, the real-time recommendation feature data correspondingly sent by the music recommendation service after the music recommendation service recommends the target music to the target client is consumed, thereby obtaining the real-time recommendation feature data; By executing each of the user behavior real-time data consumption tasks, the corresponding type of user behavior real-time data reported by the target client for the recommended target music is consumed to obtain various user behavior real-time data.

3. The method for generating music recommendation model training samples according to claim 1, characterized in that: The preset data lake includes an initial sample data lake and a training sample data lake; Accordingly, the step of splicing the real-time recommendation feature data, the first data type label, and each of the second data type labels into the same data sample, and reporting the data sample to the preset data lake includes: splicing the real-time recommendation feature data, the first data type label, and each of the second data type labels into the same data sample, and reporting the data sample to the initial sample data lake; The data samples in the initial sample data lake are processed to obtain training samples that can be used to directly perform model training, and the training samples are reported to the training sample data lake.

4. The method for generating music recommendation model training samples according to claim 3, characterized in that: Also includes: The acquired real-time recommendation feature data and various user behavior real-time data are added to the end of the corresponding data list respectively, so as to obtain the current data list corresponding to the real-time recommendation feature data and various user behavior real-time data respectively; Accordingly, the step of splicing the real-time recommendation feature data, the first data type label, and each of the second data type labels into the same data sample, and reporting the data sample to the initial sample data lake includes: Consume each of the current data lists by executing a data list consumption task, and splice each of the current data lists obtained after consumption into the same data sample, and report the original table of the spliced ​​data sample to the initial sample data lake; Each of the current data lists is configured with a corresponding data type label of real-time recommendation feature data or user behavior real-time data.

5. The method for generating music recommendation model training samples according to claim 4, characterized in that: The processing of the data samples in the initial sample data lake to obtain training samples that can be used to directly perform model training, and reporting the training samples to the training sample data lake, includes: The data sample original tables in the initial sample data lake are consumed by executing the data sample original table consumption task, and feature conversion and feature extraction are performed on each of the data sample original tables obtained after consumption to obtain training samples that can be used directly for model training, and the training samples are reported to the training sample data lake.

6. The method for generating music recommendation model training samples according to claim 1, characterized in that: Also includes: Create a target general structure; the target general structure includes a data type field corresponding to the data type of the real-time recommendation feature data and data type fields corresponding to the data types of various user behavior real-time data; Using the corresponding data type field in the target general structure to record the data type label corresponding to the real-time recommendation feature data, the first data type label and each second data type label; Accordingly, the step of concatenating the real-time recommendation feature data, the first data type label, and each of the second data type labels into the same data sample includes: The real-time recommendation feature data and the data type label corresponding to the real-time recommendation feature data recorded in the target general structure, the first data type label and each second data type label are all spliced ​​into the same data sample.

7. The method for generating music recommendation model training samples according to any one of claims 1 to 6, characterized in that: Reporting the data sample to the preset data lake includes: Determine whether the preset sample reporting conditions are currently met; If the preset sample reporting condition is currently met, the data sample is reported to the preset data lake; If the preset sample reporting condition is not met at present, the data sample is cached, and during the caching period, if the real-time user behavior data reported by the target client for the target music is obtained again, the operation of registering the next timer is triggered to collect the data type labels corresponding to the real-time user behavior data of a new round, and the data type labels corresponding to the real-time user behavior data collected in this round are merged with the cached data sample to obtain a new data sample, and jump to the step of determining whether the preset sample reporting condition is met at present; The preset sample reporting condition is a condition determined based on the relationship between the actual total cache duration of the data sample and a preset cache duration threshold.

8. A music recommendation method, characterized in that: include: Obtaining data samples corresponding to target music recommended by the music recommendation service from a preset data lake; The data sample is a data sample generated by the method according to any one of claims 1 to 7; Using the data sample, performing online and / or offline model training on a music recommendation model corresponding to the music recommendation service to obtain a corresponding optimized music recommendation model; The music recommendation service is used to perform music recommendation operations on the client using the optimized music recommendation model.

9. An electronic device, characterized in that: The electronic device comprises a processor and a memory; wherein the memory is used to store a computer program, and the computer program is loaded and executed by the processor to implement the method according to any one of claims 1 to 8.

10. A computer-readable storage medium, characterized in that: Used to store a computer program, which implements the method according to any one of claims 1 to 8 when executed by a processor.

11. A computer program product comprising a computer program / instructions, characterized in that When the computer program / instructions are executed by a processor, the method according to any one of claims 1 to 8 is implemented.

Images