Data set construction method and device, electronic equipment and storage medium

Abstract

The present disclosure relates to a data set construction method and device, electronic equipment and storage medium. The method comprises: acquiring multi-source IPTV service log information; performing unified processing on different program identification information corresponding to the same program in different program watching logs and program metadata respectively to obtain processed program watching logs and processed program metadata; cleaning the processed program watching logs according to set log cleaning constraint information to obtain target program watching logs; performing irreversible desensitization processing on user identification information in the target program watching logs and user identification information in subscription logs respectively to obtain processed target program watching logs and processed subscription logs; generating a user-program interaction data set according to the processed target program watching logs, generating a paid user data set according to the processed subscription logs, and generating a program data set according to the processed program metadata.

Description

Technical Field

[0001] This disclosure relates to the field of data processing technology, and more specifically, to a dataset construction method, a dataset construction apparatus, an electronic device, and a computer-readable storage medium. Background Technology

[0002] With the development of television services, broadcast television platforms continuously generate massive amounts of behavioral logs during user viewing. These logs reflect users' program selection and payment conversion behaviors in the home television setting, serving as an important data foundation for building broadcast television content recommendation systems and conducting research on recommendation algorithms.

[0003] Related technologies typically use publicly available recommendation datasets or online video datasets as experimental benchmarks for TV recommendation algorithms. These approaches generally include: obtaining user, content, and interaction behavior tables from existing public datasets; using movies and short videos as recommendation targets; converting behaviors such as ratings, clicks, likes, comments, and plays into tags required by the recommendation model; and conducting offline evaluation using collaborative filtering, matrix factorization, deep ranking models, or multi-task models. However, this approach suffers from problems such as insufficient scenario realism, inadequate coverage of business types, insufficient multi-task behavior, insufficient temporal context, and difficulty in reflecting true sparsity. This technology does not build a dataset based on the aforementioned massive behavioral logs, resulting in insufficient scenario realism, inadequate coverage of business types, insufficient multi-task behavior, insufficient temporal context, and difficulty in reflecting true sparsity, leading to poor dataset accuracy. Summary of the Invention

[0004] One objective of this disclosure is to provide a new technical solution for a dataset construction method.

[0005] According to a first aspect of this disclosure, a method for constructing a dataset is provided, comprising: Obtain multi-source IPTV service log information, wherein the multi-source IPTV service log information includes program viewing logs, subscription logs and program metadata generated based on each service system. The program viewing logs include user identification information, program identification information, program name, program viewing start time, program viewing end time, service system type and program playback platform. The subscription logs include user identification information and subscription type information. The program metadata includes program identification information, program name and program type information. The unified processing is performed on different program identification information corresponding to the same program in different program viewing logs to obtain the processed program viewing logs; and the unified processing is performed on different program identification information corresponding to the same program in the program metadata to obtain the processed program metadata. Based on the set log cleaning constraints, the processed program viewing logs are cleaned to obtain the target program viewing logs. Irreversible desensitization processing is performed on the user identification information in the target program viewing log and the user identification information in the order log, respectively, to obtain the processed target program viewing log and the processed order log. A user-program interaction dataset is generated based on the processed target program viewing logs, a paid user dataset is generated based on the processed subscription logs, and a program dataset is generated based on the processed program metadata.

[0006] Optionally, the process of performing unified processing on different program identifier information for the same program in different program viewing logs to obtain processed program viewing logs, and the process of performing unified processing on different program identifier information for the same program in program metadata to obtain processed program metadata, include: Obtain the program association information corresponding to each program identifier, wherein the program association information includes program name, program type information, program broadcast time, total number of episodes, program release year, and program content provider code information; Based on the program association information corresponding to each program identifier, different program identifiers corresponding to the same program are identified from multiple program viewing logs, and different program identifiers corresponding to the same program are identified from multiple program metadata. The program viewing log is processed by performing unified processing on the different program identifiers that correspond to the same program in the program viewing log. The program metadata is also processed by performing unified processing on the different program identifiers that correspond to the same program in the program metadata.

[0007] Optionally, the set log cleaning constraint information includes the program viewing end time being later than the program viewing start time, the effective range of program viewing duration, the user activity threshold, and the judgment rules for the same user's repeated viewing records of the same program.

[0008] Optionally, the user identification information in the target program viewing log and the user identification information in the subscription log are subjected to irreversible desensitization processing to obtain the processed target program viewing log and the processed subscription log, including: For the same user identifier information in different target program viewing logs and different subscription logs, generate random salt values; Based on the set hash function, each user identifier and its corresponding random salt value are processed to obtain the corresponding de-identified user identifier. Replace the original user identifier information with the anonymized user identifier information to obtain the processed target program viewing log and the processed subscription log.

[0009] Optionally, the method further includes: Based on the program viewing start time and program viewing end time in each processed target program viewing log, determine the viewing interest tags of the corresponding user for the corresponding program, and add the viewing interest tags of the corresponding user for the corresponding program to the corresponding processed target program viewing log; and / or, Based on all processed target program viewing logs for the same user, determine the viewing quality tag for the corresponding user, and add the viewing quality tag for the corresponding user to the processed target program viewing logs for the same user; and / or, Based on the order type information in each processed order log, determine the corresponding user's order tag and add the corresponding user's order tag to the corresponding processed order log.

[0010] Optionally, the method further includes: Based on the program's public knowledge base, extended data for each program is obtained, including program sub-category information, program synopsis, lead actor information, director information, release date, and release region. Based on the processed program metadata and corresponding extended data, a program dataset is generated.

[0011] According to a second aspect of this disclosure, a dataset construction apparatus is provided, comprising: The acquisition module is used to acquire multi-source IPTV service log information, wherein the multi-source IPTV service log information includes program viewing logs, subscription logs and program metadata generated based on each service system. The program viewing logs include user identification information, program identification information, program name, program viewing start time, program viewing end time, service system type and program playback platform. The subscription logs include user identification information and subscription type information. The program metadata includes program identification information, program name and program type information. The unified processing module is used to perform unified processing based on different program identification information corresponding to the same program in different program viewing logs to obtain the processed program viewing logs, and to perform unified processing based on different program identification information corresponding to the same program in program metadata to obtain the processed program metadata. The cleaning module is used to clean the processed program viewing logs according to the set log cleaning constraint information to obtain the target program viewing logs. The desensitization processing module is used to perform irreversible desensitization processing on the user identification information in the target program viewing log and the user identification information in the order log, respectively, to obtain the processed target program viewing log and the processed order log. The dataset generation module is used to generate a user-program interaction dataset based on the processed target program viewing logs, a paid user dataset based on the processed subscription logs, and a program dataset based on the processed program metadata.

[0012] According to a third aspect of this disclosure, an electronic device is provided, including a memory and a processor, the memory storing a computer program for controlling the processor to operate in order to perform the method according to any one of the first aspects.

[0013] According to a fourth aspect of this disclosure, a computer-readable storage medium is provided, on which a computer program is stored, the computer program implementing the method according to any one of the first aspects when executed by a processor.

[0014] This disclosure provides a dataset construction method. By acquiring multi-source IPTV service log information, it performs unified processing on different program identification information corresponding to the same program in different service systems, cleans the processed program viewing logs according to the set log cleaning constraints, and performs irreversible desensitization processing on user identification information to generate user and program interaction datasets, paid user datasets, and program datasets. This improves the accuracy of dataset construction and can also solve the problems of inconsistent multi-source logs, a large amount of abnormal data, and high risk of privacy leakage, thereby improving the authenticity, integrity, and security of the dataset.

[0015] The features and advantages of the embodiments of this specification will become clear from the following detailed description of exemplary embodiments with reference to the accompanying drawings. Attached Figure Description

[0016] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments of the present disclosure and, together with the description, serve to explain the principles of the disclosure.

[0017] Figure 1 This is a schematic diagram of the structure of an electronic device to which the methods of the embodiments of this disclosure can be applied.

[0018] Figure 2 This is a flowchart illustrating a dataset construction method based on some embodiments.

[0019] Figure 3 This is a schematic block diagram of a dataset construction device based on some embodiments.

[0020] Figure 4 This is a block schematic diagram of an electronic device according to some embodiments. Detailed Implementation

[0021] Various exemplary embodiments of the present disclosure will now be described in detail with reference to the accompanying drawings. It should be noted that, unless otherwise specifically stated, the relative arrangement, numerical expressions, and values ​​of the components and steps set forth in these embodiments do not limit the scope of the present disclosure.

[0022] The following description of at least one exemplary embodiment is merely illustrative and is not intended to limit the scope of this disclosure or its application or use.

[0023] Techniques, methods, and equipment known to those skilled in the art may not be discussed in detail, but where appropriate, such techniques, methods, and equipment should be considered part of the specification.

[0024] In all the examples shown and discussed herein, any specific values ​​should be interpreted as merely exemplary and not as limitations. Therefore, other examples of exemplary embodiments may have different values.

[0025] It should be noted that similar labels and letters in the following figures indicate similar items; therefore, once an item is defined in one figure, it does not need to be discussed further in subsequent figures.

[0026] This disclosure relates to a dataset construction scheme. Figure 1 This is a schematic diagram of the structure of an electronic device capable of applying the dataset construction method of the embodiments of this disclosure. The electronic device 1000 is a device capable of running computer programs. These computer programs can be local applications installed on the electronic device, or web applications, lightweight applications, or mini-programs, etc., and are not limited thereto. The electronic device 1000 can be a mobile phone, tablet computer, server, etc., and is not limited thereto.

[0027] like Figure 1 As shown, the electronic device 1000 may include a processor 1100, a memory 1200, an interface device 1300, a communication device 1400, etc. Figure 1 The hardware configuration shown is illustrative only and is not intended to limit this disclosure, its application, or its use.

[0028] The processor 1100 is used to execute computer programs, which can be written using instruction sets of architectures such as x86, Arm, RISC, MIPS, and SSE. The memory 1200 includes, for example, ROM (Read-Only Memory), RAM (Random Access Memory), and non-volatile memory such as a hard disk. The interface device 1300 includes, for example, a USB interface, a network cable interface, and a headphone jack. The communication device 1400 is capable of wired or wireless communication. The communication device 1400 may include at least one short-range communication module, such as any module for short-range wireless communication based on short-range wireless communication protocols such as Hilink, WiFi (IEEE 802.11), Mesh, Bluetooth, ZigBee, Thread, Z-Wave, NFC, UWB, and LiFi. The communication device 1400 may also include a long-range communication module, such as any module for WLAN, GPRS, or 2G / 3G / 4G / 5G long-range communication.

[0029] To address the aforementioned technical issues, this disclosure provides a dataset construction method. By acquiring multi-source IPTV service log information, it performs unified processing on different program identification information corresponding to the same program in different service systems, cleans the processed program viewing logs according to set log cleaning constraints, and performs irreversible desensitization processing on user identification information, generating user-program interaction datasets, paid user datasets, and program datasets. This improves the accuracy of dataset construction and also solves the problems of inconsistent multi-source logs, numerous abnormal data, and high risk of privacy leakage, thereby enhancing the authenticity, integrity, and security of the dataset.

[0030] The following combination Figure 1 The electronic device shown illustrates the dataset construction method provided in this disclosure.

[0031] <First Embodiment> Figure 2 This is a flowchart illustrating a dataset construction method based on some embodiments. This method can be applied to data processing servers or offline big data processing environments of IPTV (Internet Protocol Television) operating platforms, OTT (OverThe Top) video platforms, smart large-screen service platforms, and broadcast television content service platforms to construct standardized datasets for content recommendation research.

[0032] The dataset was constructed by [method / method]. Figure 1 The illustrated electronic device 1000 is implemented. For example... Figure 2 As shown, the dataset construction method of this embodiment may include the following steps S210 to S250.

[0033] Step S210: Obtain multi-source IPTV service log information, which includes program viewing logs, subscription logs, and program metadata generated by each service system. The program viewing logs include user identification information, program identification information, program name, program viewing start time, program viewing end time, service system type, and program playback platform. The subscription logs include user identification information and subscription type information. The program metadata includes program identification information, program name, and program type information.

[0034] Multi-source IPTV service log information is real service data collected from the IPTV operation platform over a continuous period of time.

[0035] The program viewing log can include live viewing logs, on-demand viewing logs, and replay viewing logs.

[0036] To avoid impacting online systems, the collection of multi-source IPTV service log information can be conducted offline in batches, exporting data from the service database or log repository on a daily, weekly, or monthly basis. By collecting real service data over continuous time periods, complete user viewing behavior patterns and program content information can be obtained, providing a data foundation for subsequent unified processing, cleaning, and anonymization.

[0037] Step S220: Perform unified processing based on different program identification information corresponding to the same program in different program viewing logs to obtain processed program viewing logs; and perform unified processing based on different program identification information corresponding to the same program in program metadata to obtain processed program metadata.

[0038] The log fields, time granularity, program encoding rules, and reporting mechanisms generated by different business systems (e.g., live streaming system, video-on-demand system, and replay system) are not entirely consistent, which results in different program identification information corresponding to the same program in different program viewing logs.

[0039] In some examples, step S220 includes steps S2211 to S2213.

[0040] Step S2211: Obtain the program association information corresponding to each program identifier information. The program association information includes program name, program type information, program broadcast time, total number of episodes, program release year, and program content provider code information.

[0041] Step S2212: Based on the program association information corresponding to each program identifier information, identify different program identifier information corresponding to the same program from multiple program viewing logs, and identify different program identifier information corresponding to the same program from multiple program metadata.

[0042] The following matching rules apply: if two program identifiers have the same program name, program type information, program broadcast time, total number of episodes, program release year, and program content provider code information, then they are determined to be the same program.

[0043] Step S2213: Perform unified processing on the identified different program identifier information corresponding to the same program in the program viewing log to obtain the processed program viewing log, and perform unified processing on the identified different program identifier information corresponding to the same program in the program metadata to obtain the processed program metadata.

[0044] Specifically, a unique program identifier (Program_ID) can be generated for different program identifiers corresponding to the same program, and the original program identifiers can be replaced with the unique program identifier in the processed program viewing logs and program metadata. This ensures that viewing records of the same program in different business systems can be associated with the same program entity.

[0045] By aligning entities through program association information, the problem of multiple encodings for the same program across different business systems can be solved. This ensures that viewing behaviors generated by live streaming, on-demand, and replay systems all fall within the same user-program interaction space, achieving consistent representation of content entities. Unified processing provides a standardized program identifier foundation for subsequent data cleaning, de-identification, and dataset generation, improving dataset usability and the accuracy of recommendation algorithms.

[0046] Step S230: Clean the processed program viewing logs according to the set log cleaning constraints to obtain the target program viewing logs.

[0047] This step involves cleaning the processed program viewing logs to remove illegal samples that violate physical constraints and business logic.

[0048] The log cleaning constraints include the following: the program viewing end time is later than the program viewing start time; the effective range of program viewing duration; the user activity threshold; and the rules for determining repeated viewing records of the same user for the same program.

[0049] In some examples, the log cleaning constraints may include the program viewing end time being later than the program viewing start time, the effective range of program viewing duration, user activity thresholds, and rules for determining repeated viewing records of the same user for the same program.

[0050] If the start time of a viewing session is later than the end time, it is considered an illegal record and is removed. This type of anomaly is usually caused by a system clock error, reversed log write order, or front-end reporting errors.

[0051] The effective viewing time range for the program is . Used to filter out obvious accidental touches, automatic preloading, or invalid playback; can be set to 5 seconds. The limit can be set based on program duration, business rules, or the platform's maximum session duration, or no limit can be set when long-term viewing for research needs are required. For example, it can be set... Set to 14400 seconds to filter out abnormally long periods of idle time.

[0052] Based on user activity thresholds, extremely inactive or occasionally active accounts are removed. For example, to preserve the research value of cold start and low-activity users, the user activity threshold can be determined based on a set number of program views per month (e.g., 3 times). When the number of program views per month is less than the set number, the corresponding processed program viewing logs are removed.

[0053] The rule for determining duplicate viewing records of the same program by the same user is as follows: if the same user, the same program, the difference between the start time and the start time is less than a set threshold, and the end time and duration of the viewing are the same or very close, then it is determined to be a duplicate report of the same processed program viewing log. Only the earliest or most complete processed program viewing log will be retained.

[0054] In these examples, multi-dimensional cleaning constraints can effectively eliminate abnormal situations such as start time being later than end time, negative viewing duration, excessively short viewing duration, duplicate records caused by network retransmission, and non-active interactions triggered by the system. At the same time, it can retain real viewing behavior, fast channel switching behavior, and long-tail program behavior to the maximum extent, so that the dataset is both usable and retains the real sparse characteristics of broadcast television scenarios, thereby improving the reliability of user interest modeling and experimental results.

[0055] Step S240: Irreversible desensitization processing is performed on the user identification information in the target program viewing log and the user identification information in the subscription log, respectively, to obtain the processed target program viewing log and the processed subscription log.

[0056] In some examples, step S240 includes steps S2411 to S2413 as follows.

[0057] Step S2411: Generate a random salt value for the same user identification information in different target program viewing logs and different subscription logs.

[0058] Specifically, a random salt value can be generated for each unique user identifier. The random salt value can be a random string of 16 or 32 bytes in length, generated using a cryptographically secure random number generator. The same user identifier uses the same salt value across all log records to ensure consistency of the anonymized user identifier across different records.

[0059] Step S2412: Based on the set hash function, process each user identifier information and its corresponding random salt value to obtain the corresponding de-identified user identifier information.

[0060] Specifically, SHA-256, SHA-512, or other cryptographic one-way hash functions can be used to perform hash operations on the concatenation result of each user identifier and its corresponding random salt value. For example, for each user identifier (uid) and random salt value (salt), the following calculation can be performed: Obtain the anonymized user identification information .

[0061] Step S2413: Replace the corresponding un-de-identified user identification information with the de-identified user identification information to obtain the processed target program viewing log and the processed subscription log respectively.

[0062] Specifically, all target program viewing logs and subscription logs are iterated through, and the original user identifier in each record is replaced with the corresponding de-identified user identifier. The replaced records retain other fields unchanged, forming the processed target program viewing logs and processed subscription logs.

[0063] To resist reverse enumeration and dictionary attacks, the following measures can be taken: First, random salt values ​​should not be released with public datasets but should be stored or destroyed offline by the data controller. Second, direct identification fields such as user mobile phone numbers, account numbers, set-top box numbers, and detailed addresses should not be output. Third, fields that may form quasi-identifier combinations should be generalized; for example, precise timestamps should be coarse-grained, generalizing second-level timestamps to hourly or daily-level timestamps. Fourth, extremely low-frequency order combinations or rare behavior combinations should be bucketed or merged to reduce the risk of re-identification.

[0064] Through the above processing, the anonymized user identifier can maintain consistency of the same user across records, supporting sequence modeling, activity analysis, and user profile construction. At the same time, the original identity identifier does not exist in the public data, making it difficult to recover the real user identity from the anonymous identifier, thus achieving a balance between data availability and privacy security.

[0065] Step S250: Generate a user and program interaction dataset based on the processed target program viewing logs, generate a paid user dataset based on the processed subscription logs, and generate a program dataset based on the processed program metadata.

[0066] Table 1 lists the fields included in each record of the user-program interaction dataset. Table 2 lists the fields included in each record of the program dataset.

[0067] Table 1

[0068] Table 2

[0069] In some examples, the method further includes: determining the viewing interest tag of the corresponding user for watching the corresponding program based on the program viewing start time and program viewing end time in each processed target program viewing log, and adding the viewing interest tag of the corresponding user for watching the corresponding program to the corresponding processed target program viewing log; and / or determining the viewing quality tag of the corresponding user based on all processed target program viewing logs corresponding to the same user, and adding the viewing quality tag of the corresponding user to the processed target program viewing log corresponding to the same user; and / or determining the subscription tag of the corresponding user based on the subscription type information in each processed subscription log, and adding the subscription tag of the corresponding user to the corresponding processed subscription log.

[0070] Specifically, we can set the viewing duration as d and the viewing interest threshold as... In one example, Seconds. If Then view interest tags This indicates that the user has a strong interest in watching the program. If... Then view interest tags This indicates that the user is quickly browsing, changing channels, or has not developed a stable interest. This threshold can be configured according to the business scenario; for example, a shorter threshold can be set for short video programs, and a longer threshold can be set for movie programs.

[0071] Viewing quality tags can be determined based on information such as viewing duration, standard program duration, and start and end times. Viewing quality tags can include extended fields such as viewing completion rate, early exit indicator, viewing progress percentile, weekend viewing indicator, and prime-time viewing indicator. For example, viewing completion rate can be defined as... Where d is the viewing duration, This refers to the standard program duration. An early departure flag can be defined as... Early departure is marked as 1, otherwise 0. Weekend viewing is determined by the weekday attribute of the viewing start time: 1 for Saturday or Sunday, 0 otherwise. Prime time viewing is determined by the hour attribute of the viewing start time: 1 for 7 PM to 10 PM, 0 otherwise. These extended fields support finer-grained retention quality modeling.

[0072] Specifically, ordering labels This can be determined based on the order type. If the order type (Order_type) is 0, then the order label... This indicates that the user only subscribes to the basic plan. If the order type is not 0, then the order tag is... This indicates that the user has subscribed to additional value-added services or premium packages. For multi-value subscription types, the subscription type is encoded as multiple binary tags, indicating whether a sports package, movie package, children's package, etc., are being subscribed to.

[0073] By constructing the above tags, the dataset can simultaneously support viewing interest prediction and paid conversion prediction, and can reflect the user's behavioral chain from content attraction to commercial conversion.

[0074] In some examples, the method further includes: obtaining extended data for each program based on a public knowledge base of programs, wherein the extended data includes program sub-category information, program synopsis, lead actor information, director information, release date, and release region; and generating a program dataset based on the processed program metadata and the corresponding extended data.

[0075] For missing fields such as program synopsis, lead actor information, director information, and distribution region, candidate completion results can be generated using authorized content libraries, public program knowledge bases, or large-scale models. These results are then manually sampled or filtered based on confidence levels before being written into the corresponding fields. Original fields and completed fields can be stored separately to ensure data traceability.

[0076] In some examples, a program knowledge graph is built based on fields such as program name, type, tags, actors, director, region, and year. This knowledge graph includes entries from the same director, same lead actors, same region, same type, same series, and similar themes. This extension can be published as a standalone enhancement file without changing the underlying dataset structure.

[0077] In some examples, the method further includes: merging the user-program interaction dataset and the paid user dataset into a first program dataset based on user identification information; and / or, merging the user-program interaction dataset and the program dataset into a second program dataset based on program identification information.

[0078] Specifically, using anonymized user identification information as the linking information, each record in the user-program interaction dataset can be associated with the corresponding user's subscription type in the paid user dataset, generating a first program dataset containing user viewing behavior and payment attributes. This dataset can be used to conduct correlation analysis between user payment behavior and viewing preferences.

[0079] Specifically, using the unified program identification information as the association information, each record in the user-program interaction dataset can be associated with the metadata of the corresponding program in the program dataset to generate a second program dataset containing user viewing behavior and program content attributes. This dataset can be used to conduct correlation analysis between program attributes and viewing performance.

[0080] By merging user identification information or program identification information, dataset views with different dimensions can be generated according to different research needs, which facilitates user-side analysis or content-side analysis.

[0081] <Second Embodiment> Figure 3 This is a schematic diagram of the structure of a dataset construction apparatus provided in an embodiment of this disclosure. Figure 3 As shown, the device may include an acquisition module 310, a unification processing module 320, a cleaning module 330, a desensitization processing module 340, and a dataset generation module 350.

[0082] The acquisition module 310 is used to acquire multi-source IPTV service log information, which includes program viewing logs, subscription logs and program metadata generated by each service system. The program viewing logs include user identification information, program identification information, program name, program viewing start time, program viewing end time, service system type and program playback platform. The subscription logs include user identification information and subscription type information. The program metadata includes program identification information, program name and program type information.

[0083] The unification processing module 320 is used to perform unification processing based on different program identification information corresponding to the same program in different program viewing logs to obtain processed program viewing logs, and to perform unification processing based on different program identification information corresponding to the same program in program metadata to obtain processed program metadata.

[0084] The cleaning module 330 is used to clean the processed program viewing logs according to the set log cleaning constraint information to obtain the target program viewing logs.

[0085] The desensitization processing module 340 is used to perform irreversible desensitization processing on the user identification information in the target program viewing log and the user identification information in the order log, respectively, to obtain the processed target program viewing log and the processed order log.

[0086] The dataset generation module 350 is used to generate a user and program interaction dataset based on the processed target program viewing logs, a paid user dataset based on the processed subscription logs, and a program dataset based on the processed program metadata.

[0087] The device may also include a program object unification module, a log cleaning and verification module, an interaction matrix construction module, a tag generation module, a privacy desensitization module, a metadata fusion module, a dataset encapsulation module, and a statistical verification module. Each module can work together to complete each stage of dataset construction.

[0088] In some examples, the unification processing module 320 is used to obtain program association information corresponding to each program identifier, wherein the program association information includes program name, program type information, program broadcast time, total number of episodes, program release year, and program content provider encoding information; based on the program association information corresponding to each program identifier, it identifies different program identifier information corresponding to the same program from multiple program viewing logs, and identifies different program identifier information corresponding to the same program from multiple program metadata; it performs unification processing on the identified different program identifier information corresponding to the same program in the program viewing logs to obtain the processed program viewing logs, and performs unification processing on the identified different program identifier information corresponding to the same program in the program metadata to obtain the processed program metadata.

[0089] In some examples, the log cleaning constraints include the program viewing end time being later than the program viewing start time, the effective range of program viewing duration, the user activity threshold, and the rules for determining repeated viewing records of the same user for the same program.

[0090] In some examples, the desensitization processing module 340 generates random salt values ​​for the same user identification information in different target program viewing logs and different subscription logs; based on a set hash function, it processes each user identification information and the corresponding random salt value to obtain the corresponding desensitized user identification information; and replaces the corresponding unsensitized user identification information with each desensitized user identification information to obtain the processed target program viewing logs and processed subscription logs respectively.

[0091] In some examples, the device also includes a tag generation module. The tag generation module is used to determine the viewing interest tag of a corresponding user for watching a corresponding program based on the program viewing start time and program viewing end time in each processed target program viewing log, and add the viewing interest tag of the corresponding user for watching the corresponding program to the corresponding processed target program viewing log; and / or, based on all processed target program viewing logs corresponding to the same user, determine the viewing quality tag of the corresponding user, and add the viewing quality tag of the corresponding user to the processed target program viewing log corresponding to the same user; and / or, based on the subscription type information in each processed subscription log, determine the subscription tag of the corresponding user, and add the subscription tag of the corresponding user to the corresponding processed subscription log.

[0092] In some examples, the device also includes a program data extension module. This module obtains extended data for each program from a public knowledge base, including program subcategories, program descriptions, lead actor information, director information, release date, and release region. Based on the processed program metadata and the corresponding extended data, a program dataset is generated.

[0093] In some examples, the device also includes a merging module. The merging module is used to merge the user-program interaction dataset and the paid user dataset into a first program dataset based on user identification information; and / or, to merge the user-program interaction dataset and the program dataset into a second program dataset based on program identification information.

[0094] <Third Embodiment> This embodiment provides an electronic device. Figure 4 A schematic diagram of the hardware structure of the electronic device is shown.

[0095] like Figure 4 As shown, the electronic device 1000 includes a processor 1100 and a memory 1200. The memory 1200 can be used to store computer programs, and the processor 1100 can be used to retrieve the computer programs from the memory 1200 to execute any method embodiment of this disclosure. The processor 1100 can be one or more, and these processors can execute instructions individually or jointly. Similarly, the memory 1200 can also be one or more, and these memories can store the aforementioned computer programs individually or jointly.

[0096] This disclosure also provides a computer-readable storage medium storing a computer program thereon, which, when executed by a processor, implements the dataset construction method in any embodiment of this disclosure. Optionally, the computer-readable storage medium may be a non-transitory storage medium, but is not limited thereto; it may also be a temporary storage medium.

[0097] This disclosure can be a system, method, and / or computer program product. A computer program product may include a computer-readable storage medium having computer-readable program instructions loaded thereon for causing a processor to implement various aspects of this disclosure.

[0098] Computer-readable storage media can be tangible devices capable of holding and storing instructions for use by an instruction execution device. Computer-readable storage media can be, for example, but not limited to, electrical storage devices, magnetic storage devices, optical storage devices, electromagnetic storage devices, semiconductor storage devices, or any suitable combination thereof. More specific examples (a non-exhaustive list) of computer-readable storage media include: portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), static random access memory (SRAM), portable compact disc read-only memory (CD-ROM), digital multifunction disc (DVD), memory sticks, floppy disks, mechanical encoding devices, such as punch cards or recessed protrusions storing instructions thereon, and any suitable combination thereof. The computer-readable storage media used herein are not to be construed as transient signals themselves, such as radio waves or other freely propagating electromagnetic waves, electromagnetic waves propagating through waveguides or other transmission media (e.g., light pulses through fiber optic cables), or electrical signals transmitted through wires.

[0099] The computer-readable program instructions described herein can be downloaded from computer-readable storage media to various computing / processing devices, or downloaded via a network, such as the Internet, local area network, wide area network, and / or wireless network, to an external computer or external storage device. The network may include copper transmission cables, fiber optic transmission, wireless transmission, routers, firewalls, switches, gateway computers, and / or edge servers. A network adapter card or network interface in each computing / processing device receives the computer-readable program instructions from the network and forwards them to the computer-readable storage media in the respective computing / processing device.

[0100] The computer program instructions used to perform the operations of this disclosure may be assembly instructions, instruction set architecture (ISA) instructions, machine instructions, machine-dependent instructions, microcode, firmware instructions, status setting data, or source code or object code written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Smalltalk, C++, etc., and conventional procedural programming languages ​​such as the "C" language or similar programming languages. The computer-readable program instructions may execute entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving remote computers, the remote computer may be connected to the user's computer via any type of network—including a local area network (LAN) or a wide area network (WAN)—or may be connected to an external computer (e.g., via the Internet using an Internet service provider). In some embodiments, electronic circuits, such as programmable logic circuits, field-programmable gate arrays (FPGAs), or programmable logic arrays (PLAs), are personalized by utilizing state information from computer-readable program instructions. These electronic circuits can execute computer-readable program instructions to implement various aspects of this disclosure.

[0101] Various aspects of this disclosure are described herein with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this disclosure. It should be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer-readable program instructions.

[0102] These computer-readable program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing apparatus to produce a machine such that, when executed by the processor of the computer or other programmable data processing apparatus, they create means for implementing the functions / actions specified in one or more blocks of the flowchart and / or block diagram. These computer-readable program instructions can also be stored in a computer-readable storage medium that causes a computer, programmable data processing apparatus, and / or other device to operate in a particular manner; thus, the computer-readable medium storing the instructions comprises an article of manufacture that includes instructions for implementing aspects of the functions / actions specified in one or more blocks of the flowchart and / or block diagram.

[0103] Computer-readable program instructions may also be loaded onto a computer, other programmable data processing apparatus, or other device to cause a series of operational steps to be performed on the computer, other programmable data processing apparatus, or other device to produce a computer-implemented process, thereby causing the instructions executed on the computer, other programmable data processing apparatus, or other device to perform the functions / actions specified in one or more boxes of a flowchart and / or block diagram.

[0104] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of the present disclosure. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of an instruction, which contains one or more executable instructions for implementing a specified logical function. In some alternative implementations, the functions marked in the blocks may occur in a different order than those marked in the drawings. For example, two consecutive blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified function or action, or using a combination of dedicated hardware and computer instructions. It will be known to those skilled in the art that implementation in hardware, implementation in software, and implementation in a combination of software and hardware are equivalent.

[0105] The various embodiments of this disclosure have been described above. These descriptions are exemplary and not exhaustive, and are not limited to the disclosed embodiments. Many modifications and variations will be apparent to those skilled in the art without departing from the scope and spirit of the described embodiments. The terminology used herein is chosen to best explain the principles, practical application, or technical improvements to the embodiments in the market, or to enable others skilled in the art to understand the embodiments disclosed herein. The scope of this disclosure is defined by the appended claims.

Claims

1. A data set construction method characterized by comprising: include: Obtain multi-source IPTV service log information, wherein the multi-source IPTV service log information includes program viewing logs, subscription logs and program metadata generated based on each service system. The program viewing logs include user identification information, program identification information, program name, program viewing start time, program viewing end time, service system type and program playback platform. The subscription logs include user identification information and subscription type information. The program metadata includes program identification information, program name and program type information. The program viewing logs are processed by performing unified processing on different program identifiers corresponding to the same program in different program viewing logs to obtain the processed program viewing logs. The program metadata is also processed by performing unified processing on different program identifiers corresponding to the same program to obtain the processed program metadata. Based on the set log cleaning constraints, the processed program viewing logs are cleaned to obtain the target program viewing logs. Irreversibly desensitize the user identification information in the target program viewing log and the user identification information in the order log, respectively, to obtain the processed target program viewing log and the processed order log. A user-program interaction dataset is generated based on the processed target program viewing logs, a paid user dataset is generated based on the processed subscription logs, and a program dataset is generated based on the processed program metadata.

2. The method of claim 1, wherein, The process of unifying different program identifier information for the same program in different program viewing logs to obtain processed program viewing logs, and the process of unifying different program identifier information for the same program in program metadata to obtain processed program metadata, include: Obtain the program association information corresponding to each program identifier, wherein the program association information includes program name, program type information, program broadcast time, total number of episodes, program release year, and program content provider code information; Based on the program association information corresponding to each program identifier, different program identifiers corresponding to the same program are identified from multiple program viewing logs, and different program identifiers corresponding to the same program are identified from multiple program metadata. The program viewing log is processed by performing unified processing on the different program identifiers that correspond to the same program in the program viewing log. The program metadata is also processed by performing unified processing on the different program identifiers that correspond to the same program in the program metadata.

3. The method of claim 1, wherein, The set log cleaning constraints include the program viewing end time being later than the program viewing start time, the effective range of program viewing duration, the user activity threshold, and the rules for determining repeated viewing records of the same user for the same program.

4. The method according to claim 1, characterized in that, The user identification information in the target program viewing log and the user identification information in the subscription log are irreversibly desensitized to obtain the processed target program viewing log and the processed subscription log, including: For the same user identifier information in different target program viewing logs and different subscription logs, generate random salt values; Based on the set hash function, each user identifier and its corresponding random salt value are processed to obtain the corresponding de-identified user identifier. Replace the original user identification information with the anonymized user identification information to obtain the processed target program viewing log and the processed subscription log.

5. The method according to claim 1, characterized in that, The method further includes: Based on the program viewing start time and program viewing end time in each processed target program viewing log, determine the viewing interest tags of the corresponding user for the corresponding program, and add the viewing interest tags of the corresponding user for the corresponding program to the corresponding processed target program viewing log; and / or, Based on all processed target program viewing logs for the same user, determine the viewing quality tag for the corresponding user, and add the viewing quality tag for the corresponding user to the processed target program viewing logs for the same user; and / or, Based on the order type information in each processed order log, determine the corresponding user's order tag and add the corresponding user's order tag to the corresponding processed order log.

6. The method according to claim 1, characterized in that, The method further includes: Based on the program's public knowledge base, extended data for each program is obtained, including program sub-category information, program synopsis, lead actor information, director information, release date, and release region. Based on the processed program metadata and corresponding extended data, a program dataset is generated.

7. The method according to claim 1, characterized in that, The method further includes: Based on user identification information, the user-program interaction dataset and the paid user dataset are merged into a first program dataset; and / or, Based on the program identification information, the user-program interaction dataset and the program dataset are merged into a second program dataset.

8. A dataset construction apparatus, characterized in that, include: The acquisition module is used to acquire multi-source IPTV service log information, wherein the multi-source IPTV service log information includes program viewing logs, subscription logs and program metadata generated based on each service system. The program viewing logs include user identification information, program identification information, program name, program viewing start time, program viewing end time, service system type and program playback platform. The subscription logs include user identification information and subscription type information. The program metadata includes program identification information, program name and program type information. The unification processing module is used to perform unification processing on different program identification information corresponding to the same program in different program viewing logs to obtain the processed program viewing logs, and to perform unification processing on different program identification information corresponding to the same program in program metadata to obtain the processed program metadata. The cleaning module is used to clean the processed program viewing logs according to the set log cleaning constraint information to obtain the target program viewing logs. The desensitization module is used to perform irreversible desensitization processing on the user identification information in the target program viewing log and the user identification information in the order log, respectively, to obtain the processed target program viewing log and the processed order log. The dataset generation module is used to generate a user-program interaction dataset based on the processed target program viewing logs, a paid user dataset based on the processed subscription logs, and a program dataset based on the processed program metadata.

9. An electronic device, characterized in that, It includes a memory and a processor, the memory storing a computer program for controlling the processor to operate in order to perform the method according to any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that, A computer program is stored on the computer-readable storage medium, which, when executed by a processor, implements the method according to any one of claims 1 to 7.

Images