A data generation method and device, electronic equipment and readable storage medium

Abstract

Embodiments of the present application provide a data generation method and device, electronic equipment and a readable storage medium, the method comprising: obtaining full data to be analyzed, the full data comprising a plurality of first data; performing feature extraction on each first data to obtain corresponding second data; determining a connector associated with each table according to the number of second data associated with each table in a predetermined N number of tables, N being an integer greater than 1; and storing the second data associated with any target table in the N number of tables into the target table based on the association relationship between the target table and the connector. According to the number of second data associated with each table, the connector associated with each table is dynamically determined, which can adapt the number of tables and connectors, avoid resource waste caused by excessive number of connectors, or accumulation of second data during transmission caused by insufficient number of connectors, and improve the efficiency of data generation.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of big data, and in particular to a data generation method and device, electronic equipment and a readable storage medium. BACKGROUND

[0002] In the process of generating iceberg data, an iceberg data connector is usually used. Based on a distributed real-time stream processing framework (Apache Flink), after pulling and processing raw data from full data, the raw data is transmitted to an iceberg data table through the iceberg data connector, so as to facilitate subsequent data analysis and the like. An operation and maintenance personnel edits a Directed Acyclic Graph (DAG) according to experience, so that the data connector and the data table are connected according to the setting.

[0003] When the traffic changes, the existing data generation method needs to manually re-edit the DAG to adjust the connection relationship between the data connector and the data table. For example, when the traffic corresponding to a certain data table is large, the number of data connectors allocated to the data table is not enough, which can easily cause task back pressure. Back pressure is a situation in real-time data processing in which the speed of data generation upstream of a certain data table is greater than the speed of data processing of the data table. The occurrence of back pressure causes data accumulation, which affects the production efficiency of data. For another example, when the traffic corresponding to a certain data table is small, the number of data connectors allocated to the data table is too large, which can easily cause resource waste. However, this manual allocation of data connectors according to experience is time-consuming and inefficient, and after re-editing the DAG each time, a restart is required to maintain the healthy operation of the data generation task to a certain extent, which results in low efficiency of data generation.

[0004] It can be seen that the prior art has the problem of low efficiency of data generation. SUMMARY

[0005] The embodiments of the present application aim to provide a data generation method, device, electronic equipment and readable storage medium to solve the problem of low efficiency of data generation in the prior art. The specific technical solutions are as follows:

[0006] In a first aspect of the present application, a data generation method is provided, which comprises:

[0007] Obtaining full data to be analyzed, the full data comprising a plurality of first data;

[0008] Performing feature extraction on each first data to obtain corresponding second data;

[0009] determine a connector associated with each of the N tables according to a quantity of second data associated with each of the N tables, N being an integer greater than 1;

[0010] store the second data associated with the target table into the target table based on the association relationship between the target table and the connector.

[0011] Optionally, the determining the connector associated with each of the N tables according to the quantity of second data associated with each of the N tables comprises:

[0012] acquiring the quantity of second data associated with an nth table of the N tables every interval of a preset time;

[0013] in a case where the quantity of second data associated with the nth table is greater than an upper limit value of a total quantity of the second data transmitted by m first connectors, determining a target connector from among connectors other than the m first connectors in M connectors, the m first connectors being connectors currently associated with the nth table in the M connectors, n being a positive integer less than or equal to N, M being an integer greater than 1, and m being a positive integer less than M;

[0014] establishing an association relationship between the target connector and the nth table.

[0015] Optionally, the determining the target connector from among the connectors other than the m first connectors in the M connectors in the case where the quantity of second data associated with the nth table is greater than the upper limit value of the total quantity of the second data transmitted by the m first connectors comprises at least one of:

[0016] in a case where the quantity of second data associated with the nth table is greater than the upper limit value of the total quantity of the second data transmitted by the m first connectors and a difference between the quantity of second data associated with the nth table and the upper limit value is greater than or equal to a first threshold value, determining a second connector other than the m first connectors in the M connectors, the second connector being a connector not associated with any of the N tables, the target connector comprising the second connector;

[0017] in a case where the quantity of second data associated with the nth table is greater than the upper limit value of the total quantity of the second data transmitted by the m first connectors and a difference between the quantity of second data associated with the nth table and the upper limit value is greater than or equal to a second threshold value, determining a third connector other than the m first connectors in the M connectors, the third connector being a connector associated with at least one table other than the nth table in the N tables, the target connector comprising the third connector, and the second threshold value being less than or equal to the first threshold value.

[0018] Optionally, determining a third connector from the M connectors, excluding the m first connectors, includes:

[0019] Obtain the number of second data associated with the q-th table, where the q-th table is any table among the N tables except the n-th table, and q is a positive integer less than or equal to N;

[0020] If the amount of second data associated with the q-th table is less than the upper limit of the total amount of the second data transmitted by the third connector, a third connector is determined from the M connectors, excluding the m first connectors. The third connector is the connector associated with the q-th table.

[0021] Optionally, determining the connector associated with each table based on the number of second data associated with each of the preset N tables further includes:

[0022] If the number of second data associated with the nth table is less than the upper limit of the total number of second data transmitted by the m first connectors, then m1 first connectors are determined from the m first connectors, where m1 is a positive integer less than m;

[0023] Cancel the association between the m1 first connectors and the nth table.

[0024] Optionally, determining the connector associated with each table based on the number of second data associated with each of the preset N tables further includes:

[0025] If the number of second data associated with the nth table is equal to the upper limit of the total number of second data transmitted by the m first connectors, the association between the m first connectors and the nth table is maintained.

[0026] Optionally, after determining the connector associated with each table based on the number of second data associated with each table in the preset N tables, and before storing the second data associated with the target table into the target table based on the association relationship between any target table and the connector in the N tables, the method further includes:

[0027] A first label for the second data is determined based on the attribute information of the second data. The first label is used to represent the target table corresponding to the second data in the N tables.

[0028] A second marker for the second data is determined based on the connector associated with the target table, and the second marker is used to represent the connector that transmits the second data;

[0029] The step of storing the second data associated with any target table in the target table into the target table based on the association relationship between any target table in the N tables and the connector includes:

[0030] Based on the first marker, determine the target table for each of the N tables associated with the second data;

[0031] Each connector associated with the second data is determined in the connectors associated with the target table based on the second mark;

[0032] Each piece of the second data is stored in the target table associated with each piece of the second data through the connector associated with each piece of the second data.

[0033] In a second aspect of the present invention, a data generation apparatus is provided, comprising:

[0034] The acquisition module is used to acquire the full data to be analyzed, which includes multiple first data entries;

[0035] The feature extraction module is used to extract features from each piece of first data to obtain the corresponding second data.

[0036] The determination module is used to determine the connector associated with each table based on the number of second data associated with each table in the preset N tables, where N is an integer greater than 1;

[0037] The storage module is used to store the second data associated with the target table into the target table based on the association relationship between any target table and the connector among the N tables.

[0038] In a third aspect of the present invention, an electronic device is provided, including a processor, a communication interface, a memory, and a communication bus, wherein the processor, the communication interface, and the memory communicate with each other through the communication bus.

[0039] Memory, used to store programs;

[0040] When a processor executes a program stored in memory, it implements the method described in the first aspect.

[0041] In a fourth aspect of the invention, a readable storage medium is provided having a program stored thereon that, when executed by a processor, implements the method described in the first aspect.

[0042] In this embodiment of the invention, by acquiring the full data in the distributed publish-subscribe messaging system and extracting features from each piece of first data in the full data to obtain second data conforming to the table storage format, and then dynamically determining the connector associated with each table based on the number of second data associated with each of the preset N tables, the number of tables and connectors can be matched, avoiding resource waste due to too many connectors or data accumulation during the transmission of second data due to too few connectors. Compared with the method of manually allocating connectors, the method provided in this application embodiment can quickly adjust the associated connectors according to the changes in traffic of different tables, enhance the timeliness of connector allocation, and improve the efficiency of data generation. Attached Figure Description

[0043] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the accompanying drawings used in the description of the embodiments or the prior art will be briefly introduced below.

[0044] Figure 1 This is one of the flowcharts illustrating a data generation method in an embodiment of the present invention;

[0045] Figure 2 This is a second schematic flowchart of a data generation method in an embodiment of the present invention;

[0046] Figure 3 This is a third flowchart illustrating a data generation method in an embodiment of the present invention;

[0047] Figure 4 This is a schematic diagram of the structure of a data generation device according to an embodiment of the present invention;

[0048] Figure 5 This is a structural diagram of an electronic device according to an embodiment of the present invention. Detailed Implementation

[0049] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0050] Before discussing the exemplary embodiments in more detail, it should be noted that some exemplary embodiments are described as processes or methods depicted as flowcharts. Although the flowcharts describe the steps as sequential processes, many of these steps can be performed in parallel, concurrently, or simultaneously. Furthermore, the order of the steps can be rearranged. A process can be terminated when its operation is complete, but it may also have additional steps not included in the figures. A process can correspond to a method, function, procedure, subroutine, subroutine, etc.

[0051] Furthermore, the terms "first," "second," etc., may be used herein to describe various directions, actions, steps, or elements, but these directions, actions, steps, or elements are not limited by these terms. These terms are only used to distinguish a first direction, action, step, or element from another direction, action, step, or element. For example, without departing from the scope of the invention, a first speed difference may be referred to as a second speed difference, and similarly, a second speed difference may be referred to as a first speed difference. Both the first speed difference and the second speed difference are speed differences, but they are not the same speed difference. The terms "first," "second," etc., should not be construed as indicating or implying relative importance or implicitly specifying the number of indicated technical features. Thus, a feature defined as "first" or "second" may explicitly or implicitly include one or more of that feature. In the description of the invention, "a plurality of" means at least two, such as two, three, etc., unless otherwise explicitly specified.

[0052] This invention provides a data generation method, such as... Figure 1 As shown, the steps of this method include:

[0053] Step 101: Obtain the full data to be analyzed. The full data includes multiple first data entries, which are message data obtained from the distributed publish-subscribe messaging system.

[0054] Full data can be data stored in a distributed publish-subscribe messaging system (such as a Kafka cluster). Based on the Kafka distributed publish-subscribe messaging system, low-latency and high-throughput data delivery can be achieved. Multiple first data entries can be data contained in a Kafka queue.

[0055] In one example, such as Figure 2As shown, the data generation system includes a controller and a connector. The full data consists of multiple first data entries, meaning the Kafka queue can contain on-demand data from multiple channels. After obtaining the on-demand data (i.e., the full data) from all channels in the Kafka queue to be analyzed, the controller uses a preset number of threads, for example, 100 threads, to pull data from the full data, obtaining 100 first data entries. It should be understood that the preset number of threads can also be other numbers; pulling data from the full data to obtain the corresponding number of first data entries can achieve the same technical effect, which will not be elaborated further here.

[0056] For example, one set of first data could be "Xiaoming watched 'Pleasant Goat and Big Big Wolf' at 8 a.m.", another set of first data could be "Xiaohong watched 'Water Margin' on the train at 2 p.m.", and yet another set of first data could be "Xiaohua watched 'Day Day Up' on her mobile phone".

[0057] Step 102: Extract features for each piece of first data to obtain the corresponding second data;

[0058] For each piece of the first data, features are extracted to obtain second data that conforms to a tabular storage format. This feature extraction can be performed using supervised learning, unsupervised learning, or semi-supervised learning methods.

[0059] In one example, the controller inputs the first data "Xiaoming watched 'Pleasant Goat and Big Big Wolf' at 8:00 AM" into the feature extraction model, and the resulting second data could be "Xiaoming, 8:00 AM, Pleasant Goat and Big Big Wolf"; the controller inputs the first data "Xiaohong watched 'Water Margin' on the train at 2:00 PM" into the feature extraction model, and the resulting second data could be "Xiaohong, 2:00 PM, Water Margin"; the controller inputs the first data "Xiaohua watched 'Day Day Up' on her mobile phone" into the feature extraction model, and the resulting second data could be "Xiaohua, mobile phone, Day Day Up". The operator edits the tables to be generated in the user interface, including Table 1, Table 2, and Table 3. Table 1 stores data in the format: Person-Title; Table 2 stores data in the format: Person-Time-Title; and Table 3 stores data in the format: Person-Terminal-Title. The terms "Xiaoming, Happy Sheep," "Xiaohong, Water Margin," and "Xiaohua, Everyday Upward" in the second set of data can be stored in Table 1, formatted as "Character-Title." Furthermore, "Xiaoming, 8 AM, Happy Sheep" and "Xiaohong, 2 PM, Water Margin" can be stored in Table 2, formatted as "Character-Time-Title." Finally, "Xiaohua, Mobile Phone, Everyday Upward" can be stored in Table 3, formatted as "Character-Terminal-Title." This allows for targeted training of the feature extraction model during the feature extraction process on the first set of data, based on the format of the tables used to store the second set of data. This improves the accuracy of feature extraction, resulting in the acquisition of suitable second set of data, which is then used for subsequent analysis and processing.

[0060] Step 103: Determine the connector associated with each table based on the number of second data associated with each of the N preset tables, where N is an integer greater than 1;

[0061] Based on the tasks set in the distributed real-time streaming framework (Apache Flink), the on-demand data of all channels is generated into a table (i.e., an iceberg table) corresponding to each preset channel. In this embodiment of the invention, the format corresponding to Table 1 (i.e., person-title) is used as an example for illustration. The N preset tables may include: a television channel table, a children's channel table, and a variety show channel table.

[0062] The second data associated with the TV channel table can be on-demand data of TV drama programs watched based on the "title". For example, after feature extraction of the first data "Xiao Hong watched 'Water Margin' on the train at 2 PM", the second data obtained could be "Xiao Hong, Water Margin". Based on the title "Water Margin", "Xiao Hong, Water Margin" is determined as the second data associated with the TV channel table. Similarly, for another first data "Xiao Zhao watched 'Romance of the Three Kingdoms' on the computer at 3 PM", "Xiao Zhao, Romance of the Three Kingdoms" is determined as the second data associated with the TV channel table based on the title "Romance of the Three Kingdoms". The second data associated with the TV channel table can also include other TV drama programs, which will not be elaborated upon here.

[0063] The second data associated with the children's channel table can be on-demand data of children's programs watched based on the "program title". For example, after feature extraction of the first data "Xiaoming watched 'Pleasant Goat and Big Big Wolf' at 8:00 AM", the second data obtained could be "Xiaoming, Pleasant Goat". Based on the program title "Pleasant Goat", "Xiaoming, Pleasant Goat" is determined as the second data associated with the children's channel table. Similarly, for another first data point, "Xiaoqian watched 'BabyBus' on a tablet at 5:00 PM", "Xiaoqian, BabyBus" is also determined as the second data associated with the children's channel table based on the program title "BabyBus". The second data associated with the children's channel table can also include other children's programs, which will not be elaborated upon here.

[0064] The second data associated with the variety show channel table can be on-demand data of viewers who watched variety shows, determined by the "show title". For example, after feature extraction of the first data "Xiaohua watched 'Day Day Up' on her phone", the second data obtained could be "Xiaohua, Day Day Up". Based on the show title "Day Day Up", "Xiaohua, Day Day Up" is determined as the second data associated with the variety show channel table. Similarly, for another first data point, "Xiaosun watched 'Open Lecture' at 10 AM", "Xiaosun, Open Lecture" is also determined as the second data associated with the variety show channel table based on the show title "Open Lecture". The second data associated with the variety show channel table can also include other variety shows, which will not be elaborated upon here.

[0065] The full dataset includes multiple first data entries. After feature extraction from these first data entries, multiple second data entries are obtained, each including keywords from its corresponding first data entry. In this embodiment, when generating corresponding channel tables based on channel type, the traffic of different channel tables is considered, i.e., the number of second data entries associated with different channel tables is taken into account. Based on the number of second data entries associated with each of the preset N tables, the connector associated with each table is determined to achieve an optimal connector resource allocation scheme.

[0066] In one example, the controller subscribes to messages in Kafka and extracts features from each piece of first data to obtain second data conforming to a table storage format. If the traffic to the TV channel table is high, meaning there is a large amount of second data associated with the TV channel table, more connectors can be added to connect to the TV channel table simultaneously. This allows multiple connectors to connect to the TV channel table simultaneously to transmit the associated second data, avoiding backpressure (i.e., data backlog) caused by insufficient connectors. For example, if there are 3 connectors associated with the TV channel table, and the maximum number of second data transmitted by each connector is 15,000 records per second, and the traffic to the TV channel table is high, with 50,000 associated second data records, the required number of connectors is calculated as 50,000 / 15,000, rounded up to 4. Therefore, one more connector needs to be added to the TV channel table. In this way, the number of connectors associated with the TV channel table is adjusted in real time based on the traffic of the TV channel table. Compared with the manual allocation method, this enhances the timeliness of connector allocation and improves data generation efficiency.

[0067] In another example, the controller subscribes to messages in Kafka and extracts features from each piece of first data to obtain second data conforming to a tabular storage format. If the traffic to the variety show channel table is low, meaning there is less second data associated with it, the number of connectors associated with the variety show channel table can be reduced, or the connectors associated with the variety show channel table can be associated with other channel tables simultaneously, improving connector resource utilization. For example, if there are 3 connectors associated with the variety show channel table, and the maximum number of second data transmitted by each connector is 15,000 messages per second, then when the traffic to the variety show channel table is low and there are only 10,000 second data messages associated with it, determining the number of connectors associated with the variety show channel table based on these 10,000 messages only requires: 10,000 / 15,000, rounded up to 1. Therefore, 2 fewer connectors can be associated with the variety show channel table. These 2 fewer connectors can be associated with other channel tables, improving connector resource utilization and thus increasing data generation efficiency.

[0068] Step 104: Based on the association between any target table and the connector among the N tables, store the second data associated with the target table into the target table.

[0069] After determining the connector associated with each table in step 103, the controller stores the second data associated with the target table into the target table through the connector associated with the target table according to the determined scheme. For example, if the connectors associated with the TV channel table are determined to be connector 1, connector 2, and connector 3 based on the number of second data associated with the TV channel table, the second data associated with the TV channel table is stored into the TV channel table using a polling method based on connectors 1, connector 2, and connector 3.

[0070] In this embodiment of the invention, by acquiring the full data in the distributed publish-subscribe messaging system and extracting features from each piece of first data in the full data to obtain second data conforming to the table storage format, and then dynamically determining the connector associated with each table based on the number of second data associated with each of the preset N tables, the number of tables and connectors can be matched, avoiding resource waste due to too many connectors or data accumulation during the transmission of second data due to too few connectors. Compared with the method of manually allocating connectors, the method provided in this application embodiment can quickly adjust the associated connectors according to the changes in traffic of different tables, enhance the timeliness of connector allocation, and improve the efficiency of data generation.

[0071] In some optional embodiments, step 103: determining the connector associated with each table based on the number of second data associated with each of the preset N tables includes:

[0072] At preset time intervals, retrieve the number of second data items associated with the nth table out of N preset tables.

[0073] If the number of second data associated with the nth table is greater than the upper limit of the total number of second data transmitted by the m first connectors, a target connector is determined among the M connectors excluding the m first connectors. The m first connectors are the connectors currently associated with the nth table among the M connectors, where n is a positive integer less than or equal to N, M is an integer greater than 1, and m is a positive integer less than M.

[0074] Establish the association between the target connector and the nth table.

[0075] In this embodiment, the Kafka full data can maintain dynamic balance. That is, feature extraction is performed on a portion of the first data to obtain second data, and this second data is stored in the corresponding table via a connector. Then, a new batch of first data is added to the full data, ensuring that the Kafka full data always remains within a certain range. The second data obtained after feature extraction on the newly added first data can be the second data associated with the nth table out of N preset tables. This causes changes in the traffic of the nth table, which in turn changes the number of second data associated with the nth table. The following example uses the TV channel table from the preset TV channel table, children's channel table, and variety channel table as an illustration:

[0076] like Figure 2 and Figure 3 As shown, the controller can obtain 64,000 records of the second data associated with the current TV channel table from the Flink task every 2 minutes. The first connectors currently associated with the TV channel table are connector 1, connector 2, and connector 3. Assuming that connectors 1, 2, and 3 have the same data processing performance, and each connector's maximum transmission capacity for the second data is 16,000 records per second, then the maximum transmission capacity of the three first connectors associated with the TV channel table is 16,000 × 3 = 48,000 records. In this case, the number of second data associated with the TV channel table is greater than the maximum transmission capacity of the three first connectors associated with the TV channel table, i.e., 64,000 > 48,000. To avoid task backpressure (i.e., data backlog during second data transmission) that reduces data generation efficiency, the number of connectors associated with the TV channel table can be increased. The controller can determine the target connector from among the M connectors excluding connectors 1, 2, and 3. Assuming that the upper limit for the number of second data transmitted by connector 5 (not shown in the figure) is also 16,000 records per second, connector 5 is designated as the target connector. An association is established between the target connector and the TV channel table, so that the TV channel table is simultaneously associated with connectors 1, 2, 3, and 5. After establishing the association between the target connector and the TV channel table, the upper limit for the number of second data transmitted by connectors 1, 2, 3, and 5 is 16,000 × 4 = 64,000 records. Thus, based on the number of second data associated with the nth table (i.e., the TV channel table) out of the preset N tables, the number of connectors associated with the nth table is dynamically determined, reducing backpressure and improving data generation efficiency.

[0077] This can be achieved by calling the target connector's Application Programming Interface (API), such as TableSource or TableSink interfaces, to connect the target connector to the TV channel table and establish a relationship between them. Before establishing this relationship, the first connectors associated with the TV channel table are Connector 1, Connector 2, and Connector 3. These can be connectors initially chosen by the developers based on experience, or they can be dynamically determined based on the amount of second data associated with the TV channel table in the previous cycle.

[0078] Among them, M connectors are multiple preset connectors, including connector 1, connector 2, connector 3, connector associated with the children's channel table, connection associated with the variety channel table, and idle connectors (i.e., connections not associated with any table).

[0079] It should be noted that, according to the determined scheme, the second data associated with any target table among the N tables is stored in the target table based on the connector associated with the target table. By optimizing the allocation scheme of the connectors associated with each table in real time, the data generation efficiency can be improved, which will not be elaborated here.

[0080] In some optional embodiments, when the number of second data associated with the nth table is greater than the upper limit of the total number of the second data transmitted by the m first connectors, determining the target connector among the M connectors excluding the m first connectors includes at least one of the following:

[0081] If the number of second data associated with the nth table is greater than the upper limit of the total number of second data transmitted by the m first connectors, and the difference between the number of second data associated with the nth table and the upper limit is greater than or equal to a first threshold, a second connector other than the m first connectors is determined among the M connectors. The second connector is a connector that is not associated with any of the N tables, and the target connector includes the second connector.

[0082] If the number of second data associated with the nth table is greater than the upper limit of the total number of second data transmitted by the m first connectors, and the difference between the number of second data associated with the nth table and the upper limit is greater than or equal to a second threshold, a third connector other than the m first connectors is determined among the M connectors. The third connector is a connector among the N tables associated with at least one table other than the nth table. The target connector includes the third connector, and the second threshold is less than or equal to the first threshold.

[0083] In one example, again taking the nth table as the TV channel table, the number of second data associated with the TV channel table is greater than the number of first connectors associated with the TV channel table (e.g., Figure 2 and Figure 3 In this context, the upper limit of the number of second data transmitted by connectors 1, 2, and 3 is defined, and the number of second data associated with the TV channel table is 80,000. The upper limit of the number of second data transmitted by the three first connectors is 1.6 × 3 = 48,000. The first threshold can be half of the upper limit of the number of second data transmitted by a single connector, i.e., 1.6 ÷ 2 = 8,000. The difference between the number of second data associated with the TV channel table and the upper limit of the number of second data transmitted by these three first connectors is 80,000 - 48,000 = 32,000 > 8,000. Therefore, the traffic of the TV channel table can be considered large. When adding a connector associated with the TV channel table, the new connector must not be associated with any of the N tables before being associated with the TV channel table. This is to avoid the new connector simultaneously transmitting second data associated with other tables and second data associated with the TV channel table, which would affect the efficiency of transmitting the second data associated with the TV channel table. The controller can determine the target connector from among the M connectors excluding connectors 1, 2, and 3. Assuming that the upper limit for the number of second data transmitted by either connector 5 or connector 6 (not shown in the figure) is 16,000 per second, and that both connector 5 and connector 6 are second connectors (i.e., connector 5 and connector 6 are not currently associated with any of the N tables), then connector 5 and connector 6 can be identified as target connectors.

[0084] In this way, after establishing the association between the target connector and the TV channel table, connectors 5 and 6 can transmit the second data associated with the TV channel table at full load. This avoids affecting the efficiency of transmitting the second data associated with the TV channel table when transmitting the second data associated with other tables at the same time, reduces the back pressure of the task, and improves the data generation efficiency of the nth table.

[0085] In another example, again using the nth table as the TV channel table, the number of second data associated with the TV channel table is greater than the number of the three first connectors associated with the TV channel table (e.g., Figure 2 and Figure 3 In the context of the M connectors (connectors 1, 2, and 3), the upper limit for the number of second data transmitted is defined. The number of second data entries associated with the TV channel table is 50,000. The upper limit for the number of second data entries transmitted by the three first connectors is 1.6 × 3 = 48,000 entries. The second threshold can be half of the upper limit for the number of second data entries transmitted by a single connector, i.e., 1.6 ÷ 2 = 8,000 entries. The difference between the number of second data entries associated with the TV channel table and the upper limit for the number of second data entries transmitted by the three first connectors is 50,000 - 48,000 = 2,000 entries < 8,000 entries. Therefore, the TV channel table's traffic can be considered relatively high. When adding connectors associated with the TV channel table, the new connector does not necessarily need to be associated with any of the N tables before being associated with the TV channel table. When the new connector simultaneously transmits second data associated with other tables and second data associated with the TV channel table, it will not affect the efficiency of transmitting the second data associated with the TV channel table. The controller can determine the target connector from among the M connectors excluding connectors 1, 2, and 3. Assuming that the upper limit for the number of second data transmitted by connector 4 is also 16,000 data entries per second, and that connector 4 is the third connector (e.g., ... Figure 2 and Figure 3 In the context of connector 4 being currently associated with both the children's channel table and the variety show channel table, connector 4 can be identified as the target connector.

[0086] In this way, after establishing the association between the target connector and the TV channel table, connector 4 can be associated with the TV channel table, children's channel table and variety channel table at the same time, which improves the data generation efficiency of the TV channel table and also improves the utilization rate of connector 4.

[0087] In determining that connector 4 is associated with the TV channel table, since connector 4 is a third connector (i.e., connector 4 is currently associated with both the children's channel table and the variety show channel table), to avoid backpressure on the children's channel table and variety show channel table after connector 4 is associated with the TV channel table, in an optional example, determining the third connector other than the m first connectors among the M connectors includes:

[0088] Obtain the number of second data associated with the q-th table, where the q-th table is any table among the N tables except the n-th table, and q is a positive integer less than or equal to N;

[0089] If the amount of second data associated with the q-th table is less than the upper limit of the total amount of the second data transmitted by the third connector, a third connector is determined from the M connectors, excluding the m first connectors. The third connector is the connector associated with the q-th table.

[0090] In this embodiment, since connector 4 is currently associated with both the children's channel table and the variety show channel table, it is necessary to obtain the number of second data associated with the q-th table out of N tables, and the number of second data associated with the w-th table out of N tables, where w is a positive integer less than N and not equal to q. In other words, it is necessary to obtain the number of second data associated with the children's channel table and the number of second data associated with the variety show channel table. Dynamically determining the connector associated with each table reduces the uneven distribution of connectors among the tables. By optimizing the connector allocation scheme for each table in real time, data generation efficiency is improved.

[0091] For example, the number of second data entries associated with the children's channel table is 5,000, and the number of second data entries associated with the variety show channel table is 10,000. The upper limit for the number of second data entries transmitted by connector 4 is 16,000. The current load of connector 4 is: 5,000 + 10,000 = 15,000 entries < 16,000 entries, which does not exceed the upper limit for the number of second data entries transmitted by connector 4. Therefore, connector 4 can be used as a third connector to associate with more other tables. Furthermore, the difference between the number of second data entries associated with the TV channel table and the upper limit for the number of second data entries transmitted by the three first connectors is: 50,000 - 48,000 = 2,000 entries. After associating connector 4 with the TV channel table, the load of connector 4 becomes: 5,000 + 10,000 + 2,000 = 17,000 entries > 16,000 entries, which exceeds the upper limit for the number of second data entries transmitted by connector 4. Therefore, connector 4 cannot be used as a third connector to associate with the TV channel table. This avoids the situation where associating connector 4 with the TV channel table causes backpressure on the tables originally associated with connector 4. Based on this, further, if the number of second data associated with the q-th table is less than a third threshold, a third connector other than the m first connectors can be determined among the M connectors, where the third threshold is half of the upper limit of the total number of the second data transmitted by the third connector.

[0092] For example, the number of second data entries associated with the children's channel table is 1,000, and the number of second data entries associated with the variety show channel table is 2,000. The upper limit for the number of second data entries transmitted by connector 4 is 16,000. The current load of connector 4 is: 1,000 + 2,000 = 3,000 entries < 16,000 ÷ 2 = 8,000 entries. Therefore, connector 4 can be used as a third connector to associate with more other tables. Furthermore, the difference between the number of second data entries associated with the TV channel table and the upper limit for the number of second data entries transmitted by the three first connectors is: 5 - 48,000 = 2,000 entries. After associating connector 4 with the TV channel table, the load of connector 4 becomes: 1,000 + 2,000 + 2,000 = 5,000 entries < 16,000 entries. Therefore, connector 4 can be used as a third connector to associate with the TV channel table. This avoids the situation where the tables originally associated with connector 4 experience backpressure after associating connector 4 with the TV channel table, improving the utilization rate of connector 4 and simultaneously improving the data generation efficiency of the TV channel table.

[0093] In some optional embodiments, step 103: determining the connector associated with each table based on the number of second data associated with each of the preset N tables, further includes:

[0094] If the number of second data associated with the nth table is less than the upper limit of the total number of second data transmitted by the m first connectors, then m1 first connectors are determined from the m first connectors, where m1 is a positive integer less than m;

[0095] Cancel the association between the m1 first connectors and the nth table.

[0096] In this embodiment, taking the nth table as the TV channel table as an example, the number of second data entries associated with the current TV channel table obtained from the Flink task every 2 minutes is 32,000. The first connectors currently associated with the TV channel table are connector 1, connector 2, and connector 3. Assuming that connectors 1, 2, and 3 have the same data processing performance, and each connector's maximum transmission capacity for second data is 16,000 entries per second, then the maximum transmission capacity of the three first connectors associated with the TV channel table is 16,000 × 3 = 48,000 entries. At this point, the number of second data entries associated with the TV channel table is less than the maximum transmission capacity of the three first connectors associated with the TV channel table, i.e., 32,000 < 48,000. To avoid idle data when the second data associated with the TV channel table is processed, resulting in resource waste, the number of connectors associated with the TV channel table can be reduced. The controller can determine one first connector (e.g., connector 3) from connectors 1, 2, and 3, cancel the association between connector 3 and the TV channel table, so that the TV channel table is simultaneously associated with connectors 1 and 2. After canceling the association between connector 3 and the TV channel table, the upper limit for the number of second data entries transmitted by connectors 1 and 2 is 1.6 × 2 = 32,000. This allows for the dynamic determination of the number of connectors associated with the nth table (i.e., the TV channel table) out of the preset N tables. This avoids wasting resources by allocating too many connectors to the nth table when the traffic corresponding to it decreases. Furthermore, connector 3, whose association with the TV channel table has been canceled, can re-establish associations with other tables in the N tables, improving overall data generation efficiency.

[0097] In some optional embodiments, step 103: determining the connector associated with each table based on the number of second data associated with each of the preset N tables, further includes:

[0098] If the number of second data associated with the nth table is equal to the upper limit of the total number of second data transmitted by the m first connectors, the association between the m first connectors and the nth table is maintained.

[0099] In this embodiment, taking the nth table as the TV channel table as an example, the number of second data entries associated with the current TV channel table obtained from the Flink task every 2 minutes is 48,000. The first connectors currently associated with the TV channel table are Connector 1, Connector 2, and Connector 3. Assuming that Connector 1, Connector 2, and Connector 3 have the same data processing performance, and each connector's maximum transmission capacity for second data is 16,000 entries per second, then the maximum transmission capacity of the three first connectors associated with the TV channel table is 16,000 × 3 = 48,000 entries. At this point, the number of second data entries associated with the TV channel table is equal to the maximum transmission capacity of the three first connectors associated with the TV channel table. The association between Connector 1, Connector 2, and Connector 3 and the TV channel table is maintained. This allows the second data associated with the TV channel table to be stored in the TV channel table through Connector 1, Connector 2, and Connector 3, improving the stability of the data generation process.

[0100] In some optional embodiments, after step 103: determining the connector associated with each table based on the number of second data associated with each table in the preset N tables, and before step 104: storing the second data associated with the target table into the target table based on the association relationship between any target table and the connector in the N tables, the method further includes:

[0101] A first label for the second data is determined based on the attribute information of the second data. The first label is used to represent the target table corresponding to the second data in the N tables.

[0102] A second marker for the second data is determined based on the connector associated with the target table, and the second marker is used to represent the connector that transmits the second data;

[0103] Step 104: Based on the association relationship between any target table among the N tables and the connector, storing the second data associated with the target table into the target table includes:

[0104] Based on the first marker, determine the target table for each of the N tables associated with the second data;

[0105] Each connector associated with the second data is determined in the connectors associated with the target table based on the second mark;

[0106] Each piece of the second data is stored in the target table associated with each piece of the second data through the connector associated with each piece of the second data.

[0107] In this embodiment, a first tag for the second data is determined based on the attribute information of the second data. For example, based on the phrase "Water Margin" in the second data, it can be determined that the target table corresponding to the second data in the TV channel table, children's channel table, and variety channel table is the TV channel table. Therefore, the second data is marked with a first tag, thus associating the second data with the TV channel table. Similarly, after marking each piece of second data with a first tag, a second tag for the second data is determined based on the connector associated with the target table. For example, through step 103, the connectors associated with the TV channel table are determined to be: connector 1, connector 2, connector 3, and connector 4. A polling method is used to determine one connector from connector 1 to connector 4 as the second tag for the second data (for example, if the polling result is connector 3, then the second tag corresponds to connector 3).

[0108] Thus, based on the first and second tags of the second data, connector 3 is selected to store the second data in the TV channel table. Similarly, after determining the target table associated with each piece of second data among N tables based on the first tag, and determining the connector associated with each piece of second data among the connectors associated with the target table based on the second tag, each piece of second data is stored in the target table associated with each piece of second data through the connector associated with each piece of second data. By optimizing the allocation scheme of connectors associated with each table in real time, data generation efficiency is improved. When a new table needs to be added, or an old table needs to be taken offline, developers only need to edit and update on the controller. The controller can calculate the new allocation scheme of connectors associated with each table based on the number of second data associated with each table, and it can be pulled by Flink tasks to achieve hot updates without downtime.

[0109] like Figure 4 As shown, this embodiment of the invention also provides a data generation device 400, comprising:

[0110] The acquisition module 401 is used to acquire the full data to be analyzed, wherein the full data includes multiple first data.

[0111] Feature extraction module 402 is used to extract features for each piece of first data to obtain the corresponding second data;

[0112] The determining module 403 is used to determine the connector associated with each table based on the number of second data associated with each table in the preset N tables, where N is an integer greater than 1.

[0113] Storage module 404 is used to store the second data associated with the target table into the target table based on the association relationship between any target table and the connector among the N tables.

[0114] Optionally, the determining module 403 includes:

[0115] The acquisition submodule is used to acquire the number of second data associated with the nth table in a preset N tables at preset time intervals;

[0116] The first determining submodule is used to determine the target connector among the connectors other than the m first connectors among the M connectors when the number of second data associated with the nth table is greater than the upper limit of the total number of second data transmitted by the m first connectors. The m first connectors are the connectors currently associated with the nth table among the M connectors, where n is a positive integer less than or equal to N, M is an integer greater than 1, and m is a positive integer less than M.

[0117] A submodule is created to establish the association between the target connector and the nth table.

[0118] Optionally, the first determined submodule includes at least one of the following:

[0119] A first determining unit is configured to determine a second connector other than the m first connectors among the M connectors when the number of second data associated with the nth table is greater than an upper limit of the total number of second data transmitted by the m first connectors, and the difference between the number of second data associated with the nth table and the upper limit is greater than or equal to a first threshold. The second connector is a connector that is not associated with any of the N tables, and the target connector includes the second connector.

[0120] The second determining unit is configured to determine a third connector other than the m first connectors among the M connectors when the number of second data associated with the nth table is greater than the upper limit of the total number of second data transmitted by the m first connectors, and the difference between the number of second data associated with the nth table and the upper limit is greater than or equal to a second threshold. The third connector is a connector among the N tables associated with at least one table other than the nth table. The target connector includes the third connector, and the second threshold is less than or equal to the first threshold.

[0121] Optionally, the second determining unit includes:

[0122] A sub-unit is used to obtain the number of second data associated with the q-th table, where the q-th table is any table among the N tables except the n-th table, and q is a positive integer less than or equal to N;

[0123] A subunit is defined for determining a third connector from the M connectors, excluding the m first connectors, when the amount of second data associated with the q-th table is less than the upper limit of the total amount of the second data transmitted by the third connector. The third connector is the connector associated with the q-th table.

[0124] Optionally, the determining module 403 further includes:

[0125] The second determining submodule is used to determine m1 first connectors from the m first connectors when the number of second data associated with the nth table is less than the upper limit of the total number of second data transmitted by the m first connectors, where m1 is a positive integer less than m.

[0126] The Cancel submodule is used to cancel the association between the m1 first connectors and the nth table.

[0127] Optionally, the determining module 403 further includes:

[0128] The maintenance submodule is used to maintain the association between the m first connectors and the nth table when the number of second data associated with the nth table is equal to the upper limit of the total number of second data transmitted by the m first connectors.

[0129] Optionally, the device further includes:

[0130] A first marking module is used to determine a first mark of the second data based on the attribute information of the second data. The first mark is used to represent the target table corresponding to the second data in the N tables.

[0131] The second marking module is used to determine a second mark of the second data based on the connector associated with the target table, wherein the second mark is used to represent the connector that transmits the second data;

[0132] Storage module 404 includes:

[0133] The third determining submodule is used to determine the target table for each of the second data associations in the N tables based on the first marker.

[0134] The fourth determining submodule is used to determine each of the second data association connectors in the connectors associated with the target table based on the second mark;

[0135] The storage submodule is used to store each piece of the second data into the target table associated with each piece of the second data through the connector associated with each piece of the second data.

[0136] The data generation device 400 provided in this embodiment of the invention can achieveFigure 1 The various processes implemented in the method embodiments shown are capable of achieving the same beneficial effects, and will not be described again here to avoid repetition.

[0137] This invention also provides an electronic device, such as... Figure 5 As shown, it includes a processor 501, a communication interface 502, a memory 503, and a communication bus 504, wherein the processor 501, the communication interface 502, and the memory 503 communicate with each other through the communication bus 504.

[0138] The memory 503 is used to store computer programs; the processor 501, when executing the program stored in the memory 503, performs the following steps:

[0139] Obtain the full dataset to be analyzed, which includes multiple first data entries;

[0140] For each piece of first data, feature extraction is performed to obtain the corresponding second data.

[0141] Based on the number of second data associated with each of the N pre-defined tables, determine the connector associated with each table, where N is an integer greater than 1;

[0142] Based on the association between any target table among the N tables and the connector, the second data associated with the target table is stored in the target table.

[0143] Optionally, determining the connector associated with each table based on the number of second data associated with each of the preset N tables includes:

[0144] At preset time intervals, retrieve the number of second data items associated with the nth table out of N preset tables.

[0145] If the number of second data associated with the nth table is greater than the upper limit of the total number of second data transmitted by the m first connectors, a target connector is determined among the M connectors excluding the m first connectors. The m first connectors are the connectors currently associated with the nth table among the M connectors, where n is a positive integer less than or equal to N, M is an integer greater than 1, and m is a positive integer less than M.

[0146] Establish the association between the target connector and the nth table.

[0147] Optionally, when the number of second data associated with the nth table is greater than the upper limit of the total number of second data transmitted by the m first connectors, determining the target connector among the M connectors excluding the m first connectors includes at least one of the following:

[0148] If the number of second data associated with the nth table is greater than the upper limit of the total number of second data transmitted by the m first connectors, and the difference between the number of second data associated with the nth table and the upper limit is greater than or equal to a first threshold, a second connector other than the m first connectors is determined among the M connectors. The second connector is a connector that is not associated with any of the N tables, and the target connector includes the second connector.

[0149] If the number of second data associated with the nth table is greater than the upper limit of the total number of second data transmitted by the m first connectors, and the difference between the number of second data associated with the nth table and the upper limit is greater than or equal to a second threshold, a third connector other than the m first connectors is determined among the M connectors. The third connector is a connector among the N tables associated with at least one table other than the nth table. The target connector includes the third connector, and the second threshold is less than or equal to the first threshold.

[0150] Optionally, determining a third connector from the M connectors, excluding the m first connectors, includes:

[0151] Obtain the number of second data associated with the q-th table, where the q-th table is any table among the N tables except the n-th table, and q is a positive integer less than or equal to N;

[0152] If the amount of second data associated with the q-th table is less than the upper limit of the total amount of the second data transmitted by the third connector, a third connector is determined from the M connectors, excluding the m first connectors. The third connector is the connector associated with the q-th table.

[0153] Optionally, determining the connector associated with each table based on the number of second data associated with each of the preset N tables further includes:

[0154] If the number of second data associated with the nth table is less than the upper limit of the total number of second data transmitted by the m first connectors, then m1 first connectors are determined from the m first connectors, where m1 is a positive integer less than m;

[0155] Cancel the association between the m1 first connectors and the nth table.

[0156] Optionally, determining the connector associated with each table based on the number of second data associated with each of the preset N tables further includes:

[0157] If the number of second data associated with the nth table is equal to the upper limit of the total number of second data transmitted by the m first connectors, the association between the m first connectors and the nth table is maintained.

[0158] Optionally, when executing a program stored in memory 503, processor 501 may also perform the following steps:

[0159] A first label for the second data is determined based on the attribute information of the second data. The first label is used to represent the target table corresponding to the second data in the N tables.

[0160] A second marker for the second data is determined based on the connector associated with the target table, and the second marker is used to represent the connector that transmits the second data;

[0161] The step of storing the second data associated with any target table in the target table into the target table based on the association relationship between any target table in the N tables and the connector includes:

[0162] Based on the first marker, determine the target table for each of the N tables associated with the second data;

[0163] Each connector associated with the second data is determined in the connectors associated with the target table based on the second mark;

[0164] Each piece of the second data is stored in the target table associated with each piece of the second data through the connector associated with each piece of the second data.

[0165] The communication bus mentioned in the above electronic devices can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. This communication bus can be divided into address bus, data bus, control bus, etc. For ease of illustration, only one thick line is used to represent it in the diagram, but this does not indicate that there is only one bus or one type of bus.

[0166] The communication interface is used for communication between the aforementioned terminal and other devices.

[0167] The memory may include random access memory (RAM) or non-volatile memory, such as at least one disk storage device. Optionally, the memory may also be at least one storage device located remotely from the aforementioned processor.

[0168] The processors mentioned above can be general-purpose processors, including central processing units (CPUs), network processors (NPs), etc.; they can also be digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components.

[0169] In another embodiment of the present invention, a computer-readable storage medium is also provided, which stores instructions that, when executed on a computer, cause the computer to perform any of the data generation methods described in the above embodiments.

[0170] In another embodiment of the present invention, a computer program product containing instructions is also provided, which, when run on a computer, causes the computer to perform any of the data generation methods described in the above embodiments.

[0171] In the above embodiments, implementation can be achieved entirely or partially through software, hardware, firmware, or any combination thereof. When implemented using software, it can be implemented entirely or partially in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the processes or functions described in the embodiments of the present invention are generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wired (e.g., coaxial cable, fiber optic, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that integrates one or more available media. The available medium can be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., DVD), or a semiconductor medium (e.g., solid state disk (SSD)).

[0172] It should be noted that, in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0173] The various embodiments in this specification are described in a related manner. Similar or identical parts between embodiments can be referred to mutually. Each embodiment focuses on describing the differences from other embodiments. In particular, the system embodiments are basically similar to the method embodiments, so the description is relatively simple; relevant parts can be referred to the descriptions of the method embodiments.

[0174] The above description is merely a preferred embodiment of the present invention and is not intended to limit the scope of protection of the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention are included within the scope of protection of the present invention.

Claims

1. A data generation method, characterized in that, The method includes: Obtain the full dataset to be analyzed, which includes multiple first data entries; For each piece of first data, feature extraction is performed to obtain the corresponding second data. The connector associated with each table is determined based on the number of second data associated with each of the N preset tables, where N is an integer greater than 1; wherein the N preset tables include: TV channel table, children's channel table and variety channel table. Based on the association between any target table among the N tables and the connector, the second data associated with the target table is stored in the target table; The step of determining the connector associated with each table based on the number of second data associated with each of the preset N tables includes: At preset time intervals, retrieve the number of second data items associated with the nth table out of N preset tables. If the number of second data associated with the nth table is greater than the upper limit of the total number of second data transmitted by the m first connectors, a target connector is determined among the M connectors excluding the m first connectors. The m first connectors are the connectors currently associated with the nth table among the M connectors, where n is a positive integer less than or equal to N, M is an integer greater than 1, and m is a positive integer less than M. Establish the association between the target connector and the nth table.

2. The method according to claim 1, characterized in that, When the number of second data associated with the nth table is greater than the upper limit of the total number of second data transmitted by the m first connectors, determining the target connector among the M connectors excluding the m first connectors includes at least one of the following: If the number of second data associated with the nth table is greater than the upper limit of the total number of second data transmitted by the m first connectors, and the difference between the number of second data associated with the nth table and the upper limit is greater than or equal to a first threshold, a second connector other than the m first connectors is determined among the M connectors. The second connector is a connector that is not associated with any of the N tables, and the target connector includes the second connector. If the number of second data associated with the nth table is greater than the upper limit of the total number of second data transmitted by the m first connectors, and the difference between the number of second data associated with the nth table and the upper limit is greater than or equal to a second threshold, a third connector other than the m first connectors is determined among the M connectors. The third connector is a connector among the N tables associated with at least one table other than the nth table. The target connector includes the third connector, and the second threshold is less than or equal to the first threshold.

3. The method according to claim 2, characterized in that, The step of determining a third connector from the M connectors, excluding the m first connectors, includes: Obtain the number of second data associated with the q-th table, where the q-th table is any table among the N tables except the n-th table, and q is a positive integer less than or equal to N; If the amount of second data associated with the q-th table is less than the upper limit of the total amount of the second data transmitted by the third connector, a third connector is determined from the M connectors, excluding the m first connectors. The third connector is the connector associated with the q-th table.

4. The method according to claim 1, characterized in that, The step of determining the connector associated with each table based on the number of second data associated with each of the preset N tables further includes: If the number of second data associated with the nth table is less than the upper limit of the total number of second data transmitted by the m first connectors, then m1 first connectors are determined from the m first connectors, where m1 is a positive integer less than m; Cancel the association between the m1 first connectors and the nth table.

5. The method according to claim 1, characterized in that, The step of determining the connector associated with each table based on the number of second data associated with each of the preset N tables further includes: If the number of second data associated with the nth table is equal to the upper limit of the total number of second data transmitted by the m first connectors, the association between the m first connectors and the nth table is maintained.

6. The method according to claim 1, characterized in that, After determining the connector associated with each table based on the number of second data associated with each table in the preset N tables, and before storing the second data associated with the target table into the target table based on the association relationship between any target table and the connector in the N tables, the method further includes: A first label for the second data is determined based on the attribute information of the second data. The first label is used to represent the target table corresponding to the second data in the N tables. A second marker for the second data is determined based on the connector associated with the target table, and the second marker is used to represent the connector that transmits the second data; The step of storing the second data associated with any target table in the target table into the target table based on the association relationship between any target table in the N tables and the connector includes: Based on the first marker, determine the target table for each of the N tables associated with the second data; Each connector associated with the second data is determined in the connectors associated with the target table based on the second mark; Each piece of the second data is stored in the target table associated with each piece of the second data through the connector associated with each piece of the second data.

7. A data generation apparatus, characterized in that, include: The acquisition module is used to acquire the full data to be analyzed, which includes multiple first data entries; The feature extraction module is used to extract features from each piece of first data to obtain the corresponding second data. The determining module is used to determine the connector associated with each table based on the number of second data associated with each table in a preset set of N tables, where N is an integer greater than 1; wherein the preset set of N tables includes: a TV channel table, a children's channel table, and a variety show channel table. The storage module is used to store the second data associated with the target table into the target table based on the association relationship between any target table and the connector among the N tables; The acquisition submodule is used to acquire the number of second data associated with the nth table in a preset N tables at preset time intervals; The first determining submodule is used to determine the target connector among the connectors other than the m first connectors among the M connectors when the number of second data associated with the nth table is greater than the upper limit of the total number of second data transmitted by the m first connectors. The m first connectors are the connectors currently associated with the nth table among the M connectors, where n is a positive integer less than or equal to N, M is an integer greater than 1, and m is a positive integer less than M. A submodule is created to establish the association between the target connector and the nth table.

8. An electronic device, characterized in that, It includes a processor, a communication interface, a memory, and a communication bus, wherein the processor, the communication interface, and the memory communicate with each other through the communication bus; Memory, used to store programs; A processor, when executing a program stored in memory, implements the method as described in any one of claims 1-6.

9. A readable storage medium having a program stored thereon, characterized in that, When the program is executed by the processor, it implements the method as described in any one of claims 1-6.

Images