Acquisition protocol multi-version application method and system

Abstract

The invention discloses an acquisition protocol multi-version application method and system, and relates to the technical field of big data acquisition, and the method comprises the steps: adding a version number identifier to an acquisition protocol and each field in the acquisition protocol; the collection SDK compares the version number of the collected data with the collection protocol metadata, marks a corresponding version number on a corresponding field and reports the version number; the cloud side correspondingly stores the field version numbers into a wide table database according to the field version numbers; and the application layer selects available version number field data as required. When an acquisition protocol is created or updated, all fields are numbered by independent versions, the versions are controlled to be fine-grained, and field data corresponding to the version numbers can be freely selected when using data is applied; unified processing is carried out according to the field version number during cloud cleaning and warehousing, iterative development does not need to be carried out on an old version cleaning and warehousing process, and manpower is saved; in the storage stage, other fields which are not modified and updated in the same protocol do not need to be additionally and repeatedly stored due to consideration of old version data value, so that the storage space is greatly saved.

Description

Technical Field

[0001] The present invention relates to the field of big data acquisition technology, and in particular to a method and system for applying multiple versions of an acquisition protocol. Background Art

[0002] In big data collection scenarios, such as those for TV terminals, the data reporting process involves multiple steps: first, operations personnel propose collection requirements, then product managers design the collection content based on these requirements, and the R&D team defines the collection protocol. TV terminal developers then develop a collection SDK based on the collection protocol. The SDK collects terminal operational data and uploads it to the cloud. Finally, data is stored through ETL processing. Collection protocols contain numerous fields to cover the collection content, and the number of protocols is typically quite large. When collection requirements change, the approach is typically to add a complete collection protocol or modify a few fields. However, when only a few fields are modified, it is difficult for TV terminals to fully update the collection SDK at once, resulting in multiple versions of the same protocol being reported. To address this issue, adaptation is required during the cloud-based ETL stage. Traditional approaches include: 1) manually writing code to correct data during the cleansing and storage phase; 2) discarding all old versions of data during the cleansing and storage phase; and 3) storing all versions of data in full, prioritizing the latest version when used, with older versions kept for archival purposes. All of the above solutions have significant drawbacks: 1) Manual maintenance of old versions increases labor costs as the number of protocols and versions increases; 2) Directly discarding old versions of data can lead to the loss of valuable information; 3) Storing all versions of data can lead to a surge in data volume, resulting in wasted storage space and a lack of assurance of data accuracy. Therefore, a method for collecting multiple versions of protocols is urgently needed that strikes a balance between resource consumption and management complexity to address these issues. Summary of the Invention

[0003] The purpose of the present invention is to provide a multi-version application method and system for an acquisition protocol, so as to solve the problems in the prior art of high labor costs, loss of valuable information or waste of storage space in the ETL stage when the acquisition protocol update only involves the adjustment of a small number of fields.

[0004] The present invention solves the above problems through the following technical solutions:

[0005] A method for applying multiple versions of an acquisition protocol, comprising:

[0006] Add version number identification to the acquisition protocol and each field in the acquisition protocol;

[0007] The collection SDK compares the version number of the collected data with the collection protocol metadata, adds the corresponding version number to the corresponding field, and then reports it to the cloud;

[0008] The cloud stores the data in a wide table database based on the field version number.

[0009] The application layer selects available version number field data as needed.

[0010] Furthermore, the adding of a version number identifier to the acquisition protocol and each field in the acquisition protocol specifically includes: adding a version number identifier to the acquisition protocol and each field in the acquisition protocol when creating or updating the acquisition protocol.

[0011] Furthermore, the collection SDK compares the version number of the collected data with the collection protocol metadata, adds the corresponding version number to the corresponding field, and then reports it to the cloud:

[0012] The terminal's collection SDK is split into the application collection SDK and the manager SDK. The application collection SDK collects data and sends it along with its own version number to the manager SDK. The manager SDK pulls the collection protocol metadata, compares the version numbers, adds the corresponding version numbers to the corresponding fields, and then reports it to the cloud.

[0013] Furthermore, before marking the corresponding field with the corresponding version number, redundant and repeated data is also removed.

[0014] Furthermore, the cloud stores the data in the wide table database according to the corresponding field version number as follows:

[0015] During the ETL stage, a key consisting of the field and version number is generated based on the version number. When data is entered into the database, a wide table database is used for storage. For the new version number field, a new field key_version is added to the wide table database and the processed new field content is stored. For the old version number field, the field content is stored in the existing corresponding version number field.

[0016] Furthermore, the application layer selects available version number field data as needed, specifically:

[0017] When configured to use data, query the collection protocol metadata and choose to use the corresponding version number field data based on business needs.

[0018] A multi-version application system for an acquisition protocol, comprising:

[0019] The metadata management platform is configured to assign a version number to the acquisition protocol when creating or updating the acquisition protocol, and to add a version number identifier to each field in the acquisition protocol;

[0020] The terminal is configured to split the collection SDK into the application SDK and the manager SDK. The application collection SDK collects data and sends it with its own version number to the manager SDK. The manager SDK pulls the collection protocol metadata, compares the version numbers, and marks the corresponding fields with the corresponding version numbers before reporting to the cloud.

[0021] The cloud is configured to perform targeted ETL based on the collected protocol metadata and reported field version numbers. Based on the version number, a key consisting of the field and the version number is generated. When data is entered into the database, a wide-table database is used for storage. For new version number fields, a new field key_version is added to the wide-table database and the processed new field content is stored. For old version number fields, the field content is stored in the existing corresponding version number field.

[0022] Furthermore, before marking the corresponding field with the corresponding version number, redundant and repeated data is also removed.

[0023] Furthermore, it also includes an application layer, which is configured to query the metadata management platform when using data and select and use corresponding version number field data from the cloud according to business needs.

[0024] Compared with the prior art, the present invention has the following advantages and beneficial effects:

[0025] (1) When creating or updating the acquisition protocol, the present invention marks all fields with independent version numbers, making version control more granular. When the application uses the data, it can freely select the field data with the corresponding version number; when cleaning and warehousing in the cloud, unified processing is performed according to the field version number, and there is no need to iteratively develop the cleaning and warehousing process of the old version, saving manpower; in the storage stage, there is no need to repeatedly store other fields in the same protocol that have not been modified or updated due to the value of the old version data, which greatly saves storage space.

[0026] (2) The present invention refines the version to each field. To reduce traffic consumption, the version number can be placed after the field name. In this way, when reporting, the version number can be judged according to the field name, and different processing can be performed.

[0027] (3) The present invention adds independent version identifiers to all fields of the acquisition protocol to achieve fine-grained version control, which fundamentally solves the problem in the existing technology that the version control granularity is too large and the storage model cannot take into account both new and old versions, resulting in the application layer having to choose to use the full amount of data of different versions, resulting in a waste of storage space. BRIEF DESCRIPTION OF THE DRAWINGS

[0028] Figure 1 This is a system principle block diagram of the present invention. DETAILED DESCRIPTION

[0029] The present invention will be further described in detail below with reference to the examples, but the embodiments of the present invention are not limited thereto.

[0030] Example 1:

[0031] A method for applying multiple versions of a collection protocol, comprising:

[0032] Step S1: During the acquisition protocol generation phase, an independent version number is set for each field in the acquisition protocol to implement field-level version management so as to accurately track the update iterations of each field.

[0033] Step S2: Determine the data storage format and store the fields in the form of "field name_version number" so that the stored data can intuitively reflect the correspondence between the fields and versions, which is convenient for subsequent retrieval and processing.

[0034] Step S3: When the acquisition protocol is updated, the version number information of the corresponding field is added to the acquisition protocol metadata. The acquisition protocol metadata completely records the protocol evolution process and provides an accurate version basis for data processing.

[0035] Step S4: The TV terminal SDK is split into the collection SDK for each application and the manager SDK for system collection and reporting: manager_sdk. The division of labor between the two is clarified. The collection SDK is responsible for data collection within the application, and the manager_sdk is responsible for protocol management and data integration and reporting.

[0036] Step S5: Each application APP collection SDK integrates the same protocol and allows different versions to exist. Manager_sdk obtains the latest collection protocol rules and corresponding version numbers from the cloud in real time to achieve dynamic adaptation of the collection protocol.

[0037] Step S6: The application collection SDK aggregates the collected data and sends it to manager_sdk. Manager_sdk integrates the collected fields with the collection protocol metadata obtained from the cloud, and adds a corresponding version number to each field to ensure that the version information is complete when the data is reported. Specifically:

[0038] After collecting data, it is sent to manager_sdk with its own version number. Manager_sdk is responsible for protocol management and data integration and reporting functions, including the manager pulling the collection protocol metadata from the cloud, comparing the collection SDK version number and the collection metadata version number (list), removing redundant and repeated data, and processing the reported fields accordingly. The latest version is not processed, and the older version adds a version number similar to price_v1.1.

[0039] Step S7: When performing cleaning calculations in the cloud, the storage table is automatically compared. If it is found that the field with the corresponding version number does not exist, the corresponding field is added to ensure the integrity of the data and avoid data loss due to version differences.

[0040] Step S8: When using data at the application layer, query and collect protocol metadata and select available version numbers based on business needs to achieve flexible call and accurate use of data.

[0041] The present invention constructs a fine-grained acquisition protocol version control mechanism based on adding field versions. On the one hand, when creating or updating an acquisition protocol, all fields are marked with independent version numbers. This approach is different from the conventional version marking of the entire acquisition protocol. It makes the version control more fine-grained. Fine-grained version control facilitates problem location and resolution, and application implementation is more flexible. Usually, the update of the protocol will only be a very small range of field changes. The present invention can avoid the storage of multiple versions of the entire protocol data or the discarding of all data of the old version due to a change in one field, thereby reducing storage costs. On the other hand, when in use, the corresponding version number field data can be freely selected, making the application implementation more flexible and more compatible. Related businesses can continue to run when not necessary without the need for subsequent iterations.

[0042] During cloud-based data cleaning and storage, data is uniformly processed based on field version numbers, eliminating the need for iterative development of the old version cleaning and storage process, reducing labor costs. Each update to the collection protocol typically affects only a small number of fields. Therefore, during storage, there's no need to duplicately store unmodified fields from the same protocol to preserve the value of old versions, significantly saving storage space.

[0043] This invention solves the problems existing in the traditional acquisition protocol version:

[0044] 1. Duplicate reporting: Each TV terminal's acquisition SDK operates independently, and even for the same acquisition protocol, different versions may be reported. This leads to duplicate reporting, wasting data traffic and local hardware resources.

[0045] 2. Each protocol version requires independent test cases and pipelines. ETL also needs to run in multiple versions. To maximize the value of data, a lot of manpower is required to make compatibility corrections before data is stored.

[0046] 3. Performance and Resource Loss: Running multiple data processing pipelines in parallel increases cluster resource utilization. Measured data shows that each additional protocol version increases Spark job memory consumption by 15%-20%. Regarding storage, while a full multi-version data storage strategy can preserve historical data, it increases HDFS storage capacity by 3-5 times.

[0047] 4. Application Issues: The addition, deletion, or format differences of fields across different protocol versions require the application layer to maintain multiple sets of parsing logic. For example, processing older protocols requires retaining compatibility logic for deprecated fields, while newer protocols require expanding field mapping rules, exponentially increasing code complexity. When the same business metric involves fields across multiple protocol versions (e.g., user behavior statistics fields are defined differently across different versions), the application layer must develop multiple sets of calculation rules.

[0048] Example 2:

[0049] Combined with attachment Figure 1 As shown, a multi-version application system for an acquisition protocol includes:

[0050] The metadata management platform is configured to assign a version number to the acquisition protocol when creating or updating the acquisition protocol, and to add a version number identifier to each field in the acquisition protocol;

[0051] The terminal is configured to split the collection SDK into the application SDK and the manager SDK. The application collection SDK collects data and sends it with its own version number to the manager SDK. The manager SDK pulls the collection protocol metadata, compares the version numbers, removes redundant data, and then marks the corresponding fields with the corresponding version numbers before reporting to the cloud.

[0052] The cloud is configured to perform targeted ETL based on the collected protocol metadata and reported field version numbers. Based on the version number, a key consisting of the field and the version number is generated. When data is entered into the database, a wide-table database is used for storage. For new version number fields, a new field key_version is added to the wide-table database and the processed new field content is stored. For old version number fields, the field content is stored in the existing corresponding version number field.

[0053] When the application layer is configured to use data, it queries the metadata management platform and selects and uses the corresponding version number field data from the cloud according to business needs.

[0054] Although the present invention is described herein with reference to illustrative embodiments of the present invention, the above embodiments are merely preferred embodiments of the present invention, and the embodiments of the present invention are not limited to the above embodiments. It should be understood that those skilled in the art can design many other modifications and implementations, which will fall within the scope and spirit of the principles disclosed in this application.

Claims

1. A method for applying multiple versions of a collection protocol, characterized in that: include: Add version number identification to the acquisition protocol and each field in the acquisition protocol; The collection SDK compares the version number of the collected data with the collection protocol metadata, adds the corresponding version number to the corresponding field, and then reports it to the cloud; The cloud stores the data in a wide table database based on the field version number. The application layer selects available version number field data as needed.

2. A method for applying multiple versions of a collection protocol according to claim 1, characterized in that: The adding of a version number identifier to the acquisition protocol and each field in the acquisition protocol specifically includes: adding a version number identifier to the acquisition protocol and each field in the acquisition protocol when creating or updating the acquisition protocol.

3. The method for applying multiple versions of a collection protocol according to claim 1, characterized in that: The collection SDK compares the version number of the collected data with the collection protocol metadata, adds the corresponding version number to the corresponding field, and then reports it to the cloud: The terminal's collection SDK is split into the application collection SDK and the manager SDK. The application collection SDK collects the data and sends it with its own version number to the manager SDK. The manager SDK pulls the collection protocol metadata, compares the version numbers, adds the corresponding version numbers to the corresponding fields, and then reports it to the cloud.

4. A method for applying multiple versions of a collection protocol according to claim 3, characterized in that: Before marking the corresponding fields with corresponding version numbers, the step also includes removing redundant and repeated data.

5. The method for applying multiple versions of a collection protocol according to claim 1, characterized in that: The cloud stores the field version number in the wide table database as follows: During the ETL stage, a key consisting of the field and version number is generated based on the version number. When data is entered into the database, a wide table database is used for storage. For the new version number field, a new field key_version is added to the wide table database and the processed new field content is stored. For the old version number field, the field content is stored in the existing corresponding version number field.

6. The method for applying multiple versions of a collection protocol according to claim 1, characterized in that: The application layer selects the available version number field data as needed: When configured to use data, query the collection protocol metadata and choose to use the corresponding version number field data based on business needs.

7. A multi-version application system for acquisition protocols, characterized in that: include: The metadata management platform is configured to assign a version number to the acquisition protocol when creating or updating the acquisition protocol, and to add a version number identifier to each field in the acquisition protocol; The terminal is configured to split the collection SDK into the application SDK and the manager SDK. The application collection SDK collects data and sends it with its own version number to the manager SDK. The manager SDK pulls the collection protocol metadata, compares the version numbers, and marks the corresponding fields with the corresponding version numbers before reporting to the cloud. The cloud is configured to perform targeted ETL based on the collected protocol metadata and reported field version numbers. Based on the version number, a key consisting of the field and the version number is generated. When data is entered into the database, a wide-table database is used for storage. For new version number fields, a new field key_version is added to the wide-table database and the processed new field content is stored. For old version number fields, the field content is stored in the existing corresponding version number field.

8. A collection protocol multi-version application system according to claim 7, characterized in that: Before marking the corresponding fields with corresponding version numbers, the step also includes removing redundant and repeated data.

9. A collection protocol multi-version application system according to claim 8, characterized in that: It also includes an application layer, which is configured to query the metadata management platform when using data and select and use corresponding version number field data from the cloud according to business needs.

Images