Data label processing method and device based on dimension conversion, equipment and medium

Abstract

The invention relates to the technical field of data processing, and provides a data label processing method and device based on dimension conversion, equipment and a medium, and the method comprises the steps: obtaining to-be-processed original data; extracting a plurality of target tags in the to-be-processed original data, and naming and classifying the plurality of target tags based on a preset tag system; performing naming and classification based on a preset tag system and the plurality of target tags, and constructing a transverse tag table by taking the user identity identifiers corresponding to the plurality of target tags as main keys and taking each target tag as a transverse field; performing dimension conversion on the data in the transverse label table, and constructing a longitudinal label table by taking user identities corresponding to a plurality of target labels as main keys and taking each target label as a longitudinal field; and performing cross aggregation calculation on the data in the longitudinal label table to group the target labels corresponding to the user identity identifier to obtain multiple groups of labels, and packaging the multiple groups of labels into objectified data to realize processing of the data labels.

Description

Technical Field

[0001] The present invention relates to the field of data processing technology, and in particular to a data label processing method, device, equipment and medium based on dimensional conversion. Background Art

[0002] In modern data-driven application scenarios, the effective management and utilization of user data has become a key challenge. In this era of traffic, the rough operation model can no longer meet the requirements of the times. The activity effect of the entire site needs to be improved, and efficient conversion and refined operation need to be achieved. The increase in this demand has prompted the exploration of more efficient and accurate extraction and processing methods for user tag information.

[0003] User labeling refers to labeling and classifying user behaviors, characteristics, preferences and other information through a certain mechanism to form labeled data that is easy to analyze. This process not only requires processing and storing a large amount of raw data, but also requires efficient computing and aggregation methods to convert raw data into structured labeled data.

[0004] However, existing labeling methods usually have the following problems: when labeling users, traditional methods usually adopt a vertical storage structure, which easily leads to inefficient query of label data when facing large-scale data; user label information needs to be able to support multi-dimensional query and analysis, but existing systems often lack a flexible data structure conversion mechanism, and the converted data has fewer dimensions, which in turn limits the use of label information; user label information usually needs to be exchanged and shared between different systems and platforms, but existing formatting and data conversion methods often lack standardization, affecting the cross-platform exchange and application of information.

[0005] Therefore, how to improve the query efficiency, data dimension and standardization of label data is an urgent problem that needs to be solved. Summary of the invention

[0006] The present invention intends to provide a data label processing method, device, equipment and medium based on dimensional conversion to solve the problems existing in the prior art on how to improve the query efficiency, data dimension and standardization of label data. The technical problem to be solved by the present invention is achieved through the following technical solution.

[0007] The present application embodiment provides a data label processing method based on dimensional conversion, the method comprising: Obtain the network environment information, hardware information and software information of the target network device; Obtaining raw data to be processed; Extracting multiple target tags from the raw data to be processed, and naming and classifying the multiple target tags based on a preset tag system; Based on the preset tag system and the multiple target tags, the user identity identifiers corresponding to the multiple target tags are used as primary keys, and each of the target tags is used as a horizontal field to construct a horizontal tag table; Perform dimension conversion on the data in the horizontal tag table, use the user identity corresponding to the multiple target tags as the primary key, and use each target tag as the vertical field to construct a vertical tag table; A cross-aggregation calculation is performed on the data in the vertical tag table to group the target tags corresponding to the user identity identifiers to obtain multiple groups of tags, and the multiple groups of tags are encapsulated into objectified data to achieve processing of data tags.

[0008] In one embodiment of the present application, the step of obtaining the raw data to be processed includes: Based on the microservice architecture, the interface services are orchestrated to obtain microservice instances; According to the microservice instance, collecting the raw data to be processed from a relational database; According to the microservice instance, a distributed data management architecture is adopted to collect the raw data to be processed from the log data source; According to the microservice instance, the raw data to be processed in the data source of the file type is collected and encrypted through the network security file transfer protocol; According to the microservice instance, the raw data to be processed is collected from an external system through an application programming interface, and the raw data to be processed is encrypted through the network security file transfer protocol.

[0009] In one embodiment of the present application, after obtaining the raw data to be processed, the method further includes: Setting the tag type, which includes fact tags, statistical tags, rule tags, and data mining tags. The fact tags indicate basic user information, the statistical tags indicate user behavior information, the rule tags indicate user characteristics and product usage, and the data mining tags indicate user preferences. Setting a tag naming rule, wherein the tag naming rule includes basic information classification, user identity, tag type number, and tag number; Constructing an initial tag metadata mapping table, wherein the initial tag metadata mapping table is used to store the metadata of the target tag in text form; Setting a naming rule for the initial tag table, where the naming rule for the initial tag table is used to standardize user portraits and functions corresponding to the initial tag table; The label system is obtained based on the label type, the label naming rule, the label metadata initial mapping table, and the naming rule of the initial label table.

[0010] In one embodiment of the present application, the cross-aggregation calculation of the data in the vertical label table includes: The cross-aggregation calculation is implemented for the data in the vertical tag table through data cleaning, data association, and data extraction to generate multiple key-value pairs, each of which corresponds to a target tag corresponding to a group of user identity identifiers.

[0011] In one embodiment of the present application, after encapsulating the multiple sets of tags into objectified data, the method further includes: The objectified data is injected into the tag metadata initial mapping table to obtain a tag metadata target mapping table, and the tag metadata target mapping table is used to perform complex query and screening operations according to the target field.

[0012] In one embodiment of the present application, the method further includes: The operations related to extracting multiple target tags from the raw data to be processed are encapsulated into a serial instruction set, and when extracting multiple target tags from the raw data to be processed, the running status of the serial instruction set is monitored in real time. If the serial instruction set fails to run or runs abnormally, an interrupt mechanism is triggered and feedback of operation failure or operation abnormality is provided.

[0013] In one embodiment of the present application, after encapsulating the multiple sets of tags into objectified data, the method further includes: Based on the preset search engine tool, an object-oriented data output interface is constructed through the hypertext transfer protocol; The objectified data is transmitted to the target client through the objectified data output interface, so that the target client can query the target tag and / or display the target tag based on the objectified data.

[0014] The embodiment of the present application further provides a data label processing device based on dimensional conversion, the device comprising: A data acquisition unit, used for acquiring raw data to be processed; A label extraction unit, used to extract multiple target labels from the raw data to be processed, and name and classify the multiple target labels based on a preset label system; A horizontal label table construction unit, configured to name and classify the plurality of target labels based on the preset label system, and to construct a horizontal label table with the user identity corresponding to the plurality of target labels as the primary key and each target label as a horizontal field; A vertical tag table construction unit is used to perform dimension conversion on the data in the horizontal tag table, and to construct a vertical tag table with the user identity corresponding to the multiple target tags as the primary key and each target tag as the vertical field; The label encapsulation unit is used to perform cross-aggregation calculation on the data in the vertical label table to group the target labels corresponding to the user identity to obtain multiple groups of labels, and encapsulate the multiple groups of labels into objectified data to realize the processing of data labels.

[0015] An embodiment of the present application further provides a computer-readable storage medium, wherein the computer-readable storage medium includes a stored computer program, wherein when the computer program is run, the data label processing method based on dimensional conversion as described in any one of the above items is executed.

[0016] An embodiment of the present application also provides an electronic device, including a memory and a processor, wherein the memory stores a computer program, and the processor is configured to execute the data label processing method based on dimensional conversion as described in any one of the above items through the computer program.

[0017] The embodiments of the present invention include the following advantages: The data label processing method, device, equipment and medium based on dimensional conversion provided by the embodiments of the present invention first obtain the original data to be processed; then extract multiple target labels in the original data to be processed, and name and classify the multiple target labels based on a preset label system; then name and classify the multiple target labels based on the preset label system, and use the user identity identifiers corresponding to the multiple target labels as the primary key, and use each of the target labels as the horizontal field to construct a horizontal label table; then perform dimensional conversion on the data in the horizontal label table, use the user identity identifiers corresponding to the multiple target labels as the primary key, and use each of the target labels as the vertical field to construct a vertical label table; finally, perform cross-aggregation calculation on the data in the vertical label table to group the target labels corresponding to the user identity identifiers to obtain multiple groups of labels, and encapsulate the multiple groups of labels into objectified data to realize the processing of data labels.

[0018] In the present invention, target tags that need attention are extracted from the original data, and are named and classified according to a preset tag system, which helps to ensure that all tags follow a unified standard and facilitates subsequent data management and analysis; a table is constructed with the user identity identifier as the primary key and each target tag as a horizontal field. The horizontal layout makes multi-dimensional information query for a specific user very direct and fast, reducing the number of data traversals during query; the horizontal tag table is converted into a form with the user identity identifier as the primary key and each target tag as the vertical field. Through the vertical layout, the relationship and pattern between different tags can be more easily discovered, which provides convenience for in-depth analysis; cross-aggregation calculations are performed on the data in the vertical tag table, and the same tags are grouped according to the user identity identifier through aggregation calculations, which can reduce the amount of data and optimize the storage structure. At the same time, the objectified data form is easier for program processing, which improves the speed and flexibility of data access. In addition, by constructing horizontal and vertical label tables, a data storage structure that combines horizontal and vertical data is formed, which not only retains the full picture of label information with user identity as the core dimension, but also realizes data reorganization and aggregation in multiple data dimensions, providing strong support for efficient query and in-depth analysis of data; by encapsulating multiple groups of labels into objectified data, the standardization of label data is improved, thereby enhancing the versatility and interchangeability of label data, laying a solid foundation for cross-platform circulation and sharing of data. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] The drawings described herein are used to provide a further understanding of the present application and constitute a part of the present application. The illustrative embodiments of the present application and their descriptions are used to explain the present application and do not constitute an improper limitation on the present application. In the drawings: Figure 1 is a schematic diagram of an application environment of a data label processing method based on dimension conversion shown in an exemplary embodiment of the present application; Figure 2 is a flowchart of a data label processing method based on dimension conversion shown in an exemplary embodiment of the present application; Figure 3 is a schematic diagram of a data label processing device based on dimension conversion shown in an exemplary embodiment of the present application; Figure 4 It is a schematic diagram of the structure of an electronic device shown in an exemplary embodiment of the present application. DETAILED DESCRIPTION

[0020] In order to enable those skilled in the art to better understand the solution of the present application, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present application.

[0021] It should be noted that the terms used herein are only for describing specific embodiments and are not intended to limit the exemplary embodiments according to the present application. As used herein, unless the context clearly indicates otherwise, the singular form is also intended to include the plural form. In addition, it should also be understood that when the terms "comprise" and / or "include" are used in this specification, it indicates the presence of features, steps, operations, devices, components and / or combinations thereof.

[0022] It should be noted that the terms "first", "second", etc. in the specification and claims of the present application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the terms used in this way can be interchangeable where appropriate, so that the embodiments of the present application described herein can be implemented in an order other than those illustrated or described herein.

[0023] In addition, the terms "include" and "have" and any variations thereof are intended to cover non-exclusive inclusions. For example, a process, method, system, product, or apparatus that includes a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units that are not explicitly listed or inherent to these processes, methods, products, or apparatuses.

[0024] In one embodiment of the present application, a data label processing method based on dimensional conversion is provided. Optionally, as an optional implementation, the data label processing method based on dimensional conversion can be applied to, but is not limited to, Figure 1 in the environment shown. Figure 1 is a schematic diagram of an application environment of a data label processing method based on dimension conversion shown in an exemplary embodiment of the present application, see Figure 1The data label processing terminal 101 may be, but is not limited to, a host computer, a desktop computer, a laptop computer, etc. with local computing capabilities. The data label processing terminal 101 may be, but is not limited to, communicating with the data acquisition terminal 102 and the client 103 through a network. The data acquisition terminal 102 and the client 103 may be, but are not limited to, performing operations on the database, such as writing data operations or reading data operations. The above-mentioned data label processing terminal 101 may be, but is not limited to, including a human-computer interaction screen, a processor, and a memory. The above-mentioned human-computer interaction screen may be, but is not limited to, used to display the process of data label processing based on dimensional conversion. The above-mentioned processor may be, but is not limited to, used to respond to the above-mentioned human-computer interaction operations, perform corresponding operations, or generate corresponding instructions, and send the generated instructions to the client 103. The above-mentioned memory is used to store relevant data, such as raw data to be processed.

[0025] As an optional method, the following steps in the data label processing method based on dimension conversion may be performed on the data label processing terminal 101: Obtaining raw data to be processed; Extracting multiple target tags from the raw data to be processed, and naming and classifying the multiple target tags based on a preset tag system; Based on the preset tag system and the multiple target tags, the user identity identifiers corresponding to the multiple target tags are used as primary keys, and each of the target tags is used as a horizontal field to construct a horizontal tag table; Perform dimension conversion on the data in the horizontal tag table, use the user identity corresponding to the multiple target tags as the primary key, and use each target tag as the vertical field to construct a vertical tag table; A cross-aggregation calculation is performed on the data in the vertical tag table to group the target tags corresponding to the user identity identifiers to obtain multiple groups of tags, and the multiple groups of tags are encapsulated into objectified data to achieve processing of data tags.

[0026] As an optional method, the client 103 can obtain the objectified data transmitted by the data tag processing terminal 101 and then perform subsequent processing.

[0027] The above method extracts the target tags that need attention from the original data, and names and classifies them according to the preset tag system, which helps to ensure that all tags follow a unified standard and facilitates subsequent data management and analysis; the user identity is the primary key, and each target tag is used as a horizontal field to build a table. The horizontal layout makes multi-dimensional information query for a specific user very direct and fast, reducing the number of data traversals during query; the horizontal tag table is converted into a form with the user identity as the primary key and each target tag as the vertical field. Through the vertical layout, the relationship and pattern between different tags can be more easily discovered, which facilitates in-depth analysis; cross-aggregation calculations are performed on the data in the vertical tag table, and the same tags are grouped according to the user identity through aggregation calculations, which can reduce the amount of data and optimize the storage structure. At the same time, the objectified data form is easier for program processing, which improves the speed and flexibility of data access. In addition, by constructing horizontal and vertical label tables, a data storage structure that combines horizontal and vertical data is formed, which not only retains the full picture of label information with user identity as the core dimension, but also realizes data reorganization and aggregation in multiple data dimensions, providing strong support for efficient query and in-depth analysis of data; by encapsulating multiple groups of labels into objectified data, the standardization of label data is improved, thereby enhancing the versatility and interchangeability of label data, laying a solid foundation for cross-platform circulation and sharing of data.

[0028] Optionally, in this embodiment, the data tag processing terminal 101 may be a terminal device configured with a target client, which may include but is not limited to at least one of the following: a laptop, a tablet computer, a PDA, a desktop computer, etc. The target client may be a client that supports providing a data tag processing application based on dimension conversion, such as a video client, an instant messaging client, a browser client, an education client, etc. The above network may include but is not limited to: a wired network, a wireless network, wherein the wired network includes: a local area network, a metropolitan area network and a wide area network, and the wireless network includes: Bluetooth, WIFI (wireless network communication technology) and other networks that realize wireless communication.

[0029] Optionally, in this embodiment, the data tag processing terminal 101 may be a server configured with a target client. The server may be a single server, or a server cluster consisting of multiple servers, or a cloud server. The above is only an example, and this embodiment does not make any limitation to this.

[0030] As an optional example, this embodiment does not limit the execution subject of the above-mentioned data label processing method based on dimensional conversion. Some or all steps of the above-mentioned data label processing method based on dimensional conversion can be executed on the data label processing terminal 101. For example, when the data label processing terminal 101 is a desktop computer, some or all steps of the above-mentioned data label processing method based on dimensional conversion can be executed on the desktop computer.

[0031] In one embodiment of the present application, a data label processing method based on dimensionality conversion is provided. Figure 2 is a flowchart of a data label processing method based on dimension conversion shown in an exemplary embodiment of the present application, see Figure 2 The data label processing method based on dimension conversion includes the following steps S210 to S250: In step S210, original data to be processed is obtained.

[0032] When obtaining the raw data to be processed, the data sources that need to be analyzed and processed are collected. These data may come from different channels, such as user behavior records, transaction records, etc.

[0033] In step S220, a plurality of target tags are extracted from the raw data to be processed, and the plurality of target tags are named and classified based on a preset tag system.

[0034] Exemplarily, valuable information points (i.e., target tags) are identified from the raw data, and these tags are named and classified according to a predefined tag system.

[0035] In step S230, based on the preset label system and the multiple target labels, the labels are named and classified, the user identities corresponding to the multiple target labels are used as primary keys, and each target label is used as a horizontal field to construct a horizontal label table.

[0036] For example, a two-dimensional table is created using the user identity as the primary key and each target tag as a column. This can clearly display the specific values ​​of each user on each tag, facilitating preliminary data viewing and filtering.

[0037] In step S240, the data in the horizontal tag table is dimensionally converted, and the vertical tag table is constructed by taking the user identity identifiers corresponding to the multiple target tags as the primary key and each target tag as the vertical field.

[0038] For example, the data in the horizontal tag table is reorganized so that each tag becomes a row, and the user identity remains unchanged. This can better support in-depth analysis of the data, such as statistical distribution of specific tags in user groups.

[0039] In step S250, cross-aggregation calculation is performed on the data in the vertical tag table to group the target tags corresponding to the user identity to obtain multiple groups of tags, and the multiple groups of tags are encapsulated into objectified data to realize the processing of data tags.

[0040] Exemplarily, an aggregation operation is performed on the vertical tag table, such as counting the frequency of occurrence of each tag by user group, etc. Then, the aggregation result is encapsulated into an easy-to-process data object to facilitate application program calling and further processing.

[0041] By adopting the above-mentioned embodiments provided by the present application, target tags that need attention are extracted from the original data, and they are named and classified according to the preset tag system, which helps to ensure that all tags follow a unified standard and facilitates subsequent data management and analysis; the user identity identifier is used as the primary key, and each target tag is used as a horizontal field to construct a table. The horizontal layout makes multi-dimensional information query for a specific user very direct and fast, reducing the number of data traversals during query; the horizontal tag table is converted into a form with the user identity identifier as the primary key and each target tag as the vertical field. Through the vertical layout, the relationship and pattern between different tags can be more easily discovered, which facilitates in-depth analysis; cross-aggregation calculations are performed on the data in the vertical tag table, and the same tags are grouped according to the user identity identifier through aggregation calculations, which can reduce the amount of data and optimize the storage structure. At the same time, the objectified data form is easier for program processing, which improves the speed and flexibility of data access. In addition, by constructing horizontal and vertical label tables, a data storage structure that combines horizontal and vertical data is formed, which not only retains the full picture of label information with user identity as the core dimension, but also realizes data reorganization and aggregation in multiple data dimensions, providing strong support for efficient query and in-depth analysis of data; by encapsulating multiple groups of labels into objectified data, the standardization of label data is improved, thereby enhancing the versatility and interchangeability of label data, laying a solid foundation for cross-platform circulation and sharing of data.

[0042] In one embodiment of the present application, the obtaining of raw data to be processed includes: Based on the microservice architecture, the interface services are orchestrated to obtain microservice instances; According to the microservice instance, collecting the raw data to be processed from a relational database; According to the microservice instance, a distributed data management architecture is adopted to collect the raw data to be processed from the log data source; According to the microservice instance, the raw data to be processed in the data source of the file type is collected and encrypted through the network security file transfer protocol; According to the microservice instance, the raw data to be processed is collected from an external system through an application programming interface, and the raw data to be processed is encrypted through the network security file transfer protocol.

[0043] Among them, based on the microservice architecture, the interface service is orchestrated to obtain a microservice instance. The microservice architecture allows a large application to be split into a group of small, autonomous services, each of which runs in an independent process and communicates using a lightweight communication mechanism. According to the microservice instance, the raw data to be processed is collected from a relational database. Relational databases are the main way to store structured data, and query languages ​​such as SQL (Structured Query Language) can efficiently extract the required data. According to the microservice instance, a distributed data management architecture is used to collect the raw data to be processed from the log data source. The distributed data management architecture can process massive data and provide efficient data storage and analysis capabilities. According to the microservice instance, the raw data to be processed in the file type data source is collected and encrypted through a network security file transfer protocol (such as SFTP, FTPS, etc.), ensuring the security and integrity of the file during transmission. According to the microservice instance, the raw data to be processed is collected from the external system through an application programming interface (API), and the raw data to be processed is encrypted through a network security file transfer protocol, which allows the microservice to be seamlessly integrated with the external system while ensuring the security of data transmission.

[0044] For example, the data collection terminal in the embodiment of the present application has a high degree of adaptability and flexibility, and can support multi-touch data collection capabilities, covering APP (Application), H5 (HTML5, the fifth generation of hypertext markup language), SDK (Software Development Kit, a set of development toolkits) and other multiple touch points. Based on the microservice architecture, the data collection terminal arranges the interface services, generates multiple microservice instances, and realizes flexible service combination and scheduling.

[0045] For example, for data sources of relational databases such as MySQL (an open source relational database management system), Sqoop (SQL to Hadoop, a tool for transmitting relational database data) is used for efficient data extraction and transmission. By exporting data from the database and importing it into a data warehouse or distributed file system, the consistency and integrity of the data are ensured. In terms of processing log data sources, a distributed data management architecture built by Flume (a distributed, reliable, and highly available system for collecting, aggregating, and transmitting large amounts of log data) and Kafka (a distributed stream processing platform, mainly used to build real-time data pipelines and stream applications) is adopted. Flume is responsible for capturing data from log sources and efficiently transmitting it to Kafka. Kafka serves as an intermediate cache layer for stream data to ensure the real-time and scalability of log data. For data from file-type data sources, the SFTP protocol (a network security file transfer protocol) is used for encrypted transmission to ensure the security and confidentiality of data during transmission. SFTP provides a secure file transfer channel to protect file data from unauthorized access. In terms of the integration of external systems and data services, a secure and efficient data interaction method is provided through the API interface (application programming interface) combined with the SFTP encrypted transmission mechanism. The API interface is used for real-time data requests and interactions, while SFTP provides encryption protection for batch data transmission.

[0046] In one embodiment of the present application, after obtaining the raw data to be processed, the method further includes: Setting the tag type, which includes fact tags, statistical tags, rule tags, and data mining tags. The fact tags indicate basic user information, the statistical tags indicate user behavior information, the rule tags indicate user characteristics and product usage, and the data mining tags indicate user preferences. Setting a tag naming rule, wherein the tag naming rule includes basic information classification, user identity, tag type number, and tag number; Constructing an initial tag metadata mapping table, wherein the initial tag metadata mapping table is used to store the metadata of the target tag in text form; Setting a naming rule for the initial tag table, where the naming rule for the initial tag table is used to standardize user portraits and functions corresponding to the initial tag table; The label system is obtained based on the label type, the label naming rule, the label metadata initial mapping table, and the naming rule of the initial label table.

[0047] Exemplarily, factual tags represent objective and stable data features, such as basic information of users (such as name, gender, age, occupation, etc.). Factual tags are designed to capture static features of users and provide basic data support for subsequent data analysis. Statistical tags are derived from user behavior data, such as statistical information of user file storage in cloud disk business scenarios, or user purchase frequency, browsing time, click-through rate, etc. Statistical tags reflect user behavior patterns and data usage in specific business scenarios through real-time or periodic data collection. Rule tags extract characteristic tags and current product usage of users based on predefined rules. For example, rule tags can include user activity level, product usage frequency, etc., which are helpful for feature extraction and user behavior pattern recognition through rule engines. Data mining tags are used to reveal users' implicit preferences (such as hobbies, shopping preferences, content preferences) and potential values. They belong to the category of value tags. For example, through association analysis technology, the correlation of user function access is mined, and iterative optimization of function display is performed to generate personalized recommendation tags. In this embodiment, through clear classification of tag types, user data can be managed and analyzed more finely; it supports analysis of users from multiple dimensions to provide support for personalized recommendations and marketing strategies.

[0048] For example, when setting the tag naming rules, a structured naming method is used (such as "basic_u_01_XXX"), where "basic" represents the basic information classification, "u" represents user-related, "01" is the classification number, and "XXX" is the specific tag number. This naming method is not only intuitive and easy to understand, but also convenient for adding new tag categories through simple number extension, without the need to redesign the entire tag system, which greatly improves the flexibility and scalability of the system.

[0049] For example, when constructing the initial mapping table for label metadata, all label metadata (including label identifiers, names, descriptions, types, etc.) are stored in a specific table (such as dim.t_up_userlabel_attr_text) in Hive (a data warehouse tool for processing large-scale structured and semi-structured data) in text form. This design makes it possible to centrally manage label metadata, which facilitates subsequent data analysis, query, and synchronization.

[0050] Exemplarily, in order to support the smooth operation of the label system and its related data processing projects, this embodiment sets the naming rules of the initial label table. The naming of all tables that support the operation of the system must follow the format of "t_up_XXX", where "t" represents table, "up" is the abbreviation of "user_portrait", indicating user portrait related, and "XXX" is specifically named according to the content or function of the actual table to clearly express the purpose or content of the table. Through this naming specification, not only the uniformity and standardization of database table naming are guaranteed, but also it helps to improve the maintainability and readability of the system, laying a solid foundation for the subsequent development and optimization of the system.

[0051] In one embodiment of the present application, performing cross-aggregation calculation on the data in the vertical label table includes: The cross-aggregation calculation is implemented for the data in the vertical tag table through data cleaning, data association, and data extraction to generate multiple key-value pairs, each of which corresponds to a target tag corresponding to a group of user identity identifiers.

[0052] Among them, during data cleaning, duplicate records can be checked and deleted to ensure that each record is unique, records with missing values ​​can be filled or deleted to ensure data integrity, erroneous data can be detected and corrected, such as outliers, format errors, etc., and data formats can be unified to ensure data consistency, such as date format, numerical units, etc. Through data cleaning, the quality of data is improved, the accuracy of subsequent analysis is ensured, invalid and erroneous data can be removed, and the impact of noise on analysis results can be reduced.

[0053] When associating data, the user identities in different tag tables are associated to ensure that the data of the same user in different tag tables can be matched; the relationships between different tags are associated, such as associating user basic information tags with behavior information tags, to generate a more comprehensive user portrait; if the data has a timestamp, it can be associated in chronological order to generate time series data. Through data association, the integrity of the data is ensured and multi-dimensional analysis is supported; data from different sources are associated to generate a more comprehensive user portrait and support comprehensive analysis.

[0054] When extracting data, select the most meaningful feature tags for the analysis target and remove irrelevant or redundant tags; aggregate related tags to generate higher-level tags. For example, aggregate multiple behavior tags into a comprehensive behavior score; use dimensionality reduction technology to reduce the dimension of the data and improve computing efficiency. Through feature selection, focus on the data that is most meaningful to the analysis target and improve the pertinence of the analysis; through data aggregation and dimensionality reduction, simplify the data structure and improve computing efficiency and model performance.

[0055] Among them, when performing cross-aggregation calculations, the data is grouped according to the user identity, and the aggregate value of each user on different tags is calculated. Commonly used aggregation functions include sum, average, maximum, minimum, etc.; aggregation is performed on multiple dimensions to generate multi-dimensional aggregation results. For example, cross-aggregation is performed by user age and gender to generate user behavior characteristics of different age groups and genders; the aggregation results are generated into key-value pairs, where the key is the user identity and the value is a set of aggregated tag values. Through multi-dimensional aggregation, user data can be analyzed from multiple angles to generate richer insights; data in key-value pair format is generated to facilitate subsequent efficient query and analysis; through aggregation calculations, a large amount of raw data can be compressed into more representative aggregation results, reducing the cost of data storage and transmission.

[0056] Exemplarily, data is stored in a predefined tag table in an orderly manner. The tag table uses the user's unique identifier id-map as the primary key, and each defined tag is used as an independent field to construct a horizontal tag table, whose horizontal axis shows a rich tag dimension. During the data storage process, the tag data that meets the business rules is calculated and packaged and stored in the corresponding field. For occasions that do not meet the rules, NULL values ​​are used to fill in. In this way, each user corresponds to a row of data, forming a one-dimensional data representation.

[0057] Exemplarily, based on the data of the one-dimensional horizontal tag table, a vertical data table is formed through dimension conversion processing. In the vertical tag table, id-map is also used as the primary key to ensure data consistency. The valid tag information in the horizontal tag table is extracted and mapped to the vertical tag table. Each user and each corresponding valid tag information constitutes a row of data in the vertical tag table. When a user has multiple valid tags, this information will exist in the vertical tag table in the form of multiple rows.

[0058] For example, in the final stage of data processing, cross-aggregation calculations are performed based on the data in the vertical tag table, grouped according to the user id-map, and the tag data information of each user is concatenated. Ultimately, this information is converted into a JSON object (including a set of unordered key-value pairs), forming a data structure containing multi-dimensional tag information, providing rich data support for complex data analysis and business decisions.

[0059] In this embodiment, based on the constraints of the tag system construction, the tag data mainly adopts a horizontal storage architecture to store the designed tag set in a structured manner. By performing operations such as data connection, aggregation, and data fusion, the tag annotation process based on the user ID mapping as the core key value is realized. This process generates a horizontal tag table, which supports multi-dimensional query functions and provides users with rich tag information.

[0060] In this embodiment, based on the horizontal label table, the coordinate transformation of the data dimension is performed to construct a vertical data structure oriented to the label dimension, namely the vertical label table. This data structure mainly transforms the coordinates of user information in the label dimension, providing structured source data for subsequent self-service data extraction and information retrieval.

[0061] In this embodiment, based on the vertical tag table, the tag information is encapsulated into a JSON object through data format conversion and data aggregation. This step converts the abstract user tag into a specific object representation, realizes capability encapsulation, aggregates the information of a single user, facilitates information exchange between different platforms, and realizes the digital empowerment of data exchange.

[0062] For example, in the specific practice based on cloud disk business, the factual data tags include user basic information, user channel information, user behavior tags and user data storage status tags. After completing the design of the data structure, after the data flow conversion and data processing, objectified user tag information is formed to form valuable data assets, and the HTTP protocol (HyperText Transfer Protocol) is supported for data extraction to achieve the output and docking of data capabilities; in terms of visualization, it supports the acquisition of portrait information of a single user, so as to handle work related to customer service complaints; at the same time, in terms of data statistics and self-service data extraction, labels can be used to complete the implementation of precision marketing strategies and improve the accuracy of marketing plans.

[0063] In one embodiment of the present application, after encapsulating the multiple groups of tags into objectified data, the method further includes: The objectified data is injected into the tag metadata initial mapping table to obtain a tag metadata target mapping table, and the tag metadata target mapping table is used to perform complex query and screening operations according to the target field.

[0064] When injecting objectified data into the tag metadata initial mapping table, ensure that the objectified data has been aggregated according to the user identity and generated data in the key-value pair format. Ensure that the tag metadata initial mapping table already contains metadata information for all tags, including tag name, tag type, tag description, data type, data source, etc. Then, match each key-value pair in the objectified data with the tag information in the tag metadata initial mapping table; inject the matched tag information and the corresponding value into the tag metadata initial mapping table to generate the tag metadata target mapping table.

[0065] When generating the tag metadata target mapping table, all key-value pairs in the objectified data are integrated with the tag information in the tag metadata initial mapping table; ensuring that all tag information corresponding to each user identity is reflected in the target mapping table. The generated tag metadata target mapping table is verified to ensure the integrity and accuracy of the data; check whether there is any missing tag information or incorrect data value; finally, the tag metadata target mapping table is stored in a structured data format to facilitate subsequent query and analysis.

[0066] Exemplarily, in the data retrieval stage, this embodiment introduces ElasticSearch (an open source search and analysis engine, which may be referred to as ES in the embodiment of this application) as a search engine. ElasticSearch is known for its efficient search speed and flexible query capabilities, and can support complex data retrieval requirements. Through data operations such as data cleaning, data association, and data extraction, the extracted data is cross-aggregated to generate a Map object with a key-value pair structure. Subsequently, during the data transfer process, the MapSet object is converted into a Json object format and stored in a data table. In the Hive environment, a data mapping table corresponding to Elasticsearch is established, and this process is called data mapping. By injecting data into the mapping table, data indexing and storage are achieved. Using the data structure defined by the mapping table, complex query and screening operations based on the target field can be performed on the data table. In order to meet the diverse query requirements, the data mapping table structure design provides a high degree of freedom, allowing customized data indexing strategies based on different field information to be implemented. This design supports the creation of multiple types of indexes, such as date indexes, user id-map indexes, etc., to optimize query performance and accuracy. In addition, through the two-dimensional data table format, the mapping table structure between Hive and ES can be flexibly designed. This design process is called data model design. This design not only supports the transformation of customized data query capabilities, but also realizes the free adaptation of diversified capability output forms. In this way, it provides strong data support and flexibility for data analysis and business decision-making, can quickly respond to various data query requests, realize rapid data location and extraction, and greatly improve the response speed and efficiency of data services.

[0067] In one embodiment of the present application, the method further includes: The operations related to extracting multiple target tags from the raw data to be processed are encapsulated into a serial instruction set, and when extracting multiple target tags from the raw data to be processed, the running status of the serial instruction set is monitored in real time. If the serial instruction set fails to run or runs abnormally, an interrupt mechanism is triggered and feedback of operation failure or operation abnormality is provided.

[0068] Exemplarily, the operations involving label extraction are encapsulated into a serial instruction set, and the running status of the program is monitored in real time during execution. If a failure or abnormal situation is detected, the interrupt mechanism will be automatically triggered and feedback will be provided to avoid the propagation of erroneous data and prevent it from affecting the subsequent data calculation chain. After completing the overall data construction task, the data integrity verification process is implemented. This process includes random sampling of the formed data and verifying the integrity of the data labels in each dimension table. Through this verification, the accuracy and consistency of the data during operation are ensured, thereby ensuring the quality of data output; the security of label data, the stability of the operating status, and the accuracy of data quality are monitored to ensure the stable operation of the entire data label processing system.

[0069] In one embodiment of the present application, after encapsulating the multiple groups of tags into objectified data, the method further includes: Based on the preset search engine tool, an object-oriented data output interface is constructed through the hypertext transfer protocol; The objectified data is transmitted to the target client through the objectified data output interface, so that the target client can query the target tag and / or display the target tag based on the objectified data.

[0070] For example, using the search engine tool ElasticSearch, the data output interface is realized through the HTTP protocol, which not only supports the visualization needs of the internal system, but also provides services for diverse operational functions, realizing the transformation of data value and data-driven decision support. In order to meet the needs of different users, two data service methods, Web interface (WebInterface, a standard method that allows communication between different software systems) and API interface, are provided. Through the data label display, business personnel can understand the people using the business. At the same time, API interface is also provided to access data, realize self-service query and download of data, and improve user experience and convenience.

[0071] It can be seen from the above embodiments that, through the method in this embodiment, the target tags that need to be paid attention to are extracted from the original data, and they are named and classified according to the preset tag system, which helps to ensure that all tags follow a unified standard and facilitate subsequent data management and analysis; the user identity is used as the primary key, and each target tag is used as a horizontal field to construct a table. The horizontal layout makes multi-dimensional information query for a specific user very direct and fast, reducing the number of data traversals during query; the horizontal tag table is converted into a form with the user identity as the primary key and each target tag as the vertical field. Through the vertical layout, the relationship and pattern between different tags can be more easily discovered, which provides convenience for in-depth analysis; the data in the vertical tag table is cross-aggregated and calculated, and the same tags are grouped according to the user identity through aggregation calculation, which can reduce the amount of data and optimize the storage structure. At the same time, the objectified data form is easier to process by the program, which improves the speed and flexibility of data access. In addition, by constructing horizontal and vertical label tables, a data storage structure that combines horizontal and vertical data is formed, which not only retains the full picture of label information with user identity as the core dimension, but also realizes data reorganization and aggregation in multiple data dimensions, providing strong support for efficient query and in-depth analysis of data; by encapsulating multiple groups of labels into objectified data, the standardization of label data is improved, thereby enhancing the versatility and interchangeability of label data, laying a solid foundation for cross-platform circulation and sharing of data.

[0072] In one embodiment of the present application, a data label processing device based on dimensional conversion is also provided. Figure 3 is a schematic diagram of a data label processing device based on dimension conversion shown in an exemplary embodiment of the present application, see Figure 3 , the device comprises: The data acquisition unit 301 is used to acquire the raw data to be processed; The tag extraction unit 302 is used to extract multiple target tags from the raw data to be processed, and name and classify the multiple target tags based on a preset tag system; A horizontal label table construction unit 303 is used to name and classify the plurality of target labels based on the preset label system, and to construct a horizontal label table with the user identity corresponding to the plurality of target labels as the primary key and each target label as a horizontal field; A vertical tag table construction unit 304 is used to perform dimension conversion on the data in the horizontal tag table, and to construct a vertical tag table by using the user identity corresponding to the multiple target tags as the primary key and each target tag as a vertical field; The tag encapsulation unit 305 is used to perform cross-aggregation calculation on the data in the vertical tag table to group the target tags corresponding to the user identity to obtain multiple groups of tags, and encapsulate the multiple groups of tags into objectified data to realize the processing of data tags.

[0073] The data label processing device based on dimensional conversion in the embodiment of the present application extracts target labels that need attention from the original data, and names and classifies them according to a preset label system, which helps to ensure that all labels follow a unified standard and facilitates subsequent data management and analysis; uses the user identity identifier as the primary key and each target label as a horizontal field to construct a table. The horizontal layout makes multi-dimensional information query for a specific user very direct and fast, reducing the number of data traversals during query; converts the horizontal label table into a form with the user identity identifier as the primary key and each target label as the vertical field. Through the vertical layout, it is easier to discover the relationship and pattern between different labels, which provides convenience for in-depth analysis; cross-aggregation calculations are performed on the data in the vertical label table, and the same labels are grouped according to the user identity identifier through aggregation calculations, which can reduce the amount of data and optimize the storage structure. At the same time, the objectified data form is easier for program processing, which improves the speed and flexibility of data access. In addition, by constructing horizontal and vertical label tables, a data storage structure that combines horizontal and vertical data is formed, which not only retains the full picture of label information with user identity as the core dimension, but also realizes data reorganization and aggregation in multiple data dimensions, providing strong support for efficient query and in-depth analysis of data; by encapsulating multiple groups of labels into objectified data, the standardization of label data is improved, thereby enhancing the versatility and interchangeability of label data, laying a solid foundation for cross-platform circulation and sharing of data.

[0074] The specific embodiments of the data label processing device based on dimensional conversion in the present application can refer to the examples shown in the above-mentioned data label processing method based on dimensional conversion, which will not be repeated here in this example.

[0075] In one embodiment of the present application, an electronic device for implementing the above-mentioned data label processing method based on dimensional conversion is also provided. The electronic device includes a memory and a processor, the memory stores a computer program, and the processor is configured to execute the above-mentioned data label processing method based on dimensional conversion through the computer program.

[0076] See also Figure 4 , Figure 44 is a schematic diagram of the structure of an electronic device shown in an exemplary embodiment of the present application. The computer system 400 includes a central processing unit (CPU) 401, which can perform various appropriate actions and processes according to the program stored in the read-only memory (ROM) 402 or the program loaded from the storage part 408 to the random access memory (RAM) 403, such as executing the method described in the above embodiment. In RAM 403, various programs and data required for system operation are also stored. CPU 401, ROM 402 and RAM 403 are connected to each other through bus 404. Input / output (I / O) interface 405 is also connected to bus 404.

[0077] The following components are connected to the I / O interface 405: an input section 406 including a keyboard, a mouse, etc.; an output section 407 including a cathode ray tube (CRT), a liquid crystal display (LCD), etc., and a speaker, etc.; a storage section 408 including a hard disk, etc.; and a communication section 409 including a network interface card such as a LAN (Local Area Network) card, a modem, etc. The communication section 409 performs communication processing via a network such as the Internet. A drive 410 is also connected to the I / O interface 405 as needed. A removable medium 411, such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, etc., is installed on the drive 410 as needed so that a computer program read therefrom is installed into the storage section 408 as needed.

[0078] In particular, according to an embodiment of the present application, the process described above with reference to the flowchart can be implemented as a computer software program. For example, an embodiment of the present application includes a computer program product, which includes a computer program carried on a computer-readable medium, and the computer program includes a computer program for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from a network through the communication part 409, and / or installed from a removable medium 411. When the computer program is executed by the central processing unit (CPU) 401, various functions defined in the system of the present application are executed.

[0079] It should be noted that the computer-readable medium shown in the embodiment of the present application may be a computer-readable signal medium or a computer-readable storage medium or any combination of the above two. The computer-readable storage medium may be, for example, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM), a flash memory, an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the present application, a computer-readable signal medium may include a data signal propagated in a baseband or as part of a carrier, which carries a computer-readable computer program. This propagated data signal may take a variety of forms, including but not limited to an electromagnetic signal, an optical signal, or any suitable combination of the above. A computer-readable signal medium may also be any computer-readable medium other than a computer-readable storage medium, which may send, propagate or transmit a program for use by or in conjunction with an instruction execution system, apparatus or device. A computer program contained on a computer-readable medium may be transmitted using any appropriate medium, including but not limited to: wireless, wired, etc., or any suitable combination of the above.

[0080] The flowchart and block diagram in the accompanying drawings illustrate the possible architecture, functions and operations of the system, method and computer program product according to various embodiments of the present application. Wherein, each box in the flowchart or block diagram can represent a module, a program segment, or a part of the code, and the above-mentioned module, program segment, or a part of the code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a different order from the order marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram or flowchart, and the combination of boxes in the block diagram or flowchart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0081] The units involved in the embodiments described in this application may be implemented by software or hardware, and the units described may also be set in a processor. The names of these units do not, in some cases, constitute limitations on the units themselves.

[0082] Another aspect of the present application further provides a computer-readable storage medium, the computer-readable storage medium including a stored computer program, wherein the computer program executes the above-mentioned data label processing method based on dimensional conversion when it is run. The computer-readable storage medium may be included in the electronic device described in the above embodiment, or may exist independently without being assembled into the electronic device.

[0083] Another aspect of the present application also provides a computer program product or a computer program, which includes a computer instruction stored in a computer-readable storage medium. A processor of a computer device reads the computer instruction from the computer-readable storage medium, and the processor executes the computer instruction, so that the computer device executes the data label processing method based on dimensional conversion provided in each of the above embodiments.

[0084] In the above detailed description, reference is made to the accompanying drawings, which form a part of this document. In the accompanying drawings, similar symbols typically identify similar components unless the context indicates otherwise. The illustrated embodiments described in the detailed description, drawings, and claims are not meant to be limiting. Other embodiments may be used, and other changes may be made, without departing from the spirit or scope of the subject matter presented herein.

[0085] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. For those skilled in the art, the present invention may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.

Claims

1. A data label processing method based on dimension conversion, characterized in that: The method comprises: Obtaining raw data to be processed; Extracting multiple target tags from the raw data to be processed, and naming and classifying the multiple target tags based on a preset tag system; Based on the preset tag system and the multiple target tags, the user identity identifiers corresponding to the multiple target tags are used as primary keys, and each of the target tags is used as a horizontal field to construct a horizontal tag table; Perform dimension conversion on the data in the horizontal tag table, use the user identity corresponding to the multiple target tags as the primary key, and use each target tag as the vertical field to construct a vertical tag table; A cross-aggregation calculation is performed on the data in the vertical tag table to group the target tags corresponding to the user identity identifiers to obtain multiple groups of tags, and the multiple groups of tags are encapsulated into objectified data to achieve processing of data tags.

2. The data label processing method based on dimension conversion according to claim 1, characterized in that: The obtaining of the raw data to be processed comprises: Based on the microservice architecture, the interface services are orchestrated to obtain microservice instances; According to the microservice instance, collecting the raw data to be processed from a relational database; According to the microservice instance, a distributed data management architecture is adopted to collect the raw data to be processed from the log data source; According to the microservice instance, the raw data to be processed in the data source of the file type is collected and encrypted through the network security file transfer protocol; According to the microservice instance, the raw data to be processed is collected from an external system through an application programming interface, and the raw data to be processed is encrypted through the network security file transfer protocol.

3. The data label processing method based on dimension conversion according to claim 1, characterized in that: After obtaining the raw data to be processed, the method further includes: Setting the tag type, which includes fact tags, statistical tags, rule tags, and data mining tags. The fact tags indicate basic user information, the statistical tags indicate user behavior information, the rule tags indicate user characteristics and product usage, and the data mining tags indicate user preferences. Setting a tag naming rule, wherein the tag naming rule includes basic information classification, user identity, tag type number, and tag number; Constructing an initial tag metadata mapping table, wherein the initial tag metadata mapping table is used to store the metadata of the target tag in text form; Setting a naming rule for the initial tag table, where the naming rule for the initial tag table is used to standardize user portraits and functions corresponding to the initial tag table; The label system is obtained based on the label type, the label naming rule, the label metadata initial mapping table, and the naming rule of the initial label table.

4. The data label processing method based on dimension conversion according to claim 1, characterized in that: The cross-aggregation calculation of the data in the vertical label table includes: The cross-aggregation calculation is implemented for the data in the vertical tag table through data cleaning, data association, and data extraction to generate multiple key-value pairs, each of which corresponds to a target tag corresponding to a group of user identity identifiers.

5. The data label processing method based on dimension conversion according to claim 1, characterized in that: After encapsulating the multiple groups of tags into objectified data, the method further includes: The objectified data is injected into the tag metadata initial mapping table to obtain a tag metadata target mapping table, and the tag metadata target mapping table is used to perform complex query and screening operations according to the target field.

6. The data label processing method based on dimension conversion according to claim 1, characterized in that: The method further comprises: The operations related to extracting multiple target tags from the raw data to be processed are encapsulated into a serial instruction set, and when extracting multiple target tags from the raw data to be processed, the running status of the serial instruction set is monitored in real time. If the serial instruction set fails to run or runs abnormally, an interrupt mechanism is triggered and feedback of operation failure or operation abnormality is provided.

7. The data label processing method based on dimension conversion according to claim 1, characterized in that: After encapsulating the multiple groups of tags into objectified data, the method further includes: Based on the preset search engine tool, an object-oriented data output interface is constructed through the hypertext transfer protocol; The objectified data is transmitted to the target client through the objectified data output interface, so that the target client can query the target tag and / or display the target tag based on the objectified data.

8. A data label processing device based on dimension conversion, characterized in that: The device comprises: A data acquisition unit, used for acquiring raw data to be processed; A label extraction unit, used to extract multiple target labels from the raw data to be processed, and name and classify the multiple target labels based on a preset label system; A horizontal label table construction unit, configured to name and classify the plurality of target labels based on the preset label system, and to construct a horizontal label table with the user identity corresponding to the plurality of target labels as the primary key and each target label as a horizontal field; A vertical tag table construction unit is used to perform dimension conversion on the data in the horizontal tag table, and to construct a vertical tag table with the user identity corresponding to the multiple target tags as the primary key and each target tag as the vertical field; The label encapsulation unit is used to perform cross-aggregation calculation on the data in the vertical label table to group the target labels corresponding to the user identity to obtain multiple groups of labels, and encapsulate the multiple groups of labels into objectified data to realize the processing of data labels.

9. A computer-readable storage medium, characterized in that: The computer-readable storage medium includes a stored computer program, wherein the computer program executes the data label processing method based on dimensional conversion according to any one of claims 1 to 7 when running.

10. An electronic device comprising a memory and a processor, characterized in that: A computer program is stored in the memory, and the processor is configured to execute the data label processing method based on dimensional conversion according to any one of claims 1 to 7 through the computer program.

Images