A data processing and analysis method for a building block big data platform

Abstract

The present invention provides a data processing and analysis method for a building block big data platform, which belongs to the field of big data processing technology, including: collecting user needs and user information data in multiple preset time periods in real time at client points; analyzing user needs in each preset time period to obtain first instructions and instruction deadlines for multiple users in the corresponding preset time period, and analyzing the first instructions and instruction deadlines to generate at least one second instruction and a sub-level coefficient for each second instruction; determining the business module and instruction type of each second instruction based on a preset instruction-module-type library, and determining the calling order of the second instruction in the business module according to the instruction type, thereby generating a calling process; dynamically updating the generated calling process based on instruction data of the next time period of the preset time period corresponding to the second instruction. Personalized analysis of the big data platform is realized.

Description

Technical Field

[0001] The present invention relates to the technical field of big data processing, and in particular to a data processing and analysis method for a building block type big data platform. Background Art

[0002] With the continuous increase of big data, traditional data processing and analysis methods can no longer meet the needs of real-time, personalization and intelligence. There are some problems with traditional big data processing and analysis methods, such as the inability to meet real-time requirements, lack of personalized analysis capabilities, lack of intelligent optimization and dynamic adjustment capabilities of the processing flow, etc. Therefore, it is necessary to study a new building block data processing and analysis method to meet the ever-changing user needs and improve the efficiency and quality of data processing and analysis.

[0003] Therefore, the present invention provides a data processing and analysis method for a building block type big data platform. Summary of the invention

[0004] The present invention provides a data processing and analysis method for a building block type big data platform, which realizes the real-time performance of data processing and analysis by collecting user needs and information data in real time at the client, analyzes user needs, obtains the first instruction and instruction deadline of each user, thereby realizing personalized analysis of user behavior, generates a call flow according to the instruction level coefficient and the call sequence, realizes intelligent optimization of the data processing flow, improves the efficiency of data processing and analysis, and dynamically updates the generated call flow by the instruction data of the next time period, thereby realizing dynamic adjustment of the data processing flow. It adapts to the dynamically changing data processing needs and improves the stability and flexibility of the system.

[0005] The present invention provides a data processing and analysis method of a building block type big data platform, comprising:

[0006] Step 1: Collect user needs and user information data in real time within multiple preset time periods at the client end;

[0007] Step 2: Analyze the user needs in each preset time period to obtain the first instructions and instruction deadlines of multiple users in the corresponding preset time period, and analyze the first instructions and instruction deadlines based on the user information data in the corresponding preset time period to generate at least one second instruction and a sub-level coefficient of each second instruction;

[0008] Step 3: Determine the business module and instruction type of each second instruction based on the preset instruction-module-type library, determine the calling order of the second instruction in the business module according to the instruction type, and generate a calling process based on the sub-level coefficient and the calling order;

[0009] Step 4: Dynamically update the generated call flow based on the instruction data of the next time period corresponding to the preset time period where the second instruction is located.

[0010] Preferably, the user needs and user information data within multiple preset time periods are collected in real time at the client embedding point, including:

[0011] Tracking is performed on the client's program interface and user interface, and the first tracking path of the user's clicks on the program interface and the second tracking path of the user's clicks on the user interface are captured;

[0012] The first burial point path and the second burial point path are respectively sent to the server for comprehensive analysis to obtain user needs and user information data.

[0013] Preferably, capturing the first buried path of a user's click on a program interface includes:

[0014] When the user clicks on the program interface, the tracking point label of the area corresponding to the center at the time of the click is pulled with the click position as the center;

[0015] Based on the tracking status of the tracking label, obtain the corresponding index path;

[0016] Determine the first burial path based on all index paths.

[0017] Preferably, capturing the second buried point path of the user clicking on the user interface includes:

[0018] Obtain static buried points and dynamic buried points within the click range of the user's click position on the user node;

[0019] Obtain a dynamic identifier corresponding to the dynamic burial point, and obtain a corresponding first pending path according to the dynamic identifier. At the same time, obtain a static identifier corresponding to the static burial point, and obtain a corresponding second pending path according to the static identifier.

[0020] Based on the first pending path and the second pending path, a second burial point path is constructed.

[0021] Preferably, generating at least one second instruction and a sub-level coefficient of each second instruction comprises:

[0022] Extract multiple demand features of user needs within each preset time period, and extract feature keywords of the demand features again to map them in a first preset word-instruction library to obtain a first instruction, and obtain an instruction deadline corresponding to the first instruction based on the first preset instruction information library;

[0023] Extracting user keywords from user information data corresponding to a preset time period, mapping the user keywords in a second preset word-instruction library to obtain an extended instruction, and expanding the first instruction based on the extended instruction to obtain a second instruction;

[0024] The sub-unit weight to which the second instruction belongs is determined based on the second preset instruction information library, and the sub-level coefficient of the second instruction is calculated based on the sub-unit weight and the deadline of the first instruction.

[0025] Preferably, the sub-level coefficient of the second instruction is calculated based on the sub-unit weight and the deadline of the first instruction, including:

[0026] ;

[0027] in, For the The sub-level coefficients of the second instruction, For the The subunit weight of the second instruction, For the The single running time of the second instruction corresponding to the deadline of the first instruction, For the The subcomplexity of the second instruction, For the The sub-urgency of the second instruction, For the The sub-adjustment coefficient of the second instruction, and , is the number of the second instruction.

[0028] Preferably, based on the sub-level coefficients and the calling sequence, generating the calling process includes:

[0029] Determining the relationship status of each second instruction in the business module based on the instruction type, wherein the relationship status includes: a dependency relationship and a non-dependency relationship;

[0030] Establishing an instruction dependency tree for the second instructions with dependencies and in combination with the sub-level coefficients of the second instructions with dependencies, and determining a first calling order of each second instruction in the instruction dependency tree based on the instruction dependency value of each second instruction in the instruction dependency tree;

[0031] Determine the instruction-independent value of the second instruction in the non-dependency relationship based on the sub-level coefficient of the second instruction in the non-dependency relationship, and generate a second calling sequence;

[0032] The first calling sequence and the second calling sequence are adjusted based on the instruction priorities of all second instructions to generate a comprehensive calling sequence;

[0033] All second instructions are organized according to the comprehensive calling sequence to generate a calling flow.

[0034] Preferably, dynamically updating the generated call flow based on instruction data of a next time period corresponding to the preset time period in which the second instruction is located includes:

[0035] According to the user demand data of each second instruction in the preset time period, the effective execution probability of the corresponding second instruction is calculated:

[0036] ;

[0037] in, For the The probability of effective execution of the second instruction is The first time within a preset time period determined based on user demand data The probability of the second instruction being executed is , is the total number of executions of all second instructions within a preset time period, For the The preset execution interval of the second instruction, For the The average execution time of the second instruction, For the The average time interval between all two adjacent executions of the second instruction in the historical process; The first time within a preset time period determined based on user demand data The number of times the second instruction is executed; is the rounding symbol;

[0038] Based on the effective execution probability of the corresponding second instruction, the first sub-contribution of the corresponding second instruction is calculated:

[0039] ;

[0040] in, For the The first sub-contribution of the second instruction, For the The execution adjustment factor of the second instruction;

[0041] Based on the user data of the next time period of the preset time period corresponding to the second instruction, a third instruction is obtained, and the corresponding second instruction and the obtained third instruction are matched to obtain the number of repetitions, and the second sub-contribution of the corresponding second instruction is calculated:

[0042] ;

[0043] in, Indicates the number of repetitions, represents the repetition weight that matches the number of repetitions, For the The second sub-contribution of the second instruction;

[0044] The first sub-contribution and the second sub-contribution of all the same second instructions are calculated to obtain the comprehensive contribution of the corresponding second instruction, and all the comprehensive contributions are sorted and filtered to obtain a filtered instruction set; based on the filtered instruction set, the generated call process is dynamically updated.

[0045] Compared with the prior art, the beneficial effects of this application are as follows:

[0046] By collecting user needs and information data in real time on the client side, the real-time data processing and analysis is realized. By analyzing user needs, the first instruction and instruction deadline of each user are obtained, thereby realizing personalized analysis of user behavior, generating personalized module call processes based on user needs, modularizing functions, and flexibly calling modules based on corresponding instructions. By dynamically updating the generated call process with the instruction data of the next time period, the data processing process is dynamically adjusted. It adapts to the dynamically changing data processing needs and improves the stability and flexibility of the system.

[0047] Other features and advantages of the present invention will be described in the following description, and partly become apparent from the description, or understood by practicing the present invention. The purpose and other advantages of the present invention can be realized and obtained by the structures particularly pointed out in the written description and the accompanying drawings.

[0048] The technical solution of the present invention is further described in detail below through the accompanying drawings and embodiments. BRIEF DESCRIPTION OF THE DRAWINGS

[0049] The accompanying drawings are used to provide a further understanding of the present invention and constitute a part of the specification. Together with the embodiments of the present invention, they are used to explain the present invention and do not constitute a limitation of the present invention. In the accompanying drawings:

[0050] Figure 1 The present invention is a flowchart of a data processing and analysis method of a building block type big data platform in an embodiment of the present invention. DETAILED DESCRIPTION

[0051] The preferred embodiments of the present invention are described below in conjunction with the accompanying drawings. It should be understood that the preferred embodiments described herein are only used to illustrate and explain the present invention, and are not used to limit the present invention.

[0052] Example 1

[0053] Reference Figure 1 The embodiment of the present invention provides a data processing and analysis method of a building block big data platform, including:

[0054] Step 1: Collect user needs and user information data in real time within multiple preset time periods at the client end;

[0055] Step 2: Analyze the user needs in each preset time period to obtain the first instructions and instruction deadlines of multiple users in the corresponding preset time period, and analyze the first instructions and instruction deadlines based on the user information data in the corresponding preset time period to generate at least one second instruction and a sub-level coefficient of each second instruction;

[0056] Step 3: Determine the business module and instruction type of each second instruction based on the preset instruction-module-type library, determine the calling order of the second instruction in the business module according to the instruction type, and generate a calling process based on the sub-level coefficient and the calling order;

[0057] Step 4: Dynamically update the generated call flow based on the instruction data of the next time period corresponding to the preset time period where the second instruction is located.

[0058] In this embodiment, user needs include user behaviors, operations, queries, etc. For example, a user needs to query information about a product or make a purchase;

[0059] In this embodiment, user information data refers to various information data generated by users in the system, including user behavior preferences, user historical query data, user behavior trajectory, user data neural records, etc.;

[0060] In this embodiment, the first instructions of multiple users are instructions obtained by feature analysis of user demand data obtained based on multiple primary operations of users within a specific time period, and the first instructions of multiple users include different instruction types, and different instruction type features are obtained through feature analysis. For example, after feature analysis, demand feature 1 and demand feature 2 have a high correlation, which are classified as class A01, and the corresponding first instruction is operation instruction 1; demand feature 3 and demand feature 4 have a high correlation, which are classified as class C04, and the corresponding first instruction is operation instruction 34, wherein operation instruction 1 includes: a combination of a data cleaning module, a data conversion module and a data application behavior analysis module, which belongs to the analysis process 01 of data application behavior, and operation instruction 34 includes: a combination of a data cleaning module, a data conversion module and anomaly detection module, which belongs to the real-time monitoring data process 02;

[0061] In this embodiment, the instruction deadline refers to the deadline for the user's operation, that is, the operation or instruction that needs to be completed before this time. For example, the user needs to complete the checkout operation of the shopping cart before a certain time point;

[0062] In this embodiment, the second instruction is obtained by expanding the first instruction and the user information data.

[0063] In this embodiment, the sub-level coefficient of the second instruction is the next operation instruction for each user and the level coefficient of the instruction generated after analyzing the first instruction and the corresponding user information. For example, preset time period 1: user A: demand feature 1, user B: demand feature 2, preset time period 2: user A: demand feature 4, user B: demand feature 5, user D: demand feature 6, the first instruction of user A is operation instruction 1, and the instruction deadline is the end time of time period 1, the first instruction of user B is operation instruction 2, and the instruction deadline is the end time of time period 1, according to user information A, the second instruction generated for user A is operation instruction 4, and the sub-level coefficients are 0.8 respectively, according to user information B, the second instruction generated for user B is operation instruction 5, and the sub-level coefficients are 0.7 respectively.

[0064] In this embodiment, the preset instruction-module-type library is a library of predefined instructions, modules and types, which is used to determine the business module and instruction type of each second instruction. For example, a certain instruction may belong to the "shopping cart module" and be of the "settlement type";

[0065] In this embodiment, the business module is a functional module or business module in the system, which is used to complete specific business functions. For example, an e-commerce system may include a product display module, a shopping cart module, a payment module, etc.

[0066] In this embodiment, the instruction type is the type or category to which the instruction belongs, and is used to describe the characteristics or attributes of the instruction. For example, an instruction may belong to a "query type" or a "transaction type";

[0067] In this embodiment, the calling order refers to the order or order relationship in which each instruction is called in a specific business module. For example, in a business module, it may be necessary to first query data and then update data;

[0068] In this embodiment, the calling process is a flowchart or process sequence for user operations within a specific time period generated according to the instruction level coefficient and the calling sequence, and is used to describe the logical process of user operations;

[0069] In this embodiment, dynamic update is to update and adjust the generated call flow in real time according to the instruction data of the next time period to adapt to user needs and system changes. For example, preset time period 1: user A: [operation instruction 1, operation instruction 4], user B: [operation instruction 2, operation instruction 5], preset time period 2: user A: operation instruction 7, user B: operation instruction 8, according to the instruction data of the next time period, the call flow of user A is updated to: user A: [operation instruction 1, operation instruction 4, operation instruction 7], and the call flow of user B is updated to: user B: [operation instruction 2, operation instruction 5, operation instruction 8].

[0070] The working principle and beneficial effects of the above technical solution are: by collecting user needs and information data in real time on the client, the real-time data processing and analysis is realized, the user needs are analyzed, and the first instruction and instruction deadline of each user are obtained, thereby realizing personalized analysis of user behavior, generating personalized module call flow based on user needs, realizing modularization of functions, and flexibly calling modules based on corresponding instructions, and dynamically updating the generated call flow by the instruction data of the next time period, realizing dynamic adjustment of the data processing flow. It adapts to the dynamically changing data processing needs and improves the stability and flexibility of the system.

[0071] Example 2

[0072] The present invention provides a data processing and analysis method for a building block big data platform, which collects user needs and user information data in multiple preset time periods in real time at the client end, including:

[0073] Tracking is performed on the client's program interface and user interface, and the first tracking path of the user's clicks on the program interface and the second tracking path of the user's clicks on the user interface are captured;

[0074] The first burial point path and the second burial point path are respectively sent to the server for comprehensive analysis to obtain user needs and user information data.

[0075] In this embodiment, the client program refers to an application or software installed on the user's device, such as a mobile phone app, a web browser, etc.;

[0076] In this embodiment, embedding in the client program interface and user interface means embedding in multiple interfaces of the client program, that is, setting trigger points for recording user behaviors on different pages or interactive interfaces. For example, when a user browses a product page, a search page, and a shopping cart page in an e-commerce app, embedding is set on each page to record the user's clicks, slides, and other operations.

[0077] In this embodiment, the first tracking path is generated according to the change of the user's click position, which can capture the user's click behavior in different interfaces and different positions in the application. For example, the user clicks on a product image on the homepage and then jumps to the product details page, generating a dynamic tracking path;

[0078] In this embodiment, the second tracking path refers to a predefined tracking point at a fixed position, which is used to record the user's click behavior on a specific interface. It is set according to the design and requirements of the application and will not change with the change of the user's click position. For example, a static tracking point is set on the Add to Cart button in the App to record the user's behavior of adding to the shopping cart.

[0079] In this embodiment, user demand data is data obtained based on the user's behavior and operation records in the client program, and is used to analyze the user's needs and behavior characteristics. For example, which products the user browsed in the App within a certain period of time, which function buttons the user clicked, etc.;

[0080] The above working principle and beneficial effects: By performing tracking operations in multiple interfaces of the client program, recording the user's click operations and behavior trajectories, and sending these data to the server for analysis, user demand data can be obtained. By generating tracking paths for user information through preset templates, user information data can be obtained more conveniently and efficiently. It helps analyze user behavior characteristics, optimize user experience, improve product functions, etc., thereby improving the system's intelligence and personalized service level.

[0081] Example 3

[0082] The present invention provides a data processing and analysis method for a building block big data platform, which captures the first buried point path of a user clicking on a program interface, including:

[0083] When the user clicks on the program interface, the tracking point label of the area corresponding to the center at the time of the click is pulled with the click position as the center;

[0084] Based on the tracking status of the tracking label, obtain the corresponding index path;

[0085] Determine the first burial path based on all index paths.

[0086] In this embodiment, the tag of the tracking point is a tag set on a specific area or element in the interface of the client program to record the user's click behavior and operation, which is used to identify and record whether the user's click operation is dynamic or static. For example, in the product list page of the App, the picture and title of each product can be set as a tracking point tag;

[0087] In this embodiment, the tag state refers to the static or dynamic attributes of the tracking point tag;

[0088] In this embodiment, the corresponding index path is an index path obtained according to the tag state, which is used to determine the embedding path of multiple pages of the client program, indicating the path or behavior trajectory of the user's clicks. For example, when the user clicks the tag of the third product on the product list page of the App, the corresponding index path may be [2], indicating that the user clicked the third product.

[0089] The working principle and beneficial effects of the above technical solution are as follows: By performing tracking operations in multiple interfaces of the client program, and obtaining tracking tags of corresponding areas according to the user's click behavior. By monitoring the status of the tag, the information of the user's click can be obtained, and the tracking paths of multiple pages of the client program can be determined according to the index path. Such a method can help record the user's click behavior and operation path, and provide accurate data for subsequent data processing and analysis. By analyzing the user's click behavior and path, the user's needs and behavioral characteristics can be better understood, thereby optimizing product functions and providing personalized services.

[0090] Example 4

[0091] The present invention provides a data processing and analysis method for a building block type big data platform, which captures a second buried point path of a user clicking on a user interface, including:

[0092] Obtain static buried points and dynamic buried points within the click range of the user's click position on the user node;

[0093] Obtain a dynamic identifier corresponding to the dynamic burial point, and obtain a corresponding first pending path according to the dynamic identifier. At the same time, obtain a static identifier corresponding to the static burial point, and obtain a corresponding second pending path according to the static identifier.

[0094] Based on the first pending path and the second pending path, a second burial point path is constructed.

[0095] In this embodiment, static tracking points refer to tracking points pre-defined in the client program. These tracking points already exist when the page is loaded and are used to record the click behavior of fixed elements or areas in the page. For example, in the App, each title of the shopping list page is set with a static tracking point to record the user's click operation;

[0096] In this embodiment, dynamic tracking points refer to tracking points that are dynamically generated in the client program based on specific conditions or user behaviors. They do not exist when the page is loaded and need to be dynamically generated based on the user's operation. They are used to record the click behavior of elements or areas generated based on user behavior in the page. For example, in an App, the recommended user area dynamically generated based on the user's follow list can be set as a dynamic tracking point.

[0097] In this embodiment, the dynamic identifier is a unique identifier used to identify the dynamic embedding point when the embedding point of the clicked area is a dynamic embedding point, so as to obtain the corresponding path when the user clicks it.

[0098] In this embodiment, the first pending path is to obtain the corresponding pending path according to the dynamic identifier, that is, the path generated according to the click behavior recorded by the dynamic embedding point. For example, according to the dynamic embedding point record, the corresponding dynamic identifier is obtained, and the dynamic identifier recording the user clicking the product picture on the homepage is "dynamic_001". The corresponding first pending path is obtained according to the dynamic identifier, and the first pending path corresponding to the dynamic identifier "dynamic_001" is "homepage-product details page";

[0099] In this embodiment, the static identifier is a unique identifier used to identify the static embedding point when the embedding point of the clicked area is a static embedding point, so as to obtain the corresponding path when the user clicks. The static identifier is a unique identifier used to distinguish different static embedding points in subsequent data processing and analysis.

[0100] In this embodiment, the second pending path is the pending path corresponding to the static identifier, that is, the path generated according to the click behavior recorded by the static embedding point. For example, the state identifier set on the Add to Cart button on the product details page is "static_001". The corresponding second pending path is obtained according to the static identifier. The second pending path corresponding to the static identifier "static_001" is "product page-Add to Cart button-shopping cart page".

[0101] In this embodiment, the second buried point path is constructed based on the first pending path and the second pending path. For example, according to the first pending path "home page - product details page" and the second pending path "product details page - add to cart button - shopping cart page", the second buried point path constructed is "home page - product details page - add to cart button - shopping cart page".

[0102] The working principle and beneficial effects of the above technical solution: By setting static and dynamic tracking points, the corresponding path is obtained when the user clicks, and the path data is uploaded to the server for analysis. This method can help record the user's click behavior and operation path, including not only fixed static elements, but also elements dynamically generated according to user behavior. By analyzing these path data, users' needs and behavioral characteristics can be better understood.

[0103] Example 5

[0104] The present invention provides a data processing and analysis method for a building block type big data platform, generating at least one second instruction and a sub-level coefficient of each second instruction, including:

[0105] Extract multiple demand features of user needs within each preset time period, and extract feature keywords of the demand features again to map them in a first preset word-instruction library to obtain a first instruction, and obtain an instruction deadline corresponding to the first instruction based on the first preset instruction information library;

[0106] Extracting user keywords from user information data corresponding to a preset time period, mapping the user keywords in a second preset word-instruction library to obtain an extended instruction, and expanding the first instruction based on the extended instruction to obtain a second instruction;

[0107] The sub-unit weight to which the second instruction belongs is determined based on the second preset instruction information library, and the sub-level coefficient of the second instruction is calculated based on the sub-unit weight and the deadline of the first instruction.

[0108] In this embodiment, the first preset word-instruction library is a corresponding library of words and instructions pre-set in the data processing and analysis method, which is used to map the keywords in the user demand characteristics to the corresponding instructions, and help convert the user's demand characteristics into specific instructions. For example, the first preset word-instruction library can contain keywords of various commodity categories and corresponding purchase instructions. The first preset word-instruction library contains the instruction "buy mobile phone" corresponding to the keyword "mobile phone". When the user's demand characteristics contain the keyword "mobile phone", it is mapped to the first instruction "buy mobile phone";

[0109] In this embodiment, user keywords are keywords generated based on user information data corresponding to a preset time period, and are used to describe the user's behavior, preferences, and other information during the time period. For example, the content posted by the user on the social platform, the pages browsed, the interactive behavior, etc. can be used as user information keywords;

[0110] In this embodiment, the extended instruction is to map the user's keyword to the keyword in the second preset word-instruction library to obtain the corresponding extended instruction. For example, the user extracts the key "mobile phone 030" from the demand characteristics within the preset time period. Through mapping, the keyword "mobile phone" is mapped to the keyword "mobile phone" in the first preset word-instruction library to obtain the first instruction "buy mobile phone 030". According to the mapping of the user's keyword "mobile phone" in the second preset-instruction library, the extended instruction "buy mobile phone 030 accessories 051" is obtained.

[0111] In this embodiment, the sub-unit to which the second instruction belongs refers to the specific category described by the sub-instruction of the second instruction, which is used to classify and summarize user needs. For example, the unit to which the sub-instruction belongs is a commodity category, a service category, etc.;

[0112] In this embodiment, the sub-unit weight is used to measure the weight of the unit to which the sub-instruction of the second instruction belongs. The units to which different second instructions belong are sorted and summarized by the unit weight;

[0113] In this embodiment, the sub-level coefficient of the second instruction is calculated based on the unit weight and the deadline of the first instruction.

[0114] The working principle and beneficial effects of the above technical solution are as follows: by analyzing and mapping the characteristics of user needs, a first instruction and instruction deadline are generated. Then, an extended instruction is generated based on the user information keyword, and the extended instruction is combined with the first instruction to generate a second instruction. The level coefficient of the second instruction is obtained by calculating the unit weight and the deadline of the first instruction, which more accurately describes the importance and priority of the user needs. The call flow can be generated more effectively based on the level of user needs.

[0115] Example 6

[0116] The present invention provides a data processing and analysis method for a building block type big data platform, which calculates the sub-level coefficient of a second instruction based on the sub-unit weight and the deadline of a first instruction, including:

[0117] ;

[0118] in, For the The sub-level coefficients of the second instruction, For the The subunit weight of the second instruction, For the The single running time of the second instruction corresponding to the deadline of the first instruction, For the The subcomplexity of the second instruction, For the The sub-urgency of the second instruction, For the The sub-adjustment coefficient of the second instruction, and , is the number of the second instruction.

[0119] In this embodiment, the sub-complexity of the second instruction represents the complexity of the sub-instructions of the second instruction, which is used to measure the resources and workload required to process the instruction. For example, instruction C1 is to train a complex machine learning algorithm on a large data set, and the complexity is determined to be in a high threshold range based on the computational amount of the instruction and the required resources;

[0120] In this embodiment, the sub-urgency of the second instruction indicates the urgency of the second instruction. For example, instruction C1 is to perform real-time analysis on an important business data to support immediate decision-making. Based on the importance and deadline of the instruction, it is determined that the complexity is in the high threshold range;

[0121] In this embodiment, the sub-level coefficient of the second instruction is calculated, for example, the parameters of instruction C02 are as follows:

[0122] is 0.1, is 0.8, is 3, is 5.12, 5s, is 700, then:

[0123] ;

[0124] Then the sub-level coefficient of instruction C02 is 1.16;

[0125] In this embodiment, the sub-adjustment coefficient of the second instruction is used to adjust the sub-level coefficient to meet specific needs or optimize the results. For example, instruction C1 is to analyze module 001. The weight of module 001 is less than 0.5, which is in a lower range, but module 001 has a large correlation with modules 005 and 007. In this case, the instruction needs to be weighted when calculating the sub-level coefficient. The sub-adjustment coefficient of the instruction is set to 1.2.

[0126] The working principle and beneficial effects of the above technical solution are as follows: by setting the sub-level coefficient, the priority of each instruction is determined according to the sub-unit weight and the deadline of the first instruction, thereby improving the overall task scheduling rate. Through the sub-complexity parameter, the computing resources and workload are reasonably allocated. Through the sub-urgency parameter, the urgent tasks are ensured to be processed in a timely manner. Through the sub-adjustment coefficient parameter, the sub-level coefficient is adjusted according to the actual needs and optimization goals to meet specific needs or optimize the processing results. By reasonably setting and adjusting the parameters, the data processing efficiency is improved and the processing time is reduced. The sub-level coefficient of the second instruction is converted into a specific numerical value by calculation, which is used as a quantitative indicator of the instruction at the sub-level to evaluate the sub-level effect of the instruction. By calculating the sub-level coefficient, the effect difference of different instructions at the sub-level is intuitively understood, and the influence of different factors on the instruction at the sub-level is obtained. Based on the sub-level coefficient, the inefficient sub-level can be improved and the investment in the efficient sub-level can be increased. By adjusting the parameters in the formula, the influence of different factors on the sub-level coefficient can be simulated. More efficient, accurate and reliable data processing and analysis are achieved.

[0127] Example 7

[0128] The present invention provides a data processing and analysis method for a building block type big data platform. Based on the sub-level coefficients and the calling sequence, a calling process is generated, including:

[0129] Determining the relationship status of each second instruction in the business module based on the instruction type, wherein the relationship status includes: a dependency relationship and a non-dependency relationship;

[0130] Establishing an instruction dependency tree for the second instructions with dependencies and in combination with the sub-level coefficients of the second instructions with dependencies, and determining a first calling order of each second instruction in the instruction dependency tree based on the instruction dependency value of each second instruction in the instruction dependency tree;

[0131] Determine the instruction-independent value of the second instruction in the non-dependency relationship based on the sub-level coefficient of the second instruction in the non-dependency relationship, and generate a second calling sequence;

[0132] The first calling sequence and the second calling sequence are adjusted based on the instruction priorities of all second instructions to generate a comprehensive calling sequence;

[0133] All second instructions are organized according to the comprehensive calling sequence to generate a calling flow.

[0134] In this embodiment, the instruction type refers to the category or type to which each second instruction belongs, including data cleaning, data conversion, data analysis, data mining, etc.

[0135] In this embodiment, the instruction dependency tree is established by constructing a tree structure to represent the dependency relationship between instructions based on the dependency relationship between instructions. The nodes of the tree represent each second instruction, and the edges represent the dependency relationship between instructions. For example, there are three instructions A, B, and C, where A depends on B, and B depends on C. When establishing the instruction dependency tree, the root node is A, the child node of A is B, and the child node of B is C. This can clearly represent the relationship that A depends on B and B depends on C;

[0136] In this embodiment, the first calling sequence refers to determining the execution order of each second instruction after establishing the instruction dependency tree, that is, which instruction is executed first and which instruction is executed later. For example, instruction A depends on instruction B, and instruction B depends on instruction C. According to the instruction dependency tree and the dependency relationship between instructions, the first calling sequence is determined to be CBA, that is, instruction C is executed first, then instruction B, and finally instruction A.

[0137] In this embodiment, the instruction independence value refers to the priority or importance of each second instruction determined according to the sub-level coefficient of the instruction in the absence of a dependency relationship, for example, there is no dependency relationship between multiple instructions. According to the sub-level coefficient of the instruction, the instruction independence value of each instruction can be determined, thereby determining their execution order. The sub-level coefficient of instruction A is 2.0, the sub-level coefficient of instruction B is 1.5, and the sub-level coefficient of instruction C is 1.0. Instruction A has the highest instruction independence value and instruction C has the lowest instruction independence value. The execution order may be ABC;

[0138] In this embodiment, the second calling order is for instructions without dependencies, and a second calling order can be generated, indicating the calling order of instructions without dependencies when executing data processing or analysis tasks.

[0139] In this embodiment, the comprehensive calling order is to determine the final execution order of each second instruction under the condition of considering the instruction dependency, instruction independence value and instruction priority. For example, multiple instructions need to be executed, including both dependent instructions and independent instructions. According to the dependency, independence value and priority of the instructions, the comprehensive calling order of each instruction can be determined. Instruction A depends on instruction B, the sub-level coefficient of instruction A is 2.0, and the sub-level number of instruction B is 1.5, then the comprehensive calling order may be BA.

[0140] In this embodiment, the calling process refers to organizing all the second instructions in a certain order of instruction execution to form a complete data processing and analysis process. According to the execution order of the instructions, all the second instructions are organized in order to form a calling process. For example, according to the comprehensive calling order, instruction A is executed after instruction B, and instruction B is executed after instruction C, then the calling process may be CBA.

[0141] The working principle and beneficial effects of the above technical solution are as follows: by determining the instruction type and dependency status, an instruction dependency tree is established, and a first calling order and a second calling order are generated. Then, the calling order is adjusted according to the priority of the second instruction to generate a comprehensive calling order. Finally, the second instruction is organized according to the comprehensive calling order to generate a calling process. Ensure that the data processing or analysis task is carried out in the correct order and process, and improve the efficiency and accuracy of the task. At the same time, by considering the priority and dependency of the instruction, the execution order of the data processing or analysis task can be optimized, and the overall data processing and analysis effect can be improved.

[0142] Example 8

[0143] The present invention provides a data processing and analysis method of a building block big data platform, which dynamically updates a generated call flow based on instruction data of a next time period corresponding to a preset time period where a second instruction is located, including:

[0144] According to the user demand data of each second instruction in the preset time period, the effective execution probability of the corresponding second instruction is calculated:

[0145] ;

[0146] in, For the The probability of effective execution of the second instruction is The first time within a preset time period determined based on user demand data The probability of the second instruction being executed is , is the total number of executions of all second instructions within a preset time period, For the The preset execution interval of the second instruction, For the The average execution time of the second instruction, For the The average time interval between all two adjacent executions of the second instruction in the historical process; The first time within a preset time period determined based on user demand data The number of times the second instruction is executed; is the rounding symbol;

[0147] Based on the effective execution probability of the corresponding second instruction, the first sub-contribution of the corresponding second instruction is calculated:

[0148] ;

[0149] in, For the The first sub-contribution of the second instruction, For the The execution adjustment factor of the second instruction;

[0150] Based on the user data of the next time period of the preset time period corresponding to the second instruction, a third instruction is obtained, and the corresponding second instruction and the obtained third instruction are matched to obtain the number of repetitions, and the second sub-contribution of the corresponding second instruction is calculated:

[0151] ;

[0152] in, Indicates the number of repetitions, represents the repetition weight that matches the number of repetitions, For the The second sub-contribution of the second instruction;

[0153] Calculate the first sub-contribution and the second sub-contribution of all the same second instructions to obtain the comprehensive contribution of the corresponding second instructions, and sort and filter all the comprehensive contributions to obtain a filtered instruction set;

[0154] The generated calling process is dynamically updated based on the screening instruction set.

[0155] In this embodiment, the comprehensive contribution of the second instruction is calculated. For example, the parameters of instruction C02 are: is 0.2, For 10 times, 5s, is 2s, 1s; For 40 times, is 0.79, is 3; is 0.6, then

[0156] ;

[0157] Then the effective execution probability of instruction C02 is 0.64;

[0158] ;

[0159] Then the first sub-contribution of instruction C02 is ;

[0160] ;

[0161] Then the second sub-contribution of instruction C02 is ;

[0162] Then the comprehensive contribution of instruction C02 is 0.5412;

[0163] In this embodiment, the effective execution probability refers to the probability of a second instruction being executed within a preset time period, which is calculated based on the user demand data;

[0164] In this embodiment, the probability of being executed refers to the probability that a second instruction is executed within a preset time period; for example, according to user demand data, the number of execution times N of instruction A within the preset time period is calculated,

[0165] The preset execution interval is I, and the probability of execution is P = N / (preset time period length / I), the preset time period length is L, then ;

[0166] In this embodiment, the first sub-contribution refers to the contribution of the sub-instruction of the second instruction to the entire call process;

[0167] In this embodiment, the execution adjustment factor refers to the execution section parameter corresponding to the second instruction adjusted according to the actual demand and optimization target, and is used to adjust the priority weight of the instruction. For example, the execution adjustment factor of the second instruction A is 0.9, and the priority weight of instruction A is adjusted;

[0168] In this embodiment, the second sub-contribution refers to the contribution degree of a third instruction related to a second instruction in the next time period of a preset time period.

[0169] In this embodiment, the contribution screening instruction set is to calculate the contribution of each instruction in the second instruction according to the user demand data of the next time period of the preset time period where the second instruction is located, and screen the contribution to obtain the instruction set. The contribution represents the contribution degree of each instruction to the user demand. By screening the instructions with high contribution, the instruction set with high contribution can be obtained. For example, in a certain time period, the user demand data shows that for data analysis, the contribution of instruction A is the highest, the contribution of instruction B is the second, and the contribution of instruction C is the lowest. Through contribution screening, the screening instruction set can be obtained as {A, B}.

[0170] In this embodiment, the third instruction data represents new instruction data generated according to user demand data of the next time period, and is used to update the calling process.

[0171] The working principle and beneficial effect of the above technical solution are: by calculating the contribution of each instruction of the second instruction and screening the contribution, an instruction set with high contribution is obtained. Then, the third instruction data and the third instruction deadline are generated based on the user demand data of the next time period, and the third instruction is updated based on the user information data to generate the fourth instruction. Finally, the repeated call sequence of the screened instruction set and the fourth instruction is extracted to generate an optimized call sequence, and then a call process is generated. By calculating the first sub-contribution and the second contribution of the second instruction to obtain the comprehensive contribution, the abstract instruction execution probability, contribution, number of repetitions, etc. can be converted into specific values ​​as quantitative indicators for evaluating and comparing the effects of different instructions, and different modules can be flexibly called based on indicators intuitively. Such a method can dynamically update the call process according to user needs to ensure the timeliness and accuracy of data processing and analysis. At the same time, the call sequence is optimized by calculating and screening the contribution. The efficiency and quality of data processing and analysis are improved.

[0172] Obviously, those skilled in the art can make various changes and modifications to the present invention without departing from the spirit and scope of the present invention. Thus, if these modifications and variations of the present invention fall within the scope of the claims of the present invention and their equivalents, the present invention is also intended to include these modifications and variations.

Claims

1. A data processing and analysis method for a building block big data platform, characterized in that: include: Step 1: Collect user needs and user information data in real time within multiple preset time periods at the client end; Step 2: Analyze the user needs in each preset time period to obtain the first instructions and instruction deadlines of multiple users in the corresponding preset time period, and analyze the first instructions and instruction deadlines based on the user information data in the corresponding preset time period to generate at least one second instruction and a sub-level coefficient of each second instruction; Step 3: Determine the business module and instruction type of each second instruction based on the preset instruction-module-type library, determine the calling order of the second instruction in the business module according to the instruction type, and generate a calling process based on the sub-level coefficient and the calling order; Step 4: dynamically updating the generated call flow based on instruction data of the next time period corresponding to the preset time period where the second instruction is located; The step of generating at least one second instruction and a sub-level coefficient of each second instruction comprises: Extract multiple demand features of user needs within each preset time period, and extract feature keywords of the demand features again to map them in a first preset word-instruction library to obtain a first instruction, and obtain an instruction deadline corresponding to the first instruction based on the first preset instruction information library; Extracting user keywords from user information data corresponding to a preset time period, mapping the user keywords in a second preset word-instruction library to obtain an extended instruction, and expanding the first instruction based on the extended instruction to obtain a second instruction; Determine the sub-unit weight to which the second instruction belongs based on the second preset instruction information base, and calculate the sub-level coefficient of the second instruction based on the sub-unit weight and the deadline of the first instruction; The sub-level coefficient of the second instruction is calculated based on the sub-unit weight and the deadline of the first instruction, including: ； in, For the The sub-level coefficients of the second instruction, For the The subunit weight of the second instruction, For the The single running time of the second instruction corresponding to the deadline of the first instruction, For the The subcomplexity of the second instruction, For the The sub-urgency of the second instruction, For the The sub-adjustment coefficient of the second instruction, and , is the number of the second instruction.

2. The data processing and analysis method of a building block big data platform according to claim 1, characterized in that: Collect user needs and user information data in real time within multiple preset time periods at the client tracking point, including: Tracking is performed on the client's program interface and user interface, and the first tracking path of the user's clicks on the program interface and the second tracking path of the user's clicks on the user interface are captured; The first burial point path and the second burial point path are respectively sent to the server for comprehensive analysis to obtain user needs and user information data.

3. The data processing and analysis method of a building block big data platform according to claim 2, characterized in that: Capture the first click path of the user on the program interface, including: When the user clicks on the program interface, the tracking point label of the area corresponding to the center at the time of the click is pulled with the click position as the center; Based on the tracking status of the tracking label, obtain the corresponding index path; Determine the first burial path based on all index paths.

4. The data processing and analysis method of a building block big data platform according to claim 3 is characterized in that: Capture the second tracking path of the user's clicks on the user interface, including: Obtain static buried points and dynamic buried points within the click range of the user's click position on the user node; Obtain a dynamic identifier corresponding to the dynamic burial point, and obtain a corresponding first pending path according to the dynamic identifier. At the same time, obtain a static identifier corresponding to the static burial point, and obtain a corresponding second pending path according to the static identifier. Based on the first pending path and the second pending path, a second burial point path is constructed.

5. The data processing and analysis method of a building block big data platform according to claim 1, characterized in that: Based on the sub-level coefficients and the calling sequence, the calling process is generated including: Determining the relationship status of each second instruction in the business module based on the instruction type, wherein the relationship status includes: a dependency relationship and a non-dependency relationship; Establishing an instruction dependency tree for the second instructions with dependencies and in combination with the sub-level coefficients of the second instructions with dependencies, and determining a first calling order of each second instruction in the instruction dependency tree based on the instruction dependency value of each second instruction in the instruction dependency tree; Determine the instruction-independent value of the second instruction in the non-dependency relationship based on the sub-level coefficient of the second instruction in the non-dependency relationship, and generate a second calling sequence; The first calling sequence and the second calling sequence are adjusted based on the instruction priorities of all second instructions to generate a comprehensive calling sequence; All second instructions are organized according to the comprehensive calling sequence to generate a calling flow.

6. The data processing and analysis method of a building block big data platform according to claim 1, characterized in that: The generated call flow is dynamically updated based on instruction data of a next time period corresponding to the preset time period where the second instruction is located, including: According to the user demand data of each second instruction in the preset time period, the effective execution probability of the corresponding second instruction is calculated: ； in, For the The probability of effective execution of the second instruction is The first time within a preset time period determined based on user demand data The probability of the second instruction being executed is , is the total number of executions of all second instructions within a preset time period, For the The preset execution interval of the second instruction, For the The average execution time of the second instruction, For the The average time interval between all two adjacent executions of the second instruction in the historical process; The first time within a preset time period determined based on user demand data The number of times the second instruction is executed; is the rounding symbol; Based on the effective execution probability of the corresponding second instruction, the first sub-contribution of the corresponding second instruction is calculated: ； in, For the The first sub-contribution of the second instruction, For the The execution adjustment factor of the second instruction; Based on the user data of the next time period of the preset time period corresponding to the second instruction, a third instruction is obtained, and the corresponding second instruction and the obtained third instruction are matched to obtain the number of repetitions, and the second sub-contribution of the corresponding second instruction is calculated: ； in, Indicates the number of repetitions, represents the repetition weight that matches the number of repetitions, For the The second sub-contribution of the second instruction; Calculate the first sub-contribution and the second sub-contribution of all the same second instructions to obtain the comprehensive contribution of the corresponding second instructions, and sort and filter all the comprehensive contributions to obtain a filtered instruction set; The generated calling process is dynamically updated based on the screening instruction set.

Images