Massive traffic flow data offline base and ad hoc dynamic hybrid label distribution calculation method

Abstract

The application provides an offline base and instant dynamic mixed label distribution calculation method for mass traffic flow data, relates to the technical field of data processing, and comprises the following steps: reading individual portrait offline base table data and instant configuration data, performing federated query to output mixed label presentation results; receiving group circle selection rule strings and parsing to generate a multi-level logical syntax tree; according to a conditional leaf node statistical parameter, the multi-level logical syntax tree is split into an offline static execution subtree and an instant dynamic execution subtree; based on the offline static execution subtree, an offline base entity bitmap is generated in an offline data warehouse; the instant dynamic execution subtree is issued to generate an instant dynamic entity bitmap; based on a preset cross-domain mapping matrix, the aforementioned bitmaps are reduced in dimension and converted into homologous target entity bitmaps; according to a non-leaf node operator, bit-by-bit operation is performed on the homologous target entity bitmaps to generate a mixed bitmap, and a target circle selection result set is output. The application reasonably allocates underlying computing resources and improves the circle selection efficiency of mass data.

Description

Technical Field

[0001] This invention relates to the field of data processing technology, and in particular to a method for calculating the offline basis and ad-hoc dynamic hybrid label distribution of massive traffic flow data. Background Technology

[0002] A vast amount of business record data has been accumulated in traffic management and operation services. To extract the valuable information contained in this data, relevant technical personnel typically use various processing models to perform label distribution calculations and group selection operations, thereby assisting in the formulation of traffic strategies or the optimization of resource scheduling. The massive records generated by various traffic entities come from multiple sources, and the data dimensions of different sources are not entirely the same. Reasonably integrating and calculating multi-source heterogeneous data is the foundation of traffic flow data processing.

[0003] Conventional data processing workflows typically employ offline batch processing or ad-hoc queries for group selection. In offline batch processing, the framework performs periodic calculations based on pre-planned dimensional conditions to create a fixed data table. This approach can be used during system downtime, but when faced with ad-hoc additions or changes in analytical needs, the fixed-generation reports struggle to provide real-time support for combined conditions. In ad-hoc dynamic query mode, the system receives externally input query rules in real time and scans and aggregates them across the entire dataset. This method retains the flexibility of query dimensions, but it requires significant memory and computing node resources when analyzing massive amounts of record data. Real-time reading of large amounts of detailed data and multi-dimensional intersection / merge operations can easily overload the system.

[0004] When performing wide table joins across multiple types of source data, the overlapping mapping relationships between the attributes and identities of the underlying entities can lead to an explosion in the data volume of intermediate result sets. Relying on relational operators for full table scans and matching increases the system's concurrent processing pressure. Exploring how to rationally allocate computing power while considering computational efficiency under complex combinations of conditions is an objective technical requirement in the field of traffic data analysis. Summary of the Invention

[0005] The purpose of this invention is to provide a method for calculating the distribution of offline basis and ad-hoc dynamic hybrid labels for massive traffic flow data, so as to solve the technical problems of existing technologies in balancing the flexibility of conditions and the consumption of system computing power when facing massive multi-source data combination queries, and the problem that direct association of cross-domain data tables can easily cause the expansion of intermediate results, resulting in poor computing efficiency.

[0006] This invention provides a method for calculating the offline basis and ad-hoc dynamic hybrid label distribution of massive traffic flow data, including: Read the offline baseline data and ad-hoc configuration data of the individual profile, perform a federated query operation based on the offline baseline data and ad-hoc configuration data of the individual profile, and output the mixed label presentation result; Receive the group selection rule string; The group selection rule string is parsed to generate a multi-level logical syntax tree; Based on the statistical parameters associated with the condition leaf nodes, the multi-level logical syntax tree is split into an offline static execution subtree and an ad-hoc dynamic execution subtree; Based on the offline static execution subtree, a transformation operation is performed in the offline data warehouse to generate an offline base entity bitmap; The ad-hoc dynamic execution subtree is sent to the real-time analysis node to generate an ad-hoc dynamic entity bitmap. Based on a preset cross-domain mapping matrix, a dimensionality reduction operation is performed on the offline base entity bitmap and the ad hoc dynamic entity bitmap to transform the offline base entity bitmap and the ad hoc dynamic entity bitmap into homogeneous target entity bitmaps with the same primary key. Perform bitwise operations on the source target entity bitmap according to the non-leaf node operators to generate the final mixed bitmap, and output the target selection result set based on the final mixed bitmap.

[0007] Optionally, reading the offline baseline data and ad-hoc configuration data of the individual profile includes: The offline processing framework is used to read vehicle business data associated with traffic entities, electronic non-stop toll collection user business data, and mobile application user business data. Batch processing calculations are performed on the vehicle business data, the electronic non-stop toll collection user business data, and the mobile application user business data according to the primary key dimension to generate the offline base table data of the individual profile. The enumerated category set and enumerated state parameters associated with the traffic entity are synchronized from the relational database as the ad hoc configuration data.

[0008] Optionally, the step of performing a federated query operation based on the offline baseline data of the individual profile and the ad-hoc configuration data, and outputting a mixed-label presentation result, includes: In the data query layer, a federated mapping view layer is constructed that spans the offline data warehouse and the relational database; The label category name and label setting value are read from the ad-hoc configuration data through the federated mapping view layer; In the offline baseline data of the individual profile, the total number of covered entities corresponding to each of the set values ​​of the labels is counted. The total number of covered entities and the label category name are combined and integrated by dimension concatenation to output the hybrid label presentation result, which includes the personnel gathering scale parameter and the activity level stratification parameter.

[0009] Optionally, the receiving group selection rule string includes: The group selection rule string, encapsulated in script object format, is obtained through the front-end interface application programming interface; Verify the outermost merge relationship identifier character included in the group selection rule string; Parse the inner nested array sequence in the group selection rule string, and extract the source table name parameter, drop-down field name parameter, field type parameter, comparison operator, and comparison assignment parameter included in the inner nested array sequence; Identify the conditional connector logical operator corresponding to the deepest nested level in the group selection rule string.

[0010] Optionally, after outputting the target selection result set, the calculation method further includes: Write the target selection result set into the group-individual relationship history table of the offline data warehouse; The total number of members in the target selection result set is aggregated according to the group identifier to generate group personnel statistics; The statistical data of the group members and the historical table of the group's individual relationships are synchronized to the columnar storage acceleration node using the primary key index model, and the latest status parameters corresponding to each group identifier are retained for the front-end interface to perform query callback operations.

[0011] Optionally, the step of splitting the multi-level logical syntax tree into an offline static execution subtree and an ad-hoc dynamic execution subtree based on the statistical parameters associated with the conditional leaf nodes includes: Extract the attribute field names bound to each condition leaf node in the multi-level logical syntax tree; Read the pre-calculated total tag coverage table in the offline data warehouse, and extract the statistical parameter that matches the attribute field name from the total tag coverage table; The conditional leaf nodes whose statistical parameters exceed the computing power consumption threshold, along with their connected parent node paths, are pruned into the offline static execution subtree. The conditional leaf nodes whose statistical parameters are not higher than the computing power consumption threshold, along with their associated parent node paths, are pruned into the ad hoc dynamic execution subtree.

[0012] Optionally, the step of performing transformation operations in the offline data warehouse based on the offline static execution subtree to generate an offline base entity bitmap includes: The underlying query condition nodes in the offline static execution subtree are converted into full table scan structured instructions adapted to the offline data warehouse; Execute the full table scan structured instruction to extract the offline source entity business primary key sequence that meets the filtering conditions; A blank binary space sequence is allocated in the memory buffer area, and the corresponding bit positions in the blank binary space sequence are set to the active state according to the ascending order of the numerical parameters in the offline source entity business primary key sequence, thereby completing the generation and construction operation of the offline base entity bitmap.

[0013] Optionally, the step of performing a dimensionality reduction operation on the offline base entity bitmap and the ad-hoc dynamic entity bitmap based on a preset cross-domain mapping matrix, transforming the offline base entity bitmap and the ad-hoc dynamic entity bitmap into homogeneous target entity bitmaps with the same primary key, includes: Determine whether the offline base entity bitmap and the ad-hoc dynamic entity bitmap belong to the same entity category identity; When the entity categories are determined to be different, the source entity bit index parameters that are set in the offline base entity bitmap and the ad hoc dynamic entity bitmap are extracted. Find the associated mapping row vector corresponding to the source entity bit index parameter in the preset cross-domain mapping matrix; Perform a bitwise OR operation on all the found associated mapping row vectors and output the same source target entity bitmap with the target entity primary key aligned.

[0014] Optionally, the construction operation of the preset cross-domain mapping matrix includes: Extract the vehicle primary key vector, electronic non-stop toll collection primary key vector, and mobile application primary key vector from the traffic entities in the offline data warehouse according to the set batch processing cycle; Based on the vehicle primary key vector, the electronic non-stop toll collection primary key vector, and the mobile application primary key vector, a bidirectional pairing mapping calculation is performed to generate a list of attribution mapping relationships between each of the traffic entities. The attribution mapping list is reorganized into a two-dimensional mapping matrix with row index parameters and column index parameters, and the two-dimensional mapping matrix is ​​loaded into the real-time analysis node as the preset cross-domain mapping matrix.

[0015] Optionally, the step of performing bitwise operations on the source target entity bitmap according to non-leaf node operators to generate a final mixed bitmap, and outputting a target selection result set based on the final mixed bitmap, includes: Extract the non-leaf node operators from the breakpoint locations of the split multi-level logical syntax tree; When the non-leaf node operator is an intersection operation instruction, a bitwise AND logical call is performed on multiple source target entity bitmaps in the same dimension to generate the final mixed bitmap that retains the overlapping set parameters. When the non-leaf node operator is a union operation instruction, a bitwise OR logical call is performed on multiple source target entity bitmaps in the same dimension to generate the final mixed bitmap that retains all set parameters. The active bit indices in the final mixed bitmap are decoded and mapped to convert them into a detailed user information list, and the target selection result set is encapsulated and output.

[0016] The present invention has achieved the following beneficial effects: The computational method disclosed in this invention generates a multi-level logical syntax tree by parsing the group selection rule string. Based on the statistical parameters associated with the condition leaf nodes, it dynamically splits the tree into offline static execution subtrees and ad-hoc dynamic execution subtrees. The large-scale and fundamentally stable filtering task is pushed down to the offline data warehouse to generate an offline base entity bitmap, while the flexible and ever-changing ad-hoc combined query task is handled by the real-time analysis node to generate an ad-hoc dynamic entity bitmap. This method retains the flexibility of multi-dimensional cross-queries while rationally allocating the underlying computational load and controlling resource consumption in the real-time computation stage. This method utilizes a pre-set cross-domain mapping matrix to perform dimensionality reduction transformation on entity bitmaps of different identity categories, completing the alignment transformation of primary keys for multi-source heterogeneous businesses and avoiding data bloat caused by direct physical association of multi-business detail tables. In the final result generation stage, bitwise operations are performed on the same-source target entity bitmap instead of traditional relational algebra matching, optimizing memory space usage in the mixed label distribution calculation process and improving the analysis response performance of massive multi-source traffic flow data.

[0017] Other features and advantages of the invention will be set forth in the following description, and will be apparent in part from the description, or may be learned by practicing the invention. The objects and other advantages of the invention may be realized and obtained by means of the structures particularly pointed out in the written description and the accompanying drawings.

[0018] The technical solution of the present invention will be further described in detail below with reference to the accompanying drawings and embodiments. Attached Figure Description

[0019] The accompanying drawings are provided to further illustrate the invention and form part of the specification. They are used in conjunction with embodiments of the invention to explain the invention and do not constitute a limitation thereof. In the drawings: Figure 1 This is the main flowchart of the offline basis and ad-hoc dynamic hybrid label distribution calculation method for massive traffic flow data in this embodiment of the invention; Figure 2This is a flowchart illustrating the process of reading base table data and executing a federated query in an embodiment of the present invention. Figure 3 This is a flowchart of receiving a group selection rule string and generating a multi-level logical syntax tree in an embodiment of the present invention; Figure 4 This is a flowchart of splitting the syntax tree and generating an entity bitmap based on conditional leaf nodes in an embodiment of the present invention; Figure 5 This is a flowchart of the process of performing dimensionality reduction operation and generating a bitmap of the same target entity based on the cross-domain mapping matrix in an embodiment of the present invention; Figure 6 This is a flowchart illustrating the process of generating a final mixed bitmap and outputting a result set by performing bitwise operations in an embodiment of the present invention. Detailed Implementation

[0020] The preferred embodiments of the present invention will be described below with reference to the accompanying drawings. It should be understood that the preferred embodiments described herein are for illustration and explanation only and are not intended to limit the present invention.

[0021] Refer to the corresponding attached diagram, as shown below. Figure 1 As shown, the invention discloses a method for calculating the distribution of offline base and ad-hoc dynamic hybrid labels for massive traffic flow data, and the transfer steps include: Step S1: Read the offline baseline data and ad-hoc configuration data of the individual profile, perform a federated query operation based on the offline baseline data and ad-hoc configuration data of the individual profile, and output the mixed label presentation result.

[0022] like Figure 2 As shown, the operation of reading the offline baseline data of the individual profile and the ad-hoc configuration data includes: using an offline processing framework to read traffic entity-related vehicle business data, electronic non-stop toll collection user business data, and mobile application user business data; performing batch processing calculations on the vehicle business data, the electronic non-stop toll collection user business data, and the mobile application user business data according to the primary key dimension to generate the offline baseline data of the individual profile; and synchronizing the traffic entity-related enumeration category set and enumeration status parameters from a relational database as the ad-hoc configuration data.

[0023] Based on the offline baseline data of the individual profiles and the ad-hoc configuration data, a federated query operation is performed to output a hybrid tag presentation result. This operation includes: constructing a federated mapping view layer across the offline data warehouse and the relational database in the data query layer; reading the ad-hoc configuration data, including tag category names and tag setting values, through the federated mapping view layer; the scheduling node counts the total number of covered entities corresponding to each of the tag setting values ​​in the offline baseline data of the individual profiles; and performing a dimension concatenation and integration operation between the total number of covered entities and the tag category names to output the hybrid tag presentation result, including personnel gathering scale parameters and activity level stratification parameters.

[0024] The results of mixed label presentation, for example, output the total number of covered entities corresponding to the set values ​​of each label, providing a data distribution reference for subsequent group selection.

[0025] Step S2: Receive the group selection rule string, perform a parsing operation on the group selection rule string, and generate a multi-level logical syntax tree.

[0026] like Figure 3 As shown, the group selection rule string is obtained through the front-end interface application programming interface and encapsulated in script object format. The lexical control unit verifies that the group selection rule string includes the outermost merging relationship identifier character; the parsing program decomposes the inner nested array sequence remaining in the group selection rule string, extracts the corresponding fragments carrying the source table name parameter, dropdown field name parameter, field type parameter, comparison operator, and comparison assignment parameter; and matches and identifies the conditional connection logic operator corresponding to the deepest nested level in the group selection rule string. Specifically, the outermost merging relationship identifier character and the conditional connection logic operator are selected from the characters "AND" or "&&" representing intersection logic, and the characters "OR" or "||" representing union logic.

[0027] Based on the parsing, the underlying business parameters are obtained, and the source table name parameter representing the filtering dimension and the comparison assignment parameter are encapsulated into condition leaf nodes. The condition connection logical operator representing the set operation is converted into a non-leaf node operator, and the multi-level logical syntax tree is instantiated using a bottom-up hierarchical mounting method.

[0028] Step S3: Based on the statistical parameters associated with the condition leaf nodes, split the multi-level logical syntax tree into an offline static execution subtree and an ad-hoc dynamic execution subtree.

[0029] like Figure 4As shown, when performing dynamic and static splitting and orchestration of the tree-structured topology, the names of the bound attribute fields of each condition leaf node in the multi-level logical syntax tree are extracted; the total number of tag coverages completed in the offline data warehouse is read, and the statistical parameter matching the attribute field name is extracted from the total number of tag coverages table. Specifically, the statistical parameter here refers to the absolute base value of the entity deduplication corresponding to the attribute field in the full individual profile (i.e., the total number of independent entity rows covered). The physical dimension of this parameter is aligned with the boundary of the total number of entity records represented by the computing power consumption threshold T, thereby ensuring the consistency of the dimensions for comparison and judgment.

[0030] The mathematical derivation steps for establishing the computing power consumption threshold include: calling the available physical memory block capacity parameter allocated to the current real-time analysis node to obtain the number of the highest concurrent worker threads enabled within the real-time analysis node; setting the equation T=(M×K) / (N×S), where T represents the computing power consumption threshold, M represents the total number of available bytes corresponding to the available physical memory block capacity parameter, N represents the number of the highest concurrent worker threads, S represents the basic byte size of the single-item bitmap index, and K represents the set safety redundancy coefficient; configuring the safety redundancy coefficient to a value between 0.6 and 0.8. In the aforementioned equation, the available physical memory block capacity parameter M is obtained by the scheduling engine calling the operating system kernel's resource control group (Cgroups) interface to read the total number of physical memory page frames allocated by the operating system to the current computing process and deducting the daemon process resident overhead. The basic byte size S of the single-item bitmap index is limited by the 64-bit physical addressing architecture of the current computing node and is fixed at 8 bytes. The core purpose of setting the safety redundancy coefficient K is to absorb the memory fragmentation overhead caused by the underlying virtual machine garbage collection mechanism. The system deploys probes within the memory allocator to continuously collect the garbage collection trigger frequency of the young generation memory space in second-level time windows. When a surge in concurrent retrieval requests is detected within a short period, and the garbage collection frequency per second exceeds the preset safety baseline, the dynamic adjustment module adaptively lowers the coefficient K to 0.6, forcibly tightening the boundary of the total number of entity bitmaps that a single concurrent thread can carry. This intercepts the overflowing data filtering pressure and pushes it down to the offline computing link, preventing memory overflow from the underlying resource allocation level. In one specific embodiment, the preset safety baseline is set to 5 to 10 times / second, or dynamically calibrated by the system during the initialization phase using the benchmark stress test module to apply extreme concurrent request tests to the hardware configuration of the current real-time analysis node. Based on the computing power consumption threshold calculated by the equation, the boundary of the total number of entity records that each concurrent worker thread can carry without causing memory overflow is defined.

[0031] In the comparison and determination phase, the conditional leaf nodes whose statistical parameters are higher than the computing power consumption threshold and the paths connecting to parent nodes are pruned and stripped into the offline static execution subtree; the conditional leaf nodes whose statistical parameters are not higher than the computing power consumption threshold and the paths connecting to associated parent nodes are retained and reorganized into the ad hoc dynamic execution subtree.

[0032] Topology pruning of multi-level logical syntax trees can disrupt the original Boolean operation chain, causing discontinuity in data flow between offline and ad-hoc computing branches. To maintain logical equivalence during heterogeneous computing cluster collaboration, the execution engine dynamically injects a shared memory proxy node encapsulated with a state spinlock at the pruning breakpoint of the original topology mount node. This proxy node internally holds a base address pointer to the shared storage area. After the offline link completes the scanning of underlying records and generates the offline base entity bitmap in the offline data warehouse, it reverses the memory mapping offset address of the physical block in the distributed file system to the aforementioned base address pointer and releases the state spinlock. The ad-hoc dynamic execution subtree performs algebraic operations in the real-time analysis node according to the post-order traversal logic. When the traversal cursor reaches the proxy node, the worker thread verifies the spinlock state. Upon detecting the unlock signal, the engine directly retrieves the offline base entity bitmap in batches across processes based on the offset address. For scenarios where offline data warehouses and real-time analytics nodes are deployed across physical clusters, the shared memory proxy node encapsulates a distributed key-value cache and a Remote Direct Memory Access (RDMA) protocol at its underlying layer. After the offline link side writes the data to disk, it registers the globally unique resource identifier (URI) and checksum hash of the physical block in the distributed key-value cache. Upon detecting an unlock signal, the real-time analytics node bypasses the operating system kernel directly using the RDMA protocol and retrieves the offline base entity bitmap across the network with zero copy based on the URI. This mechanism, driven by memory events using state pointers, effectively connects static base table resources and dynamic selection results at the operation merging operator.

[0033] Step S4: Perform a transformation operation in the offline data warehouse based on the offline static execution subtree to generate an offline base entity bitmap; send the ad-hoc dynamic execution subtree to the real-time analysis node to generate an ad-hoc dynamic entity bitmap.

[0034] To isolate and segment the offline link side, the underlying query condition nodes of the offline static execution subtree are converted into structured instructions adapted for full table scans of the offline data warehouse. The offline execution cluster allocates a computing container to load the full table scan structured instructions and extracts the offline source entity business primary key sequences that meet the judgment filtering conditions. The computing unit allocates a blank binary space sequence in the memory buffer area and performs an ascending sorting operation according to the numerical value of the offline source entity business primary key sequence records. The bits at the corresponding offset address positions in the blank binary space sequence are activated, completing the offline base entity bitmap generation and construction operation.

[0035] On the ad-hoc analysis link side, the ad-hoc dynamic execution subtree is sent to the real-time analysis node. The real-time analysis node receives the instruction, scans the buffer resources to retrieve detailed records, and generates the ad-hoc dynamic entity bitmap using the same encoding logic.

[0036] Step S5: Perform dimensionality reduction operation on the offline base entity bitmap and the ad-hoc dynamic entity bitmap based on the preset cross-domain mapping matrix, and transform the offline base entity bitmap and the ad-hoc dynamic entity bitmap into homogeneous target entity bitmaps with the same primary key.

[0037] like Figure 5 As shown, the pre-set cross-domain mapping matrix construction calculation step involves periodically extracting traffic entities from the offline data warehouse, including vehicle primary key vectors, electronic toll collection primary key vectors, and mobile application primary key vectors, according to a set batch processing cycle. Based on the vehicle primary key vectors, electronic toll collection primary key vectors, and mobile application primary key vectors, a bidirectional pairing mapping calculation is performed to generate a list of attribution mapping relationships between various traffic entities. This attribution mapping relationship list is then reorganized into a two-dimensional mapping matrix with row and column index parameters, and loaded into the real-time analysis node as the pre-set cross-domain mapping matrix.

[0038] The steps for setting the value logic of the internal elements of the two-dimensional mapping matrix include: aligning the row index parameter of the two-dimensional mapping matrix with the primary key sequence of the source entity's business, and aligning the column index parameter with the primary key sequence of the target entity's business; when it is determined that the source entity and the target entity have a physical binding mapping relationship, the matrix element at the corresponding coordinate position in the two-dimensional mapping matrix takes the value of the number 1, and when it is determined that there is no physical binding mapping relationship, the matrix element takes the value of the number 0.

[0039] The adjudication of the aforementioned physical binding mapping relationship relies not only on foreign key connections but also on the hash-based tracing of the underlying core identity credentials. The data processing module extracts the vehicle identification number from the offline table of individual profiles, the vehicle unit physical serial number from the electronic non-stop toll collection data, and the salted and anonymized mobile phone number from the mobile application user data to construct three types of specific retrieval features. The system concurrently accesses the globally unified traffic management real-name authentication benchmark library and extracts the hash value of the natural person's resident identity code or the encrypted text of the legal person's unified social credit code associated with the above three types of specific retrieval features. The system determines that the corresponding source entity and target entity belong to the same exact physical subject only when retrieval features belonging to different data source dimensions collide with a matching encrypted identity in the benchmark library. In response to the frequent occurrence of overlapping records of multiple accounts for a single vehicle due to used car transfers or device rebindings in real-world traffic scenarios, the system extracts the timestamp parameter from the historical table transaction data, performs descending sorting arbitration, and exclusively filters and retains the unique attribution link with the latest timestamp and an active registration status. This engineering judgment mechanism, which introduces encrypted identity comparison and time-series conflict cleanup, ensures the accuracy of the cross-domain mapping matrix coordinate system generation state from the physical source.

[0040] Determine whether the offline base entity bitmap and the ad-hoc dynamic entity bitmap belong to the same entity category. If they belong to different entity categories, extract the source entity bit index parameter that is set in both the offline base entity bitmap and the ad-hoc dynamic entity bitmap. The memory addressing module uses the received source entity bit index parameter to search for and retrieve the associated mapping row vector in the preset cross-domain mapping matrix.

[0041] Extract the coordinate values ​​corresponding to the numerical value 1 in the source entity bit index parameter; extract the corresponding associated mapping row vectors in the two-dimensional mapping matrix based on the coordinate values. Perform a bitwise OR operation on the values ​​of the columns corresponding to all loaded associated mapping row vectors, merge overlapping dimensions, and output the bitmap of the same source target entity with aligned dimensions.

[0042] By employing the aforementioned hardware-level bitwise OR operation logic, the Cartesian product multiplier inflation phenomenon, which is easily caused by traditional relational databases when performing multi-source wide table joins, can be offset at the register level. The two-dimensional cross-domain mapping matrix is ​​arranged in memory as a sparse tensor, with its column index strictly aligned to the unique target entity's business primary key. When massive heterogeneous businesses undergo many-to-one aggregation, for example, when a single natural person target entity is simultaneously associated with the primary keys of three mobile applications in the active state in the source entity bitmap domain, the central processing unit schedules the underlying single-instruction multiple-data-stream instruction set to vertically align the three extracted association mapping row vectors in the cache. Since the primary keys of these three source entities are all physically bound to this natural person, their row vectors all exhibit a Boolean high level at the column coordinates corresponding to this natural person. After vertical span bitwise OR merging interception, multiple redundant high-level signals are merged and calculated into a single binary value of 1 at the logic operator level. This mechanism uses in-situ state overwriting to replace the detailed row copy accumulation of relational algebra, ensuring that the memory resident scale of the entire cycle of hybrid tag distribution calculation is limited to the upper limit of the target subject's deduplication cardinality.

[0043] Step S6: Perform bitwise operations on the source target entity bitmap according to the non-leaf node operators to generate the final mixed bitmap, and output the target selection result set based on the final mixed bitmap.

[0044] like Figure 6 As shown, the non-leaf node operators are extracted from the breakpoint positions of the split multi-level logical syntax tree. When the non-leaf node operator is an intersection operation instruction, a bitwise AND logical call is performed on multiple source target entity bitmaps at the same underlying data dimension to clear the region lacking double-ended active state support for associated offsets, generating the final mixed bitmap that retains the overlapping set parameters; when the non-leaf node operator is a union operation instruction, a bitwise OR logical call is performed on multiple source target entity bitmaps at the same underlying data dimension to generate the final mixed bitmap that retains the full set superposition parameters.

[0045] The decoder performs a physical bit-to-primary key dictionary inverse conversion operation on the active site indexes in the final mixed bitmap, converts the returned primary key string into a detailed user information list, and encapsulates and outputs the target selection result set.

[0046] After outputting the target selection result set, the target selection result set is written into the group individual relationship history table in the offline data warehouse. The total number of members included in the target selection result set is summarized according to the group identifier to generate group personnel statistics. The data flow pipeline, relying on a primary key index model, synchronously writes the group personnel statistics along with the group individual relationship history table into the columnar storage accelerated computing node, using a primary key index overwrite mechanism to ensure the retention of the latest status parameters corresponding to each group identifier.

[0047] Obviously, those skilled in the art can make various modifications and variations to this invention without departing from its spirit and scope. Therefore, if these modifications and variations fall within the scope of the claims of this invention and their equivalents, this invention also intends to include these modifications and variations.

Claims

1. A method for calculating the offline basis and ad-hoc dynamic hybrid label distribution of massive traffic flow data, characterized in that, include: Read the offline baseline data and ad-hoc configuration data of the individual profile, perform a federated query operation based on the offline baseline data and ad-hoc configuration data of the individual profile, and output the mixed label presentation result; Receive the group selection rule string; The group selection rule string is parsed to generate a multi-level logical syntax tree; Based on the statistical parameters associated with the condition leaf nodes, the multi-level logical syntax tree is split into an offline static execution subtree and an ad-hoc dynamic execution subtree; Based on the offline static execution subtree, a transformation operation is performed in the offline data warehouse to generate an offline base entity bitmap; The ad-hoc dynamic execution subtree is sent to the real-time analysis node to generate an ad-hoc dynamic entity bitmap. Based on a preset cross-domain mapping matrix, a dimensionality reduction operation is performed on the offline base entity bitmap and the ad hoc dynamic entity bitmap to transform the offline base entity bitmap and the ad hoc dynamic entity bitmap into homogeneous target entity bitmaps with the same primary key. Perform bitwise operations on the source target entity bitmap according to the non-leaf node operators to generate the final mixed bitmap, and output the target selection result set based on the final mixed bitmap.

2. The method for calculating the offline basis and ad-hoc dynamic hybrid label distribution of massive traffic flow data according to claim 1, characterized in that, The process of reading offline baseline data and ad-hoc configuration data for individual profiles includes: The offline processing framework is used to read vehicle business data associated with traffic entities, electronic non-stop toll collection user business data, and mobile application user business data. Batch processing calculations are performed on the vehicle business data, the electronic non-stop toll collection user business data, and the mobile application user business data according to the primary key dimension to generate the offline base table data of the individual profile. The enumerated category set and enumerated state parameters associated with the traffic entity are synchronized from the relational database as the ad hoc configuration data.

3. The method for calculating the offline basis and ad-hoc dynamic hybrid label distribution of massive traffic flow data according to claim 2, characterized in that, The federated query operation based on the offline baseline data of the individual profile and the ad-hoc configuration data outputs a mixed-label presentation result, including: Construct a federated mapping view layer that spans the offline data warehouse and the relational database in the data query layer; The label category name and label setting value are read from the ad-hoc configuration data through the federated mapping view layer; In the offline baseline data of the individual profile, the total number of covered entities corresponding to each of the set values ​​of the labels is counted. The total number of covered entities and the label category name are combined and integrated by dimension concatenation to output the hybrid label presentation result, which includes the personnel gathering scale parameter and the activity level stratification parameter.

4. The method for calculating the offline basis and ad-hoc dynamic hybrid label distribution of massive traffic flow data according to claim 3, characterized in that, The receiving group selection rule string includes: The group selection rule string, encapsulated in script object format, is obtained through the front-end interface application programming interface; Verify the outermost merge relationship identifier character included in the group selection rule string; Parse the inner nested array sequence in the group selection rule string, and extract the source table name parameter, drop-down field name parameter, field type parameter, comparison operator, and comparison assignment parameter included in the inner nested array sequence; Identify the conditional connector logical operator corresponding to the deepest nested level in the group selection rule string.

5. The method for calculating the offline basis and ad-hoc dynamic hybrid label distribution of massive traffic flow data according to claim 4, characterized in that, After outputting the target selection result set, the calculation method further includes: Write the target selection result set into the group-individual relationship history table of the offline data warehouse; The total number of members in the target selection result set is aggregated according to the group identifier to generate group personnel statistics; The statistical data of the group members and the historical table of the group's individual relationships are synchronized to the columnar storage acceleration node using the primary key index model, and the latest status parameters corresponding to each group identifier are retained for the front-end interface to perform query callback operations.

6. The method for calculating the offline basis and ad-hoc dynamic hybrid label distribution of massive traffic flow data according to claim 1, characterized in that, Based on the statistical parameters associated with the conditional leaf nodes, the multi-level logical syntax tree is split into an offline static execution subtree and an ad-hoc dynamic execution subtree, including: Extract the attribute field names bound to each condition leaf node in the multi-level logical syntax tree; Read the pre-calculated total tag coverage table in the offline data warehouse, and extract the statistical parameter that matches the attribute field name from the total tag coverage table; The conditional leaf nodes whose statistical parameters exceed the computing power consumption threshold, along with their connected parent node paths, are pruned into the offline static execution subtree. The conditional leaf nodes whose statistical parameters are not higher than the computing power consumption threshold, along with their associated parent node paths, are pruned into the ad hoc dynamic execution subtree.

7. The method for calculating the offline basis and ad-hoc dynamic hybrid label distribution of massive traffic flow data according to claim 6, characterized in that, The step of performing transformation operations in the offline data warehouse based on the offline static execution subtree to generate an offline base entity bitmap includes: The underlying query condition nodes in the offline static execution subtree are converted into full table scan structured instructions adapted to the offline data warehouse; Execute the full table scan structured instruction to extract the offline source entity business primary key sequence that meets the filtering conditions; A blank binary space sequence is allocated in the memory buffer area, and the corresponding bit positions in the blank binary space sequence are set to the active state according to the ascending order of the numerical parameters in the offline source entity business primary key sequence, thereby completing the generation and construction operation of the offline base entity bitmap.

8. The method for calculating the offline basis and ad-hoc dynamic hybrid label distribution of massive traffic flow data according to claim 1, characterized in that, The step of performing a dimensionality reduction operation on the offline base entity bitmap and the ad-hoc dynamic entity bitmap based on a preset cross-domain mapping matrix, transforming the offline base entity bitmap and the ad-hoc dynamic entity bitmap into homogeneous target entity bitmaps with the same primary key, includes: Determine whether the offline base entity bitmap and the ad-hoc dynamic entity bitmap belong to the same entity category identity; When the entity categories are determined to be different, the source entity bit index parameters that are set in the offline base entity bitmap and the ad hoc dynamic entity bitmap are extracted. Find the associated mapping row vector corresponding to the source entity bit index parameter in the preset cross-domain mapping matrix; Perform a bitwise OR operation on all the found associated mapping row vectors and output the same source target entity bitmap with the target entity primary key aligned.

9. The method for calculating the offline basis and ad-hoc dynamic hybrid label distribution of massive traffic flow data according to claim 8, characterized in that, The construction operation of the preset cross-domain mapping matrix includes: Extract the vehicle primary key vector, electronic non-stop toll collection primary key vector, and mobile application primary key vector from the traffic entities in the offline data warehouse according to the set batch processing cycle; Based on the vehicle primary key vector, the electronic non-stop toll collection primary key vector, and the mobile application primary key vector, a bidirectional pairing mapping calculation is performed to generate a list of attribution mapping relationships between each of the traffic entities. The attribution mapping list is reorganized into a two-dimensional mapping matrix with row index parameters and column index parameters, and the two-dimensional mapping matrix is ​​loaded into the real-time analysis node as the preset cross-domain mapping matrix.

10. The method for calculating the offline basis and ad-hoc dynamic hybrid label distribution of massive traffic flow data according to claim 1, characterized in that, The step of performing bitwise operations on the source target entity bitmap according to the non-leaf node operators to generate a final mixed bitmap, and outputting a target selection result set based on the final mixed bitmap, includes: Extract the non-leaf node operators from the breakpoint locations of the split multi-level logical syntax tree; When the non-leaf node operator is an intersection operation instruction, a bitwise AND logical call is performed on multiple source target entity bitmaps in the same dimension to generate the final mixed bitmap that retains the overlapping set parameters. When the non-leaf node operator is a union operation instruction, a bitwise OR logical call is performed on multiple source target entity bitmaps in the same dimension to generate the final mixed bitmap that retains all set parameters. The active bit indices in the final mixed bitmap are decoded and mapped to convert them into a detailed user information list, and the target selection result set is encapsulated and output.

Images