Mass log data archiving method for cross-border e-commerce platform

Abstract

The invention relates to the technical field of data processing, in particular to a mass log data archiving method for a cross-border e-commerce platform, which comprises the following steps: receiving cross-border e-commerce transaction log streams, analyzing and extracting user session IDs and commodity SKU codes in logs, counting the frequency of sharing the user session IDs or the commodity SKU codes among different log entries in a buffer area, and storing the user session IDs or the commodity SKU codes in the buffer area. And establishing a log attribute association frequency table, and calculating log clustering coefficient indexes of different log entries on the clustering dimension according to the log attribute association frequency table. According to the method, the compressed sequence is generated through line number difference calculation, difference distribution density is counted to serve as a storage format decision basis, an array container format is selected to reduce storage space occupation when the distribution density is lower than a threshold value, and a bitmap container format is automatically switched to improve bit operation efficiency when the distribution density is higher than the threshold value. And the established adaptive sparse bitmap index can dynamically balance the storage cost and the retrieval speed in different data sparseness scenes.

Description

TECHNICAL FIELD

[0002] The present application relates to the technical field of data processing, and particularly relates to a mass log data archiving method of a cross-border e-commerce platform. BACKGROUND

[0003] Data processing technology covers the whole process technology set of automatic collection, conversion, cleaning, classification, summarization, storage and retrieval of raw data, which is used to convert heterogeneous data in disorder into information resources with logic and usability.

[0004] The prior art usually adopts a time series-based append write strategy for log archiving, and the data landing order follows the arrival timestamp. In the cross-border e-commerce involving long-chain transactions and asynchronous interaction scenarios, the logically strong correlation events such as order creation, payment confirmation and logistics update often have a time span, which causes the semantic related records to be stored in physically distant disk blocks, and when performing specific transaction link retrieval, a large amount of irrelevant data segments have to be scanned, causing disk input and output jitter and high latency. Therefore, improvement is needed. SUMMARY

[0005] The purpose of the present application is to solve the shortcomings in the prior art, and a mass log data archiving method of a cross-border e-commerce platform is proposed.

[0006] In order to achieve the above purpose, the technical scheme adopted by the present application is as follows: a mass log data archiving method of a cross-border e-commerce platform, comprising the following steps: Receiving a cross-border e-commerce transaction log stream, parsing and extracting the user session ID and the commodity SKU code in the log, counting the frequency of shared user session IDs or commodity SKU codes between different log entries in the buffer area, establishing a log attribute correlation frequency table, and calculating the log clustering coefficient index of different log entries in the clustering dimension according to the log attribute correlation frequency table; Comparing the log clustering coefficient index with a preset correlation threshold, screening to generate a high-correlation log group, and reorganizing the physical storage location of the high-correlation log group, storing multiple related logs in the same continuous storage space, and constructing an affinity data block structure; Scanning the advertisement source identifier column and the coupon code column in the affinity data block structure, identifying the non-empty data and the row number position thereof, extracting the row number value of the non-empty data, generating a non-empty row number list, calculating the difference value between adjacent row numbers according to the non-empty row number list, and generating a row number difference value compression sequence; The distribution density of the row number difference compression sequence is counted, and the distribution density is compared with a preset container switching threshold value. If the distribution density is lower than the container switching threshold value, the array container format is selected, otherwise the bitmap container format is selected. A container type selection instruction is generated, and the index data is encapsulated and written into the corresponding physical storage container according to the container type selection instruction, so as to establish an adaptive sparse bitmap index.

[0007] Preferably, the step of obtaining the log clustering coefficient indicator is: According to the cross-border e-commerce transaction log stream, the user session ID and the commodity SKU code are parsed, structured parsing is performed, and null values and illegal characters are removed. Log entries with the same user session ID or commodity SKU code are merged and marked with the corresponding buffer row number position, and a user session ID and commodity SKU code mapping table is generated. According to the user session ID and commodity SKU code mapping table, the buffer log entries are compared row by row, the number of pairs of log entries sharing the same user session ID or commodity SKU code is counted, the results with the number of occurrences greater than or equal to the lower limit of the number of occurrences are summarized, and the results less than the lower limit of the number of occurrences are discarded. The statistical results are sorted and the source row number position is recorded to obtain a log attribute association frequency table. According to the log attribute association frequency table, the log clustering coefficient indicator is calculated.

[0008] Preferably, the step of obtaining the high-association-degree log grouping is: According to the log clustering coefficient indicator, the association threshold value is compared piece by piece, the log entries greater than the association threshold value are marked, the user session ID, the commodity SKU code and the full-link tracking ID are recorded, the numbering is sorted in ascending order of buffer row number, and a threshold-passed log entry list is formed. According to the threshold-passed log entry list, the full-link tracking ID column is read, the same full-link tracking ID is aggregated, the aggregated results containing only a single entry are removed, the entries are merged in ascending order of log timestamp, and a high-association-degree log grouping is generated.

[0009] Preferably, the step of obtaining the affinity data block structure is: According to the high-association-degree log grouping, the length of the continuous storage space required by each grouping is calculated, the physical storage location is allocated and the block start offset is adjusted, multiple associated logs are written into the same continuous storage space in the order of log timestamp, and the affinity data block structure is formed.

[0010] Preferably, the step of obtaining the non-empty row number list is: According to the affinity data block structure, the advertisement source identifier column and the coupon code column are located, the field values are read row by row, the empty strings and the contents containing only spaces are removed, the buffer row numbers where the non-empty fields are located are recorded and are de-duplicated in the order of occurrence, and a non-empty row number list is generated.

[0011] Preferably, the obtaining step of the row number difference compression sequence is: According to the non-empty row number list, merging the repeated items of the same row number, marking the starting position and ending position of each continuous row number, and generating an ordered non-empty row number list; According to the ordered non-empty row number list, calculating the difference between the next row number and the previous row number according to the adjacent index in turn, ignoring the difference vacancy of the first row number, continuously writing the difference value sequence and deleting the repeated section of zero difference, and generating a row number difference compression sequence.

[0012] Preferably, the obtaining step of the container type selection instruction is: According to the row number difference compression sequence, extracting the numerical value of all row number difference elements, calculating the row number difference sum and counting the number of elements, and calculating the distribution density according to the overall distribution of the row number difference; According to the distribution density, reading the container switching threshold value, selecting the array container format when the distribution density is less than the container switching threshold value, and selecting the bitmap container format when the distribution density is greater than or equal to the container switching threshold value, recording the container selection logic and the corresponding storage instruction, and generating a container type selection instruction.

[0013] Preferably, the obtaining step of the adaptive sparse bitmap index is: According to the container type selection instruction, extracting the storage path and data packaging rule of the corresponding container format, writing the row number difference compression sequence and the offset position into the physical storage container according to the selected format, updating the index mapping table item, and generating an adaptive sparse bitmap index.

[0014] Compared with the prior art, the application has the advantages and positive effects that: In the application, by analyzing the cross-border e-commerce transaction log stream and extracting user session and commodity code features, a fine correlation model based on attribute sharing frequency is constructed, the log clustering coefficient index is calculated to quantify the semantic closeness between discrete log entries, and the high correlation degree group is screened according to the correlation threshold, and the physical storage location is reorganized, and multiple logically related logs are stored in the same continuous storage space to construct an affinity data block. The physical level reorganization reduces the disk seek times and random read overhead when searching multiple logs, and ensures that the storage layout is highly consistent with the business query logic. Scan the sparse fields such as advertisement source and coupon code to identify non-empty data positions, use the row number difference value to calculate the compressed sequence, and use the difference value distribution density as the basis for storage format decision, and when the distribution density is lower than the threshold, the array container format is selected to reduce the storage space occupation, and when it is higher than the threshold, the bitmap container format is automatically switched to improve the bit operation efficiency, and the adaptive sparse bitmap index can dynamically balance the storage cost and retrieval speed in different data sparsity scenarios. Through the synergistic effect of data physical aggregation and adaptive adjustment of index structure, the query response performance of the full link tracking of specific sessions or commodities is improved. BRIEF DESCRIPTION OF DRAWINGS

[0015] Figure 1 The schematic diagram of the steps of the application. DETAILED DESCRIPTION

[0016] In order to make the purpose, technical scheme and advantages of the application clearer, the application will be further described in detail below in combination with the drawings and examples. It should be understood that the specific examples described herein are only used to explain the application and not to limit the application.

[0017] Please refer to Figure 1 The application provides a technical scheme, a mass log data archiving method of a cross-border e-commerce platform, comprising the following steps: Receiving a cross-border e-commerce transaction log stream, parsing and extracting user session ID and commodity SKU code in the log, counting the frequency of sharing user session ID or commodity SKU code between different log entries in the buffer, establishing a log attribute association frequency table, and calculating the log clustering coefficient index of different log entries in the clustering dimension according to the log attribute association frequency table; Comparing the log clustering coefficient index with the preset correlation threshold to screen and generate a high correlation degree log group, and reorganizing the physical storage location of the high correlation degree log group, storing multiple associated logs in the same continuous storage space to construct an affinity data block structure; Scanning the advertisement source identifier column and the coupon code column in the affinity data block structure to identify non-empty data and the row number position, extracting the row number value of the non-empty data, generating a non-empty row number list, calculating the difference value between adjacent row numbers according to the non-empty row number list, and generating a row number difference value compressed sequence; The distribution density of the compressed sequence of row number difference is calculated, and the distribution density is compared with the preset container switching threshold. If the distribution density is lower than the container switching threshold, the array container format is selected; otherwise, the location map container format is selected. A container type selection instruction is generated, and the index data is encapsulated and written into the corresponding physical storage container according to the container type selection instruction to establish an adaptive sparse bitmap index.

[0018] The steps to obtain the log clustering coefficient metric are as follows: Based on the cross-border e-commerce transaction log stream, the user session ID and product SKU code are parsed, structured parsing is performed, and null values ​​and illegal characters are removed. Log entries with the same user session ID or product SKU code are merged and the corresponding buffer line number position is marked to generate a mapping table between user session ID and product SKU code. Based on the mapping table between user session ID and product SKU code, compare the buffer log entries line by line, count the number of times pairs of users sharing the same user session ID or product SKU code appear, summarize the results whose occurrence count is greater than or equal to the lower limit of occurrence count, discard the results whose occurrence count is less than the lower limit of occurrence count, organize the statistical results and record the source line number position to obtain the log attribute association frequency table; Based on the log attribute association frequency table, the log clustering coefficient index is calculated using the following formula: ; in, For the first The log clustering coefficient index of log entries. The user session ID value is equal to the first one. The frequency of paired occurrences of user session ID values ​​for each log entry. The product SKU code value is equal to the number of... The frequency of paired occurrences of product SKU code values ​​for each log entry. The absolute difference between two frequencies. This is the sum of the two frequencies, used to normalize the differences and ensure dimensional consistency.

[0019] Specifically, based on cross-border e-commerce transaction log streams, the system reads raw binary log data from the memory buffer in real time through a pre-defined log collection interface. The binary stream is converted into a string text according to UTF-8 encoding. Regular expressions are used to match key field locators in each line of log text, identifying user session ID fields starting with specific prefixes such as "UID:" and product SKU code fields starting with "SKU:". The string content after the colon in each field is extracted as the value to be processed. The extracted strings undergo validity checks, and a list containing common illegal characters such as "" is constructed. <script>”、"NULL”、"undefined”以及不可见控制字符的非法字符集，遍历提取到的ID与编码字符串，一旦发现字符串中包含非法字符集中的元素，则立即将该字段标记为无效并置空，同时检查字符串长度是否符合预定义的长度标准，例如用户会话ID通常为32位UUID，商品SKU编码为10位至15位数字字母组合，剔除长度不符的噪声数据，在清洗完成后，在内存中创建一个哈希映射结构，以清洗后的用户会话ID或商品SKU编码作为哈希键，以当前日志条目在缓冲区中的数组下标索引即缓冲区行号作为哈希值，遍历缓冲区内的所有日志条目，将具有相同键值的日志条目行号追加到对应的哈希值列表中，对于列表中行号数量大于1的键值对，视为存在关联的日志条目并进行合并存储操作，将同一ID下的所有行号组织成一个紧凑的整型数组，明确标注每个行号对应的物理内存地址偏移量，最终将整理好的哈希结构转换为结构化的查找表，生成用户会话ID与商品SKU编码映射表。

[0020] 根据用户会话ID与商品SKU编码映射表，读取映射表中存储的所有键值对列表，对于每一个键值对中的行号列表，采用双重循环遍历的方式生成该列表中所有行号的两两组合，例如若列表中包含行号A、B、C，则生成(A,B)、(A,C)、(B,C)三对组合，针对生成的每一对行号组合，在内存中维护一个全局的关联计数矩阵，矩阵的行索引和列索引分别对应缓冲区中的两个日志行号，将对应位置的计数器数值加一，用以表征这两条日志在同一用户会话或同一商品维度下的共现关系，遍历完整的映射表后，统计矩阵中每个非零元素的数值作为成对出现次数，读取预设的出现次数下限，该下限的设置过程为，选取过去24小时内系统处理的历史日志块作为样本数据，统计样本中所有日志对的平均关联次数以及标准差，设定出现次数下限，例如在一次实际采样计算中，平均关联次数为12次，标准差为4次，则计算下限次，将计数矩阵中数值小于10的元素置为0，表示该关联关系强度不足，属于随机噪声或弱关联，予以舍弃，保留数值大于或等于10的元素，并将这些保留下来的高频关联对提取出来，记录每一对关联日志的来源行号位置以及对应的具体的关联频次数值，将这些分散的统计数据按照行号索引进行重新排序和归集，构建一个包含源行号、目标行号及关联频次的结构化数据表，得到日志属性关联频次表。

[0021] 日志聚类系数指标计算公式中，通过对数函数衡量日志条目在用户与商品两个维度上的总活跃度，同时利用指数衰减函数衡量两个维度贡献的平衡性，以此综合评价日志的聚类紧密度；参数的获取步骤为：该参数代表第条日志条目在用户会话维度上的关联强度，具体定义为在当前分析的缓冲区周期内，与该条日志具有相同用户会话ID的其他日志条目的数量。该参数是一个无量纲的整数值，直接反映了该日志所在会话的深度。获取过程为：遍历日志属性关联频次表，检索所有包含第号日志且关联类型标记为"用户会话ID”的记录行，统计这些记录行的数量。例如，在一次针对"黑色星期五”促销活动的日志分析中，缓冲区内第50号日志条目记录了一次用户点击行为，系统通过查询频次表发现，该日志的用户会话ID在缓冲区内还出现在另外25条日志中（即存在25个关联对），这意味着该用户在当前窗口内产生了26条记录（包含自身），因此确定的值为25。

[0022] 参数的获取步骤为：该参数代表第条日志条目在商品SKU编码维度上的关联热度，具体定义为在当前缓冲区内，与该条日志具有相同商品SKU编码的其他日志条目的数量。该参数同样为无量纲整数，用于量化商品的热度。获取方式为：在日志属性关联频次表中，筛选出所有涉及第号日志且关联属性为"商品SKU编码”的条目，统计满足条件的条目总数。例如，对于上述第50号日志，其涉及的商品SKU在同一时间窗口内被其他用户的会话访问，频次表中记录了该SKU与其他45条日志存在共现关系，这表明该商品在当前批次中被高频访问，因此确定的值为45。

[0023] 根据参数进行计算：基于上述实际算例获取的参数数值，即第50号日志条目的用户会话关联计数，商品SKU关联计数，代入公式进行计算：第一步，计算两个频次的代数和，作为总活跃度指标：；第二步，计算对数项，对总活跃度进行平滑处理：；第三步，计算两个频次之间的绝对差值：；第四步，计算差异比率：；第五步，计算指数衰减项，该项取值范围在0到1之间：；第六步，将对数项与指数项相乘，得到最终的日志聚类系数指标：；计算得到的日志聚类系数指标是一个无量纲的综合评分。该结果表明第50号日志条目在当前缓冲区中处于高关联状态，且其在用户维度和商品维度的关联分布较为均衡（指数项0.7515接近1，说明差异不大）。相比于仅有单一维度高频次（例如仅有SKU爆发但无用户深度）的日志，该日志更适合作为归档块的"锚点”，系统将依据此数值将其分配至热点数据存储区，并与其关联的25条同会话日志和45条同商品日志尽可能存放在连续的物理块中。

[0024] 高关联度日志分组的获取步骤为：根据日志聚类系数指标，逐条比对关联阈值，标记大于关联阈值的日志条目，记录用户会话ID、商品SKU编码及全链路追踪ID，按缓冲区行号升序整理编号，形成阈值通过日志条目列表；根据阈值通过日志条目列表，读取全链路追踪ID列，按相同全链路追踪ID聚合，将仅含单条的聚合结果剔除，按日志时间戳升序归并条目，生成高关联度日志分组。

[0025] 具体的，根据日志聚类系数指标，读取预设的关联阈值，该阈值的设定依据为缓冲区历史日志条目的聚类系数统计分布，选取最近5个缓冲周期的日志聚类系数作为样本集合，计算样本集合的平均值与标准差，设定调节因子为0.6，计算公式为，例如在一次实际运算中，历史平均系数为3.2，标准差为1.5，则计算得出关联阈值，将当前缓冲区内计算所得的每一条日志聚类系数指标与该关联阈值进行数值比对，保留系数指标数值大于4.1的日志条目，将其标记为活跃数据，对于保留下来的日志条目，解析其内部的键值对结构，定位并提取用户会话ID字段、商品SKU编码字段以及全链路追踪ID字段，其中全链路追踪ID需通过正则表达式匹配以"Trace-Id”或"X-B3-TraceId”为键名的头部信息获取，若字段缺失则填充默认的空标识符，同时读取每条日志在原始缓冲区中的行号索引值，该索引值代表了日志进入内存的物理顺序，创建一个临时的结构体数组来存储提取出的ID信息与行号，采用快速排序算法，以缓冲区行号作为排序键值，对结构体数组进行升序排列，确保后续处理遵循原始数据的写入时序，将排序完成后的结构体对象依次追加写入到内存中的列表中，形成阈值通过日志条目列表。

[0026] 根据阈值通过日志条目列表，遍历列表中的每一条记录，识别记录中包含的全链路追踪ID字段，该字段通常为一个32位或64位的唯一十六进制字符串，用于标识一次完整的分布式请求调用链，构建一个哈希映射容器，以全链路追踪ID作为键，以包含用户会话ID、商品SKU编码及原始内容引用的日志对象列表作为值，将遍历到的每一条日志按照其全链路追踪ID分发至哈希映射容器的对应槽位中，完成初步的归类聚合，待所有日志分发完毕后，遍历哈希映射容器的所有键值对，检查每个键对应的日志对象列表的元素数量，设定聚合数量下限为2，即认为单条日志无法构成关联上下文，若列表长度小于2，则直接从哈希映射中移除该键值对，将仅含单条的聚合结果剔除，对于保留下来的每一个键值对，读取其值列表中每一条日志的时间戳字段，将时间戳字符串转换为长整型毫秒数值，调用排序函数，按照时间戳数值由小到大的顺序对列表内的日志条目进行重排，确保同一调用链内的日志按照业务发生的先后逻辑顺序排列，将排序后的各个列表合并存入一个新的动态数组中，生成高关联度日志分组。

[0027] 亲和力数据块结构的获取步骤为：根据高关联度日志分组，计算每个分组所需连续存储空间长度，分配物理存储位置并调整块起始偏移量，按日志时间戳顺序将多条关联日志写入同一连续存储空间，形成亲和力数据块结构。

[0028] 具体的，根据高关联度日志分组，预先扫描每一个高关联度日志分组内的所有日志条目，读取每一条日志的字节长度属性，累加分组内所有日志的长度值，并额外增加预设的分组头数据长度，例如每个分组头占用16字节用于存储分组ID和总长度信息，从而得出每个分组所需的连续存储空间总长度，在内存堆区中申请一块足够容纳当前批次所有分组数据的连续字节缓冲区，定义一个写指针变量指向该缓冲区的起始内存地址，遍历每一个分组，首先在写指针当前指向的位置写入分组头信息，随后将写指针向后移动16个字节，接着按照分组内日志的时间戳顺序，依次将多条日志的二进制内容通过内存拷贝操作复制到缓冲区中，每复制一条日志，根据该日志的实际字节长度将写指针向后移动相应的偏移量，确保每条日志在物理内存上是紧密相邻存储的，中间无内存碎片的间隔，待一个分组的所有日志写入完成后，记录当前分组在缓冲区中的起始偏移量与结束偏移量，作为后续索引构建的物理地址依据，重复上述过程直至所有分组均写入完毕，将这块包含紧凑排列日志数据的内存区域标记为待持久化状态，形成亲和力数据块结构。

[0029] 非空行号列表的获取步骤为：根据亲和力数据块结构，定位广告来源标识列与优惠券代码列，逐行读取字段值，剔除空字符串与仅含空白的内容，记录非空字段所在的缓冲区行号并按出现顺序去重，生成非空行号列表。

[0030] 具体的，根据亲和力数据块结构，访问存储块头部定义的数据字典与元数据区，根据预先定义的列式存储布局协议，解析出"广告来源标识”字段与"优惠券代码”字段在连续存储空间中的起始内存偏移地址与字段字节长度，初始化一个遍历指针指向数据块的首行位置，设定非空判定的字符编码有效范围，该范围依据ASCII编码标准设定，将十进制数值32（空格符）及以下的控制字符定义为无效内容，将十进制数值126（波浪号）以上的扩展字符或特定非法字符也纳入过滤范畴，启动行级扫描循环，逐行读取上述两个目标列的字节序列，首先检查字节长度属性，若长度为0则直接跳过，若长度非0则进入字符内容扫描，遍历字节序列中的每一个字节，一旦发现存在ASCII码值在33至126之间的可见字符，即判定该字段为非空有效数据，记录当前日志条目在亲和力数据块内的相对缓冲区行号（例如第5行、第12行），为了确保行号记录的唯一性并去除因单行日志同时包含广告标识与优惠券代码而产生的重复索引，在内存中构建一个布尔类型的位图（BitMap）过滤器，该过滤器的位宽等于数据块内的总行数，当扫描到有效行号时，检查位图中对应的位状态，若该位为0，则将其置为1并将行号加入临时列表，若该位已为1，则说明该行号已被记录，执行跳过操作，按照日志在物理空间中的存储顺序依次完成所有行的扫描与判定，最终输出一个不包含重复项且仅包含有效数据所在位置的非空行号列表。

[0031] 行号差值压缩序列的获取步骤为：根据非空行号列表，合并相同行号的重复项，标注每一段连续行号的起始位置与结束位置，生成有序非空行号列表；根据有序非空行号列表，按相邻索引依次计算后一行号减前一行号的差值，忽略首个行号的差值空缺，连续写入差值序列并删除零差值的重复段，生成行号差值压缩序列。

[0032] 具体的，根据非空行号列表，调用快速排序算法，以行号数值大小为比较键值，对列表中捕获的所有行号进行升序排列，将原本按出现顺序排列（可能因多列扫描导致局部乱序）的索引整理为严格单调递增的序列，遍历排序后的列表进行二次清洗，设置双指针结构，主指针指向当前处理元素，副指针指向前一个有效元素，比较两者数值，若发现相邻两个行号数值完全相等，这通常源于多线程扫描合并时的残留重复项，则剔除当前元素，仅保留一个副本，在确保列表元素唯一性的同时，启动连续性检测机制，初始化段落标记变量，逐一计算相邻行号的差值，若差值等于1，表明行号在物理上是连续紧密排列的（例如101、102、103），此时仅更新当前连续段的结束位置指针而不新增记录，若差值大于1，表明出现了数据断层（例如103之后是108），此时立即结束前一个连续段的标记，记录该段的起始行号与终止行号作为元数据，并开启一个新的段落记录，这种分段标注为后续的数据压缩提供了结构化依据，经过排序、去重及分段标注处理后，将纯净的行号序列与段落元数据打包，生成有序非空行号列表。

[0033] 根据有序非空行号列表，进入差分编码处理阶段，该阶段旨在通过存储相邻数值的差量来降低数据存储位宽，分配一个专用的输出缓冲区用于存放压缩数据，读取有序列表中的第一个行号数值作为"锚点”或"基准值”单独存储在索引头部的基准字段中，随后初始化前序值寄存器为该基准值，从列表的第二个元素开始进行迭代处理，在每次迭代中，读取当前位置的行号数值，执行减法运算，用当前行号减去前序值寄存器中的数值，得到一个正整数差值，例如对于序列[100, 102, 105]，基准为100，第一个差值为102-100=2，第二个差值为105-102=3，在计算过程中，设置零值过滤器，若计算出的差值为0，说明在之前的去重步骤中仍有遗漏或数据异常，此时直接丢弃该差值不予写入，防止产生无效的占位符，仅当差值大于0时，将其转换为紧凑的二进制格式（如Varint变长编码）追加写入差值序列中，每完成一次写入，立即更新前序值寄存器为当前行号，以便进行下一次的差值计算，重复此步骤直至列表末尾，最终得到一个由一系列小整数组成的序列，生成行号差值压缩序列。

[0034] 容器类型选择指令的获取步骤为：根据行号差值压缩序列，提取所有行号差值元素的数值，计算行号差值总和并统计元素数量，依据行号差值的整体分布计算分布密度，计算公式为：；其中，为分布密度，为行号差值压缩序列中的元素数量，为第个行号差值元素，为行号差值索引下标，为行号差值的平均值，表示行号差值序列的总体波动幅度；根据分布密度，读取容器切换阈值，分布密度小于容器切换阈值则选择数组容器格式，分布密度大于或等于容器切换阈值则选择位图容器格式，记录容器选择逻辑及对应存储指令，生成容器类型选择指令。

[0035] 具体的，分布密度计算公式中，其中分母部分结合了总跨度（即所有差值之和）与分布的不均匀性惩罚项（波动幅度），使得该密度值能够同时感知数据的稀疏性与聚集性，当差值越小且分布越均匀时，分母越小，密度值越高，反之则越低，以此精确指导后续存储容器的选型；参数的获取步骤为：该参数代表行号差值压缩序列中的元素总数量，即当前处理的数据块中包含的非空日志索引条目的个数。该参数是一个无量纲的整数值，直接决定了索引块的大小规模。获取过程为：读取内存中生成的"行号差值压缩序列”对象，调用该列表对象的计数方法，遍历序列中的所有有效差值元素并进行累加计数。例如，在一次针对"双十一”大促期间的日志归档处理中，内存中的压缩序列列表包含了从缓冲区第5行到第505行之间的所有非空记录索引，经过去重和差分处理后，列表中实际存储了5个有效的差值数据（代表5条日志记录），程序通过读取列表长度属性，确定的数值为5。

[0036] 参数的获取步骤为：该参数代表行号差值压缩序列中第个具体位置上的差值数值，反映了相邻两条关联日志在物理存储空间上的间距。该参数为无量纲的整数。获取方式为：通过循环遍历的方式访问"行号差值压缩序列”，按照索引下标从1到依次提取对应的整数值。例如，在上述"双十一”案例的序列中，实际存储的差值数据为[10, 12, 10, 8, 10]，则，，，，。这些数值表明日志索引之间的间距在10左右波动。

[0037] 参数的获取步骤为：该参数代表所有行号差值元素的平均值，用于衡量整体的平均分布间距。该参数为无量纲的实数。获取过程为：首先对序列中所有的值进行累加求和，得到差值总和，然后将该总和除以元素数量。计算公式为：。在上述案例中，差值总和为，元素数量，则计算。

[0038] 根据参数进行计算：基于上述实际算例获取的参数数值：元素数量，差值序列，平均值，代入公式进行计算：第一步，计算分母中的第一部分，即差值总和：；第二步，计算每个元素与平均值的偏差平方：；；；；；第三步，计算偏差平方和：；第四步，计算分母中的波动幅度项（惩罚项）：；第五步，计算分母总值：；第六步，计算分布密度：；计算得到的分布密度。该结果表明当前的日志索引序列呈现出较低的分布密度（稀疏分布），因为平均间距较大且存在一定波动。该数值将直接作为决策依据，用于后续步骤中容器格式的选择。数值越接近1表示数据越密集（如连续整数），数值越接近0表示数据越稀疏（间距很大）。在此例中，0.0888的低密度意味着使用位图（Bitmap）存储可能造成空间浪费，倾向于选择数组容器。

[0039] 根据分布密度，调用预置的配置读取接口获取"容器切换阈值”，该阈值的设定基于存储开销的盈亏平衡点分析，选取不同基数（Cardinality）的数据集进行测试，对比数组容器（Array Container，通常每个元素占2字节）与位图容器（Bitmap Container，固定占8KB即65536位）的内存占用情况，当数组容器的总大小超过位图容器大小时即为切换点，例如在16位整数范围内，位图固定占用8192字节，而数组每个元素2字节，平衡点为8192 / 2= 4096个元素，对应的密度阈值计算为 4096 / 65536 = 0.0625，考虑到索引压缩头部开销及CPU处理效率，将实际工程阈值设定为，将前一步骤计算所得的分布密度与该阈值进行比对，判定，符合稀疏存储特征，因此决策逻辑指向"数组容器格式”，若计算结果大于或等于0.1则指向"位图容器格式”，在内存中构建一个控制指令对象，设置其类型字段为"ARRAY_16”，并附带相关的编码参数（如是否启用Run-Length Encoding），将这一决策结果封装为结构化的操作码，生成容器类型选择指令。

[0040] 自适应稀疏位图索引的获取步骤为：根据容器类型选择指令，提取对应容器格式的存储路径与数据封装规则，将行号差值压缩序列及偏移位置按选定格式写入物理存储容器，更新索引映射表项，生成自适应稀疏位图索引。

[0041] 具体的，根据容器类型选择指令，解析指令中携带的容器格式标识码，根据标识码"ARRAY_16”加载对应的序列化驱动程序，在物理存储介质上分配一个新的索引数据页，在页头写入容器类型标记（0x01代表数组，0x02代表位图）以及基数信息，遍历内存中的行号差值压缩序列，将序列中的每一个差值整数转换为16位短整型（Short）二进制流，依次追加写入到数据页的有效载荷区域，同时维护一个当前写入位置的偏移量指针，每写入一个数据更新一次指针位置，在数据写入完成后，计算该数据页在文件系统中的全局偏移地址与长度，定位到全局索引映射表（Index Map）的对应槽位，将日志块的时间戳区间或ID范围作为键，将上述物理地址与容器类型作为值，更新映射表的元数据项，确保后续查询引擎能够通过映射表快速定位并解码该索引块，完成索引构建流程，生成自适应稀疏位图索引。

[0042] 以上，仅是本发明的较佳实施例而已，并非对本发明作其他形式的限制，任何熟悉本专业的技术人员可能利用上述揭示的技术内容加以变更或改型为等同变化的等效实施例应用于其他领域，但是凡是未脱离本发明技术方案内容，依据本发明的技术实质对以上实施例所做的任何简单修改、等同变化与改型，仍属于本发明技术方案的保护范围。< / script>

Claims

1. A method for archiving massive log data from cross-border e-commerce platforms, characterized in that, Includes the following steps: Receive cross-border e-commerce transaction log streams, parse and extract user session IDs and product SKU codes from the logs, count the frequency of sharing user session IDs or product SKU codes among different log entries in the buffer, establish a log attribute association frequency table, and calculate the log clustering coefficient index of different log entries on the clustering dimension based on the log attribute association frequency table. The log clustering coefficient index is compared with a preset association threshold to filter and generate highly associated log groups. The physical storage locations of the highly associated log groups are reorganized, and multiple associated logs are stored in the same contiguous storage space to construct an affinity data block structure. Scan the advertising source identifier column and coupon code column in the affinity data block structure, identify non-empty data and their row number positions, extract the row number values ​​of the non-empty data, generate a list of non-empty row numbers, calculate the difference between adjacent row numbers based on the list of non-empty row numbers, and generate a row number difference compressed sequence. The distribution density of the compressed sequence of row number difference is statistically analyzed, and the distribution density is compared with a preset container switching threshold. If the distribution density is lower than the container switching threshold, the array container format is selected; otherwise, the position map container format is selected. A container type selection instruction is generated, and the index data is encapsulated and written into the corresponding physical storage container according to the container type selection instruction to establish an adaptive sparse bitmap index.

2. The method for archiving massive log data of a cross-border e-commerce platform according to claim 1, characterized in that, The steps for obtaining the log clustering coefficient metric are as follows: Based on the cross-border e-commerce transaction log stream, the user session ID and product SKU code are parsed, structured parsing is performed, and null values ​​and illegal characters are removed. Log entries with the same user session ID or product SKU code are merged and the corresponding buffer line number position is marked to generate a mapping table between user session ID and product SKU code. Based on the user session ID and product SKU code mapping table, compare the buffer log entries line by line, count the number of times pairs of users sharing the same user session ID or product SKU code appear, summarize the results whose occurrence count is greater than or equal to the lower limit of occurrence count, discard the results whose occurrence count is less than the lower limit of occurrence count, organize the statistical results and record the source line number position to obtain the log attribute association frequency table. Calculate the log clustering coefficient based on the log attribute association frequency table.

3. The method for archiving massive log data of a cross-border e-commerce platform according to claim 1, characterized in that, The steps for obtaining the highly correlated log groups are as follows: Based on the log clustering coefficient index, each log entry is compared with the association threshold, and log entries that are greater than the association threshold are marked. The user session ID, product SKU code and end-to-end tracking ID are recorded and numbered in ascending order according to the buffer row number to form a list of log entries that pass the threshold. Based on the threshold, the end-to-end tracking ID column is read from the log entry list, and the entries are aggregated according to the same end-to-end tracking ID. Aggregation results containing only a single entry are removed, and the entries are merged in ascending order of log timestamp to generate highly correlated log groups.

4. The method for archiving massive log data of a cross-border e-commerce platform according to claim 1, characterized in that, The steps for obtaining the affinity data block structure are as follows: Based on the highly correlated log grouping, the length of the continuous storage space required for each group is calculated, the physical storage location is allocated and the block start offset is adjusted, and multiple correlated logs are written to the same continuous storage space in the order of log timestamps to form an affinity data block structure.

5. The method for archiving massive log data of a cross-border e-commerce platform according to claim 1, characterized in that, The steps for obtaining the list of non-empty line numbers are as follows: Based on the affinity data block structure, locate the advertising source identifier column and the coupon code column, read the field values ​​row by row, remove empty strings and content containing only blanks, record the buffer row number where the non-empty field is located and remove duplicates in the order of appearance, and generate a list of non-empty row numbers.

6. The method for archiving massive log data of a cross-border e-commerce platform according to claim 1, characterized in that, The steps for obtaining the row number difference compressed sequence are as follows: Based on the list of non-empty line numbers, duplicate items with the same line number are merged, and the start and end positions of each consecutive line number segment are marked to generate an ordered list of non-empty line numbers. Based on the ordered non-empty row number list, calculate the difference between the next row number and the previous row number in order of adjacent indices, ignore the missing difference in the first row number, continuously write the difference sequence and delete the duplicate segments with zero difference to generate a compressed row number difference sequence.

7. The method for archiving massive log data of a cross-border e-commerce platform according to claim 1, characterized in that, The steps for obtaining the container type selection instruction are as follows: Based on the compressed sequence of row number differences, extract the values ​​of all row number difference elements, calculate the sum of row number differences and count the number of elements, and calculate the distribution density based on the overall distribution of row number differences; Based on the distribution density, read the container switching threshold. If the distribution density is less than the container switching threshold, select the array container format. If the distribution density is greater than or equal to the container switching threshold, select the bitmap container format. Record the container selection logic and corresponding storage instructions, and generate a container type selection instruction.

8. The method for archiving massive log data of a cross-border e-commerce platform according to claim 1, characterized in that, The steps for obtaining the adaptive sparse bitmap index are as follows: Based on the container type selection instruction, the storage path and data encapsulation rules of the corresponding container format are extracted, the row number difference compression sequence and offset position are written into the physical storage container according to the selected format, the index mapping table entries are updated, and an adaptive sparse bitmap index is generated.

Images