E-commerce platform-oriented high-speed data stream detection method and device, terminal and storage medium

Abstract

The invention discloses a high-speed data flow detection method and device for an e-commerce platform, a terminal and a storage medium, and relates to the field of data flow detection of the e-commerce platform, the method is operated in a high-speed storage of a core server of the e-commerce platform, the high-speed storage comprises two storage structures of a large flow index table and an important element array, and the method comprises the steps that the large flow index table is stored in the high-speed storage; the two storage structures are initialized in parallel at the beginning of a monitoring period, index positions are calculated according to identifiers of data streams to which data stream elements belong, idle states of the index positions in a large stream index table are detected, and the two structures are cooperatively updated in combination with the identifiers and rules; and after the monitoring period is finished, screening the large-flow data flow from the large-flow index table, and screening the important elements from the important element array. According to the method, the large data stream and the important elements thereof can be efficiently and accurately screened from the high-speed data stream of the e-commerce platform, and the efficiency and the accuracy of data detection are improved.

Description

Technical Field

[0001] This application relates to the field of data stream detection for e-commerce platforms, and in particular to a high-speed data stream detection method, apparatus, terminal and storage medium for e-commerce platforms. Background Technology

[0002] In e-commerce platforms, content-driven marketing, represented by short videos and live streaming, has rapidly emerged, resulting in the continuous recording and conversion of user browsing, clicking, and purchasing behaviors into massive and rapidly flowing data streams. These high-speed data streams are characterized by a highly uneven distribution: a few popular products or categories account for the vast majority of traffic, known as "major flows." Real-time and accurate detection of these major flows, and further analysis of key products within them, can provide the platform with more granular and valuable reference information, which is crucial for grasping market trends, optimizing recommendation systems, and formulating precise marketing strategies.

[0003] However, traditional detection methods face severe challenges in high-speed data stream scenarios. Due to the extremely high data flow rate, the high-speed computing and storage resources available to the system are very limited, typically allowing only extremely simplified operations to be performed on each data point. Attempting to temporarily store all data in a high-speed cache for detailed analysis will result in rapid failure due to insufficient capacity; if the data is transferred to a larger general-purpose storage device, the update speed cannot match the input rate, leading to lag in real-time processing.

[0004] Therefore, existing technologies often employ a combination of sampling and offline analysis for approximate processing. However, these methods have inherent drawbacks: firstly, sampling inevitably loses key information, leading to inaccurate and incomplete detection results; secondly, offline analysis is complex and has high latency, failing to meet the urgent real-time requirements of e-commerce scenarios. More importantly, existing research typically focuses only on detecting the flow itself, neglecting in-depth analysis of important elements within the flow, making it difficult to provide more granular and valuable decision-making support for platform operations.

[0005] Therefore, there is an urgent need in this field for a lightweight real-time processing technology that can simultaneously achieve high-volume detection and effective screening of important elements in high-speed data streams within limited high-speed storage resources. Summary of the Invention

[0006] This application provides a high-speed data stream detection method, device, terminal, and storage medium for e-commerce platforms, which can significantly reduce computing and storage overhead, efficiently identify large data streams and their important elements, and provide more real-time, accurate, and fine-grained data support for content recommendation optimization, precision marketing, and sales decisions on e-commerce platforms.

[0007] Firstly, this application provides a high-speed data stream detection method for e-commerce platforms, employing the technical solution described below: A high-speed data stream detection method for e-commerce platforms is disclosed. The method operates in the high-speed storage of the core server of the e-commerce platform, which includes two storage structures. The method comprises the following steps: At the start of the current monitoring period, the two storage structures, including a high-current index table and an array of important elements, are initialized in parallel. The index position is calculated based on the data stream identifier of the data stream to which each arriving data stream element belongs. The index position points to both the large stream index table and the corresponding storage bucket in the important element array. The idle status of the index position in the large flow index table is detected, and the large flow index table and the important element array are updated collaboratively in combination with the data flow identifier and the preset data update rules; After the current monitoring period ends, the large-scale data stream is obtained by filtering from the large-scale data stream index table, and the important elements in each large-scale data stream are obtained by filtering from the important element array.

[0008] By adopting the above technical solution, the index position is calculated based on the data flow identifier, pointing to the storage buckets corresponding to both the main flow index table and the important element array, providing location for data updates. The two storage structures are updated collaboratively in conjunction with the data flow identifier, reflecting data flow and element information in real time. After the monitoring period ends, the main flow data flow and its important elements are selected from the updated two storage structures, providing e-commerce platforms with fine-grained and valuable reference information, helping to optimize recommendation systems, formulate precise marketing strategies, and improve user experience. This also avoids the shortcomings of sampling and offline analysis methods, reducing computational and storage overhead.

[0009] Preferably, at the start of the current monitoring period, the initialization of the two storage structures, which include a high-current index table and an array of important elements, specifically includes the following steps: At the start of the current monitoring cycle, two storage structures, a high-flow index table and an array of important elements, are deployed in the high-speed storage. The large flow index table includes a one-dimensional bucket array for recording each arriving data flow in real time. Each first storage bucket in the one-dimensional bucket array stores the data flow information of the data flow, which includes a data flow identifier and a data flow counter. The important element array includes a bucket array structure corresponding to the one-dimensional bucket array of the large flow index table. Each second storage bucket in the bucket array structure contains multiple storage units. Each storage unit stores element information of a data flow element. The element information includes an element identifier and an element frequency counter. Each of the first buckets in the large flow index table and each of the storage cells in the important element array are initialized to empty.

[0010] By adopting the above technical solution, a high-volume index table and an array of important elements are deployed and initialized in high-speed storage. Using a bucket array structure, data stream information and element information of data stream elements are stored respectively. This provides a foundation for subsequent real-time recording and analysis of high-speed data streams, ensuring that each arriving data stream and its elements can be accurately recorded in subsequent processing. This facilitates the subsequent filtering of high-volume data streams and their important elements, providing support for e-commerce platforms to optimize recommendation systems, formulate precise marketing strategies, and improve user experience.

[0011] Preferably, the step of calculating the index position based on the data stream identifier of the data stream to which each arriving data stream element belongs specifically includes the following steps: Monitor each data stream element arriving within the current monitoring period, determine the data stream identifier of the data stream to which each data stream element belongs, and calculate an index position using a pre-configured hash mapping function, wherein the hash mapping function is... ; in, It is the data stream identifier, It is a uniformly distributed hash function. It is a random seed. The lengths of the one-dimensional bucket array of the large flow index table and the lengths of the bucket array structure in the important element array are given.

[0012] By adopting the above technical solution, each arriving data stream element is monitored, its data stream identifier is determined, and the index position is calculated by combining the hash mapping function. This allows for the accurate location of the corresponding storage bucket in the large flow index table and important element array, providing a foundation for subsequent updates and filtering of the storage structure and facilitating efficient data processing in high-speed data stream scenarios.

[0013] Preferably, the step of detecting the idle status of the index position in the mass flow index table, and coordinating the update of the mass flow index table and the important element array in conjunction with the data flow identifier and preset data update rules, specifically includes the following steps: The idle status of the first storage bucket corresponding to the index position in the large flow index table is detected. The idle status includes empty and not empty. The currently arriving data flow element is taken as the first data flow element, corresponding to the first data flow identifier and the first data flow element identifier. If the idle state is empty, the first data stream information corresponding to the first data stream identifier is stored in the currently idle first storage bucket, and the corresponding data stream counter is set to 1; in the second storage bucket corresponding to the index position in the important element array, the first element information of the first data stream element is stored in the first storage unit of the second storage bucket, and the corresponding element frequency counter is set to 1; If the idle state is not empty, the data stream identifier stored in the first storage bucket corresponding to the index position is used as the second data stream identifier. The second data stream identifier and the first data stream identifier are compared and a comparison result is obtained. The comparison result includes the same and different. The large flow index table and the important element array are updated collaboratively based on the comparison result.

[0014] By adopting the above technical solution, within the current monitoring period, the incoming data stream elements can be updated collaboratively by combining the idle status of the storage bucket at the index position in the large flow index table, the data stream identifier, and the preset data update rules. This ensures the real-time performance and accuracy of the data, providing a foundation for subsequent screening of large flow data streams and important elements.

[0015] Preferably, the step of collaboratively updating the main flow index table and the important element array based on the comparison result specifically includes the following steps: If the comparison results are the same, increment the data stream counter corresponding to the first data stream identifier by one, and traverse all the storage units of the second storage bucket corresponding to the index position in the important element array. Determine whether there exists a data stream element identifier in all of the storage units that is the same as the first data stream element identifier; If it exists, the corresponding storage unit is used as the first storage unit, and the element frequency counter in the first storage unit is incremented by one. If the first storage unit does not exist, but there is an empty second storage unit, then the data stream element identifier in the second storage unit is replaced and updated according to the first element information, and the element frequency counter in the second storage unit is assigned a value of 1; If the first storage unit does not exist and there is no empty second storage unit, then obtain the third storage unit with the smallest element frequency counter among all the storage units in the current second storage bucket, and perform an exponential decay operation on the third storage unit according to the preset decay formula to obtain a first decay probability. Then, decrement the value of the element frequency counter in the third storage unit according to the first decay probability. If the element frequency counter in the third storage unit is 0 after decay, the data stream element identifier in the third storage unit is replaced and updated according to the first element information, and the frequency counter in the third storage unit is assigned a value of 1; otherwise, the first data stream element is discarded.

[0016] By adopting the above technical solution, when the comparison results are the same, incrementing the data stream counter by one can accurately count the data stream size; traversing the storage units and judging whether the element identifiers are the same can accurately update the element frequency; when there are identical element identifiers, incrementing the element frequency counter by one can reflect the increase in the number of times the element appears; if there is an empty storage unit, updating the element information and assigning the frequency counter to 1 can record new elements in a timely manner; if there is no empty storage unit, performing an exponential decay operation on the storage unit with the smallest frequency can filter out important elements; when the frequency is 0 after decay, updating the storage unit information can ensure that the storage unit stores important elements, avoid storing worthless elements, and discard elements that do not meet the conditions can reduce unnecessary data processing. Overall, it can achieve accurate and coordinated updates of the large-stream index table and the array of important elements.

[0017] Preferably, the step of collaboratively updating the main flow index table and the important element array based on the comparison result further includes the following steps: If the comparison results are different, then an exponential decay operation is performed on the second data stream stored in the first storage bucket corresponding to the index position according to the preset decay formula, and a second decay probability is obtained. Then, the value of the data stream counter in the current first storage bucket is decremented by one according to the second decay probability. If the value of the current data stream counter becomes 0 after decay, the second data stream information in the large flow index table is replaced and updated according to the first data stream information. After clearing all the storage units of the second storage bucket at the same index position in the important element array, the first element information of the first data stream element is inserted. If the value of the current data stream counter is greater than 0 after decay, then the first data stream element is discarded.

[0018] By adopting the above technical solution, when the comparison results are different, an exponential decay operation is performed on the data stream stored in the first storage bucket corresponding to the index position. This dynamically adjusts the value of the data stream counter, reflecting changes in the data stream in a timely manner. When the value of the data stream counter becomes 0 after decay, the old data stream information is replaced with the current data stream information, and the storage unit corresponding to the important element array is cleared before inserting the current data stream element information. This ensures the timeliness and accuracy of the data in the storage structure, storing only the currently valuable data streams and element information. When the value of the data stream counter is greater than 0 after decay, the current data stream element is discarded, avoiding the storage of invalid data, reducing storage overhead, and improving storage efficiency. Overall, this approach can efficiently identify large flows and their important elements while reducing computational and storage overhead, providing e-commerce platforms with more real-time, accurate, and fine-grained data support.

[0019] Preferably, the step of selecting the large-scale data stream from the large-scale data stream index table after the current monitoring period ends, and selecting the important elements in each large-scale data stream from the important element array, specifically includes the following steps: After the current monitoring period ends, each of the first storage buckets in the large flow index table is traversed, and the data flow identifier and the corresponding data flow counter value stored in each non-empty first storage bucket are read. The value of the data flow counter is compared with the preset large flow threshold, and all data flow identifiers corresponding to the data flow counter values ​​that are not less than the large flow threshold are selected. The data flow corresponding to the selected data flow identifier is taken as the large flow data flow. Traverse all the storage units in the second storage bucket corresponding to each of the large data streams in the important element array. If the element frequency counter corresponding to the data stream element in each storage unit is not less than the preset important element threshold, then the corresponding data stream element is regarded as an important element. The data stream information corresponding to the large data stream and the element information corresponding to the important elements are summarized and downloaded to general memory for storage.

[0020] By adopting the above technical solution, after the monitoring period ends, large-scale data streams can be filtered from the large-stream index table, and important elements in each large-stream data stream can be filtered from the important element array. The information from both is then aggregated and downloaded to general-purpose memory storage, providing e-commerce platforms with more granular and valuable reference information, assisting in optimizing recommendation systems, formulating precise marketing strategies, and improving user experience.

[0021] Secondly, this application provides a high-speed data stream detection device for e-commerce platforms, employing the following technical solution: A high-speed data stream detection device for e-commerce platforms includes the following modules: The storage structure initialization module is used to initialize two storage structures in parallel at the start of the current monitoring period. The two storage structures include a high-current index table and an array of important elements. The data element mapping module is used to calculate the index position based on the data stream identifier of the data stream to which each arriving data stream element belongs. The index position points to both the large stream index table and the corresponding storage bucket in the important element array. The storage structure update module is used to detect the idle status of the index position in the large flow index table, and to perform collaborative updates on the large flow index table and the important element array in combination with the data flow identifier and the preset data update rules; The large flow and important element detection module is used to filter out the large flow data stream from the large flow index table and filter out the important elements in each of the large flow data streams from the important element array after the current monitoring period ends.

[0022] By adopting the above technical solution, the storage structure initialization module can initialize the large flow index table and the important element array in parallel at the beginning of the monitoring period, preparing for subsequent data processing; the data element mapping module can calculate the index position based on the data flow identifier, so that the large flow index table and the important element array are associated with corresponding storage buckets; the storage structure update module combines the data flow identifier and rules to update the two storage structures collaboratively, ensuring the real-time performance and accuracy of the data; the large flow and important element detection module can filter out the large flow data stream and its important elements from the storage structure after the monitoring period ends, providing more valuable reference information for the e-commerce platform, and the whole device can run in high-speed storage, reducing computing and storage overhead and efficiently completing the high-speed data flow detection task.

[0023] Thirdly, this application provides a smart terminal, which adopts the following technical solution: A smart terminal includes a memory and a processor. The memory stores at least one instruction, at least one program, code set, or instruction set. The at least one instruction, at least one program, code set, or instruction set is loaded and executed by the processor to implement the high-speed data stream detection method for e-commerce platforms as described above.

[0024] Fourthly, this application provides a computer-readable storage medium, which adopts the following technical solution: A computer-readable storage medium storing at least one instruction, at least one program, code set, or instruction set, wherein the at least one instruction, at least one program, code set, or instruction set is loaded and executed by a processor to implement the high-speed data stream detection method for e-commerce platforms as described above.

[0025] In summary, this application includes at least the following beneficial effects: (1) This application can run in the high-speed storage of the core server of the e-commerce platform. By initializing the large flow index table and the array of important elements in parallel, and calculating the index position by combining the hash mapping function, the storage structure is updated in a coordinated manner. This can significantly reduce the computation and storage overhead while efficiently identifying the large flow and its important elements. (2) This application calculates the index position based on the data stream identifier and updates the storage structure in coordination. When the high-speed data stream arrives, it can quickly find the corresponding storage position and update the data, avoiding the problem of missing key information caused by sampling. It can comprehensively and accurately reflect the real situation and provide more real-time and accurate data support for content recommendation optimization and precision marketing. (3) This application can not only detect large flows, but also filter out important elements in each large flow data stream, providing e-commerce platforms with more granular and valuable decision-making basis, effectively improving the e-commerce platform's ability to process high-speed data streams and decision-making efficiency. Attached Figure Description

[0026] Figure 1 This is one of the overall flowcharts for high-speed data stream detection methods for e-commerce platforms; Figure 2 This is the second part of the overall flowchart for high-speed data stream detection methods for e-commerce platforms; Figure 3 These are schematic diagrams of two storage structures used in a high-speed data stream detection method for e-commerce platforms. Figure 4 This is a flowchart illustrating the processing steps for a high-speed data stream detection method for e-commerce platforms when the index position in the high-flow index table is not empty and the data stream identifier matches. Figure 5 This is a flowchart of the storage unit exponential decay operation in a high-speed data stream detection method for e-commerce platforms. Figure 6 This is a flowchart of the first bucket exponential decay operation in a high-speed data stream detection method for e-commerce platforms. Figure 7 This is a flowchart illustrating the processing steps for high-speed data stream detection methods for e-commerce platforms when the index position in the high-flow index table is not empty and the data stream identifier does not match. Figure 8 This is a structural diagram of a high-speed data stream detection device for e-commerce platforms. Detailed Implementation

[0027] This application provides a high-speed data stream detection method, apparatus, terminal, and storage medium for e-commerce platforms. To make the objectives, technical solutions, and advantages of this application clearer, the implementation methods of this application will be further described in detail below.

[0028] The following describes in further detail an embodiment of a high-speed data stream detection method for e-commerce platforms, in conjunction with the accompanying drawings.

[0029] This application discloses a high-speed data stream detection method for e-commerce platforms. The method is deployed and run in the high-speed storage of the core server of the e-commerce platform, such as the CPU cache. The high-speed storage includes two storage structures. In this embodiment, the two storage structures include a large flow index table and an important element array.

[0030] At the initial stage of a single monitoring cycle, when a high-speed data stream arrives at the core server, the system first searches the large flow index table to determine if the arriving data stream is a stored candidate large flow. Then, based on the search results in the large flow index table, the corresponding processing logic is executed in the important element array. In the important element array storage structure, an exponential decay mechanism is used to filter the elements of the candidate large flows stored in the large flow index table in real time, ensuring that only those important elements of a certain scale and valuable for subsequent analysis are saved, thereby avoiding the occupation of high-speed storage resources for non-important elements.

[0031] At the end of a single monitoring period, the flow information stored in the flow index table and the important element information of each flow stored in the important element array are synchronously downloaded to a secondary general-purpose storage device for subsequent archiving, processing and analysis, so as to support subsequent more complex data mining and decision analysis.

[0032] In practical applications, data streams and elements can be defined independently according to requirements. For ease of understanding, in this embodiment, a certain category of products on an e-commerce platform can be abstracted as a data stream, and specific products under that category can be further defined as elements of that data stream. The total number of times a user accesses this product category can be used as the size of the data stream, while the number of times each specific product under that category is accessed corresponds to the size of each element.

[0033] like Figure 1 and Figure 2 The above is a general flowchart of the method in this embodiment. The method includes the following steps: S1. At the start of the current monitoring period, initialize the two storage structures, the high-current index table and the important element array, in parallel. This includes the following steps: S11. At the start of the current monitoring period, deploy a high-flow index table in high-speed storage. and array of important elements Two storage structures. The storage structures are as follows: Figure 3 As shown.

[0034] The large-scale data stream index table is mainly used to store large-scale data streams that appear in high-speed data streams in real time; the important element array is responsible for filtering out the important elements of the large streams stored in the large-scale data stream index table.

[0035] S12. The large-scale data flow index table includes a one-dimensional bucket array for recording each arriving data flow in real time. In this embodiment, the one-dimensional bucket array is composed of... It consists of one first storage bucket; Each first bucket in the one-dimensional bucket array stores the data stream information of a data stream, including the data stream identifier. and data stream counter In this context, the data stream identifier serves as a unique identifier for each arriving data stream, and the data stream counter is used to count the total size of the data stream.

[0036] S13. The important element array includes a bucket array structure corresponding to the one-dimensional bucket array of the main flow index table; that is, the important element array also maintains a bucket array structure. A bucket array structure consisting of two second storage buckets.

[0037] In this embodiment, each second storage bucket in the bucket array structure contains Each storage unit stores the element information of one data stream element, including the element identifier. and element frequency counter Among them, the element identifier serves as a unique identifier for each element in the data stream, and the element frequency counter is used to count the number of times that element appears in the data stream.

[0038] S14. At the initial stage at the start of the current monitoring cycle, initialize each first bucket in the flow index table and each storage unit in the important element array to empty.

[0039] S2. Calculate the index position based on the data stream identifier of the data stream to which each arriving data stream element belongs. The index position points to the corresponding bucket in both the main stream index table and the important element array. This includes the following steps: S21. Monitor each data stream element arriving within the current monitoring period. In this embodiment, the data stream element is used as the monitoring unit. For example, An identifier used to indicate the data stream to which this data stream element belongs. , Identifier used to represent this data stream element .

[0040] S22. Determine the data stream identifier of each data stream element, and calculate an index position using a pre-configured hash mapping function. The hash mapping function is: ; in, It is a data stream identifier. It is a uniformly distributed hash function. It is a system-specified random seed. Let be the length of the one-dimensional bucket array of the large-flow index table and the length of the bucket array structure in the important element array, where ⊕ is the XOR operation.

[0041] S3. Detect the idle status of index positions in the mass flow index table, and perform collaborative updates on the mass flow index table and the important element array based on the data flow identifier and preset data update rules. This includes the following steps: S31. Take the currently arriving data stream element as the first data stream element, corresponding to the first data stream identifier and the first data stream element identifier.

[0042] S32. Determine the index position in the mass flow index table. The free state of the first storage bucket, which includes empty and non-empty states.

[0043] S33. If the corresponding first storage bucket is empty, then record the information of the arriving data stream in the bucket, and then insert the information of the data stream element into the first storage unit of the second storage bucket at the same index position of the important element array.

[0044] Specifically, in this embodiment, if the idle state is empty; S331. Store the first data stream information corresponding to the first data stream identifier into the currently idle first storage bucket. In the middle, the corresponding data stream counter is assigned a value of 1; Specifically, of Set as ,Right now ; Set to 1, that is .

[0045] S332, in the array of important elements, with the same index position In the corresponding second storage bucket, the first element information of the first data stream element is stored in the first storage unit of the second storage bucket, and the corresponding element frequency counter is assigned a value of 1.

[0046] In this embodiment, since before An empty value indicates that no other data streams exist, which suggests an important element array. of All storage units were also empty, therefore, directly in The first storage unit stores the arriving element information (i.e., the specified first element information).

[0047] Specifically, it is to The data stream element identifier is set to The element frequency counter is set to 1, that is, , .

[0048] S34. Otherwise, if the corresponding first storage bucket is not empty, check whether the data stream identifier stored in the bucket matches the currently arriving data stream identifier.

[0049] Specifically, in this embodiment, if the idle state is not empty, the first storage bucket corresponding to the index position is... The data stream identifier stored in the middle is used as the second data stream identifier, and the second data stream identifier and the first data stream identifier are combined. The comparison is performed and the comparison results are obtained to determine whether the two identifiers match.

[0050] S35. Based on the comparison results, the main flow index table and the important element array are updated collaboratively. The comparison results include both identical and different results.

[0051] S36. If the comparison results are the same, then the two data stream identifiers match, indicating that the second data stream already existing in the first bucket is the same as the currently arriving first data stream. At this point, the bucket... The data in the bucket is updated, and the information of the data stream element is updated in the second bucket at the same index position in the important element array. For example... Figure 4 As shown, specifically: S361. Increment the data stream counter corresponding to the first data stream identifier by one, that is, increment the first storage bucket. Data stream count value in Add one.

[0052] S362. Initialize an index to record the location of an empty storage unit. Empty. Used for subsequent traversal. of When finding a storage unit, record the empty storage unit found at the same time.

[0053] like If the value is empty, it means that no empty storage unit was found.

[0054] S363. Initialize the minimum value that has been found so far. In practical applications, it can be initialized to a value that is too small for the data stream to reach, and the index of the smallest storage unit can be initialized. Empty. Used for subsequent traversal. of When there are multiple storage units, the index of the smallest storage unit is also recorded.

[0055] S364. Initialize the traversal index Traverse the second bucket corresponding to the index position in the array of important elements. All One storage unit.

[0056] S365, Determine the storage unit Is it empty? If it is empty, record the index of the currently found empty storage unit, i.e. .

[0057] If not empty, determine if there exists a data stream element identifier in all storage units that is the same as the first data stream element identifier. That is, determine the storage unit. Is the data stream element identifier consistent with... match.

[0058] S365. If they exist, that is, the element identifiers of the two data streams match, update the element information in the storage unit.

[0059] Specifically, the corresponding storage unit is designated as the first storage unit, and the first storage unit is... Element frequency counter Perform an increment operation, which means incrementing the frequency of the element by 1.

[0060] S366. If the first storage unit does not exist, determine the storage unit. element frequency in Is it less than the minimum value already found? .

[0061] If it is less than, then the storage unit will be... The element frequency counter is set to the minimum value found so far. And modify the index where the smallest storage unit is located. ; If it is not less than, increment the index by one, i.e. Determine if the traversal index satisfies If so, return to step S365 and repeat the loop; If not satisfied Then determine the index of the location of the empty storage unit. Is it empty?

[0062] If it is empty, it means that no empty storage unit was found. For the smallest storage unit... Implement a decay mechanism; If the value is not empty, it means an empty storage unit has been found. If an empty storage unit is found, it will be used as the second storage unit. Perform the following steps:

[0063] A. If there is no first storage unit and there is an empty second storage unit, then the data stream element identifier in the second storage unit is replaced and updated according to the first element information, and the element frequency counter in the second storage unit is assigned a value of 1.

[0064] Specifically, the first element information of the current data stream is stored in the found empty second storage unit, and the... The data stream element identifier is set to Element frequency Set to 1.

[0065] B. If there is no first storage unit and no empty second storage unit, then obtain the third storage unit with the smallest element frequency counter among all storage units in the current second storage bucket. And according to the preset decay formula, an exponential decay operation is performed on the third memory cell to obtain the first decay probability, such as Figure 5 As shown.

[0066] Specifically, the attenuation formula in this embodiment is: ; in, For the third storage unit The value of the element frequency counter. For a decay parameter with a value greater than 1, This is the calculated first decay probability.

[0067] The value of the element frequency counter in the third storage unit is decremented by one based on the first decay probability, specifically as follows: Generate a random number , Compare random numbers With decay probability The size of the random number Less than or equal to decay probability , the third storage unit Decrement the value of the element frequency counter by one; otherwise, end the exponential decay operation.

[0068] The third storage unit after the decay operation The value of the element frequency counter is compared with 0; If the element frequency counter in the third storage unit is 0 after decay, then the data stream element identifier in the third storage unit is replaced and updated according to the first element information; that is, the element identifier in the third storage unit is replaced. If so, set the frequency counter in the third storage unit to 1; otherwise, discard the current first data stream element. .

[0069] S37. If the comparison results are different, the two data stream identifiers do not match, indicating that the second data stream that already exists in the first storage bucket is different from the first data stream that has just arrived.

[0070] S371, such as Figure 6 As shown, an exponential decay operation is performed on the second data stream stored in the first storage bucket corresponding to the index position according to the preset decay formula, and the second decay probability is obtained.

[0071] In this embodiment, the second attenuation probability is calculated. ; in, For the first storage bucket The value of the data stream counter stored in it, For a decay parameter with a value greater than 1, This is the calculated second decay probability.

[0072] S372. Decrement the value of the data stream counter in the first storage bucket by one according to the second decay probability.

[0073] Specifically, generate a random number. , Compare random numbers With the second decay probability The size of the random number Less than or equal to the second decay probability , the first storage bucket Decrement the value of the data stream counter by one; otherwise, end the operation.

[0074] S373, such as Figure 7 As shown, the first bucket after the decay operation will be executed. The value of the data stream counter is compared with 0.

[0075] S374, If the first storage bucket is attenuated If the value of the data stream counter is greater than 0, then the first data stream element that arrives is discarded. .

[0076] S375. Otherwise, if the value of the current data stream counter becomes 0 after decay, the second data stream information in the large stream index table is replaced and updated according to the first data stream information. After clearing all storage units of the second storage bucket at the same index position in the important element array, the first element information of the first data stream element is inserted, specifically: First storage bucket Replace the information in the first data stream and put the second storage bucket The original second data stream element stored in the first data stream is deleted, and then the information of the first data stream element is inserted.

[0077] Specifically: The first storage bucket Data stream identifier in Replace with The data stream counter is set to 1; Second storage bucket middle Clear each storage unit; The first storage unit Replace with the arriving data stream element, i.e., the first storage unit. The element identifier in the middle is set to The element frequency is set to 1.

[0078] Initialize traversal index Start clearing the second to the third One storage unit.

[0079] Clear storage unit The element information in.

[0080] Perform the traversal and increment the index by one, that is... Determine if the index satisfies If the value is less than the given value, return and clear the element information of the current storage unit. Otherwise, end the current process.

[0081] S4. Determine if the current monitoring period has ended. If the current monitoring period has ended, iterate through each first bucket in the large flow index table and filter out the large flow data stream from the large flow index table. This includes the following steps: S41. Initialize the bucket array traversal index .

[0082] S42. Determine the first bucket in the mass flow index table. Is it empty? Specifically, at index position. Starting from 0 until For each index position ,examine Is it empty?

[0083] S43, if If the position is empty, skip that position and execute S48 to check the next index position.

[0084] S44. If not empty, read the first storage bucket. Data stream identifier stored in and the value of its data stream counter .

[0085] S45. The size of the read data stream With preset high flow threshold Compare; S46, if Then report the data stream identifier. The corresponding data stream is a large stream, and its size Read the data stream from the second bucket of the important element array. The element information stored in it.

[0086] S47, Traversing the Index Add one, .

[0087] like Execute S47 directly.

[0088] S48. Determine the traversal index Is it less than the length of the bucket array? If yes, return to step S42; otherwise, end the retrieval task of the large data stream and download the large data stream identifier and size count to general memory for storage.

[0089] The large-scale data flow index table in this embodiment is mainly used for real-time storage of large-scale data flows appearing in high-speed data streams. It is responsible for real-time updating and storing data flow information that appears within the monitoring period, and saving the information of potential large flows through an attenuation mechanism. The important element array is responsible for filtering out the important elements of the large flows stored in the large-scale data flow index table, maintaining the element information of potential large flows stored in the large-scale data flow index table, and filtering out the important elements of the data flow from numerous elements through an attenuation mechanism.

[0090] S5. Traverse all storage units in the second bucket corresponding to each large data stream in the important element array, and filter out the important elements in each large data stream from the important element array. This includes the following steps: S51. Initialize the storage unit traversal index .

[0091] S52, Determine the buckets in the array of important elements medium storage unit Is it empty?

[0092] S53, if If the position is empty, skip that position and check the next index position.

[0093] S54. If not empty, read the memory unit. The element identifier and its frequency stored in .

[0094] S55. Count the frequency of the read elements. Compared with the preset important element threshold Comparison, S56. If the frequency counter of the element corresponding to the data stream element in each storage unit is not less than the preset threshold for important elements, then the corresponding data stream element is regarded as an important element.

[0095] Specifically, if Report the element identifier Important elements and their frequency .

[0096] S57, Traversing the Index Add one, .

[0097] S58. Determine the traversal index Is it less than the number of storage units? If yes, return to S52; otherwise, end the task of reading important elements and download the element identifier and frequency of the corresponding element to general memory for storage.

[0098] In this embodiment, the high flow threshold and important element threshold It can be set to a fixed value or a proportional value, depending on actual needs: For example, , ;in, This refers to the total size of the data stream that passes through during the entire monitoring period. The size of the data stream to which the element belongs. and These are two proportional parameters, both greater than 0 and less than 1.

[0099] The method in this embodiment can quickly prepare space for data recording by initializing the storage structure in parallel. By calculating the index position based on the data stream identifier and coordinating the update of the storage structure, the corresponding storage location can be quickly located and the data updated when high-speed data streams arrive, avoiding the shortcomings of traditional sampling and offline analysis methods, such as missing key information and poor real-time performance. Finally, the method filters out the large-volume data streams and important elements, providing e-commerce platforms with more granular and valuable decision-making support, effectively improving the e-commerce platform's processing capabilities and decision-making efficiency for high-speed data streams, representing a significant improvement and contribution compared to existing technologies.

[0100] Based on the same inventive concept described above, this application also discloses a high-speed data stream detection device for e-commerce platforms, the structure of which is as follows: Figure 8 As shown, the device includes the following modules: The storage structure initialization module is used to initialize two storage structures in parallel at the start of the current monitoring period. The two storage structures include a high-current index table and an array of important elements. The data element mapping module is used to calculate the index position based on the data stream identifier of the data stream to which each arriving data stream element belongs. The index position points to the corresponding storage bucket in both the main stream index table and the important element array. The storage structure update module is used to detect the idle status of the index position in the mass flow index table, and to perform collaborative updates on the mass flow index table and the important element array in combination with the data flow identifier and the preset data update rules. The large flow and important element detection module is used to filter out the large flow data stream from the large flow index table and filter out the important elements in each large flow data stream from the important element array after the current monitoring period ends.

[0101] In one specific implementation scheme, the storage structure initialization module includes the following units: The first storage structure initialization unit is used to deploy two storage structures, a high-flow index table and an important element array, in high-speed storage at the beginning of the current monitoring period. The second storage structure initialization unit is used to include a one-dimensional bucket array in the large flow index table, which is used to record each arriving data flow in real time. Each first storage bucket in the one-dimensional bucket array stores the data flow information of a data flow, including the data flow identifier and the data flow counter. The third storage structure initialization unit is used to include a bucket array structure in the important element array that corresponds to a one-dimensional bucket array of the large flow index table. Each second storage bucket in the bucket array structure contains multiple storage units. Each storage unit stores the element information of a data flow element. The element information includes the element identifier and the element frequency counter. The fourth storage structure initialization unit is used to initialize each first bucket in the mass flow index table and each storage unit in the important element array to be empty.

[0102] In one specific implementation scheme, the data element mapping module includes the following units: The data element mapping unit is used to monitor each data stream element arriving within the current monitoring period, determine the data stream identifier of the data stream to which each data stream element belongs, and calculate an index position using a pre-configured hash mapping function. The hash mapping function is as follows: ; in, It is a data stream identifier. It is a uniformly distributed hash function. It is a random seed. Let be the length of the one-dimensional bucket array of the large-flow index table and the length of the bucket array structure in the important element array.

[0103] In one specific implementation scheme, the storage structure update module includes the following units: The first storage structure update unit is used to detect the free status of the first storage bucket corresponding to the index position in the large flow index table. The free status includes empty and non-empty. The currently arriving data stream element is used as the first data stream element, corresponding to the first data stream identifier and the first data stream element identifier. The second storage structure update unit is used to store the first data stream information corresponding to the first data stream identifier into the currently idle first storage bucket if the idle state is empty, and to set the corresponding data stream counter to 1; in the second storage bucket corresponding to the index position in the important element array, the first element information of the first data stream element is stored into the first storage unit of the second storage bucket, and the corresponding element frequency counter is set to 1. The third storage structure update unit is used to, if the idle state is not empty, take the data stream identifier stored in the first storage bucket corresponding to the index position as the second data stream identifier, compare the second data stream identifier and the first data stream identifier and obtain the comparison result, the comparison result includes the same and different, and perform collaborative update of the large flow index table and the important element array based on the comparison result.

[0104] In one specific implementation scheme, the third storage structure update unit includes the following sub-units: The first storage structure update sub-unit is used to perform an increment operation on the data stream counter corresponding to the first data stream identifier if the comparison results are the same, and to traverse all storage units of the second storage bucket corresponding to the index position in the important element array. The second storage structure update sub-unit is used to determine whether there is a data stream element identifier that is the same as the first data stream element identifier in all storage units; If it exists, the corresponding storage unit is used as the first storage unit, and the element frequency counter in the first storage unit is incremented by one. If the first storage unit does not exist, but there is an empty second storage unit, then the data stream element identifier in the second storage unit is replaced and updated according to the first element information, and the element frequency counter in the second storage unit is assigned a value of 1; If there is no first storage unit and no empty second storage unit, then obtain the third storage unit with the smallest element frequency counter among all storage units in the current second storage bucket, and perform an exponential decay operation on the third storage unit according to the preset decay formula to obtain the first decay probability. Then, decrement the value of the element frequency counter in the third storage unit by one according to the first decay probability. If the element frequency counter in the third storage unit is 0 after decay, the data stream element identifier in the third storage unit is replaced and updated according to the first element information, and the frequency counter in the third storage unit is assigned a value of 1; otherwise, the first data stream element is discarded.

[0105] The third storage structure update subunit is used to perform an exponential decay operation on the second data stream stored in the first storage bucket corresponding to the index position according to the preset decay formula if the comparison results are different, and obtain the second decay probability. Based on the second decay probability, the value of the data stream counter in the current first storage bucket is decremented by one. The fourth storage structure update sub-unit is used to replace and update the second data stream information in the large flow index table according to the first data stream information if the value of the current data stream counter becomes 0 after decay. After clearing all storage units of the second storage bucket at the same index position in the important element array, the first element information of the first data stream element is inserted. If the value of the current data stream counter is greater than 0 after decay, the first data stream element is discarded.

[0106] In a specific feasible implementation, the high-flow and important element detection module includes the following units: The first large flow and important element detection unit is used to traverse each first storage bucket in the large flow index table after the current monitoring period ends, read the data flow identifier and the corresponding data flow counter value stored in each non-empty first storage bucket, compare the data flow counter value with the preset large flow threshold, filter out all data flow identifiers corresponding to the data flow counter values ​​that are not less than the large flow threshold, and take the data flow corresponding to the filtered data flow identifiers as the large flow data flow. The second major flow and important element detection unit is used to traverse all storage units in the second storage bucket corresponding to each major flow data flow in the important element array. If the element frequency counter corresponding to the data flow element in each storage unit is not less than the preset important element threshold, then the corresponding data flow element is regarded as an important element. The third major flow and important element detection unit is used to summarize the data flow information corresponding to the major flow data flow and the element information corresponding to the important elements, and download them to general memory for storage.

[0107] Based on the same inventive concept described above, this application also discloses a smart terminal, including a memory and a processor. The memory stores at least one instruction, at least one program, code set, or instruction set. The at least one instruction, at least one program, code set, or instruction set is loaded and executed by the processor to implement the high-speed data stream detection method for e-commerce platforms as described above.

[0108] Based on the same inventive concept described above, this application also discloses a computer-readable storage medium storing at least one instruction, at least one program, code set, or instruction set. The at least one instruction, at least one program, code set, or instruction set is loaded and executed by a processor to implement the high-speed data stream detection method for e-commerce platforms as described above.

[0109] Those skilled in the art will understand that all or part of the steps of the above embodiments can be implemented by hardware or by a program instructing related hardware. The program can be stored in the computer-readable storage medium, which includes, for example, various media capable of storing program code such as: USB flash drive, portable hard drive, read-only memory (ROM), random access memory (RAM), magnetic disk or optical disk.

[0110] Those skilled in the art will understand that the step numbers of the above methods or processes are only used to distinguish different steps and do not constitute an absolute restriction on the execution order. Some steps may be executed simultaneously or in a different order than the numbers.

[0111] The above description is merely an optional embodiment of this application and is not intended to limit this application. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the protection scope of this application.

Claims

1. A high-speed data stream detection method for e-commerce platforms, characterized in that, The method runs in the high-speed storage of the core server of the e-commerce platform, which includes two storage structures. The method includes the following steps: At the start of the current monitoring period, the two storage structures, including a high-current index table and an array of important elements, are initialized in parallel. The index position is calculated based on the data stream identifier of the data stream to which each arriving data stream element belongs. The index position points to both the large stream index table and the corresponding storage bucket in the important element array. The idle status of the index position in the large flow index table is detected, and the large flow index table and the important element array are updated collaboratively in combination with the data flow identifier and the preset data update rules; After the current monitoring period ends, the large-scale data stream is obtained by filtering from the large-scale data stream index table, and the important elements in each large-scale data stream are obtained by filtering from the important element array.

2. The high-speed data stream detection method for e-commerce platforms according to claim 1, characterized in that, At the start of the current monitoring period, the initialization of the two storage structures, which include a high-current index table and an array of important elements, specifically includes the following steps: At the start of the current monitoring cycle, two storage structures, a high-flow index table and an array of important elements, are deployed in the high-speed storage. The large flow index table includes a one-dimensional bucket array for recording each arriving data flow in real time. Each first storage bucket in the one-dimensional bucket array stores the data flow information of the data flow, which includes a data flow identifier and a data flow counter. The important element array includes a bucket array structure corresponding to the one-dimensional bucket array of the large flow index table. Each second storage bucket in the bucket array structure contains multiple storage units. Each storage unit stores element information of a data flow element. The element information includes an element identifier and an element frequency counter. Each of the first buckets in the large flow index table and each of the storage cells in the important element array are initialized to empty.

3. The high-speed data stream detection method for e-commerce platforms according to claim 2, characterized in that, The process of calculating the index position based on the data stream identifier of the data stream to which each arriving data stream element belongs includes the following steps: Monitor each data stream element arriving within the current monitoring period, determine the data stream identifier of the data stream to which each data stream element belongs, and calculate an index position using a pre-configured hash mapping function, wherein the hash mapping function is... ； in, It is the data stream identifier, It is a uniformly distributed hash function. It is a random seed. The lengths of the one-dimensional bucket array of the large flow index table and the lengths of the bucket array structure in the important element array are given.

4. The high-speed data stream detection method for e-commerce platforms according to claim 2, characterized in that, The step of detecting the idle status of the index position in the large flow index table, and then collaboratively updating the large flow index table and the important element array in conjunction with the data flow identifier and preset data update rules, specifically includes the following steps: The idle status of the first storage bucket corresponding to the index position in the large flow index table is detected. The idle status includes empty and not empty. The currently arriving data flow element is taken as the first data flow element, corresponding to the first data flow identifier and the first data flow element identifier. If the idle state is empty, the first data stream information corresponding to the first data stream identifier is stored in the currently idle first storage bucket, and the corresponding data stream counter is set to 1; in the second storage bucket corresponding to the index position in the important element array, the first element information of the first data stream element is stored in the first storage unit of the second storage bucket, and the corresponding element frequency counter is set to 1; If the idle state is not empty, the data stream identifier stored in the first storage bucket corresponding to the index position is used as the second data stream identifier. The second data stream identifier and the first data stream identifier are compared and a comparison result is obtained. The comparison result includes the same and different. The large flow index table and the important element array are updated collaboratively based on the comparison result.

5. The high-speed data stream detection method for e-commerce platforms according to claim 4, characterized in that, The step of collaboratively updating the high-current index table and the important element array based on the comparison results specifically includes the following steps: If the comparison results are the same, increment the data stream counter corresponding to the first data stream identifier by one, and traverse all the storage units of the second storage bucket corresponding to the index position in the important element array. Determine whether there exists a data stream element identifier in all of the storage units that is the same as the first data stream element identifier; If it exists, the corresponding storage unit is used as the first storage unit, and the element frequency counter in the first storage unit is incremented by one. If the first storage unit does not exist, but there is an empty second storage unit, then the data stream element identifier in the second storage unit is replaced and updated according to the first element information, and the element frequency counter in the second storage unit is assigned a value of 1; If the first storage unit does not exist and there is no empty second storage unit, then obtain the third storage unit with the smallest element frequency counter among all the storage units in the current second storage bucket, and perform an exponential decay operation on the third storage unit according to the preset decay formula to obtain a first decay probability. Then, decrement the value of the element frequency counter in the third storage unit according to the first decay probability. If the element frequency counter in the third storage unit is 0 after decay, the data stream element identifier in the third storage unit is replaced and updated according to the first element information, and the frequency counter in the third storage unit is assigned a value of 1; otherwise, the first data stream element is discarded.

6. The high-speed data stream detection method for e-commerce platforms according to claim 4, characterized in that, The step of collaboratively updating the large-flow index table and the important element array based on the comparison result further includes the following steps: If the comparison results are different, then an exponential decay operation is performed on the second data stream stored in the first storage bucket corresponding to the index position according to the preset decay formula, and a second decay probability is obtained. Then, the value of the data stream counter in the current first storage bucket is decremented by one according to the second decay probability. If the value of the current data stream counter becomes 0 after decay, the second data stream information in the large flow index table is replaced and updated according to the first data stream information. After clearing all the storage units of the second storage bucket at the same index position in the important element array, the first element information of the first data stream element is inserted. If the value of the current data stream counter is greater than 0 after decay, the first data stream element is discarded.

7. The high-speed data stream detection method for e-commerce platforms according to claim 2, characterized in that, After the current monitoring period ends, the process of filtering out large-scale data streams from the large-scale data stream index table and filtering out important elements in each large-scale data stream from the important element array specifically includes the following steps: After the current monitoring period ends, each of the first storage buckets in the large flow index table is traversed, and the data flow identifier and the corresponding data flow counter value stored in each non-empty first storage bucket are read. The value of the data flow counter is compared with the preset large flow threshold, and all data flow identifiers corresponding to the data flow counter values ​​that are not less than the large flow threshold are selected. The data flow corresponding to the selected data flow identifier is taken as the large flow data flow. Traverse all the storage units in the second storage bucket corresponding to each of the large data streams in the important element array. If the element frequency counter corresponding to the data stream element in each storage unit is not less than the preset important element threshold, then the corresponding data stream element is regarded as an important element. The data stream information corresponding to the large data stream and the element information corresponding to the important elements are summarized and downloaded to general memory for storage.

8. A high-speed data stream detection device for e-commerce platforms, characterized in that, Includes the following modules: The storage structure initialization module is used to initialize two storage structures in parallel at the start of the current monitoring period. The two storage structures include a high-current index table and an array of important elements. The data element mapping module is used to calculate the index position based on the data stream identifier of the data stream to which each arriving data stream element belongs. The index position points to both the large stream index table and the corresponding storage bucket in the important element array. The storage structure update module is used to detect the idle status of the index position in the large flow index table, and to perform collaborative updates on the large flow index table and the important element array in combination with the data flow identifier and the preset data update rules; The large flow and important element detection module is used to filter out the large flow data stream from the large flow index table and filter out the important elements in each of the large flow data streams from the important element array after the current monitoring period ends.

9. A smart terminal, characterized in that, The method includes a memory and a processor, wherein the memory stores at least one instruction, at least one program, code set, or instruction set, and the at least one instruction, at least one program, code set, or instruction set is loaded and executed by the processor to implement the high-speed data stream detection method for e-commerce platforms as described in any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that, The readable storage medium stores at least one instruction, at least one program, code set, or instruction set, wherein the at least one instruction, at least one program, code set, or instruction set is loaded and executed by a processor to implement the high-speed data stream detection method for e-commerce platforms as described in any one of claims 1 to 7.

Images