Read-write separated data feature storage optimization method and device and computer equipment

Abstract

The invention discloses a read-write separated data feature storage optimization method and device and computer equipment. The method comprises the following steps: acquiring event information input by an external system, analyzing the event information into a dictionary format, and extracting key information; retrieving a final calculation result of the data features from a storage system based on the key information, and determining the data features needing to be updated; grouping is carried out according to unique keys according to the data features needing to be updated, and corresponding time slices are created to generate updating request parameters; calculating a specific data feature of each unique key based on the update request parameter, and performing aggregation processing to obtain a final data feature; and storing the final data feature result into a KV database according to a unique key. By implementing the method provided by the invention, the efficiency of data feature calculation can be improved, synchronous and asynchronous updating modes are supported, and the real-time performance and flexibility of the system are enhanced, so that a more efficient solution is provided for complex data analysis.

Description

Technical Field

[0001] This invention relates to data processing methods, and more specifically to a read-write separation data feature storage optimization method, apparatus, and computer equipment. Background Technology

[0002] In modern data analytics, calculating metrics or data characteristics within a specific time period is crucial. These metrics represent behavioral statistics within a certain dimension, such as the total transaction amount of customer A over 30 days, or the number of accounts associated with the same device within a month. To effectively manage and analyze this information, a time-slice data structure is typically used to organize the data, dividing and storing it according to clock ranges such as minutes, hours, days, months, and years.

[0003] There are three main approaches in existing technologies: one is to obtain event information and target indicator identifiers from external systems and send read requests to the database for sharding and aggregation; another is to first parse the event message and extract metadata to store in memory, and then query memory and calculate credit indicators based on configuration information; the third focuses on the calculation and storage of time span values, determining time sharding information based on target indicator configuration information, calculating indicator results according to shard size, and finally aggregating them. However, all these approaches have significant drawbacks. The first two approaches lead to frequent storage access and high query overhead when calculating large amounts of data features, reducing computational efficiency; while the second approach, because it requires pre-acquiring all raw data before calculation, consumes huge amounts of storage space and network bandwidth when dealing with long time spans and large amounts of data features, and generates a large amount of redundant data transmission.

[0004] Therefore, it is necessary to design a new method to improve the efficiency of data feature calculation, while supporting synchronous and asynchronous update modes, enhancing the real-time performance and flexibility of the system, thereby providing a more efficient solution for complex data analysis. Summary of the Invention

[0005] The purpose of this invention is to overcome the shortcomings of the prior art and provide a method, apparatus and computer equipment for optimizing read-write separation data feature storage.

[0006] To achieve the above objectives, the present invention adopts the following technical solution: a read-write separation data feature storage optimization method, comprising: Acquire event information input from external systems, parse it into dictionary format, and extract key information; Based on the key information, the final calculation results of the data features are retrieved from the storage system, and the data features that need to be updated are determined. Based on the data characteristics that need to be updated, group them according to a unique key and create corresponding time slices to generate update request parameters; For each unique key, specific data features are calculated based on the update request parameters, and then aggregated to obtain the final data features. The final data feature results are stored in a KV database using a unique key.

[0007] The further technical solution is as follows: the key information includes the query primary key, the event occurrence time, and the calculated attributes.

[0008] Its further technical solution is as follows: the acquisition of event information input from an external system, parsing it into a dictionary format, and extracting key information include: Obtain event information from external systems and parse the event information into a dictionary format to obtain the extracted information; Based on the extracted information, the computational attributes required for subsequent data feature queries are determined in order to obtain key information.

[0009] The further technical solution is as follows: the final calculation result of retrieving data features from the storage system based on the key information, and determining the data features that need to be updated, includes: Parse the query primary key and determine the corresponding time slice based on the event occurrence time and feature configuration; Check whether the final calculation result of the time slice in storage is valid; If valid, the final calculation result is returned from the RESULT field stored in the KV database; If the result is invalid or does not exist, the final calculation result will be used as the data feature that needs to be updated.

[0010] The further technical solution is as follows: checking whether the final calculation result of the time slice in the storage is valid includes: Compare the time slice with the most recent update time. If the unit time that the time slice advances forward is not less than the most recent update time, the final calculation result is valid and can be read directly. If the unit time that the time slice advances forward is less than the most recent update time, the final calculation result is invalid or does not exist.

[0011] The further technical solution is as follows: The data to be updated is grouped according to a unique key based on its characteristics, and corresponding time slices are created to generate update request parameters, including: Group the data features according to their unique keys to obtain multiple sets of data features; For each set of data features, an update time slice is determined based on the time unit and event time; Specific update request parameters are generated based on the update time slice.

[0012] The further technical solution is as follows: For each unique key, specific data features are calculated based on the update request parameters, and aggregation processing is performed to obtain the final data features, including: A data statistics method is performed on each group of data features according to the update request parameters to obtain the calculation results; Each calculation result is associated with its corresponding time slice to record the time dimension of data changes, thereby obtaining intermediate results for the time slice. Aggregate the intermediate results of all the time slices to obtain the final data features; By optimizing storage methods to preserve intermediate results, and using timestamps to ensure data consistency and query efficiency.

[0013] The present invention also provides a read-write separation data feature storage optimization device, comprising: The parsing unit is used to acquire event information input from external systems, parse it into dictionary format, and extract key information; The query unit is used to retrieve the final calculation results of data features from the storage system based on the key information, and to determine the data features that need to be updated. The update unit is used to group the data that needs to be updated according to a unique key and create a corresponding time slice to generate update request parameters; The calculation unit is used to calculate specific data features for each unique key based on the update request parameters, and perform aggregation processing to obtain the final data features; A storage unit is used to store the final data feature results into a KV database using a unique key.

[0014] The present invention also provides a computer device, the computer device including a memory and a processor, the memory storing a computer program, and the processor executing the computer program to implement the above-described method.

[0015] The present invention also provides a storage medium storing a computer program that, when executed by a processor, implements the above-described method.

[0016] The advantages of this invention compared to existing technologies are as follows: This invention acquires event information input from an external system and parses it into a dictionary format to extract key information. Based on this key information, it efficiently retrieves the final calculation results of data features from the storage system and determines the data features that need to be updated. Subsequently, it intelligently groups the data features to be updated according to unique keys, creates corresponding time slices to generate update request parameters, calculates specific data features for each unique key based on these parameters, and performs aggregation processing to obtain the final data features. Finally, it stores these results in a KV database according to unique keys. This process utilizes read-write separation and time-slice storage mechanisms to optimize the calculation and storage efficiency of data features, reducing unnecessary redundant calculations and storage overhead. Simultaneously, it supports both synchronous and asynchronous update modes to meet the real-time and throughput requirements of different scenarios, thus providing a more efficient and flexible solution for complex data analysis and significantly improving the overall performance and response speed of the system.

[0017] The present invention will be further described below with reference to the accompanying drawings and specific embodiments. Attached Figure Description

[0018] To more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings used in the following description of the embodiments will be briefly introduced. Obviously, the drawings described below are some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0019] Figure 1 This is a flowchart illustrating the read-write separation data feature storage optimization method provided in an embodiment of the present invention. Figure 2 A schematic diagram of a sub-process of the read-write separation data feature storage optimization method provided in an embodiment of the present invention; Figure 3 A schematic diagram of a sub-process of the read-write separation data feature storage optimization method provided in an embodiment of the present invention; Figure 4 A schematic diagram of a sub-process of the read-write separation data feature storage optimization method provided in an embodiment of the present invention; Figure 5 A schematic diagram of a sub-process of the read-write separation data feature storage optimization method provided in an embodiment of the present invention; Figure 6 This is a schematic diagram illustrating the verification of the final calculation result of the time slice in storage, as provided in an embodiment of the present invention. Figure 7 A schematic block diagram of a read-write separation data feature storage optimization device provided in an embodiment of the present invention; Figure 8A schematic block diagram of a computer device provided for an embodiment of the present invention. Detailed Implementation

[0020] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0021] It should be understood that, when used in this specification and the appended claims, the terms "comprising" and "including" indicate the presence of the described features, integrals, steps, operations, elements and / or components, but do not exclude the presence or addition of one or more other features, integrals, steps, operations, elements, components and / or collections thereof.

[0022] It should also be understood that the terminology used in this specification is for the purpose of describing particular embodiments only and is not intended to limit the invention. As used in this specification and the appended claims, the singular forms “a,” “an,” and “the” are intended to include the plural forms unless the context clearly indicates otherwise.

[0023] It should also be further understood that the term "and / or" as used in this specification and the appended claims refers to any combination of one or more of the associated listed items and all possible combinations, and includes such combinations.

[0024] Please see Figure 1 , Figure 1 This is a schematic flowchart illustrating the read-write separation data feature storage optimization method provided in this embodiment of the invention. This method is applied to a server, acquiring external event information and parsing key content. Based on unique keys, it groups and divides the data features to be updated into time slices, performs statistical calculations, aggregates the data to obtain the final data features, and uses an optimized storage strategy to save intermediate results, ensuring data consistency and query efficiency, thus achieving the goal of improving the efficiency of data feature calculation. Simultaneously, it supports synchronous and asynchronous update modes, allowing the system to select the most suitable update method according to real-time requirements, enhancing the system's flexibility and response speed, and providing a more efficient and reliable solution for complex data analysis. This method effectively promotes the automation and intelligence of data processing, making rapid analysis of large-scale datasets possible.

[0025] Figure 1 This is a flowchart illustrating the read-write separation data feature storage optimization method provided in an embodiment of the present invention. Figure 1As shown, the method includes the following steps S110 to S150.

[0026] S110. Obtain event information input from external systems, parse it into dictionary format, and extract key information.

[0027] In this embodiment, the key information includes the query primary key, the event occurrence time, and the calculated attributes.

[0028] A primary key (PK) is a keyword used to uniquely identify a data record. For example, when processing account transaction data, the primary key might be the account number, which helps the system quickly locate all relevant data for a specific account. The existence of a primary key allows the system to perform data retrieval efficiently, avoiding the performance overhead of a full table scan.

[0029] The event occurrence time refers to the actual point in time when each event occurs. In data analysis, especially time series analysis, this timestamp is crucial. It not only determines the time slice to which the data belongs but also affects the validity of the calculation logic. For example, when calculating the total transaction amount of an account over the past 30 days, the event occurrence time is used to determine which transaction records belong to this 30-day time window.

[0030] Computed attributes refer to specific metrics or dimensions that need to be focused on based on business needs or analytical objectives. These attributes guide how data is processed and analyzed. Taking transaction records as an example, computed attributes might include transaction amount, transaction type, etc. Based on different computed attributes, the system can perform various types of calculations, such as summation, average, maximum and minimum values, etc.

[0031] In one embodiment, please refer to Figure 2 The above step S110 may include steps S111 to S112.

[0032] S111. Obtain event information from an external system and parse the event information into a dictionary format to obtain extracted information.

[0033] In this embodiment, extracting information refers to parsing key data elements from event information received from an external system and converting them into dictionary format for subsequent processing.

[0034] In this sub-step, the system first receives event information from external systems. This information may include, but is not limited to, account activity, transaction records, and user behavior. The received information may be in JSON, XML, or other structured data formats. The system needs to parse this information into a format that is easy to process internally, namely, dictionary format. This conversion makes it easier for the system to access and manipulate the various fields in this data. For example, a typical transaction record may contain information such as "account number," "transaction amount," and "transaction time." Once this information is parsed into dictionary format, the corresponding values ​​can be accessed intuitively by key name (such as 'account_id', 'transaction_amount', 'transaction_time').

[0035] S112. Based on the extracted information, determine the computational attributes required for subsequent data feature queries to obtain key information.

[0036] After obtaining the parsed event information, the next step is to determine the calculated attributes required for subsequent data feature queries based on this extracted information, thus forming key information. Here, "calculated attributes" refer to specific metrics or dimensions that need to be focused on based on business needs or data analysis objectives. For example, in the transaction record example above, "calculated attributes" might be the total transaction amount or average transaction amount within a specified time period. By analyzing specific fields in the extracted information (such as "transaction time" and "transaction amount"), the system can identify which data features need to be calculated and how to calculate them. The key to this step is correctly understanding the business logic and translating it into actionable data processing instructions. Ultimately, this process generates a set of "key information," including but not limited to the query primary key (used to uniquely identify an entity, such as an account number), the event occurrence time (very important for time series data), and calculated attributes (guiding how to process the data). This key information will be used in subsequent steps to retrieve and update relevant data features.

[0037] S120. Based on the key information, retrieve the final calculation results of the data features from the storage system and determine the data features that need to be updated.

[0038] In this embodiment, the data features that need to be updated refer to those that are outdated or no longer accurate, and therefore need to be recalculated and updated. Specifically, when the system executes step S120 to retrieve the final calculation results of the data features from the storage system based on key information, it determines which data features need to be updated based on the following aspects: Time slice validity: This is determined by comparing the current query's time slice with the most recently updated time slice. If the unit of time advancing the current time slice is less than the most recently updated time slice, it means that new data may have been generated or changed during this period, and therefore the existing data features may be invalid or inaccurate, requiring an update.

[0039] Data not found or missing: When checking the final calculation results in storage, if it is found that there is no corresponding calculation result for a specific time slice (i.e., the data does not exist), these data features also need to be marked as needing to be updated.

[0040] Changes in business logic: Sometimes, even if the data is technically "up-to-date," changes in business rules or requirements may render previously calculated results inapplicable or inconsistent with the latest business logic. In such cases, the relevant data characteristics should also be considered as needing to be updated.

[0041] Data integrity issues: If incomplete or erroneous data features are found during the inspection process, these data need to be corrected and updated to ensure that all returned data is accurate.

[0042] By using the above standards, the system can effectively identify which data features no longer meet current needs, thereby triggering the corresponding update process to ensure that the data processed and provided by the system is always up-to-date and accurate. This process not only improves the system's reliability and accuracy but also optimizes resource utilization efficiency and avoids unnecessary duplication of calculations.

[0043] In one embodiment, please refer to Figure 3 The above-mentioned step S120 may include steps S121 to S124.

[0044] S121. Parse the query primary key and determine the corresponding time slice based on the event occurrence time and feature configuration.

[0045] First, the system receives a query request containing a primary key. This primary key may include key information such as time information and specific event identifiers. The system parses this information to identify the relevant event occurrence time and feature configuration. For example, if the query is for user behavior analysis within a specific time period, the system will extract the start and end times of that period as the "event occurrence time" and determine the data dimensions to focus on based on predefined feature configurations (such as user access counts and dwell time). Then, based on this information, the system can calculate or directly obtain a specific time slice, that is, the smallest unit of time interval for the data.

[0046] S122. Check whether the final calculation result of the time slice in the storage is valid.

[0047] like Figure 6 As shown, the time slice is compared with the most recent update time. If the unit time that the time slice advances forward is not less than the most recent update time, the final calculation result is valid and can be read directly; if the unit time that the time slice advances forward is less than the most recent update time, the final calculation result is invalid or does not exist.

[0048] Once the time slice is determined, the next step is to check if a valid final calculation result exists in the storage system for that time slice. This step involves comparing the current query's time slice with the time of the most recent update. Figure 6 As shown, if the unit time advance of the time slice is not less than the most recent update time, it means that there are no new data changes during this period. Therefore, the calculation results in the existing storage can be considered valid and can be read and used directly. Conversely, if the unit time advance of the time slice is less than the most recent update time, it means that there may be new data that has not been processed. In this case, the existing calculation results are considered invalid or do not exist.

[0049] S123. If valid, return the final calculation result from the RESULT field stored in the KV database.

[0050] When step S122 confirms that the stored calculation result is valid, the system will directly read the corresponding final calculation result from the RESULT field of the key-value (KV) database. Here, the KV database is an efficient data storage method where each record consists of a unique key and a corresponding value, and the RESULT field contains the final calculation result for a specific query condition. In this way, the system can quickly respond to user query requests and provide the required information.

[0051] S124. If the result is invalid or does not exist, the final calculation result shall be used as the data feature that needs to be updated.

[0052] If the calculation result is found to be invalid or non-existent in step S122, the system will mark this data as a data feature that needs to be updated. This means that an update process will be triggered to recalculate these data features to ensure they reflect the latest state. The updated data will be stored in the storage system again, and its update time will be recorded so that the validity of the data can be accurately determined during future queries.

[0053] Through the steps S121 to S124 described above, the system can not only effectively manage the timeliness and accuracy of data, but also improve query efficiency, reduce unnecessary repetitive calculations, and thus optimize overall performance.

[0054] Specifically, after obtaining the primary key (PK) for the query, this module is responsible for retrieving the final calculation result of the data features from the storage system. During the query, the feature configuration is parsed, and the time slice to which the event occurrence time belongs is calculated based on the event occurrence time and feature configuration. The system will first check whether the final result in the storage is valid, and if the result is valid, it will directly return the result in the RESULT in the KV storage to reduce unnecessary computational overhead. If the final result of the data features is invalid, the system will proceed to the data feature update and data feature calculation module to ensure that the returned data is always the latest calculation result that conforms to the business logic.

[0055] Feature Query Request: The system receives a feature query request containing specific query conditions. Parsing Feature Information: The feature query request is parsed to extract key information. In this embodiment, the parsed information includes a unique key (PK): Account = Account 1; Time slice range: 1746028800~1748628800; Query Data: Based on the parsed unique key and time slice range, a query is performed in the stored data table. The data table contains the following: Unique Key (PK), M1 and M2: two calculated attributes storing values ​​at different time points, and RESULT: the final result column, used to store the query or calculation results. During the query process, the system extracts data matching the specified unique key and time slice range from columns M1 and M2 and performs corresponding calculations or summaries. Returning Feature Results: If the queried data is valid, the calculated results are returned to the requester. For example, for data of Account 1 within the specified time slice range, the calculated results are M1: 94, 16; M2: 13. Data validity assessment: If the retrieved data is invalid (e.g., no matching data found or the data does not meet expectations), a feature update request is triggered to further process the data and ensure its validity. Feature update request: When data is invalid, the system initiates a feature update request to update or supplement the relevant data to ensure the accuracy of subsequent queries. Through this process, the system can efficiently process feature query requests, ensuring that the returned data is accurate and valid.

[0056] When determining whether data features are valid, the following steps should be followed: Record key time parameters: First, record the required unit time slice length as timeUnit, and the current system time in seconds as T. NOW The most recent update time slice was T. OLD (Unit: seconds), the time of the newly occurring event is T. NOW(Unit: seconds). Simultaneously, the time slice offset (offsetTime) and unit time (Unit) are recorded for calculating T. NOW The unit time slice contained in the time slice to which it belongs. The feature data is valid and can be read directly: if T NOW The time slice is advanced by N units or more than or equal to T. OLD This indicates that there is no data within N time slices from the last feature update to the current event time. At this point, the first data point within the feature calculation range can be directly read, i.e., timeSlice(T) NOW ,0,timeUnit)>=T OLD The feature data is invalid and the true value needs to be recalculated: if T NOW The time slice is advanced by N units less than or equal to T. OLD This indicates that there is data within N time slices from the last feature update to the current event time, requiring recalculation, i.e., timeSlice(T) NOW (N, timeUnit)<=T OLD The process of reducing construction parameters: In the above operations, the interval timeSlice(T) for constructing parameters when features do not need to be recalculated has already been constructed. NOW ,0,timeUnit) and endTime, which are the last time slice data to which the feature belongs, are obtained through timeSlice(T NOW Setting (N, timeUnit) <= startTime reduces the parameter construction process. These steps effectively determine the validity of data features and decide whether recalculation is necessary based on the actual situation, thereby improving data processing efficiency and accuracy. In this way, the entire process not only ensures the timeliness and accuracy of the data but also optimizes system performance and reduces unnecessary waste of computational resources.

[0057] S130. Group the data that needs to be updated according to the unique key and create corresponding time slices to generate update request parameters.

[0058] In this embodiment, the update request parameters refer to a specific set of parameters generated based on the time unit and event time of each set of data features, used to query or calculate corresponding statistical results from the feature library.

[0059] In one embodiment, please refer to Figure 4 The above-mentioned step S130 may include steps S131 to S133.

[0060] S131. Group the data features according to the unique keys of the data features that need to be updated to obtain multiple groups of data features.

[0061] In this embodiment, multiple sets of data features refer to different groups of data features that need to be updated, based on the unique key of the data features.

[0062] In this step, the system first identifies all data features that need to be updated and groups them based on their unique keys. A unique key typically consists of multiple attributes that ensure the uniqueness of each data feature, such as account ID, feature type (e.g., average transaction amount, number of transfers), and a specific statistical time window (e.g., the last 30 days). In this way, the system can group all related data features with the same unique key together.

[0063] For example: Suppose we have two accounts (Account 1 and Account 2), and each account has two features that need to be updated (M1 and M2). The system will then group these features according to the account ID and feature type, forming the following groups: Group 1: Account 1's M1; Group 2: M2 of Account 1; Group 3: Account 2's M1; Group 4: M2 of Account 2.

[0064] S132. For each group of data features, determine the update time slice based on the time unit and event time.

[0065] In this embodiment, after grouping, the system needs to determine the update time slice for each group of data features. The time slice is calculated based on the feature's time unit (such as day, hour, minute, etc.) and the current event time. For example, if a feature's time unit is "day" and the current event time is "May 6, 2025", then the update time slice for that feature might be from "May 5, 2025" to "May 6, 2025".

[0066] This step is crucial because it directly affects which data will be included in the feature calculation. For example, for feature M1 (average account transaction amount over the last 30 days), its time slice is the period from the start time of the previous 30 days to the current event time.

[0067] S133. Generate specific update request parameters based on the update time slice.

[0068] The final step is to generate specific update request parameters for each set of data features based on the determined update time slice. These request parameters include, but are not limited to: The specific value or numerical range of the feature (such as transaction amount); The timestamp of the current event time; Update the start and end timestamps of the time slice; Other configuration or status information that may affect feature calculation.

[0069] These parameters will then be used to initiate queries or calculation requests to the feature library in order to update the corresponding data feature values.

[0070] Continuing with the example above, for account 1's M1 feature (average transaction amount over the last 30 days), the generated update request parameters would look something like: [200.0, 1746288000, 1744214400, 0] Here, the first parameter represents the transaction amount of the new application; the second parameter is the timestamp of the current event; the third parameter is the timestamp from 30 days ago; and the fourth parameter may represent other configuration information or status information.

[0071] Through steps S131 to S133, the system can effectively manage and update data characteristics related to each account, ensuring that these characteristics reflect the latest business dynamics while achieving a balance between computational load and resource consumption. This not only improves the system's real-time performance and accuracy but also provides strong support for subsequent risk assessment and decision analysis.

[0072] Once the data feature query is complete, the system will initiate an update request based on the retrieved data. This process first involves grouping the data features to be calculated based on the unique key of each data feature. Then, a specific time slice is determined based on the time unit and event time of each group of data features, and corresponding request parameters are generated accordingly. This ensures that a balance is found between computational load and resource consumption while meeting real-time requirements. Taking an incoming data entry as an example, this data contains key information such as account information, transaction type, transaction amount, and event time. For example, account 1 made a transfer transaction of 200 yuan on May 6, 2025. Next, the system will use this incoming data to extract relevant feature parameters from the feature library, which contains statistical results from multiple dimensions, each with its unique calculation logic and time window.

[0073] For example, feature M1 focuses on the average transaction amount of an account over the past 30 days. It constructs its request parameters using a series of parameters, including the current transaction amount, the timestamp of the current event, the timestamp from 30 days ago, and other configuration or status information. Feature M2, on the other hand, focuses on the number of transfers made by the account over the past 30 days, and its request parameters include the timestamp of the current event, the timestamp from 30 days ago, and other possible status information. For feature MN, although the specific parameters are not given, it can be inferred that they are also based on specific time slices and dimensions to reflect statistical results. The entire processing flow includes steps such as parsing incoming data, determining the time window, generating request parameters, querying the feature library, and returning results. This series of operations enables the system to efficiently extract feature information closely related to the current transaction from a large amount of data, providing strong support for subsequent risk assessment and decision analysis.

[0074] S140. Calculate specific data features for each unique key based on the update request parameters, and perform aggregation processing to obtain the final data features; The final data features refer to the dataset formed by aggregating the specific data features calculated for each unique key (such as an account ID) based on the update request parameters. This dataset integrates all relevant time slices and intermediate results, providing a comprehensive view of a specific entity (such as an account, user, etc.).

[0075] Specifically, the final data characteristics include, but are not limited to, the following aspects: Statistical data summary: The results obtained after performing calculations on the characteristics of each data set according to different statistical methods (such as average, cumulative frequency, etc.). These results reflect the changing trends or status of the data within a specific time period.

[0076] Time dimension recording: By associating each calculation result with its corresponding time slice, the time dimension of data changes is recorded. This not only helps track the development trend of data over time, but also provides a time series basis for subsequent analysis.

[0077] Information aggregation and integration: Aggregating intermediate results from all time slices to form a complete set of data features. This set provides a comprehensive data snapshot for each unique key (such as an account), containing the latest calculation results of all relevant features under that key.

[0078] Optimize storage mechanisms: Employ efficient storage methods to preserve these intermediate results and utilize timestamps to ensure data consistency and query efficiency. This step is crucial for ensuring system real-time performance, reducing computational load, and improving resource utilization.

[0079] Business application support: The final data features form the basis for further business logic such as risk assessment and decision analysis. Accurate and timely data features can greatly improve the quality and efficiency of business decisions.

[0080] In summary, the final data feature is a comprehensive dataset obtained after a series of calculations, aggregations, and optimized storage processes. It can accurately reflect the status and development trend of a specific entity over a period of time, providing strong support for subsequent business decisions.

[0081] In one embodiment, please refer to Figure 5 The above step S140 may include steps S141 to S144.

[0082] S141. Perform a data statistics method on each group of data features according to the update request parameters to obtain the calculation results.

[0083] In this embodiment, the calculation result refers to the result obtained after performing a specific data statistical method on each group of data features according to the update request parameters. For example, for features M1 and M2 under account 1, the system will select the corresponding statistical method (such as average, cumulative count, etc.) based on the time slice generated by its time unit and event time, and perform calculations based on the data within these time slices. For feature M1, it may be necessary to calculate the average of the most recent days or weeks; while for M2, it may be necessary to calculate the cumulative count of the most recent months. This process ensures that the data features can accurately reflect the latest business dynamics.

[0084] S142. Associate each calculation result with the corresponding time slice to record the time dimension of data changes, so as to obtain the intermediate results of the time slice.

[0085] In this embodiment, intermediate results refer to associating each calculation result with its corresponding time slice to record the time dimension of data changes. In this way, the system not only saves the statistical results of the data but also retains the time information corresponding to these results, which is crucial for analyzing the trend of data changes over time. For example, if the calculation result of M1 is the average of the past week, this result will be marked with a specific time range for subsequent querying and analysis.

[0086] S143. Aggregate the intermediate results of all the time slices to obtain the final data features.

[0087] This step involves aggregating the intermediate results from all time slices to obtain the final data features. This means that for each unique key (e.g., account 1), the system summarizes the intermediate results of all its features (e.g., M1 and M2) to form a comprehensive view. This aggregation process helps provide more complete information and supports higher-level decision analysis.

[0088] S144. Save intermediate results by optimizing storage methods and use timestamps to ensure data consistency and query efficiency.

[0089] To improve query efficiency and ensure data consistency, the system employs optimized storage methods to preserve intermediate results and uses timestamps to track data change history. Optimized storage can include techniques such as using efficient index structures and compression algorithms to reduce storage space while accelerating data retrieval. Timestamps ensure data consistency and reliability even under concurrent updates.

[0090] In summary, through the steps S141 to S144 described above, the system can effectively calculate, record, aggregate, and store data features, ensuring the real-time nature, accuracy, and query efficiency of the data, thereby providing strong support for application scenarios such as risk assessment and decision analysis.

[0091] After a data feature update request is triggered, each unique key is grouped together. Based on the request parameters generated by the data feature update request module, different data statistical methods are executed to obtain different results, which are then placed into time slices. The final calculation result is obtained by aggregating the time slices calculated for different features. The storage method of intermediate results can accelerate subsequent queries, and the included timestamps can improve data consistency and reliability. A feature calculation request refers to the system receiving a feature calculation request, which contains the data feature information that needs to be updated. Unique Key (PK) Extraction and Grouping: Based on the data feature information in the request, the system first extracts unique key values ​​(e.g., account ID) and groups the data features based on these unique keys. In this example, the unique key is "Account 1", and the related data features include M1 and M2, which correspond to different timestamps and values, respectively. For each group (e.g., Account 1), the system further processes its various features (e.g., M1 and M2). Taking M1 as an example, it contains a series of timestamps and corresponding values, such as {1746828000: [1800, 41], 1746515200: [-9, 31], ...}. The system determines which data needs to be included in this update based on the current event time (e.g., 1746288000) and update time (e.g., 1746374400). Update Time Slice Generation: For each data feature, the system generates a time slice for this update based on its time unit and event time. For example, for M1, it might need to consider data from the last few days or weeks; while for M2, it might be data from the last few months. Request Parameter Generation: Based on the above time slices and other relevant information, the system generates specific request parameters for each feature. These parameters will be used to query or calculate the corresponding statistical results from the feature library. Feature Library Query and Result Return: Using the generated request parameters, the system queries the corresponding feature value from the feature library and returns the query result to the subsequent processing module. For example, for M1, the query result might be a new average; for M2, it might be a cumulative count. After all relevant features have been updated, the system integrates these updated feature values ​​to form the final result set (RESULT), which is then returned to the caller for further risk assessment, decision analysis, and other business logic. Through this series of steps, the system can efficiently process massive amounts of data, ensuring the real-time nature and accuracy of data features while achieving a balance between computational load and resource consumption. This not only improves data processing efficiency but also provides a solid foundation for data-driven decision-making.

[0092] S150. Store the final data feature results in the KV database using a unique key.

[0093] After calculation, the final data feature results are stored. The final results are stored using unique keys, and the storage structure in the KV database is shown in Table 1. The unique key (PK) is used to uniquely identify the data feature and is typically composed of dimensional information; the values ​​(M1, M2, and RESULT) store the data feature calculation results within a time slice, including intermediate calculation results between time slices M1 and M2 and the final calculation result of RESULT. This storage method ensures efficient access to data features while improving query efficiency.

[0094] Table 1. Storage structure of data features in a KV database This embodiment employs a time-slice storage mechanism for efficient data feature management. Through unique key indexing, hierarchical storage, and query optimization, it achieves a significant improvement in storage and computation efficiency. Specifically, the data feature storage structure uses a key-value (KV) model, distinguishing between intermediate and final results, and performing full computation only when necessary, thereby reducing storage consumption and computational burden. During the query process, the system prioritizes matching the storage location of the final result and utilizes time-slice information to quickly locate the required data range, ensuring efficient data feature computation and retrieval.

[0095] To optimize the data feature calculation process, the method in this embodiment adopts a read-write separation mode, supporting both synchronous and asynchronous updates. This design allows the final result to be read first during data querying, avoiding unnecessary duplicate calculations; while during data updates, synchronous or asynchronous processing methods are selected according to the actual scenario, reducing the impact of the calculation process on query performance and improving the overall throughput and real-time performance of the system.

[0096] The system's data feature read / write separation architecture consists of two modules: data feature query and data feature update, to improve query efficiency and update performance. During the data feature query process, the system first categorizes and organizes data features within the same dimension based on dimensional information, ensuring efficient querying. When event information is input, the system iterates through each feature, retrieves the corresponding data from its configured dimensional information, and concatenates them to form a unique key (PK) for that feature dimension. This key is used to quickly query all data features belonging to that feature, avoiding traversal queries and improving query performance.

[0097] During the data feature update process, upon receiving event information, the system first performs filtering based on the logical calculation conditions configured for the data features to determine the data features that need to be updated. Then, it iterates through each data feature, generating corresponding time-slice data according to its configured time unit and quantity, and efficiently organizing it within the storage structure. The system supports both synchronous and asynchronous update modes. Synchronous updates are suitable for low-latency scenarios, ensuring that data features are updated immediately after changes; while asynchronous updates are suitable for high-throughput scenarios, reducing storage pressure and improving overall system performance through batch processing.

[0098] Specifically, during the data feature query process, the system first categorizes and organizes data features within the same dimension based on their dimensional information to ensure efficient querying. When event information is input, the system receives the data and iterates through each feature, retrieving the corresponding data for its configured dimension within the event information. The system then concatenates the dimension identifier with its corresponding value to form the unique key (PK) for that feature dimension. The relationship between the unique key and the data feature is one-to-many. This unique key is used to quickly query all data features belonging to it, avoiding traversal queries and improving query performance.

[0099] During the data feature update process, after receiving event information, the system first calculates the logical expression based on the configured logical conditions for the data features, selecting data features whose logical results are true. These are the data features that need to be updated for this event. This initial filtering avoids unnecessary calculations. Subsequently, each data feature is iterated over, and time slice data corresponding to its configured time unit and number of units is generated for efficient organization in the storage structure. To meet different business needs, the system supports both synchronous and asynchronous update modes. Synchronous updates are suitable for low-latency scenarios, ensuring that data features are updated immediately after changes, while asynchronous updates are suitable for high-throughput scenarios, reducing storage pressure and improving overall system performance through batch processing.

[0100] This embodiment addresses the core issues in current data feature computation—namely, computation efficiency and data storage structure optimization. It proposes a data feature computation method that combines read-write merging and read-write separation, allowing these two approaches to mutually enhance overall computational performance. Particularly in large-scale data scenarios, this method extracts the necessary computational data based on the data feature computation logic, reducing storage overhead, and employs a hierarchical storage strategy to improve query efficiency. Furthermore, by storing intermediate and final results hierarchically, it reduces data redundancy and storage overhead, and quickly locates query results based on a dictionary structure. This approach improves both computational and query efficiency while reducing computational resource consumption, making it highly suitable for large-scale data analysis and real-time computing applications.

[0101] The aforementioned indicators or data characteristics refer to the statistical results of a certain behavior under a certain dimension within a certain period of time. For example, the total transaction amount of customer A within 30 days, or the account numbers associated with the same device within one month.

[0102] A time slice is a data structure that stores data within a clock range such as minutes, hours, days, months, or years. For example, a day time slice records data from 00:00:00 to 23:59:59 today.

[0103] The aforementioned read-write separation data feature storage optimization method acquires event information from external systems and parses it into a dictionary format to extract key information. Based on this key information, it efficiently retrieves the final calculation results of data features from the storage system and determines the data features that need to be updated. Then, it intelligently groups the data features to be updated by unique keys, creates corresponding time slices to generate update request parameters, calculates specific data features for each unique key based on these parameters, and performs aggregation processing to obtain the final data features. Finally, it stores these results in a key-value database by unique keys. This process utilizes read-write separation and time-slice storage mechanisms to optimize the calculation and storage efficiency of data features, reducing unnecessary redundant calculations and storage overhead. Simultaneously, it supports both synchronous and asynchronous update modes to meet the real-time and throughput requirements of different scenarios, thus providing a more efficient and flexible solution for complex data analysis and significantly improving the overall system performance and response speed.

[0104] Figure 7 This is a schematic block diagram of a read-write separation data feature storage optimization device 300 provided in an embodiment of the present invention. Figure 7 As shown, corresponding to the above-described read-write separation data feature storage optimization method, the present invention also provides a read-write separation data feature storage optimization apparatus 300. This read-write separation data feature storage optimization apparatus 300 includes a unit for executing the above-described read-write separation data feature storage optimization method, and the apparatus can be configured in a server. Specifically, please refer to... Figure 7 The read-write separation data feature storage optimization device 300 includes a parsing unit 301, a query unit 302, an update unit 303, a calculation unit 304, and a storage unit 305.

[0105] The parsing unit 301 is used to acquire event information input from an external system, parse it into a dictionary format, and extract key information; the query unit 302 is used to retrieve the final calculation results of data features from the storage system based on the key information, and determine the data features that need to be updated; the update unit 303 is used to group the data features that need to be updated according to unique keys, and create corresponding time slices to generate update request parameters; the calculation unit 304 is used to calculate specific data features for each unique key based on the update request parameters, and perform aggregation processing to obtain the final data features; the storage unit 305 is used to store the final data feature results in a KV database according to unique keys.

[0106] In one embodiment, the parsing unit 301 includes: The extraction subunit is used to acquire event information from an external system and parse the event information into a dictionary format to obtain extracted information; the attribute determination subunit is used to determine the calculated attributes required for subsequent data feature queries based on the extracted information to obtain key information.

[0107] In one embodiment, the query unit 302 includes: The parsing subunit is used to parse the query primary key and determine the corresponding time slice based on the event occurrence time and feature configuration; the checking subunit is used to check whether the final calculation result of the time slice in the storage is valid; the return subunit is used to return the final calculation result from the RESULT field in the KV database if it is valid; and the determination subunit is used to treat the final calculation result as the data feature that needs to be updated if the result is invalid or does not exist.

[0108] In one embodiment, the return subunit is used to compare the time slice with the most recent update time. If the unit time during which the time slice advances forward is not less than the most recent update time, the final calculation result is valid and can be read directly; if the unit time during which the time slice advances forward is less than the most recent update time, the final calculation result is invalid or does not exist.

[0109] In one embodiment, the update unit 303 includes: The grouping subunit is used to group the data features to be updated according to their unique keys to obtain multiple groups of data features; the time slice determination subunit is used to determine the update time slice for each group of data features based on the time unit and event time; and the generation subunit is used to generate specific update request parameters based on the update time slice.

[0110] In one embodiment, the computing unit 304 includes: The statistics subunit is used to perform data statistics on each group of data features according to the update request parameters to obtain calculation results; the association subunit is used to associate each calculation result with the corresponding time slice to record the time dimension of data changes to obtain intermediate results of the time slice; the aggregation subunit is used to aggregate the intermediate results of all the time slices to obtain the final data features; and the storage subunit is used to save the intermediate results by optimizing the storage method and using time stamps to ensure data consistency and query efficiency.

[0111] It should be noted that those skilled in the art can clearly understand that the specific implementation process of the above-mentioned read-write separation data feature storage optimization device 300 and each unit can be referred to the corresponding description in the foregoing method embodiments. For the sake of convenience and brevity, it will not be repeated here.

[0112] The aforementioned read / write separation data feature storage optimization device 300 can be implemented as a computer program, which can, for example... Figure 8 It runs on the computer device shown.

[0113] Please see Figure 8 , Figure 8 This is a schematic block diagram of a computer device provided in an embodiment of this application. The computer device 500 can be a server, wherein the server can be a standalone server or a server cluster composed of multiple servers.

[0114] See Figure 8 The computer device 500 includes a processor 502, a memory, and a network interface 505 connected via a system bus 501. The memory may include a non-volatile storage medium 503 and internal memory 504.

[0115] The non-volatile storage medium 503 may store an operating system 5031 and a computer program 5032. The computer program 5032 includes program instructions that, when executed, cause the processor 502 to perform a read-write separated data feature storage optimization method.

[0116] The processor 502 provides computing and control capabilities to support the operation of the entire computer device 500.

[0117] The internal memory 504 provides an environment for the operation of the computer program 5032 in the non-volatile storage medium 503. When the computer program 5032 is executed by the processor 502, the processor 502 can execute a read-write separation data feature storage optimization method.

[0118] This network interface 505 is used for network communication with other devices. Those skilled in the art will understand that... Figure 8The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the computer device 500 to which the present application is applied. The specific computer device 500 may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0119] The processor 502 is used to run a computer program 5032 stored in the memory to perform the following steps: The system acquires event information from external systems, parses it into a dictionary format, and extracts key information. Based on the key information, it retrieves the final calculation results of data features from the storage system and determines the data features that need to be updated. The system groups the data features that need to be updated according to unique keys and creates corresponding time slices to generate update request parameters. For each unique key, it calculates specific data features based on the update request parameters and performs aggregation processing to obtain the final data features. The final data feature results are stored in a key-value database according to unique keys.

[0120] The key information includes the query primary key, the event occurrence time, and the calculated attributes.

[0121] In one embodiment, when the processor 502 implements the step of acquiring event information input from an external system, parsing it into a dictionary format, and extracting key information, the specific steps are as follows: Event information from an external system is acquired and parsed into a dictionary format to obtain extracted information; based on the extracted information, the computational attributes required for subsequent data feature queries are determined to obtain key information.

[0122] In one embodiment, when the processor 502 implements the step of retrieving the final calculation result of data features from the storage system based on the key information and determining the data features that need to be updated, it specifically implements the following steps: Parse the query primary key and determine the corresponding time slice based on the event occurrence time and feature configuration; check whether the final calculation result of the time slice in the storage is valid; if valid, return the final calculation result from the RESULT field in the KV database storage; if the result is invalid or does not exist, use the final calculation result as the data feature that needs to be updated.

[0123] In one embodiment, when implementing the step of checking whether the final calculation result of the time slice in storage is valid, the processor 502 specifically implements the following steps: Compare the time slice with the most recent update time. If the unit time that the time slice advances forward is not less than the most recent update time, the final calculation result is valid and can be read directly. If the unit time that the time slice advances forward is less than the most recent update time, the final calculation result is invalid or does not exist.

[0124] In one embodiment, when the processor 502 implements the step of grouping the data features to be updated according to a unique key and creating corresponding time slices to generate update request parameters, the specific implementation steps are as follows: The data features to be updated are grouped according to their unique keys to obtain multiple groups of data features; for each group of data features, an update time slice is determined based on the time unit and event time; specific update request parameters are generated based on the update time slice.

[0125] In one embodiment, when the processor 502 implements the step of calculating specific data features for each unique key based on the update request parameters and performing aggregation processing to obtain the final data features, the following steps are specifically implemented: According to the update request parameters, a data statistics method is performed on each group of data features to obtain calculation results; each calculation result is associated with the corresponding time slice to record the time dimension of data changes to obtain intermediate results of the time slice; the intermediate results of all time slices are aggregated to obtain the final data features; the intermediate results are saved by optimizing the storage method, and time stamps are used to ensure data consistency and query efficiency.

[0126] It should be understood that in the embodiments of this application, the processor 502 may be a central processing unit (CPU), or it may be other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor may be a microprocessor or any conventional processor.

[0127] It will be understood by those skilled in the art that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program includes program instructions and can be stored in a storage medium, which is a computer-readable storage medium. The program instructions are executed by at least one processor in the computer system to implement the process steps of the embodiments of the above methods.

[0128] Therefore, the present invention also provides a storage medium. This storage medium can be a computer-readable storage medium. The storage medium stores a computer program, wherein when executed by a processor, the computer program causes the processor to perform the following steps: The system acquires event information from external systems, parses it into a dictionary format, and extracts key information. Based on the key information, it retrieves the final calculation results of data features from the storage system and determines the data features that need to be updated. The system groups the data features that need to be updated according to unique keys and creates corresponding time slices to generate update request parameters. For each unique key, it calculates specific data features based on the update request parameters and performs aggregation processing to obtain the final data features. The final data feature results are stored in a key-value database according to unique keys.

[0129] The key information includes the query primary key, the event occurrence time, and the calculated attributes.

[0130] In one embodiment, when the processor executes the computer program to acquire event information input from an external system, parse it into a dictionary format, and extract key information, it specifically implements the following steps: Event information from an external system is acquired and parsed into a dictionary format to obtain extracted information; based on the extracted information, the computational attributes required for subsequent data feature queries are determined to obtain key information.

[0131] In one embodiment, when the processor executes the computer program to achieve the final calculation result of retrieving data features from the storage system based on the key information and to determine the data features that need to be updated, it specifically implements the following steps: Parse the query primary key and determine the corresponding time slice based on the event occurrence time and feature configuration; check whether the final calculation result of the time slice in the storage is valid; if valid, return the final calculation result from the RESULT field in the KV database storage; if the result is invalid or does not exist, use the final calculation result as the data feature that needs to be updated.

[0132] In one embodiment, when the processor executes the computer program to implement the step of checking whether the final calculation result of the time slice in storage is valid, it specifically implements the following steps: Compare the time slice with the most recent update time. If the unit time that the time slice advances forward is not less than the most recent update time, the final calculation result is valid and can be read directly. If the unit time that the time slice advances forward is less than the most recent update time, the final calculation result is invalid or does not exist.

[0133] In one embodiment, when the processor executes the computer program to implement the step of grouping the data features to be updated according to a unique key and creating corresponding time slices to generate update request parameters, the specific implementation includes the following steps: The data features to be updated are grouped according to their unique keys to obtain multiple groups of data features; for each group of data features, an update time slice is determined based on the time unit and event time; specific update request parameters are generated based on the update time slice.

[0134] In one embodiment, when the processor executes the computer program to perform the step of calculating specific data features for each unique key based on the update request parameters and performing aggregation processing to obtain the final data features, the processor specifically implements the following steps: According to the update request parameters, a data statistics method is performed on each group of data features to obtain calculation results; each calculation result is associated with the corresponding time slice to record the time dimension of data changes to obtain intermediate results of the time slice; the intermediate results of all time slices are aggregated to obtain the final data features; the intermediate results are saved by optimizing the storage method, and time stamps are used to ensure data consistency and query efficiency.

[0135] The storage medium can be any computer-readable storage medium capable of storing program code, such as a USB flash drive, portable hard drive, read-only memory (ROM), magnetic disk, or optical disk.

[0136] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of both. To clearly illustrate the interchangeability of hardware and software, the components and steps of the various examples have been generally described in terms of functionality in the foregoing description. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementations should not be considered beyond the scope of this invention.

[0137] In the several embodiments provided by this invention, it should be understood that the disclosed apparatus and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative. For example, the division of each unit is merely a logical functional division, and there may be other division methods in actual implementation. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed.

[0138] The steps in the method of this invention can be adjusted, merged, or reduced in order according to actual needs. The units in the device of this invention can be merged, divided, or reduced according to actual needs. Furthermore, the functional units in the various embodiments of this invention can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit.

[0139] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a storage medium. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, a terminal, or a network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention.

[0140] The above description is merely a specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any person skilled in the art can easily conceive of various equivalent modifications or substitutions within the technical scope disclosed in the present invention, and these modifications or substitutions should all be covered within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims.

Claims

1. A read-write separation data feature storage optimization method, characterized in that, include: Acquire event information input from external systems, parse it into dictionary format, and extract key information; Based on the key information, the final calculation results of the data features are retrieved from the storage system, and the data features that need to be updated are determined. Based on the data characteristics that need to be updated, group them according to a unique key and create corresponding time slices to generate update request parameters; For each unique key, specific data features are calculated based on the update request parameters, and then aggregated to obtain the final data features. The final data feature results are stored in a KV database using a unique key.

2. The read-write separation data feature storage optimization method according to claim 1, characterized in that, The key information includes the query primary key, the event occurrence time, and the calculated attributes.

3. The read-write separation data feature storage optimization method according to claim 2, characterized in that, The process of acquiring event information input from external systems, parsing it into a dictionary format, and extracting key information includes: Obtain event information from external systems and parse the event information into a dictionary format to obtain the extracted information; Based on the extracted information, the computational attributes required for subsequent data feature queries are determined in order to obtain key information.

4. The read-write separation data feature storage optimization method according to claim 1, characterized in that, The final calculation result of retrieving data features from the storage system based on the key information, and determining the data features that need to be updated, includes: Parse the query primary key and determine the corresponding time slice based on the event occurrence time and feature configuration; Check whether the final calculation result of the time slice in storage is valid; If valid, the final calculation result is returned from the RESULT field stored in the KV database; If the result is invalid or does not exist, the final calculation result will be used as the data feature that needs to be updated.

5. The read-write separation data feature storage optimization method according to claim 4, characterized in that, The step of checking whether the final calculation result of the time slice in storage is valid includes: Compare the time slice with the most recent update time. If the unit time that the time slice advances forward is not less than the most recent update time, the final calculation result is valid and can be read directly. If the unit time that the time slice advances forward is less than the most recent update time, the final calculation result is invalid or does not exist.

6. The read-write separation data feature storage optimization method according to claim 1, characterized in that, The step of grouping the data based on the required update features according to a unique key and creating corresponding time slices to generate update request parameters includes: Group the data features according to their unique keys to obtain multiple sets of data features; For each set of data features, an update time slice is determined based on the time unit and event time; Specific update request parameters are generated based on the update time slice.

7. The read-write separation data feature storage optimization method according to claim 1, characterized in that, The process of calculating specific data features for each unique key based on the update request parameters and performing aggregation processing to obtain the final data features includes: A data statistics method is performed on each group of data features according to the update request parameters to obtain the calculation results; Each calculation result is associated with its corresponding time slice to record the time dimension of data changes, thereby obtaining intermediate results for the time slice. Aggregate the intermediate results of all the time slices to obtain the final data features; By optimizing storage methods to preserve intermediate results, and using timestamps to ensure data consistency and query efficiency.

8. A read-write separated data feature storage optimization device, characterized in that, include: The parsing unit is used to acquire event information input from external systems, parse it into dictionary format, and extract key information; The query unit is used to retrieve the final calculation results of data features from the storage system based on the key information, and to determine the data features that need to be updated. The update unit is used to group the data that needs to be updated according to a unique key and create a corresponding time slice to generate update request parameters; The calculation unit is used to calculate specific data features for each unique key based on the update request parameters, and perform aggregation processing to obtain the final data features; A storage unit is used to store the final data feature results into a KV database using a unique key.

9. A computer device, characterized in that, The computer device includes a memory and a processor, the memory storing a computer program, and the processor executing the computer program to implement the method as described in any one of claims 1 to 7.

10. A storage medium, characterized in that, The storage medium stores a computer program that, when executed by a processor, implements the method as described in any one of claims 1 to 7.

Images