Electronic forensics data management method and device based on real-time data warehouse, and storage medium

Abstract

The invention provides an electronic evidence collection data management method and device based on a real-time data warehouse and a storage medium, and the method comprises the steps: storing original data in a data source layer (ODS), and storing the original data in a corresponding ODS table according to a theme; the data warehouse detail layer DWD records the preprocessed original data according to the dimension of the data packet, stores the preprocessed original data in a DWD table, summarizes the preprocessed original data according to the file dimension in a preset time period, and stores the summarized original data in a DWD time period table; the data warehouse service layer DWS stores the preprocessed original data into a DWS table according to the dimension of an account number, warehousing and statistical separation are carried out, the number of collection times is accumulated, and data exceeding a preset threshold value are counted twice; and the data application layer ADS acquires data from the data source layer ODS, the data warehouse detail layer DWD or the data warehouse service layer DWS according to the configured indexes. According to the method, the layering thought of the data warehouse is used, the library table of electronic data governance analysis is designed, the data governance efficiency can be improved, resource consumption is reduced, and stability and expansibility are improved.

Description

Technical Field

[0001] The present application relates to the field of data processing technology, and in particular to a method, device and storage medium for electronic forensic data management based on a real-time data warehouse. Background Art

[0002] At present, the largest daily structured electronic forensic data processing volume in the country has reached TB level, and these data are collected and uploaded in real time. In the face of massive data, high-performance real-time management and storage and real-time analysis are required. Massive data needs to be stored and used for a long time, and the hardware resources required are very large (more than 100 servers for 500 billion data). Not only must the servers be fully scheduled and utilized, but the performance of the cluster must also be guaranteed to be stable.

[0003] In terms of business functions, deducing the unknown from the known is the most important idea for electronic forensic data analysis. Suspicious information that has not been collected can be discovered based on the collected data, such as suspicious friend accounts. In business, such as single-person portraits, there are suspicious friends who have been collected multiple times based on the full database data, whether they are common friends of the same case, etc. In specific implementation, it is necessary to calculate data such as the number of related cases, the number of related persons, and the total number of collections. If each query is to check the full database data, the tables that need to be associated according to this association relationship are: person-main account-friend account-all friend relationships-other main accounts, and finally the results are counted. If the data is not governed in advance, each query will consume a lot of computing resources, and bottlenecks will occur if the governance is improper.

[0004] The analysis of electronic forensic data must also adapt to new environmental changes. With the emergence of new requirements such as changes in techniques and tactics, the analysis results can be delivered quickly. For example, if there is a sudden need to pay attention to the active time of friends, or to filter out people who are active in the middle of the night, etc., data governance must also be flexible and scalable. If there is no prior data governance in this regard, temporary statistics of the entire database will be required, or even re-governance will be required, which is time-consuming and laborious.

[0005] The main methods of data governance at present are:

[0006] (1) In traditional governance methods, either the database is not layered, data jumps, and scalability is poor; or the data layering is not reasonable, and the database is built directly based on business functions. Although the amount of computing during governance may be small, when there are new changes in demand, massive historical data must be reprocessed, which is a huge workload. The scalability is not enough to meet the changes in demand. In addition, because the business requires complex statistical analysis, a large amount of computing resources are occupied, affecting the normal use of other businesses.

[0007] (2) Offline data warehouses are generally used to calculate daily, weekly, and monthly data, and are often implemented using technologies such as Spark and Hive. The biggest drawback is limited timeliness: offline data warehouses usually process batches of historical data and cannot provide real-time or near real-time data processing and analysis, and do not provide sufficient support for business scenarios that require rapid response and decision-making.

[0008] (3) Real-time data warehouses generally use streaming data processing and real-time statistics, and often use flink, kafka, elasticsearch, etc. This solves the timeliness problem, but the amount of calculation is large and requires continuous calculation. When new data comes in, it must be associated with historical data in real time, which requires a large amount of computing resources and is prone to performance bottlenecks. This method is the current big data processing architecture Kappa.

[0009] (4) Lambda big data processing architecture. Lambda combines the big data architecture of batch processing (offline data warehouse) and stream processing (real-time data warehouse), stores two sets of data, divides the processing into different tasks, generates different results, integrates them, and the results can be mutually verified. However, it requires doubling the hardware resources. Summary of the invention

[0010] In order to solve the above technical problems, the present invention provides an electronic forensic data management method, device and storage medium based on a real-time data warehouse. The layered idea of ​​the data warehouse is used to design the library table for electronic data management and analysis, which can improve the efficiency of data management, reduce resource consumption, and improve stability and scalability.

[0011] The present invention adopts the following technical solution:

[0012] On the one hand, a method for electronic forensic data governance based on a real-time data warehouse is applied to a real-time data warehouse model including a data source layer ODS, a data warehouse detail layer DWD, a data warehouse service layer DWS and a data application layer ADS; the method includes:

[0013] Parse and extract the electronic forensic data source, and push the parsed and extracted raw data to the data source layer ODS;

[0014] The data source layer ODS stores the original data and stores the original data in the corresponding ODS table according to the subject;

[0015] The data warehouse detail layer DWD records the pre-processed raw data according to the dimensions of the data package, retains the preset key fields and lightweight calculation fields, and stores them in the DWD table; at the same time, the pre-processed raw data is summarized according to the file dimension and the preset time period, and then stored in the DWD time period table;

[0016] The data warehouse service layer DWS stores the pre-processed raw data in the DWS table according to the account dimension, separates the storage and statistics, accumulates the number of collections, and counts the data that exceeds the preset threshold twice, without associating all incremental data with all existing historical data;

[0017] The data application layer ADS obtains data from the data source layer ODS, the data warehouse detail layer DWD or the data warehouse service layer DWS according to the configured indicators.

[0018] Preferably, the preprocessing includes cleaning and converting the data.

[0019] Preferably, the data warehouse detail layer DWD further includes: using the data of the DWD daily table to aggregate and supplement the DWD table.

[0020] Preferably, the data of the DWD daily table is used to aggregate and supplement the DWD table, specifically including:

[0021] After the data of the entire data package is stored in the database, the DWD daily table is queried by the data package ID, and the record fields that exist in the DWD daily table but not in the DWD table are upsert to DWDFriend.

[0022] Preferably, the processing of the data warehouse service layer DWS includes two stages, as follows:

[0023] In the first phase, upsert is performed to store the specified fields. In the second phase, data in the specified fields that meet the preset constraints are selected for statistical update.

[0024] Preferably, the electronic forensic data is friend data. 7. The electronic forensic data management method based on real-time data warehouse according to claim 6 is characterized in that the specific implementation of the first stage is as follows:

[0025] The data is stored in the database through upsert, and the fields of collection times, number of cases involved, and number of related persons are accumulated. If they do not exist in the database, the value is 1, and if they exist, the value is accumulated. The earliest collection time and the latest collection time fields are updated by comparing the minimum and maximum values ​​during upsert. The friend account, nickname, and update time fields are directly updated during upsert, and the remarks field for others is updated by appending.

[0026] Preferably, the second stage is specifically implemented as follows:

[0027] Through scheduled task scheduling, the scheduled task interval specifies the time, records the current time as the "this statistical time", and uses it as a condition to query the update time field of the DWS table. If the update time value∈{last statistical end time, this statistical time} is within the time window, then the number of collections is greater than 1. After obtaining the friend account field, use it as a condition to query the corresponding DWS table, count the number of collections, the number of cases involved, the number of related people, other people's notes + times, and the number of main accounts involved in various types of cases, and confirm whether there are common numbers for similar cases. After the statistics are completed, update the corresponding DWS table. If a friend account is updated multiple times in a time window, it only needs to be counted once. Before the end of a single scheduled task, change the "this statistical time" to the "last statistical end time" for the next task scheduling.

[0028] In another aspect, an electronic forensic data management device includes:

[0029] one or more processors;

[0030] memory; and,

[0031] One or more computer programs, wherein the one or more computer programs are stored in the memory and configured to be executed by the processor to implement the electronic forensic data governance method based on real-time data warehouse.

[0032] On the other hand, a computer-readable storage medium stores a computer program, which enables a computer to execute to implement the electronic forensic data governance method based on real-time data warehouse.

[0033] The present invention has the following beneficial effects:

[0034] (1) The present invention draws on the concept of data warehouse and combines the characteristics of electronic forensic data to design data in a reasonable layered manner. Data is aggregated level by level, and computing resources are reasonably allocated to each level, thereby narrowing the query scope and improving statistical efficiency and stability. Data at different levels provide different degrees of data granularity for use in different business scenarios. Based on high-level processed data, very complex query analysis can be performed to reduce development complexity. If the demand changes, only the individual data at the corresponding level needs to be expanded without affecting the existing functions.

[0035] (2) The present invention is improved on the basis of real-time data warehouse. The DWD layer takes the subject record as the dimension, first stores the basic data, aggregates the daily table, and then performs lightweight statistics; the DWS layer separates storage and statistics, accumulates the number of collections, and performs secondary statistics on the data exceeding the threshold, without the need to associate all incremental data with all existing historical data; the Ads layer directly faces the business, reduces the difficulty of development, and flexibly adapts to new needs; in general, the large query is split into multiple levels of small-scale calculations, which reduces the fluctuation of resource usage and improves stability, and will not cause abnormalities due to the sudden occupation of a large number of resources by individual complex operations;

[0036] (3) In terms of resource usage, the present invention reduces data storage and computing resources by half compared to the Lambda architecture, and also achieves real-time management and real-time analysis, thereby improving the efficiency of business query analysis;

[0037] (4) The present invention is not limited to the use of specific technical implementations. Fink, Spark, Storm, or native Java projects can be used for data preprocessing. Database storage can be mongo, greenplum, doris, or mysql, postgrep. Middleware and databases can be selected as needed.

[0038] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work. BRIEF DESCRIPTION OF THE DRAWINGS

[0039] Figure 1 It is a flow chart of the electronic forensic data management method based on real-time data warehouse according to an embodiment of the present invention;

[0040] Figure 2 A schematic diagram of a model of an electronic forensics data management method based on a real-time data warehouse according to an embodiment of the present invention;

[0041] Figure 3 It is a structural diagram of the electronic forensic data management device according to an embodiment of the present invention. DETAILED DESCRIPTION

[0042] The technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention; it is obvious that the described embodiments are only part of the embodiments of the present invention, rather than all the embodiments, and all other embodiments obtained by ordinary technicians in this field based on the embodiments of the present invention without making creative work are within the scope of protection of the present invention.

[0043] In the description of the present invention, it should be noted that the terms "comprises", "includes" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also includes other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, an element defined by the sentence "comprises a ..." does not exclude the existence of other identical elements in the process, method, article or device including the element.

[0044] In the description of the present invention, it should be noted that, unless otherwise clearly specified and limited, step identifiers S101, S102, S103, etc. are only used for convenient expression and do not represent the execution order. The corresponding execution order can be adjusted as needed.

[0045] The data warehouse is generally divided into four layers: ODS, DWD, DWS, and ADS. Among them, ODS is the aggregation of original data; DWD is the detail layer to extract the relationship between key data; DWS is the aggregation layer, storing the subject data; ADS is the business layer, providing data support for each business unit.

[0046] See also Figure 1 and Figure 2 As shown, this embodiment is an electronic forensic data governance method based on a real-time data warehouse, which is applied to a real-time data warehouse model including a data source layer ODS, a data warehouse detail layer DWD, a data warehouse service layer DWS and a data application layer ADS; the method is as follows.

[0047] S101, parse and extract the electronic forensic data source, and push the parsed and extracted original data to the data source layer ODS.

[0048] Specifically, parsing and extraction mainly involves connecting to the electronic forensic data source (data packages compressed from various data files) and reading the content of the data source in a certain standard format.

[0049] S102, the data source layer ODS stores the original data, and stores the original data into the corresponding ODS table according to the subject.

[0050] Specifically, the data source layer ODS is the preparation area of ​​the data warehouse, which stores the original data that has not been cleaned and converted, and maintains the same format and structure as the source data, that is, stores the most original data, including all fields.

[0051] The following is a detailed explanation using the example of electronic forensic data being friend data.

[0052] As shown in Table 1, the ODS table is a friend ODS table ODSFriend.

[0053] Table 1

[0054] Field Name type describe account text Master Account accountId string Account Id addTime date Join Time age integer age area string Country Code areaCode string Area Code backgroundPath string Background media relative path backgroundUrl string Background Media URL bloodType string blood type friendAccount text Friend Account friendGroup text Friends Group friendId string FriendId friendLabel text Label friendNickname text Friend Nickname friendRemark text Friend Notes graduateSchool text Graduate School headUrl string Avatar Url hometown text hometown hsAccount string phone number ……

[0055] Friend data ODSFriend, such as Weibo friends, mobile phone contacts, instant messaging friends, email contacts, etc., are accessed as raw data at this stage and divided into multiple tables. Some types of data in instant messaging friends may have a large volume, such as WeChat friends, which account for a large proportion and can be split out.

[0056] S103, the data warehouse detail layer DWD records the pre-processed raw data according to the dimensions of the data packet, retains the preset key fields and lightweight calculation fields, and stores them in the DWD table; at the same time, the pre-processed raw data is summarized according to the file dimension in the preset time period and stored in the DWD time period table.

[0057] Specifically, the data in the ODS layer is first cleaned, deduplicated, and formatted, such as merging duplicate data, deleting empty data, transcoding text, etc., to ensure the consistency of the same data in the DWD layer. Then, the data is recorded in the dimension of the electronic forensic data packet, retaining key fields and lightweight calculation fields. The dimension of the data packet is specifically that the data is merged by package layer. If there are two identical numbers in the same package, two pieces of data are stored in the ODS layer and merged into one piece in the DWD layer.

[0058] The following description is made by taking the DWD table being the DWD friend table DWDFriend and the DWD time period table being the friend communication date table DWDFriendDate as an example.

[0059] (1) DWDFriend, the friend table of the DWD layer, is constructed. The dimension record data of the collected records is retained, the key fields are retained, and lightweight calculation fields such as the number of contacts are added. The data can be split and merged according to the data volume level (according to the quantity distribution of various types of data in the electronic forensic data, it is recommended to split into: WeChat friends, QQ friends, mobile phone address book, other friends), and the number of tables is less than that of the ODS layer. For example, WeChat friends, QQ friends, mobile phone address book, other friends (including Weibo friends, email contacts, other instant messaging friends, etc.), if the number of these friends is small, they can be merged into one table. The statistics of the number of contacts depend on the friend contact daily table. The structure of DWDFriend is shown in Table 2 below.

[0060] Table 2

[0061] Field Name type describe _id String Primary Key packageId String Packet ID account String Master Account nickname String Nick name friendAccount String Friend Account isFriend Boolean Friendship friendNickname String Friend Nickname friendRemark String Friend Notes knish String Kinship firstTime Date First contact time latestTime Date Last contact time communicationCount Integer Number of contacts (statistics)

[0062] (2) In general, in an application scenario, the amount of communication data between friends in the electronic forensic data accounts for about 30%, and its repeated collection rate is very low. From the perspective of space utilization, it is not suitable to store another copy at the detail layer. Therefore, when managing the storage, it is summarized by file-day. When adding the summary of the file dimension (electronic forensic data standard file), it is considered that the data can be directly overwritten when it is repeatedly stored (the data of the same file is abnormally interrupted and the data packet is re-stored) to ensure data consistency. Therefore, a friend communication day table DWDFriendDate is established, and the fields used to calculate the ID are: data packet ID, file name, main account, friend account, account type, date, etc. In addition, referring to the design of the zipper table, the friend communication day table is expanded to have 24-hour fields, and the number of communications in each hour is summarized according to the time point of the communication. Other accumulated fields are the number of messages sent between the main account and the friend account, the number of messages received, the total number of communications, etc., as shown in Table 3 below.

[0063] Table 3

[0064] Field Name type describe _id String Primary Key packageId String Packet ID account String Master Account accountId String Account Id nickname String Nick name friendId String FriendId friendAccount String Friend Account friendNickname String Nickname of the member sending the message sourceFileName String Source Files sendDate String Date sent (yyyyMMdd) communicateCount Integer Number of contacts callingCount Integer Number of outgoing calls calledCount Integer Number of called calls sendCount Integer Number of messages sent receiveCount Integer Number of received messages firstTime Date First contact time latestTime Date Last contact time H00 Integer 0 point contact number …… H23 Integer 23 points of contact

[0065] In one application scenario, for more than 300 million friend communication data, only more than 4 million friend communication daily tables were generated, saving about 70 times the storage space. To count the number of communication times of two accounts, it is only necessary to query and summarize the more than 4 million daily table data, without querying and summarizing more than 300 million data, which improves the calculation speed. The performance improvement is particularly obvious when doing time axis (counting the situation every day within a period of time).

[0066] (3) You can aggregate and supplement data to the DWDFriend table. In some scenarios, there may be omissions when collecting data, that is, the main account has a communication record with a friend, but it is not collected in the address book. At this time, you need to use the data from the DWDFriendDate table to aggregate and supplement DWDFriend to ensure data integrity. After the entire data package dw data is entered into the database, you can query DWDFriendDate by data package ID, select the necessary main account, friend account, "number of communications" and other fields, and upsert them to DWDFriend.

[0067] S104, the data warehouse service layer DWS stores the pre-processed raw data into the DWS table according to the account dimension, separates the storage and statistics, accumulates the number of collection times, and counts the data exceeding the preset threshold twice, without associating all incremental data with all existing historical data.

[0068] The data warehouse service layer DWS no longer aggregates data in the dimension of data packets, but stores it in the dimension of accounts. It is the most critical layer of data, which connects the past and the future, aggregates the data of the DWD layer, and provides a data source for the ADS layer. If it is properly managed, it can flexibly provide data of various dimensions, which will reduce the pressure of data management at the ADS layer and greatly improve the scalability of the system.

[0069] Let's continue to take friend data as an example. When data is stored in DWDFriend, a copy of it also needs to be stored in the DWS table DWSFriend. When it is stored, the main account is no longer stored, and only the friend account information is retained. Most friend accounts will not be collected repeatedly, and only a small number of friend accounts are collected repeatedly, that is, a friend account is a friend of multiple collected main accounts (common friends). If the friend account is only collected once, there will be no statistical indicators such as the number of notes and whether there are common numbers in similar cases that need to be calculated. Therefore, it is distinguished when entering the database, and the data entering the DWS layer is divided into two steps. In the first stage, upsert is entered into the database to accumulate the number of collections (appearing in an electronic forensics data package is counted as one collection), etc. In the second stage, data with a collection number greater than 1 is selected for statistical update. This will narrow the scope, and most of the friends who will not be repeated will be entered into the database in the first step, and only a small number of common friend accounts need to be counted in the second step. The structure of DWSFriend is shown in Table 4 below.

[0070] Table 4

[0071]

[0072]

[0073] Specifically, in the first stage, data is stored in the database through upsert, and the number of collections, number of cases involved, and number of related persons need to be accumulated. If it does not exist in the database, it is 1. If it exists, the number of times is accumulated. The earliest collection time and the latest collection time are updated by comparing the maximum and minimum values ​​during upsert. The friend account, nickname, and update time are directly updated during upsert, and other people's notes are added. At this time, if a friend account is stored for the first time, only the values ​​of the previous few fields will be available. Number of collections 1, number of cases involved (1 or 0), number of related persons (1 or 0), friend account, friend nickname, etc. At this time, if the second stage is entered according to the "Update Time" window to filter the records with a collection time greater than 1 for statistics, this data will not be queried. If a friend account is stored for the second time and the number of collections is accumulated to 2 after upsert, it will be filtered out and enter the second stage of statistics.

[0074] In the second stage, scheduled tasks are scheduled. For example, the scheduled tasks are scheduled every 10 minutes, and the current time is recorded as "this statistical time". It is used as a condition to query the data collected more than 1 times in the time window of DWSFriend.updateTime∈{last statistical end time, this statistical time}. After obtaining the friend account, the corresponding DWDFriend table is used as a condition to count the number of collections, the number of cases involved, and the number of related people (the number of cases involved and the number of related people are not repeated in the first stage of accumulation, and are uniformly treated as accumulation). Here, the statistics are corrected. You can also count the number of other people's notes + times, the number of main accounts involved in various cases, and confirm whether they are common numbers for similar cases. After the statistics are completed, update DWSFriend. If the friend account is updated for the third time, it will be counted again. However, if it is updated multiple times in a time window, it only needs to be counted once. Before the end of a single scheduled task, change "this statistical time" to "last statistical end time" in preparation for the next task scheduling.

[0075] S105, the data application layer ADS acquires data from the data source layer ODS, the data warehouse detail layer DWD or the data warehouse service layer DWS according to the configured indicators.

[0076] With the data statistics of the DWS layer, the data of the ADS layer no longer needs to be counted from the entire database. You can scan the DWS layer regularly and filter the data that meets the requirements, which reduces the amount of calculation. For example, if you want to check the notes of a certain account by others to determine its real name, just query "other people's notes + times". If you want to pick out all the common numbers of similar cases among friends as important clues, just take the data of DWSFriend.isSameCase=true and make a small amount of association. If you need to query details, query the data of DWDFriend, and query ODSFriend to trace the source.

[0077] Table 5 below is a business table generated according to configuration indicators in the data application layer ADS, taking friend data as an example.

[0078] Table 5

[0079]

[0080]

[0081] and Figure 1 The embodiment shown is consistent with that shown in FIG. Figure 3 , which is a schematic diagram of the structure of the electronic forensic data management device provided in the embodiment. The electronic forensic data management device 300 includes a processor 310, a memory 320 and a communication interface 330. The processor 310, the memory 320 and the communication interface 330 are interconnected via a bus 350.

[0082] The memory 320 stores a computer program 340 , and the computer program 340 is configured to be executed by the processor 310 .

[0083] The computer program 340 includes instructions for performing the following steps:

[0084] Parse and extract the electronic forensic data source, and push the parsed and extracted raw data to the data source layer ODS;

[0085] The data source layer ODS stores the original data and stores the original data in the corresponding ODS table according to the subject;

[0086] The data warehouse detail layer DWD records the pre-processed raw data according to the dimensions of the data package, retains the preset key fields and lightweight calculation fields, and stores them in the DWD table; at the same time, the pre-processed raw data is summarized according to the file dimension and the preset time period, and then stored in the DWD time period table;

[0087] The data warehouse service layer DWS stores the pre-processed raw data in the DWS table according to the account dimension, separates the storage and statistics, accumulates the number of collections, and counts the data that exceeds the preset threshold twice, without associating all incremental data with all existing historical data;

[0088] The data application layer ADS obtains data from the data source layer ODS, the data warehouse detail layer DWD or the data warehouse service layer DWS according to the configured indicators.

[0089] In addition, an embodiment of the present application also provides a computer-readable storage medium, which stores a computer program, and the computer program enables a computer to execute to implement the electronic forensic data management method based on a real-time data warehouse.

[0090] The above is only a preferred specific implementation of the present invention; however, the protection scope of the present invention is not limited thereto. Any technician familiar with the technical field can make equivalent replacements or changes according to the technical solution and its improved conception within the technical scope disclosed by the present invention, which should be covered by the protection scope of the present invention.

Claims

1. A method for managing electronic forensic data based on real-time data warehouse, characterized in that: The real-time data warehouse model is applied to the data source layer ODS, the data warehouse detail layer DWD, the data warehouse service layer DWS and the data application layer ADS. The method includes: Parse and extract the electronic forensic data source, and push the parsed and extracted raw data to the data source layer ODS; The data source layer ODS stores the original data and stores the original data in the corresponding ODS table according to the subject; The data warehouse detail layer DWD records the pre-processed raw data according to the dimensions of the data package, retains the preset key fields and lightweight calculation fields, and stores them in the DWD table; at the same time, the pre-processed raw data is summarized according to the file dimension and the preset time period, and then stored in the DWD time period table; The data warehouse service layer DWS stores the pre-processed raw data in the DWS table according to the account dimension, separates the storage and statistics, accumulates the number of collections, and counts the data that exceeds the preset threshold twice, without associating all incremental data with all existing historical data; The data application layer ADS obtains data from the data source layer ODS, the data warehouse detail layer DWD or the data warehouse service layer DWS according to the configured indicators.

2. The electronic forensic data management method based on real-time data warehouse according to claim 1 is characterized in that: The preprocessing includes cleaning and converting the data.

3. The electronic forensic data management method based on real-time data warehouse according to claim 1 is characterized in that: The data warehouse detail layer DWD also includes: using the data of the DWD daily table to aggregate and supplement the DWD table.

4. The electronic forensic data management method based on real-time data warehouse according to claim 3 is characterized in that: Use the data of DWD daily table to aggregate and supplement the DWD table, including: After the data of the entire data package is stored in the database, the DWD daily table is queried by the data package ID, and the record fields that exist in the DWD daily table but not in the DWD table are upsert to DWDFriend.

5. The electronic forensic data management method based on real-time data warehouse according to claim 1 is characterized in that: The processing of the data warehouse service layer DWS includes two stages, as follows: In the first phase, upsert is performed to store the specified fields. In the second phase, data in the specified fields that meet the preset constraints are selected for statistical update.

6. The electronic forensic data management method based on real-time data warehouse according to claim 5 is characterized in that: The electronic forensic data is friend data.

7. The electronic forensic data management method based on real-time data warehouse according to claim 6 is characterized in that: The specific implementation of the first stage is as follows: The data is stored in the database through upsert, and the fields of collection times, number of cases involved, and number of related persons are accumulated. If the data does not exist in the database, the value is 1, and if the data exists, the value is accumulated. The earliest collection time and the latest collection time fields are updated by comparing the minimum and maximum values ​​during upsert. The friend account, nickname, and update time fields are directly updated during upsert, and the other person's note fields are updated in an appended manner.

8. The electronic forensic data management method based on real-time data warehouse according to claim 7 is characterized in that: The specific implementation of the second stage is as follows: Through scheduled task scheduling, the scheduled task interval specifies the time, records the current time as "this statistical time", and uses it as a condition to query the update time field of the DWS table. If the update time value∈{last statistical end time, this statistical time} is within the time window, then the number of collections is greater than 1. After obtaining the friend account field, use it as a condition to query the corresponding DWS table, count the number of collections, the number of cases involved, the number of related people, other people's notes + times, and the number of main accounts involved in various cases, and confirm whether there are common numbers for similar cases. After statistics are completed, update the corresponding DWS table; If a friend account is updated multiple times in a time window, it only needs to be calculated once; before the end of a single scheduled task, the "current statistical time" will be changed to the "last statistical end time" for the next task scheduling.

9. An electronic forensic data management device, characterized in that: include: one or more processors; Memory; as well as, One or more computer programs, wherein the one or more computer programs are stored in the memory and configured to be executed by the processor to implement the electronic forensic data governance method based on real-time data warehouse as described in any one of claims 1 to 8.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, which enables a computer to execute to implement the electronic forensic data management method based on a real-time data warehouse as described in any one of claims 1 to 8.

Images