Data storage method and system based on legal knowledge service platform and storage medium
Through data feature extraction and clustering analysis, an adaptive multi-level storage architecture is built, combined with ETL and incremental storage technology, the complexity and timeliness of data management in the legal knowledge service platform is solved, and efficient and low-cost data storage is achieved and visit.
Patent Information
- Application Number
- CN202410986815.5
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-07-23
- Publication Date
- 2024-11-15
- Estimated Expiration
- Not applicable · inactive patent
AI Technical Summary
Legal knowledge service platforms face the problems of complexity in management of various types of data, uneven access to hot and cold data, and high data timeliness, and traditional data storage methods are difficult to effectively manage and store.
Adopting data feature extraction and clustering analysis based on data type, usage frequency, importance and timeliness, build an adaptive multi-level storage architecture model, and standardize mapping and version control through ETL technology, combining incremental storage and automatic data Migration strategy to realize intelligent storage.
Optimize the utilization of storage resources, reduce costs, improve data access efficiency and user experience, ensure data consistency and traceability, and meet the needs of legal knowledge service platforms for efficient data access and management.
Smart Images

Figure CN118964496A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of data storage technology, and in particular to a data storage method and system based on a legal knowledge service platform. Background Art
[0002] In the era of rapid development of informatization and digitalization, legal knowledge service platform plays an increasingly important role as an important tool for legal information integration and application. The main task of legal knowledge service platform is to collect, organize, store and provide legal information to support legal practitioners and the public to quickly find and efficiently use legal information. However, with the explosive growth of legal data, traditional data storage and management methods face many challenges: legal knowledge service platform needs to process multiple types of data, including laws, cases, legal documents, legal reviews, academic papers, etc. These data types are different, and structured, semi-structured and unstructured data coexist, which brings complexity to data management; there is an obvious hot and cold distribution in the access frequency of legal data. For example, the latest laws and regulations and hot cases have a high access frequency, while the historical data and uncommon legal clauses have a low access frequency. How to effectively manage these hot and cold uneven data is an important issue; the timeliness of legal data is high, especially for laws and regulations and precedents, timely update and version management are particularly important. Summary of the invention
[0003] Based on this, it is necessary for the present invention to provide a data storage method and system based on a legal knowledge service platform to solve at least one of the above technical problems.
[0004] To achieve the above purpose, a data storage method based on a legal knowledge service platform includes the following steps:
[0005] Step S1: Acquire legal knowledge service data; extract data features based on data type, usage frequency, importance and timeliness of the legal knowledge service data, and perform cluster analysis to obtain classification result data; construct an adaptive multi-level storage architecture model based on the classification result data;
[0006] Step S2: Determine the storage solutions for the hot data layer, the warm data layer, and the cold data layer according to the multi-level storage architecture model, and establish a data migration strategy model based on historical access patterns, so as to obtain automatic data migration strategies between different layers;
[0007] Step S3: Establish standardized mapping of legal knowledge service data to reduce semantic differences through ETL technology, thereby obtaining legal standardized data; perform version control based on time dimension management and query control based on geographic spatial index on legal standardized data, thereby obtaining version controlled data;
[0008] Step S4: New data is stored in the legal standardization data according to the version control data through the incremental storage technology, thereby obtaining version management model data; storage level allocation is performed on the newly stored data according to the automatic data migration strategy, thereby obtaining initial storage allocation data;
[0009] Step S5: Construct an unstructured data management platform based on a distributed document database, an object storage system, and a full-text search engine according to the legal standardization data and the initial storage allocation data, and perform semantic-based intelligent indexing according to the version management model data to obtain intelligent indexing data to realize intelligent storage of data.
[0010] The present invention can better understand and manage legal knowledge service data through data feature extraction and cluster analysis, and the classification result data provides a structured view of the data, which is convenient for subsequent storage and processing. Based on the adaptive multi-level storage architecture model, different categories of data can be stored in appropriate storage levels, thereby optimizing storage resources and reducing costs. Hot data can be stored on storage media with higher performance, while cold data can be migrated to storage media with lower costs. After storing data in layers according to access patterns and features, users' access requirements for different data categories can be better met. Hot data can respond to user query requests more quickly, while cold data can be stored for a long time at a lower cost. According to the determination of the storage scheme, data can be stored at the most suitable level, balancing performance requirements and cost-effectiveness. Hot data can be efficiently stored at a higher performance level, while cold data can be migrated to a cheaper storage level. By establishing a data migration strategy model based on historical access patterns, automatic data migration can be achieved. This ensures that frequently accessed data is in a higher performance storage level, while infrequently accessed data can be automatically migrated to a lower cost storage level, thereby improving the utilization efficiency of storage resources. Through the establishment of standardized mapping, semantic differences in legal knowledge service data can be eliminated, data consistency and standardization can be ensured, and data quality and credibility can be improved. Version control based on the time dimension can help track and manage the change history of data, enabling users to access and compare data versions at different time points, which helps to maintain the integrity and traceability of data; through query control based on geospatial indexes, geographical location-related queries and analysis of legal standardized data can be achieved, which is very useful for scenarios that require legal research and analysis based on geographical location. Through incremental storage technology, incremental updates of legal standardized data can be efficiently processed, avoiding repeated processing and storage of the entire data set, and improving processing speed and efficiency. Through automatic data migration strategies, newly entered data is allocated to the appropriate storage tier, which can ensure that frequently accessed data is stored in a higher-performance tier, while infrequently accessed data can be stored in a lower-cost tier, achieving optimization of storage resources and cost reduction. After storing newly entered data in the appropriate storage tier, the access efficiency of data can be improved, and hot data is stored in a higher-performance tier, which can respond to user query requests more quickly and provide a better user experience. The automated storage model can realize automated storage management of legal knowledge service data. Data storage and migration will be automatically performed according to the version and storage level, reducing the workload and complexity of manual management. The automated access model can optimize access to legal knowledge service data. According to the user's access needs and the data storage level, the most appropriate storage level is automatically selected for data access, improving data access efficiency and user experience.By automating the storage and access model, the utilization of storage resources and data access efficiency can be optimized, thereby improving the overall performance of the system, which helps meet users' needs for efficient access and processing of legal knowledge services. According to the automatic data migration strategy, infrequently accessed data can be stored in a lower-cost tier to reduce storage costs, which can help organizations effectively manage storage resources and control costs based on the value and access needs of the data.
[0011] The present invention also provides a data storage system based on a legal knowledge service platform, which is used to execute the data storage method based on a legal knowledge service platform as described above. The data storage system based on a legal knowledge service platform includes:
[0012] The storage architecture module is used to obtain legal knowledge service data; extract data features based on data type, usage frequency, importance and timeliness of legal knowledge service data, and perform cluster analysis to obtain classification result data; and build an adaptive multi-level storage architecture model based on the classification result data;
[0013] The multi-layer storage strategy module is used to determine the storage schemes for the hot data layer, warm data layer, and cold data layer according to the multi-level storage architecture model, and to establish a data migration strategy model based on historical access patterns, thereby obtaining automatic data migration strategies between different layers;
[0014] The version control module is used to establish a standardized mapping of legal knowledge service data to reduce semantic differences through ETL technology, thereby obtaining legal standardized data; perform version control based on time dimension management and query control based on geographic spatial index on legal standardized data, thereby obtaining version controlled data;
[0015] The incremental level allocation storage module is used to store new data in the legal standardization data according to the version control data through the incremental storage technology, so as to obtain the version management model data; the storage level allocation is performed on the newly stored data according to the automatic data migration strategy, so as to obtain the initial storage allocation data;
[0016] The intelligent indexing module is used to build an unstructured data management platform based on distributed document databases, object storage systems and full-text search engines according to legal standardization data and initial storage allocation data, and to perform semantic-based intelligent indexing based on version management model data to obtain intelligent indexing data to achieve intelligent storage of data.
[0017] The storage architecture module of the present invention can classify and cluster the data according to the characteristics of the legal knowledge service data, such as data type, frequency of use, importance and timeliness; this helps to extract the characteristics and requirements of the data and provide a basis for subsequent storage and management. The construction of an adaptive multi-level storage architecture model can allocate data to different storage levels according to the classification result data. This can reasonably allocate storage resources according to the access frequency and importance of the data and improve the access efficiency of the data. The multi-layer storage strategy module determines the storage scheme of the hot data layer, the warm data layer and the cold data layer according to the multi-level storage architecture model, so that the data can be stored in the most suitable storage layer according to the access mode and importance of the data, thereby improving the utilization efficiency of storage resources. The data migration strategy model based on the historical access pattern can automatically determine the time and method of data migration from one level to another according to the access mode and trend of the data, which helps to realize the dynamic migration of data and optimize the utilization of storage resources. The version control module uses the ETL technology to establish a standardized mapping of the legal knowledge service data to reduce semantic differences, thereby obtaining legal standardized data, which helps to ensure the consistency and standardization of the data and improve the quality and credibility of the data. Version control based on time dimension management and query control based on geospatial index make the management and retrieval of legal standardization data more flexible and accurate, and can conveniently manage historical versions of data, while supporting query operations based on time and geographic location. The incremental hierarchical allocation storage module realizes the storage of new legal standardization data through incremental storage technology, which helps to merge the newly generated data with the existing data to maintain the integrity and consistency of the data. According to the automatic data migration strategy, the storage level of the newly stored data can be allocated, and the new data can be stored in the most appropriate level according to the characteristics and requirements of the data, which helps to optimize the utilization of storage resources and the access efficiency of data. The intelligent indexing module builds an unstructured data management platform based on legal standardization data and initial storage allocation data. Such a platform can support technologies such as distributed document databases, object storage systems and full-text search engines, and provide efficient data management and query functions. Semantic-based intelligent indexing based on version management model data can better understand and organize data; through intelligent indexing, the semantic information of the data can be extracted, indexes can be established, and more accurate and intelligent data search and retrieval can be achieved.
[0018] Preferably, the present invention also provides a storage medium for data storage based on a legal knowledge service platform, storing a computer program, and when the computer program is executed, it implements the above-mentioned data storage method based on a legal knowledge service platform. BRIEF DESCRIPTION OF THE DRAWINGS
[0019] Other features, objects and advantages of the present invention will become more apparent from the detailed description of non-limiting embodiments thereof made with reference to the following drawings:
[0020] Figure 1 A schematic diagram of the steps of the data storage method based on the legal knowledge service platform of the present invention;
[0021] Figure 2 for Figure 1 Detailed step flow diagram of step S1;
[0022] Figure 3 for Figure 1 Detailed step flow chart of step S2 in FIG. DETAILED DESCRIPTION
[0023] The technical method of the present invention is described clearly and completely below in conjunction with the accompanying drawings. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by technicians in this field without creative work are within the scope of protection of the present invention.
[0024] In addition, the accompanying drawings are only schematic illustrations of the present invention and are not necessarily drawn to scale. The same reference numerals in the figures represent the same or similar parts, and their repeated description will be omitted. Some of the block diagrams shown in the accompanying drawings are functional entities and do not necessarily correspond to physically or logically independent entities. The functional entities can be implemented in software form, or implemented in one or more hardware modules or integrated circuits, or implemented in different networks and / or processor methods and / or microcontroller methods.
[0025] It should be understood that, although the terms "first", "second", etc. may be used herein to describe various units, these units should not be limited by these terms. These terms are used only to distinguish one unit from another unit. For example, without departing from the scope of the exemplary embodiments, the first unit may be referred to as the second unit, and similarly the second unit may be referred to as the first unit. The term "and / or" used herein includes any and all combinations of one or more of the listed associated items.
[0026] To achieve this, please refer to Figures 1 to 3 The present invention provides a data storage method based on a legal knowledge service platform, the method comprising the following steps:
[0027] Step S1: Acquire legal knowledge service data; extract data features based on data type, usage frequency, importance and timeliness of the legal knowledge service data, and perform cluster analysis to obtain classification result data; construct an adaptive multi-level storage architecture model based on the classification result data;
[0028] Step S2: Determine the storage solutions for the hot data layer, the warm data layer, and the cold data layer according to the multi-level storage architecture model, and establish a data migration strategy model based on historical access patterns, so as to obtain automatic data migration strategies between different layers;
[0029] Step S3: Establish standardized mapping of legal knowledge service data to reduce semantic differences through ETL technology, thereby obtaining legal standardized data; perform version control based on time dimension management and query control based on geographic spatial index on legal standardized data, thereby obtaining version controlled data;
[0030] Step S4: New data is stored in the legal standardization data according to the version control data through the incremental storage technology, thereby obtaining version management model data; storage level allocation is performed on the newly stored data according to the automatic data migration strategy, thereby obtaining initial storage allocation data;
[0031] Step S5: Construct an unstructured data management platform based on a distributed document database, an object storage system, and a full-text search engine according to the legal standardization data and the initial storage allocation data, and perform semantic-based intelligent indexing according to the version management model data to obtain intelligent indexing data to realize intelligent storage of data.
[0032] In the embodiment of the present invention, reference Figure 1 The above is a schematic diagram of the steps of a data storage method based on a legal knowledge service platform of the present invention. In this example, the data storage method based on a legal knowledge service platform includes the following steps:
[0033] Step S1: Acquire legal knowledge service data; extract data features based on data type, usage frequency, importance and timeliness of the legal knowledge service data, and perform cluster analysis to obtain classification result data; construct an adaptive multi-level storage architecture model based on the classification result data;
[0034] The embodiment of the present invention needs to obtain legal knowledge service data, which can come from legal documents, legal clauses, case databases or other law-related data sources. Suitable data extraction tools or APIs can be used to collect these data and store them in a data set for subsequent processing. According to the content and structure of the data, the data is classified, such as legal clauses, case descriptions, regulations, etc. The access frequency of the data can be analyzed by counting the number of times users access different types of data or using logs. According to the importance of the data in the legal field, the authority, applicability and wide application of the data can be considered. Indicators such as the data’s timeliness, that is, the data’s update speed and validity period, can be evaluated. Based on the extracted data features, cluster analysis methods can be used to classify legal knowledge service data. Common clustering algorithms include K-means, hierarchical clustering, etc. Cluster analysis can divide data into different categories or clusters, each category representing a data set with similar characteristics. The clustering results can be used as input for subsequent steps to build a storage architecture model.
[0035] Step S2: Determine the storage solutions for the hot data layer, the warm data layer, and the cold data layer according to the multi-level storage architecture model, and establish a data migration strategy model based on historical access patterns, so as to obtain automatic data migration strategies between different layers;
[0036] The embodiment of the present invention determines the storage scheme of the hot data layer, the warm data layer and the cold data layer according to the multi-level storage architecture model. Generally, the hot data layer stores frequently accessed data, the warm data layer stores less frequently accessed but still important data, and the cold data layer stores less frequently accessed and less important data. Select appropriate storage media and technologies, such as high-speed disk storage, flash storage, tape storage, etc., to meet the data access requirements and cost-effectiveness. Based on historical access patterns and data characteristics, a data migration strategy model is established to automatically determine the timing and method of migrating data from one level to another, and analyze the access patterns and trends of the data. For example, the access frequency of certain data may increase within a specific time period, and it needs to be migrated from the warm data layer to the hot data layer; combined with the results of data feature extraction, the migration strategy of data between different levels is determined according to factors such as data type, frequency of use, and importance; considering the availability and cost of storage resources, the proportion of data storage in each level is reasonably allocated.
[0037] Step S3: Establish standardized mapping of legal knowledge service data to reduce semantic differences through ETL technology, thereby obtaining legal standardized data; perform version control based on time dimension management and query control based on geographic spatial index on legal standardized data, thereby obtaining version controlled data;
[0038] The embodiment of the present invention uses ETL (Extract, Transform, Load) technology to establish a standardized mapping of legal knowledge service data to reduce semantic differences. The ETL process includes data extraction, transformation and loading. By cleaning, normalizing and unifying the data format, the consistency and standardization of the data are ensured. A suitable ETL tool or programming language (such as Python) can be used to implement the data standardization process. Legal standardized data is version controlled based on the time dimension to ensure the historical traceability and change management of the data. A version control system (such as Git) can be used to manage different versions of the data and record the modification, author, time and other information of each version. Based on the geospatial index, a query control mechanism is established for the legal standardized data to achieve efficient data retrieval. An appropriate database management system (such as MySQL, Elasticsearch) can be used to create a geospatial index, and a query language (such as SQL, Elasticsearch query statements) can be used to perform geospatial query operations.
[0039] Step S4: New data is stored in the legal standardization data according to the version control data through the incremental storage technology, thereby obtaining version management model data; storage level allocation is performed on the newly stored data according to the automatic data migration strategy, thereby obtaining initial storage allocation data;
[0040] The embodiment of the present invention uses incremental storage technology to store new legal standardization data according to version control data. Incremental storage can effectively handle data updates and changes, and only store newly added or modified parts, reducing storage resource usage and data processing time costs. According to the automatic data migration strategy, the newly stored data is allocated to the storage level, and the data is allocated to the appropriate hot data layer, warm data layer or cold data layer according to the characteristics and access mode of the data to achieve optimal utilization of storage resources.
[0041] Step S5: Construct an unstructured data management platform based on a distributed document database, an object storage system, and a full-text search engine according to the legal standardization data and the initial storage allocation data, and perform semantic-based intelligent indexing according to the version management model data to obtain intelligent indexing data to realize intelligent storage of data.
[0042] The embodiment of the present invention constructs an unstructured data management platform based on legal standardized data and initial storage allocation data. The platform may include components such as a distributed document database (such as MongoDB), an object storage system (such as Amazon S3) and a full-text search engine (such as Elasticsearch) for storing and managing legal knowledge service data. Based on version management model data and semantic analysis technology, legal standardized data is intelligently indexed, and natural language processing (NLP) technology and machine learning algorithms can be used to extract information such as keywords, entity recognition and semantic associations to enhance the searchability and intelligence of data.
[0043] The present invention can better understand and manage legal knowledge service data through data feature extraction and cluster analysis, and the classification result data provides a structured view of the data, which is convenient for subsequent storage and processing. Based on the adaptive multi-level storage architecture model, different categories of data can be stored in appropriate storage levels, thereby optimizing storage resources and reducing costs. Hot data can be stored on storage media with higher performance, while cold data can be migrated to storage media with lower costs. After storing data in layers according to access patterns and features, users' access requirements for different data categories can be better met. Hot data can respond to user query requests more quickly, while cold data can be stored for a long time at a lower cost. According to the determination of the storage scheme, data can be stored at the most suitable level, balancing performance requirements and cost-effectiveness. Hot data can be efficiently stored at a higher performance level, while cold data can be migrated to a cheaper storage level. By establishing a data migration strategy model based on historical access patterns, automatic data migration can be achieved. This ensures that frequently accessed data is in a higher performance storage level, while infrequently accessed data can be automatically migrated to a lower cost storage level, thereby improving the utilization efficiency of storage resources. Through the establishment of standardized mapping, semantic differences in legal knowledge service data can be eliminated, data consistency and standardization can be ensured, and data quality and credibility can be improved. Version control based on the time dimension can help track and manage the change history of data, enabling users to access and compare data versions at different time points, which helps to maintain the integrity and traceability of data; through query control based on geospatial indexes, geographical location-related queries and analysis of legal standardized data can be achieved, which is very useful for scenarios that require legal research and analysis based on geographical location. Through incremental storage technology, incremental updates of legal standardized data can be efficiently processed, avoiding repeated processing and storage of the entire data set, and improving processing speed and efficiency. Through automatic data migration strategies, newly entered data is allocated to the appropriate storage tier, which can ensure that frequently accessed data is stored in a higher-performance tier, while infrequently accessed data can be stored in a lower-cost tier, achieving optimization of storage resources and cost reduction. After storing newly entered data in the appropriate storage tier, the access efficiency of data can be improved, and hot data is stored in a higher-performance tier, which can respond to user query requests more quickly and provide a better user experience. The automated storage model can realize automated storage management of legal knowledge service data. Data storage and migration will be automatically performed according to the version and storage level, reducing the workload and complexity of manual management. The automated access model can optimize access to legal knowledge service data. According to the user's access needs and the data storage level, the most appropriate storage level is automatically selected for data access, improving data access efficiency and user experience.By automating the storage and access model, the utilization of storage resources and data access efficiency can be optimized, thereby improving the overall performance of the system, which helps meet users' needs for efficient access and processing of legal knowledge services. According to the automatic data migration strategy, infrequently accessed data can be stored in a lower-cost tier to reduce storage costs, which can help organizations effectively manage storage resources and control costs based on the value and access needs of the data.
[0044] Preferably, step S1 comprises the following steps:
[0045] Step S11: Obtain legal knowledge service data, and perform deduplication and format unification on the legal knowledge service data, thereby obtaining legal cleansing data;
[0046] Step S12: extracting data features of the legal cleansing data based on data type, usage frequency, importance and timeliness, thereby obtaining legal comprehensive feature data;
[0047] Step S13: assign weights to different features of the legal comprehensive feature data, determine the number of clusters 3, and perform cluster analysis based on K-means, thereby generating classification result data in which each data object has a cluster label, wherein the cluster labels include hot data labels, warm data labels, and cold data labels;
[0048] Step S14: Determine the number of storage levels and level characteristics based on the classification result data, so as to construct an adaptive multi-level storage architecture model.
[0049] As an embodiment of the present invention, refer to Figure 2 As shown, Figure 1 Detailed step flow diagram of step S1 in the embodiment of the present invention, step S1 includes the following steps:
[0050] Step S11: Obtain legal knowledge service data, and perform deduplication and format unification on the legal knowledge service data, thereby obtaining legal cleansing data;
[0051] The embodiment of the present invention obtains legal knowledge service data through channels such as relevant databases in the legal field, legal documents, legal publications, government public data, and contributions from legal experts. Web crawler tools (such as Python's Scrapy framework) can be used for automated data collection. The obtained legal knowledge service data is deduplicated to exclude duplicate data records. At the same time, the data is formatted in a unified manner to ensure data consistency and standardization. Data cleaning tools (such as Python's pandas library) can be used to perform data deduplication and format conversion operations.
[0052] Step S12: extracting data features of the legal cleansing data based on data type, usage frequency, importance and timeliness, thereby obtaining legal comprehensive feature data;
[0053] The embodiments of the present invention extract data type characteristics, such as case data, regulatory data, judgment document data, etc., based on the content and structure of legal knowledge service data, and can use natural language processing technology and text classification algorithms to automatically identify and classify data types. To analyze the frequency with which legal knowledge service data is accessed, statistics and analysis can be performed based on the data access log or user behavior data; based on the access frequency of the data, usage frequency characteristics are extracted, such as popular data, commonly used data, unpopular data, etc. Based on the importance of the legal knowledge service data, the importance characteristics of the data are extracted, and the importance of the data can be evaluated based on factors such as the source, authority, and legal influence of the data. To analyze the timeliness of legal knowledge service data, the timeliness characteristics of the data are extracted, and the timeliness of the data can be evaluated based on information such as the data release date and update frequency.
[0054] Step S13: assign weights to different features of the legal comprehensive feature data, determine the number of clusters 3, and perform cluster analysis based on K-means, thereby generating classification result data in which each data object has a cluster label, wherein the cluster labels include hot data labels, warm data labels, and cold data labels;
[0055] The embodiment of the present invention assigns weights to the comprehensive legal feature data to determine the degree of influence of each feature on the clustering result. The weight value can be determined by subjective evaluation or data analysis-based methods; the comprehensive legal feature data is clustered using the K-means clustering algorithm. The data object is divided into 3 clusters, corresponding to hot data, warm data, and cold data respectively.
[0056] Step S14: Determine the number of storage levels and level characteristics based on the classification result data, so as to construct an adaptive multi-level storage architecture model.
[0057] The embodiment of the present invention determines the number of storage levels in a multi-level storage architecture based on the classification result data and business requirements. The appropriate number of levels is determined based on the ratio and access requirements of hot data, warm data, and cold data. The characteristics and storage strategy of each storage level are determined based on the classification result data and the number of storage levels. For example, hot data can be stored in a high-performance storage medium (such as a solid-state drive), warm data can be stored in a medium-performance storage medium (such as a mechanical hard drive), and cold data can be stored in a low-cost storage medium (such as a tape library). Based on the determined number of storage levels and the characteristics of the levels, an adaptive multi-level storage architecture model is designed and constructed. A storage management system (such as Hadoop, HDFS) can be used to implement multi-level storage, and the data can be stored in the corresponding storage level based on the access mode and requirements of the data.
[0058] The deduplication operation of the present invention can eliminate duplicates in the data set, ensure the accuracy and consistency of the data, and the format unification can convert the data into a unified format, making it easier to process and analyze. By removing duplicate data and unifying the format, the quality of the data can be improved, and the cleaned data is more reliable, easier to understand and use, which helps to improve the accuracy and credibility of subsequent data processing and analysis. The format unification process can ensure that the data maintains a consistent format and structure throughout the data set, which helps to avoid problems caused by data inconsistency and provides a more consistent data foundation. By extracting features from legal cleansing data, key feature information of the data, such as data type, frequency of use, importance and timeliness, can be extracted, which helps to understand and describe the characteristics of the data and provide a basis for subsequent data classification and storage level division. The extracted data features can provide a basis for subsequent data classification and storage level division. According to the characteristics of the data, the data can be better organized and managed to meet different data types and usage requirements. Based on the extracted data features, further data analysis and mining can be performed, which helps to discover the correlation, regularity and potential value between data, and provide deeper insights and decision support for legal knowledge services. Through cluster analysis, legal comprehensive feature data can be divided into different cluster clusters, and corresponding cluster labels can be assigned to each data object. This helps to classify and organize data, making it easier to manage and access. Cluster labels can be used to determine storage tier divisions, such as hot data, warm data, and cold data, which helps to store data at the appropriate level and optimize storage resources and control costs based on the access frequency and importance of data. According to the results of cluster analysis, hot data can be stored in a higher-performance level to improve data access efficiency and response speed, and warm data and cold data can be stored in a lower-performance level to save storage costs. Based on the classification result data and the labels of cluster analysis, the number and level characteristics of storage levels can be determined, which helps to build an adaptive multi-level storage architecture model so that data can be stored in different levels based on its access frequency, importance, and timeliness. By determining the storage tier, storage resources can be allocated to the appropriate tier according to the characteristics of the data. Hot data can be stored in a high-performance storage tier to provide fast access and response time; warm data can be stored in a medium-performance tier to balance performance and cost; cold data can be stored in a low-performance but lower-cost tier to save storage costs. By storing data on the appropriate tier, data access efficiency and response speed can be improved. Frequently accessed hot data can be obtained faster, while infrequently accessed cold data can be retrieved when needed. Allocating data to different storage tiers based on its importance and access frequency can achieve storage cost control. High-performance storage tiers are suitable for data with higher performance requirements, while low-performance storage tiers are suitable for data with lower performance requirements, thereby optimizing storage resource utilization and cost-effectiveness.
[0059] Preferably, step S12 comprises the following steps:
[0060] Step S121: extracting features of the legal cleansing data based on the degree of structure and text length, thereby obtaining type feature data;
[0061] The embodiment of the present invention analyzes and extracts the structured degree of legal cleansing data, and can evaluate the structured degree of data based on structured information such as fields, attributes, and tags of the data; for example, the number of fields in the data, the diversity of field types, and other indicators can be calculated as structured degree features. The length of the text content of legal cleansing data is counted and analyzed, and indicators such as the number of characters and words in the data text can be calculated as text length features.
[0062] Step S122: extracting frequency characteristics of the number of accesses and the most recent access time according to the legal cleansing data, thereby obtaining frequency characteristic data;
[0063] The embodiment of the present invention is based on the access log or user behavior data of the legal cleansing data, and the number of times the data is accessed can be calculated, and indicators such as the total number of accesses or the average number of accesses to the data can be calculated as the number of accesses feature. The time when the data was last accessed can be analyzed, and the most recent access time feature can be extracted based on the timestamp information in the access log or user behavior data.
[0064] Step S123: extracting the number of citations and the legal effectiveness level of the legal cleansing data, and assigning weights to the number of citations and the legal effectiveness level, thereby generating importance feature data;
[0065] The embodiment of the present invention counts and analyzes the number of times data is cited in other legal documents or citation networks, and can calculate the number of citations of data through text matching or network analysis methods. The legal cleansing data is divided and extracted into legal effectiveness levels, and the data is divided into corresponding legal effectiveness levels, such as the Constitution, laws, administrative regulations, local regulations, etc., according to the classification standards and rules in the legal system. Weights are assigned to the number of citations and legal effectiveness levels, and their respective weight values are determined based on business needs and professional judgment. Subjective evaluation, expert opinions or data analysis methods can be used to determine the weight values.
[0066] Step S124: extracting timeliness features from the legal cleansing data through the time difference between the effective time and the expiration time, thereby obtaining timeliness feature data;
[0067] The embodiment of the present invention analyzes the effective time and expiration time information in the legal cleansing data and calculates the time difference between the two. The time difference can be calculated using a date calculation tool or a date function in a programming language. According to the time difference between the effective time and the expiration time, the timeliness feature of the data is extracted; the time difference can be converted into units such as days, months or years as the timeliness feature.
[0068] Step S125: combining the type feature data, the frequency feature data, the importance feature data and the timeliness feature data into comprehensive feature data.
[0069] The embodiment of the present invention merges the type feature data, frequency feature data, importance feature data and timeliness feature data extracted in steps S121, S122, S123 and S124 to form comprehensive feature data, and data processing tools (such as Python's pandas library) can be used to merge and organize the data.
[0070] The type feature data of the present invention provides structured information about legal texts, such as the type, format, and text length of the document; these features can be used for further analysis and processing, such as selecting an appropriate processing method based on the text structure, or adjusting the processing logic of the algorithm based on the text length. Frequency feature data reflects the access mode and usage frequency of legal texts. These features can be used to optimize the storage and access strategy of legal texts, such as placing frequently accessed texts in a higher-performance storage layer to improve access efficiency. Importance feature data measures the importance of legal texts in the entire system. The number of citations and the level of legal effectiveness can be used to evaluate the authority and influence of the text and provide a reference for subsequent legal analysis and decision-making. Timeliness feature data provides timeliness information of legal texts, that is, the validity period of legal texts between effectiveness and expiration. These features can be used to manage and update legal texts, such as timely discovering expired texts or texts that need to be updated to ensure the accuracy and effectiveness of legal information. Comprehensive feature data combines multiple feature dimensions together to provide a feature representation that comprehensively describes legal texts. These comprehensive features can be used for further legal analysis, classification, recommendation, and other tasks, thereby supporting higher-level legal decision-making and applications. The effects of the above steps include providing structured information, optimizing storage and access, evaluating importance, managing timeliness, and supporting comprehensive analysis and decision-making. Through these feature extraction and synthesis, the characteristics and value of legal cleansing data can be better understood and utilized, providing support for applications in the legal field.
[0071] Preferably, step S2 comprises the following steps:
[0072] Step S21: using the data with hot data labels in the classification result data as the hot data layer of the multi-level storage architecture model, and storing the hot data layer through the memory database, thereby obtaining a hot data layer storage solution;
[0073] Step S22: using the data with warm data labels in the classification result data as the warm data layer of the multi-level storage architecture model, and storing them in a relational database mainly based on solid state drives, thereby obtaining a warm data layer storage solution;
[0074] Step S23: using the data with cold data layer labels in the classification result data as the cold data layer of the multi-level storage architecture model, and storing the cold data layer through a distributed file system, thereby obtaining a cold data layer storage solution;
[0075] Step S24: integrating the hot data layer storage solution, the warm data layer storage solution, and the cold data layer storage solution, thereby obtaining comprehensive storage solution data of the multi-level storage architecture model;
[0076] Step S25: formulating initial data allocation rules between different storage layers based on the classification result data and the comprehensive storage solution data, and defining trigger conditions for data migration from one storage layer to another, thereby obtaining migration rule data;
[0077] Step S26: Establish a data migration strategy model based on historical access patterns according to the migration rule data and the comprehensive storage solution data, so as to obtain automatic data migration strategies between different levels.
[0078] As an embodiment of the present invention, refer to Figure 3 As shown, Figure 1 Detailed step flow diagram of step S2 in the embodiment of the present invention, step S2 includes the following steps:
[0079] Step S21: using the data with hot data labels in the classification result data as the hot data layer of the multi-level storage architecture model, and storing the hot data layer through the memory database, thereby obtaining a hot data layer storage solution;
[0080] The embodiment of the present invention selects a memory database suitable for high performance and low latency as the storage solution for the hot data layer. Some common memory databases include Redis, Memcached, etc., and a suitable memory database is selected according to specific needs and technical requirements. According to the classification result data, the data marked as hot data is labeled and classified, and machine learning algorithms, rule engines or manually defined rules can be used to classify the data. The data marked as hot data is stored in the memory database, and the data is written to the memory database using corresponding tools or programming languages according to the API and operation mode of the memory database.
[0081] Step S22: using the data with warm data labels in the classification result data as the warm data layer of the multi-level storage architecture model, and storing them in a relational database mainly based on solid state drives, thereby obtaining a warm data layer storage solution;
[0082] The embodiment of the present invention selects a relational database suitable for storing a large amount of data as the storage solution for the warm data layer. Considering performance and reliability, a solid state drive (SSD) can be selected as the storage medium. Common relational databases include MySQL, PostgreSQL, etc. According to the classification result data, the data marked as warm data is labeled and classified. The data marked as warm data is stored in a relational database, and a database connection library in a database management tool or a programming language is used to create a table or collection, and the data is inserted into the corresponding table.
[0083] Step S23: using the data with cold data layer labels in the classification result data as the cold data layer of the multi-level storage architecture model, and storing the cold data layer through a distributed file system, thereby obtaining a cold data layer storage solution;
[0084] The embodiment of the present invention selects a distributed file system suitable for large-scale storage and processing of cold data as a storage solution for the cold data layer. Some common distributed file systems include Hadoop HDFS, Ceph, etc. According to the classification result data, the data marked as cold data is labeled and classified. The data marked as cold data is stored in the distributed file system, and the data is written into the file system using corresponding tools or programming languages according to the specific operation mode of the distributed file system.
[0085] Step S24: integrating the hot data layer storage solution, the warm data layer storage solution, and the cold data layer storage solution, thereby obtaining comprehensive storage solution data of the multi-level storage architecture model;
[0086] The embodiment of the present invention comprehensively considers the storage solutions of the hot data layer, the warm data layer and the cold data layer, and formulates a comprehensive storage solution based on factors such as data access frequency, capacity requirements and cost. The specific comprehensive storage solution may include data distribution strategy, backup strategy, cache strategy, etc.
[0087] Step S25: formulating initial data allocation rules between different storage layers based on the classification result data and the comprehensive storage solution data, and defining trigger conditions for data migration from one storage layer to another, thereby obtaining migration rule data;
[0088] The embodiment of the present invention formulates the initial allocation rules of data between different storage layers based on the classification result data and the comprehensive storage solution data, and determines which storage layer the data should be allocated to by considering the characteristics, access mode and performance requirements of the data. The trigger conditions for data migration from one storage layer to another are defined, and the data migration operation can be triggered according to indicators such as data access frequency and storage space utilization.
[0089] Step S26: Establish a data migration strategy model based on historical access patterns according to the migration rule data and the comprehensive storage solution data, so as to obtain automatic data migration strategies between different levels.
[0090] The embodiment of the present invention establishes a data migration strategy model based on historical access patterns based on migration rule data and comprehensive storage solution data; machine learning algorithms, time series analysis and other methods can be used to analyze historical access patterns and predict future data access patterns; according to the prediction results and migration rules, an automatic data migration strategy is formulated to realize automatic migration of data between different storage layers.
[0091] The hot data layer storage solution of the present invention stores data with high frequency access in a memory database to provide fast read and write access performance; the memory database has high data access speed and low latency, which can meet the real-time processing and query requirements of hot data. The warm data layer storage solution uses faster storage media such as solid-state drives to store data with low access frequency but still high access requirements; solid-state drives have high read and write performance and low access latency, which can provide faster data retrieval and processing speed. The cold data layer storage solution stores data with low access frequency in a distributed file system to achieve high capacity and scalability; the distributed file system can effectively manage large-scale data, provide high reliability and fault tolerance, and is suitable for storing cold data and archived data. The comprehensive storage solution data integrates different storage solutions for the hot data layer, warm data layer, and cold data layer into a multi-level storage architecture model; this comprehensive storage solution can store data in layers according to the access mode and requirements of the data to achieve more efficient data management and optimization of storage costs. Migration rule data determines the initial allocation rules of data between different storage layers and the triggering conditions for data migration; these rules can achieve reasonable stratification and dynamic migration of data based on data characteristics and business needs to optimize the utilization of storage resources and the efficiency of data access. The data migration strategy model based on historical access patterns uses past data access patterns and behavior analysis to predict future data access needs; through this model, it can automatically determine when to migrate data from one storage layer to another to meet the performance requirements and cost-effectiveness of data access.
[0092] Preferably, step S26 comprises the following steps:
[0093] Step S261: Obtain historical data access logs; extract access pattern features from the historical data access logs to obtain access pattern feature data, wherein the access pattern features include access frequency features, time distribution features, access timing features, access source features, and associated data access features;
[0094] The embodiment of the present invention obtains historical data access logs through system log records or other related mechanisms. The logs should contain key information, such as access time, access path, access source, etc. Feature extraction is performed on the historical data access logs to obtain access pattern feature data. The following are some common access pattern features: Count the frequency of each data being accessed, and you can use counting or statistical access times within a time period; Analyze the distribution of data access in different time periods, such as daily, weekly or monthly access patterns; Observe the time series pattern of data access, such as access time interval, access duration, etc.; Analyze the source of data access, such as different users, IP addresses or device types; Consider the correlation between data, such as data sets accessed at the same time or the access pattern of related data. According to specific needs and data characteristics, select appropriate feature extraction methods and tools, such as data analysis libraries (such as Pandas, Numpy), time series analysis tools (such as statsmodels), etc.
[0095] Step S262: Perform pattern clustering analysis on the historical data access logs according to the access pattern feature data by using the K-Means clustering algorithm, thereby obtaining access pattern data, wherein the clustering results represent different data access patterns, including high-frequency short-term access patterns, periodic regular access patterns, peak-period concentrated access patterns, long-term stable access patterns, and low-frequency occasional access patterns;
[0096] The embodiment of the present invention preprocesses the extracted access pattern feature data, such as normalization, standardization or dimensionality reduction, to ensure that the feature data is clustered on the same scale. The preprocessed access pattern feature data is clustered using the K-Means algorithm, which is a commonly used clustering algorithm that can divide the data set into K clusters, each cluster representing an access pattern. Based on the clustering results of the K-Means algorithm, the access patterns represented by each cluster are analyzed, such as high-frequency short-term access patterns, periodic regular access patterns, peak-period concentrated access patterns, long-term stable access patterns, and low-frequency occasional access patterns. The interpretation and naming of the clustering results can be customized according to business needs and data characteristics.
[0097] Step S263: setting data migration trigger conditions according to the migration rule data and the access mode data, thereby obtaining a migration trigger condition parameter set;
[0098] The embodiment of the present invention defines migration trigger conditions for different access modes based on existing migration rule data and business requirements. The migration rule data may include data access frequency thresholds, storage layer capacity utilization thresholds, etc. The access mode data obtained in the aforementioned step S262 is used to associate each data with a corresponding access mode. In combination with the migration rule data and the access mode data, a parameter set of the migration trigger condition is set. These parameter sets may include access modes, access frequency thresholds, storage layer capacity utilization thresholds, etc.
[0099] Step S264: establishing a data migration strategy model according to the migration trigger condition parameter set, thereby obtaining automatic data migration strategies between different levels.
[0100] The embodiment of the present invention establishes a data migration strategy model based on a migration trigger condition parameter set. The model can be a rule-based logical model or a machine learning-based model. The following is a possible implementation method of the data migration strategy model: use conditional statements or rule engines and other methods to formulate a data migration strategy according to the migration trigger condition parameter set. For example, if the access frequency of certain data exceeds a set threshold and the storage layer capacity utilization exceeds a set threshold, the data is migrated to a storage layer corresponding to a low-frequency occasional access mode; use a machine learning algorithm, such as a decision tree, a random forest or a neural network, to train a data migration model according to the migration trigger condition parameter set; the training data can include historical data access logs, access pattern data and corresponding migration results. Through the model, it can be predicted whether new data needs to be migrated and to which storage layer it needs to be migrated.
[0101] The present invention can understand the behavior patterns and laws of data access by obtaining historical data access logs and extracting access pattern features; the access frequency features can indicate the frequency of data access and help identify hot data and cold data; the time distribution features can reveal the distribution of data access in different time periods, which helps to determine peak and trough periods; the access timing features can describe the time sequence of data access and help understand the evolution and trend of data access; the access source features can indicate the source of data access, such as different users, applications or system components; the associated data access features can capture the correlation between data and are used to identify data sets with similar access patterns. Through the K-Means clustering algorithm, the access pattern feature data can be divided into different clustering groups, each group representing a data access pattern; cluster analysis can help identify and summarize common patterns of data access and provide an overall understanding of data access behavior; different access patterns can be used to guide the formulation of data migration strategies to meet data access requirements under different patterns. According to the migration rule data and access mode data, the data migration trigger conditions can be set, that is, when to trigger the data migration operation; the migration trigger condition parameter set includes parameters set according to specific circumstances, such as data access frequency threshold, data access time distribution law, data access peak value, etc.; by setting appropriate migration trigger conditions, an automated data migration strategy can be implemented to improve the efficiency and performance of the data storage level. Based on the migration trigger condition parameter set, a data migration strategy model can be established, which automatically determines when to migrate data from one storage layer to another; the data migration strategy model can consider factors such as data access mode, migration rules, and storage cost, comprehensively weigh and automatically decide on data migration.
[0102] Preferably, step S3 comprises the following steps:
[0103] Step S31: Analyze the source of legal knowledge service data and identify the characteristics of different data sources, thereby generating a data source characteristic report;
[0104] The embodiment of the present invention analyzes different sources of legal knowledge service data, such as legal documents, legal provisions, legal cases, and regulatory databases, and determines the type and characteristics of the data source for subsequent feature identification and data processing. For different data sources, their characteristics are identified, including but not limited to the following aspects: analyzing the organizational structure of the data, such as tables, texts, relational databases, etc., determining the format of the data, such as JSON, XML, CSV, etc., identifying key fields in the data, such as case numbers, parties, and judgment results in legal documents, evaluating the quality and accuracy of the data, identifying possible errors or omissions, understanding the meaning and context of the data, and identifying legal terms, concepts, and relationships. The feature identification results for each data source are sorted and summarized to generate a data source feature report that describes the characteristics and attributes of each data source.
[0105] Step S32: Design a data extraction strategy based on the data source feature report, formulate data cleaning rules, and design a data loading solution to generate ETL process design data;
[0106] The embodiment of the present invention formulates a data extraction strategy based on the data source feature report, determines which data to extract from each data source and how to extract the data, and considers factors such as data availability, data volume, and data update frequency. According to data quality assessment and data cleaning requirements, formulate data cleaning rules, such as removing duplicate data, processing missing values, correcting erroneous data, and normalizing data formats. Design a data loading plan to determine how to load the cleaned data into the target system or database, considering the performance, security, and scalability of data loading. Based on the data extraction strategy, data cleaning rules, and data loading plan, generate ETL (extraction, transformation, loading) process design data to describe the steps and processes of data processing.
[0107] Step S33: Establish a standardized dictionary of legal terms and concepts based on the ETL process design data, and perform a unified structure conversion based on field definitions and data types, thereby obtaining legal standardized data;
[0108] The embodiment of the present invention establishes a standardized dictionary of legal terms and concepts, defines and explains commonly used legal terms, and ensures the consistency and accuracy of data, for example, defines and explains various types of crimes in criminal law. According to the standardized dictionary and data structure definition, the legal knowledge service data is converted into a unified structure, the fields of different data sources are mapped to a unified data structure, and converted and normalized according to the data type. According to the results of the time dimension feature extraction based on timestamp and the construction of the version control system, a query interface is designed to enable users to query legal standardized data at specific moments or historical moments according to conditions such as time range and version number. According to the results of geographic information extraction and geographic spatial index structure design, a query interface is designed to enable users to query legal standardized data related to geographic information according to conditions such as geographic location and geographic range. In combination with time version control data and geographic index data, a query interface based on composite query is designed to enable users to conduct comprehensive queries according to time, geographic location and other conditions to obtain legal standardized data that meets multiple conditions.
[0109] Step S34: extracting time dimension features based on timestamps from legal standardization data, and constructing a version control system, thereby obtaining time version control data;
[0110] The embodiment of the present invention extracts the time dimension feature of legal standardization data, that is, extracts time-related information from the data; for example, extracts the timestamp fields such as the creation time, modification time, and effective time of the data, and formats and standardizes them. Based on the results of the time dimension feature extraction, a version control system is constructed to manage different versions and historical records of legal standardization data; the version control system can track the change history of data, including operations such as adding data, modifying data, and deleting data.
[0111] Step S35: extracting geographic information from legal standardization data and designing a geographic spatial index structure to obtain geographic index data;
[0112] The embodiment of the present invention extracts geographic information from legal standardized data, that is, identifies and extracts information related to geographic location from the data, for example, extracts geographic fields such as the address of the parties and the location of the case, and performs geocoding and geographic data cleaning. According to the characteristics of geographic information and query requirements, a suitable geospatial index structure is designed to support efficient geospatial queries; commonly used geospatial index structures include R-trees, quadtrees, etc., which are used to organize and manage the spatial relationships of geographic data.
[0113] Step S36: Design a query interface based on compound query according to the time version control data and the geographic index data, so as to obtain the version control data.
[0114] The embodiment of the present invention combines the characteristics of time version control data and the version control system to design a query interface, allowing users to query legal standardization data at specific moments or historical moments based on conditions such as time range and version number. The interface must support functions such as time filtering and version selection. Combined with geographic information extraction and geographic spatial index structure, a query interface is designed to enable users to query legal standardization data related to geographic information based on conditions such as geographic location and geographic range. The interface must support functions such as geographic location filtering and spatial relationship query. Combined with time version control data and geographic index data, a query interface based on composite query is designed to enable users to conduct comprehensive queries based on time, geographic location and other conditions to obtain legal standardization data that meets multiple conditions. The interface must support multi-condition combination query and result filtering.
[0115] Through source analysis, the present invention can determine the various data sources of legal knowledge service data and understand its authority, reliability, scope of application and other characteristics; feature identification of different data sources can identify the structure, format, data type, data quality and other characteristics of the data source, and help subsequent data processing and analysis; generating a data source feature report can provide a comprehensive overview of the data source characteristics, providing a reference and basis for subsequent steps. According to the data source feature report, a suitable data extraction strategy can be designed to determine the method and frequency of extracting data from each data source; formulating data cleaning rules can clean up and deal with noise, duplicates, missing values and other problems in the data, and improve the quality and consistency of the data; designing a data loading solution can determine the method and process of loading data into the legal knowledge service system, ensuring the effective import and update of data; ETL (Extract, Transform, Load) process design data describes the entire process from data extraction to data loading, providing a basis for subsequent data processing and analysis. Establishing a standardized dictionary of legal terms and concepts can unify the expression of legal terms and concepts in the data and improve the consistency and comprehensibility of the data; performing a unified structural conversion based on field definitions and data types can map and convert fields from different data sources to make them conform to a unified data structure and specification; after obtaining legal standardized data, the legal information in the data will be easier to manage, search and understand, providing a basis for subsequent data analysis and application. Time dimension feature extraction based on timestamps can extract and record time-related information of the data, such as creation time, modification time and expiration time, etc., to provide support for time analysis and backtracking of data; building a version control system can track and manage different versions of legal standardized data, including historical versions and current versions, to ensure the traceability and consistency of data. Time version control data can be used for historical query, audit and backtracking analysis of legal data. Geographic information extraction can extract information related to geographic location from legal standardized data, such as the location of the court, the location of the case, etc., to provide support for geographic spatial analysis and query. Designing a geographic spatial index structure can optimize the storage and query efficiency of geographic data and improve the retrieval and analysis performance of geographic data; geographic index data can be used for query operations related to geographic location, such as location-based case retrieval, geographic distribution statistics, etc. Based on time version control data and geographic index data, the query interface can be designed to support complex query requirements based on time and geographic location; the query interface design can include functions such as filtering, sorting and aggregation for time and geographic dimensions, providing flexible query operations; the query interface for version control data can help users query legal data versions within a specific time period, as well as retrieve legal information based on geographic location.
[0116] Preferably, step S4 comprises the following steps:
[0117] Step S41: extracting difference data from legal standardization data according to version control data, thereby obtaining newly added data to be stored;
[0118] The embodiment of the present invention needs to compare the differences between different versions based on version control data, and extract the newly added data to be stored in the database; use the version control system (such as Git) to obtain different versions of legal standardization data, compare the data differences between adjacent versions, and identify the newly added data records; extract the newly added data records and save them as data to be stored in the database.
[0119] Step S42: Designing an incremental storage solution for the newly added data to be stored based on the historical access pattern, thereby obtaining an incremental storage solution, wherein the incremental storage solution includes a storage structure and an update strategy;
[0120] The embodiment of the present invention needs to design an incremental storage solution based on historical access patterns to effectively store and update the newly added data to be stored in the warehouse; analyze the historical access patterns and data access patterns to understand the frequency and method of access to the newly added data; design a suitable storage structure, such as database tables, indexes, etc., according to the characteristics and access patterns of the data to support efficient data access and update; define the update strategy, including how to handle the insertion, update and deletion operations of the newly added data, and the method of data version management.
[0121] Step S43: writing new data to be stored in the version control data according to the incremental storage solution, updating the version control data and adjusting the version tree structure, thereby obtaining version management model data;
[0122] The embodiment of the present invention needs to write the newly added data to be stored into the version control data according to the incremental storage scheme, and update the version control data and adjust the version tree structure. According to the update strategy defined in the incremental storage scheme, the newly added data to be stored is written to the corresponding position in the version control data; the information of the relevant data records in the version control data is updated, such as the update timestamp, version number, etc.; the version tree structure is adjusted to reflect the insertion of the newly added data and the change of the version relationship.
[0123] Step S44: performing storage level evaluation on the newly added data to be stored in the warehouse according to the automatic data migration strategy, thereby obtaining storage level evaluation data;
[0124] The embodiment of the present invention needs to evaluate the storage requirements of the newly added data to be stored, and perform storage level evaluation according to the automatic data migration strategy. Analyze the properties and size of the newly added data to be stored, and evaluate its storage requirements for different storage levels; according to the automatic data migration strategy, consider the performance, cost and capacity of the storage levels, and perform storage level evaluation on the newly added data.
[0125] Step S45: Perform storage level allocation according to the storage level evaluation data and the current resource usage of each storage layer, thereby obtaining initial storage allocation data.
[0126] The embodiment of the present invention needs to allocate storage levels according to the storage level evaluation data and the current usage of each storage level resource, and determine the initial storage location of the newly added data. According to the storage level evaluation data, determine the storage capacity and performance requirements of each storage level; analyze the current resource usage of each storage level, including the used storage capacity, performance indicators, etc.; according to the evaluation data and resource usage, allocate storage levels, determine the initial storage location of the newly added data, and record the initial storage allocation data.
[0127] The present invention can identify the new content in the legal standardization data, that is, the new data compared with the previous version, by comparing the difference of the version control data; extracting the new data to be stored can extract these new contents separately to prepare for the subsequent incremental storage and update operations. The incremental storage scheme based on the historical access mode can design a suitable storage structure and update strategy according to the user's access mode to the legal standardization data; the design of the storage structure can optimize the storage method of the data, improve the storage efficiency and query performance, and the design of the update strategy can effectively manage the update and maintenance of the new data to be stored, and ensure the consistency and availability of the data. According to the incremental storage scheme, the new data to be stored is written into the version control data to ensure the correct storage and association of the new data; updating the version control data can record the change information of the new data, maintain the integrity and accuracy of the version management, and adjust the version tree structure according to the relationship and dependency of the new data, update the structure of the version tree, and provide support for subsequent version management and query. The automatic data migration strategy can evaluate the storage requirements of the new data to be stored at different storage levels according to the importance and access frequency of the data; the storage level evaluation data provides a quantitative evaluation of the storage requirements of the new data, and provides a basis for the subsequent storage level allocation. Based on the storage level assessment data and the current storage layer resource usage, storage level allocation can reasonably allocate storage resources for new data to meet the needs of data storage and access; the initial storage allocation data determines the storage location and allocation ratio of new data at each storage level, providing a basis for subsequent storage management and expansion.
[0128] Preferably, step S5 comprises the following steps:
[0129] Step S51: Designing an integrated architecture of a distributed document database, an object storage system, and a full-text search engine based on the legal standardization data and the initial storage allocation data, thereby obtaining a comprehensive storage architecture solution;
[0130] The embodiment of the present invention needs to design a comprehensive storage architecture solution based on legal standardized data and initial storage allocation data, including the integration of distributed document database, object storage system and full-text search engine. Analyze the characteristics, structure and access requirements of legal standardized data, as well as the storage level allocation of initial storage allocation data; select appropriate storage technologies and tools such as distributed document database, object storage system and full-text search engine according to the requirements and analysis results; design a comprehensive storage architecture, determine the functions of each storage component and the integration method between them, such as data storage, retrieval and indexing; define the data flow rules and data synchronization mechanism between each storage component to ensure data consistency and integrity.
[0131] Step S52: Load the legal standardization data into the storage system according to the comprehensive storage architecture solution, thereby obtaining a data loading report;
[0132] The embodiment of the present invention needs to load the legal standardization data into the corresponding storage system according to the comprehensive storage architecture solution and generate a data loading report. According to the comprehensive storage architecture solution, use the corresponding tools and methods to import the legal standardization data into the distributed document database and object storage system; monitor the data loading process to ensure the correctness and integrity of the data; generate a data loading report, including the amount of data successfully loaded, loading time, error records and other information for subsequent verification and analysis.
[0133] Step S53: Integrate the version management model into the distributed document database and object storage system described in step S51, thereby generating version management integration test data;
[0134] The embodiment of the present invention needs to integrate the version management model into the distributed document database and object storage system designed in step S51 to generate version management integration test data. According to the characteristics and requirements of the version management model, the structure and function of the distributed document database and object storage system are modified or expanded to support the needs of version management; the data structure of the version management model is mapped with the distributed document database and object storage system to ensure that the version management data can be correctly stored and retrieved; for each function of version management, corresponding integration test data is designed, including operations such as version creation, version comparison, and version merging.
[0135] Step S54: Intelligent indexing is performed according to the version management integrated test data through a preset semantic analysis engine based on natural language processing, thereby obtaining intelligent indexing data to realize intelligent storage of data.
[0136] The embodiment of the present invention needs to use a preset semantic analysis engine based on natural language processing to perform intelligent indexing according to the version management integrated test data, so as to obtain intelligent indexing data to realize intelligent storage of data. Prepare a preset semantic analysis engine based on natural language processing, which can understand the semantic information in the text and extract keywords and topics; use the version management integrated test data as input, and process and analyze the data text through the semantic analysis engine; the semantic analysis engine will extract keywords and topics according to the semantic information of the text, and use them as intelligent indexing data; the intelligent indexing data can be used for intelligent storage of data, such as retrieval, classification and recommendation based on keywords or topics.
[0137] The present invention can reasonably integrate legal standardization data into distributed document databases, object storage systems and full-text search engines according to the requirements of initial storage allocation data through the design of a comprehensive storage architecture solution; the comprehensive storage architecture solution takes into account the characteristics, access patterns and storage requirements of the data, optimizes the storage structure and data distribution, and improves the storage efficiency and query performance of the data. The legal standardization data is loaded into the storage system according to the requirements of the comprehensive storage architecture solution to ensure that the data is correctly stored and meets the storage requirements; the data loading report provides detailed information on the loading process, including the amount of data successfully loaded, the amount of data failed to load, etc., which is used to verify the accuracy and completeness of the data loading. The version management model is integrated into the distributed document database and object storage system to realize the version management function of the legal standardization data; the version management integration test data is generated by applying the version management model to the data in the storage system, which is used to verify the correctness and effectiveness of the version management function. The version management integration test data is intelligently indexed and formulated using a semantic analysis engine based on natural language processing, and keywords and tags can be automatically extracted according to the semantic and content characteristics of the data; the intelligent indexing data contains keyword and tag information processed by the semantic analysis engine, which provides support for intelligent storage and intelligent search of data and improves the efficiency of data discovery and utilization. Combining the results of the above steps, the following goals can be achieved: design a comprehensive storage architecture solution, reasonably integrate legal standardization data into the storage system, improve storage efficiency and query performance; load legal standardization data into the storage system to ensure correct storage and integrity of the data; integrate the version management model into the storage system to realize the version management function of the data; use the semantic analysis engine for intelligent indexing to improve the intelligent storage and search capabilities of the data.
[0138] The present invention also provides a data storage system based on a legal knowledge service platform, which is used to execute the data storage method based on a legal knowledge service platform as described above. The data storage system based on a legal knowledge service platform includes:
[0139] The storage architecture module is used to obtain legal knowledge service data; extract data features based on data type, usage frequency, importance and timeliness of legal knowledge service data, and perform cluster analysis to obtain classification result data; and build an adaptive multi-level storage architecture model based on the classification result data;
[0140] The multi-layer storage strategy module is used to determine the storage schemes for the hot data layer, warm data layer, and cold data layer according to the multi-level storage architecture model, and to establish a data migration strategy model based on historical access patterns, thereby obtaining automatic data migration strategies between different layers;
[0141] The version control module is used to establish a standardized mapping of legal knowledge service data to reduce semantic differences through ETL technology, thereby obtaining legal standardized data; perform version control based on time dimension management and query control based on geographic spatial index on legal standardized data, thereby obtaining version controlled data;
[0142] The incremental level allocation storage module is used to store new data in the legal standardization data according to the version control data through the incremental storage technology, so as to obtain the version management model data; the storage level allocation is performed on the newly stored data according to the automatic data migration strategy, so as to obtain the initial storage allocation data;
[0143] The intelligent indexing module is used to build an unstructured data management platform based on distributed document databases, object storage systems and full-text search engines according to legal standardization data and initial storage allocation data, and to perform semantic-based intelligent indexing based on version management model data to obtain intelligent indexing data to achieve intelligent storage of data.
[0144] The storage architecture module of the present invention can classify and cluster the data according to the characteristics of the legal knowledge service data, such as data type, frequency of use, importance and timeliness; this helps to extract the characteristics and requirements of the data and provide a basis for subsequent storage and management. The construction of an adaptive multi-level storage architecture model can allocate data to different storage levels according to the classification result data. This can reasonably allocate storage resources according to the access frequency and importance of the data and improve the access efficiency of the data. The multi-layer storage strategy module determines the storage scheme of the hot data layer, the warm data layer and the cold data layer according to the multi-level storage architecture model, so that the data can be stored in the most suitable storage layer according to the access mode and importance of the data, thereby improving the utilization efficiency of storage resources. The data migration strategy model based on the historical access pattern can automatically determine the time and method of data migration from one level to another according to the access mode and trend of the data, which helps to realize the dynamic migration of data and optimize the utilization of storage resources. The version control module uses the ETL technology to establish a standardized mapping of the legal knowledge service data to reduce semantic differences, thereby obtaining legal standardized data, which helps to ensure the consistency and standardization of the data and improve the quality and credibility of the data. Version control based on time dimension management and query control based on geospatial index make the management and retrieval of legal standardization data more flexible and accurate, and can conveniently manage historical versions of data, while supporting query operations based on time and geographic location. The incremental hierarchical allocation storage module realizes the storage of new legal standardization data through incremental storage technology, which helps to merge the newly generated data with the existing data to maintain the integrity and consistency of the data. According to the automatic data migration strategy, the storage level of the newly stored data can be allocated, and the new data can be stored in the most appropriate level according to the characteristics and requirements of the data, which helps to optimize the utilization of storage resources and the access efficiency of data. The intelligent indexing module builds an unstructured data management platform based on legal standardization data and initial storage allocation data. Such a platform can support technologies such as distributed document databases, object storage systems and full-text search engines, and provide efficient data management and query functions. Semantic-based intelligent indexing based on version management model data can better understand and organize data; through intelligent indexing, the semantic information of the data can be extracted, indexes can be established, and more accurate and intelligent data search and retrieval can be achieved.
[0145] Preferably, the present invention also provides a storage medium for data storage based on a legal knowledge service platform, storing a computer program, and when the computer program is executed, it implements the above-mentioned data storage method based on a legal knowledge service platform.
[0146] Therefore, the embodiments should be regarded as illustrative and non-restrictive from all points, and the scope of the present invention is limited by the appended claims rather than the above description, and it is therefore intended that all changes falling within the meaning and range of equivalent elements of the application documents are included in the present invention.
[0147] The above description is only a specific embodiment of the present invention, so that those skilled in the art can understand or implement the present invention. Various modifications to these embodiments will be apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present invention. Therefore, the present invention will not be limited to the embodiments shown herein, but should conform to the widest scope consistent with the principles and novel features invented herein.
Claims
1. A data storage method based on a legal knowledge service platform, characterized in that: The following steps are involved: Step S1: Obtain legal knowledge service data; Extract data features of legal knowledge service data based on data type, usage frequency, importance and timeliness, and perform cluster analysis to obtain classification result data; Build an adaptive multi-level storage architecture model based on the classification result data; Step S2: Determine the storage solutions for the hot data layer, the warm data layer, and the cold data layer according to the multi-level storage architecture model, and establish a data migration strategy model based on historical access patterns, so as to obtain automatic data migration strategies between different layers; Step S3: Establish standardized mapping of legal knowledge service data to reduce semantic differences through ETL technology, thereby obtaining legal standardized data; perform version control based on time dimension management and query control based on geographic spatial index on legal standardized data, thereby obtaining version controlled data; Step S4: New data is stored in the legal standardization data according to the version control data through the incremental storage technology, thereby obtaining version management model data; storage level allocation is performed on the newly stored data according to the automatic data migration strategy, thereby obtaining initial storage allocation data; Step S5: Construct an unstructured data management platform based on a distributed document database, an object storage system, and a full-text search engine according to the legal standardization data and the initial storage allocation data, and perform semantic-based intelligent indexing according to the version management model data to obtain intelligent indexing data to realize intelligent storage of data.
2. The data storage method based on the legal knowledge service platform according to claim 1 is characterized in that: Step S1 includes the following steps: Step S11: Obtain legal knowledge service data, and perform deduplication and format unification on the legal knowledge service data, thereby obtaining legal cleansing data; Step S12: extracting data features based on data type, usage frequency, importance and timeliness from the legal cleansing data, thereby obtaining legal comprehensive feature data; Step S13: assign weights to different features of the legal comprehensive feature data, determine the number of clusters 3, and perform cluster analysis based on K-means, thereby generating classification result data in which each data object has a cluster label, wherein the cluster labels include hot data labels, warm data labels, and cold data labels; Step S14: Determine the number of storage levels and level characteristics based on the classification result data, so as to construct an adaptive multi-level storage architecture model.
3. The data storage method based on the legal knowledge service platform according to claim 2 is characterized in that: Step S12 includes the following steps: Step S121: extracting features of the legal cleansing data based on the degree of structure and text length, thereby obtaining type feature data; Step S122: extracting frequency characteristics of the number of accesses and the most recent access time according to the legal cleansing data, thereby obtaining frequency characteristic data; Step S123: extracting the number of citations and the legal effectiveness level of the legal cleansing data, and assigning weights to the number of citations and the legal effectiveness level, thereby generating importance feature data; Step S124: extracting timeliness features from the legal cleansing data through the time difference between the effective time and the expiration time, thereby obtaining timeliness feature data; Step S125: combining the type feature data, the frequency feature data, the importance feature data and the timeliness feature data into comprehensive feature data.
4. The data storage method based on the legal knowledge service platform according to claim 3 is characterized in that: Step S2 includes the following steps: Step S21: using the data with hot data labels in the classification result data as the hot data layer of the multi-level storage architecture model, and storing the hot data layer through the memory database, thereby obtaining a hot data layer storage solution; Step S22: using the data with warm data labels in the classification result data as the warm data layer of the multi-level storage architecture model, and storing them in a relational database mainly based on solid state drives, thereby obtaining a warm data layer storage solution; Step S23: using the data with cold data layer labels in the classification result data as the cold data layer of the multi-level storage architecture model, and storing the cold data layer through a distributed file system, thereby obtaining a cold data layer storage solution; Step S24: integrating the hot data layer storage solution, the warm data layer storage solution, and the cold data layer storage solution, thereby obtaining comprehensive storage solution data of the multi-level storage architecture model; Step S25: formulating initial data allocation rules between different storage layers based on the classification result data and the comprehensive storage solution data, and defining trigger conditions for data migration from one storage layer to another, thereby obtaining migration rule data; Step S26: Establish a data migration strategy model based on historical access patterns according to the migration rule data and the comprehensive storage solution data, so as to obtain automatic data migration strategies between different levels.
5. The data storage method based on the legal knowledge service platform according to claim 4 is characterized in that: Step S26 includes the following steps: Step S261: Obtain historical data access logs; extract access pattern features from the historical data access logs to obtain access pattern feature data, wherein the access pattern features include access frequency features, time distribution features, access timing features, access source features, and associated data access features; Step S262: Perform pattern clustering analysis on the historical data access logs according to the access pattern feature data by using the K-Means clustering algorithm, thereby obtaining access pattern data, wherein the clustering results represent different data access patterns, including high-frequency short-term access patterns, periodic regular access patterns, peak-period concentrated access patterns, long-term stable access patterns, and low-frequency occasional access patterns; Step S263: setting data migration trigger conditions according to the migration rule data and the access mode data, thereby obtaining a migration trigger condition parameter set; Step S264: establishing a data migration strategy model according to the migration trigger condition parameter set, thereby obtaining automatic data migration strategies between different levels.
6. The data storage method based on the legal knowledge service platform according to claim 5 is characterized in that: Step S3 includes the following steps: Step S31: Analyze the source of legal knowledge service data and identify the characteristics of different data sources, thereby generating a data source characteristic report; Step S32: Design a data extraction strategy based on the data source feature report, formulate data cleaning rules, and design a data loading solution to generate ETL process design data; Step S33: Establish a standardized dictionary of legal terms and concepts based on the ETL process design data, and perform a unified structure conversion based on field definitions and data types, thereby obtaining legal standardized data; Step S34: extracting time dimension features based on timestamps from legal standardization data, and constructing a version control system, thereby obtaining time version control data; Step S35: extracting geographic information from legal standardization data and designing a geographic spatial index structure to obtain geographic index data; Step S36: Design a query interface based on compound query according to the time version control data and the geographic index data, so as to obtain the version control data.
7. The data storage method based on the legal knowledge service platform according to claim 6 is characterized in that: Step S4 includes the following steps: Step S41: extracting difference data from legal standardization data according to version control data, thereby obtaining newly added data to be stored; Step S42: Designing an incremental storage solution for the newly added data to be stored based on the historical access pattern, thereby obtaining an incremental storage solution, wherein the incremental storage solution includes a storage structure and an update strategy; Step S43: writing new data to be stored in the version control data according to the incremental storage solution, updating the version control data and adjusting the version tree structure, thereby obtaining version management model data; Step S44: performing storage level evaluation on the newly added data to be stored in the warehouse according to the automatic data migration strategy, thereby obtaining storage level evaluation data; Step S45: Perform storage level allocation according to the storage level evaluation data and the current resource usage of each storage layer, thereby obtaining initial storage allocation data.
8. The data storage method based on the legal knowledge service platform according to claim 7 is characterized in that: Step S5 includes the following steps: Step S51: Designing an integrated architecture of a distributed document database, an object storage system, and a full-text search engine based on the legal standardization data and the initial storage allocation data, thereby obtaining a comprehensive storage architecture solution; Step S52: Load the legal standardization data into the storage system according to the comprehensive storage architecture solution, thereby obtaining a data loading report; Step S53: Integrate the version management model into the distributed document database and object storage system described in step S51, thereby generating version management integration test data; Step S54: Intelligent indexing is performed according to the version management integrated test data through a preset semantic analysis engine based on natural language processing, thereby obtaining intelligent indexing data to realize intelligent storage of data.
9. A data storage system based on a legal knowledge service platform, characterized in that: Used to execute the data storage method based on the legal knowledge service platform according to claim 1, the data storage system based on the legal knowledge service platform comprises: The storage architecture module is used to obtain legal knowledge service data; extract data features based on data type, usage frequency, importance and timeliness of legal knowledge service data, and perform cluster analysis to obtain classification result data; and build an adaptive multi-level storage architecture model based on the classification result data; The multi-layer storage strategy module is used to determine the storage schemes for the hot data layer, warm data layer, and cold data layer according to the multi-level storage architecture model, and to establish a data migration strategy model based on historical access patterns, thereby obtaining automatic data migration strategies between different layers; The version control module is used to establish a standardized mapping of legal knowledge service data to reduce semantic differences through ETL technology, thereby obtaining legal standardized data; perform version control based on time dimension management and query control based on geographic spatial index on legal standardized data, thereby obtaining version controlled data; The incremental level allocation storage module is used to store new data in the legal standardization data according to the version control data through the incremental storage technology, so as to obtain the version management model data; the storage level allocation is performed on the newly stored data according to the automatic data migration strategy, so as to obtain the initial storage allocation data; The intelligent indexing module is used to build an unstructured data management platform based on distributed document databases, object storage systems and full-text search engines according to legal standardization data and initial storage allocation data, and to perform semantic-based intelligent indexing based on version management model data to obtain intelligent indexing data to achieve intelligent storage of data.
10. A data storage medium based on a legal knowledge service platform, storing a computer program, characterized in that: When the computer program is executed, it implements a data storage method based on a legal knowledge service platform as described in any one of claims 1 to 8.
Citation Information
Patent Citations
Distributed multi-level storage system and method based on HDFS
CN109284258A
Data migration method and device, equipment and storage medium
CN116974470A
Intelligent archiving system for legal documents
CN117407360A
Data management method and system in cloud native environment and storage medium
CN117874141A
Construction method and system of intelligent data medium station
CN118152378A
Cited By
Intelligent management method and system based on domestic data packet
CN119271684A
Multi-level quality file management method based on big data
CN119537324A
Method and system for hierarchical storage of block chain data in urban update field
CN119537382A
A method and system for hierarchical storage of blockchain data in the field of urban renewal
CN119537382B
Network service method and device and medium
CN119583657A
