Lossless compression storage method and system based on heterogeneous data under xinxin environment

Abstract

The application discloses a lossless compression storage method and system based on heterogeneous data in a Xinhua environment, first extracts the file type and structure feature mapping relationship by scanning the target file, classifies and groups, and then carries out preliminary lossless compression, further adopts a sliding window matching and sequence matching algorithm to optimize the compressed data for the redundant mode, then generates a metadata structure containing a check code and compression parameters, encapsulates into a data packet supporting cross-platform transmission, finally transmits to the target storage device through a secure write protocol, and verifies the data consistency and recovery path feasibility by using the check code. The application significantly improves the data compression efficiency, ensures the compatibility of cross-platform transmission and the integrity of storage data, and provides efficient and reliable technical support for complex electronic file processing.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of data storage, and particularly discloses a lossless compression storage method and system for heterogeneous data based on a China-specific environment. BACKGROUND

[0002] Under the background of rapid development of information technology, data storage technology, as an important cornerstone supporting digital transformation, is self-evident in its importance. Especially in the information environment constructed with domestic software and hardware as the core, the storage and management of massive data are directly related to the running efficiency and economic cost of the system. How to reduce the storage space occupation as much as possible while ensuring data integrity has become a key field that needs to be tackled.

[0003] Research in this field not only affects the operating costs of enterprises, but also puts higher requirements on information security and system stability. However, some existing storage compression methods often fail to fully exert their effectiveness in the context of domestic information environment. These methods are mostly not adapted to the characteristics of domestic processor architecture and operating system, resulting in limited compression efficiency and running speed when dealing with complex data.

[0004] The deeper problem is that some solutions may ignore the complete protection of the original information of the data when pursuing compression effects, or lack effective verification mechanisms during the long-term storage and recovery of compressed data, thereby burying security risks. Focusing on technical difficulties, the core problem is how to design an algorithm that can guarantee data without any loss and significantly improve compression effects in the context of domestic software and hardware environment. Especially for the storage of electronic files, the types and structures of files are different, and the adaptability of compression algorithms to the characteristics of different files puts high requirements. If the content and format characteristics of the files cannot be processed specifically, the compression process may result in resource waste, and even affect the accurate recovery of subsequent data. This unsolved technical factor directly leads to the difficulty of achieving ideal storage space utilization in practical applications, and also increases the complexity of data management. Taking a specific business scenario as an example, in an archive management system using domestic processors, facing the storage demand of a large number of electronic documents, if the compression algorithm cannot adapt to the file types, such as using the same processing method for text documents and image documents, it may result in poor compression effect of text documents and loss of some details after recovery of image documents. This situation not only increases the burden of storage devices, but also poses risks to the long-term preservation and invocation of data.

[0005] Therefore, how to design a storage method that can efficiently compress and ensure complete data recovery in the context of domestic information environment, according to the diversity characteristics of electronic files, has become a key problem that needs to be solved. SUMMARY

[0006] The application provides a lossless compression storage method and system based on heterogeneous data in a Xinxing environment, aiming to solve at least one of the defects in the prior art.

[0007] An aspect of the application relates to a lossless compression storage method based on heterogeneous data in a Xinxing environment, comprising the following steps:

[0008] S100, by scanning the target electronic file, extracting the file type identifier and the internal data structure mode from the target electronic file, obtaining the mapping relationship of file type and structure characteristics, wherein the mapping relationship includes file format standard and data block distribution characteristics;

[0009] S200, according to the mapping relationship, classifying and grouping the file type and structure characteristics, using a lossless encoding method to perform preliminary compression on the repetitive mode part, determining the data block sequence after preliminary compression, and classifying and grouping based on file content similarity;

[0010] S300, if there is a redundant mode in the data block sequence after preliminary compression, the similarity threshold between the data block sequences is compared to determine whether the data block sequence after preliminary compression is redundant, and the fusion information processing link is involved in the data mode matching optimization, and then the sequence matching algorithm is applied to the sliding window matching to obtain the optimized compression data;

[0011] S400, for the optimized compression data, generating a metadata structure containing data check code and compression parameters, merging and packaging the optimized compression data and the metadata structure to form a packaged data packet, and the packaged data packet supports cross-platform transmission;

[0012] S500, parse the storage path information from the packaged data packet, use the safe and compatible platform write protocol to transmit the packaged data packet to the target storage device, and get the write confirmation signal;

[0013] S600, according to the write confirmation signal, query the integrity of the packaged data packet in the storage device, verify the data consistency in the fusion information processing link through the check code comparison, judge the feasibility of the recovery path, and obtain the final lossless storage record.

[0014] Further, step S100 comprises:

[0015] S110, by scanning the target electronic file, obtaining the header identification information inside the target electronic file, extracting the file type identifier and the corresponding format standard from the header identification information, comparing with the preset rule library, and obtaining the preliminary classification result of the file type;

[0016] S120, according to the preliminary classification result, the internal data structure mode of each file type is parsed, the distribution characteristics and storage mode of the data block are obtained, and the mapping relationship between the preliminary file type and the structure characteristics is determined by comparing the preset data block template;

[0017] S130, if the structure characteristics do not match the preset template, a byte stream analysis tool is used to segmentally disassemble the data block of the target electronic file, hidden format identifiers and data distribution rules are extracted therefrom, it is judged whether it conforms to the known extended format standard, and a corrected mapping relationship is obtained;

[0018] S140, by integrating the corrected mapping relationship and the preliminary classification result, a data comparison tool is used to finally verify the file type and the structure characteristics, and the corresponding relationship between the complete file format standard and the data block distribution characteristics is obtained.

[0019] Further, step S200 includes:

[0020] S210, according to the mapping relationship, the file type and the structure characteristics are classified and grouped, a content similarity comparison tool is used to extract feature data from the target electronic file, an initial result of classification and grouping is determined by comparing a preset threshold range;

[0021] S220, according to the initial result of classification and grouping, a lossless encoding tool is used to scan the repeated mode part in the group, the distribution rule of the repeated data block is obtained, and a preliminary compressed data block sequence is obtained.

[0022] Further, step S300 includes:

[0023] S310, according to the preliminary compressed data block sequence, an information comparison tool is used to scan the data block distribution, and the distribution rule of the redundant mode is obtained;

[0024] S320, if the distribution rule of the redundant mode exceeds the preset threshold range, a sequence matching tool is used to segmentally compare the data block sequence, and the data segment with high similarity is determined;

[0025] S330, for the data segment with high similarity, a sliding window tool is used for one-by-one matching, and the data interval that can be further compressed is determined;

[0026] S340, the further compressed data interval is integrated by a fusion processing tool, a sequence optimization method is applied to recombine the data block, and the optimized compressed data is obtained.

[0027] Further, step S400 includes:

[0028] S410, according to the optimized compressed data, the corresponding data check code is generated by using the check tool, the compression parameter is obtained, and the mapping relationship between the data check code and the compression parameter is determined;

[0029] S420, for the mapping relationship between the data check code and the compression parameter, the unified metadata structure is integrated by using the construction tool, if the format of the metadata structure does not reach the preset threshold, the adjustment tool is used for correction, and the standardized metadata structure is obtained;

[0030] S430, the standardized metadata structure is bound with the optimized compressed data, the initial encapsulation data packet is generated by using the encapsulation tool, whether it meets the cross-platform transmission requirement is judged, and the compatibility of the initial encapsulation data packet is obtained;

[0031] S440, if the compatibility of the initial encapsulation data packet does not meet the preset standard, the transmission protocol is adjusted by using the protocol adaptation tool, and the final encapsulation data packet is obtained.

[0032] Further, step S500 comprises:

[0033] S510, the storage path information is obtained from the encapsulation data packet, the format of the storage path information is checked by using the path analysis tool, if it does not meet the preset threshold, the format conversion tool is used for adjustment, and the standardized path data is obtained;

[0034] S520, according to the standardized path data, the transmission protocol matching is carried out for the interface characteristics of the target storage device by using the protocol adaptation tool, whether it meets the safety requirement is judged, and the adapted protocol configuration is obtained;

[0035] S530, according to the adapted protocol configuration, the encapsulation data packet is transmitted to the target storage device;

[0036] S540, the write confirmation feedback returned by the target storage device is obtained, the legality of the write confirmation feedback is judged by using the verification tool, and the write confirmation signal is obtained.

[0037] Further, step S600 comprises:

[0038] S610, the storage device identifier is obtained from the write confirmation signal, the data positioning is carried out for the target storage device by using the query tool, and the storage position of the encapsulation data packet is obtained;

[0039] S620, according to the storage position of the encapsulation data packet, the check code information of the encapsulation data packet is extracted by using the comparison tool, if the check code of the encapsulation data packet does not match the preset threshold, the integrity correction is carried out by using the repair tool, and the integrity of the data packet is determined;

[0040] S630, according to the complete state of the data packet, obtaining the comparison record in the fusion information processing link, using a verification tool to judge the data consistency, and obtaining a consistency verification result;

[0041] S640, according to the consistency verification result, using a path analysis tool to evaluate the feasibility of the recovery path, if the path meets the preset threshold, then generating a final lossless storage record.

[0042] Another aspect of the application relates to a lossless compression storage system based on heterogeneous data in a Xingchu environment, for executing the lossless compression storage method based on heterogeneous data in a Xingchu environment as described above, comprising:

[0043] The mapping relationship acquisition module is used for extracting the file type identifier and the internal data structure mode from the target electronic file by scanning the target electronic file, obtaining the mapping relationship of the file type and the structure characteristics, and the mapping relationship includes the file format standard and the data block distribution characteristics.

[0044] The data block sequence determination module is used for classifying and grouping the file type and the structure characteristics according to the mapping relationship, performing preliminary compression on the repeated mode part by using a lossless encoding method, determining the data block sequence after preliminary compression, and classifying and grouping based on the file content similarity.

[0045] The optimized compression data acquisition module is used for judging in the fusion information processing link by comparing the similarity threshold between the data block sequences if there is a redundant mode in the data block sequence after preliminary compression, wherein the fusion information processing link involves data mode matching optimization, and then applying a sequence matching algorithm to perform sliding window matching to obtain optimized compression data.

[0046] The encapsulated data packet forming module is used for generating a metadata structure containing a data check code and a compression parameter for the optimized compression data, merging and encapsulating the optimized compression data and the metadata structure to form an encapsulated data packet, and the encapsulated data packet supports cross-platform transmission.

[0047] The write confirmation signal acquisition module is used for parsing the storage path information from the encapsulated data packet, transmitting the encapsulated data packet to the target storage device by using a safe and compatible platform write protocol, and obtaining a write confirmation signal.

[0048] The lossless storage record acquisition module is used for querying the integrity of the encapsulated data packet in the storage device according to the write confirmation signal, verifying the data consistency in the fusion information processing link by check code comparison, judging the feasibility of the recovery path, and obtaining a final lossless storage record.

[0049] The application has the following beneficial effects:

[0050] This invention provides a lossless compression and storage method and system for heterogeneous data in a domestically developed information technology environment. Addressing the comprehensive business challenges of data redundancy, cross-platform transmission compatibility, and storage integrity verification in electronic document processing, it constructs a complete solution by deeply integrating file type identifier extraction, data structure pattern analysis, and optimized compression techniques. First, the invention extracts the mapping relationship between file type and structural features by scanning target files. After classification and grouping, preliminary lossless compression is performed. For redundancy patterns, sliding window matching and sequence matching algorithms are further employed to optimize the compressed data. Subsequently, a metadata structure containing checksums and compression parameters is generated, encapsulated into a data packet supporting cross-platform transmission, and finally transmitted to the target storage device via a secure write protocol. The checksum is used to verify data consistency and recovery path feasibility. This invention significantly improves data compression efficiency, ensures cross-platform transmission compatibility and the integrity of stored data, and provides efficient and reliable technical support for complex electronic document processing. Attached Figure Description

[0051] Figure 1 This is a flowchart illustrating an embodiment of a lossless compression and storage method for heterogeneous data based on the information technology innovation environment of the present invention.

[0052] Figure 2 This is a functional block diagram of an embodiment of a lossless compression storage system for heterogeneous data based on the information technology innovation environment of the present invention.

[0053] Explanation of icon numbers:

[0054] 10. Mapping Relationship Acquisition Module; 20. Data Block Sequence Determination Module; 30. Optimized and Compressed Data Acquisition Module; 40. Encapsulated Data Packet Formation Module; 50. Write Confirmation Signal Acquisition Module; 60. Lossless Storage Record Acquisition Module. Detailed Implementation

[0055] To better understand the above technical solutions, the following will provide a detailed explanation of the technical solutions in conjunction with the accompanying drawings and specific implementation methods.

[0056] like Figure 1 As shown, the first embodiment of the present invention proposes a lossless compression and storage method for heterogeneous data based on the information technology innovation environment, including the following steps:

[0057] Step S100: By scanning the target electronic file, extract the file type identifier and internal data structure pattern from the target electronic file to obtain the mapping relationship between file type and structural features, wherein the mapping relationship includes file format standards and data block distribution characteristics.

[0058] The target electronic file (such as document, image, database file and other heterogeneous data) in the Xinchuang environment is taken as the processing object, the file header identification, field structure and data organization form are traversed through the file scanning tool, and two types of core information are extracted: one is the file type identification (such as file suffix, format signature, protocol identification, etc., which is used to distinguish the format category to which the file belongs), and the other is the internal data structure mode (such as data block division rule, field arrangement order, redundant data distribution position, etc., which is used to represent the organization logic of the internal data of the file). The two types of information are associated and matched to form a mapping relationship between the file type and the structure characteristics. The mapping relationship specifically includes the file format standard (corresponding to the official / industry format specification of various types of files, clearly specifying the syntax rules, field meaning and verification standard of data storage) and the data block distribution characteristics (corresponding to the division method, size proportion, storage location and association of data in the file), which provides accurate characteristic basis for subsequent targeted compression based on file type and structure.

[0059] In step S200, according to the mapping relationship, the file type and structure characteristics are classified and grouped, the lossless encoding method is used to perform preliminary compression on the repetitive mode part, and the data block sequence after preliminary compression is determined. The classification and grouping are based on the similarity of file content.

[0060] Based on the file type and structure characteristic mapping relationship established in step S100, the content similarity algorithm (such as cosine similarity, edit distance, etc.) is used to classify and group heterogeneous electronic files. The files with consistent file format standard, similar data block distribution characteristics and content repetition degree meeting the preset conditions are grouped into the same group, ensuring that the files in the same group have similar compression adaptability. For the internal data of each group of files, the mode recognition tool is used to locate the repeatedly appearing data segments (such as repeated paragraphs in documents, similar pixel blocks in images, repeated fields in databases, etc.), and the lossless encoding method (such as Huffman coding, LZ77 (Lempel-Ziv 1977 Algorithm, a lossless data compression algorithm based on sliding window) / LZ78 (Lempel-Ziv 1978 Algorithm, a self-adaptive dictionary encoding algorithm based on dictionary tree) variant, DEFLATE (a lossless data compression algorithm based on LZ77 algorithm and Huffman coding) algorithm, etc.) suitable for Xinchuang environment is selected to compress these repetitive mode parts, and the original structure and information of non-repetitive data are retained. After compression, the ordered data block sequence, i.e. the data block sequence after preliminary compression, is formed according to the data block division rule and grouping logic of the original file. This sequence not only eliminates part of the redundant data, but also completely retains the core information and recoverability of the original data.

[0061] Step S300, if there is a redundant mode in the preliminary compressed data block sequence, the similarity threshold between the data block sequences is compared to determine in the fusion information processing link, the fusion information processing link involves data mode matching optimization, and then a sequence matching algorithm is applied for sliding window matching to obtain optimized compressed data.

[0062] The preliminary compressed data block sequence obtained in step S200 is taken as a processing object, and a redundant mode detection tool is used to check the repeated data segments (i.e. redundant mode) in the sequence that are not excluded by preliminary compression. After entering the fusion information processing link, based on the data mode matching optimization logic built in the link, a preset similarity threshold (dynamically set according to the file type and structure feature mapping relationship) is called to quantitatively compare the similarity between the data block sequences. If the similarity between the sequences is higher than the threshold, it is determined that there is a redundant mode that needs to be further optimized. Then, a sequence matching algorithm (such as KMP (Knuth-Morris-Pratt Algorithm) algorithm, BM (Boyer-Moore) algorithm, etc.) is applied to accurately locate and match the redundant mode through a sliding window mechanism (the window size is adapted to the data block distribution characteristics) to eliminate the redundant representation of repeated data and only keep the core data and matching index information. Finally, the optimized compressed data with lower redundancy and higher storage efficiency is formed, and the data information is guaranteed not to be lost throughout the process, meeting the lossless storage requirements.

[0063] Step S400, for the optimized compressed data, a metadata structure containing data check code and compression parameters is generated, and the optimized compressed data and the metadata structure are merged and packaged to form a packaged data packet, and the packaged data packet supports cross-platform transmission.

[0064] Taking the optimized compression data obtained in step S300 as the core, first, a matching metadata structure is generated, which contains two types of key information, one is data check code (generated by using a check algorithm compatible with the ChinaSoft environment, such as CRC32 (Cyclic Redundancy Check 32-bit), SHA-256 (Secure Hash Algorithm 256-bit), etc., which is used for subsequent verification of data integrity), and the other is compression parameters (including lossless encoding type, sequence matching algorithm identifier, sliding window size, similarity threshold, etc., which are used for parameter reuse during data recovery). Subsequently, using the standardized packaging protocol compatible with the ChinaSoft environment (compatible with the interface of domestic operating systems, databases and storage devices), the optimized compression data and the metadata structure are merged and packaged to form a packaged data packet with unified structure and standardized format. The packaged data packet has built-in cross-platform adaptation fields, supports seamless transmission between ChinaSoft devices (such as x86, ARM (Advanced RISCMachine)) and operating systems of different architectures, and through the binding of metadata and core data, ensures that the data is traceable and verifiable during transmission.

[0065] Step S500, parse the storage path information from the packaged data packet, and use the secure compatible platform write protocol to transmit the packaged data packet to the target storage device to obtain a write confirmation signal.

[0066] Taking the packaged data packet formed in step S400 as the processing object, the built-in storage path information (including standardized path parameters such as target storage device address, partition directory, storage file name, and storage resource management specification compatible with the ChinaSoft environment) is extracted through a data packet analysis tool. Using a compatible platform write protocol (such as a domestic SMB3.0 adaptation protocol, an IPFS (InterPlanetary File System) ChinaSoft customized protocol, etc., supporting interface compatibility between domestic operating systems and storage devices) that meets the ChinaSoft security standards, data encryption (such as the SM4 (SM4 Block Cipher Algorithm) national encryption algorithm) and transmission verification mechanism are enabled during transmission to ensure the security and integrity of the data packet in the transmission link. The packaged data packet is transmitted to the specified target storage device (such as a domestic distributed storage cluster, a local ChinaSoft hard disk, etc.) according to the parsed storage path. After the device confirms that the data packet is completely received and successfully written, a write confirmation signal containing the write status and storage location check code is returned as the closed-loop feedback basis for data storage landing.

[0067] Step S600, according to the write confirmation signal, query the encapsulated data packet integrity in the storage device, verify the data consistency in the fusion information processing link through the check code comparison, judge the feasibility of the recovery path, and obtain the final lossless storage record.

[0068] The write confirmation signal obtained in step S500 is triggered, and the query instruction is called through the storage device interface to obtain the actual storage state (such as data block integrity, storage address matching degree, etc.) of the encapsulated data packet written in the target storage device. Enter the fusion information processing link, extract the preset check code of the metadata structure in the encapsulated data packet, and accurately compare it with the actual check code of the data packet in the storage device to verify whether the data is lost or tampered during transmission and writing, and ensure data consistency. At the same time, based on the compression parameters and file structure feature mapping relationship in the metadata, simulate the data recovery process to judge whether the recovery path is smooth (such as the feasibility of reverse analysis of compression algorithm, the integrity of data block sequence recombination, etc.). If the check code comparison is consistent and the recovery path is feasible, it means that the data has realized lossless storage, and the system automatically generates the final lossless storage record containing the storage location, check result, recovery parameter and timestamp, which is used as the core basis for subsequent data calling and recovery.

[0069] Further, the lossless compression storage method for heterogeneous data based on the signal creation environment proposed in the embodiment comprises steps S100:

[0070] Step S110, by scanning the target electronic file, obtaining the head identification information inside the target electronic file, extracting the file type identification and the corresponding format standard from the head identification information, comparing with the preset rule library to obtain the preliminary classification result of the file type.

[0071] The following formula is used to quantify the complexity and information content of the head identification information, which helps to determine the reliability of file type identification:

[0072] (1),

[0073] In formula (1), represents the head information entropy of the target electronic file, represents the total number of bytes of the head identification information, represents the probability of the th byte.

[0074] The following formula is used to obtain the preliminary classification result by weighted voting mechanism:

[0075] (2),

[0076] In formula (2), represents the final file type classification result, This indicates the total number of predefined file types. This indicates the total number of rules in the rule base. Indicates the first The confidence weight of each rule Representation rules Pointing to category The indicator function is 1 if the rule supports the category, and 0 otherwise.

[0077] When processing a target electronic file named "report.pdf", the first step is to read its initial byte sequence using a file scanning tool such as a Hex editor. The header identification information is usually located in the first few bytes of the file. For example, the header identification of a PDF file is "%PDF-1.4". The file type identifier "PDF" and the corresponding format standard, such as Adobe PDF specification version 1.4, are extracted from it. Then, this information is compared with a preset rule base. This rule base is a database containing signatures and standard descriptions of common file types. For example, the rule base stores the signature pattern of PDF and its correspondence with the ISO 32000 standard. After comparison using a string matching algorithm, the preliminary classification result is "Portable Document Format". This step ensures rapid identification of common files and avoids complex subsequent parsing.

[0078] S120. Based on the preliminary classification results, analyze the internal data structure pattern of each file type to obtain the distribution characteristics and storage method of data blocks. By comparing with the preset data block template, determine the preliminary mapping relationship between file type and structural characteristics.

[0079] The initial mapping relationship between file type and structural characteristics is derived using the following formula:

[0080] (3),

[0081] In formula (3), Indicate file type With structural features The strength of the mapping relationship between them This represents the total number of feature dimensions used for comparison. Represents the base of the natural constant. Representation type In the Feature values ​​in each dimension Representation structure In the Feature values ​​in each dimension This represents the mapping sensitivity parameter. The function represents the calculation of feature similarity.

[0082] Based on the above preliminary classification results, for PDF file type, further analyze its internal data structure mode, specifically, the data structure of PDF file includes objects, cross-reference table and page tree, etc., get the distribution characteristics of data block such as the order distribution of object number and storage mode such as indirect reference, by comparing the preset data block template, for example, the template defines the standard object structure mode of PDF, if matched, determine the mapping relationship of file type and structure characteristics such as "PDF type corresponds to hierarchical object storage".

[0083] S130, if the structure characteristics do not match the preset template, use byte stream analysis tool to segmentally disassemble the data block of the target electronic file, extract the hidden format identifier and data distribution rule therefrom, judge whether it conforms to the known extended format standard, and get the corrected mapping relationship.

[0084] The hidden format identifier extracted from the target electronic file is obtained by the following formula:

[0085] (4),

[0086] In formula (4), represents the hidden format identifier extracted from the target electronic file, represents the set of all possible format identifiers, represents the probability of format under the given hidden data condition, represents the degree of conformity of format to the known extended format standard, represents the format identifier corresponding to the maximum value of the subsequent expression.

[0087] The corrected mapping relationship is obtained by the following formula:

[0088] (5),

[0089] In formula (5), represents the corrected mapping relationship, represents the original mapping relationship, represents the correction strength coefficient, represents the total number of data distribution rules, represents the th data distribution rule, represents the correction weight of the th distribution rule.

[0090] ​If the structural features do not match the preset template, such as an abnormal distribution of objects within a disguised PDF file, a byte stream analysis tool such as Binwalk is used to break down the data blocks segment by segment. This byte stream analysis tool will divide the file byte stream into independent segments, extract hidden format identifiers such as embedded JPEG identifiers and data distribution patterns such as repeating compressed block patterns, and determine whether it conforms to known extended format standards such as PDF / A (PDF / Archive, electronic archive preservation format) extension, thereby obtaining the corrected mapping relationship, which helps to identify variant file formats.

[0091] S140. By integrating the corrected mapping relationship with the preliminary classification results, a data comparison tool is used to perform final verification of file type and structural characteristics, and obtain the complete correspondence between file format standards and data block distribution characteristics.

[0092] The correspondence between file format standards and data block distribution characteristics can be established using the following formula:

[0093] (6),

[0094] In formula (6), This represents the correspondence between complete file format standards and data block distribution characteristics. Indicates the total number of data blocks. Indicates the first The standard value of a file format, Indicates the first The distribution values ​​of the distribution characteristics of each data block.

[0095] By integrating the corrected mapping relationship and the preliminary classification result, the final verification of the file type and structural characteristics is performed using a data comparison tool such as the Diff tool. For example, the extended PDF feature in the corrected relationship is compared with the initial PDF classification, and after confirming the consistency, the complete file format standard such as PDF1.7 and the corresponding relationship of the data block distribution feature such as "compressed image block corresponds to page content stream" are obtained. This integration can improve the accuracy of file identification and bring higher evidence reliability in digital forensics business. For example, suppose that an "image.jpg" file is processed, the header is identified as "FFD8FFE0", the type "JPEG" and the standard "JFIF" are extracted, and the rule library comparison gives the preliminary result "Joint Photographic Experts Group format". The data structure such as SOI (Start Of Image, image start marker) marker and paragraph distribution is parsed, and the mapping such as "JPEG corresponds to segmented compression storage" is determined by comparing the template. For example, if there is no match, the hidden EXIF (Exchangeable Image File Format) identifier is extracted using the byte stream tool to correct the mapping to "extended JPEG".

[0096] The complete correspondence such as "JPEG standard and metadata block distribution" is obtained through integration and verification, which ensures format compatibility in media file management. For example, for a "docx" file, the header "PK0304" identifies "Office Open XML", and the preliminary classification is "document format".

[0097] The ZIP structure and XML (eXtensible Markup Language) part are parsed, and the template mapping "DOCX corresponds to packed XML storage" is compared. For example, when there is no match, the hidden OOXML (Office OpenXML) identifier is extracted to correct the mapping. Finally, the "DOCX standard and relationship file block distribution" is obtained through integration and verification, which improves the efficiency of document processing.

[0098] Preferably, the lossless compression storage method for heterogeneous data based on the Xinhua environment proposed in the embodiment comprises the following steps:

[0099] S210, according to the mapping relationship, the file type and structural characteristics are classified and grouped, and the feature data is extracted from the target electronic file using the content similarity comparison tool. By comparing the preset threshold range, the initial result of classification and grouping is determined.

[0100] The initial result of classification and grouping is obtained by the following formula:

[0101] (7),

[0102] In formula (7), denotes the file type denotes the set of classification grouping results, denotes the th electronic file, denotes the minimum value of the preset threshold, denotes the maximum value of the preset threshold, denotes the feature data value extracted from the file denotes the set of file indexes belonging to the type denotes the set of file indexes belonging to the type

[0103] According to the established mapping relationship between file types and structural features, a plurality of target electronic files are classified and grouped. For example, when processing a batch of mixed format file sets, first, feature data is extracted using a content similarity comparison tool such as a Cosine similarity calculator. This tool converts the byte sequence inside the file into a vector representation, and then calculates the cosine value between the vectors to measure the degree of similarity. Specifically, for a PDF file and a DOCX file, their header identification and data block mode are extracted as feature vectors. If the cosine similarity is higher than a preset threshold, such as 0.8, they are grouped into the same group. This ensures that the initial grouping result is based on structural similarity rather than just relying on file extensions, thereby improving the accuracy of classification in digital archive management business. For example, when processing enterprise report files, assuming that the mapping relationship shows that the PDF type corresponds to hierarchical object storage, and the DOCX corresponds to packed XML storage, after extracting the feature data through the content similarity comparison tool, if the similarity is within the threshold range, such as 0.7 to 0.9, the initial grouping is "document type", which helps subsequent batch processing and avoids confusion between files of different formats.

[0104] S220, according to the initial result of classification and grouping, using lossless encoding tool to scan the repeated mode part in the group, to obtain the distribution rule of repeated data block, get the preliminary compressed data block sequence.

[0105] The following formula is used to evaluate the complexity distribution characteristics of the preliminary compressed data block sequence:

[0106] (8),

[0107] In formula (8), denotes the complexity of the data block sequence, denotes the number of different types of repeated patterns, denotes the weight coefficient of the th pattern, denotes the number of repetitions of the th pattern in the sequence. ​

[0108] Based on the initial results of the above classification and grouping, lossless encoding tools such as the LZ77 algorithm implementer are used to scan the repetitive pattern portions within the groups. These lossless encoding tools identify repetitive byte sequences through a sliding window mechanism. For example, when scanning the content streams of multiple PDF files in the "Document" group, if repetitive page tree object patterns are found, their distribution patterns, such as compressed blocks appearing at fixed byte intervals, are recorded, thus obtaining a preliminary compressed data block sequence. This reduces redundant data and improves efficiency in storage optimization. For instance, for a JPEG file group containing repetitive image elements, the distribution pattern obtained after scanning shows that compressed blocks repeat every 500 bytes. The compressed sequence then removes these repetitions, generating a more compact sequence. This connection between grouping and compression processes lies in the fact that the initial results directly guide the scanning range. For example, the initial grouping ensures that repetitive patterns are only found within files with similar structures, avoiding cross-type interference. This, in turn, enables efficient deduplication storage in media file backup, resulting in a reduction in data volume. In actual forensic analysis, for the grouping of disguised files, abnormal structures are first isolated by similarity comparison, and then repeating patterns are scanned to reveal hidden data patterns. The final compressed sequence is convenient for further decoding to verify the integrity of the evidence.

[0109] Furthermore, the lossless compression and storage method for heterogeneous data based on the information technology innovation environment proposed in this embodiment includes step S300 as follows:

[0110] S310. Based on the initially compressed data block sequence, an information comparison tool is used to scan the data block distribution to obtain the distribution pattern of the redundancy pattern.

[0111] The spatial distribution of redundancy patterns in a data block sequence is described by the following formula:

[0112] (9),

[0113] In formula (9), Indicates the redundancy mode at location The probability distribution at that location, Indicates the length of the scan window. Indicates the first Redundancy strength parameters at each location, Indicates the first Data density at each location It represents the base of the natural constant.

[0114] For the sequence of data blocks after preliminary compression, first, an information comparison tool is used to scan the distribution of data blocks. This information comparison tool is essentially a software module based on statistical analysis, which calculates the frequency and position interval of repeated blocks by traversing each data block in the sequence. For example, when processing a batch of electronic report files that have been preliminarily compressed in a digital archive management service, the information comparison tool will check each byte block in the sequence one by one, and identify the pattern distribution rules of redundant metadata tags or text paragraphs. Specifically, assuming that the sequence contains multiple compressed blocks, the information comparison tool will generate a distribution chart, marking out redundant patterns such as identical hash value fragments that appear every 100 bytes, and thus derive the overall rules of these patterns, such as the average repetition rate or distribution density, which helps to determine whether further optimization is needed.

[0115] S320, if the distribution rule of the redundant pattern exceeds the preset threshold range, then the sequence matching tool is used to compare the data block sequence segment by segment to determine data segments with high similarity.

[0116] The data segments with high similarity are determined by the following formula:

[0117] (10),

[0118] In formula (10), represents the threshold decision value of sequence matching, represents the number of common features of two data segments, represents the total number of features, represents an adjustment parameter, represents the current matching accuracy, represents the reference accuracy value, when is greater than the set value, it is considered that the data segments have high similarity.

[0119] When processing enterprise financial data backup services, if the scanned redundant pattern distribution rule shows that the repetition rate exceeds the preset threshold, such as 0.5, the next step is triggered, and the sequence matching tool is used to compare the data block sequence segment by segment. This sequence matching tool is an implementation of a string comparison algorithm, which divides the sequence into fixed-length paragraphs, such as every 256 bytes, and then calculates the edit distance or similarity score between paragraphs to determine data segments with high similarity. For example, when backing up multiple Excel file sequences, the sequence matching tool finds that the similarity of some numerical table paragraphs is above 0.85, and thus marks these segments as potential optimization targets.

[0120] S330, for data segments with high similarity, a sliding window tool is used for one-by-one matching to determine data intervals that can be further compressed.

[0121] The total matching degree of the sliding window matching is obtained by the following formula:

[0122] (11),

[0123] In formula (11), denotes the total matching degree of the sliding window matching, denotes the total length of the target data sequence, denotes the size of the pattern window, denotes the current sliding position, denotes the offset within the window, denotes the data at position in the target sequence, denotes the data at position in the pattern window, denotes the matching degree function of the two data points.

[0124] For these data segments with high similarity, a sliding window tool is used for one-by-one matching. This sliding window tool dynamically adjusts the window size, such as expanding from 16 bytes to 64 bytes, to compare the sub-sequences within the segments one by one, and determine the data intervals that can be further compressed. For example, in a media file storage service, for similar image data segments, the tool identifies continuous pixel pattern intervals after sliding comparison. These intervals are determined to be compressible because they contain highly repetitive color value sequences.

[0125] S340, the data intervals that can be further compressed are integrated by a fusion processing tool, and a sequence optimization method is applied to reorganize the data blocks to obtain optimized compressed data.

[0126] The formula integrates the outputs of multiple fusion processing tools by weighted average to effectively integrate the data intervals that can be further compressed:

[0127] (12),

[0128] In formula (12), denotes the fusion processing result of the th data interval, denotes the total number of available fusion tools, denotes the weight coefficient of the th fusion tool, denotes the compression capability evaluation value of the th tool for the data interval .

[0129] The overall performance of the compressed data obtained after fusion processing and sequence optimization is evaluated by the following formula:

[0130] (13),

[0131] In formula (13), represents the final optimized compression data quality index, represents the volume size of the original data, represents the volume size of the compressed data, represents the compression efficiency coefficient, represents the data integrity retention factor.

[0132] The further compressible data intervals are integrated by a fusion processing tool, which is a data merging module that merges adjacent or similar intervals into larger blocks, such as using a joint algorithm to connect scattered intervals, and then applying a sequence optimization method to reorganize the data blocks to obtain optimized compressed data. For example, in cloud storage services, the sequence after reorganization removes redundant redundancy to generate more compact final data streams, which in actual operation ensures the improvement of storage efficiency.

[0133] Preferably, the lossless compression storage method for heterogeneous data based on the Xinchuang environment according to the embodiment comprises the following steps:

[0134] S410, according to the optimized compressed data, a corresponding data check code is generated by using a check tool, and the compression parameters are obtained, and the mapping relationship between the data check code and the compression parameters is determined.

[0135] The following formula is used to generate a comprehensive check code by XOR operation of multiple check algorithms:

[0136] (14),

[0137] In formula (14), represents the generated data check code, represents the hash function operation on the compressed data, represents the cyclic redundancy check calculation on the compressed data, represents the first bit of the compressed data, represents the total number of bits of the compressed data.

[0138] The mapping relationship between the data check code and the compression parameters is obtained by the following formula:

[0139] (15),

[0140] In formula (15), represents the mapping relationship function between the data check code and the compression parameters , 、 、 denotes a mapping weight coefficient, denotes a check code modulo parameter set size operation, denotes a hash operation on compression parameters, denotes an encoding value of compression parameters, denotes a bitwise AND operation.

[0141] For optimizing compressed data, first, a corresponding data check code is generated using a check tool, which is a software component based on a hash function that calculates a check value such as MD5 (Message-Digest Algorithm 5) or SHA-256 (Secure Hash Algorithm 256-bit) by performing a byte-by-byte scan of the data block to ensure data integrity, while extracting parameters such as compression ratio and algorithm type from the compression process, for example, when processing enterprise financial report backups, the check tool scans the optimized data stream to generate a 128-bit check code and records compression parameters including the LZW (Lempel-Ziv-Welch) algorithm used and a compression ratio of 1.5, thereby establishing a correspondence between the two. Specifically, this mapping relationship is determined through a key-value pair structure, where the check code is the key and the compression parameters are the value, forming a simple association table, such as in digital archive management services, if the data check code is a specific hash string, the corresponding parameters are displayed as gzip (GNU zip, GNU free software file compression program) method and 0.7 ratio, which helps subsequent verification and recovery.

[0142] S420, for the mapping relationship between the data check code and the compression parameters, a construction tool is used to integrate into a unified metadata structure, if the format of the metadata structure does not reach the preset threshold, an adjustment tool is used to correct, and a standardized metadata structure is obtained.

[0143] The following formula describes the process of integrating the mapping relationship between the check code and the compression parameters into a unified metadata structure by the construction tool:

[0144] (16),

[0145] In formula (16), denotes a unified metadata structure, denotes a set of data check codes, denotes a set of compression parameters, denotes a mapping function between the check code and the compression parameters.

[0146] The process of modifying the non-compliant metadata structure by adjusting the tool to obtain the standardized result is described by the following formula:

[0147] (17),

[0148] In formula (17), represents the standardized metadata structure, represents the original metadata structure, represents the correction function of the adjusting tool, represents the correction parameter vector.

[0149] For the mapping relationship between the data check code and the compression parameter, the construction tool integrates it into a unified metadata structure. This construction tool is an XML or JSON formatted module that organizes mapping information into a hierarchical tree structure. For example, in the media file storage scenario, the construction tool packages the check code and the parameter into a JSON object, including fields such as "checksum" and "params". If the structure format does not meet the preset threshold, such as a minimum of 100 bytes, the adjusting tool intervenes to correct it by adding padding or restructuring the field to obtain the standardized metadata structure.

[0150] S430, bind the standardized metadata structure with the optimized compression data, generate an initial encapsulation data packet using an encapsulation tool, judge whether it meets the cross-platform transmission requirements, and obtain the compatibility of the initial encapsulation data packet.

[0151] The compatibility of the initial encapsulation data packet is obtained by the following formula:

[0152] (18),

[0153] In formula (18), represents the cross-platform compatibility index of the initial encapsulation data packet, represents the total number of target platforms, represents the compatibility score of the th platform, represents the weight factor of the th platform, represents the transmission requirement threshold of the th platform, represents the standardization coefficient of cross-platform transmission.

[0154] In cloud storage services, after the standardized metadata structure is bound with the optimized compressed data, an initial encapsulation data packet is generated by an encapsulation tool, which is an implementation of a packaging algorithm. The encapsulation tool places the metadata in the packet header, followed by the compressed data body, forming a complete file packet. Then it determines whether the packet meets the cross-platform transmission requirements, such as checking if it supports decoding for Windows and Linux, and obtaining a compatibility score. If the score is below 0.8, it is considered not to meet the standard.

[0155] S440, if the compatibility of the initial encapsulation data packet does not meet the preset standard, adjust the transmission protocol through a protocol adaptation tool to obtain a final encapsulation data packet.

[0156] The process of adjusting the data packet through the protocol adaptation tool is described by the following formula:

[0157] (19),

[0158] In formula (19), represents the encapsulation data packet adjusted by the protocol adaptation tool, represents the protocol adaptation function, represents the original data packet, represents the protocol parameter vector that needs to be adjusted, represents the transmission protocol conversion function, represents the data packet re-encapsulation operation.

[0159] If the compatibility of the initial encapsulation data packet is insufficient, the protocol adaptation tool will adjust the transmission protocol. This protocol adaptation tool is a network protocol conversion module that switches from HTTP (HyperText Transfer Protocol) to FTP (File Transfer Protocol) or adds a compatibility layer. For example, when handling multi-version electronic report transmission, the protocol adaptation tool detects that the packet fails to parse on the iOS platform and modifies it to use SFTP (Secure File Transfer Protocol) protocol to ensure cross-device compatibility of the packet. Finally, an adjusted encapsulation data packet is obtained, which continues the coherence of data processing in the actual backup process.

[0160] Further, the lossless compression storage method for heterogeneous data based on the signal creation environment proposed in this embodiment includes the following steps:

[0161] S510, obtain the storage path information from the encapsulation data packet, and use a path parsing tool to perform format verification on the storage path information. If it does not meet the preset threshold, adjust it through a format conversion tool to obtain standardized path data.

[0162] The standardized path data is obtained by the following formula:

[0163] (20),

[0164] In formula (20), represents the standardized path data obtained after processing by the format conversion tool, represents the original storage path information, represents the set of format conversion rules, represents the adjustment parameter, represents the conversion function of the format conversion tool.

[0165] The process of obtaining storage path information from the encapsulated data packet involves parsing the metadata header in the packet. For example, in the enterprise financial report backup service, the path information may be embedded in a JSON structure. The path parsing tool is a regular expression-based software module that first extracts the path string such as " / backup / finance / 2023 / report.dat" and then checks whether its format conforms to the preset rules, such as the path length not exceeding 256 characters and not containing illegal symbols. If the path contains a backslash instead of a forward slash, the tool judges that it does not meet the threshold value, such as a format consistency score below 0.9. At this time, the format conversion tool intervenes, which is a string processing component that replaces the backslash with a forward slash and normalizes the separator, finally obtaining the standardized path data such as " / backup / finance / 2023 / report.dat", which ensures the compatibility of subsequent transmission.

[0166] The format verification mechanism of the above path parsing tool is based on the principle of preset threshold value, which defines a scoring system, for example, the threshold value is 0.85, the path parsing tool checks the path components such as directory depth and character encoding, if the depth exceeds 5 layers, it will be deducted, when the threshold value is below, it will trigger the adjustment of the conversion tool, which continues the standardization of the path when processing multi-platform backup, ensures seamless migration from Windows to Linux, while the specific adjustment process of the format conversion tool includes parsing the path string, identifying non-standard elements such as mixed case, and unifying it to lowercase form, obtaining data such as " / archives / media / 2023 / video.mp4", which is connected with the aforementioned protocol matching, because the standard path directly affects the selection of the protocol, avoiding transmission failure caused by path error.

[0167] S520, according to the standardized path data, using protocol adaptation tool to match transmission protocol according to the interface characteristics of the target storage device, judge whether it meets the safety requirements, obtain the adapted protocol configuration.

[0168] The adapted protocol configuration is derived by the following equation:

[0169] (21),

[0170] In equation (21), represents the optimal protocol configuration, represents the set of all possible protocol configurations, represents the balancing factor between performance and compatibility, represents the reliability score of the configuration , represents the performance score of the configuration , represents the parameter selection that maximizes the objective function. The control logic of equation (21) is to make a weighted trade-off between two objectives (reliability and performance ) to determine the optimal protocol configuration .

[0171] According to the standardized path data, the protocol adaptation tool performs transmission protocol matching for the interface characteristics of the target storage device. For example, in a digital archive management scenario, the tool analyzes the interface of the device pointed to by the path, such as a NAS (Network Attached Storage) server, which supports the SMB (Server Message Block) protocol, and then matches protocols such as FTP or HTTPS to determine whether security requirements are met, which involves checking whether the protocol enables encryption. If it does not meet the requirements, such as lacking a TLS layer, it is adjusted to add a security layer, and the adapted protocol configuration such as "HTTPS with TLS 1.3" is obtained.

[0172] The transmission protocol matching of the protocol adaptation tool involves detailed analysis of interface characteristics. For example, for the interface of an S3 (Amazon Simple Storage Service) compatible storage device, the tool queries the API (Application Programming Interface) document to confirm the list of supported protocols, and then makes a judgment based on security requirements such as the need for an authentication token. If the initial match is HTTP but the security is insufficient, it is adapted to a configuration with an added OAuth (Open Authorization) layer, which ensures the confidentiality of data in electronic report transmission and provides a basis for subsequent transmission.

[0173] S530, transmit the encapsulated data packet to the target storage device according to the adapted protocol configuration.

[0174] The probability of successfully transmitting an encapsulated data packet to the target storage device is calculated using the following formula:

[0175] (twenty two),

[0176] In formula (22), This indicates the probability that a data packet was successfully transmitted to the target storage device. This indicates the total number of nodes in the transmission path. Indicates the first Failure rate of each node Indicates the first The data processing capacity of each node This indicates the receive fault tolerance parameters of the storage device. Indicates the first The success probability of each node, the control logic of formula (22) is to apply it to all nodes. The success probabilities of each step are multiplied together to obtain the final probability of the encapsulated data packet being successfully transmitted to the target storage device. .

[0177] The process of transmitting encapsulated data packets to the target storage device through the adapted protocol configuration includes initializing the connection and sending data. For example, in media file storage services, the configured HTTPS protocol is used to push the packet body to the cloud server to ensure that the data is not intercepted during transmission.

[0178] The process of transmitting encapsulated data packets can be extended to chunked transmission. For example, when processing large backup files, the protocol configuration guidance tool can split the packet into multiple chunks, with each chunk being transmitted to the device independently. This enhances the reliability of the transmission and is linked to a write acknowledgment feedback mechanism, as the acknowledgments of each chunk accumulate to form an overall signal.

[0179] S540. Obtain the write confirmation feedback returned by the target storage device, use a verification tool to judge the legality of the write confirmation feedback, and obtain the write confirmation signal.

[0180] The write confirmation signal is derived using the following formula:

[0181] (twenty three),

[0182] In formula (23), This indicates the final write confirmation signal generated. The score representing the legality assessment. This represents the threshold for legality judgment; when the legality judgment score is... Greater than or equal to the legality threshold 1, otherwise 0. The control logic of equation (23) is a binary decision made by a conditional statement: if the final generated write acknowledgement signal is greater than or equal to a threshold value , then the write acknowledgement feedback is determined to be valid, and the output is ; if the final generated write acknowledgement signal is less than the threshold value , then the write acknowledgement feedback is determined to be invalid, and the output is .

[0183] After obtaining the write acknowledgement feedback returned by the target storage device, the verification tool performs a validity judgment on the feedback. The verification tool is a signature verification module that compares the digital signature in the feedback with an expected value. If the comparison is a match, the write acknowledgement signal is obtained. In cloud storage services, the feedback such as "Write Success with ID: 12345" is verified to be valid after verification.

[0184] The validity judgment of the write acknowledgement feedback by the verification tool includes parsing the feedback structure such as an XML tag and checking whether the timestamp is within an expected range. If the feedback delay exceeds 10 seconds, it is considered invalid. After obtaining the acknowledgement signal, the continuation of the business process is supported. For example, in archive management, this signal triggers the update of the archive log.

[0185] Preferably, the lossless compression storage method for heterogeneous data based on the Xinda environment proposed in the embodiment comprises the following steps:

[0186] S610, obtaining the storage device identifier from the write acknowledgement signal, using a query tool to locate data for the target storage device, and obtaining the storage location of the encapsulated data packet.

[0187] The storage device identifier is obtained from the write acknowledgement signal by the following formula:

[0188] (24),

[0189] In formula (24), represents the storage device identifier extracted from the write acknowledgement signal, represents the response function of the write acknowledgement signal, represents the timestamp, and the function is used to generate a unique device identifier code.

[0190] The storage location of the encapsulated data packet is obtained by the following formula:

[0191] (25),

[0192] In formula (25), represents the storage location of the encapsulated data packet, represents the scanning result of the storage block, represents the offset calculated based on the data packet header information.

[0193] The process of obtaining the storage device identifier from the write confirmation signal involves parsing a specific field within the signal. For example, in enterprise data backup services, the signal may contain a device ID such as "NAS-001". The query tool is a database query module that connects to the storage management system based on this ID. It first sends a query request to locate the data packet. Specifically, the query tool scans the device's directory structure and matches the data packet's metadata tags to ultimately obtain the location information such as " / data / backup / 2023 / file.pkg".

[0194] S620, according to the storage location of the packaged data packet, the comparison tool is used to extract the check code information of the packaged data packet, if the check code of the packaged data packet does not match the preset threshold, the integrity correction is carried out through the repair tool to determine the integrity state of the data packet.

[0195] The following formula is used to calculate and extract the check code information from the packaged data packet at the specified storage location through the comparison tool:

[0196] (26),

[0197] In formula (26), represents the check code value extracted from the packaged data packet, represents the total length of the data packet, represents the data byte at the th position in the data packet, represents the hash operation on the th data byte, represents the modulus reference value of the check code.

[0198] The following formula is used to determine whether the check code of the packaged data packet matches the preset threshold, thereby deciding whether repair operation is needed:

[0199] (27),

[0200] In formula (27), represents the start flag of the repair tool, when the value is 1, the integrity correction is started, when the value is 0, no repair is needed; represents the check code value of the current data packet, represents the preset check code threshold, represents the allowed error range.

[0201] The final integrity state of the entire data packet is evaluated and determined by the following formula after the repair tool completes the integrity correction:

[0202] (28),

[0203] In formula (28), represents the integrity state evaluation value of the data packet, represents the total number of detected data segments, represents the weight coefficient of the th data segment, represents the integrity verification result of the th data segment.

[0204] Based on the obtained storage location of the encapsulated data packet, the process of extracting the check code information of the encapsulated data packet by the comparison tool includes reading the tail check segment of the data packet. For example, in the digital archive storage scene, the comparison tool is a hash calculation component that extracts the MD5 check code such as "a1b2c3d4e5f6g7h8", and then compares it with the preset threshold. If it does not match, such as the hash value difference exceeds the similarity threshold of 0.1, the repair tool, which is a data recovery module, is activated to correct the integrity by recalculating and replacing the damaged part to determine the integrity state of the data packet as "repaired".

[0205] S630, according to the integrity state of the data packet, obtain the comparison record in the fusion information processing link, and use the verification tool to judge the data consistency to obtain the consistency verification result.

[0206] The consistency verification result is obtained by the following formula:

[0207] (29),

[0208] In formula (29), represents the final data consistency verification result, represents the total number of test types executed by the verification tool, represents the number of errors found in the th type of test, represents the total number of check items in the th type of test, represents the importance index of the th type of test.

[0209] According to the complete state of the data packet, the process of obtaining the comparison record in the fusion information processing link needs to understand the fusion information processing link first, which is a data integration stage, and the comparison record is a log file generated in the previous processing. For example, in the media content management business, the verification tool is a consistency checking software that loads these records such as "comparison log: hash_match=true", and then judges whether the data block is consistent item by item to get the verification result such as "consistency passed".

[0210] S640, according to the consistency verification result, using a path analysis tool to evaluate the feasibility of the recovery path, if the path meets the preset threshold, a final lossless storage record is generated;

[0211] The generation index of the final lossless storage record is obtained by the following formula:

[0212] (30),

[0213] In formula (30), represents the generation index of the lossless storage record, represents the original data integrity, represents the data integrity after recovery, represents the data integrity index, represents the minimum integrity requirement, represents the quality evaluation value, represents the standard quality requirement, represents the Dirac function to ensure that the quality meets the standard. The control logic of formula (30) is to calculate the lossless loss by comprehensively calculating three-dimensional indicators, represents the data volume consistency dimension, represents the integrity dimension, represents the quality deviation dimension, and finally takes the minimum value of the "data volume consistency" and "integrity" two dimensions, and combines the influence of "quality deviation" to get the generation index of the lossless storage record The smaller the generation index of the lossless storage record, the better the losslessness of the data in the dimension, integrity and quality; the larger the generation index of the lossless storage record, the more serious the loss.

[0214] According to the consistency verification result, the process of evaluating the feasibility of the recovery path by using the path analysis tool involves path simulation. For example, in a cloud file recovery service, the path analysis tool is a routing evaluation component that checks the availability of a recovery path such as “ / restore / media / 2023 / video.mp4”, including verifying node connectivity and permissions. If the evaluation score of the path exceeds a preset threshold such as 0.9, a final lossless storage record is generated, which is in the form of a JSON file containing all verification details to support subsequent audits.

[0215] See Figure 2 The embodiment provides a lossless compression storage system based on heterogeneous data in a Xingcheng environment, which is used for executing the lossless compression storage method based on heterogeneous data in a Xingcheng environment. The system comprises a mapping relationship acquisition module 10, a data block sequence determination module 20, an optimized compression data acquisition module 30, an encapsulated data packet forming module 40, a write confirmation signal acquisition module 50 and a lossless storage record acquisition module 60. The mapping relationship acquisition module 10 is used for extracting a file type identifier and an internal data structure mode from a target electronic file by scanning the target electronic file, obtaining a mapping relationship between the file type and the structure characteristics, wherein the mapping relationship comprises a file format standard and a data block distribution characteristic. The data block sequence determination module 20 is used for classifying and grouping the file type and the structure characteristics according to the mapping relationship, performing preliminary compression on a repeated mode part by using a lossless coding method, determining a data block sequence after preliminary compression, and classifying and grouping based on file content similarity. The optimized compression data acquisition module 30 is used for judging in a fusion information processing link by comparing the similarity threshold between the data block sequences if there is a redundant mode in the data block sequence after preliminary compression, wherein the fusion information processing link involves data mode matching optimization, and then applying a sequence matching algorithm to perform sliding window matching to obtain optimized compression data. The encapsulated data packet forming module 40 is used for generating a metadata structure containing a data check code and a compression parameter for the optimized compression data, merging and encapsulating the optimized compression data and the metadata structure to form an encapsulated data packet, and supporting cross-platform transmission of the encapsulated data packet. The write confirmation signal acquisition module 50 is used for parsing storage path information from the encapsulated data packet, transmitting the encapsulated data packet to a target storage device by using a safe and compatible platform write protocol, and obtaining a write confirmation signal. The lossless storage record acquisition module 60 is used for querying the integrity of the encapsulated data packet in the storage device according to the write confirmation signal, verifying data consistency in the fusion information processing link by check code comparison, judging the feasibility of the recovery path, and obtaining a final lossless storage record.

[0216] Compared with the prior art, the method and system for lossless compression storage of heterogeneous data based on the Xinhua environment provided by the embodiment first extracts the mapping relationship between the file type and the structural features by scanning the target file, performs preliminary lossless compression after classification and grouping, further optimizes the compressed data by using the sliding window matching and sequence matching algorithm for the redundant mode, then generates the metadata structure containing the check code and the compression parameters, encapsulates into the data packet supporting cross-platform transmission, finally transmits to the target storage device through the secure write protocol, and verifies the data consistency and the recovery path feasibility by using the check code. The embodiment significantly improves the data compression efficiency, ensures the compatibility of cross-platform transmission and the integrity of stored data, and provides efficient and reliable technical support for complex electronic file processing.

[0217] While the preferred embodiments of the application have been described, additional variations and modifications can be made to the embodiments by those skilled in the art once they learn of the basic inventive concepts. Therefore, the appended claims are intended to encompass within their scope all such variations and modifications as are included within the scope of the application. It should be apparent that the application is not limited to the specific embodiments described herein, but can be practiced with modification within the spirit and scope of the present application. Accordingly, though the present application has been described in some detail to provide a thorough understanding thereof, it will be apparent that certain minuscule changes can be made in the application without departing from the spirit and the scope of the application. It is intended to cover all such modifications as fall within the scope of the application.

Claims

1. A lossless compression and storage method for heterogeneous data based on the information technology innovation environment, characterized in that, Includes the following steps: S100. By scanning the target electronic file, extract the file type identifier and internal data structure pattern from the target electronic file to obtain the mapping relationship between file type and structural features, wherein the mapping relationship includes file format standards and data block distribution characteristics; S200. Based on the mapping relationship, file types and structural features are classified and grouped. A lossless encoding method is used to perform preliminary compression on the repetitive pattern part to determine the data block sequence after preliminary compression. The classification and grouping are based on file content similarity. S300. If there are redundant patterns in the initially compressed data block sequence, the similarity threshold between the data block sequences is compared in the fusion information processing stage to make a judgment. The fusion information processing stage involves data pattern matching optimization, and then a sequence matching algorithm is applied to perform sliding window matching to obtain optimized compressed data. S400. For the optimized compressed data, generate a metadata structure containing data checksum and compression parameters. By merging and encapsulating the optimized compressed data with the metadata structure, an encapsulated data packet is formed. The encapsulated data packet supports cross-platform transmission. S500: Parse the storage path information from the encapsulated data packet, and transmit the encapsulated data packet to the target storage device using a secure and compatible platform write protocol to obtain a write confirmation signal; S600. Based on the write confirmation signal, query the integrity of the encapsulated data packet in the storage device, verify the data consistency in the fusion information processing stage by comparing the check code, determine the feasibility of the recovery path, and obtain the final lossless storage record. Step S300 includes: S310. Based on the initially compressed data block sequence, use an information comparison tool to scan the data block distribution to obtain the distribution pattern of the redundancy pattern. S320. If the distribution pattern of the redundancy pattern exceeds the preset threshold range, the data block sequence is compared segment by segment using a sequence matching tool to determine the data segments with high similarity. S330. For data segments with high similarity, use a sliding window tool to match them one by one to determine the data range that can be further compressed. S340. By using a fusion processing tool to integrate data ranges that can be further compressed, and applying a sequence optimization method to reorganize the data blocks, optimized compressed data is obtained.

2. The lossless compression and storage method for heterogeneous data based on the information technology innovation environment according to claim 1, characterized in that, Step S100 includes: S110. By scanning the target electronic file, obtain the header identification information inside the target electronic file, extract the file type identifier and the corresponding format standard from the header identification information, and compare them using a preset rule base to obtain a preliminary classification result of the file type. S120. Based on the preliminary classification results, analyze the internal data structure pattern of each file type to obtain the distribution characteristics and storage method of data blocks, and determine the preliminary mapping relationship between file type and structural characteristics by comparing with the preset data block template. S130. If the structural features do not match the preset template, a byte stream analysis tool is used to disassemble the data blocks of the target electronic file segment by segment, extract the hidden format identifiers and data distribution patterns, determine whether they conform to the known extended format standards, and obtain the corrected mapping relationship. S140. By integrating the corrected mapping relationship with the preliminary classification results, a data comparison tool is used to perform final verification of file type and structural characteristics, and obtain the complete correspondence between file format standards and data block distribution characteristics.

3. The lossless compression and storage method for heterogeneous data based on the information technology innovation environment according to claim 1, characterized in that, Step S200 includes: S210. Based on the mapping relationship, classify and group the files according to their type and structural features. Use a content similarity comparison tool to extract feature data from the target electronic files. Determine the initial results of the classification and grouping by comparing with a preset threshold range. S220. Based on the initial classification and grouping results, a lossless coding tool is used to scan the repetitive pattern portion within the group to obtain the distribution pattern of the repetitive data blocks and obtain the preliminary compressed data block sequence.

4. The lossless compression and storage method for heterogeneous data based on the information technology innovation environment according to claim 1, characterized in that, Step S400 includes: S410. Based on the optimized compressed data, a corresponding data verification code is generated using a verification tool, and compression parameters are obtained simultaneously to determine the mapping relationship between the data verification code and the compression parameters. S420. Based on the mapping relationship between the data verification code and the compression parameters, a construction tool is used to integrate them into a unified metadata structure. If the format of the metadata structure does not reach the preset threshold, it is corrected by an adjustment tool to obtain a standardized metadata structure. S430. Bind the optimized compressed data with a standardized metadata structure, use a packaging tool to generate an initial packaged data packet, determine whether it meets the cross-platform transmission requirements, and obtain the compatibility of the initial packaged data packet. S440. If the compatibility of the initial encapsulated data packet does not meet the preset standard, the transmission protocol is adjusted by a protocol adaptation tool to obtain the final encapsulated data packet.

5. The lossless compression and storage method for heterogeneous data based on the information technology innovation environment according to claim 1, characterized in that, Step S500 includes: S510. Obtain storage path information from the encapsulated data packet, and use a path parsing tool to perform format verification on the storage path information. If it does not meet the preset threshold, adjust it using a format conversion tool to obtain standardized path data. S520. Based on standardized path data, use a protocol adaptation tool to match the transmission protocol to the interface characteristics of the target storage device, determine whether the security requirements are met, and obtain the adapted protocol configuration. S530. Based on the adapted protocol configuration, the encapsulated data packet is transmitted to the target storage device; S540. Obtain the write confirmation feedback returned by the target storage device, and use a verification tool to determine the legality of the write confirmation feedback to obtain a write confirmation signal.

6. The lossless compression and storage method for heterogeneous data based on the information technology innovation environment according to claim 1, characterized in that, Step S600 includes: S610. Obtain the storage device identifier from the write confirmation signal, use a query tool to locate the data on the target storage device, and obtain the storage location of the encapsulated data packet; The storage device identifier can be obtained from the write acknowledgment signal using the following formula: ； in, This indicates the storage device identifier extracted from the write confirmation signal. This represents the response function for the write confirmation signal. Represents a timestamp. This function is used to generate a unique device identifier. The storage location of the encapsulated data packet is determined by the following formula: ； in, Indicates the storage location of the encapsulated data packet. This indicates the scan results of the storage blocks. This represents the offset calculated based on the packet header information; S620. Based on the storage location of the encapsulated data packet, a comparison tool is used to extract the check code information of the encapsulated data packet. If the check code of the encapsulated data packet does not match the preset threshold, the integrity is corrected by a repair tool to determine the integrity status of the data packet. The following formula is used to calculate and extract checksum information from encapsulated data packets at a specified storage location using a comparison tool: ； in, This represents the checksum value extracted from the encapsulated data packet. Indicates the total length of the data packet. Indicates the first in the data packet Data bytes at each location, Indicates the first Hash operation performed on each data byte The modulus base value representing the check code; The following formula is used to determine whether the checksum of the encapsulated data packet does not match a preset threshold, thereby deciding whether a repair operation is needed: ； in, This indicates the activation flag for the repair tool. A value of 1 activates integrity correction, while a value of 0 indicates that no repair is needed. This represents the checksum value of the current data packet. This indicates the preset checksum threshold. Indicates the allowable error range; The following formula is used to comprehensively evaluate and determine the final integrity status of the entire data packet after the repair tool has completed integrity correction: ； in, This represents the complete state assessment value of the data packet. Indicates the total number of data segments detected. Indicates the first The weighting coefficients of each data segment. Indicates the first The integrity verification results of each data segment; S630. Based on the complete state of the data packet, obtain the comparison records in the fusion information processing stage, use the verification tool to judge the data consistency, and obtain the consistency verification result; S640. Based on the consistency verification results, the feasibility of the recovery path is evaluated using a path analysis tool. If the path meets the preset threshold, the final lossless storage record is generated.

7. The lossless compression and storage method for heterogeneous data based on the information technology innovation environment according to claim 6, characterized in that, In step S630, the consistency verification result is obtained using the following formula: ； in, This indicates the final data consistency verification result. This indicates the total number of test types executed by the verification tool. Indicates the first The number of errors found in class tests Indicates the first Total number of checks in class tests Indicates the first Importance index of class tests.

8. The lossless compression and storage method for heterogeneous data based on the information technology innovation environment according to claim 7, characterized in that, In step S640, the final lossless storage record generation index is obtained using the following formula: ； in, Indicates the generation metrics for lossless storage records. Indicates the completeness of the original data. Indicates the integrity of the recovered data. Indicates the data integrity index. Indicates the minimum integrity requirement. Indicates the quality assessment value. Indicates standard quality requirements. This indicates that the Dirac function ensures that the quality meets the standards.

9. A lossless compression and storage system for heterogeneous data under a domestically developed information technology environment, used to execute the lossless compression and storage method for heterogeneous data under a domestically developed information technology environment as described in any one of claims 1 to 8, characterized in that, include: The mapping relationship acquisition module (10) is used to extract the file type identifier and internal data structure pattern from the target electronic file by scanning the target electronic file, and obtain the mapping relationship between the file type and the structural features, wherein the mapping relationship includes the file format standard and the data block distribution features; The data block sequence determination module (20) is used to classify and group file types and structural features according to the mapping relationship, perform preliminary compression on the repetitive pattern part using a lossless encoding method, and determine the data block sequence after preliminary compression. The classification and grouping are based on file content similarity. The optimized compressed data acquisition module (30) is used to make a judgment in the fusion information processing stage by comparing the similarity threshold between data block sequences if there are redundant patterns in the initially compressed data block sequence. The fusion information processing stage involves data pattern matching optimization, and then the sequence matching algorithm is applied to perform sliding window matching to obtain optimized compressed data. The data packet forming module (40) is used to generate a metadata structure containing a data check code and compression parameters for the optimized compressed data, and to form a data packet by merging the optimized compressed data with the metadata structure. The data packet supports cross-platform transmission. The write confirmation signal acquisition module (50) is used to parse the storage path information from the encapsulated data packet, and transmit the encapsulated data packet to the target storage device using the security-compatible platform write protocol to obtain the write confirmation signal. The lossless storage record acquisition module (60) is used to query the integrity of the encapsulated data packet in the storage device according to the write confirmation signal, verify the data consistency in the fusion information processing stage by comparing the check code, determine the feasibility of the recovery path, and obtain the final lossless storage record.

Images