Lossless compression storage method and system for heterogeneous data based on credential environment

Abstract

The invention discloses a lossless compression storage method and system based on heterogeneous data in a credential environment, and the method comprises the steps: firstly, extracting a mapping relation between a file type and a structural feature through scanning a target file, carrying out the preliminary lossless compression after classification and grouping, and further employing a sliding window matching and sequence matching algorithm to optimize compressed data for a redundancy mode, and then generating a metadata structure containing a check code and a compression parameter, packaging the metadata structure into a data packet supporting cross-platform transmission, finally transmitting the data packet to a target storage device through a secure write-in protocol, and verifying data consistency and recovering path feasibility by using the check code. According to the method, the data compression efficiency is remarkably improved, the compatibility of cross-platform transmission and the integrity of stored data are ensured, and efficient and reliable technical support is provided for complex electronic file processing.

Description

Technical Field

[0001] This invention relates to the field of data storage technology, and in particular discloses a lossless compression storage method and system for heterogeneous data based on the information technology innovation environment. Background Technology

[0002] Against the backdrop of rapid development in information technology, data storage technology, as a crucial cornerstone supporting digital transformation, is undeniably important. Especially in information environments built around domestically developed hardware and software, the storage and management of massive amounts of data directly impacts system efficiency and economic costs. How to ensure data integrity while minimizing storage space usage has become a critical area that urgently needs to be addressed.

[0003] Research in this field not only impacts enterprise operating costs but also places higher demands on information security and system stability. However, some existing storage compression methods often fail to fully realize their potential in domestically developed information environments. Most of these methods are not adapted to the characteristics of domestic processor architectures and operating systems, resulting in limitations in compression efficiency and operating speed when processing complex data.

[0004] A deeper problem lies in the fact that some solutions, in pursuing compression efficiency, may neglect the complete protection of the original data information, or lack effective verification mechanisms during long-term storage and recovery of compressed data, thus creating security vulnerabilities. Focusing on the technical challenges, the core issue is how to design an algorithm that can significantly improve compression efficiency while ensuring no data loss in a domestically developed hardware and software environment. This is especially true for electronic document storage, where file types and structures vary greatly, and the characteristics of different files place extremely high demands on the adaptability of compression algorithms. If targeted processing is not performed based on file content and format characteristics, the compression process may lead to wasted resources and even affect the accurate recovery of subsequent data. This unresolved technical factor directly results in storage space utilization failing to reach ideal levels in practical applications, while also increasing the complexity of data management. For example, in a specific business scenario, in an archive management system using a domestically developed processor, facing the storage needs of a large number of electronic documents, if the compression algorithm cannot adapt to file types—for example, using the same processing method for text documents and image documents—the compression effect of text documents may be poor, while some details of image documents may be lost after recovery. This situation not only increases the burden on storage devices, but also puts the long-term preservation and retrieval of data at risk.

[0005] Therefore, how to design a storage method that can both compress efficiently and ensure the complete recovery of data, taking into account the diverse characteristics of electronic documents in a domestic information environment, has become a key issue that urgently needs to be addressed. Summary of the Invention

[0006] This invention provides a lossless compression storage method and system for heterogeneous data based on the information technology innovation environment, aiming to solve at least one of the defects existing in the above-mentioned prior art.

[0007] One aspect of the present invention relates to a lossless compression and storage method for heterogeneous data based on the information technology innovation environment, comprising the following steps: S100. By scanning the target electronic file, extract the file type identifier and internal data structure pattern from the target electronic file to obtain the mapping relationship between file type and structural features, wherein the mapping relationship includes file format standard and data block distribution characteristics; S200. Based on the mapping relationship, classify and group the file types and structural features, use lossless coding methods to perform preliminary compression on the repetitive pattern parts, determine the data block sequence after preliminary compression, and classify and group based on file content similarity. S300. If there are redundant patterns in the initially compressed data block sequence, the similarity threshold between the data block sequences is compared in the fusion information processing stage. The fusion information processing stage involves data pattern matching optimization, and then the sequence matching algorithm is applied to perform sliding window matching to obtain optimized compressed data. S400: For optimized compressed data, a metadata structure containing data check codes and compression parameters is generated. The optimized compressed data and the metadata structure are merged and encapsulated to form an encapsulated data packet. The encapsulated data packet supports cross-platform transmission. S500: Parse the storage path information from the encapsulated data packet, and use the security-compatible platform write protocol to transmit the encapsulated data packet to the target storage device to obtain a write confirmation signal; S600: Based on the write confirmation signal, query the integrity of the encapsulated data packet in the storage device, verify the data consistency in the fusion information processing stage by comparing the check code, determine the feasibility of the recovery path, and obtain the final lossless storage record.

[0008] Further, step S100 includes: S110. By scanning the target electronic file, obtain the header identification information inside the target electronic file, extract the file type identifier and the corresponding format standard from the header identification information, and compare them using a preset rule base to obtain the preliminary classification result of the file type. S120. Based on the preliminary classification results, analyze the internal data structure pattern of each file type to obtain the distribution characteristics and storage method of data blocks. By comparing with the preset data block template, determine the preliminary mapping relationship between file type and structural characteristics. S130. If the structural features do not match the preset template, a byte stream analysis tool is used to disassemble the data blocks of the target electronic file segment by segment, extract the hidden format identifiers and data distribution patterns, determine whether they conform to the known extended format standards, and obtain the corrected mapping relationship. S140. By integrating the corrected mapping relationship with the preliminary classification results, a data comparison tool is used to perform final verification of file type and structural characteristics, and obtain the complete correspondence between file format standards and data block distribution characteristics.

[0009] Further, step S200 includes: S210. Based on the mapping relationship, classify and group the files according to their type and structural features. Use a content similarity comparison tool to extract feature data from the target electronic files. Determine the initial results of the classification and grouping by comparing with a preset threshold range. S220. Based on the initial classification and grouping results, a lossless coding tool is used to scan the repetitive pattern portion within the group to obtain the distribution pattern of the repetitive data blocks and obtain the preliminary compressed data block sequence.

[0010] Further, step S300 includes: S310. Based on the initially compressed data block sequence, use an information comparison tool to scan the data block distribution to obtain the distribution pattern of the redundancy pattern. S320. If the distribution pattern of the redundancy pattern exceeds the preset threshold range, the data block sequence is compared segment by segment using a sequence matching tool to determine the data segments with high similarity. S330. For data segments with high similarity, use a sliding window tool to match them one by one to determine the data range that can be further compressed. S340. By using a fusion processing tool to integrate data ranges that can be further compressed, and applying a sequence optimization method to reorganize the data blocks, optimized compressed data is obtained.

[0011] Further, step S400 includes: S410. Based on the optimized compressed data, a verification tool is used to generate the corresponding data verification code. At the same time, the compression parameters are obtained, and the mapping relationship between the data verification code and the compression parameters is determined. S420. Regarding the mapping relationship between data verification codes and compression parameters, a construction tool is used to integrate them into a unified metadata structure. If the format of the metadata structure does not meet the preset threshold, it is corrected by an adjustment tool to obtain a standardized metadata structure. S430: Bind the standardized metadata structure with the optimized compressed data, use the encapsulation tool to generate the initial encapsulated data packet, determine whether it meets the cross-platform transmission requirements, and obtain the compatibility of the initial encapsulated data packet; S440. If the compatibility of the initial encapsulated data packet does not meet the preset standard, the transmission protocol is adjusted through the protocol adaptation tool to obtain the final encapsulated data packet.

[0012] Further, step S500 includes: S510. Obtain storage path information from the encapsulated data packet, use a path parsing tool to perform format verification on the storage path information, and if it does not meet the preset threshold, adjust it using a format conversion tool to obtain standardized path data. S520. Based on standardized path data, use a protocol adaptation tool to match the transmission protocol to the interface characteristics of the target storage device, determine whether the security requirements are met, and obtain the adapted protocol configuration. S530: Based on the adapted protocol configuration, the encapsulated data packet is transmitted to the target storage device; S540. Obtain the write confirmation feedback returned by the target storage device, use a verification tool to judge the legality of the write confirmation feedback, and obtain the write confirmation signal.

[0013] Further, step S600 includes: S610. Obtain the storage device identifier from the write confirmation signal, use a query tool to locate the data on the target storage device, and obtain the storage location of the encapsulated data packet; S620. Based on the storage location of the encapsulated data packet, a comparison tool is used to extract the check code information of the encapsulated data packet. If the check code of the encapsulated data packet does not match the preset threshold, the integrity is corrected by a repair tool to determine the integrity status of the data packet. S630. Based on the complete state of the data packet, obtain the comparison records in the fusion information processing stage, use the verification tool to judge the data consistency, and obtain the consistency verification result; S640. Based on the consistency verification results, the feasibility of the recovery path is evaluated using a path analysis tool. If the path meets the preset threshold, the final lossless storage record is generated.

[0014] Another aspect of the present invention relates to a lossless compression and storage system for heterogeneous data under the domestic information technology innovation environment, for performing the above-described lossless compression and storage method for heterogeneous data under the domestic information technology innovation environment, comprising: The mapping relationship acquisition module is used to extract the file type identifier and internal data structure pattern from the target electronic file by scanning the target electronic file, and obtain the mapping relationship between the file type and structural features, wherein the mapping relationship includes the file format standard and data block distribution characteristics; The data block sequence determination module is used to classify and group files according to their type and structural features based on the mapping relationship. It uses a lossless encoding method to perform preliminary compression on the repetitive pattern part and determines the data block sequence after preliminary compression. The classification and grouping are based on the file content similarity. The optimized compressed data acquisition module is used to make a judgment in the fusion information processing stage by comparing the similarity threshold between data block sequences if there are redundant patterns in the initially compressed data block sequence. The fusion information processing stage involves data pattern matching optimization, and then applies a sequence matching algorithm to perform sliding window matching to obtain optimized compressed data. The data packet forming module is used to generate a metadata structure containing data check codes and compression parameters for optimized compressed data. By merging and encapsulating the optimized compressed data and the metadata structure, an encapsulated data packet is formed. The encapsulated data packet supports cross-platform transmission. The write confirmation signal acquisition module is used to parse the storage path information from the encapsulated data packet, transmit the encapsulated data packet to the target storage device using a security-compatible platform write protocol, and obtain the write confirmation signal. The lossless storage record acquisition module is used to query the integrity of the encapsulated data packets in the storage device based on the write confirmation signal, verify the data consistency in the fusion information processing stage by comparing the check code, determine the feasibility of the recovery path, and obtain the final lossless storage record.

[0015] The beneficial effects achieved by this invention are as follows: This invention provides a lossless compression and storage method and system for heterogeneous data in a domestically developed information technology environment. Addressing the comprehensive business challenges of data redundancy, cross-platform transmission compatibility, and storage integrity verification in electronic document processing, it constructs a complete solution by deeply integrating file type identifier extraction, data structure pattern analysis, and optimized compression techniques. First, the invention extracts the mapping relationship between file type and structural features by scanning target files. After classification and grouping, preliminary lossless compression is performed. For redundancy patterns, sliding window matching and sequence matching algorithms are further employed to optimize the compressed data. Subsequently, a metadata structure containing checksums and compression parameters is generated, encapsulated into a data packet supporting cross-platform transmission, and finally transmitted to the target storage device via a secure write protocol. The checksum is used to verify data consistency and recovery path feasibility. This invention significantly improves data compression efficiency, ensures cross-platform transmission compatibility and the integrity of stored data, and provides efficient and reliable technical support for complex electronic document processing. Attached Figure Description

[0016] Figure 1 This is a flowchart illustrating an embodiment of a lossless compression and storage method for heterogeneous data based on the information technology innovation environment of the present invention. Figure 2This is a functional block diagram of an embodiment of a lossless compression storage system for heterogeneous data based on the information technology innovation environment of the present invention.

[0017] Explanation of icon numbers: 10. Mapping Relationship Acquisition Module; 20. Data Block Sequence Determination Module; 30. Optimized and Compressed Data Acquisition Module; 40. Encapsulated Data Packet Formation Module; 50. Write Confirmation Signal Acquisition Module; 60. Lossless Storage Record Acquisition Module. Detailed Implementation

[0018] To better understand the above technical solutions, the following will provide a detailed explanation of the technical solutions in conjunction with the accompanying drawings and specific implementation methods.

[0019] like Figure 1 As shown, the first embodiment of the present invention proposes a lossless compression and storage method for heterogeneous data based on the information technology innovation environment, including the following steps: Step S100: By scanning the target electronic file, extract the file type identifier and internal data structure pattern from the target electronic file to obtain the mapping relationship between file type and structural features, wherein the mapping relationship includes file format standards and data block distribution characteristics.

[0020] Taking target electronic files (such as heterogeneous data like documents, images, and database files) in a domestically developed information technology environment as the processing object, a file scanning tool is used to traverse the file header identifier, field structure, and data organization form to extract two types of core information: first, file type identifiers (such as file extensions, format signatures, and protocol identifiers, used to distinguish the file's format category); and second, internal data structure patterns (such as data block partitioning rules, field arrangement order, and redundant data distribution locations, used to characterize the internal data organization logic of the file). These two types of information are then correlated and matched to form a mapping relationship between file type and structural characteristics. This mapping relationship specifically includes file format standards (corresponding to the official / industry format specifications for various types of files, clarifying the syntax rules, field meanings, and verification standards for data storage) and data block distribution characteristics (corresponding to the data partitioning method, size ratio, storage location, and relationships within the file), providing accurate feature basis for subsequent targeted compression based on file type and structure.

[0021] Step S200: Based on the mapping relationship, classify and group the file types and structural features, use lossless coding methods to perform preliminary compression on the repetitive pattern parts, determine the data block sequence after preliminary compression, and classify and group based on file content similarity.

[0022] Based on the file type and structural feature mapping relationship established in step S100, heterogeneous electronic files are classified and grouped using content similarity algorithms (such as cosine similarity, edit distance, etc.). Files with consistent file format standards, similar data block distribution characteristics, and content repetition that meets preset conditions are grouped together to ensure that files in the same group have similar compression adaptability. For the internal data of each group of files, recurring data segments (such as repeated paragraphs in documents, similar pixel blocks in images, repeated fields in databases, etc.) are located using pattern recognition tools. Lossless coding methods adapted to the information technology innovation environment (such as Huffman coding, LZ77 (Lempel-Ziv 1977 Algorithm, a lossless data compression algorithm based on sliding window) / LZ78 (Lempel-Ziv 1978 Algorithm, an adaptive dictionary coding algorithm based on trie), DEFLATE (a lossless data compression algorithm based on LZ77 algorithm and Huffman coding, etc.) are used to compress these recurring pattern parts, preserving the original structure and information of non-repetitive data. After compression, the data blocks are organized into an ordered sequence according to the original file's data block division rules and grouping logic. This sequence is the initially compressed data block sequence, which eliminates some redundant data while fully preserving the core information and recoverability of the original data.

[0023] Step S300: If there are redundant patterns in the initially compressed data block sequence, the similarity threshold between the data block sequences is compared in the fusion information processing stage to make a judgment. The fusion information processing stage involves data pattern matching optimization, and then a sequence matching algorithm is applied to perform sliding window matching to obtain optimized compressed data.

[0024] Taking the initially compressed data block sequence obtained in step S200 as the processing object, the redundancy pattern detection tool is first used to check for duplicate data segments (i.e., redundancy patterns) that were not initially compressed and removed from the sequence. After entering the fusion information processing stage, based on the data pattern matching optimization logic built into this stage, a preset similarity threshold (dynamically set according to the mapping relationship between file type and structural features) is called to quantify and compare the similarity between data block sequences. If the similarity between sequences is higher than the threshold, it is determined that there are redundant patterns that need further optimization. Subsequently, a sequence matching algorithm (such as KMP (Knuth-Morris-Pratt Algorithm) algorithm, BM (Boyer-Moore string search algorithm) algorithm, etc.) is applied to accurately locate and match the redundant patterns through a sliding window mechanism (the window size adapts to the distribution characteristics of the data blocks), remove the redundant representations of duplicate data, and retain only the core data and matching index information. Finally, optimized compressed data with lower redundancy and higher storage efficiency is formed, and data information is not lost throughout the process, meeting the requirements of lossless storage.

[0025] Step S400: For the optimized compressed data, generate a metadata structure containing data checksum and compression parameters. By merging and encapsulating the optimized compressed data and the metadata structure, an encapsulated data packet is formed. The encapsulated data packet supports cross-platform transmission.

[0026] Using the optimized compressed data obtained in step S300 as the core, a corresponding metadata structure is first generated. This metadata structure contains two types of key information: first, a data checksum (generated using verification algorithms compatible with the domestic IT environment, such as CRC32 (Cyclic Redundancy Check 32-bit) and SHA-256 (Secure Hash Algorithm 256-bit), used for subsequent verification of data integrity); and second, compression parameters (including lossless encoding type, sequence matching algorithm identifier, sliding window size, similarity threshold, etc., used for parameter reuse during data recovery). Subsequently, a standardized encapsulation protocol adapted to domestic IT environments (compatible with domestic operating systems, databases, and storage device interfaces) is used to merge and package the optimized compressed data and the metadata structure, forming a uniformly structured and formatted encapsulated data package. This encapsulated data package has built-in cross-platform adaptation fields, supporting seamless transmission between domestically developed devices with different architectures (such as x86, ARM (Advanced RISC Machine) architecture) and operating systems. At the same time, by binding metadata with core data, it ensures that the data is traceable and verifiable during transmission.

[0027] Step S500: Parse the storage path information from the encapsulated data packet, and use the security-compatible platform write protocol to transmit the encapsulated data packet to the target storage device to obtain a write confirmation signal.

[0028] Using the encapsulated data packet formed in step S400 as the processing object, the built-in storage path information (including standardized path parameters such as target storage device address, partition directory, and storage file name, adapted to the storage resource management specifications under the domestic IT innovation environment) is extracted using a data packet parsing tool. A compatible platform write protocol conforming to domestic IT innovation security standards (such as the domestic SMB3.0 adaptation protocol, the IPFS (InterPlanetary File System) domestic IT innovation customized protocol, etc., supporting interface compatibility between domestic operating systems and storage devices) is adopted. During transmission, data encryption (such as the national cryptographic algorithm SM4 (SM4Block Cipher Algorithm)) and transmission verification mechanisms are enabled to ensure the security and integrity of the data packet in the transmission link. The encapsulated data packet is transmitted to the designated target storage device (such as a domestic distributed storage cluster, local domestic IT innovation hard drive, etc.) according to the parsed storage path. After the device confirms that the data packet has been completely received and successfully written, it returns a write confirmation signal containing the write status and storage location verification code, serving as a closed-loop feedback basis for data storage landing.

[0029] Step S600: Based on the write confirmation signal, query the integrity of the encapsulated data packet in the storage device, verify the data consistency in the fusion information processing stage by comparing the check code, determine the feasibility of the recovery path, and obtain the final lossless storage record.

[0030] Using the write confirmation signal obtained in step S500 as the trigger, a query command is invoked through the storage device interface to obtain the actual storage status (such as data block integrity, storage address matching, etc.) of the encapsulated data packet already written to the target storage device. Entering the fusion information processing stage, a preset checksum of the metadata structure in the encapsulated data packet is extracted and precisely compared with the actual checksum of the data packet in the storage device to verify whether the data was lost or tampered with during transmission and writing, ensuring data consistency. Simultaneously, based on the compression parameters and file structure feature mapping relationships in the metadata, a data recovery process is simulated to determine if the recovery path is feasible (such as the feasibility of reverse parsing of the compression algorithm, the integrity of data block sequence reconstruction, etc.). If the checksum comparison matches and the recovery path is feasible, it indicates that the data has achieved lossless storage. The system automatically generates a final lossless storage record containing the storage location, verification result, recovery parameters, and timestamp, serving as the core basis for subsequent data retrieval and recovery.

[0031] Furthermore, the lossless compression and storage method for heterogeneous data based on the information technology innovation environment proposed in this embodiment includes step S100 as follows: Step S110: By scanning the target electronic file, obtain the header identification information inside the target electronic file, extract the file type identifier and the corresponding format standard from the header identification information, and compare them using a preset rule base to obtain the preliminary classification result of the file type.

[0032] The following formula is used to quantify the complexity and information content of header identification information, helping to determine the reliability of file type identification: (1), In formula (1), This represents the header entropy of the target electronic document. This indicates the total number of bytes in the header identification information. Indicates the first The probability of each byte appearing.

[0033] The preliminary classification results are obtained using the following formula through a weighted voting mechanism: (2), In formula (2), This indicates the final file type classification result. This indicates the total number of predefined file types. This indicates the total number of rules in the rule base. Indicates the first The confidence weight of each rule Representation rules Pointing to category The indicator function is 1 if the rule supports the category, and 0 otherwise.

[0034] When processing a target electronic file named "report.pdf", the first step is to read its initial byte sequence using a file scanning tool such as a Hex editor. The header identification information is usually located in the first few bytes of the file. For example, the header identification of a PDF file is "%PDF-1.4". The file type identifier "PDF" and the corresponding format standard, such as Adobe PDF specification version 1.4, are extracted from it. Then, this information is compared with a preset rule base. This rule base is a database containing signatures and standard descriptions of common file types. For example, the rule base stores the signature pattern of PDF and its correspondence with the ISO 32000 standard. After comparison using a string matching algorithm, the preliminary classification result is "Portable Document Format". This step ensures rapid identification of common files and avoids complex subsequent parsing.

[0035] S120. Based on the preliminary classification results, analyze the internal data structure pattern of each file type to obtain the distribution characteristics and storage method of data blocks. By comparing with the preset data block template, determine the preliminary mapping relationship between file type and structural characteristics.

[0036] The initial mapping relationship between file type and structural characteristics is derived using the following formula: (3), In formula (3), Indicate file type With structural features The strength of the mapping relationship between them This represents the total number of feature dimensions used for comparison. Represents the base of the natural constant. Representation type In the Feature values ​​in each dimension Representation structure In the Feature values ​​in each dimension This represents the mapping sensitivity parameter. The function represents the calculation of feature similarity.

[0037] Based on the preliminary classification results above, for file types such as PDF, we further analyze their internal data structure patterns. Specifically, the data structure of a PDF file includes objects, cross-reference tables, and page trees. We obtain the distribution characteristics of data blocks, such as the sequential distribution of object numbers and the storage method, such as indirect references. By comparing with a preset data block template, such as a template that defines the standard object structure pattern of PDF, if a match is found, we determine the mapping relationship between the file type and the structural features, such as "PDF type corresponds to hierarchical object storage".

[0038] S130. If the structural features do not match the preset template, a byte stream analysis tool is used to disassemble the data blocks of the target electronic file segment by segment, extract the hidden format identifiers and data distribution patterns, determine whether they conform to the known extended format standards, and obtain the corrected mapping relationship.

[0039] The hidden format identifiers extracted from the target electronic document are derived using the following formula: (4), In formula (4), This indicates hidden formatting identifiers extracted from the target electronic document. Represents the set of all possible format identifiers. Indicates that given hidden data Conditional format The probability, Representation format Compared with known extended format standards The degree of conformity, This indicates the format identifier that causes the subsequent expression to take the maximum value.

[0040] The corrected mapping relationship is derived using the following formula: (5), In formula (5), This represents the corrected mapping relationship. This represents the original mapping relationship. Indicates the corrected strength coefficient. This represents the total number of data distribution patterns. Indicates the first The data distribution pattern Indicates the first Corrected weights for each distribution pattern.

[0041] If the structural features do not match the preset template, such as an abnormal distribution of objects within a disguised PDF file, a byte stream analysis tool such as Binwalk is used to break down the data blocks segment by segment. This byte stream analysis tool will divide the file byte stream into independent segments, extract hidden format identifiers such as embedded JPEG identifiers and data distribution patterns such as repeating compressed block patterns, and determine whether it conforms to known extended format standards such as PDF / A (PDF / Archive, electronic archive preservation format) extension, thereby obtaining the corrected mapping relationship, which helps to identify variant file formats.

[0042] S140. By integrating the corrected mapping relationship with the preliminary classification results, a data comparison tool is used to perform final verification of file type and structural characteristics, and obtain the complete correspondence between file format standards and data block distribution characteristics.

[0043] The correspondence between file format standards and data block distribution characteristics can be established using the following formula: (6), In formula (6), This represents the correspondence between complete file format standards and data block distribution characteristics. Indicates the total number of data blocks. Indicates the first The standard value of a file format, Indicates the first The distribution values ​​of the distribution characteristics of each data block.

[0044] By integrating the corrected mapping relationships with the preliminary classification results, data comparison tools such as Diff tools are used to perform final verification of file type and structural features. For example, the extended PDF features in the corrected relationship are compared with the initial PDF classification. After confirming consistency, the complete correspondence between file format standards such as PDF 1.7 and data block distribution features is obtained, such as "compressed image blocks correspond to page content streams." This integration can improve the accuracy of file recognition and bring higher evidence reliability in digital forensics. For example, assuming a "image.jpg" file is being processed, with the header identifier "FFD8FFE0," the type "JPEG" and the standard "JFIF" are extracted. The rule base comparison gives the preliminary result "Joint Image Experts Group format." Its data structure, such as the SOI (Start Of Image) marker and paragraph distribution, is parsed, and the template is compared to determine the mapping, such as "JPEG corresponds to segmented compressed storage." For example, if there is a mismatch, a byte stream tool is used to extract the hidden EXIF ​​(Exchangeable Image File Format) identifier and correct it to an "Extended JPEG" mapping.

[0045] The integrated verification process obtains complete correspondences such as "JPEG standard and metadata block distribution," which ensures format compatibility in media file management. For example, for a "docx" file, the header "PK0304" identifier extracts "Office Open XML," providing a preliminary classification as a "document format."

[0046] The ZIP structure and XML (eXtensible Markup Language) parts are parsed, and the template mapping "DOCX corresponds to packaged XML storage" is compared. For example, if there is a mismatch, the hidden OOXML (Office OpenXML, the open XML format for Microsoft Office documents) identifiers are extracted and the mapping is corrected. Finally, the "DOCX standard and relational file block distribution" are verified and integrated, improving document processing efficiency.

[0047] Preferably, the lossless compression and storage method for heterogeneous data under the information technology innovation environment proposed in this embodiment includes step S200: S210. Based on the mapping relationship, classify and group the files according to their type and structural features. Use a content similarity comparison tool to extract feature data from the target electronic files. By comparing with a preset threshold range, determine the initial results of the classification and grouping.

[0048] The initial results of the classification and grouping are obtained using the following formula: (7), In formula (7), Indicate file type The collection of classification and grouping results Indicates the first One electronic file, This represents the minimum value of the preset threshold. This represents the maximum value of the preset threshold. Indicates from file Feature data values ​​extracted from them Indicates belonging to type A collection of file indexes.

[0049] Based on the established mapping relationship between file types and structural features, multiple target electronic files are classified and grouped. For example, when processing a batch of mixed-format files, a content similarity comparison tool such as the Cosine similarity calculator is first used to extract feature data. This tool converts the byte sequence within a file into a vector representation and then calculates the cosine value between the vectors to measure the degree of similarity. Specifically, for a PDF file and a DOCX file, their header identifier and data block pattern are extracted as feature vectors. If the cosine similarity is higher than a preset threshold, such as 0.8, they are grouped into the same group. This ensures that the initial grouping results are based on structural similarity rather than solely on file extensions, thereby improving the accuracy of classification in digital archive management. For example, when processing corporate report files, assuming the mapping relationship shows that PDF corresponds to hierarchical object storage and DOCX corresponds to packaged XML storage, after extracting feature data using a content similarity comparison tool, if the similarity is within a threshold range, such as 0.7 to 0.9, they are initially grouped into the "document class." This helps with subsequent batch processing and avoids confusing files of different formats.

[0050] S220. Based on the initial classification and grouping results, a lossless coding tool is used to scan the repetitive pattern portion within the group to obtain the distribution pattern of the repetitive data blocks and obtain the preliminary compressed data block sequence.

[0051] The following formula is used to evaluate the complexity distribution characteristics of the data block sequence after initial compression: (8), In formula (8), Indicates the complexity of a data block sequence. This indicates the number of different repetition patterns. Indicates the first The weighting coefficients of the various patterns Indicates the first The number of times a pattern is repeated in a sequence.

[0052] Based on the initial results of the above classification and grouping, lossless encoding tools such as the LZ77 algorithm implementer are used to scan the repetitive pattern portions within the groups. These lossless encoding tools identify repetitive byte sequences through a sliding window mechanism. For example, when scanning the content streams of multiple PDF files in the "Document" group, if repetitive page tree object patterns are found, their distribution patterns, such as compressed blocks appearing at fixed byte intervals, are recorded, thus obtaining a preliminary compressed data block sequence. This reduces redundant data and improves efficiency in storage optimization. For instance, for a JPEG file group containing repetitive image elements, the distribution pattern obtained after scanning shows that compressed blocks repeat every 500 bytes. The compressed sequence then removes these repetitions, generating a more compact sequence. This connection between grouping and compression processes lies in the fact that the initial results directly guide the scanning range. For example, the initial grouping ensures that repetitive patterns are only found within files with similar structures, avoiding cross-type interference. This, in turn, enables efficient deduplication storage in media file backup, resulting in a reduction in data volume. In actual forensic analysis, for the grouping of disguised files, abnormal structures are first isolated by similarity comparison, and then repeating patterns are scanned to reveal hidden data patterns. The final compressed sequence is convenient for further decoding to verify the integrity of the evidence.

[0053] Furthermore, the lossless compression and storage method for heterogeneous data based on the information technology innovation environment proposed in this embodiment includes step S300 as follows: S310. Based on the initially compressed data block sequence, an information comparison tool is used to scan the data block distribution to obtain the distribution pattern of the redundancy pattern.

[0054] The spatial distribution of redundancy patterns in a data block sequence is described by the following formula: (9), In formula (9), Indicates the redundancy mode at location The probability distribution at that location, Indicates the length of the scan window. Indicates the first Redundancy strength parameters at each location, Indicates the first Data density at each location, It represents the base of the natural constant.

[0055] For the initially compressed data block sequence, an information comparison tool is first used to scan the distribution of the data blocks. This tool is essentially a statistical analysis-based software module. It traverses each data block in the sequence, calculating the frequency and positional intervals of repetition between blocks. For example, in digital archive management, when processing a batch of initially compressed electronic report files, the information comparison tool examines the byte blocks in the sequence one by one, identifying patterns such as repeated metadata tags or text segments. Specifically, assuming the sequence contains multiple compressed blocks, the information comparison tool generates a distribution chart, marking redundant patterns such as segments with the same hash value appearing every 100 bytes, thus deriving the overall pattern of these patterns, such as the average repetition rate or distribution density. This helps in determining whether further optimization is needed.

[0056] S320. If the distribution pattern of the redundancy pattern exceeds the preset threshold range, the data block sequence is compared segment by segment using a sequence matching tool to determine the data segments with high similarity.

[0057] The following formula is used to identify data segments with high similarity: (10) In formula (10), This represents the threshold value for sequence matching. This represents the number of common features between two data segments. Indicates the total number of features. Indicates the adjustment parameter. Indicates the current matching precision. Indicates the reference accuracy value, when If the similarity of the data segments exceeds the set value, the data segments are considered to have a high degree of similarity.

[0058] When processing enterprise financial data backup, if the redundancy pattern distribution obtained from the scan shows that the repetition rate exceeds a preset threshold, such as 0.5, the next step is triggered. The data block sequence is compared segment by segment using a sequence matching tool. This sequence matching tool is an implementation of a string comparison algorithm. It divides the sequence into segments of fixed length, such as every 256 bytes, and then calculates the edit distance or similarity score between each segment to identify data segments with high similarity. For example, in backing up multiple Excel file sequences, the sequence matching tool finds that the similarity of some numerical table segments is above 0.85, thus marking these segments as potential optimization targets.

[0059] S330. For data segments with high similarity, a sliding window tool is used to match them one by one to determine the data range that can be further compressed.

[0060] The total matching degree of the sliding window matching is obtained by the following formula: (11), In formula (11), This represents the total matching degree of the sliding window. Indicates the total length of the target data sequence. Indicates the size of the modal window. Indicates the current sliding position. This represents the offset within the window. Indicates the position in the target sequence Data, Indicates position in the modal window Data, The matching degree function for two data points.

[0061] For these highly similar data segments, a sliding window tool is used for one-by-one matching. This sliding window tool dynamically adjusts the window size, such as gradually expanding from 16 bytes to 64 bytes, to compare the subsequences within the segment one by one and determine the data range that can be further compressed. For example, in media file storage services, for similar image data segments, the tool identifies continuous pixel pattern ranges after sliding comparison. These ranges are determined to be compressible because they contain highly repetitive color value sequences.

[0062] S340. By using a fusion processing tool to integrate data ranges that can be further compressed, and applying a sequence optimization method to reorganize the data blocks, optimized compressed data is obtained.

[0063] The formula integrates the outputs of multiple fusion processing tools through a weighted average, achieving effective integration of data ranges that can be further compressed. (12), In formula (12), Indicates the first The result of fusion processing of data intervals, This indicates the total number of available fusion tools. Indicates the first The weighting coefficients of each fusion tool, Indicates the first A tool for data range The compression capacity assessment value.

[0064] The following formula is used to evaluate the overall performance of the compressed data obtained after fusion processing and sequence optimization: (13) In formula (13), This represents the final optimized compressed data quality metric. Indicates the size of the original data. Indicates the size of the compressed data. This represents the compression efficiency coefficient. This represents the data integrity preservation factor.

[0065] These data intervals that can be further compressed are integrated by a fusion processing tool. This fusion processing tool is a data merging module that merges adjacent or similar intervals into larger blocks. For example, it uses a joint algorithm to connect scattered intervals and then applies a sequence optimization method to reorganize the data blocks to obtain optimized compressed data. For example, in cloud storage services, the reorganized sequence removes redundant information and generates a more compact final data stream, which ensures improved storage efficiency in practice.

[0066] Preferably, the lossless compression and storage method for heterogeneous data under the information technology innovation environment proposed in this embodiment includes step S400 as follows: S410. Based on the optimized compressed data, a verification tool is used to generate the corresponding data verification code. At the same time, the compression parameters are obtained, and the mapping relationship between the data verification code and the compression parameters is determined.

[0067] The following formula is used to generate a comprehensive check code through XOR operations of multiple check algorithms: (14) In formula (14), This represents the generated data verification code. This represents the hash function operation on compressed data. This indicates the calculation of cyclic redundancy check for compressed data. The first part represents the compressed data. 1 bit This indicates the total number of bits in the compressed data.

[0068] The mapping relationship between the data checksum and compression parameters is derived using the following formula: (15) In formula (15), Indicates data verification code With compression parameters The mapping relationship function between them , , Represents the mapping weight coefficients. This indicates that the checksum is obtained by modulo the size of the parameter set. This represents a hash operation on the compression parameters. This represents the encoded value of the compression parameters. This indicates a bitwise AND operation.

[0069] To optimize compressed data, a verification tool is first used to generate a corresponding data checksum. This verification tool is a software component based on a hash function. It scans the data block byte by byte and calculates a checksum such as MD5 (Message-Digest Algorithm 5) or SHA-256 (Secure Hash Algorithm 256-bit) to ensure data integrity. At the same time, it extracts parameters such as compression ratio and algorithm type from the compression process. For example, when processing backups of corporate financial reports, the verification tool scans the optimized data stream, generates a 128-bit checksum, and records the compression parameters, including the LZW (Lempel-Ziv-Welch) algorithm used and the compression ratio of 1.5, thereby establishing a correspondence between the two. Specifically, this mapping relationship is determined through a key-value pair structure, where the checksum is the key and the compression parameter is the value, forming a simple association table. For example, in digital archive management, if the data checksum is a specific hash string, the corresponding parameter is displayed as the gzip (GNU zip, a file compression program from GNU free software) method and a ratio of 0.7, which helps with subsequent verification and recovery.

[0070] S420. Regarding the mapping relationship between data verification codes and compression parameters, a construction tool is used to integrate them into a unified metadata structure. If the format of the metadata structure does not meet the preset threshold, it is corrected by an adjustment tool to obtain a standardized metadata structure.

[0071] The following formula describes the process of integrating the mapping relationship between checksums and compression parameters into a unified metadata structure using a construction tool: (16) In formula (16), This represents a unified metadata structure. Represents a set of data verification codes. Represents a set of compression parameters. This represents the mapping function between the checksum and the compression parameters.

[0072] The following formula describes the process of correcting substandard metadata structures using tuning tools to obtain standardized results: (17) In formula (17), This represents the standardized metadata structure. This represents the original metadata structure. This represents the correction function for the adjustment tool. This represents the correction parameter vector.

[0073] Regarding the mapping relationship between data checksums and compression parameters, the build tool integrates them into a unified metadata structure. This build tool is an XML or JSON formatted module that organizes the mapping information into a hierarchical tree structure. For example, in the case of media file storage, the build tool packages the checksum and parameters into a JSON object, including fields such as "checksum" and "params". If the structure format, such as the field length, does not reach the preset threshold, such as the minimum of 100 bytes, the adjustment tool will intervene and correct it by adding padding or reconstructing fields to obtain a standardized metadata structure.

[0074] S430: Bind the standardized metadata structure with optimized compressed data, use the encapsulation tool to generate the initial encapsulated data packet, determine whether it meets the cross-platform transmission requirements, and obtain the compatibility of the initial encapsulated data packet.

[0075] The compatibility of the initial encapsulated data packet is determined by the following formula: (18) In formula (18), This represents the cross-platform compatibility index of the initially encapsulated data packet. Indicates the total number of target platforms. Indicates the first Compatibility ratings for each platform Indicates the first Weighting factors for each platform Indicates the first Transmission requirement threshold for each platform, This represents the standardization factor for cross-platform transmission.

[0076] In cloud storage services, after the standardized metadata structure is bound to the optimized compressed data, the encapsulation tool generates the initial encapsulated data packet. This encapsulation tool is an implementation of a packaging algorithm. It places the metadata in the header of the data packet, followed by the compressed data body, forming a complete file packet. Then, it determines whether it meets the cross-platform transmission requirements, such as checking whether it supports decoding for Windows and Linux. If the compatibility score is below 0.8, it is considered not to meet the standard.

[0077] S440. If the compatibility of the initial encapsulated data packet does not meet the preset standard, the transmission protocol is adjusted through the protocol adaptation tool to obtain the final encapsulated data packet.

[0078] The following formula describes the process of protocol adaptation of data packets using a protocol adaptation tool: (19) In formula (19), This indicates the encapsulated data packet after being adjusted by the protocol adaptation tool. Indicates the protocol adaptation function. Indicates the original data packet. This represents the vector of protocol parameters that need to be adjusted. This represents the transport protocol conversion function. This indicates a data packet re-encapsulation operation.

[0079] If the initial packaged data packet lacks compatibility, the protocol adaptation tool will adjust the transmission protocol. This protocol adaptation tool is a network protocol conversion module that switches from HTTP (Hypertext Transfer Protocol) to FTP (File Transfer Protocol) or adds a compatibility layer. For example, when processing the transmission of multiple versions of electronic reports, if the protocol adaptation tool detects that the packet fails to be parsed on the iOS platform, it will switch to using SFTP (Secure File Transfer Protocol) to ensure cross-device compatibility of the packet and finally obtain the adjusted packaged data packet. This maintains the continuity of data processing in the actual backup process.

[0080] Furthermore, the lossless compression and storage method for heterogeneous data based on the information technology innovation environment proposed in this embodiment includes step S500: S510. Obtain storage path information from the encapsulated data packet, use a path parsing tool to perform format verification on the storage path information, and if it does not meet the preset threshold, adjust it using a format conversion tool to obtain standardized path data.

[0081] Standardized path data is derived using the following formula: (20) In formula (20), This represents the standardized path data obtained after processing by a format conversion tool. Indicates the original storage path information. This represents a set of format conversion rules. This indicates that the parameters are being adjusted. This represents the conversion function of the format conversion tool.

[0082] The process of obtaining storage path information from the encapsulated data packet involves parsing the metadata header within the packet. For example, in enterprise financial report backup operations, path information may be embedded in a JSON structure. The path parsing tool is a software module based on regular expressions. It first extracts the path string, such as " / backup / finance / 2023 / report.dat", and then checks whether its format conforms to preset rules, such as the path length not exceeding 256 characters and not containing illegal symbols. If the path contains backslashes instead of forward slashes, the tool determines that it does not meet the threshold, such as a format consistency score below 0.9. At this point, the format conversion tool intervenes. This is a string processing component that replaces backslashes with forward slashes and normalizes the delimiters, ultimately obtaining standardized path data such as " / backup / finance / 2023 / report.dat", which ensures compatibility in subsequent transmissions.

[0083] The format verification mechanism of the aforementioned path parsing tool is based on a preset threshold principle, which defines a scoring system, for example, a threshold of 0.85. The path parsing tool checks path components such as directory depth and character encoding item by item. If the depth exceeds 5 levels, points are deducted. If it is below the threshold, the conversion tool is triggered to adjust. This maintains the standardization of paths when handling multi-platform backups, ensuring seamless migration from Windows to Linux. The specific adjustment process of the format conversion tool includes parsing the path string, identifying non-standard elements such as mixed case, and unifying them to lowercase, resulting in data such as " / archives / media / 2023 / video.mp4". This is connected with the aforementioned protocol matching, because the standard path directly affects the choice of protocol, avoiding transmission failures caused by path errors.

[0084] S520. Based on standardized path data, a protocol adaptation tool is used to match the transmission protocol to the interface characteristics of the target storage device, determine whether the security requirements are met, and obtain the adapted protocol configuration.

[0085] The adapted protocol configuration is derived using the following formula: (twenty one), In formula (21), Indicates the optimal protocol configuration. Represents the set of all possible protocol configurations. A factor representing the balance between performance and compatibility. Indicates configuration Reliability rating Indicates configuration Performance rating This indicates the parameter selection that maximizes the objective function. The control logic of formula (21) is based on two objectives (reliability) and performance A weighted trade-off is made among these factors to determine the optimal protocol configuration. .

[0086] Based on standardized path data, the protocol adaptation tool performs transmission protocol matching based on the interface characteristics of the target storage device. For example, in a digital archive management scenario, the tool analyzes whether the interface of the device pointed to by the path, such as a NAS (Network Attached Storage) server, supports the SMB (Server Message Block) protocol. Then, it matches it with protocols such as FTP or HTTPS. To determine whether the security requirements are met, it checks whether the protocol is encrypted. If not, such as lacking a TLS layer, it adjusts to add a security layer and obtains the adapted protocol configuration, such as "HTTPS with TLS 1.3".

[0087] Protocol adaptation tools involve detailed analysis of interface characteristics when matching transport protocols. For example, for interfaces of S3 (Amazon Simple Storage Service) compatible storage devices, the tool queries the API (Application Programming Interface) documentation to confirm the list of supported protocols. Then, it makes a judgment based on security requirements, such as the requirement for authentication tokens. If the initial match is HTTP but the security is insufficient, it adapts to a configuration that adds an OAuth (Open Authorization) layer. This ensures the confidentiality of data in electronic report transmissions and provides a foundation for subsequent transmissions.

[0088] S530, according to the adapted protocol configuration, transmits the encapsulated data packet to the target storage device.

[0089] The probability of successfully transmitting an encapsulated data packet to the target storage device is calculated using the following formula: (twenty two), In formula (22), This indicates the probability that a data packet was successfully transmitted to the target storage device. This indicates the total number of nodes in the transmission path. Indicates the first Failure rate of each node Indicates the first The data processing capacity of each node This indicates the receive fault tolerance parameters of the storage device. Indicates the first The success probability of each node, the control logic of formula (22) is to apply it to all nodes. The success probabilities of each step are multiplied together to obtain the final probability of the encapsulated data packet being successfully transmitted to the target storage device. .

[0090] The process of transmitting encapsulated data packets to the target storage device through the adapted protocol configuration includes initializing the connection and sending data. For example, in media file storage services, the configured HTTPS protocol is used to push the packet body to the cloud server to ensure that the data is not intercepted during transmission.

[0091] The process of transmitting encapsulated data packets can be extended to chunked transmission. For example, when processing large backup files, the protocol configuration guidance tool can split the packet into multiple chunks, with each chunk being transmitted to the device independently. This enhances the reliability of the transmission and is linked to a write acknowledgment feedback mechanism, as the acknowledgments of each chunk accumulate to form an overall signal.

[0092] S540. Obtain the write confirmation feedback returned by the target storage device, use a verification tool to judge the legality of the write confirmation feedback, and obtain the write confirmation signal.

[0093] The write confirmation signal is derived using the following formula: (twenty three), In formula (23), This indicates the final write confirmation signal generated. The score representing the legality assessment. This represents the threshold for legality judgment; when the legality judgment score is... Greater than or equal to the legality threshold If the output is 1, a confirmation signal is output; otherwise, 0 is output. The control logic of formula (23) is to achieve binary decision-making through a conditional judgment: if the final generated write confirmation signal is 1, then the output is 0. Greater than or equal to the threshold If the write confirmation feedback is valid, then output the following: If the final write confirmation signal is generated Less than the threshold If the write confirmation feedback is invalid, then output the following: .

[0094] After receiving the write confirmation feedback from the target storage device, the verification tool checks the validity of the feedback. The verification tool is a signature verification module that compares the digital signature in the feedback with the expected value. If they match, a write confirmation signal is obtained. In cloud storage services, feedback such as "Write Success with ID: 12345" is confirmed to be valid after verification.

[0095] The validation tool verifies the validity of the write confirmation feedback by parsing the feedback structure, such as XML tags, checking whether the timestamp is within the expected range, and considering the feedback as invalid if the delay exceeds 10 seconds. After receiving the confirmation signal, the business process can continue. For example, in document management, this signal triggers the update of archived logs.

[0096] Preferably, the lossless compression and storage method for heterogeneous data under the information technology innovation environment proposed in this embodiment includes step S600 as follows: S610. Obtain the storage device identifier from the write confirmation signal, use a query tool to locate the data on the target storage device, and obtain the storage location of the encapsulated data packet.

[0097] The storage device identifier can be obtained from the write acknowledgment signal using the following formula: (twenty four), In formula (24), This indicates the storage device identifier extracted from the write confirmation signal. This represents the response function for the write confirmation signal. Represents a timestamp. This function is used to generate a unique device identifier.

[0098] The storage location of the encapsulated data packet is determined by the following formula: (25), In formula (25), Indicates the storage location of the encapsulated data packet. This indicates the scan results of the storage blocks. This represents the offset calculated based on the packet header information.

[0099] The process of obtaining the storage device identifier from the write confirmation signal involves parsing specific fields within the signal. For example, in enterprise data backup operations, the signal may contain a device ID (Identifier) ​​such as "NAS-001". The query tool is a database query module that connects to the storage management system based on this ID. It first sends a query request to locate the location of the data packet. Specifically, the query tool scans the device's directory structure, matches the metadata tags of the data packet, and finally obtains the location information such as " / data / backup / 2023 / file.pkg".

[0100] S620. Based on the storage location of the encapsulated data packet, a comparison tool is used to extract the checksum information of the encapsulated data packet. If the checksum of the encapsulated data packet does not match the preset threshold, an integrity correction is performed using a repair tool to determine the integrity status of the data packet.

[0101] The following formula is used to calculate and extract checksum information from encapsulated data packets at a specified storage location using a comparison tool: (26) In formula (26), This represents the checksum value extracted from the encapsulated data packet. Indicates the total length of the data packet. Indicates the first in the data packet Data bytes at each location, Indicates the first Hash operation performed on each data byte The modulus base value represents the check code.

[0102] The following formula is used to determine whether the checksum of the encapsulated data packet does not match a preset threshold, thereby deciding whether a repair operation is needed: (27) In formula (27), This indicates the activation flag for the repair tool. A value of 1 activates integrity correction, while a value of 0 indicates that no repair is needed. This represents the checksum value of the current data packet. This indicates the preset checksum threshold. This indicates the allowable error range.

[0103] The following formula is used to comprehensively evaluate and determine the final integrity status of the entire data packet after the repair tool has completed integrity correction: (28) In formula (28), This represents the complete state assessment value of the data packet. Indicates the total number of data segments detected. Indicates the first The weighting coefficients of each data segment. Indicates the first The integrity verification results of each data segment.

[0104] Based on the storage location of the acquired encapsulated data packet, the process of extracting the checksum information of the encapsulated data packet using a comparison tool includes reading the tail checksum segment of the data packet. For example, in a digital archive storage scenario, the comparison tool is a hash calculation component that extracts an MD5 checksum such as "a1b2c3d4e5f6g7h8" and then compares it with a preset threshold. If the hash value difference exceeds a similarity threshold of 0.1, the repair tool is activated. This is a data recovery module that corrects the integrity by recalculating and replacing the damaged parts, determining the integrity status of the data packet as "repaired".

[0105] S630. Based on the complete status of the data packet, obtain the comparison record in the fusion information processing stage, use the verification tool to judge the data consistency, and obtain the consistency verification result.

[0106] The consistency verification result is obtained using the following formula: (29) In formula (29), This indicates the final data consistency verification result. This indicates the total number of test types executed by the verification tool. Indicates the first The number of errors found in class tests Indicates the first Total number of checks in class tests Indicates the first Importance index of class tests.

[0107] Based on the complete state of the data packet, the process of obtaining the comparison records in the fusion information processing stage requires understanding the fusion information processing stage first. This is a data integration stage, in which the comparison records are log files generated in the previous processing. For example, in media content management, the verification tool is a consistency check software. It loads these records such as "comparison log: hash_match=true", and then judges whether the data blocks are consistent item by item, and obtains the verification result such as "consistency passed".

[0108] S640. Based on the consistency verification results, the feasibility of the recovery path is evaluated using a path analysis tool. If the path meets the preset threshold, the final lossless storage record is generated. The final metrics for generating lossless storage records are derived using the following formula: (30) In formula (30), Indicates the generation metrics for lossless storage records. Indicates the completeness of the original data. Indicates the integrity of the recovered data. Indicates the data integrity index. Indicates the minimum integrity requirement. Indicates the quality assessment value. Indicates standard quality requirements. The Dirac function indicates that the quality meets the standards. The control logic of formula (30) is to calculate the non-destructive loss by comprehensively considering the indicators of three dimensions. Indicates the dimension of data consistency. Represents the integrity dimension. The quality deviation dimension is represented by the minimum value of the two dimensions, "data volume consistency" and "integrity," which is then combined with the impact of "quality deviation" to obtain the generation index of lossless storage records. The smaller the generation index of lossless storage records, the better the losslessness of the data in terms of dimensions, integrity, and quality; the larger the generation index of lossless storage records, the more severe the loss.

[0109] Based on the consistency verification results, the process of evaluating the feasibility of the recovery path using a path analysis tool involves path simulation. For example, in cloud file recovery services, a path analysis tool is a routing evaluation component that checks the availability of a recovery path such as " / restore / media / 2023 / video.mp4", including verifying node connectivity and permissions. If the path's evaluation score exceeds a preset threshold such as 0.9, a final lossless storage record is generated. This record is in the form of a JSON file, containing all verification details to support subsequent auditing.

[0110] Please see Figure 2This embodiment provides a lossless compression storage system for heterogeneous data under the domestic IT innovation environment, used to execute the aforementioned lossless compression storage method for heterogeneous data under the domestic IT innovation environment. It includes a mapping relationship acquisition module 10, a data block sequence determination module 20, an optimized compression data acquisition module 30, a packaged data packet formation module 40, a write confirmation signal acquisition module 50, and a lossless storage record acquisition module 60. The mapping relationship acquisition module 10 is used to scan the target electronic file, extract the file type identifier and internal data structure pattern from the target electronic file, and obtain the mapping relationship between file type and structural features. The mapping relationship includes file format standards and data block distribution characteristics. The data block sequence determination module 20 is used to classify and group file types and structural features according to the mapping relationship, perform preliminary compression on the repetitive pattern part using a lossless encoding method, and determine the data block sequence after preliminary compression. The classification and grouping are based on file content similarity. The optimized compression data acquisition module 30 is used to determine the data block sequence after preliminary compression if... If redundant patterns exist in the data block sequence, a similarity threshold between the data block sequences is compared during the fusion information processing stage to determine the pattern. This fusion information processing stage involves data pattern matching optimization, followed by the application of a sequence matching algorithm for sliding window matching to obtain optimized compressed data. A data packet forming module 40 generates a metadata structure containing data checksums and compression parameters for the optimized compressed data. The optimized compressed data and the metadata structure are merged and encapsulated to form a packaged data packet, which supports cross-platform transmission. A write confirmation signal acquisition module 50 parses the storage path information from the packaged data packet and transmits it to the target storage device using a secure and compatible platform write protocol to obtain a write confirmation signal. A lossless storage record acquisition module 60 queries the integrity of the packaged data packet in the storage device based on the write confirmation signal, verifies data consistency through checksum comparison during the fusion information processing stage, determines the feasibility of the recovery path, and obtains the final lossless storage record.

[0111] This embodiment provides a lossless compression and storage method and system for heterogeneous data in a domestically developed information technology environment. Compared with existing technologies, it first extracts the mapping relationship between file type and structural features by scanning target files, classifies and groups them, and then performs preliminary lossless compression. For redundant patterns, it further optimizes the compressed data using sliding window matching and sequence matching algorithms. Subsequently, it generates a metadata structure containing checksums and compression parameters, encapsulates it into a data packet that supports cross-platform transmission, and finally transmits it to the target storage device through a secure write protocol. The checksum is used to verify data consistency and the feasibility of recovery paths. This embodiment significantly improves data compression efficiency, ensures cross-platform transmission compatibility and the integrity of stored data, and provides efficient and reliable technical support for the processing of complex electronic documents.

[0112] Although preferred embodiments of the invention have been described, those skilled in the art, upon learning the basic inventive concept, can make other changes and modifications to these embodiments. Therefore, the appended claims are intended to be interpreted as including both the preferred embodiments and all changes and modifications falling within the scope of the invention. Clearly, those skilled in the art can make various alterations and modifications to the invention without departing from its spirit and scope. Thus, if these modifications and modifications of the invention fall within the scope of the claims and their equivalents, the invention is also intended to include these modifications and modifications.

Claims

1. A lossless compression storage method based on heterogeneous data in a Xinhua environment, characterized in that, The method comprises the following steps: S100, extracting a file type identifier and an internal data structure mode from the target electronic file by scanning the target electronic file, to obtain a mapping relationship between the file type and the structure characteristics, wherein the mapping relationship comprises a file format standard and data block distribution characteristics; S200, classifying and grouping the file type and the structure characteristics according to the mapping relationship, performing preliminary compression on a repeated mode part by using a lossless coding method, and determining a data block sequence after preliminary compression, wherein the classification and grouping are based on file content similarity; S300, if there is a redundant mode in the data block sequence after preliminary compression, judging in a fusion information processing link by comparing a similarity threshold value between data block sequences, wherein the fusion information processing link involves data mode matching optimization, and then applying a sequence matching algorithm to perform sliding window matching to obtain optimized compression data; S400, generating a metadata structure containing a data check code and compression parameters for the optimized compression data, merging and packaging the optimized compression data and the metadata structure to form a packaged data packet, and supporting cross-platform transmission of the packaged data packet; S500, parsing storage path information from the packaged data packet, transmitting the packaged data packet to a target storage device by using a safe and compatible platform writing protocol, and obtaining a writing confirmation signal; S600, querying the integrity of the packaged data packet in the storage device according to the writing confirmation signal, verifying data consistency in the fusion information processing link by check code comparison, judging the feasibility of a recovery path, and obtaining a final lossless storage record.

2. The lossless compression storage method for heterogeneous data based on the Xinwen environment according to claim 1, characterized in that, Step S100 comprises: S110, obtaining head identifier information inside the target electronic file by scanning the target electronic file, extracting a file type identifier and a corresponding format standard from the head identifier information, comparing by using a preset rule base, and obtaining a preliminary classification result of the file type; S120, according to the preliminary classification result, analyzing a file internal data structure mode for each type of file type, obtaining distribution characteristics and storage modes of data blocks, comparing a preset data block template, and determining a preliminary mapping relationship between the file type and the structure characteristics; S130, if the structure characteristics do not match the preset template, using a byte stream analysis tool to segmentally disassemble data blocks of the target electronic file, extracting hidden format identifiers and data distribution rules therefrom, judging whether they conform to known extended format standards, and obtaining a corrected mapping relationship; S140, integrating the corrected mapping relationship and the preliminary classification result, using a data comparison tool to finally verify the file type and the structure characteristics, and obtaining a complete corresponding relationship between the file format standard and the data block distribution characteristics.

3. The lossless compression storage method for heterogeneous data based on the Xinwen environment according to claim 1, characterized in that, Step S200 comprises: S210, according to the mapping relationship, classifying and grouping the file type and the structure characteristics, extracting feature data from the target electronic file by using a content similarity comparison tool, comparing a preset threshold range, and determining an initial result of the classification and grouping; S220, according to the initial result of the classified grouping, a lossless coding tool is used to scan the repeated mode part in the grouping, the distribution rule of the repeated data block is obtained, and a preliminary compressed data block sequence is obtained.

4. The lossless compression storage method for heterogeneous data based on the Xinwen environment according to claim 1, characterized in that, Step S300 includes: S310, according to the preliminary compressed data block sequence, an information comparison tool is used to scan the data block distribution, and the distribution rule of the redundant mode is obtained; S320, if the distribution rule of the redundant mode exceeds the preset threshold range, a sequence matching tool is used to compare the data block sequence segment by segment, and a data segment with high similarity is determined; S330, for the data segment with high similarity, a sliding window tool is used for one-by-one matching, and a data interval that can be further compressed is determined; S340, the data interval that can be further compressed is integrated by a fusion processing tool, a sequence optimization method is applied to recombine the data block, and optimized compressed data is obtained.

5. The lossless compression storage method for heterogeneous data based on the Xinwen environment according to claim 1, characterized in that, Step S400 includes: S410, according to the optimized compressed data, a check tool is used to generate a corresponding data check code, and a compression parameter is obtained, and a mapping relationship between the data check code and the compression parameter is determined; S420, for the mapping relationship between the data check code and the compression parameter, a construction tool is used to integrate into a unified metadata structure, if the format of the metadata structure does not reach the preset threshold, an adjustment tool is used for correction, and a standardized metadata structure is obtained; S430, the standardized metadata structure is bound with the optimized compressed data, an encapsulation tool is used to generate an initial encapsulation data packet, whether it meets the cross-platform transmission requirement is judged, and the compatibility of the initial encapsulation data packet is obtained; S440, if the compatibility of the initial encapsulation data packet does not meet the preset standard, a protocol adaptation tool is used to adjust the transmission protocol, and a final encapsulation data packet is obtained.

6. The lossless compression storage method for heterogeneous data based on the Xinwen environment according to claim 1, characterized in that, Step S500 includes: S510, the storage path information is obtained from the encapsulation data packet, a path analysis tool is used to perform format checking on the storage path information, if it does not meet the preset threshold, a format conversion tool is used for adjustment, and standardized path data is obtained; S520, according to the standardized path data, a protocol adaptation tool is used to match the transmission protocol according to the interface characteristics of the target storage device, whether it meets the safety requirement is judged, and the adapted protocol configuration is obtained; S530, according to the adapted protocol configuration, the encapsulation data packet is transmitted to the target storage device; S540, the write confirmation feedback returned by the target storage device is obtained, a verification tool is used to judge the legality of the write confirmation feedback, and a write confirmation signal is obtained.

7. The lossless compression storage method for heterogeneous data based on the Xinwen environment according to claim 1, characterized in that, Step S600 includes: S610, the storage device identifier is obtained from the write confirmation signal, a query tool is used to locate the data of the target storage device, and the storage position of the encapsulation data packet is obtained; The storage device identifier is obtained from the write confirmation signal by the following formula: ， wherein, represents a storage device identification extracted from the write confirmation signal, represents a response function of the write confirmation signal, represents a time stamp, the function serves to generate a unique device identification code; The storage position of the encapsulation data packet is obtained by the following formula: ， wherein, represents a storage location of the encapsulated data packet, represents a scanning result of the storage block, represents an offset calculated based on the data packet header information; S620, according to the storage location of the packaged data packet, the check code information of the packaged data packet is extracted by using the comparison tool, if the check code of the packaged data packet does not match the preset threshold, integrity correction is carried out through the repair tool to determine the integrity state of the data packet; The following formula is used to calculate and extract the check code information from the packaged data packet in the specified storage location by the comparison tool: ， wherein, represents a check code value extracted from the package data, represents the total length of the package data, represents a data byte at the i-th position in the package data, represents a hash operation on the i-th data byte, represents a hash operation on the i-th data byte, represents a hash operation on the i-th data byte, represents a modulus reference value of the check code; The following formula is used to determine whether the check code of the packaged data packet matches the preset threshold, so as to decide whether the repair operation is needed: ， wherein, represents a flag for starting the repair tool, and when the value is 1, integrity correction is started, and when the value is 0, repair is not needed; represents a check code value of a current data packet, represents a preset check code threshold value, represents an allowable error range; The following formula is used to comprehensively evaluate and determine the final integrity state of the entire data packet after the integrity correction is completed by the repair tool: ， wherein, represents a complete state evaluation value of the data packet, represents a total number of detected data segments, represents a weight coefficient of the th data segment, represents an integrity verification result of the th data segment; S630, according to the integrity state of the data packet, the comparison record in the fusion information processing link is obtained, the data consistency is judged by using the verification tool, and the consistency verification result is obtained; S640, according to the consistency verification result, the feasibility of the recovery path is evaluated by using the path analysis tool, if the path meets the preset threshold, the final lossless storage record is generated.

8. The lossless compression storage method of heterogeneous data based on Xinwen environment according to claim 7, characterized in that, In step S630, the consistency verification result is obtained by the following formula: ， wherein, represents the final data consistency verification result, represents the total number of test types performed by the verification tool, represents the number of errors found in the first class test, represents the total number of checks in the first class test, represents the importance index of the first class test.

9. The lossless compression storage method for heterogeneous data based on the Xinwen environment according to claim 8, characterized in that, In step S640, the generation index of the final lossless storage record is obtained by the following formula: ， wherein, represents a lossless storage record generation indicator, represents an original data completeness, represents a post-recovery data completeness, represents a data integrity index, represents a minimum integrity requirement, represents a quality assessment value, represents a standard quality requirement, represents a Dirac function ensuring quality compliance.

10. A lossless compression storage system for heterogeneous data based on a Xing creation environment, configured to perform the lossless compression storage method for heterogeneous data based on a Xing creation environment according to any one of claims 1 to 9. It includes: The mapping relationship acquisition module (10) is used to extract the file type identifier and internal data structure mode from the target electronic file by scanning the target electronic file, and obtain the mapping relationship of file type and structure characteristics, wherein the mapping relationship includes file format standard and data block distribution characteristics; The data block sequence determination module (20) is used to classify and group the file type and structure characteristics according to the mapping relationship, perform preliminary compression on the repeated mode part by using lossless coding method, determine the data block sequence after preliminary compression, and the classification and grouping is based on file content similarity; The optimized compression data acquisition module (30) is used to judge in the fusion information processing link by comparing the similarity threshold between data block sequences, if there is redundant mode in the data block sequence after preliminary compression, wherein the fusion information processing link involves data mode matching optimization, and then the sequence matching algorithm is applied to sliding window matching to obtain the optimized compression data; The packaged data packet formation module (40) is used to generate the metadata structure containing data check code and compression parameters for the optimized compression data, and form the packaged data packet by merging and packaging the optimized compression data and the metadata structure, the packaged data packet supports cross-platform transmission; The write confirmation signal acquisition module (50) is used to parse the storage path information from the packaged data packet, transmit the packaged data packet to the target storage device by using the safe and compatible platform write protocol, and obtain the write confirmation signal; The lossless storage record acquisition module (60) is used to query the integrity of the packaged data packet in the storage device according to the write confirmation signal, verify the data consistency in the fusion information processing link by check code comparison, judge the feasibility of the recovery path, and obtain the final lossless storage record.

Images