Data compression method based on shipbuilding mobile management platform

Abstract

The invention discloses a data compression method based on a shipbuilding mobile management platform, and belongs to the technical field of data processing. In order to solve the problems of low data transmission efficiency, slow data processing speed and inconvenient data management and use, the method comprises the following steps of characteristic analysis, compression algorithm selection, adaptive data segmentation, data classification, data compression and mobile management platform embedding. A compression algorithm is selected according to data characteristics, a self-adaptive data segmentation strategy is created according to the data characteristics, different types of data are classified so as to further improve the compression efficiency, and meanwhile, an efficient data compression and decompression module is designed and embedded into a mobile management platform. And a worker can check and process the compressed data at any time through the mobile management platform, and real-time compression and decompression of the data are ensured, so that the storage and transmission cost is remarkably reduced on the premise that the data integrity is not lost.

Description

Technical Field

[0001] The present invention relates to the technical field of data processing, and in particular to a data compression method based on a shipbuilding mobile management platform. Background Art

[0002] Shipbuilding is a complex project involving multiple fields, covering the entire process from ship design to actual construction. In this process, the designer will determine the structure, size and performance parameters of the ship based on factors such as the purpose of the ship, sailing conditions and customer needs. Subsequently, the construction team will use steel, aluminum alloy and other materials based on these design parameters, and gradually complete the construction of the hull, shell, frame and other parts through welding, cutting, forming and other processes.

[0003] During the shipbuilding process, a large amount of data is generated. This data covers information from raw material procurement, production progress, quality control to final delivery. This data is crucial for shipbuilding companies. It not only helps monitor production progress and optimize production processes, but also provides strong support for quality control and after-sales service.

[0004] However, there are still the following problems in actual operation: Uncompressed ship construction data may be very time-consuming to transmit, especially when the data volume is huge. The limitation of network bandwidth may lead to low transmission efficiency. A large amount of uncompressed data will increase the burden on computers to process data. Traditional data processing methods may not be able to fully utilize system resources, resulting in waste of resources or low processing efficiency. Summary of the invention

[0005] The purpose of the present invention is to provide a data compression method based on a shipbuilding mobile management platform to solve the problems raised in the above background technology.

[0006] To achieve the above object, the present invention provides the following technical solution: a data compression method based on a shipbuilding mobile management platform, comprising the following steps: Characteristic analysis, obtaining ship construction data and performing data analysis on the ship construction data, preprocessing the ship construction data, and caching the ship construction data according to the data type after preprocessing; Compression algorithm selection, based on the analysis results of characteristic analysis, different compression algorithms are selected for different types of ship construction data; Adaptive data segmentation: segment the shipbuilding data into several subsets based on the data type and data volume. When processing text data, segmentation is based on paragraphs, and when processing image and video data, segmentation is based on key frames. Data classification processing: according to the data type and compression requirements obtained after feature analysis, compression algorithm selection and adaptive data segmentation, the data is classified into different categories, and different compression strategies are generated based on the data categories; Data compression: construct a data compression module and compress the data subsets in the shipbuilding data based on the compression strategy, and store the compressed data in the compressed data storage module. When performing data compression, multi-threaded processing is used to simultaneously process the compression and decompression tasks of multiple data subsets; Embedded in the mobile management platform, the data compression module is embedded in the ship construction mobile management platform. The ship construction data is collected, compressed, transmitted, stored and decompressed in real time through the mobile management platform. The staff can view and process the compressed data at any time through the mobile management platform.

[0007] Furthermore, the characteristic analysis further includes the following steps: Collect shipbuilding data from all aspects of shipbuilding, and distinguish the shipbuilding data into key data and non-key data, wherein the key data includes design drawings and process parameters, and the non-key data includes other data except design drawings and process parameters; Perform type analysis on the collected non-key data, and visualize the proportion, distribution and storage characteristics of the non-key data through data visualization tools based on the data type, wherein the data type includes text data, image data, video data and structured data, and the visualization is performed by drawing one or more of a bar chart, a pie chart and a graph, and at the same time, the text data is counted for word frequency and keywords are extracted through a text analysis tool; Analyze the correlation between various links in ship construction data, identify redundant information in ship construction data, and use data deduplication and compression algorithms to identify and remove redundant information; Based on the data type of the ship construction data, key data and non-key data are cached by type.

[0008] Furthermore, the compression algorithm selection specifically includes the following steps: Select different compression algorithms based on the type of shipbuilding data; Wherein, a lossless compression algorithm is used for key data, and the lossless compression algorithm is one of the LZ77 compression algorithm and the Huffman coding compression algorithm; A lossy compression algorithm is used for non-critical data, and the lossy compression algorithm includes Huffman coding compression algorithm, H.264 compression algorithm and column storage. Among them, Huffman coding compression algorithm is selected when processing text data; H.264 compression algorithm is selected when processing image data and video data; column storage is selected when processing structured data.

[0009] Furthermore, the adaptive data segmentation further comprises the following steps: Identifying paragraphs in text data, wherein the paragraphs are identified by identifying line breaks, indents, and paragraph marks in the text; Parse the content of each paragraph in the text data, wherein the content parsing is achieved through keyword extraction and topic identification; Based on the length of the paragraphs in the text data and the data volume of the entire text, a segmentation point is determined, and the text data in the ship construction data is segmented into a plurality of subsets based on the segmentation point; Decode and convert the video data into a format, extract key frames from the video, and perform content evaluation on the extracted key frames, wherein the content evaluation is achieved through image recognition and target detection; According to the content and data size of the key frames, the video data in the ship construction data are divided into several subsets, wherein each subset contains a series of key frames and some video frames between the key frames.

[0010] Furthermore, the adaptive data segmentation further comprises the following steps: For large ship construction data files and data streams, after completing data segmentation, the subsets are compressed in blocks; For small ship construction data files and short data segments, overall compression is performed. At the same time, the segmentation strategy of each subset of ship construction data blocks is dynamically adjusted according to the real-time and importance of the data.

[0011] Furthermore, the data classification process further includes the following steps: Determine the appropriate compression level based on the importance of the data and the need for storage space. For data that is of high importance and needs to save storage space, choose a high compression level. Otherwise, choose a low compression level. According to the requirements of the selected compression algorithm, corresponding compression parameters are set, wherein the compression parameters include block size and dictionary size; Create corresponding compression strategies based on data classification and compression parameters.

[0012] Furthermore, the data compression further comprises the following steps: Treat each data subset as an independent compression task and assign it to different threads for processing; Maintain synchronization and communication between threads in multi-threaded processing, and dynamically adjust the number of threads according to system resources and compression task volume; After the compression is completed, the compressed data is stored in a compressed data storage module, wherein the compressed data storage module is one of a disk and a database; When the compressed data is stored, a unique identifier is assigned to each compressed data subset.

[0013] Furthermore, the embedded mobile management platform specifically includes the following steps: Embed the data compression module into the mobile management platform through the API interface; A real-time data collection function is set up in the mobile management platform to collect data in the ship construction process in real time. After the collected ship construction data is processed by characteristic analysis, compression algorithm selection, adaptive data segmentation and data classification, it is automatically transmitted to the data compression module for compression processing; The compressed ship construction data is transmitted and stored via the mobile management platform, and the compressed ship construction data is encrypted during the transmission process; Staff can view and process compressed data at any time through the mobile management platform. When the data needs to be viewed, the platform will automatically decompress the compressed data and display it on the user interface.

[0014] Furthermore, the correlation between various links in the shipbuilding data is analyzed, including: A pair of link data with a correlation relationship is selected from the ship construction data as first link data and second link data; Extract the characteristic index of the first link data and calculate the first reliability coefficient of the characteristic index in the characteristic attribute set:

[0015] in, K 1 is the first reliability coefficient; b a is a characteristic indicator; F 1 For feature indicators in the feature attribute set A Membership function on ; U 1 is the domain of the characteristic index; n is the characteristic attribute set A The number of all indicators above; b x is the xth index in the feature attribute set A; T (b a ,b x ) is the matching degree between the feature index and the xth index in the feature attribute set A; According to the correlation between the first link data and the second link data, determining the matching index in the second link data that has a correlation with the characteristic index; Calculate the second reliability coefficient of the matching index on the relative feature attribute set:

[0016] in, K 2 is the second reliability coefficient; b c is the matching indicator; F 2 For matching indicators in the relative feature attribute set (R-A) Membership function on m; m is the relative feature attribute set (R-A) The number of all indicators above; b y is a relative feature attribute set (R-A) The yth index in ; T (b c ,b y ) is a set of matching indicators and relative feature attributes (R-A) The matching degree of the yth indicator in ; U 2 The domain where the matching indicator is located; R is the total set of characteristic attributes; According to the first reliability coefficient and the second reliability coefficient, the third reliability coefficient of the correlation between the first link data and the second link data is:

[0017] in, K 3 is the third reliability coefficient; The third reliability coefficient is compared with a preset threshold, and the association relationship whose third reliability coefficient is greater than or equal to the preset threshold is regarded as a valid relationship, and the association relationship whose third reliability coefficient is less than the preset threshold is eliminated.

[0018] Furthermore, the ship construction data is preprocessed, including: Extract strings from ship construction data and determine key strings; Establish a pattern matching automaton based on the pattern set and regular pattern string; Inputting the key string into the pattern matching automaton to perform an initial match with the regular pattern string, dividing the regular pattern string into multiple regular substrings based on a preset rule; and matching the multiple regular substrings with the key string respectively; Determine the special substring in the key string that does not match the regular substring, match the special substring with the abnormal string in the abnormal database, and determine the abnormal data in the ship construction data based on the matching information; Determining portions of the ship construction data that are associated with the abnormal data; Smoothing and data repairing are performed on some data based on the least squares method to determine the overall data; Determine mapping data corresponding to the abnormal data in the overall data according to the association relationship between the abnormal data and the partial data; Performing interpolation calculation on the mapped data and performing sequence identification to determine a first identification data sequence; Performing interpolation calculation on the abnormal data and performing sequential identification to determine a second identification data sequence; Determine at least three matching points between the first identification data sequence and the second identification data sequence, including a first matching point and a last matching point; A correspondence between the first identification data sequence and the second identification data sequence is established according to the first matching point and the last matching point, and the abnormal data is repaired according to the corresponding relationship to obtain repaired ship construction data.

[0019] Compared with the prior art, the present invention has the following beneficial effects: 1. The present invention analyzes the characteristics of shipbuilding data, selects a compression algorithm according to the data characteristics, creates an adaptive data segmentation strategy according to the data characteristics, and classifies and processes different types of data to further improve the compression efficiency. At the same time, an efficient data compression and decompression module is designed and embedded in the mobile management platform. The staff can view and process the compressed data at any time through the mobile management platform to ensure real-time compression and decompression of the data, thereby significantly reducing the storage and transmission costs without losing data integrity.

[0020] 2. The present invention can more accurately understand the nature and characteristics of the data by performing characteristic analysis on the ship construction data, which is helpful to select appropriate compression algorithms and strategies according to the characteristics of the data in the subsequent data processing process, thereby improving the efficiency and accuracy of data processing. The data is classified and processed according to the data type and compression requirements, and different compression strategies are generated, which can further ensure the security and reliability of the data. For data that needs to be frequently accessed and modified, a fast algorithm with a slightly lower compression ratio is used to ensure the real-time and availability of the data; for data that does not need to be frequently accessed but needs to be stored for a long time, an algorithm with a relatively high compression ratio but a slightly slower decompression speed is used to ensure the long-term storage and security of the data.

[0021] 3. The present invention performs block compression on the segmented subsets, which can effectively reduce the storage space occupied by the data. For large ship construction data files and data streams, this block compression method can further reduce the storage cost while maintaining the integrity and accessibility of the data. By dynamically adjusting the segmentation strategy of each subset of the data block, different processing can be performed according to the real-time and importance of the data. For real-time data that requires a quick response, a smaller segmentation unit and a lower compression ratio can be used to ensure the timeliness and accuracy of the data; for less important or non-real-time data, a larger segmentation unit and a higher compression ratio can be used to further save storage space.

[0022] 4. In the present invention, the staff can view and process the compressed data at any time through the mobile management platform without performing additional data decompression or conversion operations, which not only improves the user experience, but also reduces the difficulty and complexity of operation, making the management and use of data more convenient. The real-time collection, compression, transmission, storage and decompression of data through the mobile management platform can ensure the continuity and traceability of the data, so that the enterprise can better track the source and change process of the data, and provide strong support for subsequent data analysis and decision-making. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] Figure 1 It is a schematic diagram of the data compression process of the shipbuilding mobile management platform of the present invention. DETAILED DESCRIPTION

[0024] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0025] In order to solve the technical problem that the transmission of uncompressed shipbuilding data may be very time-consuming, especially when the data volume is huge, the limitation of network bandwidth may lead to low transmission efficiency, a large amount of uncompressed data will increase the burden of computer processing data, and traditional data processing methods may not fully utilize system resources, resulting in resource waste or low processing efficiency, please refer to Figure 1 , the present invention provides the following technical solutions: The data compression method based on the shipbuilding mobile management platform comprises the following steps: Characteristic analysis, obtaining ship construction data and performing data analysis on the ship construction data, preprocessing the ship construction data, and caching the ship construction data according to the data type after preprocessing; Compression algorithm selection, based on the analysis results of characteristic analysis, different compression algorithms are selected for different types of ship construction data; Adaptive data segmentation: segment the shipbuilding data into several subsets based on the data type and data volume. When processing text data, segmentation is based on paragraphs, and when processing image and video data, segmentation is based on key frames. Data classification processing: according to the data type and compression requirements obtained after feature analysis, compression algorithm selection and adaptive data segmentation, the data is classified into different categories, and different compression strategies are generated based on the data categories; Data compression: construct a data compression module and compress the data subsets in the shipbuilding data based on the compression strategy, and store the compressed data in the compressed data storage module. When performing data compression, multi-threaded processing is used to simultaneously process the compression and decompression tasks of multiple data subsets; Embedded in the mobile management platform, the data compression module is embedded in the ship construction mobile management platform. The ship construction data is collected, compressed, transmitted, stored and decompressed in real time through the mobile management platform. The staff can view and process the compressed data at any time through the mobile management platform.

[0026] Specifically, by analyzing the characteristics of ship construction data, selecting compression algorithms according to data characteristics, creating adaptive data segmentation strategies based on data characteristics, and classifying different types of data to further improve compression efficiency, an efficient data compression and decompression module is designed and embedded in the mobile management platform to ensure real-time compression and decompression of data, thereby significantly reducing storage and transmission costs without losing data integrity.

[0027] In the above embodiment, through characteristic analysis and compression algorithm selection, the most suitable compression algorithm can be selected for different types of shipbuilding data, thereby improving data processing efficiency. At the same time, the adaptive data segmentation technology can segment the data into several subsets according to the data type and data volume, further improving the flexibility and efficiency of data processing.

[0028] In the above embodiment, the data compression module is embedded in the ship construction mobile management platform, so that the staff can view and process the compressed data through the mobile management platform at any time, which greatly improves the practicality and convenience of the mobile management platform and makes the data management in the ship construction process more efficient and flexible.

[0029] Characteristic analysis, obtaining ship construction data and performing data analysis on the ship construction data, preprocessing the ship construction data, and caching the ship construction data according to the data type after preprocessing; Characteristic analysis also includes the following steps: Collect shipbuilding data from all aspects of shipbuilding, and distinguish the shipbuilding data into key data and non-key data, wherein the key data includes design drawings and process parameters, and the non-key data includes other data except design drawings and process parameters; Perform type analysis on the collected non-key data, and visualize the proportion, distribution and storage characteristics of the non-key data through data visualization tools based on the data type, wherein the data type includes text data, image data, video data and structured data, and the visualization is performed by drawing one or more of a bar chart, a pie chart and a graph, and at the same time, the text data is counted for word frequency and keywords are extracted through a text analysis tool; Analyze the correlation between various links in ship construction data, identify redundant information in ship construction data, and use data deduplication and compression algorithms to identify and remove redundant information; Based on the data type of the ship construction data, key data and non-key data are cached by type.

[0030] In the above embodiment, by performing characteristic analysis on the ship construction data, including distinguishing key data from non-key data, analyzing data types, data proportions, distribution, and storage characteristics, etc., the nature and characteristics of the data can be understood more accurately, which helps to select appropriate compression algorithms and strategies according to the characteristics of the data in the subsequent data processing process, thereby improving the efficiency and accuracy of data processing.

[0031] In the above embodiment, by visually displaying non-critical data and performing word frequency statistics and other analyses, the distribution of data can be more intuitively understood, and redundant information can be identified and removed, which can not only reduce storage space usage and storage costs, but also improve data transmission efficiency and reduce transmission time and network bandwidth consumption.

[0032] In the above embodiment, data is classified and processed according to data type and compression requirements, and different compression strategies are generated, which can further ensure the security and reliability of the data. For data that needs to be frequently accessed and modified, a fast algorithm with a slightly lower compression ratio is used to ensure the real-time and availability of the data; for data that does not need to be frequently accessed but needs to be stored for a long time, an algorithm with a higher compression ratio but a slightly slower decompression speed is used to ensure the long-term storage and security of the data.

[0033] Compression algorithm selection, based on the analysis results of characteristic analysis, different compression algorithms are selected for different types of ship construction data; Compression algorithm selection specifically includes the following steps: Select different compression algorithms based on the type of shipbuilding data; Wherein, a lossless compression algorithm is used for key data, and the lossless compression algorithm is one of the LZ77 compression algorithm and the Huffman coding compression algorithm; A lossy compression algorithm is used for non-critical data, and the lossy compression algorithm includes Huffman coding compression algorithm, H.264 compression algorithm and column storage. Among them, Huffman coding compression algorithm is selected when processing text data; H.264 compression algorithm is selected when processing image data and video data; column storage is selected when processing structured data.

[0034] In the above embodiments, for key data, such as design drawings and process parameters, a lossless compression algorithm such as the LZ77 compression algorithm or the Huffman coding compression algorithm is selected to ensure that the integrity of the data is not lost during the compression process, and to ensure that the key data still maintains its original precision and accuracy after compression, thereby providing reliable data support for subsequent ship construction work. For non-critical data, a lossy compression algorithm such as the Huffman coding compression algorithm, the H.264 compression algorithm, and column storage are used, which can reduce the storage space occupied by the data to a certain extent, and achieve compression by removing redundant information in the data or reducing the precision of the data, thereby reducing storage costs.

[0035] In the above embodiment, selecting a suitable compression algorithm for different types of ship construction data can make the data more efficient during the processing process. Through reasonable data compression, the burden of data transmission and processing can be reduced, thereby improving the response speed and performance of the mobile management platform, which is crucial for real-time monitoring and remote management of the ship construction process, and helps to improve management efficiency and decision-making level.

[0036] Adaptive data segmentation: segment the shipbuilding data into several subsets based on the data type and data volume. When processing text data, segmentation is based on paragraphs, and when processing image and video data, segmentation is based on key frames. Adaptive data segmentation specifically includes the following steps: Identifying paragraphs in text data, wherein the paragraphs are identified by identifying line breaks, indents, and paragraph marks in the text; Parse the content of each paragraph in the text data, wherein the content parsing is achieved through keyword extraction and topic identification; Based on the length of the paragraphs in the text data and the data volume of the entire text, a segmentation point is determined, and the text data in the ship construction data is segmented into a plurality of subsets based on the segmentation point; Decode and convert the video data into a format, extract key frames from the video, and perform content evaluation on the extracted key frames, wherein the content evaluation is achieved through image recognition and target detection; According to the content and data size of the key frames, the video data in the ship construction data are divided into several subsets, wherein each subset contains a series of key frames and some video frames between the key frames.

[0037] For large ship construction data files and data streams, after completing data segmentation, the subsets are compressed in blocks; For small ship construction data files and short data segments, overall compression is performed. At the same time, the segmentation strategy of each subset of ship construction data blocks is dynamically adjusted according to the real-time and importance of the data.

[0038] In the above embodiment, by identifying paragraphs in text data and key frames in video data, the scheme can adaptively divide the shipbuilding data into several subsets according to the data type and data volume, so that subsequent data processing and analysis can be performed on smaller and more concentrated data units, thereby improving processing efficiency. Block compression of the divided subsets can effectively reduce the storage space occupied by the data. For large shipbuilding data files and data streams, this block compression method can further reduce storage costs while maintaining data integrity and accessibility.

[0039] In the above embodiment, by dynamically adjusting the segmentation strategy of each subset of the data block, different processing can be performed according to the real-time and importance of the data. For real-time data that requires a quick response, a smaller segmentation unit and a lower compression ratio can be used to ensure the timeliness and accuracy of the data; for less important or non-real-time data, a larger segmentation unit and a higher compression ratio can be used to further save storage space.

[0040] Data classification processing: according to the data type and compression requirements obtained after feature analysis, compression algorithm selection and adaptive data segmentation, the data is classified into different categories, and different compression strategies are generated based on the data categories; The data classification process further includes the following steps: Determine the appropriate compression level based on the importance of the data and the need for storage space. For data that is of high importance and needs to save storage space, choose a high compression level. Otherwise, choose a low compression level. According to the requirements of the selected compression algorithm, corresponding compression parameters are set, wherein the compression parameters include block size and dictionary size; Create corresponding compression strategies based on data classification and compression parameters.

[0041] In the above embodiment, by classifying and processing the data, different compression strategies can be formulated according to the characteristics and compression requirements of different types of data, ensuring that important data can be compressed more efficiently while avoiding unnecessary compression losses, thereby meeting the diverse data processing needs in the shipbuilding process. According to the importance of the data and the demand for storage space, the appropriate compression level is determined to minimize the storage space occupancy. For data that is highly important and needs to save storage space, using a high compression level can significantly reduce the data volume and improve storage and transmission efficiency; for data that is relatively less important or does not require excessive compression, selecting a low compression level can ensure the integrity and readability of the data.

[0042] In the above embodiment, according to the requirements of the selected compression algorithm, the corresponding compression parameters, such as block size and dictionary size, are set to further optimize the compression quality and efficiency. By adjusting these parameters, the characteristics of different types of data can be adapted to improve the adaptability and flexibility of the compression algorithm. By creating corresponding compression strategies based on data classification and compression parameters, data management can be made more targeted. Different compression strategies are used for different data categories, which helps to better manage and utilize data resources in the shipbuilding process and improve management efficiency and decision-making level.

[0043] Data compression: construct a data compression module and compress the data subsets in the shipbuilding data based on the compression strategy, and store the compressed data in the compressed data storage module. When performing data compression, multi-threaded processing is used to simultaneously process the compression and decompression tasks of multiple data subsets; Data compression specifically includes the following steps: Treat each data subset as an independent compression task and assign it to different threads for processing; Maintain synchronization and communication between threads in multi-threaded processing, and dynamically adjust the number of threads according to system resources and compression task volume; After the compression is completed, the compressed data is stored in a compressed data storage module, wherein the compressed data storage module is one of a disk and a database; When the compressed data is stored, a unique identifier is assigned to each compressed data subset.

[0044] In the above embodiment, by treating each data subset as an independent compression task and assigning it to different threads for processing, parallel compression can be achieved, computing resources are fully utilized, the speed of compression processing is greatly accelerated, and the overall compression efficiency is improved, making the compression of large-scale shipbuilding data more efficient. In multi-threaded processing, synchronization and communication between threads are maintained, and the number of threads is dynamically adjusted according to system resources and compression task volume, which can ensure the reasonable allocation and efficient use of system resources. This avoids waste of resources and also ensures the smooth progress of compression tasks.

[0045] In the above embodiment, after compression is completed, the compressed data is stored in a compressed data storage module, which can be a disk or a database, providing flexibility for data storage and subsequent use. At the same time, a unique identifier is assigned to each compressed data subset, which facilitates data identification, retrieval and management, and improves the convenience of data management.

[0046] In the above embodiment, the stability and reliability of the data compression process can be ensured through multi-threaded processing and dynamic resource adjustment. Even in the face of a large number of data compression tasks, the system can maintain efficient and stable operation, reducing the risk of system crash or performance degradation caused by compression tasks. Due to the improvement of compression efficiency and the optimal use of resources, the mobile management platform can respond more quickly when processing user requests, reducing the user's waiting time, helping to improve user experience, and enhancing user satisfaction and trust in the platform.

[0047] Embedded in the mobile management platform: The data compression module is embedded in the mobile management platform for shipbuilding. The mobile management platform can collect, compress, transmit, store and decompress the shipbuilding data in real time. The staff can view and process the compressed data at any time through the mobile management platform. Embedding the mobile management platform also includes the following steps: Embed the data compression module into the mobile management platform through the API interface; A real-time data collection function is set up in the mobile management platform to collect data in the ship construction process in real time. After the collected ship construction data is processed by characteristic analysis, compression algorithm selection, adaptive data segmentation and data classification, it is automatically transmitted to the data compression module for compression processing; The compressed ship construction data is transmitted and stored via the mobile management platform, and the compressed ship construction data is encrypted during the transmission process; Staff can view and process compressed data at any time through the mobile management platform. When the data needs to be viewed, the platform will automatically decompress the compressed data and display it on the user interface.

[0048] Furthermore, the correlation between various links in the shipbuilding data is analyzed, including: A pair of link data with a correlation relationship is selected from the ship construction data as first link data and second link data; Extract the characteristic index of the first link data and calculate the first reliability coefficient of the characteristic index in the characteristic attribute set:

[0049] in, K 1 is the first reliability coefficient; b a is a characteristic indicator; F 1 For feature indicators in the feature attribute set A Membership function on ; U 1 is the domain of the characteristic index; n is the characteristic attribute set A The number of all indicators above; b x is the xth index in the feature attribute set A; T (b a ,b x ) is the matching degree between the feature index and the xth index in the feature attribute set A; According to the correlation between the first link data and the second link data, determining the matching index in the second link data that has a correlation with the characteristic index; Calculate the second reliability coefficient of the matching index on the relative feature attribute set:

[0050] in, K 2 is the second reliability coefficient; b c is the matching indicator; F 2 For matching indicators in the relative feature attribute set (R-A) Membership function on m; m is the relative feature attribute set (R-A) The number of all indicators above; b y is a relative feature attribute set (R-A) The yth index in ; T (b c ,b y ) is a set of matching indicators and relative feature attributes (R-A) The matching degree of the yth indicator in ; U 2 The domain where the matching indicator is located; R is the total set of characteristic attributes; According to the first reliability coefficient and the second reliability coefficient, the third reliability coefficient of the correlation between the first link data and the second link data is:

[0051] in, K 3 is the third reliability coefficient; The third reliability coefficient is compared with a preset threshold, and the association relationship whose third reliability coefficient is greater than or equal to the preset threshold is regarded as a valid relationship, and the association relationship whose third reliability coefficient is less than the preset threshold is eliminated.

[0052] Working principle of the above technical solution: In this embodiment, the shipbuilding data includes data of various links. A pair of link data with associated relationship is selected from the shipbuilding data as the first link data and the second link data.

[0053] In this embodiment, statistical methods (such as descriptive statistics, data visualization, etc.) are used to explore the distribution, trends and relationships of the first link data. Through EDA, it is possible to preliminarily determine which features may be important and the relationship between them. Feature selection is performed based on domain knowledge, statistical tests (such as chi-square test, t-test, ANOVA, etc.) or machine learning algorithms (such as random forest, gradient boosting machine, etc.) to determine the most important feature indicators. The feature attribute set describes the information contained in each record in the table, the data type and constraints of this information, and also describes the characteristics or status of each category.

[0054] In this embodiment, the membership function on the feature attribute set refers to a function used to describe the degree of membership of the feature index on the feature attribute set. In fuzzy logic, the membership function is usually used to characterize the degree or probability that an element belongs to different fuzzy sets. In feature engineering, the membership function can be used to evaluate and describe the relationship and attributes between features. The membership function can be a Gaussian membership function:

[0055] in, x represents the eigenvalue, μ represents the mean, σ represents the standard deviation; e is a natural constant. By adjusting the values ​​of the mean and standard deviation, the membership of the feature on the feature attribute set can be described. In addition to the Gaussian membership function, there are other different forms of membership functions, such as triangular membership function, trapezoidal membership function, etc.

[0056] Domain is an important concept in mathematical logic. It refers to the set of all individuals or objects involved in a reasoning or discussion. It usually consists of three parts: a non-empty set of elements M', including the basic elements of the domain. A non-empty set of functions on M', each of which has an M' or Cartesian product of multiple M's as its domain and M' as its range. A non-empty set of propositions about M', each of which expresses the logical relationship between elements of M', between functions, and between elements and functions.

[0057] In this embodiment, the second reliability coefficient of the matching index in the relative feature attribute set is calculated, and the relative feature attribute set is a relative set of the total feature attribute set excluding the feature attribute set. The domain where the matching index is located is consistent with the domain where the feature index is located.

[0058] The beneficial effects of the above technical solution are: accurately calculating the first reliability coefficient of the characteristic index in the characteristic attribute set and the second reliability coefficient of the matching index in the relative characteristic attribute set, and then accurately calculating the third reliability coefficient of the correlation relationship between the first link data and the second link data, thereby improving the accuracy of judging the third reliability coefficient and the preset threshold value, judging whether the correlation relationship between the data of each link is a valid relationship or an invalid relationship, realizing the analysis of the correlation between the various links in the ship construction data, and realizing the sorting of the correlation relationship.

[0059] Furthermore, the ship construction data is preprocessed, including: Extract strings from ship construction data and determine key strings; Establish a pattern matching automaton based on the pattern set and regular pattern string; Inputting the key string into the pattern matching automaton to perform an initial match with the regular pattern string, dividing the regular pattern string into multiple regular substrings based on a preset rule; and matching the multiple regular substrings with the key string respectively; Determine the special substring in the key string that does not match the regular substring, match the special substring with the abnormal string in the abnormal database, and determine the abnormal data in the ship construction data based on the matching information; Determining portions of the ship construction data that are associated with the abnormal data; Smoothing and data repairing are performed on some data based on the least squares method to determine the overall data; Determine mapping data corresponding to the abnormal data in the overall data according to the association relationship between the abnormal data and the partial data; Performing interpolation calculation on the mapped data and performing sequence identification to determine a first identification data sequence; Performing interpolation calculation on the abnormal data and performing sequential identification to determine a second identification data sequence; Determine at least three matching points between the first identification data sequence and the second identification data sequence, including a first matching point and a last matching point; A correspondence between the first identification data sequence and the second identification data sequence is established according to the first matching point and the last matching point, and the abnormal data is repaired according to the corresponding relationship to obtain repaired ship construction data.

[0060] The working principle of the above technical solution: In this embodiment, key information is extracted from ship construction data based on string extraction technology.

[0061] In this embodiment, pattern matching is implemented using a deterministic finite automaton (DFA) or a non-deterministic finite automaton (NFA). Determine the type of automaton: First, it is necessary to determine whether to use DFA or NFA to construct the pattern matching automaton. DFA usually has a higher space overhead, but a faster matching speed; while NFA has a lower space overhead, but the matching speed may be slightly slower. Construct a state transition diagram: Construct a state transition diagram of the automaton based on a given regular pattern string. Each state represents a match to a certain prefix pattern string, and the state transition is performed according to the different input characters. Determine the accepting state: Mark which states in the automaton are accepting states, that is, the complete pattern string is matched. State transition function: Define the transition function between states, including the rule for transferring to the next state based on the input characters. Matching algorithm: Implement the matching algorithm, that is, perform state transitions character by character according to the input text, and finally determine whether the pattern string is matched. The preset rule is to divide the minimum substring of each string.

[0062] In this embodiment, the data matching the normal substring is verified without any other processing. However, the special substring in the key string that does not match the regular substring is verified to be abnormal data. The special substring is matched with the abnormal string in the abnormal database, and the abnormal data in the ship construction data is determined based on the matching information.

[0063] In this embodiment, the partial data is smoothed and repaired based on the least squares method to determine the overall data, including: selecting a suitable least squares model: in the data smoothing and repairing process, the least squares method is used to fit the data and find the best fitting curve. Fitting data: using the least squares method to fit the partial data to obtain a fitting curve or function. The core idea of ​​the least squares method is to adjust the model parameters by minimizing the residual sum of squares so that the fitting curve optimally conforms to the data distribution. Data repair processing: data repair processing is performed on missing values ​​or outliers based on the fitting curve or function. Missing data is filled or abnormal data is repaired according to the predicted value of the fitting curve, so that the overall data is more complete and accurate. Determination of overall data: based on the partial data and the fitting curve after the repair processing, the overall data can be determined. According to the actual situation, the overall data can be inferred and filled by the fitting curve to obtain a complete data set.

[0064] In this embodiment, the mapping data is the correct data portion corresponding to the abnormal data, and the data volume of the mapping data is generally greater than the data volume of the abnormal data.

[0065] In this embodiment, the mapping data is interpolated and sequentially marked to determine the first marked data sequence, and the mapping data is interpolated by an interpolation algorithm to fill in missing data or perform data smoothing. Common interpolation algorithms include linear interpolation, polynomial interpolation, spline interpolation, etc. Sequential marking: Sequential marking of the interpolated data can be performed in chronological order or other specified order. Ensure that each data point in the data is correctly marked and arranged in a certain order.

[0066] In this embodiment, the method of performing interpolation calculation on abnormal data is consistent with the method of performing interpolation calculation on mapping data, and will not be described in detail.

[0067] In this embodiment, the correspondence between the first identification data sequence and the second identification data sequence is established according to the first matching point and the last matching point, that is, the correspondence between the two is determined, and the abnormal data is repaired according to the correspondence to obtain the repaired ship construction data.

[0068] The beneficial effects of the above technical solution are as follows: based on the pattern matching automaton, a special substring in the key string that does not match the regular substring is determined, the special substring is matched with the abnormal string in the abnormal database, the abnormal data in the ship construction data is determined, and the abnormal data is supplemented through partial data associated with the abnormal data to obtain the overall data, and the mapping data corresponding to the abnormal data is determined in the overall data based on the association between the abnormal data and the partial data; by establishing a corresponding relationship between the first identification data sequence and the second identification data sequence, the abnormal data is repaired according to the corresponding relationship to obtain the repaired ship construction data, thereby improving the accuracy of the ship construction data.

[0069] In the above embodiment, the staff can view and process the compressed data at any time through the mobile management platform without performing additional data decompression or conversion operations, which not only improves the user experience, but also reduces the difficulty and complexity of operation, making the management and use of data more convenient. Real-time data collection, compression, transmission, storage and decompression through the mobile management platform can ensure the continuity and traceability of data, allowing enterprises to better track the source and change process of data, and provide strong support for subsequent data analysis and decision-making.

[0070] The above description is only a preferred specific implementation manner of the present invention, but the protection scope of the present invention is not limited thereto. Any technician familiar with the technical field can make equivalent replacements or changes according to the technical scheme and inventive concept of the present invention within the technical scope disclosed by the present invention, which should be covered by the protection scope of the present invention.

Claims

1. A data compression method based on a shipbuilding mobile management platform, characterized in that: The following steps are involved: Characteristic analysis, obtaining ship construction data and performing data analysis on the ship construction data, preprocessing the ship construction data, and caching the ship construction data according to the data type after preprocessing; Compression algorithm selection, based on the analysis results of characteristic analysis, different compression algorithms are selected for different types of ship construction data; Adaptive data segmentation: segment the shipbuilding data into several subsets based on the data type and data volume. When processing text data, segmentation is based on paragraphs, and when processing image and video data, segmentation is based on key frames. Data classification processing: data is classified into different categories according to the data type and compression requirements obtained after feature analysis, compression algorithm selection and adaptive data segmentation, and different compression strategies are generated based on the data categories; Data compression: construct a data compression module and compress the data subsets in the shipbuilding data based on the compression strategy, and store the compressed data in the compressed data storage module. When performing data compression, multi-threaded processing is used to simultaneously process the compression and decompression tasks of multiple data subsets; Embedded in the mobile management platform, the data compression module is embedded in the ship construction mobile management platform. The ship construction data is collected, compressed, transmitted, stored and decompressed in real time through the mobile management platform. The staff can view and process the compressed data at any time through the mobile management platform.

2. The data compression method based on the shipbuilding mobile management platform according to claim 1, characterized in that: The characteristic analysis specifically further includes the following steps: Collect shipbuilding data from all aspects of shipbuilding, and distinguish the shipbuilding data into key data and non-key data, wherein the key data includes design drawings and process parameters, and the non-key data includes other data except design drawings and process parameters; Perform type analysis on the collected non-key data, and visualize the proportion, distribution and storage characteristics of the non-key data through data visualization tools based on the data type, wherein the data type includes text data, image data, video data and structured data, and the visualization is performed by drawing one or more of a bar chart, a pie chart and a graph, and at the same time, the text data is counted for word frequency and keywords are extracted through a text analysis tool; Analyze the correlation between various links in ship construction data, identify redundant information in ship construction data, and use data deduplication and compression algorithms to identify and remove redundant information; Based on the data type of the ship construction data, key data and non-key data are cached by type.

3. The data compression method based on the shipbuilding mobile management platform according to claim 1, characterized in that: The compression algorithm selection specifically further includes the following steps: Select different compression algorithms based on the type of shipbuilding data; Wherein, a lossless compression algorithm is used for key data, and the lossless compression algorithm is one of the LZ77 compression algorithm and the Huffman coding compression algorithm; A lossy compression algorithm is used for non-critical data, and the lossy compression algorithm includes Huffman coding compression algorithm, H.264 compression algorithm and column storage. Among them, Huffman coding compression algorithm is selected when processing text data; H.264 compression algorithm is selected when processing image data and video data; column storage is selected when processing structured data.

4. The data compression method based on the shipbuilding mobile management platform according to claim 1, characterized in that: The adaptive data segmentation further comprises the following steps: Identifying paragraphs in text data, wherein the paragraphs are identified by identifying line breaks, indents, and paragraph marks in the text; Parse the content of each paragraph in the text data, wherein the content parsing is achieved through keyword extraction and topic identification; Based on the length of the paragraphs in the text data and the data volume of the entire text, a segmentation point is determined, and the text data in the ship construction data is segmented into a plurality of subsets based on the segmentation point; Decode and convert the video data into a format, extract key frames from the video, and perform content evaluation on the extracted key frames, wherein the content evaluation is achieved through image recognition and target detection; According to the content and data size of the key frames, the video data in the ship construction data are divided into several subsets, wherein each subset contains a series of key frames and some video frames between the key frames.

5. The data compression method based on the shipbuilding mobile management platform according to claim 4, characterized in that: The adaptive data segmentation further comprises the following steps: For large ship construction data files and data streams, after completing data segmentation, the subsets are compressed in blocks; For small ship construction data files and short data segments, overall compression is performed. At the same time, the segmentation strategy of each subset of ship construction data blocks is dynamically adjusted according to the real-time and importance of the data.

6. The data compression method based on the shipbuilding mobile management platform according to claim 1, characterized in that: The data classification process specifically further includes the following steps: Determine the appropriate compression level based on the importance of the data and the need for storage space. For data that is highly important and needs to save storage space, select a high compression level, otherwise select a low compression level. According to the requirements of the selected compression algorithm, corresponding compression parameters are set, wherein the compression parameters include block size and dictionary size; Create corresponding compression strategies based on data classification and compression parameters.

7. The data compression method based on the shipbuilding mobile management platform according to claim 1, characterized in that: The data compression specifically further comprises the following steps: Treat each data subset as an independent compression task and assign it to different threads for processing; Maintain synchronization and communication between threads in multi-threaded processing, and dynamically adjust the number of threads according to system resources and compression task volume; After the compression is completed, the compressed data is stored in a compressed data storage module, wherein the compressed data storage module is one of a disk and a database; When the compressed data is stored, a unique identifier is assigned to each compressed data subset.

8. The data compression method based on the shipbuilding mobile management platform according to claim 1, characterized in that: The embedded mobile management platform specifically further includes the following steps: Embed the data compression module into the mobile management platform through the API interface; A real-time data collection function is set up in the mobile management platform to collect data in the ship construction process in real time. After the collected ship construction data is processed by characteristic analysis, compression algorithm selection, adaptive data segmentation and data classification, it is automatically transmitted to the data compression module for compression processing; The compressed ship construction data is transmitted and stored via the mobile management platform, and the compressed ship construction data is encrypted during the transmission process; Staff can view and process compressed data at any time through the mobile management platform. When the data needs to be viewed, the platform will automatically decompress the compressed data and display it on the user interface.

9. The data compression method based on the shipbuilding mobile management platform according to claim 2, characterized in that: Analyze the correlation between various links in ship construction data, including: A pair of link data with a correlation relationship is selected from the ship construction data as first link data and second link data; Extract the characteristic index of the first link data and calculate the first reliability coefficient of the characteristic index in the characteristic attribute set: ； in, K 1 is the first reliability coefficient; b a is a characteristic indicator; F 1 For feature indicators in the feature attribute set A Membership function on ; U 1 is the domain of the characteristic index; n is the characteristic attribute set A The number of all indicators above; b x is the xth index in the feature attribute set A; T (b a ,b x ) is the matching degree between the feature index and the xth index in the feature attribute set A; According to the correlation between the first link data and the second link data, determining the matching index in the second link data that has a correlation with the characteristic index; Calculate the second reliability coefficient of the matching index on the relative feature attribute set: ； in, K 2 is the second reliability coefficient; b c is the matching indicator; F 2 For matching indicators in the relative feature attribute set (R- A) Membership function on m; m is the relative feature attribute set (RA) The number of all indicators above; b y is a relative feature attribute set (RA) The yth index in ; T (b c ,b y ) is a set of matching indicators and relative feature attributes (RA) The matching degree of the yth indicator in ; U 2 The domain where the matching indicator is located; R is the total set of characteristic attributes; According to the first reliability coefficient and the second reliability coefficient, the third reliability coefficient of the correlation between the first link data and the second link data is: ； in, K 3 is the third reliability coefficient; The third reliability coefficient is compared with a preset threshold, and the association relationship whose third reliability coefficient is greater than or equal to the preset threshold is regarded as a valid relationship, and the association relationship whose third reliability coefficient is less than the preset threshold is eliminated.

10. The data compression method based on the shipbuilding mobile management platform according to claim 1, characterized in that: Preprocessing of ship construction data, including: Extract strings from ship construction data and determine key strings; Establish a pattern matching automaton based on the pattern set and regular pattern string; Inputting the key string into the pattern matching automaton to perform an initial match with the regular pattern string, dividing the regular pattern string into multiple regular substrings based on a preset rule; and matching the multiple regular substrings with the key string respectively; Determine the special substring in the key string that does not match the regular substring, match the special substring with the abnormal string in the abnormal database, and determine the abnormal data in the ship construction data based on the matching information; Determining portions of the ship construction data that are associated with the abnormal data; Smoothing and data repairing are performed on some data based on the least squares method to determine the overall data; Determine mapping data corresponding to the abnormal data in the overall data according to the association relationship between the abnormal data and the partial data; Performing interpolation calculation on the mapped data and performing sequence identification to determine a first identification data sequence; Performing interpolation calculation on the abnormal data and performing sequential identification to determine a second identification data sequence; Determine at least three matching points between the first identification data sequence and the second identification data sequence, including a first matching point and a last matching point; A correspondence between the first identification data sequence and the second identification data sequence is established according to the first matching point and the last matching point, and the abnormal data is repaired according to the corresponding relationship to obtain repaired ship construction data.

Images