Data processing method for large-scale project management

Abstract

The invention relates to the technical field of data compression, in particular to a data processing method for large-scale project management, which comprises the following steps: acquiring all characters in water conservancy coded data, de-duplicating all character strings to obtain a character category set, setting character extension scales, and acquiring character connection digital quantity under each character extension scale, arranging the character connection digital quantity to obtain a character connection digital sequence of each character, equally dividing the character connection digital sequence to obtain a deviation value distance set of each character, and obtaining the optimal connection length of each character according to the distribution characteristics of elements in the deviation value distance set; and obtaining the optimal sliding window length of the water conservancy coded data according to the optimal serial length, and completing water conservancy project data compression in combination with an LZ77 algorithm. The invention aims to improve the compression efficiency and compression quality of hydraulic engineering data and realize large-scale project management.

Description

Technical Field

[0001] The present invention relates to the technical field of data compression, and in particular to a data processing method for large-scale engineering management. Background Art

[0002] Water conservancy projects are the core means for human society to comprehensively develop, effectively utilize and strictly protect natural water resources. They cover a series of key areas from flood control and drought relief to irrigation and water supply, hydropower generation, channel dredging and ecological and environmental protection. Scientific water conservancy management can finely dispatch water resources, adjust water source allocation according to seasonal changes, climate change and social and economic needs, maximize the efficiency and economic benefits of water resources, and avoid environmental pressure and social costs caused by resource waste. Water conservancy project management is crucial to preventing natural disasters, especially floods. Good water conservancy project management also pays attention to ecological protection and restoration, and pays attention to the balance between water resource development and utilization and ecological and environmental protection. Rigorous and meticulous daily management and technical maintenance help to extend the service life of water conservancy facilities and reduce economic losses caused by equipment aging or failure.

[0003] BIM, or building information model. It is a method of coordinating the process of building design, construction and operation through digital technology, integrated software and collaborative platform to achieve building information sharing and visualization. However, due to the large scale of the project, the large amount of various drawings, models and data, the required data storage space is large, so data compression processing is required. However, since all information is relatively important, when compression is adopted, a lossless compression method needs to be used. LZ77 is a commonly used lossless compression algorithm. Since the compression rate of the LZ77 compression algorithm depends on the length of the algorithm sliding window, for the sliding window of the original LZ77 algorithm, a larger window can improve the compression efficiency, but reduce the compression time. A smaller compression window improves the compression time, but reduces the compression efficiency. Since the data volume of BIM water conservancy project data is large and highly complex, the compression processing of BIM water conservancy project data is not applicable. Summary of the invention

[0004] In order to solve the above technical problems, the present invention provides a data processing method for large-scale project management to solve the existing problems.

[0005] A data processing method for large-scale project management of the present invention adopts the following technical solution: An embodiment of the present invention provides a data processing method for large-scale project management, the method comprising the following steps: Get all characters in the water conservancy coding data; All characters are deduplicated to obtain a character category set; multiple character extension scales are set; semantic character string sequences of characters are obtained according to the character extension scales; string frequency sets of each character under each character extension scale are obtained according to the frequencies of occurrence of each semantic character string sequence; character connection digital quantities under each character extension scale are obtained according to the semantic character string sequence and the string frequency set; character connection digital quantities under all character extension scales are arranged to obtain a character connection digital sequence of each character; and each sub-character connection digital sequence is obtained according to the maximum value distribution of elements in the character connection digital sequence; All sub-character connection digital sequences are summarized to obtain the deviation distance set of each character; the optimal series length of each character is obtained according to the distribution of elements in the deviation distance set; the optimal sliding window length of water conservancy coding data is obtained according to the optimal series length, and the water conservancy project management is completed by combining the LZ77 compression algorithm and the optimal sliding window length.

[0006] Preferably, obtaining the semantic character string sequences of the characters according to the extension scales of the characters includes: For each character in the character category set, each character extension scale is extended backward in the water conservancy coding data, and the character sequence obtained after the extension is used as each semantic character string sequence.

[0007] Preferably, the step of obtaining a set of string frequencies of each character at each character extension scale according to the frequencies of occurrence of each semantic string sequence includes: For each character in the character category set, the frequency of occurrence of each semantic string sequence in the water conservancy coding data at each character extension scale is counted, and the frequency of occurrence of all semantic string sequences at the same character extension scale is taken as the string frequency set of each character at each character extension scale.

[0008] Preferably, the character connection digital quantity under the extension scale of each character is obtained according to the semantic character string sequence and the character string frequency set, and the expression is: In the formula, Represents a character in a character class set In character extension scale The following characters are connected to the numerical quantity, Representing characters In character extension scale The number of different semantic string sequences is obtained by Representing characters In character extension scale The first string frequency in the following set elements, Indicates the character extension scale, Representing characters In character extension scale The total number of semantic string sequences is obtained as follows: Indicates the floor symbol, is the adjustment factor.

[0009] Preferably, the step of arranging the character connection digital quantities of all characters under the extended scale to obtain the character connection digital sequence of each character includes: For each character in the character category set, the character connection digital quantities under all character extension scales are arranged in ascending order of the character extension scales as the character connection digital sequence of each character.

[0010] Preferably, the step of obtaining each sub-character connection digital sequence according to the maximum value distribution of elements in the character connection digital sequence includes: For each character connection digital sequence, the maximum value, the second maximum value and the third maximum value in the character connection digital sequence are obtained, and all the elements remaining in the character connection digital sequence after eliminating the maximum value, the second maximum value and the third maximum value are equally divided into a first part, a second part and a third part according to their positions in the character connection digital sequence, and the sequences consisting of the maximum value and the first part, the second maximum value and the second part, and the third maximum value and the third part are respectively used as the first sub-character connection digital sequence, the second sub-character connection digital sequence, and the third sub-character connection digital sequence.

[0011] Preferably, the step of aggregating all sub-character connection digital sequences to obtain a set of offset distances of each character includes: Calculate the sum of the first element in the first sub-character connected digital sequence, twice the first element in the second sub-character connected digital sequence, and three times the first element in the third sub-character connected digital sequence, calculate the ratio of the sum to 6, and use the ratio as the first element in the deviation distance set. Use the same acquisition method as the first element in the deviation distance set to calculate other elements in the deviation distance set.

[0012] Preferably, the optimal series length of each character is obtained according to the distribution of elements in the bias distance set, and the expression is: In the formula, Represents a character in a character class set The optimal series length is Represents the median value of all elements in the bias distance set, Represents the first elements, Represents the first The index value of the element, represents the mean of all elements in the deviation distance set, Represents the maximum value of all elements in the bias distance set, Represents the minimum value of all elements in the bias distance set, Represents the number of all elements in the deviation distance set.

[0013] Preferably, the optimal sliding window length of the water conservancy coding data is obtained according to the optimal series length, and the expression is: In the formula, represents the optimal sliding window length for water conservancy coded data, represents the rounding function, Indicates the first character in the character class set The optimal run length of characters, Indicates the number of characters contained in the character category set. Indicates the first character in the character class set The frequency of occurrence of characters in water conservancy coding data, Represents the mean frequency of all characters in the character category set.

[0014] Preferably, the water conservancy project management is completed by combining the LZ77 compression algorithm and the optimal sliding window length, including: The optimal sliding window length of water conservancy coding data is used as the sliding window length of the improved LZ77 compression algorithm, and the water conservancy coding data is compressed using the improved LZ77 compression algorithm.

[0015] The present invention has at least the following beneficial effects: The present invention mainly analyzes the character string behind each character in the water conservancy coding data, calculates the character connection digital quantity of each character, and reflects the connection relationship between the character and the surrounding characters in the water conservancy engineering data. The sub-character connection digital sequence is constructed through the largest, second largest, and third largest character connection digital quantities, and the optimal series length of each character is calculated to reduce the influence of the local optimal solution, reflecting the effective connection mode of the characters in the water conservancy engineering data. The optimal sliding window length of the water conservancy coding data is calculated through the optimal series length of each character in the water conservancy engineering data, which can reduce unnecessary calculation burden while ensuring effective information acquisition, and help to compress the water conservancy engineering data more efficiently and quickly. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] In order to more clearly illustrate the technical solutions and advantages in the embodiments of the present invention or the prior art, the drawings required for use in the embodiments or the prior art descriptions are briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0017] Figure 1 A flowchart of a data processing method for large-scale project management provided by one embodiment of the present invention; Figure 2 Construct a flow chart for the optimal sliding window length. DETAILED DESCRIPTION

[0018] In order to further explain the technical means and effects adopted by the present invention to achieve the predetermined invention purpose, the data processing method for large-scale project management proposed by the present invention, its specific implementation method, structure, features and effects are described in detail as follows in conjunction with the accompanying drawings and preferred embodiments. In the following description, different "one embodiment" or "another embodiment" does not necessarily refer to the same embodiment. In addition, specific features, structures or characteristics in one or more embodiments may be combined in any suitable form.

[0019] Unless defined otherwise, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention belongs.

[0020] The following is a detailed description of a data processing method for large-scale project management provided by the present invention in conjunction with the accompanying drawings.

[0021] See also Figure 1 , which shows a flow chart of a data processing method for large-scale project management provided by an embodiment of the present invention, the method comprising the following steps: Step S001, collect water conservancy coding data and perform preprocessing.

[0022] All continuous binary water conservancy data in the BIM water conservancy project database are obtained, divided into 16 bits, and the 16-bit binary water conservancy data are mapped into Chinese characters through Unicode encoding technology. As water conservancy encoding data, Unicode encoding is a well-known technology, and the specific conversion project will not be repeated here.

[0023] Step S002, by analyzing the distribution characteristics of each character in the water conservancy coding data and the frequency of occurrence in the water conservancy coding data, the optimal series length of each character is obtained, and the optimal sliding window length of the water conservancy coding data is obtained according to the optimal series length, and the water conservancy project data is compressed using the LZ77 compression algorithm.

[0024] Specifically, this embodiment will obtain all characters in the water conservancy coding data, remove all duplicates from all character strings to obtain a character category set, set a character extension scale, obtain the character connection digital quantity under each character extension scale, arrange the character connection digital quantity to obtain a character connection digital sequence of each character, divide the character connection data sequence into equal parts to obtain a deviation distance set of each character, obtain the optimal series length of each character according to the distribution characteristics of the elements in the deviation distance set, obtain the optimal sliding window length of the water conservancy coding data according to the optimal series length, and complete the water conservancy project data compression in combination with the LZ77 algorithm. The optimal sliding window length construction flow chart is shown in the following figure. Figure 2 The construction process of the optimal sliding window length for water conservancy coding data is as follows: In the water conservancy coding data, firstly, all characters are deduplicated, and all different characters are counted, and all different characters are taken as the character category set CH. Then, for each character in the character category set, it is put into the original water conservancy coding data for analysis, and the character extension scale i is set. In this embodiment, the value range of i is [2,50]. The implementer can set the value range according to the actual situation, and this embodiment does not limit this. Each character in the character category set is taken as a character For example, all characters in the original water conservancy coding data are Extend backward i characters, i is an integer, starting from 2 and ending at 50 or the end of the water conservancy code data. Each value of i gets a character The semantic string sequence at the beginning contains multiple semantic strings. Statistics about characters The frequency of all semantic strings in the water conservancy coding data is taken as the frequency set of the strings of each character under each character extension scale, that is, the frequency of all semantic strings under the same character extension scale, i.e., the frequency set of the strings under each character extension scale, denoted as .

[0025] According to the semantic string sequence and string frequency set, the character connection digital quantity under each character extension scale is constructed, and the expression is: In the formula, Represents a character in a character class set In character extension scale The following characters are connected to the numerical quantity, Representing characters In character extension scale The number of different semantic strings is obtained by Representing characters In character extension scale The first string frequency in the following set element indicating the character extension scale indicating the character under the character extension scale the total number of semantic strings obtained indicating the floor symbol is an adjustment factor. In this embodiment , the implementer can set it according to the actual situation, and this embodiment does not limit it

[0026] When the number of occurrences of the semantic string is more, that is the larger the value of [[[ID=]], the more it indicates that the character has a higher degree of association with the semantic string, and it indicates that the semantic string of the character is more likely to be a fixed combination in the water conservancy data. When the number of different semantic strings the larger the value of [[[ID=]], the more it indicates that the character has a low degree of association with other characters and is more likely to be a particle in the Chinese context, such as characters like "de", "he", etc. When is smaller, making the values of each element in [[[ID=]] larger, and the character connection digital quantity of the character is also larger. When the character connection digital quantity of the character is larger, it indicates a higher degree of association between characters. When selecting the string length during data compression, the compression speed can be improved

[0027] Arrange the character connection digital quantities of all character extension scales of the character in ascending order of the character extension scale as the character connection digital sequence of the character denoted as CHQ. Since a single character connection digital quantity cannot reflect the best connection length of the character, the best character connection length should be calculated within a certain local area to make the calculation result more stable and reduce errors

[0028] For the character A character connection digital sequence is obtained by arranging all elements in the character connection digital sequence from large to small, taking the first element after the arrangement as the maximum value, recorded as B, the second element after the arrangement as the second maximum value, recorded as C, and the third element after the arrangement as the third maximum value, recorded as D. The maximum value, the second maximum value and the third maximum value are removed from the character connection digital sequence, and all the remaining elements after removing the maximum value, the second maximum value and the third maximum value from the character connection digital sequence are divided into three parts, which are recorded as the first part, the second part and the third part respectively according to their positions in the character connection digital sequence, and B is placed before the first part, C is placed before the second part, and D is placed before the third part as the first sub-character connection digital sequence, the second sub-character connection digital sequence, and the third sub-character connection digital sequence, and the first sub-character connection digital sequence, the second sub-character connection digital sequence, and the third sub-character connection digital sequence are collectively referred to as sub-character connection digital sequence.

[0029] For the first sub-character connected digital sequence, the second sub-character connected digital sequence, and the third sub-character connected digital sequence, the offset distance set of each character is obtained, and the expression is: In the formula, represents the kth element in the bias distance set, Indicates that the first sub-character connects the kth element in the digital sequence, Indicates that the second sub-character connects the kth element in the digital sequence, Indicates that the third sub-character connects the k-th element in the digital sequence.

[0030] The optimal string length of each character is constructed according to the distribution of elements in the bias distance set. The expression is: In the formula, Represents a character in a character class set The optimal series length is Represents the median value of all elements in the bias distance set, Represents the first elements, Represents the first The index value of the element, represents the mean of all elements in the deviation distance set, Represents the maximum value of all elements in the bias distance set, Represents the minimum value of all elements in the bias distance set, Represents the number of all elements in the deviation distance set.

[0031] For each element in the sub-character connection digital sequence, the higher the influence, the greater the weight attached to the element in the calculated bias distance set, indicating that the character is more important, so the additional weight of the element in the bias distance set is The bigger, The larger the value, the more characters are calculated. Optimal series length The larger the value, the more characters The optimal series length is also larger.

[0032] The above steps are used to calculate the optimal series length of all characters in the character category set of the water conservancy coding data. Combined with the frequency of each character in the character category set in the water conservancy coding data, the optimal sliding window length of the water conservancy coding data is constructed. The expression is: In the formula, represents the optimal sliding window length for water conservancy coded data, represents the rounding function, Indicates the first character in the character class set The optimal run length of characters, Indicates the number of characters contained in the character category set. Indicates the first character in the character class set The frequency of occurrence of characters in water conservancy coding data, Represents the mean frequency of all characters in the character category set.

[0033] The higher the frequency of a character, the more important it is to the water conservancy coding data. The larger it is, the larger the corresponding optimal sliding window length is.

[0034] Step S003, combining the optimal sliding window length of the water conservancy coding data with the LZ77 compression algorithm to complete the compression of the BIM water conservancy project data.

[0035] The optimal sliding window length As the sliding window length of the LZ77 compression algorithm, the improvement of the LZ77 compression algorithm is completed, and the BIM water conservancy project data is used as the input of the improved LZ77 compression algorithm, and the output result is the compressed BIM water conservancy project data, so as to realize the management of large-scale water conservancy projects. The optimal sliding window length value calculated by the optimal series length of each character and its accompanying weight is more suitable for BIM water conservancy project data, which improves the compression efficiency and compression quality of the LZ77 compression algorithm.

[0036] In summary, the embodiments of the present invention solve the problem of low compression efficiency and poor quality of BIM water conservancy project data due to too long or too short sliding window length when the LZ77 compression algorithm compresses BIM water conservancy project data. By analyzing the distribution characteristics of characters in water conservancy project data, the optimal sliding window length of water conservancy coding data is determined, thereby improving the compression efficiency and compression quality.

[0037] It should be noted that the sequence of the above embodiments of the present invention is only for description and does not represent the advantages and disadvantages of the embodiments. The above is a description of a specific embodiment of this specification. In addition, the processes depicted in the accompanying drawings do not necessarily require the specific order or continuous order shown to achieve the desired results. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0038] The various embodiments in this specification are described in a progressive manner, and the same or similar parts between the various embodiments can be referenced to each other, and each embodiment focuses on the differences from other embodiments.

[0039] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the principles of the present invention should be included in the protection scope of the present invention.

Claims

1. A data processing method for large-scale project management, characterized in that: The method comprises the following steps: Get all characters in the water conservancy coding data; All characters are deduplicated to obtain a character category set; multiple character extension scales are set; semantic character string sequences of characters are obtained according to the character extension scales; string frequency sets of each character under each character extension scale are obtained according to the frequencies of occurrence of each semantic character string sequence; character connection digital quantities under each character extension scale are obtained according to the semantic character string sequence and the string frequency set; character connection digital quantities under all character extension scales are arranged to obtain a character connection digital sequence of each character; and each sub-character connection digital sequence is obtained according to the maximum value distribution of elements in the character connection digital sequence; All sub-character connection digital sequences are summarized to obtain the deviation distance set of each character; the optimal series length of each character is obtained according to the distribution of elements in the deviation distance set; the optimal sliding window length of water conservancy coding data is obtained according to the optimal series length, and the water conservancy project management is completed by combining the LZ77 compression algorithm and the optimal sliding window length; The step of summarizing all sub-character connection digital sequences to obtain a set of offset distances for each character includes: Calculate the sum of the first element in the first sub-character connected digital sequence, twice the first element in the second sub-character connected digital sequence, and three times the first element in the third sub-character connected digital sequence, calculate the ratio of the sum to 6, and use the ratio as the first element in the deviation distance set. Use the same acquisition method as the first element in the deviation distance set to calculate other elements in the deviation distance set.

2. A data processing method for large-scale project management according to claim 1, characterized in that: The step of obtaining the semantic character string sequences of the characters according to the extension scale of the characters includes: For each character in the character category set, each character extension scale is extended backward in the water conservancy coding data, and the character sequence obtained after the extension is used as each semantic character string sequence.

3. A data processing method for large-scale project management according to claim 1, characterized in that: The method of obtaining a string frequency set of each character under each character extension scale according to the frequency of occurrence of each semantic string sequence includes: For each character in the character category set, the frequency of occurrence of each semantic string sequence in the water conservancy coding data at each character extension scale is counted, and the frequency of occurrence of all semantic string sequences at the same character extension scale is taken as the string frequency set of each character at each character extension scale.

4. A data processing method for large-scale project management according to claim 1, characterized in that: The character connection digital quantity under each character extension scale is obtained according to the semantic character string sequence and the character string frequency set, and the expression is: In the formula, Represents a character in a character class set In character extension scale The following characters are connected to the numerical quantity, Representing characters In character extension scale The number of different semantic string sequences is obtained by Representing characters In character extension scale The first string frequency in the following set elements, Indicates the character extension scale, Representing characters In character extension scale The total number of semantic string sequences is obtained as follows: Indicates the floor symbol, is the adjustment factor.

5. A data processing method for large-scale project management according to claim 1, characterized in that: The step of arranging the character connection digital quantities under the extended scale of all characters to obtain the character connection digital sequence of each character includes: For each character in the character category set, the character connection digital quantities under all character extension scales are arranged in ascending order of the character extension scales as the character connection digital sequence of each character.

6. A data processing method for large-scale project management according to claim 1, characterized in that: The step of obtaining each sub-character connection digital sequence according to the maximum value distribution of elements in the character connection digital sequence includes: For each character connection digital sequence, the maximum value, the second maximum value and the third maximum value in the character connection digital sequence are obtained, and all the elements remaining in the character connection digital sequence after eliminating the maximum value, the second maximum value and the third maximum value are equally divided into a first part, a second part and a third part according to their positions in the character connection digital sequence, and the sequences consisting of the maximum value and the first part, the second maximum value and the second part, and the third maximum value and the third part are respectively used as the first sub-character connection digital sequence, the second sub-character connection digital sequence, and the third sub-character connection digital sequence.

7. A data processing method for large-scale project management according to claim 1, characterized in that: The optimal series length of each character is obtained according to the distribution of elements in the bias distance set, and the expression is: In the formula, Represents a character in a character class set The optimal series length, Represents the median value of all elements in the bias distance set, Represents the first elements, Represents the first The index value of the element, represents the mean of all elements in the deviation distance set, Represents the maximum value of all elements in the bias distance set, Represents the minimum value of all elements in the bias distance set, Represents the number of all elements in the deviation distance set.

8. A data processing method for large-scale project management according to claim 1, characterized in that: The optimal sliding window length of the water conservancy coding data is obtained according to the optimal series length, and the expression is: In the formula, represents the optimal sliding window length for water conservancy coded data, represents the rounding function, Indicates the first character in the character class set The optimal run length of characters, Indicates the number of characters contained in the character category set. Indicates the first character in the character class set The frequency of occurrence of characters in water conservancy coding data, Represents the mean frequency of all characters in the character category set.

9. A data processing method for large-scale project management according to claim 1, characterized in that: The water conservancy project management is completed by combining the LZ77 compression algorithm and the optimal sliding window length, including: The optimal sliding window length of water conservancy coding data is used as the sliding window length of the improved LZ77 compression algorithm, and the water conservancy coding data is compressed using the improved LZ77 compression algorithm.

Images