A fast storage method for multidimensional data based on big data

Abstract

The present invention relates to the field of data processing technology, and more specifically to a method for fast storage of multidimensional data based on big data. The method comprises: obtaining common coefficients of the dimensional data based on the amount of dimensional data contained in different data systems, the time when any dimensional data is retrieved, and the number of times the dimensional data is retrieved by users; dividing the dimensional data into different types of dimensional data, including public dimensional data, private dimensional data, and pending dimensional data, based on the size of the common coefficients of the dimensional data; obtaining the semi-public likelihood of the pending dimensional data based on whether the pending dimensional data has been retrieved in the past; dividing the pending dimensional data into semi-public dimensional data and private dimensional data based on the size of the semi-public likelihood, and storing the different data separately. The present invention effectively improves the storage efficiency of different dimensional data by optimizing the storage strategy.

Description

Technical Field

[0001] The present invention relates to the field of data processing technology, and in particular to a method for fast storage of multidimensional data based on big data. Background Art

[0002] The smart campus platform uses big data analysis technology to provide teachers and students with a comprehensive and collaborative smart living campus environment, and provides intelligent, personalized and convenient information services for teaching, scientific research, management and learning life.

[0003] The core part of the smart campus platform is data organization and storage, that is, the establishment of a database. However, since the data in the smart campus platform comes from multiple data systems and there are complex correlations between different data, the consistency and integrity of the data content are challenged when the data is stored and called in each data system. The storage method of data of different dimensions greatly affects the actual effect of the data system when retrieving and storing data. If the storage method is inappropriate, the storage and calling effect of the data in the corresponding data system will be unsatisfactory. Summary of the Invention

[0004] The present invention provides a fast storage method for multidimensional data based on big data to solve the existing problems.

[0005] The present invention provides a fast multi-dimensional data storage method based on big data using the following technical solutions:

[0006] An embodiment of the present invention provides a method for fast storage of multidimensional data based on big data, the method comprising the following steps:

[0007] Obtaining multiple dimensional data contained in multiple data systems;

[0008] Based on the amount of dimension data contained in different data systems, the time for retrieving any dimension data, and the number of times the dimension data is retrieved by users, the common coefficients of the dimension data are obtained, and according to the size of the common coefficients of the dimension data, the dimension data are divided into different types of dimension data, including public dimension data, private dimension data, and undetermined dimension data;

[0009] According to the call status of the pending dimension data in the past, the semi-public possibility of the pending dimension data is obtained, and the pending dimension data is divided into semi-public dimension data and private dimension data according to the size of the semi-public possibility;

[0010] Public dimension data, private dimension data, semi-public dimension data and private dimension data are stored separately.

[0011] Furthermore, the method of obtaining the common coefficients of the dimensional data by combining the number of dimensional data included in different data systems, the time of retrieving any dimensional data, and the number of times the dimensional data is retrieved by the user includes the following specific methods:

[0012] According to the number of data systems containing any dimension data and the number of times the dimension data is retrieved by users within a preset time, respectively obtain the public frequency and retrieval frequency of the dimension data;

[0013] Obtain the scaling factor for each dimension data based on the number of identical dimension data in different data systems and the time it takes to retrieve any dimension data;

[0014] The common coefficient of the dimensional data is obtained according to the common frequency, the retrieval frequency and the scaling factor of the dimensional data, wherein the common frequency, the retrieval frequency and the scaling factor are all proportional to the common coefficient.

[0015] Furthermore, the specific method for obtaining the public frequency and the retrieval frequency of the dimensional data is as follows:

[0016] For any dimension data, obtain the number of data systems containing the dimension data, the number of all data systems, the number of times the dimension data was retrieved by users within a preset historical time period, and the total number of times the dimension data was retrieved by users;

[0017] The ratio of the number of data systems containing dimension data to the number of all data systems is recorded as the common frequency of dimension data;

[0018] The ratio of the number of times the dimension data is retrieved by the user within a preset historical time period to the total number of times the dimension data is retrieved by the user is recorded as the retrieval frequency of the dimension data.

[0019] Furthermore, the scaling factor of each dimension data is obtained according to the number of identical dimension data in different data systems and the time of retrieving any dimension data, including the specific method of:

[0020] The specific calculation method of the scaling factor of dimensional data is:

[0021]

[0022] Where, Indicates the The scaling factor of the dimensional data, Indicates the The data system The retrieval time of each dimension data, Indicates the A data system for the The retrieval time of each dimension data, Indicates the A data system and The number of elements in the intersection of dimensional data between data systems, Indicates the A data system and The number of elements in the union of dimensional data between the data systems, To traverse A data system and The first mean obtained after all combinations of data systems, To traverse A data system and The second mean obtained from all combinations of data systems, Represents an exponential function with a natural constant as its base.

[0023] Furthermore, the dimensional data is divided into different types of dimensional data according to the size of the common coefficient of the dimensional data, including public dimensional data, private dimensional data, and undetermined dimensional data, including the specific method of:

[0024] Preset first threshold and the second threshold , the dimensional data whose public coefficient is greater than or equal to the first threshold is recorded as public dimensional data, the dimensional data whose public coefficient is less than the first threshold and greater than or equal to the second threshold is recorded as pending dimensional data, and the dimensional data whose public coefficient is less than or equal to the second threshold is recorded as private dimensional data.

[0025] Furthermore, the method of obtaining the semi-public possibility of the pending dimension data according to the situation in which the pending dimension data was called in the past time includes the following specific methods:

[0026] According to the call status of the pending dimension data in the past, obtain the historical call reference value, the historical average call frequency of the pending dimension data, the synchronization parameter, and the total historical call frequency of all data systems calling the pending dimension data;

[0027] The historical call reference value is the average of the historical average call frequencies of all undetermined dimension data;

[0028] For any pending dimensional data, obtain the ratio of the historical average call frequency of the pending dimensional data to the historical call reference value, and record it as the first ratio of the pending dimensional data; obtain the ratio of the coherence parameter of the pending dimensional data to the total historical call frequency of all data systems calling the pending dimensional data, and record it as the second ratio of the pending dimensional data; obtain the semi-public possibility of the pending dimensional data based on the first ratio and the second ratio of the pending dimensional data, and both the first ratio and the second ratio are positively correlated with the semi-public possibility of the pending dimensional data.

[0029] Furthermore, the specific method for obtaining the historical average call frequency is:

[0030] A historical time period and a historical time interval are preset. The historical time interval includes several historical time periods. The number of times the pending dimension data is called by all data systems within a historical time period is recorded as the historical call parameter of the pending dimension data in the historical time period. The average historical call parameter of the pending dimension data in all historical time periods is recorded as the historical average call frequency of the pending dimension data.

[0031] Furthermore, the specific method for obtaining the coherence parameters is:

[0032] The number of times that the pending dimension data and any public dimension data are simultaneously retrieved by all data systems within the historical time interval is recorded as the synchronization factor between the pending dimension data and the public dimension data, and the average of the synchronization factors between the pending dimension data and all public dimension data is recorded as the synchronization parameter of the pending dimension data.

[0033] Furthermore, the specific calculation method of the semi-public possibility of the undetermined dimensional data is:

[0034] The specific calculation expression of semi-public possibility is:

[0035]

[0036] in, Indicates the The semi-public possibility of undetermined dimension data, Indicates the The historical average call frequency of undetermined dimension data, Indicates the historical call reference value, Indicates the The homology parameters of the data with undetermined dimensions, Indicates that all data system calls The total frequency of historical calls to undetermined dimension data, Represents the softsign normalization function.

[0037] Furthermore, the public dimension data, private dimension data, semi-public dimension data and private dimension data are stored separately, including the specific method of:

[0038] Set up several servers, use the server with the largest storage space as the master server and assign one server to each data system;

[0039] Private dimension data is stored in the server corresponding to the data system to which it belongs, all private dimension data is stored in the server corresponding to the data system that has called the private dimension data, and public dimension data is stored in the main server; semi-public data dimensions are stored in the main server and the server corresponding to the data system to which they belong.

[0040] The beneficial effects of the technical solution of the present invention are as follows: by analyzing the repetitiveness and call frequency of each data in numerous subsystems and combining the correlation between each data, the data is divided into public dimension data, private dimension data, semi-public dimension data and private dimension data. By dividing the dimension data types and storing them according to different characteristics, the efficiency of data access can be improved. Public dimension data and private dimension data are stored on different servers respectively, which reduces data redundancy and provides faster data access speed, which is conducive to unified processing of public dimension data and semi-public dimension data in the main server, avoiding the loss of part of the data when delayed replication is performed to maintain final consistency during decentralized storage; and for private data and private dimension data, they are stored in their respective data systems, avoiding the waste of resources caused by redundant storage when storing them, that is, By storing different types of dimensional data on different servers, they can be reasonably allocated according to their call frequency and publicness, reducing the waste of storage space. Private dimensional data is stored in the server corresponding to the corresponding data system, ensuring the security and privacy of the data. Only users with corresponding permissions can access it, which is conducive to improving data security and preventing data loss due to accidental deletion or modification during a system operation. At the same time, by dynamically adjusting the data storage strategy based on the call frequency and historical usage of the dimensional data, access to commonly used data can be made more real-time and responsive. By using the concept of semi-public possibility to analyze and classify the dimensional data to be determined, the data storage strategy can be adjusted more intelligently, making data storage more flexible and efficient. Therefore, by optimizing the storage strategy, the storage effect of different dimensional data is effectively improved. BRIEF DESCRIPTION OF THE DRAWINGS

[0041] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0042] Figure 1 This is a flowchart of the steps of a method for fast storage of multidimensional data based on big data according to the present invention;

[0043] Figure 2 This is a schematic diagram of the smart campus platform. DETAILED DESCRIPTION

[0044] To further illustrate the technical means and effectiveness of the present invention to achieve its intended purpose, the following, in conjunction with the accompanying drawings and preferred embodiments, describes in detail a method for rapidly storing multidimensional data based on big data, including its specific implementation, structure, features, and effectiveness. In the following description, different references to "one embodiment" or "another embodiment" do not necessarily refer to the same embodiment. Furthermore, specific features, structures, or characteristics of one or more embodiments may be combined in any suitable manner.

[0045] Unless defined otherwise, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention belongs.

[0046] The specific solution of the multi-dimensional data fast storage method based on big data provided by the present invention is described in detail below with reference to the accompanying drawings.

[0047] See also Figure 1 , which shows a flowchart of a method for fast storage of multidimensional data based on big data provided by one embodiment of the present invention, the method comprising the following steps:

[0048] Step S001: Acquire a plurality of dimensional data respectively contained in a plurality of data systems.

[0049] It should be noted that the smart campus platform is a campus management platform formed by a collection of multiple systems, usually including several data systems, such as: administrative system, teaching system, scientific research system and library system, and each system is aimed at different users, such as Figure 2The following diagram shows a schematic diagram of the smart campus platform, which includes students, teachers, and staff. Therefore, different users have multi-dimensional data information in each system. For example, in the administrative system, there is student and teacher identity information, department information, staff identity information, and campus department information, namely the Logistics Management Department. The Logistics Management Department includes several functional departments and business service departments. Therefore, to improve the storage efficiency of user data in the smart campus platform, it is necessary to analyze the multi-dimensional data in the smart campus platform.

[0050] Specifically, in order to implement the multi-dimensional data fast storage method based on big data proposed in this embodiment, it is necessary to first collect data of several dimensions. The specific process is as follows:

[0051] Through the database in the smart campus platform, several dimensional data of all users in all data systems in the smart campus platform are obtained.

[0052] So far, several dimensional data of all users in several data systems have been obtained through the above method.

[0053] Step S002: Based on the number of dimensional data contained in different data systems, the time for retrieving any dimensional data, and the number of times the dimensional data is retrieved by the user, the common coefficients of the dimensional data are obtained, and according to the size of the common coefficients of the dimensional data, the dimensional data are divided into different types of dimensional data, including public dimensional data, private dimensional data, and pending dimensional data.

[0054] Step 2.1: Acquire common coefficients of the dimensional data based on the amount of dimensional data included in different data systems, the time for retrieving any dimensional data, and the number of times the dimensional data is retrieved by the user.

[0055] As an embodiment, the specific calculation method of the common coefficient is:

[0056] First, according to the number of data systems containing any dimensional data and the number of times the dimensional data is retrieved by users within a preset time, the public frequency and the retrieval frequency of the dimensional data are respectively obtained.

[0057] Then, the scaling factor of each dimension data is obtained according to the number of identical dimension data in different data systems and the time of retrieving any dimension data.

[0058] Finally, the common coefficient of the dimensional data is obtained according to the common frequency, the access frequency and the scaling factor of the dimensional data, wherein the common frequency, the access frequency and the scaling factor are all proportional to the common coefficient.

[0059] As an embodiment, the specific expression of the common coefficient of the dimensional data is:

[0060]

[0061] in, Represents the common coefficients of dimensional data; Indicates the number of data systems that contain dimensional data, represents the number of all data systems, Indicates the number of times dimension data is retrieved by users within the preset historical time period. Indicates the total number of times dimension data is retrieved by users. Indicates the scaling factor of the dimensional data; Represents the common frequency of dimensional data, Indicates the frequency of retrieval of dimension data.

[0062] The method for obtaining the scaling factor of the dimensional data is:

[0063] Obtain the response time of each data system of the smart campus platform when retrieving any dimensional data, which is recorded as the retrieval time of the dimensional data by the data system; obtain the scaling coefficient of the dimensional data based on the set relationship between the dimensional data of different data systems of the smart campus platform and the difference in retrieval time of different data systems retrieving the same dimensional data.

[0064] As an embodiment, the specific calculation method of the scaling coefficient of the dimensional data is:

[0065]

[0066] Where, Indicates the The scaling factor of the dimensional data, Indicates the The data system The retrieval time of each dimension data, Indicates the The data system The retrieval time of each dimension data, Indicates the A data system and The number of elements in the intersection of dimensional data between data systems, Indicates the A data system and The number of elements in the union of dimensional data between the data systems, To traverse A data system and The first mean obtained after all combinations of data systems, To traverse A data system and The second mean obtained from all combinations of data systems, Represents an exponential function with a natural constant as its base.

[0067] It should be noted that due to the different design architectures of data in different data systems, when users call the same dimensional data in different data systems, there will be different system response times. In this embodiment, the data of the same dimension that is frequently called in different data systems during the system architecture design process is regarded as a kind of public data. Therefore, the closer the call time for the same dimensional data in different data systems is, the more likely the dimensional data is to be public data, and the larger the scaling coefficient of the dimensional data is, to ensure that the public coefficient of the dimensional data is larger. In addition, the more the same dimensional data is contained between different data systems in the campus smart platform, the higher the degree of sharing of the dimensional data in the data system, and the larger the corresponding public coefficient.

[0068] Step 2.2: Divide the dimensional data into different types of dimensional data according to the size of the public coefficients of the dimensional data, including public dimensional data, private dimensional data, and pending dimensional data.

[0069] The specific division method is: preset the first threshold and the second threshold , the dimensional data whose public coefficient is greater than or equal to the first threshold is recorded as public dimensional data, the dimensional data whose public coefficient is less than the first threshold and greater than or equal to the second threshold is recorded as pending dimensional data, and the dimensional data whose public coefficient is less than or equal to the second threshold is recorded as private dimensional data.

[0070] It should be noted that the first threshold is preset based on experience , the second threshold , can be adjusted according to actual conditions, and this embodiment does not specifically limit it.

[0071] It should be noted that, since different dimensional data are called by different data systems, and the calling situations of different dimensional data are also different during the calling process, this embodiment divides the dimensional data into different categories by analyzing the calling situation of dimensional data in all data systems. In the obtained division results, public dimensional data indicates that the dimensional data is called by multiple data systems, and private dimensional data indicates that the corresponding dimensional data is only called by individual subsystems. By dividing the different dimensional data, it is convenient to subsequently store the dimensional data by type according to the calling situation of the dimensional data in the data system, thereby improving the storage effect of different dimensional data. That is, through the division of dimensional data, some basic information and dimensional data common to each data system are stored in the overall database, which is conducive to unified processing of these dimensional data and avoids data loss caused by decentralized storage. As for private dimensional data, since it only exists in individual data systems, the private dimensional data can be stored separately in the servers of their corresponding data systems to avoid resource waste caused by redundant storage of private dimensional data.

[0072] So far, the public dimension data, private dimension data, and pending dimension data have been obtained through the above method.

[0073] Step S003: According to the situation in which the pending dimension data has been called in the past, the semi-public possibility of the pending dimension data is obtained, and the pending dimension data is divided into semi-public dimension data and private dimension data according to the size of the semi-public possibility.

[0074] It should be noted that the smart campus platform often contains some infrequently used but relatively important data systems. When distributed storage is used, the dimensional data within these systems may not be updated when users retrieve the data due to its infrequent use. To determine whether these dimensional data need to be stored with the public dimensional data and backed up within the corresponding server of the data system, this embodiment selects these dimensional data based on how frequently they have been retrieved. The more frequently they are retrieved, the more important they are. This is irrelevant to whether the data system they belong to is frequently used or important. For example, English scores may be verified when registering for certain competitions, but the CET-4 and CET-6 registration systems are only used once or twice a year. In addition, the deeper the correlation between the pending dimensional data and the public dimensional data, that is, the pending dimensional data will always appear when the public data appears, the more fundamental and important the pending dimensional data is. Therefore, when the dimensional data is subsequently stored, it needs to be stored with the public dimensional data and backed up within its respective data system.

[0075] Specifically, first, according to the situation that the pending dimension data was called in the past, the semi-public possibility of obtaining the pending dimension data is obtained.

[0076] As a preferred embodiment, the method for obtaining the semi-public probability of the undetermined dimensional data is:

[0077] For any pending dimensional data, obtain the ratio of the historical average call frequency of the pending dimensional data to the historical call reference value, and record it as the first ratio of the pending dimensional data; obtain the ratio of the coherence parameter of the pending dimensional data to the total historical call frequency of all data systems calling the pending dimensional data, and record it as the second ratio of the pending dimensional data; obtain the semi-public possibility of the pending dimensional data based on the first ratio and the second ratio of the pending dimensional data, and both the first ratio and the second ratio are positively correlated with the semi-public possibility of the pending dimensional data.

[0078] As an optional embodiment, a specific method for calculating the semi-public probability of the undetermined dimensional data is as follows:

[0079]

[0080] in, Indicates the The semi-public possibility of undetermined dimension data, Indicates the The historical average call frequency of undetermined dimension data, Indicates the historical call reference value, Indicates the The homology parameters of the data with undetermined dimensions, Indicates that all data system calls The total frequency of historical calls to undetermined dimension data, Represents the softsign normalization function.

[0081] The method for obtaining the historical average call frequency of the pending dimension data is as follows: presetting a historical time period and a historical time interval, where the historical time interval includes several historical time periods, recording the number of times the pending dimension data is called by all data systems within a historical time period as the historical call parameter of the pending dimension data in the historical time period, and recording the average historical call parameter of the pending dimension data in all historical time periods as the historical average call frequency of the pending dimension data.

[0082] It should be noted that, based on experience, the historical time period is preset to one month and the historical time interval is preset to one year, which can be adjusted according to circumstances and are not specifically limited in this embodiment.

[0083] The historical call reference value is the average of the historical average call frequencies of all pending dimension data.

[0084] The method for obtaining the synchronization parameter of the pending dimensional data is as follows: the number of times the pending dimensional data and any public dimensional data are simultaneously retrieved by all data systems within a historical time interval is recorded as the synchronization factor of the pending dimensional data and the public dimensional data dimension, and the average of the synchronization factors of the pending dimensional data and all public dimensional data dimensions is recorded as the synchronization parameter of the pending dimensional data.

[0085] The method for obtaining the total historical call frequency of the pending dimension data is as follows: the total number of times the pending dimension data is called by all data systems within a historical time interval is recorded as the total historical call frequency of the pending dimension data.

[0086] It should be noted that It is used to describe the frequency with which the pending dimension data has been retrieved in the past. The more times the pending dimension data is retrieved relative to the number of times all the pending dimension data are retrieved, the more frequently the pending dimension data is used, and the more likely it is to be public data. It is used to describe the association between the frequency of retrieval of pending dimension data and a large number of public dimension data. When the pending dimension data is called, if more public dimension data are called at the same time, it means that the correlation between the pending dimension data and the public dimension data is higher. The more commonly the pending dimension data is used and the higher the correlation with the public dimension data, the greater the possibility that it can be used as a semi-public data dimension.

[0087] Then, the undetermined dimension data is divided into semi-public dimension data and private dimension data according to the size of the semi-public possibility.

[0088] As an embodiment, the specific method for obtaining semi-public dimension data and private dimension data is: preset a semi-public threshold, record the pending dimension data with a semi-public possibility greater than or equal to the semi-public threshold as semi-public dimension data, and record the pending dimension data with a semi-public possibility less than the semi-public threshold as private dimension data.

[0089] It should be noted that, in this embodiment, the semi-public threshold is preset to 0.5 based on experience, and can be adjusted according to actual conditions. This embodiment does not impose any specific limitation.

[0090] It should be noted that this embodiment further divides the obtained pending dimensional data. If the semi-public probability of the pending dimensional data is greater than or equal to the semi-public threshold, it means that the pending dimensional data is more likely to be frequently used, so it is regarded as semi-public dimensional data; on the contrary, if the semi-public probability of the pending dimensional data is less than the semi-public threshold, the dimensional data is considered to be infrequently used data, so it is regarded as private dimensional data. This embodiment further divides the dimensional data into semi-public dimensional data and private dimensional data so that when data is subsequently stored, the relatively commonly used semi-public dimensional data can be stored in the general database and backed up in the corresponding data system, which is conducive to improving the security of data storage. As for the divided private dimensional data, since this dimensional data is relatively infrequently used but is called by many data systems, it should be stored multiple times when data is subsequently stored to ensure that the data will not be lost.

[0091] At this point, the semi-public dimension data and private dimension data are obtained through the above method.

[0092] Step S004: store the public dimension data, private dimension data, semi-public dimension data and private dimension data respectively.

[0093] Specifically, first, several servers are set up, the server with the largest storage space is used as the main server, and each data system corresponds to one server.

[0094] Secondly, the private dimension data is stored in a server corresponding to the data system to which it belongs, and all private dimension data are stored in a server corresponding to the data system that has called the private dimension data.

[0095] Finally, the public dimension data is stored in the main server; the semi-public data dimension is stored in the main server and the server corresponding to the data system to which it belongs.

[0096] It should be noted that since private dimension data is only called in the data system to which it belongs, in order to facilitate storage and calling, this embodiment stores it in the server corresponding to the data system to which each private dimension data belongs; although private dimension data is not frequently used, it will be called in multiple data systems. Therefore, it is necessary to ensure that the private dimension data can be quickly called and stored when called by the data system. Therefore, this embodiment chooses to store the private dimension data in the server corresponding to the called data system; since public dimension data is called frequently, as a kind of public data, it is the basic data of students, faculty and staff, and is called by multiple data systems. Therefore, to avoid the situation where the public dimension data is distributed and requires frequent synchronization, this embodiment stores it in the main server; since semi-public dimension data is called frequently, in order to ensure that semi-public data will not have storage errors or data loss problems, this embodiment chooses to store the semi-public data in the service corresponding to the main server and the data system to which it belongs to ensure storage security.

[0097] At this point, this embodiment is completed.

[0098] It should be noted that the The model is only used to represent negative correlation and constrain the output of the model to be in In the specific implementation, it can be replaced by other models with the same purpose. This embodiment is only based on The model is described as an example without any specific limitation. is the input to the model.

[0099] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the principles of the present invention should be included in the scope of protection of the present invention.

Claims

1. A fast multi-dimensional data storage method based on big data, characterized in that: The method comprises the following steps: Obtaining multiple dimensional data contained in multiple data systems; Based on the amount of dimension data contained in different data systems, the time for retrieving any dimension data, and the number of times the dimension data is retrieved by users, the common coefficients of the dimension data are obtained, and according to the size of the common coefficients of the dimension data, the dimension data are divided into different types of dimension data, including public dimension data, private dimension data, and undetermined dimension data; According to the situation in which the undetermined dimension data has been called in the past, the semi-public possibility of the undetermined dimension data is obtained, and the undetermined dimension data is divided into semi-public dimension data and private dimension data according to the size of the semi-public coefficient; Public dimension data, private dimension data, semi-public dimension data and private dimension data are stored separately; The method for obtaining the semi-public possibility of the pending dimension data is as follows: based on the calling situation of the pending dimension data in the past, respectively obtaining the historical call reference value and the historical average calling frequency of the pending dimension data, the coherence parameter, and the total historical calling frequency of all data systems calling the pending dimension data; The historical call reference value is the average of the historical average call frequencies of all undetermined dimension data; The specific calculation method of the semi-common coefficient of the data with undetermined dimensions is: in, Indicates the The semi-public coefficients of data with undetermined dimensions, Indicates the The historical average call frequency of undetermined dimension data, Indicates the historical call reference value, Indicates the The homology parameters of the data with undetermined dimensions, Indicates that all data system calls The total frequency of historical calls to undetermined dimension data, Represents the softsign normalization function.

2. The method for fast storage of multidimensional data based on big data according to claim 1, characterized in that: The method of obtaining the common coefficients of the dimensional data by combining the number of dimensional data included in different data systems, the time of retrieving any dimensional data, and the number of times the dimensional data is retrieved by the user includes the following specific methods: According to the number of data systems containing any dimension data and the number of times the dimension data is retrieved by users within a preset time, respectively obtain the public frequency and retrieval frequency of the dimension data; Obtain the scaling factor for each dimension data based on the number of identical dimension data in different data systems and the time it takes to retrieve any dimension data; The common coefficient of the dimensional data is obtained according to the common frequency, the retrieval frequency and the scaling factor of the dimensional data, wherein the common frequency, the retrieval frequency and the scaling factor are all proportional to the common coefficient.

3. The method for fast storage of multidimensional data based on big data according to claim 2, characterized in that: The specific method for obtaining the public frequency and retrieval frequency of the dimension data is as follows: For any dimension data, obtain the number of data systems containing the dimension data, the number of all data systems, the number of times the dimension data was retrieved by users within a preset historical time period, and the total number of times the dimension data was retrieved by users; The ratio of the number of data systems containing dimension data to the number of all data systems is recorded as the common frequency of dimension data; The ratio of the number of times the dimension data is retrieved by the user within a preset historical time period to the total number of times the dimension data is retrieved by the user is recorded as the retrieval frequency of the dimension data.

4. The method for fast storage of multidimensional data based on big data according to claim 2, characterized in that: The specific method for obtaining the scaling factor of each dimension data according to the amount of the same dimension data in different data systems and the time of retrieving any dimension data is as follows: The specific calculation method of the scaling factor of dimensional data is: Where, Indicates the The scaling factor of the dimensional data, Indicates the A data system for the The retrieval time of each dimension data, Indicates the A data system for the The retrieval time of each dimension data, Indicates the A data system and The number of elements in the intersection of dimensional data between data systems, Indicates the A data system and The number of elements in the union of dimensional data between the data systems, represents the mean function, Represents an exponential function with a natural constant as its base.

5. The method for fast storage of multidimensional data based on big data according to claim 1, characterized in that: The method of dividing the dimensional data into different types of dimensional data according to the size of the common coefficient of the dimensional data, including public dimensional data, private dimensional data, and undetermined dimensional data, includes the following specific methods: Preset first threshold and the second threshold , the dimensional data whose public coefficient is greater than or equal to the first threshold is recorded as public dimensional data, the dimensional data whose public coefficient is less than the first threshold and greater than or equal to the second threshold is recorded as pending dimensional data, and the dimensional data whose public coefficient is less than or equal to the second threshold is recorded as private dimensional data.

6. The method for fast storage of multidimensional data based on big data according to claim 1, characterized in that: The specific method for obtaining the historical average call frequency is: A historical time period and a historical time interval are preset. The historical time interval includes several historical time periods. The number of times the pending dimension data is called by all data systems within a historical time period is recorded as the historical call parameter of the pending dimension data in the historical time period. The average historical call parameter of the pending dimension data in all historical time periods is recorded as the historical average call frequency of the pending dimension data.

7. The method for fast storage of multidimensional data based on big data according to claim 6, characterized in that: The specific method for obtaining the coherence parameters is: The number of times that the pending dimension data and any public dimension data are simultaneously retrieved by all data systems within the historical time interval is recorded as the synchronization factor between the pending dimension data and the public dimension data, and the average of the synchronization factors between the pending dimension data and all public dimension data is recorded as the synchronization parameter of the pending dimension data.

8. The method for fast storage of multidimensional data based on big data according to claim 6, characterized in that: The specific method for obtaining the total historical call frequency of all data systems calling the undetermined dimension data is: The total number of times the pending dimension data is called by all data systems within the historical time interval is recorded as the total historical call frequency of the pending dimension data.

9. The method for fast storage of multidimensional data based on big data according to claim 1, characterized in that: The specific method of storing the public dimension data, private dimension data, semi-public dimension data and private dimension data separately includes: Set up several servers, use the server with the largest storage space as the master server and assign one server to each data system; The private dimension data is stored in the server corresponding to the data system to which it belongs, the private dimension data is stored in the server corresponding to the data system that has called the private dimension data, and the public dimension data is stored in the main server; The semi-public data dimensions are stored in the main server and the servers corresponding to the data systems to which they belong.

Images