Data isolation method of integrated test service platform and platform

Abstract

The invention provides a data isolation method and platform for an integrated test service platform, and relates to the technical field of data processing. The method comprises the following steps: acquiring historical user data, and performing data storage amount analysis based on application types to form database space use feature data; obtaining current user use information, and performing space division in combination with the database space use feature data to form multi-user database division information; and according to the multi-user database division information, performing data optimization based on type similarity to form multi-user shared application isolation database data. According to the method, the historical data of the platform is used for feature extraction to achieve reasonable storage space division of the current user, the utilization rate of the database storage space is improved, and meanwhile effective data isolation between different users is practically guaranteed.

Description

Technical Field

[0001] The present invention relates to the field of data processing technology, and in particular to a data isolation method and platform for an integrated testing service platform. Background Art

[0002] With the development of science and technology, the methods and approaches for AI model training have gradually matured, and the main factors affecting the training model have gradually shifted from the method to the use and processing of computing power and training data volume. At present, in order to apply model training in actual engineering situations, some companies have provided an integrated testing service platform that can provide model training services for multiple tenants.

[0003] Of course, for the integrated testing platform, although it can integrate algorithms and data information efficiently and quickly, its multi-user characteristics will inevitably lead to the situation where data between different users are stored together in the platform's database, which may lead to data confusion among different users. This is not conducive to training processing and also has the risk of exposing important user data information.

[0004] Therefore, designing a data isolation method and platform for an integrated testing service platform, realizing reasonable storage space division for current users by utilizing the platform's historical data for feature extraction, improving the utilization rate of database storage space while also ensuring effective data isolation between different users, is an urgent problem to be solved. Summary of the invention

[0005] The purpose of the present invention is to provide a data isolation method for an integrated testing service platform, by providing a relatively independent storage space in the database for different users, and storing all the data corresponding to the user in this relatively independent storage space. To achieve this method, it is first necessary to determine the amount of data storage that the user may have to ensure that the independent storage space divided fully meets the user's data storage needs. On the other hand, it is necessary to ensure that the data in the storage space can be uniquely and stably corresponded to the user to ensure that there will be no mutual extraction and confusion of data in different storage spaces. Considering these two aspects, this application estimates the data storage of the current user by extracting features from the data storage of historical users on the platform to determine the size of the storage space division and establish an identity connection to be closely associated with the storage space. In addition, considering that the data information of different users is partially the same when the training models are similar, the repeated storage of these data wastes the storage space of the database, so the same data can be optimized for storage to improve the utilization of the database storage space, and confusion in data use can also be avoided.

[0006] The purpose of the present invention is also to provide a data isolation platform for an integrated testing service platform, which obtains historical user data through a data acquisition unit, and extracts feature information through an analysis and processing unit, while realizing an estimate of the current user's data usage, thereby ensuring that a reasonable size of independent storage space is provided to the user, so that user data is confidential and avoids confusion and interaction with other users' data. Different units are closely connected, effectively realizing the mutual isolation of data use for multiple users, effectively ensuring the security of user data use, and also improving the utilization efficiency of the platform database space, which is an important material basis for achieving effective data isolation.

[0007] In the first aspect, the present invention provides a data isolation method for an integrated testing service platform, including: obtaining historical user data, performing data storage volume analysis based on application type, and forming database space usage characteristic data; obtaining current user usage information, and performing space division in combination with the database space usage characteristic data to form multi-user database division information; based on the multi-user database division information, performing data optimization based on type similarity to form multi-user shared application isolation database data.

[0008] In the present invention, the method provides a relatively independent storage space in the database for different users, and stores all the data corresponding to the user in this relatively independent storage space. To achieve this method, it is first necessary to determine the amount of data storage that the user may have to ensure that the independent storage space divided fully meets the user's data storage needs. On the other hand, it is necessary to ensure that the data in the storage space can be uniquely and stably corresponded to the user to ensure that there will be no mutual extraction and confusion of data in different storage spaces. Considering these two aspects, this application estimates the data storage of the current user by extracting features from the data storage of historical users on the platform to determine the size of the storage space division and establish an identity connection to closely associate with the storage space. In addition, considering that the data information of different users is partially the same when the training models are similar, the repeated storage of these data wastes the storage space of the database. Therefore, the same data can be optimized for storage to improve the utilization of the database storage space, and at the same time, confusion in data usage can be avoided.

[0009] As a possible implementation method, historical user data is obtained, and data storage capacity analysis based on application type is performed to form database space usage characteristic data, including: extracting storage usage information corresponding to different historical users with the same application type in the historical user data to form storage usage information of the same type of historical users; performing storage occupancy feature analysis based on the influx of single training data on the storage usage information of all the same type of historical users of the same application type to determine the type envelope occupancy corresponding to the application type; and aggregating the type envelope occupancy corresponding to different application types to form database space usage characteristic data.

[0010] In the present invention, in order to achieve data isolation between different users on the integrated testing service platform, it is first necessary to provide reasonable storage space for different users in the database to ensure the data storage needs of the users. Therefore, the data storage of the users must be accurately and reasonably estimated to reasonably divide the storage space in the database. Here, the corresponding storage space estimation reference data is established by extracting the space size characteristics of the usage data of the historical users of the platform to ensure the storage space of the current user. It can be understood that the types of training required for different users are different, and thus the amount of training data provided may be different, especially the influx of training data provided at a single time is quite different. Therefore, when extracting the data storage feature information of historical users, the application type of the model is used as the clustering condition, which can more reasonably extract the features of the user's data storage, and ensure that the subsequent storage space size that fully meets the needs of the current user is provided. Make a reasonable data reference. In addition, considering that users do not provide all training data at once when using data storage, but provide it continuously in batches, the size of the storage space used has a certain dynamic stability. Therefore, when extracting feature information, it is necessary to fully consider the characteristics of this multiple data influx to provide a more reasonable estimate of the size of the storage space.

[0011] As a possible implementation method, the storage usage information of all historical users of the same type of the same application type is analyzed based on the storage feature of a single training data influx, and the type envelope storage corresponding to the application type is determined, including: the storage usage information of different historical users of the same type under the same application type is extracted based on the intersection stability of the training data influx items, and the stable storage of the same type of users corresponding to the historical users is determined. ; According to the stable share of users of the same type corresponding to different historical users under the same application type , conduct average inventory analysis based on data clustering to determine the type envelope inventory corresponding to the application type.

[0012] In the present invention, in order to determine the reasonable data storage corresponding to an application type, we should first determine the amount of data that has been occupied by historical users of this type for a long time, and combine these data amounts to perform an average storage analysis based on big data, and then obtain the representative data storage that can reflect this type.

[0013] As a possible implementation method, the storage usage information of different historical users of the same type under the same application type is extracted based on the intersection stability of the training data influx sub-items to determine the stable storage usage of the same type of users corresponding to the historical users. , including: storing usage information for different historical users of the same type, extracting the average training data influx of a single user for each training session during the entire usage cycle, the corresponding training data continuous storage cycle duration, and the basic storage capacity of the training model According to the duration of the continuous storage period of all training data of a single training and the corresponding average influx of training data of a single user, determine the maximum influx of user training data during the period when the influx of training data in the usage cycle does not overlap with the influx of training data in the adjacent items in time. and the amount of overlapping user training data inflow corresponding to each overlapping period during which the training data inflow in the usage cycle overlaps in time with the training data inflow in the adjacent sub-item , n represents the number of different application types, m represents the number of different historical users under the application type numbered n, and k represents the sequential number of the overlapped training data corresponding to the historical user numbered m under the application type numbered n; the influx of training data of different overlapping users corresponding to the historical users Perform effective stability analysis in the following way: Collect all overlapping user training data influx , forming a user training data overlap influx set ; Overlap influx set of user training data ,like , and the largest influx of overlapping user training data in the set If the overlap duration reaches the stable overlap duration threshold, the largest overlap user training data influx Determine the effective stable overlap influx for users Otherwise, determine the effective stable overlap influx of users =0, Indicates the set of overlapping influx of user training data The variance of Indicates the stable variance limit of the overlapping influx data; based on the basic storage capacity of the training model , the maximum influx of user training data And the effective and stable overlapping influx of users , determine the stable share of users of the same type corresponding to historical users ,in .

[0014] In the present invention, the common method of historical users providing training data in batches is considered, so the extraction of the characteristic quantity of the occupied space size needs to fully consider this dynamic stability. Here, different historical users with the same application type need to obtain information on the amount of training data provided each time to determine the maximum single training data influx. However, after each training data volume is provided, there may be a situation where the training data volume provided last time has not been completely used and consumed, and the next training data volume will flow in again, so the size of the space occupied by the data is not just the size of the single data influx. Because when obtaining the user's single maximum data influx, it is also necessary to confirm the data volume during the storage period when adjacent data influx items overlap. Of course, it is understandable that the divided independent storage space is not exactly the same as the maximum storage space that has ever occurred in the history of users, but will provide a certain redundancy. Therefore, whether the amount of data during the overlap of adjacent items needs to be considered needs to confirm whether there is often a certain amount of overlap. After all, the redundancy of the space occupied by occasional or small amounts of data can be temporarily compensated, while the long-term repeated storage of a large space needs to consider that there will be other data input during the storage, resulting in insufficient space storage size. Whether the data volume during such a long-term and large repetition period is considered is determined by judging the stability of the data volume and the duration of the repetition cycle. The stability is reflected in the form of variance, and the length of the occupation cycle is determined by a reasonable duration threshold. The stable variance limit and stable overlap duration threshold for the overlapping influx data can be set according to actual conditions, or can be determined based on big data analysis.

[0015] As a possible implementation method, according to the stable stock of the same type of users corresponding to different historical users under the same application type, an average stock analysis based on data clustering is performed to determine the type envelope stock corresponding to the application type, including: for different historical users under the same application type, according to the corresponding stable stock of the same type of users , perform the following cluster average inventory analysis: For each user of the same type, the stable inventory , and other similar users have a stable share of the stock The total difference is sorted from small to large, and the stable share of users of the same type corresponding to the total difference of the first half is taken. The average value is calibrated as the type envelope share corresponding to the application type.

[0016] In the present invention, the stable storage of different historical users under the same application type in the historical data will have a certain degree of difference in the amount of data due to the trained model, and the features that need to be extracted are big data features, which mainly reflect the average level of storage used by users under this type, and this average level is related to the size of the gap between the stable storage of different users. Therefore, this application obtains the difference between the stable storage of different users to extract the stable storage data of the part with a smaller gap to determine the storage space required for the corresponding application type. Here, the number extracted according to the total difference can be half of the stable storage of users, or it can be reasonably extracted according to the size difference between the total differences. It can be understood that the part with a smaller difference in the total difference indicates a higher degree of aggregation, and the obtained data can fully reflect the space usage characteristics of the application type.

[0017] As a possible implementation method, current user usage information is obtained, and space is divided in combination with database space usage characteristic data to form multi-user database division information, including: determining current application types corresponding to different current users based on current user usage information of different current users; determining type envelope occupancy corresponding to application types that are the same as the current application type based on current application types corresponding to different current users and in combination with database space usage characteristic data, and determining the type envelope occupancy as the user preset occupancy corresponding to the current user; obtaining the total occupancy of the platform database, and dividing the user occupancy space according to the user preset occupancy corresponding to different current users to form multi-user database division information.

[0018] In the present invention, with feature data information as a reference for the size of the space for dividing independent storage spaces among users, after obtaining the training model types of different current users, the corresponding type envelope occupancy can be determined based on the application type. Dividing the storage space of different current users in the database based on the type envelope occupancy can provide a certain amount of space redundancy to cope with the space usage that cannot be covered by the big data. Of course, more importantly, after completing the division and storing the current user's usage data in the divided storage space, it is also necessary to calibrate the storage space with the user's identity information as a label to ensure that the use of the data in the storage space can only be achieved through the correct user identity information authentication, so as to achieve a better data isolation effect.

[0019] As a possible implementation method, the total storage space occupied by the platform database is obtained, and the user storage space is divided according to the preset storage space corresponding to different current users to form multi-user database division information, including: User presets corresponding to different current users , the user space is divided in the following way, i represents the number of different current users: , then the user presets corresponding to different current users occupy For the space division amount, the independent space corresponding to each current user is divided in the platform database, marked as the user independent space, and the current usage data of different current users is stored in the corresponding user independent space. α is the space division restriction factor; if , then the user presets corresponding to different current users occupy Mutual comparison platform database The space of different sizes is divided proportionally to determine the independent user storage space corresponding to different current users, and the current usage data of different current users are stored in the corresponding independent user storage space; the independent user storage space is marked with the identity information of the corresponding current user, and the identity information of all independent user storage spaces and the corresponding current users are extracted to form an isolated space identity contact form; the independent user storage space corresponding to different current users and the isolated space identity contact form are combined to form multi-user database division information.

[0020] In the present invention, of course, since the space size of the database of the integrated test service platform is also limited, the total space size of the database also needs to be considered when dividing the storage space size of different users. After all, the database not only needs to store the usage data of different users, but also needs to provide storage space for its own data information analysis and processing as well as necessary data such as different model data. Therefore, when dividing the independent storage space of different users, it is also necessary to fully consider it. This application divides the space according to the size of the envelope space occupied by the feature data when there is enough space. When the database space is insufficient, the space division of different users is carried out according to the proportional relationship of the envelope space occupied. Of course, in this case, it is necessary to provide the platform with a limit on the increase in the number of users and a size limit on the current user data influx. For the space division limit factor, it can be set according to the size of the necessary data actually stored by the platform.

[0021] As a possible implementation method, data optimization based on type similarity is performed based on multi-user database division information to form multi-user shared application isolation database data, including: according to the current application types corresponding to different current users, current users with the same current application type are determined, and they are grouped into a set of users with the same type of usage; for different sets of users with the same type of usage, the current usage data in the user-independent storage space corresponding to all current users in the set is obtained, and the same data is extracted and shared to form multi-user shared application isolation database data.

[0022] In the present invention, current users of the same application type may have the same data input when training the model because the trained models are similar or identical. Although different users store the same data independently to ensure data isolation, the utilization rate of the storage space will be reduced, especially when the amount of the same data is large. The present application establishes a shared storage space in the database for unified storage of the same data on the basis of reasonable and effective isolation of the data of different users, avoiding waste of space caused by multiple storage.

[0023] As a possible implementation method, for different sets of users of the same type, the current usage data in the user-independent storage space corresponding to all current users in the set is obtained, and the same data is extracted and shared to form multi-user shared application isolation database data, including: for all current usage data corresponding to the same set of users, if the same data information exists at the same time, the same data information is extracted and stored in the shared storage space of the platform database, and the same data information is associated with the identity information of the corresponding current user; the identity association information of all the same data in the shared storage space is extracted to form a shared data form; all the same data in the shared storage space and the shared data form are combined to form multi-user shared application isolation database data.

[0024] In the present invention, it is considered that the same data is for different users. Therefore, in order to avoid errors in sharing the same data, it is necessary to mark the same data with the identity information of the corresponding user. Therefore, all data in the entire shared storage space are identity-identifiable and can be obtained through identity information when used by users, which ensures that the data is effectively isolated and improves the utilization rate of the storage space.

[0025] In a second aspect, the present invention provides a data isolation platform for an integrated testing service platform, comprising: a data acquisition unit for acquiring historical user data and current user usage information; an analysis and processing unit for performing data storage capacity analysis based on application type on the historical user data acquired by the data acquisition unit to form database space usage characteristic data, performing spatial division of the current user usage information in combination with the database space usage characteristic data to form multi-user database division information, and performing shared optimization processing of the same data based on the current user usage information of the current user to form multi-user shared isolated database data; a database storage unit for independently storing the current user usage information based on the multi-user database division information, and performing shared storage of the same data based on the multi-user shared isolated database data.

[0026] In the present invention, the platform obtains historical user data through the data collection unit, and extracts feature information through the analysis and processing unit, while realizing the estimation of the current user's data storage volume, thereby ensuring that a reasonable size of independent storage space is provided to the user, so that the user data is confidential and avoids confusion and interaction with other users' data. Different units are closely connected, effectively realizing the mutual isolation of data use for multiple users, effectively ensuring the security of user data use, and also improving the utilization efficiency of the platform database space, which is an important material basis for realizing effective data isolation.

[0027] The data isolation method and platform of an integrated testing service platform provided by the present invention have the following beneficial effects: This method provides a relatively independent storage space in the database for different users, and stores all the data corresponding to the user in this relatively independent storage space. To implement this method, it is first necessary to determine the amount of data storage that the user may have to ensure that the independent storage space divided fully meets the user's data storage needs. On the other hand, it is necessary to ensure that the data in the storage space can be uniquely and stably corresponded to the user to ensure that there will be no mutual extraction and confusion of data in different storage spaces. Considering these two aspects, this application estimates the data storage of the current user by extracting features from the data storage of historical users on the platform to determine the size of the storage space division and establish an identity connection to closely associate with the storage space. In addition, considering that the data information of different users is partially the same when the training models are similar, the repeated storage of these data wastes the storage space of the database. Therefore, the same data can be optimized for storage to improve the utilization of the database storage space, while also avoiding confusion in data usage.

[0028] The platform obtains historical user data through the data collection unit, and extracts feature information through the analysis and processing unit, while estimating the current user's data storage, thereby ensuring that users are provided with a reasonable size of independent storage space, so that user data is confidential and avoids confusion and interaction with other users' data. Different units are closely connected, effectively realizing the mutual isolation of multi-user data use, effectively ensuring the security of user data use, and also improving the utilization efficiency of the platform database space, which is an important material basis for achieving effective data isolation. BRIEF DESCRIPTION OF THE DRAWINGS

[0029] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings required for use in the embodiments of the present invention are briefly introduced below. It should be understood that the following drawings only show certain embodiments of the present invention and therefore should not be regarded as limiting the scope. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without creative work.

[0030] Figure 1 A step diagram of a data isolation method for an integrated testing service platform provided by an embodiment of the present invention; Figure 2 A schematic diagram of the structure of a data isolation platform of an integrated testing service platform provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0031] The technical solutions in the embodiments of the present invention will be described below in conjunction with the accompanying drawings in the embodiments of the present invention.

[0032] With the development of science and technology, the methods and approaches for AI model training have gradually matured, and the main factors affecting the training model have gradually shifted from the method to the use and processing of computing power and training data volume. At present, in order to apply model training in actual engineering situations, some companies have provided an integrated testing service platform that can provide model training services for multiple tenants.

[0033] Of course, for the integrated testing platform, although it can integrate algorithms and data information efficiently and quickly, its multi-user characteristics will inevitably lead to the situation where data between different users are stored together in the platform's database, which may lead to data confusion among different users. This is not conducive to training processing and also has the risk of exposing important user data information.

[0034] refer to Figure 1-Figure 2, an embodiment of the present invention provides a data isolation method for an integrated testing service platform, which provides a relatively independent storage space in the database for different users, and stores all the data corresponding to the user in this relatively independent storage space. To achieve this method, it is first necessary to determine the amount of data storage that the user may have to ensure that the independent storage space divided fully meets the user's data storage needs. On the other hand, it is necessary to ensure that the data in the storage space can be uniquely and stably corresponded to the user to ensure that there will be no mutual extraction and confusion of data in different storage spaces. Considering these two aspects, this application estimates the data storage of the current user by extracting features from the data storage of historical users on the platform to determine the size of the storage space division and establish an identity connection to be closely associated with the storage space. In addition, considering that the data information of different users is partially the same when the training models are similar, the repeated storage of these data causes a waste of the storage space of the database, so the same data can be optimized for storage to improve the utilization of the database storage space, and at the same time, confusion in data use can be avoided.

[0035] The data isolation method of the integrated testing service platform specifically includes the following steps: S1: Obtain historical user data, perform data storage volume analysis based on application types, and form database space usage feature data.

[0036] Obtain historical user data, perform data storage capacity analysis based on application type, and form database space usage characteristic data, including: extracting storage usage information corresponding to different historical users of the same application type in the historical user data to form storage usage information of the same type of historical users; performing storage occupancy feature analysis based on the influx of single training data on the storage usage information of all the same type of historical users of the same application type to determine the type envelope occupancy corresponding to the application type; and aggregating the type envelope occupancy corresponding to different application types to form database space usage characteristic data.

[0037] To achieve data isolation between different users on the integrated testing service platform, it is first necessary to provide different users with reasonable storage space in the database to ensure the user's data storage needs. Therefore, the user's data storage volume must be accurately and reasonably estimated to reasonably divide the storage space in the database. Here, the corresponding storage space estimation reference data is established by extracting the space size characteristics of the platform's historical users' usage data to ensure the current user's storage space. It can be understood that the types of training required for different users are different, and therefore the amount of training data provided may be different, especially the amount of training data provided at a time is quite different. Therefore, when extracting data storage feature information for historical users, the application type of the model is used as the clustering condition, which can more reasonably extract features of the user's data storage volume, and ensure that the subsequent storage space size that fully meets the needs of the current user is provided. Make a reasonable data reference. In addition, considering that users do not provide all training data at once when using data storage, but provide it continuously in batches, the size of the storage space used has a certain dynamic stability. Therefore, when extracting feature information, it is necessary to fully consider the characteristics of this multiple data influx to provide a more reasonable estimate of the size of the storage space.

[0038] Performing storage feature analysis based on a single training data influx on the storage usage information of all historical users of the same application type to determine the type envelope storage corresponding to the application type, including: extracting storage feature based on the intersection stability of the training data influx items of storage usage information of different historical users of the same application type to determine the stable storage of the same type of users corresponding to the historical users ; According to the stable share of users of the same type corresponding to different historical users under the same application type , conduct average inventory analysis based on data clustering to determine the type envelope inventory corresponding to the application type.

[0039] To determine the reasonable data usage corresponding to an application type, we should first determine the amount of data that has been historically used by users of this type over a long period of time, and then conduct an average usage analysis based on big data based on these data, and then obtain the representative data usage that can reflect this type.

[0040] The storage usage information of different historical users of the same type under the same application type is extracted based on the intersection stability of the training data influx sub-items to determine the stable storage usage of the same type of users corresponding to the historical users. , including: storing usage information for different historical users of the same type, extracting the average training data influx of a single user for each training session during the entire usage cycle, the corresponding training data continuous storage cycle duration, and the basic storage capacity of the training model According to the duration of the continuous storage period of all training data of a single training and the corresponding average influx of training data of a single user, determine the maximum influx of user training data during the period when the influx of training data in the usage cycle does not overlap with the influx of training data in the adjacent items in time. and the amount of overlapping user training data inflow corresponding to each overlapping period during which the training data inflow in the usage cycle overlaps in time with the training data inflow in the adjacent sub-item , n represents the number of different application types, m represents the number of different historical users under application type numbered n, and k represents the sequential number of overlapping training data corresponding to historical users numbered m under application type numbered n; the influx of training data of different overlapping users corresponding to historical users Perform effective stability analysis in the following way: Collect all overlapping user training data influx , forming a user training data overlap influx set ; Overlap influx set of user training data ,like , and the largest influx of overlapping user training data in the set If the overlap duration reaches the stable overlap duration threshold, the largest overlap user training data influx Determine the effective stable overlap influx for users Otherwise, determine the effective stable overlap influx of users =0, Indicates the set of overlapping influx of user training data The variance of Indicates the stable variance limit of the overlapping influx data; based on the basic storage capacity of the training model , the maximum influx of user training data And the effective and stable overlapping influx of users , determine the stable share of users of the same type corresponding to historical users ,in .

[0041] Considering the common method of historical users providing training data in batches, the extraction of the space occupied feature needs to fully consider this dynamic stability. Here, different historical users with the same application type need to obtain information on the amount of training data provided each time to determine the maximum single training data influx. However, considering that after each training data is provided, there may be a situation where the last training data has not been fully used and consumed, and the next training data will flow in again. In this way, the space occupied by the data is not just the size of the single data influx. Because when obtaining the user's maximum single data influx, it is also necessary to confirm the data size during the storage period when adjacent data influx items overlap. Of course, it is understandable that the divided independent storage space is not exactly the same as the maximum storage space that has ever occurred in the history of users, but will provide a certain redundancy. Therefore, whether the amount of data during the overlap of adjacent items needs to be considered needs to confirm whether there is often a certain amount of overlap. After all, the redundancy of the space occupied by occasional or small amounts of data can be temporarily compensated, while the long-term repeated storage of a large space needs to consider that there will be other data input during the storage, resulting in insufficient space storage size. Whether the data volume during such a long-term and large repetition period is considered is determined by judging the stability of the data volume and the duration of the repetition cycle. The stability is reflected in the form of variance, and the length of the occupation cycle is determined by a reasonable duration threshold. The stable variance limit and stable overlap duration threshold for the overlapping influx data can be set according to actual conditions, or can be determined based on big data analysis.

[0042] According to the stable stock of the same type of users corresponding to different historical users under the same application type, an average stock analysis based on data clustering is performed to determine the type envelope stock corresponding to the application type, including: for different historical users under the same application type, according to the corresponding stable stock of the same type of users , perform the following cluster average inventory analysis: For each user of the same type, the stable inventory , and other similar users have a stable share of the stock The total difference is sorted from small to large, and the stable share of users of the same type corresponding to the total difference of the first half is taken. The average value is calibrated as the type envelope share corresponding to the application type.

[0043] For the stable stock of different historical users under the same application type in the historical data, there will be a certain degree of difference in the amount of data due to the trained model, and the features that need to be extracted are big data features, which mainly reflect the average level of stock used by users under this type, and this average level is related to the size of the gap between the stable stocks of different users. Therefore, this application obtains the difference between the stable stocks of different users to extract the stable stock data of the part with a smaller gap to determine the stock space required for the corresponding application type. Here, the number extracted according to the total difference can be half of the stable stock of the users, or it can be reasonably extracted according to the size difference between the total differences. It can be understood that the part with a smaller difference in the total difference indicates a higher degree of aggregation, and the obtained data can fully reflect the space occupancy characteristics of the application type.

[0044] S2: Obtain the current user usage information, and perform space division in combination with the database space usage feature data to form multi-user database division information.

[0045] The current user usage information is obtained, and the space is divided in combination with the database space usage characteristic data to form multi-user database division information, including: according to the current user usage information of different current users, the current application types corresponding to different current users are determined; according to the current application types corresponding to different current users, in combination with the database space usage characteristic data, the type envelope occupancy corresponding to the application type that is the same as the current application type is determined, and the type envelope occupancy is determined as the user preset occupancy corresponding to the current user; the total occupancy of the platform database is obtained, and the user occupancy space is divided according to the user preset occupancy corresponding to different current users to form multi-user database division information.

[0046] With feature data information as a reference for the size of the space for dividing independent storage space for users, after obtaining the training model types of different current users, the corresponding type envelope occupancy can be determined based on the application type. Dividing the database storage space for different current users based on the type envelope occupancy can provide a certain amount of space redundancy to cope with the space usage that cannot be covered by big data. Of course, more importantly, after completing the division and storing the current user's usage data in the divided storage space, it is also necessary to calibrate the storage space with the user's identity information as a label to ensure that the use of the data in the storage space can only be achieved through the correct user identity information authentication, so as to achieve a better data isolation effect.

[0047] Obtain the total storage space of the platform database, and divide the user storage space according to the preset storage space corresponding to different current users to form multi-user database division information, including: User presets corresponding to different current users , the user space is divided in the following way, i represents the number of different current users: , then the user presets corresponding to different current users occupy For the space division amount, the independent space corresponding to each current user is divided in the platform database, marked as the user independent space, and the current usage data of different current users is stored in the corresponding user independent space. α is the space division restriction factor; if , then the user presets corresponding to different current users occupy Mutual comparison platform database The space of different sizes is divided proportionally to determine the independent user storage space corresponding to different current users, and the current usage data of different current users are stored in the corresponding independent user storage space; the independent user storage space is marked with the identity information of the corresponding current user, and the identity information of all independent user storage spaces and the corresponding current users are extracted to form an isolated space identity contact form; the independent user storage space corresponding to different current users and the isolated space identity contact form are combined to form multi-user database division information.

[0048] Of course, since the space size of the database of the integrated testing service platform is also limited, the total space size of the database also needs to be considered when dividing the storage space size of different users. After all, the database not only needs to store the usage data of different users, but also needs to provide storage space for its own data information analysis and processing as well as necessary data such as different model data. Therefore, when dividing the independent storage space of different users, it is also necessary to fully consider it. This application divides the space according to the size of the envelope space occupied by the feature data when there is enough space. When the database space is insufficient, the space division of different users is carried out according to the proportional relationship of the envelope space occupied. Of course, in this case, it is necessary to provide the platform with a limit on the increase in the number of users and a size limit on the current user data influx. For the space division limit factor, it can be set according to the size of the necessary data actually stored by the platform.

[0049] S3: Based on the multi-user database partition information, data optimization based on type homogeneity is performed to form multi-user shared application isolation database data.

[0050] According to the multi-user database division information, data optimization based on type similarity is performed to form multi-user shared application isolation database data, including: according to the current application types corresponding to different current users, current users with the same current application type are determined, and they are grouped into user sets of the same type; for different user sets of the same type, the current usage data in the user-independent storage space corresponding to all current users in the set is obtained, and the same data is extracted and shared to form multi-user shared application isolation database data.

[0051] Current users of the same application type may have the same data input when training the model because the trained models are similar or identical. Although different users store the same data independently to ensure data isolation, the utilization rate of storage space will be reduced, especially when the amount of the same data is large. This application establishes a shared storage space in the database for unified storage of the same data on the basis of reasonable and effective isolation of data of different users, avoiding waste of space caused by multiple storage.

[0052] For different sets of users of the same type, the current usage data in the user-independent storage space corresponding to all current users in the set is obtained, and the same data is extracted and shared to form multi-user shared application isolation database data, including: for all current usage data corresponding to the same set of users, if the same data information exists at the same time, the same data information is extracted and stored in the shared storage space of the platform database, and the same data information is associated with the identity information of the corresponding current user; the identity association information of all the same data in the shared storage space is extracted to form a shared data form; all the same data in the shared storage space and the shared data form are combined to form multi-user shared application isolation database data.

[0053] Considering that the same data is for different users, in order to avoid errors in sharing the same data, it is necessary to mark the same data with the corresponding user's identity information. Therefore, all data in the entire shared storage space are identity-identifiable and can be obtained through identity information when used by users, ensuring effective isolation of data while improving the utilization of storage space.

[0054] The present invention also provides a data isolation platform for an integrated testing service platform, which includes: a data acquisition unit, which is used to obtain historical user data and current user usage information; an analysis and processing unit, which is used to analyze the data storage volume of the historical user data acquired by the data acquisition unit based on the application type, form database space usage characteristic data, perform spatial division of the current user usage information in combination with the database space usage characteristic data, form multi-user database division information, and perform shared optimization processing of the same data according to the current user usage information of the current user, to form multi-user shared isolated database data; a database storage unit, which is used to independently store the current user usage information according to the multi-user database division information, and perform shared storage of the same data according to the multi-user shared isolated database data.

[0055] The platform obtains historical user data through the data collection unit, and extracts feature information through the analysis and processing unit, while estimating the current user's data storage, thereby ensuring that users are provided with a reasonable size of independent storage space, so that user data is confidential and avoids confusion and interaction with other users' data. Different units are closely connected, effectively realizing the mutual isolation of multi-user data use, effectively ensuring the security of user data use, and also improving the utilization efficiency of the platform database space, which is an important material basis for achieving effective data isolation.

[0056] In summary, the data isolation method and platform of the integrated testing service platform provided by the embodiment of the present invention have the following beneficial effects: This method provides a relatively independent storage space in the database for different users, and stores all the data corresponding to the user in this relatively independent storage space. To implement this method, it is first necessary to determine the amount of data storage that the user may have to ensure that the independent storage space divided fully meets the user's data storage needs. On the other hand, it is necessary to ensure that the data in the storage space can be uniquely and stably corresponded to the user to ensure that there will be no mutual extraction and confusion of data in different storage spaces. Considering these two aspects, this application estimates the data storage of the current user by extracting features from the data storage of historical users on the platform to determine the size of the storage space division and establish an identity connection to closely associate with the storage space. In addition, considering that the data information of different users is partially the same when the training models are similar, the repeated storage of these data wastes the storage space of the database. Therefore, the same data can be optimized for storage to improve the utilization of the database storage space, while also avoiding confusion in data usage.

[0057] In the embodiments of the present application, "indication" may include direct indication and indirect indication, and may also include explicit indication and implicit indication. The information indicated by a certain information is called information to be indicated. In the specific implementation process, there are many ways to indicate the information to be indicated, such as but not limited to, the information to be indicated can be directly indicated, such as the information to be indicated itself or the index of the information to be indicated. The information to be indicated can also be indirectly indicated by indicating other information, wherein there is an association between the other information and the information to be indicated. It is also possible to indicate only a part of the information to be indicated, while the other parts of the information to be indicated are known or agreed in advance. For example, the indication of specific information can also be achieved by means of the arrangement order of each piece of information agreed in advance (for example, specified by the protocol), thereby reducing the indication overhead to a certain extent. At the same time, the common parts of each piece of information can also be identified and indicated uniformly to reduce the indication overhead caused by indicating the same information separately.

[0058] In addition, the specific indication method may also be various existing indication methods, such as but not limited to the above-mentioned indication methods and various combinations thereof. The specific details of the various indication methods can refer to the prior art and will not be repeated herein. As can be seen from the above, for example, when it is necessary to indicate multiple information of the same type, different indication methods may be used for different information. In the specific implementation process, the desired indication method can be selected according to specific needs. The embodiment of the present application does not limit the selected indication method. In this way, the indication method involved in the embodiment of the present application should be understood to cover various methods that can enable the party to be indicated to obtain the information to be indicated.

[0059] It should be understood that the information to be indicated can be sent as a whole, or divided into multiple sub-information and sent separately, and the sending period and / or sending time of these sub-information can be the same or different. The specific sending method is not limited in the embodiment of the present application. Among them, the sending period and / or sending time of these sub-information can be pre-defined, for example, pre-defined according to a protocol, or can be configured by the sending end device by sending configuration information to the receiving end device.

[0060] "Pre-definition" or "pre-configuration" can be implemented by pre-saving corresponding codes, tables or other methods that can be used to indicate relevant information in the device, and the embodiments of the present application do not limit the specific implementation method. Among them, "saving" can mean saving in one or more memories. The one or more memories can be set separately or integrated in an encoder or decoder, a processor, or a communication device. The one or more memories can also be partially set separately and partially integrated in a decoder, a processor, or a communication device. The type of memory can be any form of storage medium, which is not limited by the embodiments of the present application.

[0061] The "protocol" involved in the embodiments of the present application may refer to a protocol family in the communication field, a standard protocol with a similar protocol family frame structure, or a related protocol used in future communication systems, and the embodiments of the present application do not make specific limitations on this.

[0062] In the embodiments of the present application, descriptions such as "when...", "in the case of...", "if" and "if" all mean that the device will make corresponding processing under certain objective circumstances. It does not limit the time, nor does it require the device to have a judgment action when implementing it, nor does it mean that there are other limitations.

[0063] In the description of the embodiments of the present application, unless otherwise specified, " / " indicates that the objects associated before and after are in an "or" relationship. For example, A / B can represent A or B; "and / or" in the embodiments of the present application is only a description of the association relationship of the associated objects, indicating that there can be three relationships. For example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone. These three situations, where A and B can be singular or plural. In addition, in the description of the embodiments of the present application, unless otherwise specified, "multiple" refers to two or more than two. "At least one of the following" or similar expressions refers to any combination of these items, including any combination of single items or plural items. For example, at least one of a, b or c can represent: a, b, c, ab, ac, bc, or abc, where a, b, c can be single or multiple. In addition, in order to facilitate the clear description of the technical solution of the embodiments of the present application, in the embodiments of the present application, the words "first" and "second" are used to distinguish the same or similar items with basically the same functions and effects. Those skilled in the art will appreciate that words such as "first" and "second" do not limit the quantity and execution order, and words such as "first" and "second" do not necessarily limit the difference. At the same time, in the embodiments of the present application, words such as "exemplary" or "for example" are used to indicate examples, illustrations or explanations. Any embodiment or design described as "exemplary" or "for example" in the embodiments of the present application should not be interpreted as being more preferred or more advantageous than other embodiments or designs. Specifically, the use of words such as "exemplary" or "for example" is intended to present related concepts in a concrete way for easy understanding.

[0064] It should be understood that the processor in the embodiment of the present application may be a central processing unit (CPU), and the processor may also be other general-purpose processors, digital signal processors (DSP), application specific integrated circuits (ASIC), field programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor may be a microprocessor or the processor may also be any conventional processor, etc.

[0065] It should also be understood that the memory in the embodiments of the present application may be a volatile memory or a non-volatile memory, or may include both volatile and non-volatile memories. Among them, the non-volatile memory may be a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), or a flash memory. The volatile memory may be a random access memory (RAM), which is used as an external cache. By way of example and not limitation, many forms of random access memory (RAM) are available, such as static RAM (SRAM), dynamic random access memory (DRAM), synchronous DRAM (SDRAM), double data rate synchronous dynamic random access memory (DDR SDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synchronous link DRAM (SLDRAM), and direct rambus RAM (DR RAM).

[0066] The above embodiments can be implemented in whole or in part by software, hardware (such as circuits), firmware or any other combination. When implemented using software, the above embodiments can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions or computer programs. When the computer instructions or computer programs are loaded or executed on a computer, the process or function described in the embodiment of the present application is generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions can be transmitted from one website, computer, server or data center to another website, computer, server or data center by wired (such as infrared, wireless, microwave, etc.). The computer-readable storage medium can be any available medium that can be accessed by a computer or a data storage device such as a server or data center that contains one or more available media sets. The available medium can be a magnetic medium (for example, a floppy disk, a hard disk, a tape), an optical medium (for example, a DVD), or a semiconductor medium. The semiconductor medium can be a solid-state hard disk.

[0067] It should be understood that the term "and / or" in this article is only a description of the association relationship of associated objects, indicating that there can be three relationships. For example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone. A and B can be singular or plural. In addition, the character " / " in this article generally indicates that the associated objects before and after are in an "or" relationship, but it may also indicate an "and / or" relationship. Please refer to the context for specific understanding.

[0068] In this application, "at least one" means one or more, and "plurality" means two or more. "At least one of the following" or similar expressions refers to any combination of these items, including any combination of single items or plural items. For example, at least one of a, b, or c can mean: a, b, c, ab, ac, bc, or abc, where a, b, and c can be single or multiple.

[0069] It should be understood that in the various embodiments of the present application, the size of the serial numbers of the above-mentioned processes does not mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present application.

[0070] Those of ordinary skill in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.

[0071] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the systems, devices and units described above can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0072] In the several embodiments provided in the present application, it should be understood that the disclosed systems, devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0073] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0074] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit.

[0075] If the functions are implemented in the form of software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application can be essentially or partly embodied in the form of a software product that contributes to the prior art. The computer software product is stored in a storage medium and includes several instructions for a computer device (which can be a personal computer, server, or network device, etc.) to perform all or part of the steps of the methods described in each embodiment of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM), random access memory (RAM), disk or optical disk, and other media that can store program codes.

[0076] The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any person skilled in the art who is familiar with the present technical field can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

Claims

1. A data isolation method for an integrated testing service platform, characterized in that: include: Obtain historical user data, perform data storage volume analysis based on application types, and form database space usage feature data; The acquisition of historical user data, analysis of data storage volume based on application type, and formation of database space usage feature data include: Extracting storage usage information corresponding to different historical users with the same application type from the historical user data to form storage usage information of historical users of the same type; Performing a storage feature analysis based on a single training data influx on all the same type of historical user storage usage information of the same application type to determine the type envelope storage corresponding to the application type; Gather the type envelope occupancy corresponding to different application types to form the database space usage characteristic data; Acquire current user usage information, and perform space division in combination with the database space usage characteristic data to form multi-user database division information; According to the multi-user database division information, data optimization based on type sameness is performed to form multi-user shared application isolation database data.

2. The data isolation method of the integrated testing service platform according to claim 1, characterized in that: The performing of storage feature analysis based on a single training data influx on all the same type of historical user storage usage information of the same type of application to determine the type envelope storage corresponding to the application type includes: The storage usage information of different historical users of the same type under the same application type is extracted based on the intersection stability of the training data influx items to determine the stable storage usage of the same type of users corresponding to the historical users. ; According to the stable share of users of the same type corresponding to different historical users under the same application type , perform average inventory analysis based on data clustering to determine the type envelope inventory corresponding to the application type.

3. The data isolation method of the integrated testing service platform according to claim 2, characterized in that: The storage usage information of different historical users of the same type under the same application type is extracted based on the intersection stability of the training data influx items to determine the stable storage usage of the same type of users corresponding to the historical users ,include: For different historical users of the same type, storage usage information is extracted to extract the average training data influx of a single user for each training in the entire usage cycle, the corresponding training data continuous storage cycle duration, and the basic storage capacity of the training model. ; According to the duration of the continuous storage period of all training data of a single training and the corresponding average training data influx of a single user, the maximum user training data influx during the period when the training data influx in the usage cycle does not overlap with the training data influx in the adjacent items in time is determined. and the amount of overlapping user training data inflow corresponding to each overlapping period during which the training data inflow in the usage cycle overlaps in time with the training data inflow in the adjacent sub-item , n represents the number of different application types, m represents the number of the historical user not used in the application type numbered n, and k represents the sequential number of the overlapping multiple training data corresponding to the historical user numbered m in the application type numbered n; The influx of training data of different overlapping users corresponding to the historical users Perform effective stability analysis in the following ways: Collect all the overlapping user training data influx , forming a user training data overlap influx set ; The user training data overlaps the influx set ,like , and the largest influx of overlapping user training data in the set If the overlap duration reaches the stable overlap duration threshold, the largest overlap user training data influx Determine the effective stable overlap influx for users Otherwise, determine the effective stable overlap influx of the user =0, Indicates determining the user training data overlap influx set The variance of It represents the stable variance limit of coincident influx data; According to the basic storage capacity of the training model , the maximum amount of user training data influx And the effective stable overlap influx of the user , determine the stable share of the same type of users corresponding to the historical users ,in .

4. The data isolation method of the integrated testing service platform according to claim 3 is characterized in that: The average share analysis based on data clustering is performed according to the stable share of the same type of users corresponding to different historical users under the same application type to determine the type envelope share corresponding to the application type, including: For different historical users of the same application type, the corresponding stable share of users of the same type is calculated. , perform cluster average inventory analysis in the following ways: Stable inventory for each user of the same type , and indeed the same type of users as other mentioned above occupy a stable share of the stock The total difference is sorted from small to large, and the average value of the stable stock of the same type of users corresponding to the total difference of the first half is taken, and it is marked as the type envelope stock corresponding to the application type.

5. The data isolation method of the integrated testing service platform according to claim 4 is characterized in that: The obtaining of the current user usage information and combining the database space usage characteristic data to perform space division to form multi-user database division information includes: Determining the current application types corresponding to different current users according to the current user usage information of different current users; According to the current application types corresponding to different current users and in combination with the database space usage characteristic data, determine the type envelope occupancy corresponding to the application type that is the same as the current application type, and determine the type envelope occupancy as the user preset occupancy corresponding to the current user; The total storage space occupied by the platform database is obtained, and user storage space is divided according to the preset storage spaces corresponding to different current users to form the multi-user database division information.

6. The data isolation method of the integrated testing service platform according to claim 5, characterized in that: The acquiring of the total storage space occupied by the platform database and the partitioning of user storage space according to the preset storage spaces occupied by different current users to form the multi-user database partitioning information includes: According to the total inventory of the platform database The preset user storage space corresponding to different current users is divided into user storage space in the following manner, where i represents the number of different current users: like , then the user preset inventory corresponding to the different current users For the occupied space division amount, an independent occupied space corresponding to each current user is divided in the platform database, marked as user independent occupied space, and the current usage data of different current users is stored in the corresponding user independent occupied space, α is the space division restriction factor; like , then the user preset inventory corresponding to the different current users Mutual comparison platform database The space of different sizes is proportionally divided to determine the user independent storage spaces corresponding to different current users, and the current usage data of different current users are stored in the corresponding user independent storage spaces; Mark the user's independent occupied space with the corresponding identity information of the current user, and extract all the user's independent occupied spaces and the corresponding identity information of the current user to form an isolated space identity contact form; The multi-user database partition information is formed by combining the user independent storage spaces corresponding to different current users and the isolation space identity contact forms.

7. The data isolation method of the integrated testing service platform according to claim 6, characterized in that: The step of performing data optimization based on type homogeneity according to the multi-user database partition information to form multi-user shared application isolation database data includes: According to the current application types corresponding to different current users, the current users with the same current application type are determined and grouped into a user set of the same type; For different sets of users of the same type, current usage data in the user independent storage space corresponding to all current users in the set is obtained, and the same data is extracted and shared to form the multi-user shared application isolation database data.

8. The data isolation method of the integrated testing service platform according to claim 7, characterized in that: The method of obtaining the current usage data in the independent user storage space corresponding to all current users in the set for different sets of users of the same type and performing extraction and sharing processing of the same data to form the multi-user shared application isolation database data includes: For all the current usage data corresponding to the same type of user set, if the same data information exists at the same time, the same data information is extracted and stored in the shared storage space of the platform database, and the same data information is associated with the identity information of the corresponding current user; Extracting identity association information of all identical data in the shared storage space to form a shared data form; All the same data in the shared storage space and the shared data table are combined to form the multi-user shared application isolation database data.

9. A data isolation platform for an integrated testing service platform, using the data isolation method for an integrated testing service platform according to any one of claims 1 to 8, characterized in that: include: Data collection unit, used to obtain historical user data and current user usage information; an analysis and processing unit, configured to analyze the data storage volume of the historical user data acquired by the data acquisition unit based on the application type to form database space usage characteristic data, perform spatial division on the current user usage information in combination with the database space usage characteristic data to form multi-user database division information, and perform sharing optimization processing on the same data according to the current user usage information of the current user to form multi-user shared isolated database data; The performing of data storage volume analysis based on application type on the historical user data acquired by the data acquisition unit to form database space usage feature data includes: Extracting storage usage information corresponding to different historical users with the same application type from the historical user data to form storage usage information of historical users of the same type; Performing a storage feature analysis based on a single training data influx on all the same type of historical user storage usage information of the same application type to determine the type envelope storage corresponding to the application type; Gather the type envelope occupancy corresponding to different application types to form the database space usage characteristic data; The database storage unit is used to independently store the current user's usage information according to the multi-user database division information, and to share the same data according to the multi-user shared isolation database data.

Images