Management method and system for corpus delivery and application rules

Abstract

The present invention provides a management method and system for corpus delivery and application rules, the method comprising: obtaining the user's delivery requirements, determining the target corpus according to the delivery requirements, and preparing the target corpus for delivery; after completing the delivery preparation, selecting the corresponding delivery method according to the user's requirements, and delivering the target corpus to the user; after completing the delivery, obtaining the application requirements of the target corpus, deploying the environment according to the application requirements and importing the delivered target corpus, testing and applying the target corpus based on the deployed environment; and monitoring and recording the use of the target corpus in real time. The present invention can select the corresponding target corpus according to the user's requirements and effectively manage it, thereby improving the efficiency of corpus management and facilitating the quality of user model training.

Description

Technical Field

[0001] The present invention belongs to the field of big data technology, and in particular relates to a management method and system for corpus delivery and application rules. Background Art

[0002] Artificial intelligence big models refer to "large parameter" models trained using large-scale data and powerful computing power. These models are usually highly versatile and generalized, and can be applied to natural language processing, image recognition, speech recognition and other fields. They can be divided into large language models, large visual models, multimodal large models, and basic large models.

[0003] Corpus is an important material used to train large artificial intelligence models. Corpus generally refers to examples and data sets used in linguistic research and natural language processing. It can be written text, oral records or other structured data, usually used to analyze language phenomena, support machine translation, speech recognition, automatic text summarization and other tasks. A common corpus is a set of text or speech data that has been collected, organized and annotated. These data are used in linguistic research to analyze language usage patterns, vocabulary changes and grammatical structures. In natural language processing, corpus is the basic data source for training and testing models, supporting machine translation, speech recognition, sentiment analysis and other functions.

[0004] In the existing technology, there is a lack of effective management of the corpus for training large models, and the quality of the corpus cannot be efficiently managed. There are also potential safety hazards, which can easily affect the quality of the large model obtained through final training. Summary of the invention

[0005] In view of the above-mentioned shortcomings of the prior art, an object of the present invention is to provide a method and system for managing corpus delivery and application rules, so as to solve the problem of low efficiency of corpus management in the prior art.

[0006] To achieve the above objectives and other related objectives, the present invention provides a method for managing corpus delivery and application rules, comprising:

[0007] Obtaining the user's delivery requirements, determining the target corpus according to the delivery requirements, and preparing the target corpus for delivery;

[0008] After the delivery preparation is completed, a corresponding delivery method is selected according to the user's needs, and the target corpus is delivered to the user;

[0009] After the delivery is completed, the application requirements of the target corpus are obtained, the environment is deployed according to the application requirements, and the delivered target corpus is imported, and the target corpus is tested and applied based on the deployed environment;

[0010] The usage of the target corpus is monitored and recorded in real time.

[0011] In one embodiment of the present invention, the delivery requirements include corpus type, corpus quantity, corpus format, delivery time and application scenario, and determining the target corpus according to the delivery requirements and preparing the target corpus for delivery include:

[0012] Selecting the target corpus according to the corpus type, the corpus format and the application scenario, and generating a compilation delivery document for the target corpus, the delivery document including a corpus list, metadata description, delivery time and usage guide;

[0013] Perform virus detection on the target corpus to remove potential safety hazards in the target corpus;

[0014] Performing an integrity check on the target corpus after the check.

[0015] In one embodiment of the present invention, the virus detection and elimination of the target corpus to remove the potential safety hazard in the target corpus includes:

[0016] Acquire feature information and path information of the target corpus, and calculate the complexity of the feature information and the sensitivity of the path information;

[0017] Dividing the target corpus into a plurality of first corpus groups according to the complexity, and dividing the target corpus into a plurality of second corpus groups according to the sensitivity, wherein the number of the first corpus groups is the same as the number of the second corpus groups;

[0018] Calling the historical virus database and the result storage library, respectively calculating the proportion of each of the first corpus group and the second corpus group that is similar to the virus information and malicious code in the historical virus database, so as to obtain a first similarity coefficient and a second similarity coefficient, respectively calculating the proportion of each of the first corpus group and the second corpus group that has the same corpus as that in the result storage library, so as to obtain a first hidden danger coefficient and a second hidden danger coefficient, respectively, the historical virus database is used to store the virus information and malicious code information found in the previous corpus, and the result storage library is used to store the corpus information found to contain viruses or malicious codes in the previous inspection;

[0019] A first danger value of the first corpus group is calculated according to the first similarity coefficient and the first hidden danger coefficient, a second danger value of the second corpus group is calculated according to the first similarity coefficient and the second hidden danger coefficient, the first corpus group and the corpus group are sorted according to the first danger value and the second danger value, respectively, and different degrees of safety checks are performed according to the sorting results.

[0020] In one embodiment of the present invention, the first corpus group and the second corpus group are sorted according to the first risk value and the second risk value respectively, and different degrees of security checks are performed according to the sorting results, including:

[0021] The first corpus group is sorted according to the size of the first danger value to obtain a first sorted corpus set, and the second corpus group is sorted according to the size of the second danger value to obtain a second sorted corpus set, and the first sorted corpus set and the second sorted corpus set are sorted in the same manner, that is, from large to small or from small to large;

[0022] Matching the first corpus group of the first sorted corpus set and the second corpus group of the second sorted corpus set one by one in order to create combinations to form a plurality of corpus combinations;

[0023] Obtaining overlapping corpora of the first corpus group and the second corpus group in each of the corpus combinations, and calculating the sum of the first risk value and the second risk value in each of the overlapping corpora to obtain a target risk value;

[0024] The overlapping corpus corresponding to the corpus combination corresponding to the target risk value being greater than or equal to the safety threshold is used as the first category of corpus, the overlapping corpus corresponding to the corpus combination corresponding to the target risk value being less than the safety threshold is used as the second category of corpus, and the corpus in the target corpus excluding the first category of corpus and the second category of corpus is used as the third category of corpus;

[0025] A first check is performed on the first category of corpus, a second check is performed on the second category of corpus, and a third check is performed on the third category of corpus, wherein the priorities and inspection scopes of the first check, the second check, and the third check are decreased in sequence.

[0026] In one embodiment of the present invention, the step of selecting a corresponding delivery method according to user needs and delivering the target corpus to the user includes:

[0027] Obtaining the data security level of the target corpus, and selecting a corresponding delivery channel according to the data security level;

[0028] After establishing a connection with the user, a key serial number in a key library is obtained, an encryption key selected by the user is determined according to the key serial number, the target corpus is encrypted according to the encryption key to obtain an encrypted corpus, the key serial number is configured in the encrypted corpus to obtain a mixed encrypted corpus, and the mixed encrypted corpus is sent to the user;

[0029] After it is determined that the user has received the mixed encrypted corpus, the delivery is completed.

[0030] In one embodiment of the present invention, the method further includes:

[0031] In the process of testing and applying the target corpus, obtaining the privacy information in the target corpus;

[0032] The private information is classified to obtain category information, and access rights are configured for the classified private information according to the category information.

[0033] In one embodiment of the present invention, the real-time monitoring and recording of the usage of the target corpus includes:

[0034] Obtaining usage of each of the target corpora, including usage frequency, usage time, and usage objects;

[0035] Performing a value assessment on the target corpus according to the usage frequency, the usage time, and the user;

[0036] Setting corresponding metering and billing standards for the target corpus according to the evaluation results;

[0037] A corpus usage report is generated regularly and sent to the user.

[0038] In one embodiment of the present invention, the method further includes:

[0039] Optimizing and adjusting the target corpus according to the evaluation results, and generating an operation report accordingly;

[0040] A corpus formula library is established according to the usage of the target corpus to record the corpus formulas in different application scenarios.

[0041] In one embodiment of the present invention, the value assessment of the target corpus according to the usage frequency, the usage time and the user includes:

[0042] Retrieving past usage information of past corpora, and calculating weight information of the past corpora according to the past usage information;

[0043] Performing a value assessment on the target corpus according to the weight information to obtain a corresponding assessment result;

[0044] The optimizing and adjusting the target corpus according to the evaluation result includes:

[0045] When the evaluation result is greater than the maximum value of the standard preset interval, increasing the proportion of the corpus corresponding to the evaluation result in the target corpus;

[0046] When the evaluation result is within the standard preset interval, keeping the current target corpus;

[0047] When the evaluation result is less than the minimum value of the standard preset interval, the proportion of the corpus corresponding to the evaluation result in the target corpus is reduced.

[0048] The present invention also provides a management system for corpus delivery and application rules, including:

[0049] A delivery preparation module is used to obtain the user's delivery requirements, determine the target corpus according to the delivery requirements, and prepare the target corpus for delivery;

[0050] A delivery selection module is used to select a corresponding delivery method according to user needs after completing the delivery preparation, and deliver the target corpus to the user;

[0051] An import module is used to obtain application requirements of the target corpus after delivery is completed, deploy an environment according to the application requirements and import the delivered target corpus, and test and apply the target corpus based on the deployed environment;

[0052] The monitoring module is used to monitor and record the usage of the target corpus in real time.

[0053] As described above, the method and system for managing corpus delivery and application rules of the present invention have the following beneficial effects:

[0054] By obtaining the user's needs, the corresponding target document is determined, and the target corpus is prepared for delivery. While eliminating security risks, the efficiency of corpus security checks is improved, ensuring the security and privacy of users' use of the corpus. At the same time, it is convenient for users to choose different delivery methods according to different needs, which is more flexible to use. During use, the use of the target corpus is recorded in real time and dynamically adjusted, which is conducive to fully meeting the usage needs of different users, realizing effective management of the corpus, and improving the effect of users using large models of corpus training data. BRIEF DESCRIPTION OF THE DRAWINGS

[0055] Figure 1 Shown is a flow chart of a method for managing corpus delivery and application rules in one embodiment of the present invention.

[0056] Figure 2 Shown is a structural block diagram of a management system for corpus delivery and application rules in one embodiment of the present invention.

[0057] Figure 3 Shown is a structural block diagram of a terminal used by the method for managing corpus delivery and application rules of the present invention. DETAILED DESCRIPTION

[0058] The following describes the embodiments of the present invention by specific examples, and those skilled in the art can easily understand other advantages and effects of the present invention from the contents disclosed in this specification. The present invention can also be implemented or applied through other different specific embodiments, and the details in this specification can also be modified or changed in various ways based on different viewpoints and applications without departing from the spirit of the present invention. It should be noted that the following embodiments and features in the embodiments can be combined with each other without conflict.

[0059] It should be noted that the illustrations provided in the following embodiments are only schematic illustrations of the basic concept of the present invention, and thus the drawings only show components related to the present invention rather than being drawn according to the number, shape and size of components in actual implementation. In actual implementation, the type, quantity and proportion of each component may be changed arbitrarily, and the component layout may also be more complicated.

[0060] The corpus delivery and application rule management method and system of the present invention determine the corresponding target document after obtaining the user's needs, and prepare the target corpus for delivery, thereby eliminating the potential security risks therein and improving the efficiency of corpus security checks, ensuring the security and privacy of users' use of corpora, and facilitating users to choose different delivery methods according to different needs, making the use more flexible. During use, the use of the target corpus is recorded in real time and dynamically adjusted, which is conducive to fully meeting the use needs of different users, realizing effective management of the corpus, and improving the effect of users using large models of corpus training data.

[0061] like Figure 1 As shown, in one embodiment, the present invention provides a method for managing corpus delivery and application rules, comprising the following steps:

[0062] S100: Obtain the user's delivery requirements, determine the target corpus according to the delivery requirements, and prepare the target corpus for delivery.

[0063] In some embodiments, the delivery requirements include corpus type, corpus quantity, corpus format, delivery time, and application scenario, and determining the target corpus according to the delivery requirements and preparing the target corpus for delivery include:

[0064] Selecting the target corpus according to the corpus type, the corpus format and the application scenario, and generating a compilation delivery document for the target corpus, the delivery document including a corpus list, metadata description, delivery time and usage guide;

[0065] Perform virus detection on the target corpus to remove potential safety hazards in the target corpus;

[0066] Performing an integrity check on the target corpus after the check.

[0067] In this embodiment, the corresponding target corpus is selected according to different user needs, and the corresponding compilation and delivery document is generated according to the target corpus. In order to ensure that the user can accurately understand and use the target corpus, the generated compilation and delivery document includes a corpus list, metadata description and user guide. The corpus list is used to describe the corpus type contained in the target corpus, and the metadata description is used to describe the attribute information of each corpus in the target corpus, including storage information and introduction information, which is used to support storage location, historical data, resource search, file record and other functions. The delivery time is used to describe the time when the target corpus is delivered to the user, and the user guide is used to describe the usage requirements and scenarios of each target corpus, so that the user can use the target corpus reasonably.

[0068] After generating the compilation delivery document of the target corpus, the target corpus is checked for viruses to eliminate security risks, and the integrity of the target corpus after the check is checked is checked to avoid incompleteness in the target corpus after the check, which may affect the subsequent use of the target corpus.

[0069] In some embodiments, the virus detection and elimination of the target corpus to remove potential safety hazards in the target corpus includes:

[0070] Acquire feature information and path information of the target corpus, and calculate the complexity of the feature information and the sensitivity of the path information;

[0071] Dividing the target corpus into a plurality of first corpus groups according to the complexity, and dividing the target corpus into a plurality of second corpus groups according to the sensitivity, wherein the number of the first corpus groups is the same as the number of the second corpus groups;

[0072] Calling the historical virus database and the result storage library, respectively calculating the proportion of each of the first corpus group and the second corpus group that is similar to the virus information and malicious code in the historical virus database, so as to obtain a first similarity coefficient and a second similarity coefficient, respectively calculating the proportion of each of the first corpus group and the second corpus group that has the same corpus as that in the result storage library, so as to obtain a first hidden danger coefficient and a second hidden danger coefficient, respectively, the historical virus database is used to store the virus information and malicious code information found in the previous corpus, and the result storage library is used to store the corpus information found to contain viruses or malicious codes in the previous inspection;

[0073] A first danger value of the first corpus group is calculated according to the first similarity coefficient and the first hidden danger coefficient, a second danger value of the second corpus group is calculated according to the first similarity coefficient and the second hidden danger coefficient, the first corpus group and the corpus group are sorted according to the first danger value and the second danger value, respectively, and different degrees of safety checks are performed according to the sorting results.

[0074] In this embodiment, in order to further quickly detect and eliminate the target corpus, the feature information of the target corpus is first obtained to facilitate the subsequent calculation of the complexity of the feature information; at the same time, the path information of each target corpus is calculated, and the path information is used to record the working path and storage path of each corpus, so as to determine the sensitivity corresponding to the corresponding corpus according to the path information.

[0075] Specifically, when calculating the complexity of the feature information, first obtain the descriptive features and work behavior features of each corpus in the target corpus, and add up the number of bytes of the information of the descriptive features and work behavior features to obtain the corresponding feature information value, then calculate the cumulative sum of the feature information values ​​of each corpus in the target corpus and average them to obtain the average information value, and by calculating the ratio of the feature information value of each corpus in the target corpus to the average information value, the complexity corresponding to each corpus can be obtained. That is, the more descriptive features and work behavior features corresponding to a corpus, the higher the complexity of the feature information of the target corpus is determined to be.

[0076] Similarly, the path information of the target corpus includes a storage path and a working path. After determining the path information of the corpus, the sensitive path information pre-set by the user is retrieved, and the sensitivity of each corpus is calculated by comparing the overlap ratio between the storage path and the working path of the corpus and the sensitive path information.

[0077] After obtaining the complexity of the feature information and the sensitivity of the path information of each corpus, the target corpus is divided into multiple first corpus groups according to the complexity according to a preset number, and the target corpus is divided into multiple second corpus groups according to the sensitivity, and the number of the first corpus groups and the second corpus groups is the same.

[0078] Then, the historical virus database and the result storage library are called to calculate the proportion of each of the first corpus group and the second corpus group that has corpus similar to the virus information and malicious code in the historical virus database, so as to obtain the first similarity coefficient and the second similarity coefficient respectively, and the proportion of each of the first corpus group and the second corpus group that has the same corpus as the result storage library is calculated respectively, so as to obtain the first hidden danger coefficient and the second hidden danger coefficient respectively. The historical virus database is used to store the virus information and malicious code information found in the previous corpus, and the result storage library is used to store the corpus information that has been checked in the past and has viruses or malicious codes. In this way, the similarity coefficient and hidden danger coefficient of each first corpus group and the second corpus group are obtained, so as to facilitate the subsequent sorting of the first corpus group and the second corpus group of different groups, and improve the subsequent detection and killing efficiency of the corpus.

[0079] Then, a first danger value of the first corpus group is calculated according to the first similarity coefficient and the first hidden danger coefficient, and a second danger value of the second corpus group is calculated according to the first similarity coefficient and the second hidden danger coefficient. The first corpus group and the corpus group are sorted according to the first danger value and the second danger value, respectively, and different degrees of safety checks are performed according to the sorting results.

[0080] In this embodiment, the calculation process of the first risk value P satisfies the following formula:

[0081] P = 1 / (A × B)

[0082] Among them, A is the first similarity coefficient and B is the first hidden danger coefficient.

[0083] The calculation process of the second risk value Q satisfies the following formula:

[0084] Q = 1 / (α×β)

[0085] Among them, α is the second similarity coefficient, and β is the second hidden danger coefficient.

[0086] After obtaining the first risk value of the first corpus group and the second risk value of the second corpus group, the corpus can be sorted according to the first risk value and the second risk value, and security checks can be performed on different corpora according to the sorting results.

[0087] In some other embodiments, the first corpus group and the second corpus group are sorted according to the first risk value and the second risk value respectively, and different degrees of security checks are performed according to the sorting results, including:

[0088] The first corpus group is sorted according to the size of the first danger value to obtain a first sorted corpus set, and the second corpus group is sorted according to the size of the second danger value to obtain a second sorted corpus set, and the first sorted corpus set and the second sorted corpus set are sorted in the same manner, that is, from large to small or from small to large;

[0089] Matching the first corpus group of the first sorted corpus set and the second corpus group of the second sorted corpus set one by one in order to create combinations to form a plurality of corpus combinations;

[0090] Obtaining overlapping corpora of the first corpus group and the second corpus group in each of the corpus combinations, and calculating the sum of the first risk value and the second risk value in each of the overlapping corpora to obtain a target risk value;

[0091] The overlapping corpus corresponding to the corpus combination corresponding to the target risk value being greater than or equal to the safety threshold is used as the first category of corpus, the overlapping corpus corresponding to the corpus combination corresponding to the target risk value being less than the safety threshold is used as the second category of corpus, and the corpus in the target corpus excluding the first category of corpus and the second category of corpus is used as the third category of corpus;

[0092] A first check is performed on the first category of corpus, a second check is performed on the second category of corpus, and a third check is performed on the third category of corpus, wherein the priorities and inspection scopes of the first check, the second check, and the third check are decreased in sequence.

[0093] In this embodiment, the first corpus group is first sorted according to the size of the first danger value to obtain a first sorted corpus set, and the second corpus group is sorted according to the size of the second danger value to obtain a second sorted corpus set, and the sorting method of the first sorted corpus and the second sorted corpus is ensured to be the same, so as to facilitate the subsequent establishment of a combination according to the sorting results.

[0094] After the sorting is completed, the first sorted corpus and the second sorted corpus are combined together in the order of the first danger value and the second danger value from large to small or from small to large, that is, the first corpus with the largest first danger value in the first sorted corpus and the second corpus with the largest second danger value in the second sorted set are combined together, the first corpus with the second largest first danger value in the first sorted corpus and the second corpus with the second largest second danger value in the second sorted set are combined together,..., the first corpus with the smallest first danger value in the first sorted corpus and the second corpus with the smallest second danger value in the second sorted set are combined together.

[0095] Then, the overlapping corpus of the first corpus group and the second corpus group in each of the corpus combinations is obtained, and the sum of the first danger value and the second danger value in each of the overlapping corpus is calculated to obtain the target danger value of the overlapping corpus; the overlapping corpus corresponding to the corpus combination corresponding to the target danger value is greater than or equal to the safety threshold is used as the first category of corpus, the overlapping corpus corresponding to the corpus combination corresponding to the target danger value is less than the safety threshold is used as the second category of corpus, and the corpus in the target corpus excluding the first category of corpus and the second category of corpus is used as the third category of corpus. The first category of corpus is first checked, the second category of corpus is second checked, and the third category of corpus is third checked, wherein the priority and inspection scope of the first check, the second check, and the third check are successively reduced, thereby realizing differentiated security checks on the entire target corpus, while ensuring the checking effect, improving the efficiency of security checks on the target corpus, and facilitating the management of the corpus.

[0096] Exemplarily, a global check is performed on the first category of corpora, a local check is performed on the second category of corpora, and a keyword check is performed on the third category of corpora, thereby achieving differentiated security checks on different target corpora and improving the inspection efficiency.

[0097] Furthermore, when performing an integrity check on the target corpus, when it is detected that there is incomplete corpus in the target corpus, the position of the incomplete corpus in the target corpus is first determined, and marked according to the corresponding position, so as to facilitate the subsequent timely supplementation of the incomplete or missing corpus.

[0098] S200: After completing the delivery preparation, select a corresponding delivery method according to the user's needs, and deliver the target corpus to the user.

[0099] In some embodiments, selecting a corresponding delivery method according to user needs and delivering the target corpus to the user includes:

[0100] Obtaining the data security level of the target corpus, and selecting a corresponding delivery channel according to the data security level;

[0101] After establishing a connection with the user, a key serial number in a key library is obtained, an encryption key selected by the user is determined according to the key serial number, the target corpus is encrypted according to the encryption key to obtain an encrypted corpus, the key serial number is configured in the encrypted corpus to obtain a mixed encrypted corpus, and the mixed encrypted corpus is sent to the user;

[0102] After it is determined that the user has received the mixed encrypted corpus, the delivery is completed.

[0103] In this embodiment, in order to select the delivery method corresponding to the user, the data security level of the target corpus is first obtained, and the corresponding delivery channel is selected according to the security level, such as email, FTP, cloud storage and API interface, which can be selected according to the security level requirements. At the same time, the key serial number in the pre-established key library is obtained, and after establishing a connection with the user, the encryption key selected by the user is determined according to the key requirements, and the target corpus is encrypted according to the encryption key to obtain the encrypted corpus. At the same time, the key serial number is configured in the encrypted corpus to mix and obtain the mixed encrypted corpus, which ensures the security of the corpus transmission process. At the same time, because the key library is stored by the user, even if the encrypted corpus is obtained during the corpus transmission process, it is not easy to crack the corpus information therein, which ensures the security of the entire corpus.

[0104] After the user receives the mixed encrypted corpus, he / she will select the corresponding key to decrypt it according to the key requirements to obtain the final target corpus, thus completing the corpus delivery process. After the delivery is completed, the data reception status is further confirmed with the user to ensure that the data is complete and correct. At the same time, the delivery process is recorded in a record sheet, including delivery time, delivery method, receipt confirmation and other information, to ensure the traceability of the delivery process.

[0105] S300: After the delivery is completed, the application requirements of the target corpus are obtained, the environment is deployed according to the application requirements, and the delivered target corpus is imported, and the target corpus is tested and applied based on the deployed environment.

[0106] When deploying the specific environment, configure the corresponding computing and storage environment, such as servers, databases, and cloud platforms, according to the application requirements of the target corpus to ensure high availability and high performance of the environment and meet the processing requirements of the corpus in the application. Import the delivered corpus into the application environment to ensure the accuracy and completeness of the data import process, and then integrate the corpus into the test application system to ensure that the corpus can be effectively called and used. The test application system is mainly used to call and use the corpus to test the corpus to ensure the correctness and effectiveness of the corpus in the system.

[0107] In some further embodiments, the method further comprises:

[0108] In the process of testing and applying the target corpus, obtaining the privacy information in the target corpus;

[0109] The private information is classified to obtain category information, and access rights are configured for the classified private information according to the category information.

[0110] After obtaining the private information of the target corpus, the private information is classified according to its privacy level, so that different access rights can be configured for the classified private information according to the classification information obtained in the subsequent classification. The rules for classifying private information are pre-set by the user and can be selected according to actual needs. This solution does not specifically limit this.

[0111] S400: Monitor and record the usage of the target corpus in real time.

[0112] In some embodiments, the real-time monitoring and recording of the usage of the target corpus includes:

[0113] Obtaining usage of each of the target corpora, including usage frequency, usage time, and usage objects;

[0114] Performing a value assessment on the target corpus according to the usage frequency, the usage time, and the user;

[0115] Setting corresponding metering and billing standards for the target corpus according to the evaluation results;

[0116] A corpus usage report is generated regularly and sent to the user.

[0117] In this embodiment, in the process of monitoring the target corpus, the frequency of use, time of use and user of each target corpus are obtained in real time, so as to facilitate the value assessment of the target corpus according to the frequency of use, time of use and user of the target corpus, and set the corresponding metering and billing standards for the target corpus according to the assessment results, so as to ensure the rationality and fairness of resource use and avoid resource runs. After the metering and billing standards are formulated, the user's actual usage is charged to ensure the transparency and rationality of the charges. At the same time, the user's corpus usage report is generated regularly and sent to the user, so that the user can understand his own corpus usage.

[0118] In some embodiments, the performing value assessment on the target corpus according to the usage frequency, the usage time and the user includes:

[0119] Retrieving past usage information of past corpora, and calculating weight information of the past corpora according to the past usage information;

[0120] The target corpus is evaluated for value according to the weight information to obtain a corresponding evaluation result.

[0121] In this embodiment, by calling the past usage information of the past corpus, the weight information of the past corpus is calculated using the past usage information, including the weight information of the usage frequency, usage time and usage object, and then the usage frequency, usage time and usage object of the target corpus are weighted and summed according to the weight information to obtain the evaluation result of each target corpus, so as to facilitate the subsequent adjustment of different target corpora according to the evaluation results.

[0122] It should be noted that the process of calculating weight information for the frequency of use, usage time and usage objects of the corpus in the present application adopts the hierarchical analysis method in the prior art, and other weight calculation methods in the prior art may also be adopted. This scheme does not make any special limitations on this and will not be repeated here.

[0123] The optimizing and adjusting the target corpus according to the evaluation result includes:

[0124] When the evaluation result is greater than the maximum value of the standard preset interval, increasing the proportion of the corpus corresponding to the evaluation result in the target corpus;

[0125] When the evaluation result is within the standard preset interval, keeping the current target corpus;

[0126] When the evaluation result is less than the minimum value of the standard preset interval, the proportion of the corpus corresponding to the evaluation result in the target corpus is reduced.

[0127] The standard preset interval corresponds to an interval from a preset minimum value to a preset maximum value.

[0128] In some embodiments, the method further comprises:

[0129] Optimizing and adjusting the target corpus according to the evaluation results, and generating an operation report accordingly;

[0130] A corpus formula library is established according to the usage of the target corpus to record the corpus formulas in different application scenarios.

[0131] By establishing and formulating corpus formulas, the proportions and combinations of different types of corpora can be clarified. This can not only ensure the scientificity and rationality of the formulas and meet the needs of different application scenarios. In addition, by establishing a corpus formula library, the corpus formulas and usage effects of different application scenarios can be recorded. Regularly updating and optimizing the corpus formula can ensure the timeliness and applicability of the formulas and better meet the needs of users.

[0132] It should be noted that the protection scope of the corpus delivery and application rule management method described in the present invention is not limited to the execution order of the steps listed in this embodiment, and all solutions implemented by adding, reducing, or replacing steps in the prior art based on the principles of the present invention are included in the protection scope of the present invention.

[0133] The present invention also discloses a management system for corpus delivery and application rules, referring to Figure 2 ,include:

[0134] The delivery preparation module 201 is used to obtain the user's delivery requirements, determine the target corpus according to the delivery requirements, and prepare the target corpus for delivery;

[0135] A delivery selection module 202 is used to select a corresponding delivery method according to user needs after completing the delivery preparation, and deliver the target corpus to the user;

[0136] The import module 203 is used to obtain the application requirements of the target corpus after the delivery is completed, deploy the environment according to the application requirements and import the delivered target corpus, and test the target corpus based on the deployed environment;

[0137] The monitoring module 204 is used to monitor and record the usage of the target corpus in real time.

[0138] Since each module of the above-mentioned management system for corpus delivery and application rules corresponds one-to-one to the above-mentioned management method process for corpus delivery and application rules, they will not be described in detail here.

[0139] It should be noted that it should be understood that the division of the various modules of the above device is only a division of logical functions. In actual implementation, they can be fully or partially integrated into one physical entity, or they can be physically separated. And these modules can all be implemented in the form of software called by processing elements; they can also be all implemented in the form of hardware; some modules can also be implemented in the form of software called by processing elements, and some modules can be implemented in the form of hardware. For example, the x module can be a separately established processing element, or it can be integrated in a chip of the above device. In addition, it can also be stored in the memory of the above device in the form of program code, and called and executed by a processing element of the above device. The implementation of other modules is similar. In addition, these modules can be fully or partially integrated together, or they can be implemented independently. The processing element described here can be an integrated circuit with signal processing capabilities. In the implementation process, each step of the above method or each module above can be completed by an integrated logic circuit of hardware in the processor element or instructions in the form of software.

[0140] For example, the above modules may be one or more integrated circuits configured to implement the above methods, such as one or more application specific integrated circuits (ASICs), or one or more digital signal processors (DSPs), or one or more field programmable gate arrays (FPGAs). For another example, when a module is implemented in the form of a processing element scheduling program code, the processing element may be a general-purpose processor, such as a central processing unit (CPU) or other processors that can call program code. For another example, these modules may be integrated together and implemented in the form of a system-on-a-chip (SOC).

[0141] The storage medium of the present invention stores a computer program, which, when executed by a processor, implements the above-mentioned method for managing corpus delivery and application rules. The storage medium includes: ROM, RAM, disk, USB flash drive, memory card or optical disk, etc., which can store program codes.

[0142] like Figure 3 As shown, the terminal of the present invention includes a processor 31 and a memory 32.

[0143] The memory 32 is used to store computer programs. Preferably, the memory 32 includes: ROM, RAM, disk, USB flash drive, memory card or optical disk, etc., various media that can store program codes.

[0144] The processor 31 is connected to the memory 32 and is used to execute the computer program stored in the memory 32 so that the terminal executes the above-mentioned corpus delivery and application rule management method.

[0145] Preferably, the processor 31 can be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it can also be a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components.

[0146] It should be noted that the management system of corpus delivery and application rules of the present invention can implement the management method of corpus delivery and application rules of the present invention, but the implementation device of the management method of corpus delivery and application rules of the present invention includes but is not limited to the structure of the management system of corpus delivery and application rules listed in this embodiment. All structural deformations and replacements of the prior art made according to the principles of the present invention are included in the protection scope of the present invention.

[0147] In summary, the management method and system of corpus delivery and application rules of the present invention, by obtaining the needs of users, determines the corresponding target document, and prepares the target corpus for delivery, while eliminating the potential safety hazards, improves the efficiency of corpus security inspection, ensures the security and privacy of users using corpus, and facilitates users to choose different delivery methods according to different needs, making it more flexible to use. In the process of use, the use of target corpus is recorded in real time and adjusted dynamically, which is conducive to fully meeting the use needs of different users, realizing effective management of corpus, and improving the effect of users using corpus training data large model. Therefore, the present invention effectively overcomes various shortcomings in the prior art and has high industrial utilization value.

[0148] The above embodiments are merely illustrative of the principles and effects of the present invention, and are not intended to limit the present invention. Anyone familiar with the art may modify or alter the above embodiments without departing from the spirit and scope of the present invention. Therefore, all equivalent modifications or alterations made by a person of ordinary skill in the art without departing from the spirit and technical concept disclosed by the present invention shall still be covered by the claims of the present invention.

Claims

1. A method for managing corpus delivery and application rules, characterized in that: include: Obtaining the user's delivery requirements, determining the target corpus according to the delivery requirements, and preparing the target corpus for delivery; After the delivery preparation is completed, a corresponding delivery method is selected according to the user's needs, and the target corpus is delivered to the user; After the delivery is completed, the application requirements of the target corpus are obtained, the environment is deployed according to the application requirements, and the delivered target corpus is imported, and the target corpus is tested and applied based on the deployed environment; Monitor and record the usage of the target corpus in real time; The delivery requirements include corpus type, corpus quantity, corpus format, delivery time and application scenario. Determining the target corpus according to the delivery requirements and preparing the target corpus for delivery include: Selecting the target corpus according to the corpus type, the corpus format and the application scenario, and generating a compilation delivery document for the target corpus, the delivery document including a corpus list, metadata description, delivery time and usage guide; Perform virus detection on the target corpus to remove potential safety hazards in the target corpus; Performing integrity check on the target corpus after the check; The virus detection and elimination of the target corpus to remove the potential safety hazard in the target corpus includes: Acquire feature information and path information of the target corpus, and calculate the complexity of the feature information and the sensitivity of the path information; Dividing the target corpus into a plurality of first corpus groups according to the complexity, and dividing the target corpus into a plurality of second corpus groups according to the sensitivity, wherein the number of the first corpus groups is the same as the number of the second corpus groups; Calling the historical virus database and the result storage library, respectively calculating the proportion of each of the first corpus group and the second corpus group that is similar to the virus information and malicious code in the historical virus database, so as to obtain a first similarity coefficient and a second similarity coefficient, respectively calculating the proportion of each of the first corpus group and the second corpus group that has the same corpus as that in the result storage library, so as to obtain a first hidden danger coefficient and a second hidden danger coefficient, respectively, the historical virus database is used to store the virus information and malicious code information found in the previous corpus, and the result storage library is used to store the corpus information that has been checked in the past to contain viruses or malicious codes; A first danger value of the first corpus group is calculated according to the first similarity coefficient and the first hidden danger coefficient, a second danger value of the second corpus group is calculated according to the first similarity coefficient and the second hidden danger coefficient, the first corpus group and the second corpus group are sorted according to the first danger value and the second danger value, respectively, and different degrees of safety checks are performed according to the sorting results.

2. The method for managing corpus delivery and application rules according to claim 1, characterized in that: The first corpus group and the second corpus group are sorted according to the first risk value and the second risk value respectively, and different degrees of security checks are performed according to the sorting results, including: The first corpus group is sorted according to the size of the first danger value to obtain a first sorted corpus set, and the second corpus group is sorted according to the size of the second danger value to obtain a second sorted corpus set, and the first sorted corpus set and the second sorted corpus set are sorted in the same manner, that is, from large to small or from small to large; Matching the first corpus group of the first sorted corpus set and the second corpus group of the second sorted corpus set one by one in order to create combinations to form a plurality of corpus combinations; Obtaining overlapping corpora of the first corpus group and the second corpus group in each of the corpus combinations, and calculating the sum of the first risk value and the second risk value in each of the overlapping corpora to obtain a target risk value; The overlapping corpus corresponding to the corpus combination corresponding to the target risk value being greater than or equal to the safety threshold is used as the first category of corpus, the overlapping corpus corresponding to the corpus combination corresponding to the target risk value being less than the safety threshold is used as the second category of corpus, and the corpus in the target corpus excluding the first category of corpus and the second category of corpus is used as the third category of corpus; A first check is performed on the first category of corpus, a second check is performed on the second category of corpus, and a third check is performed on the third category of corpus, wherein the priorities and inspection scopes of the first check, the second check, and the third check are decreased in sequence.

3. The method for managing corpus delivery and application rules according to claim 1, characterized in that: The selecting a corresponding delivery method according to user needs and delivering the target corpus to the user includes: Obtaining the data security level of the target corpus, and selecting a corresponding delivery channel according to the data security level; After establishing a connection with the user, a key serial number in a key library is obtained, an encryption key selected by the user is determined according to the key serial number, the target corpus is encrypted according to the encryption key to obtain an encrypted corpus, the key serial number is configured in the encrypted corpus to obtain a mixed encrypted corpus, and the mixed encrypted corpus is sent to the user; After it is determined that the user has received the mixed encrypted corpus, the delivery is completed.

4. The method for managing corpus delivery and application rules according to claim 1, characterized in that: The method further comprises: In the process of testing and applying the target corpus, obtaining the privacy information in the target corpus; The private information is classified to obtain category information, and access rights are configured for the classified private information according to the category information.

5. The method for managing corpus delivery and application rules according to claim 1, characterized in that: The real-time monitoring and recording of the usage of the target corpus includes: Obtaining usage of each of the target corpora, including usage frequency, usage time, and usage objects; Performing a value assessment on the target corpus according to the usage frequency, the usage time, and the user; Setting corresponding metering and billing standards for the target corpus according to the evaluation results; A corpus usage report is generated regularly and sent to the user.

6. The method for managing corpus delivery and application rules according to claim 5, characterized in that: The method further comprises: Optimizing and adjusting the target corpus according to the evaluation results, and generating an operation report accordingly; A corpus formula library is established according to the usage of the target corpus to record the corpus formulas in different application scenarios.

7. The method for managing corpus delivery and application rules according to claim 6, characterized in that: The performing value assessment on the target corpus according to the usage frequency, the usage time and the user includes: Retrieving past usage information of past corpora, and calculating weight information of the past corpora according to the past usage information; Performing a value assessment on the target corpus according to the weight information to obtain a corresponding assessment result; The optimizing and adjusting the target corpus according to the evaluation result includes: When the evaluation result is greater than the maximum value of the standard preset interval, increasing the proportion of the corpus corresponding to the evaluation result in the target corpus; When the evaluation result is within the standard preset interval, keeping the current target corpus; When the evaluation result is less than the minimum value of the standard preset interval, the proportion of the corpus corresponding to the evaluation result in the target corpus is reduced.

8. A management system for corpus delivery and application rules, characterized in that: include: A delivery preparation module is used to obtain the user's delivery requirements, determine the target corpus according to the delivery requirements, and prepare the target corpus for delivery; A delivery selection module is used to select a corresponding delivery method according to user needs after completing the delivery preparation, and deliver the target corpus to the user; An import module is used to obtain application requirements of the target corpus after delivery is completed, deploy an environment according to the application requirements and import the delivered target corpus, and test and apply the target corpus based on the deployed environment; A monitoring module, used to monitor and record the usage of the target corpus in real time; The delivery requirements include corpus type, corpus quantity, corpus format, delivery time and application scenario. The delivery preparation module is further used to select the target corpus according to the corpus type, the corpus format and the application scenario, and generate a compilation delivery document for the target corpus. The delivery document includes a corpus list, metadata description, delivery time and usage guide. Perform virus detection on the target corpus to remove potential safety hazards in the target corpus; Performing integrity check on the target corpus after the check; The virus detection and elimination of the target corpus to remove the potential safety hazard in the target corpus includes: Acquire feature information and path information of the target corpus, and calculate the complexity of the feature information and the sensitivity of the path information; Dividing the target corpus into a plurality of first corpus groups according to the complexity, and dividing the target corpus into a plurality of second corpus groups according to the sensitivity, wherein the number of the first corpus groups is the same as the number of the second corpus groups; Calling the historical virus database and the result storage library, respectively calculating the proportion of each of the first corpus group and the second corpus group that is similar to the virus information and malicious code in the historical virus database, so as to obtain a first similarity coefficient and a second similarity coefficient, respectively calculating the proportion of each of the first corpus group and the second corpus group that has the same corpus as that in the result storage library, so as to obtain a first hidden danger coefficient and a second hidden danger coefficient, respectively, the historical virus database is used to store the virus information and malicious code information found in the previous corpus, and the result storage library is used to store the corpus information that has been checked in the past to contain viruses or malicious codes; A first danger value of the first corpus group is calculated according to the first similarity coefficient and the first hidden danger coefficient, a second danger value of the second corpus group is calculated according to the first similarity coefficient and the second hidden danger coefficient, the first corpus group and the second corpus group are sorted according to the first danger value and the second danger value, respectively, and different degrees of safety checks are performed according to the sorting results.

Images