Corpus directory management method and system for industrial large model

Abstract

The invention provides a corpus directory management method and system for an industrial large model, and the method comprises the steps: obtaining basic information of a target corpus, and carrying out the preprocessing of the target corpus according to the basic information, so as to obtain a standard corpus; performing scene analysis on the standard corpus to obtain corresponding label information; performing directory value calculation on the standard corpus according to the label information to obtain a standard directory value of the standard corpus; and generating a corresponding target catalogue according to the standard catalogue value, establishing mapping between the standard corpus and the target catalogue, and storing the standard corpus and the target catalogue. According to the method, the corpora are effectively managed, the use efficiency of the corpora can be improved, and the user experience is enhanced. According to the method, the storage and calling efficiency of the corpora is effectively improved, and the use efficiency of the corpora is improved.

Description

Technical Field

[0001] The present invention belongs to the technical field of big data, and in particular, relates to a corpus catalog management method and system for industrial big models. Background Art

[0002] Artificial intelligence big models refer to "large parameter" models trained using large-scale data and powerful computing power. These models are usually highly versatile and generalized, and can be applied to natural language processing, image recognition, speech recognition and other fields. They can be divided into large language models, large visual models, multimodal large models, and basic large models. Corpus is an important material used to train large artificial intelligence models. Corpus generally refers to examples and data sets used in linguistic research and natural language processing. It can be written text, oral records or other structured data, usually used to analyze language phenomena, support machine translation, speech recognition, automatic text summarization and other tasks. A common corpus is a set of text or speech data that has been collected, organized and annotated. These data are used in linguistic research to analyze language usage patterns, vocabulary changes and grammatical structures. In natural language processing, corpus is the basic data source for training and testing models, supporting machine translation, speech recognition, sentiment analysis and other functions.

[0003] Due to the huge amount of corpora, they are generally stored in a classified manner, and when storing, they are stored by establishing multi-level directories. When storing and searching, the corresponding directories must be determined layer by layer. Due to the current refined division of corpora, the directories of many corpora have too much content, resulting in low efficiency in storing and searching corpora, which is not conducive to the storage and call of corpora. Summary of the invention

[0004] In view of the above-mentioned shortcomings of the prior art, an object of the present invention is to provide a corpus directory management method and system for industrial large models, so as to solve the problem of low efficiency of corpus storage and call in the prior art.

[0005] To achieve the above-mentioned and other related purposes, the present invention provides a corpus catalog management method for industrial large models, comprising the following steps: Acquiring basic information of the target corpus, and preprocessing the target corpus according to the basic information to obtain standard corpus; Performing scenario analysis on the standard corpus to obtain corresponding label information; Calculating the catalog value of the standard corpus according to the tag information to obtain a standard catalog value of the standard corpus; A corresponding target directory is generated according to the standard directory value, and the standard corpus is mapped with the target directory and then stored.

[0006] In some embodiments, preprocessing the target corpus according to the basic information to obtain a standard corpus includes: Performing field scanning on the target corpus to remove invalid fields and noise fields in the target corpus to obtain a first corpus; Acquire a sensitive word library corresponding to the target corpus according to the basic information, compare the first corpus with the sensitive word library, and replace sensitive words in the first corpus with label words to obtain a corresponding standard corpus; The label words correspond to each of the sensitive words one by one.

[0007] In some embodiments, the basic information includes at least industry information, modality information and language information, and the performing scenario analysis on the standard corpus to obtain corresponding label information includes: Acquire the industry type of the target corpus according to the industry information, and generate a first label matrix corresponding to the standard corpus after converting the industry type into a preset coding format; Acquiring a modality type in the target corpus according to the modality information, and converting the modality type into a preset encoding format to generate a second label matrix corresponding to the standard corpus; Acquire the language type of the target corpus according to the language information, and generate a third label matrix corresponding to the standard corpus after converting the language type into a preset encoding format; Integrate the first label matrix, the second label matrix and the third label matrix in order to form a label matrix to obtain the label information; The rows and columns of the first label matrix, the second label matrix and the third label matrix are the same.

[0008] In some embodiments, calculating the catalog value of the standard corpus according to the tag information to obtain the standard catalog value of the standard corpus includes: Calculating the same-position average of the first label matrix, the second label matrix and the third label matrix to obtain an average label matrix; Acquire a corpus sequence of the standard corpus and generate a corpus index value of the corpus sequence; Respectively calculating the ratios between the eigenvalues ​​corresponding to the first label matrix, the second label matrix, the third label matrix, and the average label matrix and the corpus index value to obtain a first directory value, a second directory value, a third directory value, and an average directory value, respectively; combining the first catalog value, the second catalog value, the third catalog value, and the average catalog value to generate a catalog sequence; The directory index value of the directory sequence is calculated, the standard directory value of the standard corpus is determined according to the corpus index value and the directory index value, and a mapping relationship between the standard directory value and the standard corpus is established.

[0009] In some embodiments, generating a corresponding target directory according to the standard directory value, mapping the standard corpus with the target directory and storing the mappings thereafter, comprises: Obtaining a corresponding directory path in a directory library according to the standard directory value, and generating a target directory corresponding to the standard corpus according to the directory path; After mapping the standard corpus with the target directory, the standard corpus is stored in the target directory.

[0010] In some embodiments, acquiring a corresponding directory path according to the standard directory value, and generating a corresponding target directory according to the directory path, includes: Splitting the standard directory value into a corresponding number of multiple sub-directory values, and sequentially generating sub-paths from the multiple sub-directory values; The plurality of sub-paths are combined together to obtain the directory path, and a corresponding target directory is generated according to the directory path.

[0011] In some embodiments, combining the first directory value, the second directory value, the third directory value, and the average directory value to generate a directory sequence comprises: The directory sequence is generated by inserting separators at the beginning and the end of the first directory value, the second directory value, the third directory value and the average directory value, and arranging them in sequence.

[0012] In some embodiments, the standard directory value of the standard corpus is a sequence combination value of the corpus index value and the directory index value.

[0013] The present invention also provides a corpus catalog management system for industrial large models, comprising: A preprocessing module, used to obtain basic information of the target corpus, and preprocess the target corpus according to the basic information to obtain a standard corpus; An analysis module, used for performing scene analysis on the standard corpus to obtain corresponding label information; A calculation module, configured to calculate a catalog value of the standard corpus according to the tag information to obtain a standard catalog value of the standard corpus; The directory generation module is used to generate a corresponding target directory according to the standard directory value, and store the standard corpus after mapping with the target directory.

[0014] As described above, the corpus catalog management method and system for industrial large models described in the present invention have the following beneficial effects: The present invention obtains a standard corpus by preprocessing the target corpus, performs scene analysis on the standard corpus to obtain label information corresponding to the standard corpus, and then calculates a standard directory value corresponding to the target corpus according to the label information. Since the standard directory value is calculated according to the label information of the standard corpus and is unique, a corresponding standard directory can be generated according to the standard directory value. The target corpus can be directly stored in the standard directory, and the corresponding corpus can also be quickly queried according to the standard directory. There is no need to search in sequence according to the classification directory of the corpus, which effectively improves the storage and call efficiency of the corpus and can improve the use efficiency of the corpus. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] Figure 1 Shown is a flow chart of the corpus catalog management method for industrial large models of the present invention.

[0016] Figure 2 Shown is a structural block diagram of a corpus catalog management system for industrial large models according to the present invention. DETAILED DESCRIPTION

[0017] The following describes the embodiments of the present invention through specific embodiments, and those skilled in the art can easily understand other advantages and effects of the present invention from the contents disclosed in this specification. The present invention can also be implemented or applied through other different specific embodiments, and the details in this specification can also be modified or changed in various ways based on different viewpoints and applications without departing from the spirit of the present invention. It should be noted that the following embodiments and features in the embodiments can be combined with each other without conflict.

[0018] It should be noted that the illustrations provided in the following embodiments are only schematic illustrations of the basic concept of the present invention. The illustrations only show components related to the present invention rather than being drawn according to the number, shape and size of components in actual implementation. In actual implementation, the type, quantity and proportion of each component may be changed arbitrarily, and the component layout may also be more complicated.

[0019] The corpus catalog management method and system for industrial large models of the present invention obtain standard corpus by preprocessing the target corpus, and perform scene analysis on the standard corpus to obtain label information corresponding to the standard corpus, and then calculate the standard catalog value corresponding to the target corpus according to the label information. Since the standard catalog value is calculated according to the label information of the standard corpus and is unique, the corresponding standard catalog can be generated according to the standard catalog value. The target corpus can be directly stored in the standard catalog, and the corresponding corpus can be quickly queried according to the standard catalog. There is no need to search in sequence according to the classification catalog of the corpus, which effectively improves the storage and call efficiency of the corpus and can improve the use efficiency of the corpus.

[0020] The storage medium of the present invention stores a computer program, and when the computer program is executed by a processor, the following corpus management method is implemented. The storage medium includes: a read-only memory (ROM), a random access memory (RAM), a disk, a USB flash drive, a memory card, or an optical disk, etc., which can store program codes.

[0021] Any combination of one or more storage media may be used. The storage medium may be a computer-readable signal medium or a computer-readable storage medium. The computer-readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination thereof. More specific examples (a non-exhaustive list) of computer-readable storage media include: an electrical connection with one or more wires, a portable computer disk, a hard disk, a RAM, a ROM, an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination thereof. In this document, a computer-readable storage medium may be any tangible medium containing or storing a program that may be used by or in conjunction with an instruction execution system, device, or device.

[0022] Computer-readable signal media may include a data signal propagated in baseband or as part of a carrier wave, which carries a computer-readable program code. Such propagated data signals may take a variety of forms, including, but not limited to, electromagnetic signals, optical signals, or any suitable combination of the above. Computer-readable signal media may also be any computer-readable medium other than a computer-readable storage medium, which may send, propagate, or transmit a program for use by or in conjunction with an instruction execution system, apparatus, or device.

[0023] The program code embodied on the computer readable medium may be transmitted using any appropriate medium, including but not limited to wireless, wireline, optical fiber cable, RF, etc., or any suitable combination of the foregoing.

[0024] Computer program code for performing the operation of the present invention may be written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Java, Smalltalk, C++, etc., and conventional procedural programming languages ​​such as "C" or similar programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, as a separate software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., through the Internet using an Internet service provider).

[0025] The present invention will be described below with reference to the flowcharts and / or block diagrams of the methods, devices (systems) and computer program products according to the embodiments of the present invention. It should be understood that each box of the flowchart and / or block diagram and the combination of the boxes in the flowchart and / or block diagram can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer or other programmable data processing device, so as to produce a machine, so that when these computer program instructions are executed by the processor of the computer or other programmable data processing device, a device for implementing the functions / actions specified in one or more boxes in the flowchart and / or block diagram is generated.

[0026] These computer program instructions may also be stored in a computer-readable medium, which enables a computer, other programmable data processing apparatus, or other device to operate in a specific manner, so that the instructions stored in the computer-readable medium produce an article of manufacture including instructions for implementing the functions / actions specified in one or more blocks in the flowchart and / or block diagram.

[0027] Computer program instructions may also be loaded onto a computer, other programmable data processing apparatus, or other device so that a series of operational steps are performed on the computer, other programmable data processing apparatus, or other device to produce a computer-implemented process, so that the instructions executed on the computer or other programmable device provide a process for implementing the functions / actions specified in one or more boxes in the flowchart and / or block diagram.

[0028] The terminal of the present invention includes a processor and a memory.

[0029] The memory is used to store computer programs; preferably, the memory includes: ROM, RAM, disk, USB flash drive, memory card or CD, etc., which can store program codes.

[0030] The processor is connected to the memory and is used to execute the computer program stored in the memory so that the terminal executes the following corpus catalog management method for industrial large models.

[0031] Preferably, the processor can be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it can also be a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components.

[0032] like Figure 1 As shown, in one embodiment, the present invention discloses a corpus catalog management method for an industrial large model, comprising the following steps: S100: Acquire basic information of a target corpus, and preprocess the target corpus according to the basic information to obtain a standard corpus.

[0033] In this embodiment, when the target corpus needs to be stored, the basic information of the target corpus is first obtained, so that the target corpus can be preprocessed according to the basic information to ensure the efficiency of the final storage.

[0034] In some embodiments, preprocessing the target corpus according to the basic information to obtain a standard corpus includes: Performing field scanning on the target corpus to remove invalid fields and noise fields in the target corpus to obtain a first corpus; Acquire a sensitive word library corresponding to the target corpus according to the basic information, compare the first corpus with the sensitive word library, and replace sensitive words in the first corpus with label words to obtain a corresponding standard corpus; The label words correspond to each of the sensitive words one by one.

[0035] After determining the target corpus, first perform a field scan on the target corpus to remove invalid fields and noise fields to obtain the first corpus, avoiding the adverse effects of invalid fields and noise fields on subsequent processing. Invalid fields include blank fields and repeated fields, and noise fields include common noise information, such as advertisements, promotion links, special symbols, etc. Then, compare and match the sensitive word library based on the basic information of the target corpus to replace sensitive words with label words that meet the requirements to obtain the corresponding standard corpus, avoiding the appearance of sensitive words in the target corpus during storage and affecting the use of the corpus.

[0036] It should be noted that after obtaining the standard corpus, the standard corpus is also converted into a standard format according to a standard encoding format to ensure that the text in the standard corpus has a unified format, so as to facilitate subsequent processing of the standard corpus and ensure the consistency of the text.

[0037] S200: Perform scenario analysis on the standard corpus to obtain corresponding label information.

[0038] In some further embodiments, the basic information includes at least industry information, modality information and language information, and the performing scenario analysis on the standard corpus to obtain corresponding label information includes: Acquire the industry type of the target corpus according to the industry information, and generate a first label matrix corresponding to the standard corpus after converting the industry type into a preset coding format; Acquiring a modality type in the target corpus according to the modality information, and converting the modality type into a preset encoding format to generate a second label matrix corresponding to the standard corpus; Acquire the language type of the target corpus according to the language information, and generate a third label matrix corresponding to the standard corpus after converting the language type into a preset encoding format; Integrate the first label matrix, the second label matrix and the third label matrix in order to form a label matrix to obtain the label information; The rows and columns of the first label matrix, the second label matrix and the third label matrix are the same.

[0039] In this embodiment, since the standard corpus generally includes industry information, modal information and language information, the industry information is the industry using the standard corpus. For example, the corpus of the financial industry may contain information such as loans, investments, and financial products; the corpus of the medical industry may contain information such as diseases, drugs, and treatment methods. Modal information is organized according to the type and usage scenario of the corpus data, including text modality, image modality, and audio modality. Text modality includes text information such as articles, news, and blogs; image modality includes image information such as photos, images, and videos; audio modality includes audio information such as audio, telephone recordings, and podcasts. Language information organizes corpora according to language type and region. Corpus data in different languages ​​such as English, Chinese, and Spanish can be organized in different language directories.

[0040] After determining the basic information of the target corpus, the industry type of the target corpus is obtained according to the industry information, and the industry type is converted into a preset coding format to generate a first label matrix corresponding to the standard corpus. The industry type is the industry name corresponding to the target corpus, and the industry code is obtained by converting the industry type into the corresponding preset coding format. Then, according to the preset matrix size, the industry code is used to establish the first label matrix, so as to facilitate the subsequent calculation and processing of the first label matrix.

[0041] Similarly, after determining the corresponding language modality information and language information based on the basic information of the target corpus, the modality type and language type are respectively converted into the modality code and language code of the corresponding preset encoding format, and respectively converted into the corresponding second label matrix and third label matrix.

[0042] It should be noted that the sizes of the first label matrix, the second label matrix and the third label matrix are determined according to preset rules, or according to the size of the target corpus, mainly to ensure subsequent unified calculations. If there is an insufficient number of codes, blank codes are used to supplement them to ensure the consistency of the final matrix specifications and facilitate subsequent calculations.

[0043] S300: Calculate the catalog value of the standard corpus according to the tag information to obtain a standard catalog value of the standard corpus.

[0044] In some other embodiments, calculating the catalog value of the standard corpus according to the tag information to obtain the standard catalog value of the standard corpus includes: Calculating the same-position average of the first label matrix, the second label matrix and the third label matrix to obtain an average label matrix; Acquire a corpus sequence of the standard corpus and generate a corpus index value of the corpus sequence; Respectively calculating the ratios between the eigenvalues ​​corresponding to the first label matrix, the second label matrix, the third label matrix, and the average label matrix and the corpus index value to obtain a first directory value, a second directory value, a third directory value, and an average directory value, respectively; combining the first catalog value, the second catalog value, the third catalog value, and the average catalog value to generate a catalog sequence; The directory index value of the directory sequence is calculated, the standard directory value of the standard corpus is determined according to the corpus index value and the directory index value, and a mapping relationship between the standard directory value and the standard corpus is established.

[0045] In this embodiment, after the first label matrix A, the second label matrix B, and the third label matrix C are calculated, the elements of the corresponding rows and columns of the first label matrix A, the second label matrix B, and the third label matrix C are first accumulated and averaged to obtain the average label matrix D. The specific formula is as follows: , Among them, i represents the i-th row of the matrix, and j represents the j-th column of the matrix.

[0046] After the average label matrix is ​​calculated, the corpus sequence of the standard corpus is obtained, and the corpus index value of the corpus sequence is generated. Specifically, the standard corpus is first encoded according to a preset format, and then the encoded corpus sequence is obtained by arranging it in the encoding order, and then the corpus index value, such as the sequence mean, is calculated through the corpus sequence. After that, the eigenvalues ​​corresponding to the first label matrix, the second label matrix, the third label matrix, and the average label matrix are respectively calculated to obtain the first eigenvalue, the second eigenvalue, the third eigenvalue, and the average eigenvalue, and the first eigenvalue, the second eigenvalue, the third eigenvalue, and the average eigenvalue are respectively calculated with the corpus index value to obtain the first directory value, the second directory value, the third directory value, and the average directory value. Then, the first directory value, the second directory value, the third directory value and the average directory value are combined together to form a directory sequence. Since the first label matrix, the second label matrix, the third label matrix, the average label matrix and the corpus sequence corresponding to different target corpora are all different, the first directory value, the second directory value, the third directory value and the average directory value are obtained by calculating the ratio between the eigenvalues ​​of the first label matrix, the second label matrix, the third label matrix and the corpus index value of the corpus sequence, respectively. By combining the first directory value, the second directory value, the third directory value and the average directory value, a directory sequence with unique attributes of the target corpus can be obtained, so as to quickly locate the directory of the target corpus.

[0047] Further, the step of combining the first directory value, the second directory value, the third directory value and the average directory value to generate a directory sequence includes: The directory sequence is generated by inserting separators at the beginning and the end of the first directory value, the second directory value, the third directory value and the average directory value, and arranging them in sequence.

[0048] After obtaining the directory sequence, the directory index value of the directory sequence is calculated, so as to determine the standard directory value of the standard corpus according to the corpus index value and the directory index value, and establish a mapping relationship between the standard directory value and the standard corpus.

[0049] In this embodiment, since the directory index value is a unique identifier of the directory sequence, in order to further determine the unique directory, the standard directory value uniquely corresponding to the standard corpus is determined based on the corpus index value of the standard corpus and the directory index value of the directory sequence. After establishing a mapping relationship between the standard directory value and the standard corpus, the standard directory value uniquely corresponding to the standard corpus can be determined, which is convenient for generating the corresponding target directory according to the standard directory value, and also convenient for quickly querying the corresponding standard corpus according to the standard directory value, thereby realizing rapid query and storage of the standard corpus.

[0050] Furthermore, the standard directory value of the standard corpus is a sequence combination value of the corpus index value and the directory index value. For example, if the corpus index value is X and the directory index value is Y, then the standard directory value is (X, Y), which further ensures the uniqueness of the standard directory value.

[0051] The corpus index value and the directory index value are both sequences, thereby ensuring that standard directory values ​​will not be repeated.

[0052] S400: Generate a corresponding target directory according to the standard directory value, and map the standard corpus with the target directory and store them.

[0053] In some embodiments, generating a corresponding target directory according to the standard directory value, mapping the standard corpus with the target directory and storing the mappings thereafter, comprises: Obtaining a corresponding directory path in a directory library according to the standard directory value, and generating a target directory corresponding to the standard corpus according to the directory path; After mapping the standard corpus with the target directory, the standard corpus is stored in the target directory.

[0054] After determining the standard directory value of the standard corpus, the corresponding directory path is obtained in the directory library according to the standard directory value, so as to generate the target directory corresponding to the standard corpus according to the directory path, and then the standard corpus is directly stored in the target directory after mapping the standard corpus with the target directory, thereby completing the rapid storage of the standard corpus. When it is necessary to search, it is only necessary to determine the standard directory value of the corpus, and then the corresponding directory path can be determined, and the required corpus can be quickly found according to the directory path. There is no need to search and save in order according to the classification order of the corpus, which effectively improves the efficiency of saving and searching the corpus.

[0055] In some embodiments, acquiring a corresponding directory path according to the standard directory value, and generating a corresponding target directory according to the directory path, includes: Splitting the standard directory value into a corresponding number of multiple sub-directory values, and sequentially generating sub-paths from the multiple sub-directory values; The plurality of sub-paths are combined together to obtain the directory path, and a corresponding target directory is generated according to the directory path.

[0056] In order to reduce complexity, the standard directory value is split into a preset number of sub-directory values ​​in sequence, and then multiple sub-paths are generated in sequence at the same time for multiple sub-directory values, and the multiple sub-paths are combined together in the original order to obtain a directory path. In this way, the corresponding target directory can be generated according to the directory path to improve the efficiency of generating the target directory.

[0057] It should be noted that the protection scope of the corpus catalog management method for industrial large models described in the present invention is not limited to the execution order of the steps listed in this embodiment, and all solutions implemented by adding, reducing, or replacing steps in the prior art based on the principles of the present invention are included in the protection scope of the present invention.

[0058] like Figure 2 As shown, in one embodiment, the present invention provides a corpus catalog management system for industrial large models, including: A preprocessing module 201 is used to obtain basic information of a target corpus, and preprocess the target corpus according to the basic information to obtain a standard corpus; An analysis module 202 is used to perform scene analysis on the standard corpus to obtain corresponding tag information; A calculation module 203 is used to calculate the catalog value of the standard corpus according to the tag information to obtain a standard catalog value of the standard corpus; The directory generation module 204 is used to generate a corresponding target directory according to the standard directory value, and to map the standard corpus with the target directory and store them.

[0059] It should be noted that the structure and principle of the above-mentioned corpus catalog management system of the industrial large model correspond one to one with the steps in the above-mentioned corpus catalog management method of the industrial large model, so they will not be repeated here.

[0060] It should be noted that it should be understood that the division of the various modules of the above system is only a division of logical functions. In actual implementation, they can be fully or partially integrated into one physical entity, or they can be physically separated. And these modules can all be implemented in the form of software called by processing elements; they can also be all implemented in the form of hardware; some modules can also be implemented in the form of software called by processing elements, and some modules can be implemented in the form of hardware. For example, the x module can be a separately established processing element, or it can be integrated in a certain chip of the above system for implementation. In addition, it can also be stored in the memory of the above system in the form of program code, and called and executed by a certain processing element of the above system. The implementation of other modules is similar. In addition, these modules can be fully or partially integrated together, or they can be implemented independently. The processing element described here can be an integrated circuit with signal processing capabilities. In the implementation process, each step of the above method or each module above can be completed by an integrated logic circuit of hardware in the processor element or instructions in the form of software.

[0061] For example, the above modules may be one or more integrated circuits configured to implement the above methods, such as one or more application specific integrated circuits (ASIC), or one or more digital signal processors (DSP), or one or more field programmable gate arrays (FPGA). For another example, when a module above is implemented in the form of a processing element scheduling program code, the processing element may be a general-purpose processor, such as a central processing unit (CPU) or other processor that can call program code. For another example, these modules may be integrated together and implemented in the form of a system-on-a-chip (SOC).

[0062] It should be noted that the corpus catalog management system for industrial large models of the present invention can implement the corpus catalog management method for industrial large models of the present invention, but the implementation device of the corpus catalog management method for industrial large models of the present invention includes but is not limited to the structure of the corpus management system listed in this embodiment, and all structural deformations and replacements of the prior art made according to the principles of the present invention are included in the protection scope of the present invention.

[0063] In summary, the corpus directory management method and system for industrial large models of the present invention obtains standard corpus by preprocessing the target corpus, and performs scene analysis on the standard corpus to obtain label information corresponding to the standard corpus, and then calculates the standard directory value corresponding to the target corpus according to the label information. Since the standard directory value is calculated according to the label information of the standard corpus and is unique, the corresponding standard directory can be generated according to the standard directory value. The target corpus can be directly stored in the standard directory, and the corresponding corpus can be quickly queried according to the standard directory. There is no need to search in sequence according to the classification directory of the corpus, which effectively improves the storage and call efficiency of the corpus and can improve the utilization efficiency of the corpus. Therefore, the present invention effectively overcomes the various shortcomings in the prior art and has a high industrial utilization value.

[0064] The above embodiments are merely illustrative of the principles and effects of the present invention, and are not intended to limit the present invention. Anyone familiar with the art may modify or alter the above embodiments without departing from the spirit and scope of the present invention. Therefore, all equivalent modifications or alterations made by a person of ordinary skill in the art without departing from the spirit and technical concept disclosed by the present invention shall still be covered by the claims of the present invention.

Claims

1. A corpus catalog management method for industrial large models, characterized in that: The steps include: Acquiring basic information of the target corpus, and preprocessing the target corpus according to the basic information to obtain standard corpus; Performing scenario analysis on the standard corpus to obtain corresponding label information; Calculating the catalog value of the standard corpus according to the tag information to obtain a standard catalog value of the standard corpus; A corresponding target directory is generated according to the standard directory value, and the standard corpus is mapped with the target directory and then stored.

2. The method for managing a corpus catalog for an industrial large model according to claim 1, characterized in that: The preprocessing of the target corpus according to the basic information to obtain the standard corpus includes: Performing field scanning on the target corpus to remove invalid fields and noise fields in the target corpus to obtain a first corpus; Acquire a sensitive word library corresponding to the target corpus according to the basic information, compare the first corpus with the sensitive word library, and replace sensitive words in the first corpus with label words to obtain a corresponding standard corpus; The label words correspond to each of the sensitive words one by one.

3. The method for managing corpus catalogues for industrial large models according to claim 1, characterized in that: The basic information includes at least industry information, modality information and language information, and the scene analysis of the standard corpus to obtain corresponding label information includes: Acquire the industry type of the target corpus according to the industry information, and generate a first label matrix corresponding to the standard corpus after converting the industry type into a preset coding format; Acquiring a modality type in the target corpus according to the modality information, and converting the modality type into a preset encoding format to generate a second label matrix corresponding to the standard corpus; Acquire the language type of the target corpus according to the language information, and generate a third label matrix corresponding to the standard corpus after converting the language type into a preset encoding format; Integrate the first label matrix, the second label matrix and the third label matrix in order to form a label matrix to obtain the label information; The rows and columns of the first label matrix, the second label matrix and the third label matrix are the same.

4. The method for managing corpus catalogues for industrial large models according to claim 3 is characterized in that: The calculating the catalog value of the standard corpus according to the tag information to obtain the standard catalog value of the standard corpus includes: Calculating the same-position average of the first label matrix, the second label matrix and the third label matrix to obtain an average label matrix; Acquire a corpus sequence of the standard corpus and generate a corpus index value of the corpus sequence; Respectively calculating the ratios between the eigenvalues ​​corresponding to the first label matrix, the second label matrix, the third label matrix, and the average label matrix and the corpus index value to obtain a first directory value, a second directory value, a third directory value, and an average directory value, respectively; combining the first catalog value, the second catalog value, the third catalog value, and the average catalog value to generate a catalog sequence; The directory index value of the directory sequence is calculated, the standard directory value of the standard corpus is determined according to the corpus index value and the directory index value, and a mapping relationship between the standard directory value and the standard corpus is established.

5. The method for managing corpus catalogues for industrial large models according to claim 4, characterized in that: The step of generating a corresponding target directory according to the standard directory value and mapping the standard corpus with the target directory for storage includes: Obtaining a corresponding directory path in a directory library according to the standard directory value, and generating a target directory corresponding to the standard corpus according to the directory path; After mapping the standard corpus with the target directory, the standard corpus is stored in the target directory.

6. The method for managing corpus catalogues for industrial large models according to claim 5, characterized in that: The step of acquiring a corresponding directory path according to the standard directory value and generating a corresponding target directory according to the directory path includes: Splitting the standard directory value into a corresponding number of multiple sub-directory values, and sequentially generating sub-paths from the multiple sub-directory values; The plurality of sub-paths are combined together to obtain the directory path, and a corresponding target directory is generated according to the directory path.

7. The method for managing corpus catalogues for industrial large models according to claim 5, characterized in that: The step of combining the first directory value, the second directory value, the third directory value and the average directory value to generate a directory sequence comprises: The directory sequence is generated by inserting separators at the beginning and the end of the first directory value, the second directory value, the third directory value and the average directory value, and arranging them in sequence.

8. The method for managing corpus catalogues for industrial large models according to claim 5, characterized in that: The standard catalog value of the standard corpus is a sequence combination value of the corpus index value and the catalog index value.

9. A corpus catalog management system for industrial large models, characterized in that: include: A preprocessing module, used to obtain basic information of the target corpus, and preprocess the target corpus according to the basic information to obtain a standard corpus; An analysis module, used for performing scene analysis on the standard corpus to obtain corresponding label information; A calculation module, configured to calculate a catalog value of the standard corpus according to the tag information to obtain a standard catalog value of the standard corpus; The directory generation module is used to generate a corresponding target directory according to the standard directory value, and store the standard corpus after mapping with the target directory.

Images