Corpus management method, electronic equipment, storage medium and program product

Abstract

The invention provides a corpus management method, electronic equipment, a storage medium and a program product. The method comprises the following steps: storing corpus data into a database; analyzing the corpus data, and determining at least one corpus tag associated with the corpus data from a preset tag information table; defining at least one corpus tag as a tag group, and performing index association with corpus data to form a data set; corpus data under the label group is monitored, and when the corpus data changes, the corpus label and the changed corpus data are re-indexed and associated, so that updating of the data set is completed. The invention designs a flexible dynamic label management mechanism capable of responding to data change in real time, is used for indexing and managing massive corpus data, can adapt to continuously changing corpus data and user requirements, and ensures that a corpus data set is always updated. Through the optimized index strategy, the speed and accuracy of data retrieval are remarkably improved, and the user can quickly position the required data.

Description

Technical Field

[0001] The present invention relates to the field of computer technology, and in particular to a corpus management method, electronic equipment, storage medium and program product. Background Art

[0002] In natural language processing (NLP) and machine learning projects, corpus data management is a crucial link because it directly affects the effectiveness and efficiency of model training.

[0003] In existing corpus data management technologies, the indexing and retrieval of corpus data usually rely on a static tagging system, which is inefficient when processing large-scale data. At the same time, when the tagging system needs to be updated, the time it takes to update the tagging system is proportional to the amount of data, which is very expensive for PB-level data. Summary of the invention

[0004] The present invention provides a corpus management method, electronic device, storage medium and program product, designs a flexible dynamic tag management mechanism that can respond to data changes in real time, is used to index and manage massive corpus data, can adapt to the constantly changing corpus data and user needs, and ensure that the corpus data set is always updated.

[0005] In order to achieve the above purpose and other advantages, the present invention adopts the following technical solutions:

[0006] In a first aspect, the present invention provides a corpus management method, comprising:

[0007] Put the corpus data into the database;

[0008] Parsing the corpus data, and determining at least one corpus tag associated with the corpus data from a preset tag information table;

[0009] The at least one corpus tag is defined as a tag group, and is indexed and associated with the corpus data to form a data set;

[0010] The corpus data under the tag group is monitored, and when the corpus data changes, the corpus tag is re-indexed and associated with the changed corpus data to complete the update of the data set.

[0011] According to a corpus management method provided by the present invention, the step of storing corpus data in the corpus includes:

[0012] Preprocessing the original corpus data to obtain corpus data;

[0013] The corpus data is uploaded to the distributed search engine and object storage service through the client to complete the corpus storage.

[0014] According to a corpus management method provided by the present invention, the step of parsing the corpus data and determining at least one corpus tag associated with the corpus data from a preset tag information table includes:

[0015] Parsing the corpus data to obtain feature vectors in the corpus data;

[0016] According to the feature vector, calculating the correlation score between each category label in the label information table and the feature vector;

[0017] When the relevance score is greater than or equal to a preset threshold, the category label is determined to be the corpus label.

[0018] According to a corpus management method provided by the present invention, the step of defining the at least one corpus tag as a tag group and forming a data set after indexing and associating it with the corpus data includes:

[0019] Construct a label data index table;

[0020] Constructing the tag group and the corresponding corpus data into mapping relationship data;

[0021] Cache the mapping relationship data into the label data index table corresponding to the data set;

[0022] Based on the label data index table, the corpus data associated with the corpus labels in the label group are searched, and the corpus data are combined into the data set.

[0023] A corpus management method according to the present invention further includes:

[0024] The tag information table is stored in a cache pool in the form of key-value pairs using cache technology.

[0025] A corpus management method according to the present invention further includes:

[0026] The tag group is monitored, and when a corpus tag of the tag group is added or updated, the tag table in the cache pool is synchronously updated.

[0027] A corpus management method according to the present invention further includes:

[0028] The update task is triggered periodically to re-read the corpus data of the data set, and then the corpus tags bound to the data set are updated.

[0029] In a second aspect, the present invention provides an electronic device, the electronic device comprising:

[0030] One or more processors; and a memory storing computer program instructions, wherein the computer program instructions, when executed by the processor, implement the steps of any of the above-described corpus management methods.

[0031] In a third aspect, the present invention provides a computer-readable storage medium having a computer program / instruction stored thereon, wherein the computer program / instruction, when executed by a processor, implements the steps of any of the above-described corpus management methods.

[0032] In a fourth aspect, the present invention provides a computer program product, comprising a computer program / instruction, which, when executed by a processor, implements the steps of any of the above-described corpus management methods.

[0033] The present invention provides a corpus management method, electronic device, storage medium and program product, which are as follows: corpus data is stored in the database; the corpus data is parsed, and at least one corpus tag associated with the corpus data is determined from a preset tag information table; at least one corpus tag is defined as a tag group, and is indexed and associated with the corpus data to form a data set; the corpus data under the tag group is monitored, and when the corpus data changes, the corpus tag is re-indexed and associated with the changed corpus data to complete the update of the data set. The present invention designs a flexible dynamic tag management mechanism that can respond to data changes in real time, which is used to index and manage massive corpus data, can adapt to constantly changing corpus data and user needs, and ensure that the corpus data set is always updated. Through the optimized indexing strategy, the speed and accuracy of data retrieval are significantly improved, so that users can quickly locate the required data. The dynamic tag management mechanism has cross-domain application capabilities and can serve the data management and analysis needs of different industries. BRIEF DESCRIPTION OF THE DRAWINGS

[0034] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other implementation methods can be obtained based on these drawings without paying any creative work.

[0035] Figure 1 is a flow chart of a corpus management method provided by an embodiment of the present invention;

[0036] Figure 2 is a system architecture diagram of a corpus management system provided by an embodiment of the present invention;

[0037] Figure 3is a schematic diagram of a corpus management process based on a corpus management system provided by an embodiment of the present invention;

[0038] Figure 4 It is a structural schematic diagram of the electronic device provided by the present invention. DETAILED DESCRIPTION

[0039] The above description is only an overview of the technical solution of the present invention. In order to more clearly understand the technical means of the present invention, it can be implemented in accordance with the contents of the specification. In order to make the above and other purposes, features and advantages of the present invention more obvious and easy to understand, the following specifically cites a preferred embodiment and describes it in detail with the accompanying drawings as follows.

[0040] It should be noted that it is explicitly and implicitly understood by those of ordinary skill in the art that the embodiments described in the present invention can be combined with other embodiments without conflict. Unless otherwise defined, the technical terms or scientific terms involved in the present invention should be the usual meanings understood by people with ordinary skills in the technical field to which the present invention belongs. The words "one", "a", "a", "the" and the like involved in the present invention do not indicate a quantitative limitation and may represent the singular or plural. The terms "including", "comprising", "having" and any of their variations involved in the present invention are intended to cover non-exclusive inclusions; the terms "first", "second", "third", etc. involved in the present invention are merely to distinguish similar objects and do not represent a specific ordering of objects.

[0041] When it comes to method steps, the sequence illustrated in this document represents an exemplary solution, but does not mean a limitation on the sequence.

[0042] Reference Figure 1 As shown, an embodiment of the present invention provides a corpus management method, including:

[0043] Step S1: Store the corpus data into the database.

[0044] In this embodiment, step S1 specifically includes:

[0045] Step S101: preprocessing the original corpus data to obtain corpus data;

[0046] Step S102: Upload the corpus data to the distributed search engine and object storage service through the client to complete the corpus storage.

[0047] Specifically, the original corpus data can be derived from the user's corpus, or the corresponding original corpus data can be collected through the network. The source of the original corpus data is not specifically limited in this application. Corpus data can include various types, such as text, audio, picture, video, etc. However, the corpus data actually collected is usually full of noise, errors and inconsistencies, which brings huge challenges to the application of subsequent modules (such as data analysis modules, machine learning modules, etc.), and the value of the data cannot be effectively brought into play. Therefore, before the corpus data is stored, it is necessary to pre-process the corpus data, including corpus cleaning and natural language processing, and obtain finished corpus data by removing duplicate, invalid or low-quality data, so as to improve the accuracy of the corpus data.

[0048] For massive corpus data, the finished corpus data is uploaded to distributed search engines (such as elasticSearch and other distributed search engines) and object storage services (such as OSS, COS and other object storage services) through the client. Object storage services are suitable for storing large amounts of unstructured data. Corpus data usually exists in unstructured or semi-structured forms, so uploading it to object storage services is a good choice. Therefore, by utilizing the high scalability and reliability of distributed search engines and object storage services, the secure storage of massive corpus data is achieved. Through the distributed search function of elasticSearch, the corpus data stored therein can be quickly retrieved and queried, which improves data processing efficiency.

[0049] Step S2: Parse the corpus data and determine at least one corpus tag associated with the corpus data from a preset tag information table.

[0050] In this embodiment, step S2 specifically includes:

[0051] Step S201: parsing the corpus data to obtain feature vectors in the corpus data;

[0052] Step S202: Calculate the correlation score between each category label and the feature vector in the label information table according to the feature vector;

[0053] Step S203: when the relevance score is greater than or equal to a preset threshold, determining the category label as a corpus label.

[0054] Specifically, natural language processing technology is used to parse and process the corpus data, such as word segmentation, part-of-speech tagging, named entity recognition, etc., to extract key information, text features or structures in the corpus data. The parsed corpus data will be converted into a data format that is easy to process, such as a feature vector. The cosine similarity algorithm is used to calculate the correlation between each category label and the feature vector in the label information table. For each category label, if its correlation score with the feature vector is greater than or equal to a preset threshold, the category label is determined as the label of the corpus data. One corpus data can correspond to one or more corpus labels to form an association relationship between the corpus data and the corpus label. Therefore, matching corresponding corpus labels according to different corpus data is a flexible dynamic label management mechanism that can respond to data changes in real time. Dynamic corpus labels can be regarded as a dynamic entity that can exist and be continuously updated independently of corpus data, so that the definition of the data set is no longer limited to a pre-set static binding mode.

[0055] In practical applications, a user feedback mechanism can be introduced to continuously optimize the accuracy and efficiency of tag matching. The setting of the threshold can be adjusted according to the actual application scenario to achieve the best tag matching effect. This embodiment does not make specific limitations.

[0056] Step S3: define at least one corpus tag as a tag group, and index and associate it with the corpus data to form a data set.

[0057] In this embodiment, step S3 specifically includes:

[0058] Step S301: construct a tag data index table;

[0059] Step S302: constructing mapping relationship data between the tag group and the corresponding corpus data;

[0060] Step S303: caching the mapping relationship data into a label data index table corresponding to the data set;

[0061] Step S304: based on the label data index table, search for corpus data associated with the corpus labels in the label group, and combine the corpus data into a data set.

[0062] Specifically, a data structure is created to store the mapping relationship between corpus tags and the corpus data associated therewith. The data structure includes an inverted index, a B-tree, a hash table, etc. According to the characteristics of the corpus data and the definition of the tag, the tag group to which each corpus data belongs is determined, and a mapping relationship between them is established. The mapping relationship is written into the tag data index table. According to the mapping relationship in the index table, the corpus data matching the specified corpus tag is quickly located. These data are combined into a data set, which contains all the corpus data associated with the specified tag group. The combined data set is returned to the user or system for calling for further analysis, processing or display. Therefore, this dynamic data set generation mechanism greatly improves the flexibility and efficiency of data management.

[0063] When the amount of data in the tag data index table continues to increase, resulting in a decrease in query performance, the index table can be properly optimized, such as redesigning the structure of the index table and establishing partitions.

[0064] The present invention takes into account the indexing and management requirements of massive data, adopts efficient indexing technology, and can efficiently match the corpus data corresponding to the tag group, thereby generating a data set.

[0065] Furthermore, user-defined tag groups can be flexibly combined to integrate multi-dimensional data features through different types of corpus tags to support complex queries and multi-dimensional dataset construction. This enriches the analysis depth of the dataset and supports users to explore and understand the data from different perspectives.

[0066] Step S4: monitor the corpus data under the tag group, and when the corpus data changes, re-index and associate the corpus tag with the changed corpus data to complete the update of the data set.

[0067] For scenarios where corpus data is frequently updated, it is necessary to select appropriate monitoring technologies or frameworks, such as database triggers, file system monitoring tools, etc., so that these changes can be captured immediately when the corpus data changes. When the listener detects a change in the corpus data (such as a new addition, modification, or deletion operation), it triggers an event. In response to the event triggered by the listener, the changed corpus information is extracted and written to the corresponding data set. The system uses a label data index table to speed up queries. After the corpus data changes, the index table also needs to be updated accordingly to complete the update of the data set to ensure the accuracy and timeliness of the data set.

[0068] Therefore, this embodiment provides a more intuitive, flexible and powerful corpus data management solution for users. This solution can adapt to the ever-changing data environment, meet the growing data management needs, and has broad application prospects and practical value.

[0069] In this embodiment, the corpus management method further includes:

[0070] The cache technology is used to store the tag information table in the cache pool in the form of key-value pairs.

[0071] Specifically, this embodiment uses Redis cache. In the cache, data structures are usually stored in the form of key-value pairs. By storing the tag information table in the cache, the system can access the data faster. When tag information is frequently requested, it will place a heavy burden on the database if it is retrieved from the database every time. Using cache can reduce the number of accesses to the database, thereby reducing its load and enabling it to handle other tasks more efficiently. Compared with traditional relational databases, using a cache pool will greatly improve the efficiency of dynamic tag management.

[0072] In this embodiment, the corpus management method further includes:

[0073] Monitor the tag group. When the corpus tags of the tag group are added or updated, the tag table in the cache pool is updated synchronously.

[0074] Specifically, through the monitoring mechanism, the addition and update operations of corpus tags in the tag group can be captured in real time, and these changes can be synchronized to the cache pool immediately, which ensures that the data in the cache is always up-to-date and accurate. Therefore, the automated tag update mechanism further enhances the responsiveness of the system, allowing the data set to be dynamically adjusted according to the real-time updated tags, ensuring the timeliness and accuracy of the data set, and providing users with the most cutting-edge data resources.

[0075] A corpus management method according to the present invention further includes:

[0076] The update task is triggered regularly to re-read the corpus data of the dataset, and then update the corpus tags bound to the dataset.

[0077] Specifically, a task is automatically started at a preset time interval or a specific time point. When a task is executed, it will access the data set and count the size of the corpus under the data set (the size of the storage space occupied by the data set) and the number of corpuses (the number of independent data items in the data set, which can be text paragraphs, sentences, words, images, audio clips, etc.). Scheduled tasks achieve automated updates, reduce the need for manual intervention, and save labor costs. The trigger time and frequency of scheduled tasks can be flexibly set according to actual needs to meet different update requirements.

[0078] Figure 2 is a schematic diagram of the architecture of a corpus management system provided by an embodiment of the present invention, Figure 3 Schematic diagram of the corpus management process based on the corpus management system provided by the embodiment of the present invention. Figure 2 , 3 As shown, the corpus management system includes: a dynamic labeling system and a data set updating system.

[0079] The dynamic tag system includes: a dynamic tag management module and a tag-data association module.

[0080] The dynamic tag management module is used to manage tag creation, update, and deletion operations, and uses the cache pool to achieve dynamic tag management. When a tag is updated or added, the tag-data association module is notified to process and synchronize the tags in the tag information table.

[0081] The tag-data association module is used to parse the corpus data, determine at least one corpus tag associated with the corpus data from the tag information table, and send the corpus tag associated with the corpus data to the data set update system.

[0082] The dataset updating system includes: a dataset building module and a dataset updating module.

[0083] The data set construction module is used to receive the label information from the label-data association module, define at least one corpus label as a label group, and form a data set after indexing and associating it with the corpus data using the index construction technology.

[0084] The dataset update module is used to monitor the corpus data under the tag group. When the corpus data changes, the corpus tag is re-indexed and associated with the changed corpus data to complete the update of the dataset. It is also used to trigger the update task at a fixed time, re-read the corpus data of the dataset, and then update the corpus tag bound to the dataset.

[0085] The corpus management system can also automatically identify invalid corpus data through machine learning models. For invalid corpus data, an API interface for the invalid corpus is provided. The API interface is called to delete the specified invalid corpus. The system will perform a deletion operation to remove the specified invalid corpus from the database or storage system. After the deletion operation is completed, the data set should be updated to ensure that the deleted invalid corpus no longer appears in the data set.

[0086] In summary, a corpus management method provided by an embodiment of the present invention is to store corpus data; parse the corpus data, determine at least one corpus tag associated with the corpus data from a preset tag information table; define at least one corpus tag as a tag group, and form a data set after indexing and associating it with the corpus data; monitor the corpus data under the tag group, and when the corpus data changes, re-index and associate the corpus tag with the changed corpus data to complete the update of the data set. The present invention designs a flexible dynamic tag management mechanism that can respond to data changes in real time, which is used to index and manage massive corpus data, can adapt to the ever-changing corpus data and user needs, and ensure that the corpus data set is always updated. Through the optimized indexing strategy, the speed and accuracy of data retrieval are significantly improved, so that users can quickly locate the required data. Therefore, the present invention provides users with a brand-new data indexing and management method centered on user needs. The use of this method can significantly improve the depth and breadth of data analysis, and provide researchers and developers with powerful tools to support their work in natural language processing and other related fields.

[0087] In addition, some embodiments of the present application also provide an electronic device. The electronic device may be a digital computer in various forms, such as a laptop computer, a desktop computer, a workbench, a personal digital assistant, a server, a blade server, a mainframe computer, etc. The electronic device may also be a mobile device in various forms, such as a personal digital processing, a cellular phone, a smart phone, a wearable device, and other similar computing devices.

[0088] The electronic device includes: one or more processors; and a memory storing computer program instructions, and the computer program instructions, when executed by the processor, implement the steps of the corpus management method provided in the above embodiment. Figure 4 An exemplary structural diagram of the electronic device is disclosed. Figure 4As shown, the electronic device includes: one or more processors 1101, memory 1102, and interfaces for connecting various components, including high-speed interfaces and low-speed interfaces. The various components are connected to each other using different buses and can be installed on a common mainboard or installed in other ways as needed. The processor can process instructions executed in the electronic device, including instructions stored in or on the memory to display graphical information of the GUI on an external input / output device (such as a display device coupled to the interface). In some other embodiments, if necessary, multiple processors and / or multiple buses can be used together with multiple memories and multiple memories. Similarly, multiple electronic devices can be connected, and each device provides some necessary operations (for example, as a server array, a group of blade servers, or a multi-processor system). Among them, the components shown in this article, their connections and relationships, and their functions are only used as examples, and are not intended to limit the implementation of the present application described and / or required herein.

[0089] The electronic device may further include: an input device 1103 and an output device 1104. The processor 1101, the memory 1102, the input device 1103 and the output device 1104 may be connected via a bus or other means. Figure 4 The example of connecting through bus is taken in the following.

[0090] The input device 1103 can receive input digital or character information, and generate key signal input related to the user settings and function control of the electronic device, such as a touch screen, a keypad, a mouse, a track pad, a touch pad, an indicator bar, one or more mouse buttons, a trackball, a joystick and other input devices. The output device 1104 may include a display device, an auxiliary lighting device (e.g., an LED) and a tactile feedback device (e.g., a vibration motor), etc. The display device may include, but is not limited to, a liquid crystal display (LCD), a light emitting diode (LED) display and a plasma display. In some embodiments, the display device may be a touch screen.

[0091] To provide interaction with a user, the electronic device may be a computer. The computer has: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and a pointing device (e.g., a mouse or a trackball), through which the user can provide input to the computer. Other types of devices may also be used to provide interaction with the user; for example, the feedback provided to the user may be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user may be received in any form (including acoustic input, voice input, or tactile input).

[0092] In the embodiment of the present application, a computer program / instruction is stored on a computer-readable medium, and when the computer program / instruction is executed by a processor, the steps of the corpus management method provided in the above embodiment are implemented, or the steps of the agricultural action planning method provided in the above embodiment 2 are executed. The computer-readable medium may be included in the electronic device described in the above embodiment; or it may exist independently without being assembled into the device. The above computer-readable medium carries one or more computer-readable instructions.

[0093] The memory 1102 can be used as a non-transient computer-readable storage medium, which can be used to store non-transient software programs, non-transient computer executable programs and modules. The processor 1101 executes various functional applications and data processing of the server by running the non-transient software programs, instructions and modules stored in the memory 1102, so as to implement the program instructions / modules corresponding to the method provided by any one or more embodiments in the embodiments of the present application.

[0094] The memory 1102 may include a program storage area and a data storage area, wherein the program storage area may store an operating system, an application required for at least one function; the data storage area may store data created according to the use of the electronic device, etc. In addition, the memory 1102 may include a high-speed random access memory, and may also include a non-transient memory, such as at least one disk storage device, a flash memory device, or other non-transient solid-state storage device. In some embodiments, the memory 1102 may optionally include a memory remotely arranged relative to the processor 1101, and these remote memories may be connected to the electronic device via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0095] It should be noted that more specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination thereof. In the present application, a computer-readable storage medium may be any tangible medium containing or storing a program that may be used by or in conjunction with an instruction execution system, apparatus, or device.

[0096] Computer readable storage media include permanent and non-permanent, removable and non-removable media, and can be implemented by any method or technology to store information. Information can be computer readable instructions, data structures, modules of programs or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, read-only compact disk (CD-ROM), digital versatile disk (DVD) or other optical storage, magnetic cassettes, magnetic tape disk storage or other magnetic storage devices or any other non-transmission medium that can be used to store information that can be accessed by a computing device.

[0097] Computer program code for performing the operations of the present application may be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages, such as Java, Smalltalk, C++, and conventional procedural programming languages, such as "C" or similar programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, as a separate software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving a remote computer, the remote computer may be connected to the user's computer via any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0098] In the above-described embodiments, it can be implemented in whole or in part by software, hardware, firmware or any combination thereof. For example, it can be implemented by using an application specific integrated circuit (ASIC), a general-purpose computer or any other similar hardware device. In certain embodiments, the software program of the present application can be executed by a processor to implement the above steps or functions. Similarly, the software program of the present application (including relevant data structures) can be stored in a computer-readable recording medium, for example, a RAM memory, a magnetic or optical drive or a floppy disk and similar devices. In addition, some steps or functions of the present application can be implemented by hardware, for example, as a circuit that cooperates with a processor to perform each step or function.

[0099] The computer program product provided in the embodiment of the present application includes one or more computer programs / instructions, and when the computer program / instructions are executed by the processor, the process or function described in the embodiment of the present application is generated in whole or in part. The computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable devices. The computer instructions may be stored in a computer-readable storage medium, or transmitted from a computer-readable storage medium to another computer-readable storage medium, for example, the computer instructions may be transmitted from a website site, a computer, a server or a data center by wired (e.g., coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) mode to another website site, computer, server or data center. The computer-readable storage medium may be any available medium that a computer can access or a data storage device such as a server or a data center that includes one or more available media integrations. The available medium may be a magnetic medium, (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a DVD), or a semiconductor medium (e.g., a solid state disk (SSD)), etc.

[0100] The flow chart or block diagram in the accompanying drawings shows the possible architecture, function and operation of the equipment, method and computer program product according to various embodiments of the present application. In this regard, each box in the flow chart or block diagram can represent a module, a program segment or a part of a code, and the module, the program segment or a part of the code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a different order from the order marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram and / or flow chart, and the combination of the boxes in the block diagram and / or flow chart can be implemented with a dedicated system for hardware that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0101] The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any person skilled in the art can easily mention changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims, and the above embodiments should be regarded as exemplary and non-restrictive.

Claims

1. A corpus management method, characterized in that: include: Put the corpus data into the database; Parsing the corpus data, and determining at least one corpus tag associated with the corpus data from a preset tag information table; The at least one corpus tag is defined as a tag group, and is indexed and associated with the corpus data to form a data set; The corpus data under the tag group is monitored, and when the corpus data changes, the corpus tag is re-indexed and associated with the changed corpus data to complete the update of the data set.

2. The corpus management method according to claim 1, characterized in that: The step of storing the corpus data includes: Preprocessing the original corpus data to obtain corpus data; The corpus data is uploaded to the distributed search engine and object storage service through the client to complete the corpus storage.

3. The corpus management method according to claim 1, characterized in that: The step of parsing the corpus data and determining at least one corpus tag associated with the corpus data from a preset tag information table includes: Parsing the corpus data to obtain feature vectors in the corpus data; According to the feature vector, calculating the correlation score between each category label in the label information table and the feature vector; When the relevance score is greater than or equal to a preset threshold, the category label is determined to be the corpus label.

4. The corpus management method according to claim 1, characterized in that: The step of defining the at least one corpus tag as a tag group and indexing and associating it with the corpus data to form a data set comprises: Construct a label data index table; Constructing the tag group and the corresponding corpus data into mapping relationship data; Cache the mapping relationship data into the label data index table corresponding to the data set; Based on the label data index table, the corpus data associated with the corpus labels in the label group are searched, and the corpus data are combined into the data set.

5. The corpus management method according to claim 1, characterized in that: Also includes: The tag information table is stored in a cache pool in the form of key-value pairs using cache technology.

6. The corpus management method according to claim 5, characterized in that: Also includes: The tag group is monitored, and when a corpus tag of the tag group is added or updated, the tag table in the cache pool is synchronously updated.

7. The corpus management method according to claim 1, characterized in that: Also includes: The update task is triggered periodically to re-read the corpus data of the data set, and then the corpus tags bound to the data set are updated.

8. An electronic device, characterized in that: The electronic device comprises: One or more processors; and a memory storing computer program instructions, wherein the computer program instructions, when executed by the processor, implement the steps of the corpus management method according to any one of claims 1 to 7.

9. A computer-readable storage medium having a computer program / instruction stored thereon, characterized in that: When the computer program / instructions are executed by a processor, the steps of the corpus management method according to any one of claims 1 to 7 are implemented.

10. A computer program product comprising a computer program / instructions, characterized in that When the computer program / instructions are executed by a processor, the steps of the corpus management method according to any one of claims 1 to 7 are implemented.

Images