Text object indexing method, object storage system and related devices

Abstract

This application relates to a text object indexing method, an object storage system and related devices. Among them, the method includes: after the front-end service module writes a text object into the key-value storage database system, it adds a write success message of the text object to the text write message queue; the text analysis scheduling module consumes the write success message in the text write message queue, generates a text analysis task for the text object and schedules the text analysis module to execute the text analysis task, obtains a set of representative keywords of the text object as the metadata of the text object and updates it to the key-value storage database system, and adds a metadata update message of the text object to the metadata update message queue; so that the index service module reads the metadata of the text object from the key-value storage database system according to the metadata update message of the text object, constructs an inverted index of the text object, and provides an index service. Through the present invention, complex condition retrieval of text objects in an object storage system is realized.

Description

Technical Field

[0001] This application relates to the field of computer technology, and particularly to a method for indexing text objects, an object storage system, and related devices. Background Art

[0002] This section aims to provide background or context for the embodiments of the present invention described in the claims. The description herein should not be admitted as prior art merely because it is included in this section.

[0003] With the development of artificial intelligence technology and the intelligence of cloud product functions and services, it is necessary for cloud object storage systems to provide more perfect product services for user scenarios on the basis of providing basic object storage services to enhance the competitiveness of cloud object storage systems. Most current cloud object storage products do not provide automated solutions for building text search products, and existing object storage systems only support filtering objects based on tags and basic file information such as file size, submission time, and change time after users input tags themselves. The overall function of the index service provided by current object storage systems is relatively simple, and it is difficult to implement complex conditional retrievals in specific scenarios. Summary of the Invention

[0004] Embodiments of the present invention provide a method for indexing text objects, an object storage system, and related devices, which at least solve the problem that the object storage system in related technologies cannot implement complex conditional retrievals.

[0005] A method for indexing text objects includes:

[0006] A front-end service module receives a text object written by a user, writes the text object into a key-value storage database system, and adds a write success message of the text object to a text write message queue;

[0007] A text analysis scheduling module consumes the write success message in the text write message queue, reads the text object from the key-value storage database system according to the write success message, generates a text analysis task for the text object, and schedules the text analysis module to execute the text analysis task;

[0008] The text analysis module executes the text analysis task and returns a set of representative keywords obtained by text analysis to the text analysis scheduling module;

[0009] The text analysis scheduling module updates the set of representative keywords as metadata of the text object to the key-value storage database system, and adds a text object metadata update message to a metadata update message queue;

[0010] The index service module consumes the metadata update message queue, reads the metadata of the text object from the key-value storage database system according to the text object metadata update message, constructs an inverted index of the text object, persists the inverted index, and provides an index service based on the inverted index.

[0011] In some embodiments, the text object indexing method further includes:

[0012] The front-end service module receives the metadata of the storage bucket, where each storage bucket is used to manage user data, and the user data includes text objects;

[0013] The text analysis scheduling module reads the metadata of the user's storage bucket from the management control service module, and reads all the text objects in the corresponding storage bucket from the key-value storage database system according to the metadata of the storage bucket, and generates a text analysis task for all the text objects, where the management control service module stores the metadata of the storage buckets of different users.

[0014] In some embodiments, the text object indexing method further includes:

[0015] The text analysis scheduling module traverses all the storage buckets according to the metadata of the storage buckets of all users provided by the management control service module to generate text analysis tasks for all the text objects in all the storage buckets.

[0016] In some embodiments, the text object indexing method further includes:

[0017] The front-end service module receives the metadata of the storage bucket, where each storage bucket is used to manage user data, and the user data includes text objects;

[0018] The index service module reads the metadata of the user's storage bucket from the management control service module, reads the metadata of all the text objects in the corresponding storage bucket from the key-value storage database system according to the metadata of the storage bucket, and constructs an inverted index of all the text objects, where the management control service module stores the metadata of the storage buckets of different users.

[0019] In some embodiments, the text object indexing method further includes:

[0020] The index service module traverses all the storage buckets according to the metadata of the storage buckets of all users provided by the management control service module to construct an inverted index of the metadata of all the text objects in all the storage buckets.

[0021] In some embodiments, the text analysis module performs text analysis based on natural language processing technology.

[0022] In some of these embodiments, the text object indexing method further includes:

[0023] The front-end service module receives the query message of the user and sends the query message to the indexing service module;

[0024] The indexing service module performs word segmentation on the query message, obtains a query word set according to the word segmentation result, uses the indexing service to query the query result of the text object corresponding to the query word set in the user's storage bucket, and returns the query result to the user.

[0025] An object storage system includes: a front-end service module, a text analysis scheduling module, a text analysis module, a text write message queue, a metadata update message queue, an indexing service module, and a key-value storage database system; the object storage system is used to implement the above-mentioned text object indexing method.

[0026] In some of these embodiments, the object storage system further includes a management and control service module, and the object storage system is further used to implement the above-mentioned text object indexing method.

[0027] An electronic device includes: a processor and a memory storing a program, wherein the program includes instructions that, when executed by the processor, cause the processor to execute the above-mentioned method.

[0028] A non-transitory machine-readable medium storing computer instructions, wherein the computer instructions are used to cause the computer to execute the above-mentioned method.

[0029] In the text object indexing method, object storage system and related devices provided by the embodiments of the present invention, after the front-end service module writes a text object into the key-value storage database system, it adds a write success message of the text object to the text write message queue; the text analysis scheduling module consumes the write success message in the text write message queue, generates a text analysis task of the text object, schedules the text analysis module to execute the text analysis task, obtains a representative keyword set of the text object as metadata of the text object, updates it to the key-value storage database system, and adds a text object metadata update message to the metadata update message queue; so that the indexing service module reads the metadata of the text object from the key-value storage database system according to the text object metadata update message, constructs an inverted index of the text object, persists the inverted index, and provides an indexing service based on the inverted index, realizing complex condition retrieval of text objects in the object storage system.

[0030] Details of one or more embodiments of the present invention are set forth in the following drawings and description to make other features, objects, and advantages of the present invention more concise and understandable. Brief Description of the Drawings

[0031] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the following will briefly introduce the drawings required in the description of the embodiments or the prior art. Obviously, the drawings in the following description are only some embodiments of the present invention. For those of ordinary skill in the art, without creative efforts, other embodiments can be obtained based on these drawings.

[0032] Figure 1 It is a system structure diagram of an object storage system according to an embodiment of the present invention.

[0033] Figure 2 It is a system structure diagram of another object storage system according to an embodiment of the present invention.

[0034] Figure 3 It is a schematic diagram of the working process of the object storage system according to an embodiment of the present invention.

[0035] Figure 4 It is a flowchart of a text object indexing method according to an embodiment of the present invention.

[0036] Figure 5 It is a flowchart of the text object writing process according to an embodiment of the present invention.

[0037] Figure 6 It is a flowchart of the incremental text analysis process of the text object according to an embodiment of the present invention.

[0038] Figure 7 It is a flowchart of the incremental index construction process of the metadata of the text object according to an embodiment of the present invention.

[0039] Figure 8 It is a flowchart of the full - volume text analysis process of the text object according to an embodiment of the present invention.

[0040] Figure 9 It is a flowchart of the full - volume index construction process of the metadata of the text object according to an embodiment of the present invention.

[0041] Figure 10 It is a flowchart of the user query process according to an embodiment of the present invention.

[0042] Figure 11 It is a schematic diagram of the structure of an electronic device according to an embodiment of the present invention. Detailed Embodiments

[0043] The embodiments of the present embodiment will be described in more detail with reference to the accompanying drawings. Although some embodiments of the present embodiment are shown in the drawings, it should be understood that the present embodiment can be implemented in various forms and should not be construed as limited to the embodiments set forth herein. On the contrary, these embodiments are provided to more thoroughly and completely understand the present embodiment. It should be understood that the drawings and embodiments of the present embodiment are only for exemplary purposes and are not used to limit the protection scope of the present embodiment.

[0044] Object storage: It can be understood as a storage structure of key and value, which is a storage system in the cloud. If compared with a file storage system, an object is a file, the key of the object is the file name, and the value of the object is the content of the file. However, different from the file storage system, files have a directory and hierarchical structure, while the object storage system is all flat. Each object is at the same level in a flat address space called a bucket, and an object does not belong to the next level of another object.

[0045] Object: An object in the object storage system can be understood as a file.

[0046] Bucket: A bucket in the object storage system is a container for storing objects. The keys of objects under the same bucket cannot be the same, but the keys of objects under different buckets can be the same. The same object can only belong to a unique bucket; different users can create different buckets, and the names of each bucket are different, so that the data of different users can be distinguished. A user can create multiple buckets.

[0047] Text analysis: Text analysis refers to the representation of text and the selection of its feature items; text analysis is a basic problem in text mining and information retrieval, which quantifies the feature words extracted from the text to represent text information.

[0048] word2vec: It is a group of related models for generating word vectors. These models are shallow and two-layer neural networks used to train and reconstruct linguistic word texts.

[0049] NLP: Natural Language Processing is a discipline that studies the language problems of human-computer interaction. According to the different difficulties of technical implementation, such systems can be divided into three types: simple matching type, fuzzy matching type, and paragraph understanding type.

[0050] Index: An index is a separate, physical storage structure that sorts the values of one or more columns in a database table. It is a collection of the values of one or more columns in a table and a list of logical pointers that point to the data pages in the table that physically identify these values.

[0051] OSS (Object Storage Service) object storage is a cloud storage service that can store massive amounts of unstructured data, such as pictures, audio, video, etc. OSS adopts a distributed storage architecture and has characteristics such as high reliability, high availability, high scalability, and high security. Users can upload, download, manage, and access data through methods such as API, SDK, and console. OSS also provides multiple storage types and billing methods, and users can choose suitable storage types and billing methods according to their own needs.

[0052] Currently, OSS already supports the metadata search of basic storage objects (objects). This function provides a simple text index function based on text-based metadata information and user manual tagging of the text. Its function is limited to indexing the storage type, read and write permissions, file name, file size, modification time, multi-version, and object tags of the metadata, and cannot provide text content indexing, unable to cover the complex retrieval needs of users for text content.

[0053] Users can build their own index service on the user side based on the object storage system, but its development and maintenance costs are high, which is not conducive to users quickly building a text index service.

[0054] To solve the problem that the object storage system of related technologies cannot implement complex conditional retrieval, an object storage system is provided in an embodiment of the present invention. Figure 1 It is the system structure diagram of an object storage system according to an embodiment of the present invention, as Figure 1 shown. The object storage system includes: a front-end service module 11, a text analysis scheduling module 12, a text analysis module 13, a text write message queue 14, a metadata update message queue 15, an index service module 16, and a key-value storage database system 17.

[0055] The front-end service module 11 is used to receive the text object written by the user, write the text object into the key-value storage database system 17, and add the write success message of the text object to the text write message queue 14. The front-end service module 11 can be layered into an object storage front-end service layer, which provides a web service for the object storage function interface and directly faces users. The front-end service module 11 provides an entry for writing text object files. In some other embodiments, a query entry can also be provided, as well as an entry for configuring the control data of the storage bucket (i.e., bucket).

[0056] The text analysis scheduling module 12 is used to consume the write success messages in the text write message queue 14, read text objects from the key-value storage database system 17 according to the write success messages, generate text analysis tasks for the text objects, and schedule the text analysis module 13 to execute the text analysis tasks. The text analysis scheduling module 12 is used to coordinate resources, schedule, and execute text analysis tasks for text objects in the entire system.

[0057] The text analysis module 13 is used to execute text analysis tasks and return the set of representative keywords obtained from the text analysis to the text analysis scheduling module 12. The text analysis module 13 is responsible for content analysis of text objects, such as performing NLP analysis to calculate the set of representative keywords for the content of the text objects.

[0058] The text analysis scheduling module 12 is also used to update the set of representative keywords as metadata of the text object to the key-value storage database system 17, and add the text object metadata update message to the metadata update message queue 15.

[0059] The index service module 16 is used to consume the metadata update message queue 15, read the metadata of the text object from the key-value storage database system 17 according to the text object metadata update message, construct the inverted index of the text object, persist the inverted index, and provide index services based on the inverted index.

[0060] The above-mentioned text write message queue 14 and metadata update message queue 15 are mainly used to provide message transfer. The queues support the producer and consumer models and are used to complete the update transfer of incremental data (i.e., each written text object).

[0061] Through the above object storage system, the text analysis scheduling module 12 and the text analysis module 13 are added to the object storage system in the related technology, and the index service module, the original computing power, and the message queue of the object storage system are utilized. When a user writes a new text object to the key-value storage database system 17, it triggers the text analysis of the text object to obtain the set of representative keywords of the text object, and updates it as the metadata of the text object to the key-value storage database system 17. When updating the metadata of the text object, it triggers the index service module to generate the inverted index of the text object and provide index services based on the inverted index, thus realizing the automatic construction of the text object index, and the constructed text object index can support complex condition retrieval.

[0062] In order to distinguish objects stored by different users, a management and control service module is also included in the object storage system. Figure 2 It is the system structure diagram of another object storage system according to an embodiment of the present invention. Compared with Figure 1 compared with Figure 2It also includes a control service module 18.

[0063] Figure 1 The object storage system shown supports text analysis and indexing of incremental text objects. Figure 2 The object storage system with the control service module 18 shown can also implement text analysis and indexing of all text objects of a user or all users.

[0064] In some of the embodiments, the object storage system implements text analysis of all text objects of a user.

[0065] Among them, the front-end service module 11 is further configured to receive metadata of a storage bucket, where each storage bucket is used to manage user data, and the user data includes text objects.

[0066] Among them, the control service module 18 is used to store metadata of storage buckets of different users.

[0067] Among them, the text analysis scheduling module 12 is further configured to read metadata of a user's storage bucket from the control service module 18, and read all text objects in the corresponding storage bucket from the key-value storage database system 17 according to the metadata of the storage bucket, and generate a text analysis task for all text objects.

[0068] Among them, after generating a text analysis task for all text objects, the text analysis scheduling module 12 will schedule the text analysis module 13 to execute the text analysis task. The text analysis module 13 executes the text analysis task and returns a set of representative keywords obtained from the text analysis to the text analysis scheduling module 12. The text analysis scheduling module 12 updates the set of representative keywords as metadata of the text object to the key-value storage database system 17, and adds a text object metadata update message to the metadata update message queue 15. The index service module 16 consumes the metadata update message queue 15, reads metadata of the text object from the key-value storage database system 17 according to the text object metadata update message, constructs an inverted index of the text object, persists the inverted index, and provides an index service based on the inverted index.

[0069] Among them, the inverted index is a text indexing technology. It takes each word in the text as a keyword and establishes a mapping from the keyword to the document, that is, records each word that appears in the document and points to the document containing the word. This indexing method is different from the traditional forward index. The forward index takes the document as the keyword and establishes a mapping from the document to the keyword. The inverted index can quickly locate the document containing a certain keyword. It can support advanced search functions such as multi-keyword query, phrase query, and fuzzy query, and can improve the retrieval efficiency and accuracy of the search engine.

[0070] The process of building an inverted index mainly includes the following steps: Step 1, tokenize the documents to extract each word in the documents. Step 2, create an inverted list for each word, recording the document numbers or document addresses that contain the word. Step 3, merge all the inverted lists into an inverted index table, with the word as the keyword and the list of documents containing the word as the value.

[0071] In some of these embodiments, by traversing the full-text objects of all users, text analysis and indexing of the full-text objects of all users are realized.

[0072] Among them, the text analysis scheduling module 12 is further configured to traverse all storage buckets according to the metadata of the storage buckets of all users provided by the management and control service module 18 to generate text analysis tasks for all the full-text objects in all the storage buckets.

[0073] Among them, after generating the text analysis tasks for all the full-text of all users, the text analysis scheduling module 12 will schedule the text analysis module 13 to execute the text analysis tasks. The text analysis module 13 executes the text analysis tasks and returns the set of representative keywords obtained from the text analysis to the text analysis scheduling module 12. The text analysis scheduling module 12 updates the set of representative keywords as the metadata of the text object to the key-value storage database system 17, and adds the text object metadata update message to the metadata update message queue 15. The index service module 16 consumes the metadata update message queue 15, reads the metadata of the text object from the key-value storage database system 17 according to the text object metadata update message, constructs the inverted index of the text object, persists the inverted index, and provides index services based on the inverted index.

[0074] Since the full-text analysis tasks involve a large amount of metadata updates, which may cause congestion in the metadata update message queue 15 and result in performance bottlenecks. In some other embodiments, when the text analysis scheduling module 12 performs text analysis for one user or all users, instead of sending the metadata update message for each text object to the metadata update message queue 15, after the metadata updates of all text objects of one user or all users are completed, a metadata update message is sent to the metadata update message queue 15, along with the identifier of the storage bucket or the identifier indicating that the metadata updates of all text objects of all users are completed. Thereafter, the index service module 16 consumes the metadata update message queue 15 to complete index construction and provide index services.

[0075] In some of these embodiments, the index service module 18 can also perform index construction and provide index services for all text objects of users in a timely manner without being triggered by metadata update messages.

[0076] Among them, the front-end service module 11 is further configured to receive the metadata of the storage bucket, where each storage bucket is used to manage user data, and the user data includes text objects.

[0077] Among them, the management and control service module 18 is used to store the metadata of the storage buckets of different users.

[0078] Among them, the index service module 16 is further configured to read the metadata of the user's storage bucket from the management and control service module 18, and read the metadata of all text objects in the corresponding storage bucket from the key-value storage database system 17 according to the metadata of the storage bucket, and construct an inverted index of all text objects.

[0079] Among them, after constructing the inverted index of all text objects, the index service module 16 persists the inverted index and provides an index service based on the inverted index.

[0080] In some embodiments, by traversing the metadata of all text objects of all users, an inverted index of all text objects of all users is constructed.

[0081] Among them, the index service module 16 is further configured to traverse all storage buckets according to the metadata of the storage buckets of all users provided by the management and control service module 18 to construct an inverted index of the metadata of all text objects in all storage buckets.

[0082] Among them, after constructing the inverted index of all text objects of all users, the index service module 16 persists the inverted index and provides an index service based on the inverted index.

[0083] In some embodiments, the text analysis module 13 performs text analysis based on natural language processing (NLP) technology.

[0084] Among them, the NLP technology includes but is not limited to the following methods:

[0085] Method 1: Keyword extraction based on word frequency: Count the frequency of each word in the text, and select the words with higher frequencies as keywords. This method is simple and easy to implement.

[0086] Method 2: Keyword extraction based on TF-IDF: TF-IDF is a commonly used text feature extraction method, which can measure the importance of a word in the text. By calculating the TF-IDF value of each word, select the words with higher TF-IDF values as keywords.

[0087] Method 3: Keyword extraction based on TextRank: TextRank is a graph-based ranking algorithm that can select the most important words as keywords by calculating the similarity and weight between words. This method can effectively avoid the influence of stop words, but has a high computational complexity.

[0088] Method 4: Keyword extraction based on deep learning: By training a neural network model (such as word2vec), keywords of the text are automatically extracted. This method requires a large amount of data and computing resources, and when the model is trained properly, it has a better text analysis effect among the four methods provided in this embodiment.

[0089] The key-value store database in this embodiment is a data storage system based on key-value pairs. It stores data as a set of key-value pairs, where each key is unique and corresponds to a value. Key-value store databases usually use data structures such as hash tables or B-trees to achieve fast data lookup and storage, and have characteristics such as high performance, high scalability, and high availability. Key-value store databases are usually used to store large-scale unstructured data, such as cache data, session data, log data, etc. It can support fast read and write operations and is suitable for application scenarios with high concurrency and high throughput. At the same time, key-value store databases can also improve data scalability and reliability through methods such as data sharding and data backup to ensure data security and availability.

[0090] In some of these embodiments, the key-value store database system 17 is a distributed system. In this embodiment, a distributed system is used to implement the key-value store database system, further improving the throughput and concurrency of the database system.

[0091] Based on the indexing service provided by the object storage system in this embodiment, users can achieve retrieval of complex conditions through the front-end service module 11.

[0092] In some of these embodiments, the front-end service module 11 is further configured to receive a query message from a user and send the query message to the indexing service module 16.

[0093] Among them, the indexing service module 16 is configured to segment the query message and obtain a query word set according to the segmentation result, use the indexing service to query the query result of the text object corresponding to the query word set in the user's storage bucket, and return the query result to the user.

[0094] Figure 3 is a schematic diagram of the working process of the object storage system of the embodiment of the present invention. Figure 3The text analysis and index construction of incremental text objects, the text analysis and index construction of full-text objects, and the query process of the index implemented by the object storage system based on the above embodiments are shown.

[0095] The text analysis and index construction of incremental text objects, the text analysis and index construction of full-text objects, and the query process of the index will be described below in conjunction with the method embodiments.

[0096] An embodiment of the present invention provides a text object indexing method based on the above object storage system. Figure 4 is a flowchart of the text object indexing method according to an embodiment of the present invention. This process mainly implements the text analysis and indexing process of incremental text objects. As Figure 4 shown, and with reference to Figure 3 at the same time, this process includes the following steps:

[0097] Step S401, the front-end service module receives the text object written by the user, writes the text object into the key-value storage database system, and adds the write success message of the text object to the text write message queue.

[0098] Step S402, the text analysis scheduling module consumes the write success message in the text write message queue, reads the text object from the key-value storage database system according to the write success message, generates a text analysis task for the text object, and schedules the text analysis module to execute the text analysis task.

[0099] Step S403, the text analysis module executes the text analysis task and returns the set of representative keywords obtained from the text analysis to the text analysis scheduling module.

[0100] Step S404, the text analysis scheduling module updates the set of representative keywords as the metadata of the text object to the key-value storage database system, and adds the text object metadata update message to the metadata update message queue.

[0101] Step S405, the index service module consumes the metadata update message queue, reads the metadata of the text object from the key-value storage database system according to the text object metadata update message, constructs an inverted index for the text object, persists the inverted index, and provides an index service based on the inverted index.

[0102] Through the above steps, when a user writes a new text object into the key-value storage database system, text analysis of the text object is triggered to obtain a set of representative keywords of the text object, which are updated as metadata of the text object to the key-value storage database system. When updating the metadata of the text object, the index service module is triggered to generate an inverted index of the text object and provide an index service based on the inverted index, thereby realizing the construction of an automated text object index, and the constructed text object index can support retrieval of complex conditions.

[0103] Figure 5 is a flowchart of the text object writing process according to an embodiment of the present invention, corresponding to the above step S401. Figure 5 The shown process mainly includes the following steps: Step 1, the user uploads text data using the object storage system API. Step 2, the object storage front-end service writes the text data into the KV distributed database. Step 3, the object storage front-end service generates an object write success message to the message queue.

[0104] Figure 6 is a flowchart of the incremental text analysis process of the text object according to an embodiment of the present invention, corresponding to the above step S402. Figure 6 The shown process mainly includes the following steps: Step 1, read the object write information from the message queue. Step 2, parse the object write information and read the object data. Step 3, generate a text analysis task. Step 4, schedule and distribute the task for execution.

[0105] Figure 7 is a flowchart of the incremental index construction process of the metadata of the text object according to an embodiment of the present invention, corresponding to the above step S405. Figure 7 The shown process mainly includes the following steps: Step 1, read the Object update information from the message queue. Step 2, read the Object metadata. Step 3, construct a multi-dimensional inverted index according to the object metadata information. Step 4, persist the inverted index to a fixed address. Step 5, the index service reads the inverted index for loading and then constructs the index service.

[0106] Refer to Figure 3 , in some of these embodiments, full-text analysis of the user's text object can also be performed. The text object index method further includes: the front-end service module receives the metadata of the storage bucket, wherein each storage bucket is used to manage user data, and the user data includes text objects. The management control service module stores the metadata of the storage buckets of different users. The text analysis scheduling module reads the metadata of the user's storage bucket from the management control service module, and reads all the text objects in the corresponding storage bucket from the key-value storage database system according to the metadata of the storage bucket, and generates a text analysis task for all the text objects.

[0107] In some of these embodiments, full-text analysis can also be performed on the text objects of all users. The text object indexing method further includes: the text analysis scheduling module traverses all storage buckets according to the metadata of the storage buckets of all users provided by the management and control service module to generate text analysis tasks for all full-text objects within all storage buckets.

[0108] Figure 8 is a flowchart of the full-text analysis process of the text objects in the embodiments of the present invention. As Figure 8 shown, this process mainly includes the following steps: Step 1, read the basic information of the bucket from the management and control service. Step 2, traverse all object data under the bucket. Step 3, read the object data. Step 4, generate text analysis tasks. Step 5, schedule and distribute the tasks for execution.

[0109] Refer to Figure 3 , in some of these embodiments, full-index construction can also be performed on the metadata of the text objects of users. The text object indexing method further includes: the front-end service module receives the metadata of the storage bucket, where each storage bucket is used to manage user data, and the user data includes text objects. The management and control service module stores the metadata of the storage buckets of different users. The indexing service module reads the metadata of the storage buckets of users from the management and control service module, and reads the metadata of all full-text objects within the corresponding storage buckets from the key-value storage database system according to the metadata of the storage buckets, and constructs an inverted index of all full-text objects.

[0110] In some of these embodiments, full-index construction can also be performed on the metadata of the text objects of all users. The text object indexing method further includes: the indexing service module traverses all storage buckets according to the metadata of the storage buckets of all users provided by the management and control service module to construct an inverted index of the metadata of all full-text objects within all storage buckets.

[0111] Figure 9 is a flowchart of the full-index construction process of the metadata of the text objects in the embodiments of the present invention. As Figure 9 shown, this process mainly includes the following steps: Step 1, read the bucket information. Step 2, traverse the bucket and read the object metadata. Step 3, construct a multi-dimensional inverted index according to the object metadata information. Step 4, persist the inverted index to a fixed address. Step 5, the indexing service reads the inverted index for loading and then constructs the indexing service.

[0112] In some of these embodiments, the text analysis module performs text analysis based on natural language processing technology.

[0113] Among them, NLP technologies include but are not limited to the following methods:

[0114] Method 1: Keyword extraction based on word frequency. Count the frequency of each word in the text and select the words with higher frequencies as keywords. This method is simple and easy to implement.

[0115] Method 2: Keyword extraction based on TF-IDF. TF-IDF is a commonly used text feature extraction method that can measure the importance of a word in the text. By calculating the TF-IDF values of each word, select the words with higher TF-IDF values as keywords.

[0116] Method 3: Keyword extraction based on TextRank. TextRank is a graph-based ranking algorithm that can select the most important words as keywords by calculating the similarity and weight between words. This method can effectively avoid the influence of stop words, but the computational complexity is relatively high.

[0117] Method 4: Keyword extraction based on deep learning. Automatically extract keywords from the text by training a neural network model. This method requires a large amount of data and computing resources. When the model is trained properly, it has better text analysis effect among the four methods provided in this embodiment.

[0118] In some of these embodiments, the key-value storage database system is a distributed system. In this embodiment, a distributed system is adopted to implement the key-value storage database system, which further improves the throughput and concurrency of the database system.

[0119] After constructing the inverted index of the text object and providing the index service based on the above methods, users can implement complex condition retrieval through the front-end service module. In some of these embodiments, the text object indexing method further includes: the front-end service module receives the user's query message and sends the query message to the index service module; the index service module performs word segmentation on the query message and obtains a query word set according to the word segmentation result, and uses the index service to query the query result of the text object corresponding to the query word set in the user's storage bucket, and returns the query result to the user.

[0120] Figure 10 is the flowchart of the user query process in the embodiment of the present invention, as Figure 10 shown, and at the same time refer to Figure 3, the process of user query mainly includes the following steps: Step 1, the user sends query information using the API. Step 2, the indexing service performs query analysis and word segmentation on the query information sent by the user to obtain a set of related word retrieval. Step 3, according to the set of related word retrieval, the indexing service is called to trigger an index query, and a set of associated object lists is obtained. Step 4, metadata information filtering and relevance calculation are performed on the set of object lists and then sorted. Step 5, the final result is returned to the user.

[0121] In the above embodiments of the present invention, on the basis of the existing object storage system, modules such as a text analysis module are added, and the indexing module is added or upgraded. On the basis of the existing object storage service function, an automated integrated text content indexing service is provided for users, which can reduce the development and operation and maintenance costs of users. At the same time, because the indexing service uses a text analysis method to generate rich metadata, the retrieval of complex conditions and complex text content data is also realized.

[0122] In addition, the above text object indexing method can be used as an optional service for users. Different users can choose to turn on or off the text object indexing service separately. For users, there is no need to develop and maintain the indexing service separately, but only need to pay attention to whether to turn on the above text object indexing service for their own users in the object storage system.

[0123] The embodiments of the present invention support the construction of text content indexing after data storage on the basis of integrating the user data storage solution, and provide a text content indexing query service, which can effectively solve the need for quickly retrieving data while storing large-scale data text, and reduce the reachable cost of users for big data retrieval.

[0124] The embodiments of the present invention also provide an electronic device, including: at least one processor; and a memory communicatively connected to the at least one processor. The above memory stores a computer program that can be executed by the at least one processor, and the above computer program is used to cause the electronic device to execute the method of the embodiments of the present invention when executed by the at least one processor.

[0125] The embodiments of the present invention also provide a non-transitory machine-readable medium storing a computer program, wherein the above computer program is used to cause the above computer to execute the method of the embodiments of the present invention when executed by a processor of the computer.

[0126] Reference Figure 11, the structural block diagram of an electronic device that can be a server or a client according to an embodiment of the present invention will now be described. It is an example of a hardware device that can be applied to various aspects of the present invention. The electronic device is intended to represent various forms of digital electronic computer devices, such as, laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as, personal digital processors, cellular phones, smart phones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present invention described herein and / or claimed.

[0127] As Figure 11 shown, the electronic device includes a computing unit 1101, which can perform various appropriate actions and processes according to a computer program stored in a read-only memory (ROM) 1102 or a computer program loaded from a storage unit 1108 into a random access memory (RAM) 1103. In the RAM 1103, various programs and data required for the operation of the electronic device can also be stored. The computing unit 1101, the ROM 1102, and the RAM 1103 are connected to each other through a bus 1104. An input / output (I / O) interface 1105 is also connected to the bus 1104.

[0128] A plurality of components in the electronic device are connected to the I / O interface 1105, including: an input unit 1106, an output unit 1107, a storage unit 1108, and a communication unit 1109. The input unit 1106 can be any type of device that can input information into the electronic device. The input unit 1106 can receive input digital or character information, and generate key signal inputs related to the user settings and / or function controls of the electronic device. The output unit 1107 can be any type of device that can present information, and can include but is not limited to a display, a speaker, a video / audio output terminal, a vibrator, and / or a printer. The storage unit 1108 can include but is not limited to a magnetic disk, an optical disk. The communication unit 1109 allows the electronic device to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks, and can include but is not limited to a modem, a network card, an infrared communication device, a wireless communication transceiver, and / or a chipset, such as a Bluetooth device, a WiFi device, a WiMax device, a cellular communication device, and / or the like.

[0129] The computing unit 1101 can be various general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of the computing unit 1101 include, but are not limited to, a CPU, a graphics processing unit (GPU), various dedicated artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, a digital signal processor (DSP), and any suitable processor, controller, microcontroller, etc. The computing unit 1101 executes the various methods and processes described above. For example, in some embodiments, the method embodiments of the present invention can be implemented as a computer program, which is tangibly contained in a machine-readable medium, such as the storage unit 1108. In some embodiments, part or all of the computer program can be loaded and / or installed onto the electronic device via the ROM 1102 and / or the communication unit 1109. In some embodiments, the computing unit 1101 can be configured to execute the above-described method in any other suitable manner (e.g., by means of firmware).

[0130] The computer program for implementing the method of the embodiments of the present invention can be written in any combination of one or more programming languages. These computer programs can be provided to the processor or controller of a general-purpose computer, a special-purpose computer, or other programmable data processing device, such that when the computer programs are executed by the processor or controller, the functions / operations specified in the flowchart and / or block diagram are implemented. The computer program can be executed entirely on the machine, partially on the machine, as an independent software package partially on the machine and partially on a remote machine, or entirely on a remote machine or server.

[0131] In the context of the embodiments of the present invention, the machine-readable medium can be a tangible medium that can contain or store a program for use by or in connection with an instruction execution system, apparatus, or device. The machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. The machine-readable signal medium can include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any suitable combination of the foregoing. More specific examples of the machine-readable storage medium would include an electrical connection based on one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0132] It should be noted that the term "including" and its variants used in the embodiments of the present invention are open-ended, that is, "including but not limited to". The term "based on" means "at least partially based on". The term "one embodiment" means "at least one embodiment"; the term "another embodiment" means "at least one additional embodiment"; the term "some embodiments" means "at least some embodiments". The modifications of "one" and "a plurality" mentioned in the embodiments of the present invention are illustrative rather than restrictive. Those skilled in the art should understand that unless clearly stated otherwise in the context, it should be understood as "one or more".

[0133] The user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data for analysis, stored data, displayed data, etc.) involved in the embodiments of the present invention are all information and data that have been authorized by the user or fully authorized by all parties. And the collection, use, and processing of relevant data need to comply with the relevant laws, regulations, and standards of the relevant countries and regions, and corresponding operation entrances are provided for users to choose to authorize or reject.

[0134] The various steps described in the method embodiments provided by the embodiments of the present invention can be executed in different orders and / or executed in parallel. In addition, the method embodiments may include additional steps and / or omit the steps shown. The protection scope of the present invention is not limited in this regard.

[0135] The term "embodiment" in this specification means that the specific features, structures, or characteristics described in combination with the embodiment may be included in at least one embodiment of the present invention. The phrase appears in various positions in the specification does not necessarily mean the same embodiment, nor does it mean being independent or alternative to other embodiments and mutually exclusive. The various embodiments in this specification are described in a related manner, and the same or similar parts between the various embodiments are referred to each other. In particular, for device, equipment, and system embodiments, since they are basically similar to method embodiments, the description is relatively simple, and the relevant parts refer to the partial description of the method embodiments.

[0136] The above-described embodiments only represent several implementation manners of the present invention, and their descriptions are relatively specific and detailed, but should not be construed as limiting the scope of patent protection. It should be noted that for those of ordinary skill in the art, without departing from the concept of the present invention, several modifications and improvements can still be made, and these all belong to the protection scope of the present invention. Therefore, the protection scope of the present invention should be subject to the appended claims.

Claims

1. A method for indexing text objects, comprising: The front-end service module receives a text object written by a user, writes the text object into a key-value storage database system, and adds a write success message of the text object to a text write message queue; The text analysis scheduling module consumes the write success message in the text write message queue, reads the text object from the key-value storage database system according to the write success message, generates a text analysis task for the text object, and schedules the text analysis module to execute the text analysis task; The text analysis module executes the text analysis task and returns a set of representative keywords obtained from the text analysis to the text analysis scheduling module; The text analysis scheduling module updates the set of representative keywords as metadata of the text object to the key-value storage database system, and adds a text object metadata update message to a metadata update message queue; The index service module consumes the metadata update message queue, reads the metadata of the text object from the key-value storage database system according to the text object metadata update message, constructs an inverted index of the text object, persists the inverted index, and provides an index service based on the inverted index.

2. The text object indexing method according to claim 1, wherein, The method for indexing text objects further includes: The front-end service module receives metadata of a storage bucket, where each storage bucket is used to manage user data, and the user data includes text objects; The text analysis scheduling module reads the metadata of the user's storage bucket from the management control service module, and reads all text objects in the corresponding storage bucket from the key-value storage database system according to the metadata of the storage bucket, and generates a text analysis task for all the text objects, where the management control service module stores the metadata of storage buckets of different users.

3. The text object indexing method according to claim 2, wherein, The method for indexing text objects further includes: The text analysis scheduling module traverses all storage buckets according to the metadata of storage buckets of all users provided by the management control service module to generate text analysis tasks for all text objects in all storage buckets.

4. The text object indexing method according to claim 1, wherein, The method for indexing text objects further includes: The front-end service module receives metadata of a storage bucket, where each storage bucket is used to manage user data, and the user data includes text objects; The index service module reads the metadata of the user's storage bucket from the management control service module, reads the metadata of all text objects in the corresponding storage bucket from the key-value storage database system according to the metadata of the storage bucket, and constructs an inverted index of all the text objects, where the management control service module stores the metadata of storage buckets of different users.

5. The text object indexing method according to claim 4, wherein The method for indexing text objects further includes: The index service module traverses all storage buckets according to the metadata of storage buckets of all users provided by the management control service module to construct an inverted index of the metadata of all text objects in all storage buckets.

6. The text object indexing method according to any one of claims 1 to 5, wherein The text analysis module performs text analysis based on natural language processing technology.

7. The text object indexing method according to any one of claims 1 to 5, wherein The method for indexing text objects further includes: The front-end service module receives the query message of the user and sends the query message to the index service module; The index service module performs word segmentation on the query message and obtains a query word set according to the word segmentation result, and uses the index service to query the query results of text objects corresponding to the query word set in the user's storage bucket, and returns the query results to the user.

8. An object storage system, comprising: A front-end service module, a text analysis scheduling module, a text analysis module, a text writing message queue, a metadata update message queue, an index service module, and a key-value storage database system; the object storage system is used to implement the text object indexing method according to any one of claims 1, 3, 5-7.

9. An object storage system, wherein, The object storage system includes a control service module, and the object storage system is also used to implement the text object indexing method according to claim 2 or 4.

10. An electronic device, comprising: A processor and a memory storing a program, wherein the program includes instructions that, when executed by the processor, cause the processor to execute the method according to any one of claims 1 to 7.

11. A non-transitory machine-readable medium storing computer instructions, wherein, The computer instructions are used to cause the computer to execute the method according to any one of claims 1 to 7.

Images