A method for accelerating batch file search model based

Abstract

The application discloses a method based on an accelerated batch file search model, which comprises file index and file metadata associated with the file index, and the permission control of the file index comprises directory space, tenant information, user information and file permission; the creation of the file index comprises tenant, file service and index service, the index service is created and updated in an asynchronous mode, in addition to corresponding attempt processing, a compensation mechanism is added for updating the missing or abnormal files at a fixed time. Through the file gathering to the unified file index, the application supports the file keyword query, realizes the low-delay file data update, supports the keyword segmentation search, the permission control, the file path search and multiple condition searches, is decoupled from the business file information metadata query, preview, editing and other businesses, pays attention to the unified summary of data, pays attention to the query efficiency, pays attention to the high coverage rate and precision of the query result and the permission control.

Description

Technical Field

[0001] This invention belongs to the field of file search technology, specifically relating to a method based on an accelerated batch file search model. Background Technology

[0002] In the current standardized management of the company, knowledge base storage is an extremely important standardized management link. Knowledge base management is used to record information and knowledge, which facilitates the team to accumulate experience, share resources, collaborate and manage security, form a complete knowledge system and continuously evolve.

[0003] Currently, a large number of the company's files are stored on the server terminal, resulting in scattered files and difficulties in retrieving unstructured data. Common file searches are for querying structured file metadata, while current Elasticsearch retrieval applications are mainly in log analysis and website blogs. Applying full-text search to knowledge base file storage is not frequently used.

[0004] The company's current knowledge base tools can manage company files and set access permissions through access control. However, file search is mainly achieved through fuzzy matching of file names. This means that the search results must accurately know the keywords contained in the file names; otherwise, the required files cannot be found. Summary of the Invention

[0005] To overcome the shortcomings of existing technologies, this invention provides a solution based on an accelerated batch file search model to address the problem that existing company knowledge bases cannot search for file matching ranges using file keywords.

[0006] A method based on an accelerated batch file search model includes a file index and file metadata associated with the file index. Access control of the file index is controlled through the association relationships of the file metadata, and this access control includes directory space, tenant information, user information, and file permissions.

[0007] The file metadata is used to store the file's name attribute, file type attribute, file size attribute, and address attribute of the associated file library;

[0008] The directory space is used to store metadata for tables, indexes, and other objects, including the table name, column names, column data types, and index name information;

[0009] The tenant information is used to control the physical isolation of file metadata and the logical isolation of document indexes;

[0010] The user information is used to control the ownership and permissions of documents;

[0011] The file permissions are used to control the file authorization process, authorization information, and external link sharing, and are the finest-grained permissions in the file index.

[0012] Furthermore, the creation of the file index includes tenants, file services, and index services. The index service is created and updated asynchronously. In addition to the corresponding trial processing, a compensation mechanism is added for abnormal update data, and missing or abnormal files are updated periodically.

[0013] Furthermore, the creation of the file index includes the following tasks:

[0014] Space maintenance: Data maintained in any space directory under the tenant information must be pushed to the index service through the file service;

[0015] File Upload: In addition to storing the file in the file storage space as usual, the file upload process also needs to store the file's metadata information and document content recognition information in the index service. The current operation is implemented asynchronously to avoid affecting the file storage process and to decouple it from the business.

[0016] Access control is a fine-grained control that sets the visibility range for a document or directory.

[0017] File recycling: This should also be reflected in the indexing service; recycled files will no longer be visible.

[0018] Furthermore, the file index creation operation method class is FileConsumer, which consumes document operations in different spatial directories asynchronously, and sends messages by identifying tenants, file identifiers, operation identifiers, etc. through FileProducer; the specific steps for creating the file index include:

[0019] S1. Based on the tenant ID, file metadata ID, and file editing type as input parameters, query the metadata and detailed information of files under different tenants;

[0020] S2. The file synchronization process is handled according to the file editing type strategy: When adding, first check if the index exists under the tenant, so that the index can be initialized during the first synchronization; when editing, not only should the content be updated, but the permission changes should also be synchronized to the index; when deleting, the document information in the index is physically deleted and not retained.

[0021] S3. If there is an error in the processing of document information in the index, it needs to be recorded in the database and the database needs to be retried before updating again.

[0022] Specifically, in the space maintenance, an empty space directory is meaningless to the indexing service; only space directories containing file entities can be added as attributes to the indexed document.

[0023] Furthermore, the file search permission control supports SaaS version multi-tenant permission isolation levels, including setting directory permissions, updating document permissions, setting document permissions, updating document permissions, and modifying document permissions;

[0024] The file contains implicit permission rules when it is created, the file's storage space contains the file's organizational scope permissions, and the file's source contains the file's ownership permissions.

[0025] By setting file permissions, you can restrict the organization and the people who can see and search the file. Setting organizational permissions not only updates the organization's information to the index service, but also updates the personnel information under the organization to the permission items in the index service.

[0026] Compared with the prior art, the present invention has the following advantages:

[0027] (1) By aggregating files into a unified file index server, file keyword queries are supported, low-latency file data updates are achieved, keyword segmentation search is supported, access control is supported, and multiple conditions such as file path search are supported. This solution is decoupled from business operations such as file information metadata query, preview, and editing, and focuses on unified data aggregation, query efficiency, high coverage and accuracy of query results, and access control.

[0028] (2) The dependence on business application systems is reduced, and the performance pressure on the database is relieved; scattered file data is aggregated, file search is portable and reusable, file search performance is improved, full-text search is applied to the knowledge base, a new search method is realized, and the accuracy of file search is achieved. Attached Figure Description

[0029] Figure 1 This is a block diagram of the model of the present invention;

[0030] Figure 2 Flowchart for creating an index for the documents of this invention;

[0031] Figure 3 This is a permission control diagram for document searching in this invention.

[0032] Figure 4 This is a data structure diagram in an embodiment of the present invention;

[0033] Figure 5 This is a class diagram illustrating the implementation of an embodiment of the present invention. Detailed Implementation

[0034] The invention will now be further described with reference to the accompanying drawings.

[0035] like Figure 1 As shown, a method based on an accelerated batch file search model includes a file index and file metadata associated with the file index. Access control of the file index is controlled through the association relationships of the file metadata, which includes directory space, tenant information, user information, and file permissions. Specifically, the file metadata stores file name attributes, file type attributes, file size attributes, and the address attributes of the associated file library; the directory space stores metadata for tables, indexes, and other objects, including table names, column names, column data types, and index name information; tenant information controls the physical isolation of file metadata and the logical isolation of document indexes; user information controls ownership permissions for documents; and file permissions control the file authorization process, authorization information, and external link sharing, representing the finest-grained permissions within the file index.

[0036] like Figure 2 As shown, the creation of a file index involves tenants, file services, and index services. The index service is created and updated asynchronously. In addition to handling failed updates, a compensation mechanism is added to periodically update missing or abnormal files. The creation of a file index includes the following tasks:

[0037] Space maintenance: Data maintained in any space directory under the tenant information must be pushed to the index service through the file service.

[0038] File Upload: In addition to storing the file in the file storage space as usual, the file upload process also needs to store the file's metadata information and document content identification information in the index service. The current operation is implemented asynchronously to avoid affecting the file storage process and to decouple it from the business.

[0039] Access control is a fine-grained control that sets the visibility range for a document or directory.

[0040] File recycling: This should also be reflected in the indexing service; recycled files will no longer be visible.

[0041] like Figure 3 As shown, file search permission control supports multi-tenant permission isolation levels in the SaaS version, including setting directory permissions, updating document permissions, setting document permissions, updating document permissions, and modifying document permissions. The permission rules are implicitly defined when a file is created; the file's storage space contains the file's organization-wide permissions; and the file's source contains the file's ownership permissions. By setting file permissions, files can be searched within their organization and for specific users. Organizational permission settings not only update the organization's information to the indexing service but also update the permissions entries for users within that organization in the indexing service.

[0042] Example

[0043] The business requirement is for the company to implement a global knowledge base search function in a multi-tenant scenario, limiting search based on the user's directory space and the space of the user's project or product, and performing a global search on document information set through access control.

[0044] like Figure 4 The diagram shows the data structure for internal company retrieval. The main method class for document index creation is `FileConsumer`, which consumes documents from different directories asynchronously. `FileProducer` identifies the tenant, file identifier, and operation identifier before sending messages.

[0045] like Figure 5 The implementation class diagram shown illustrates the specific steps involved in creating a file index:

[0046] (1) Based on the tenant ID, file metadata ID, and file editing type as input parameters, query the metadata and detailed information of files under different tenants;

[0047] (2) The file synchronization process is handled according to the file editing type strategy: When adding, first check whether there is an index under the tenant, so that the index can be initialized during the first synchronization; when editing, not only should the content be updated, but the change of permissions should also be synchronized to the index; when deleting, the document information in the index is physically deleted and not retained.

[0048] (3) If there is an abnormal situation in the processing of document information in the index, it is necessary to record it in the database and wait for a retry to update again.

[0049] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some or all of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method based on an accelerated batch file search model, characterized in that, This includes a file index and file metadata associated with the file index. Access control for the file index is controlled through the association relationships within the file metadata. This access control includes directory space, tenant information, user information, and file permissions. The file metadata is used to store the file's name attribute, file type attribute, file size attribute, and address attribute of the associated file library; The directory space is used to store metadata for tables, indexes, and other objects, including the table name, column names, column data types, and index name information; The tenant information is used to control the physical isolation of file metadata and the logical isolation of document indexes; The user information is used to control the ownership and permissions of documents; The file permissions are used to control the file authorization process, authorization information, and external link sharing, and are the finest-grained permissions in the file index; The creation of the file index includes tenants, file services, and index services. The index service is created and updated asynchronously. In addition to the corresponding trial processing, a compensation mechanism is added for abnormal update data, and missing or abnormal files are updated periodically. The access control supports multi-tenant access isolation levels for SaaS versions, including setting directory permissions, updating document permissions, setting document permissions, updating document permissions, and modifying document permissions. When a file is created, it contains implicit permission rules. The file's storage space contains the file's organization-wide permissions, and the file's source contains the file's ownership permissions. By setting file permissions, you can enable the file to be visible and searchable by the organization and the people within it. Setting organization permissions requires updating the organization's information in the index service, as well as updating the permission items of the people under the organization in the index service.

2. The method based on an accelerated batch file search model according to claim 1, characterized in that, The creation of the file index includes the following tasks: Space maintenance: Data maintained in any space directory under the tenant information must be pushed to the index service through the file service; File Upload: In addition to storing the file in the file storage space as usual, the file upload process also needs to store the file's metadata information and document content recognition information in the index service. The current operation is implemented asynchronously to avoid affecting the file storage process and to decouple it from the business. Access control is a fine-grained control that sets the visibility range for a document or directory. File recycling: This should also be reflected in the indexing service; recycled files will no longer be visible.

3. The method based on an accelerated batch file search model according to claim 2, characterized in that, The file index creation operation method class is FileConsumer, which consumes document operations in different space directories in an asynchronous manner, and uses FileProducer to identify tenants, file identifiers, and operation identifiers to send messages; The specific steps for creating the file index include: S1. Based on the tenant ID, file metadata ID, and file editing type as input parameters, query the metadata and detailed information of files under different tenants; S2. The file synchronization process is handled according to the file editing type strategy: When adding, first check if the index exists under the tenant, so that the index can be initialized during the first synchronization; when editing, not only should the content be updated, but the permission changes should also be synchronized to the index; when deleting, the document information in the index is physically deleted and not retained. S3. If there is an error in the processing of document information in the index, it needs to be recorded in the database and the database needs to be retried before updating again.

4. The method based on an accelerated batch file search model according to claim 2, characterized in that, In the space maintenance process, empty space directories are meaningless to the indexing service; only space directories containing file entities can be added as attributes to the indexed documents.

Images