Universal library building method, device and equipment and storage medium

Abstract

The invention relates to the technical field of computers, and discloses a general library building method, device and equipment and a storage medium, the method comprises the following steps: obtaining a basic workflow corresponding to a library building task, the basic workflow being used for representing a library building flow through a flow relationship between basic library building services, instantiating the basic workflow according to a library building service corresponding to the library building task to obtain a service workflow, the service workflow being used for representing a library building process of the library building service, and building an index library corresponding to the library building service according to the service workflow; according to the method and the device, the basic workflow for representing the general library building process is pre-configured, the business workflow is formed after the basic workflow is instantiated, and the business workflow is directly used for the library building task, so that the library building process can be modularized, the library building efficiency can be improved, and the user experience can be improved.

Description

Technical Field

[0001] The present invention relates to the field of computer technology, and in particular, to a general database building method, device, equipment and storage medium. Background Art

[0002] In recent years, with the rapid development of the Internet, the amount of web page data has exceeded tens of billions, and therefore, there are a wide variety of data services. In the prior art, the relevant database building methods are that manual workers use different methods for different types of data services to build databases, which has the defects of cumbersome work and low efficiency. Summary of the Invention

[0003] The main object of the present invention is to provide a general database building method, device, equipment and storage medium, aiming to solve the technical problem that in the prior art, the relevant database building methods are that manual workers use different methods for different types of data services to build databases, resulting in cumbersome work and low efficiency.

[0004] To achieve the above object, the present invention provides a general database building method, and the general database building method includes:

[0005] Obtain a basic workflow corresponding to the database building task, where the basic workflow is used to represent the database building process through the process relationship between basic database building services;

[0006] Instantiate the basic workflow according to the database building service instance corresponding to the database building task to obtain a service workflow, where the service workflow is used to represent the database building process of the database building service;

[0007] Build an index database corresponding to the database building service according to the service workflow.

[0008] Optionally, the building of the index database corresponding to the database building service according to the service workflow includes:

[0009] Obtain the configuration information of the database building service;

[0010] Build an index database corresponding to the database building service according to the configuration information and the service workflow.

[0011] Optionally, the building of the index database corresponding to the database building service according to the configuration information and the service workflow includes:

[0012] Modify the configuration information according to the user configuration operation to obtain modified configuration information;

[0013] Build an index database corresponding to the database building service according to the modified configuration information and the service workflow.

[0014] Optionally, the building of the index database corresponding to the database building service according to the service workflow includes:

[0015] Schedule the task execution script according to the service workflow, and execute the distributed parallel task through the task execution script to establish the index library corresponding to the library building service. When executing the distributed parallel task, parallelly execute the steps of reading the library building source data from the first distributed database, processing the library building source data to generate intermediate library building data, storing the intermediate library building data in the second distributed database, and generating index data and saving the index data to the third distributed database.

[0016] Optionally, the first distributed database includes HBase, the second distributed database includes Alluxio, and the third distributed database includes HDFS.

[0017] Optionally, the basic library building service includes a common library building service and a special library building service. The common library building service is used to represent the library building services required by multiple library building services, and the special library building service is used to represent the library building services required by special library building services. Before obtaining the basic workflow corresponding to the library building task, it further includes:

[0018] Split the multiple library building services required by the library building service into a common library building service and a special library building service;

[0019] Combine the common library building service and the special library building service to obtain the basic workflow.

[0020] Optionally, the combining the common library building service and the special library building service to obtain the basic workflow includes:

[0021] Receive the interaction operations of the user for the common library building service and the special library building service;

[0022] Combine the common library building service and the special library building service according to the interaction operations to obtain the basic workflow.

[0023] Optionally, after establishing the index library corresponding to the library building service according to the service workflow, it further includes:

[0024] Obtain the library building information during the library building process of the library building service;

[0025] Generate the library building progress and library building log of the library building service according to the library building information.

[0026] Optionally, after establishing the index library corresponding to the library building service according to the service workflow, it further includes:

[0027] When the library building fails, generate an alarm message according to the library building failure information;

[0028] Send the alarm information to the management terminal corresponding to the database building service.

[0029] Optionally, after establishing the index database corresponding to the database building service according to the service workflow, the method further includes:

[0030] After the database building is completed, test the index database to obtain a test result;

[0031] After the test result is passed, deploy the index database.

[0032] Optionally, the step of testing the index database after the database building is completed to obtain a test result includes:

[0033] After the database building is completed, select some data based on a preset gray rule;

[0034] Perform a gray test on the index database based on the part of the data to obtain a test result.

[0035] In addition, to achieve the above object, the present invention also proposes a general database building device, which includes:

[0036] An acquisition module, configured to acquire a basic workflow corresponding to a database building task, where the basic workflow is used to represent the database building process through the process relationship between basic database building services;

[0037] An instantiation module, configured to instantiate the basic workflow according to the database building service instance corresponding to the database building task to obtain a service workflow, where the service workflow is used to represent the database building process of the database building service;

[0038] A database building module, configured to establish an index database corresponding to the database building service according to the service workflow.

[0039] Optionally, the database building module is further configured to acquire configuration information of the database building service; establish an index database corresponding to the database building service according to the configuration information and the service workflow.

[0040] Optionally, the database building module is further configured to modify the configuration information according to user configuration operations to obtain modified configuration information; establish an index database corresponding to the database building service according to the modified configuration information and the service workflow.

[0041] Optionally, the database building module is further configured to schedule a task execution script according to the service workflow, and execute a distributed parallel task through the task execution script to build an index database corresponding to the database building service. When executing the distributed parallel task, steps of reading database building source data from a first distributed database in parallel, processing the database building source data to generate database building intermediate data, and storing the database building intermediate data in a second distributed database, and steps of generating index data and saving the index data to a third distributed database are executed in parallel.

[0042] Optionally, the first distributed database includes HBase, the second distributed database includes Alluxio, and the third distributed database includes HDFS.

[0043] Optionally, the basic database building service includes a common database building service and a special database building service. The common database building service is used to represent database building services required by multiple database building operations. The special database building service is used to represent database building services required by special database building operations. The general database building device further includes:

[0044] A workflow module, configured to split multiple database building services required for a database building operation into a common database building service and a special database building service; and combine the common database building service and the special database building service to obtain a basic workflow.

[0045] Optionally, the workflow module is further configured to receive an interaction operation of a user on the common database building service and the special database building service; and combine the common database building service and the special database building service according to the interaction operation to obtain a basic workflow.

[0046] In addition, to achieve the above object, the present invention further provides a general database building device, which includes a memory, a processor, and a general database building program stored on the memory and executable on the processor. The general database building program is configured to implement the general database building method as described above.

[0047] In addition, to achieve the above object, the present invention further provides a storage medium, on which a general database building program is stored. When the general database building program is executed by a processor, the general database building method as described above is implemented.

[0048] In the present invention, a basic workflow corresponding to a library building task is disclosed. The basic workflow is used to represent the library building process through the process relationship between basic library building services. The basic workflow is instantiated according to the library building business instance corresponding to the library building task to obtain a business workflow. The business workflow is used to represent the library building process of the library building business, and an index library corresponding to the library building business is established according to the business workflow. Since the basic workflow for representing the general library building process is pre-configured in the present invention, and after instantiation, the basic workflow forms a business workflow, and the business workflow is directly used for the library building task, the library building process can be modularized, the library building efficiency can be improved, and the user experience can be enhanced accordingly. BRIEF DESCRIPTION OF THE DRAWINGS

[0049] Figure 1 is a schematic structural diagram of a general library building device in the hardware operating environment involved in the solution of the embodiment of the present invention;

[0050] Figure 2 is a schematic flowchart of the first embodiment of the general library building method of the present invention;

[0051] Figure 3 is a general distributed library building architecture diagram of an embodiment of the general library building method of the present invention;

[0052] Figure 4 is an instantiation schematic diagram of an embodiment of the general library building method of the present invention;

[0053] Figure 5 is a library building execution schematic diagram of an embodiment of the general library building method of the present invention;

[0054] Figure 6 is a schematic flowchart of the second embodiment of the general library building method of the present invention;

[0055] Figure 7 is a platform scheduling schematic diagram of an embodiment of the general library building method of the present invention;

[0056] Figure 8 is a schematic flowchart of the third embodiment of the general library building method of the present invention;

[0057] Figure 9 is a structural block diagram of the first embodiment of the general library building device of the present invention.

[0058] The realization, functional features and advantages of the object of the present invention will be further described with reference to the embodiments and the accompanying drawings. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0059] It should be understood that the specific embodiments described herein are only used to explain the present invention and are not used to limit the present invention.

[0060] Refer to Figure 1 , Figure 1Schematic diagram of the general library building device structure for the hardware operating environment involved in the embodiment of the present invention.

[0061] As Figure 1 shown, the general library building device may include: a processor 1001, such as a central processing unit (CPU), a communication bus 1002, a user interface 1003, a network interface 1004, and a memory 1005. Among them, the communication bus 1002 is used to realize the connection and communication between these components. The user interface 1003 may include a display screen (Display), and optionally, the user interface 1003 may further include a standard wired interface and a wireless interface. For the wired interface of the user interface 1003, it may be a USB interface in the present invention. The network interface 1004 may optionally include a standard wired interface and a wireless interface (such as a wireless fidelity (Wi-Fi) interface). The memory 1005 may be a high-speed random access memory (Random Access Memory, RAM), or a stable memory (Non-volatile Memory, NVM), such as a disk memory. Optionally, the memory 1005 may also be a storage device independent of the aforementioned processor 1001.

[0062] Those skilled in the art can understand that Figure 1 the structure shown in

[0063] does not constitute a limitation on the general library building device, and may include more or fewer components than shown, or combine certain components, or have different component arrangements. Figure 1 As

[0064] shown, the memory 1005 regarded as a computer storage medium may include an operating system, a network communication module, a user interface module, and a general library building program. Figure 1 In the general library building device shown in

[0065] , the network interface 1004 is mainly used to connect to the background server and perform data communication with the background server; the user interface 1003 is mainly used to connect to the user device; the general library building device calls the general library building program stored in the memory 1005 through the processor 1001 and executes the general library building method provided by the embodiment of the present invention.

[0066] Based on the above hardware structure, an embodiment of the general library building method of the present invention is proposed. Figure 2 , Figure 2 is the flowchart of the first embodiment of the general library building method of the present invention, and the first embodiment of the general library building method of the present invention is proposed.

[0067] It should be understood that in recent years, with the rapid development of the Internet, the amount of web page data has exceeded tens of billions. To address this challenge, some search engines have adopted a unique web page hierarchical marking system. According to the importance of web pages, this system divides web pages into three levels, marked as 1, 2, and 3 respectively. Among them, 1 represents that the web page is particularly important, 2 is of secondary importance, and 3 is regarded as less important. In online search services, the system will first recall data from the level library marked as 1. When the recalled data from the level 1 library is insufficient, the system will recall data from the level 2 and 3 libraries in sequence. This hierarchical recall mechanism ensures the accuracy and efficiency of search results. The data volume of the level library is huge, and the update cycle is 15 days, that is, a round of the level library is built in a rolling manner in 15 days. In addition to the level library, some search engines have also introduced advanced technologies such as the daily library and the fast line to further improve the quality and speed of search services. Among them, the daily library is built for the web page data newly added every day, and the fast line is built for the real-time newly added data. In addition, there are some vertical search requirements, such as the document library, good doctor, image search, etc., covering a variety of data types and search requirements. It can be seen that there are a variety of data services in the actual library building process.

[0068] In the prior art, the relevant library building method is that manual work is carried out in different ways for different types of data services, which has the defects of cumbersome work and low efficiency.

[0069] Therefore, in order to overcome the above defects, a basic workflow for representing the general library building process is pre-configured in this embodiment. After the basic workflow is instantiated, a business workflow is formed, and the business workflow is directly used for the library building task, so as to modularize the library building process, improve the library building efficiency, and further improve the user experience.

[0070] In the first embodiment, the general library building method includes:

[0071] Step S10: Obtain the basic workflow corresponding to the library building task, and the basic workflow is used to represent the library building process through the process relationship between basic library building services.

[0072] It can be understood that the execution subject of this embodiment can be a general library building device with data processing, network communication, and program running functions, such as a computer, a server, etc., or other electronic devices that can implement the same or similar functions. This embodiment does not limit this.

[0073] In specific implementation, the general library building device can implement the above general library building method through a general distributed library building architecture. For the convenience of understanding, reference Figure 3 is made for illustration, but it does not limit the present invention. Figure 3This is a general distributed database building architecture diagram for an embodiment of the general database building method of the present invention. In the figure, the general distributed database building architecture includes a database building platform, a business management layer, a fly task scheduling center, a database building service layer, and a storage layer.

[0074] Among them, the database building platform can be used to provide a friendly user interface, enabling users to achieve the database building process through simple operations, reducing the time and labor costs for business access; the database building platform can also be used to manage multiple basic workflows, and users can select them at will according to their needs; the database building platform can also be used to set permissions for users, and each person can only see the tasks they have permissions for.

[0075] The business management layer can be used to uniformly manage each database building business according to the settings of the database building platform. Among them, the types of database building businesses include but are not limited to types such as full-scale big search, image search, document library, big search increment, good doctor, and sky library, and this embodiment does not limit this.

[0076] The fly task scheduling center can be used to schedule the database building service according to the basic workflow configured by the user on the database building platform.

[0077] The database building service layer includes various basic database building services (such as library scanning, general engine database building, vector database building, KV database building, and incremental database building, etc.). Among them, the basic database building services can be responsible for specific functions, can be deployed, extended, and tested separately, and the basic database building services can be obtained by splitting the various database building services required for the existing database building business. For example, the basic database building services include public database building services and special database building services. The public database building services are used to represent the database building services required by multiple database building businesses, and the special database building services are used to represent the database building services required by special database building businesses. In this embodiment, a unified database building framework is provided, and the database building services required for the existing database building business are split into basic database building services, enabling the database building process to be customized according to needs, improving the flexibility and customizability of the database building process.

[0078] The storage layer includes but is not limited to distributed databases such as HBase, Alluxio, and HDFS. Among them, HBase is used to store the source data for database building, Alluxio is used to store the intermediate data for database building, and HDFS is used to store the index data.

[0079] Of course, to implement other functions, in this embodiment, the general distributed database building architecture may further include other modules. For example, the CICD module: by automatically performing building, testing, and deployment after code submission, the testing and deployment process is greatly shortened; the monitoring and alarm module: real-time monitoring of the database building progress and the database building completion rate, and when the database building fails, sending an alarm to the person in charge; the log module: the front-end page collects the database building logs in real time and displays them to the user; the reusable component module: by using encapsulated modules to implement the same functions, facilitating the quick access of the business; the gray-scale database building module: before releasing a new service or changing an existing service, small-batch data gray-scale testing can be performed, and this embodiment does not limit this.

[0080] It should be noted that the database building task can be initiated by the user through the database building platform or automatically initiated by the general database building device according to the requirements, and this embodiment does not limit this. Similarly, the basic workflow corresponding to the database building task can be specified by the user through the database building platform or selected by the general database building device according to the database building task, and this embodiment does not limit this.

[0081] The basic workflow may include a common database building service and a special database building service. The common database building service is used to represent the database building services required by multiple database building operations, and the special database building service is used to represent the database building services required by special database building operations. In this embodiment, the basic workflow can be obtained by combining the common database building service and the special database building service.

[0082] Step S20: Instantiate the basic workflow according to the database building operation corresponding to the database building task to obtain a business workflow, and the business workflow is used to represent the database building process of the database building operation.

[0083] For ease of understanding, reference Figure 4 is made for illustration, but it does not limit the present invention. Figure 4 This is an instantiation schematic diagram of an embodiment of the general database building method of the present invention. In the figure, module is the basic module (i.e., the basic database building service). Assume that the basic workflow corresponding to the database building task specified by the user is module A → module B → module C, and the database building operations corresponding to the database building task are operation 1 and operation 2. Then, the process of instantiating the basic workflow according to the database building operations corresponding to the database building task to obtain the business workflow is as Figure 4 shown, where the business workflow can be directly used for the database building task.

[0084] Step S30: Establish an index database corresponding to the database building operation according to the business workflow.

[0085] It can be understood that the index library corresponding to the database building service established according to the business workflow can be the index library corresponding to the database building service established by calling the task execution script according to the business workflow. The task execution script can be preset in this embodiment, and this embodiment does not limit it.

[0086] In this embodiment, a basic workflow corresponding to the database building task is disclosed. The basic workflow is used to represent the database building process through the process relationship between basic database building services. The basic workflow is instantiated according to the database building service corresponding to the database building task to obtain a business workflow. The business workflow is used to represent the database building process of the database building service. An index library corresponding to the database building service is established according to the business workflow. Since the basic workflow for representing the general database building process is pre-configured in this embodiment, and after the basic workflow is instantiated, a business workflow is formed. The business workflow is directly used for the database building task, so that the database building process can be modularized, the database building efficiency can be improved, and the user experience can be enhanced.

[0087] Further, in order to accelerate the database building speed, step S30 includes: scheduling a task execution script according to the business workflow, and establishing an index library corresponding to the database building service by executing a distributed parallel task through the task execution script. When executing the distributed parallel task, steps of reading database building source data from a first distributed database in parallel, processing the database building source data to generate database building intermediate data, and storing the database building intermediate data in a second distributed database, and generating index data and saving the index data to a third distributed database are executed in parallel.

[0088] In a specific implementation, the distributed parallel task can be a MapReduce task. The MapReduce task can be used to process large-scale data sets and has good parallel computing capabilities. By starting the MapReduce task through the task execution script, its parallel processing capabilities can be utilized to accelerate the database building speed.

[0089] Further, in order to make full use of the optimized performance of various types of distributed databases in each stage of data reading, processing, and storage to accelerate the entire database building process. In this embodiment, through the combination and utilization of different storage and computing components, the data flow can flow efficiently in the system, improving the overall efficiency of the database building task. In a specific implementation, the first distributed database includes HBase, the second distributed database includes Alluxio, and the third distributed database includes HDFS.

[0090] It can be understood that HBase is a column-oriented distributed database with fast random read / write performance. Storing the database construction source data in HBase can obtain the required data through fast query operations, improving the speed of reading the database construction source data; Alluxio is a distributed memory file system with data stored in memory, so it has extremely high access speed. Storing the intermediate data of database construction in Alluxio can accelerate the read / write operations of the intermediate data of database construction, improving the efficiency of data processing; HDFS is a distributed file system with high reliability and high durability. Storing the index data in HDFS can ensure the secure storage of data and provide support for long-term storage at the same time.

[0091] For the sake of easy understanding, reference is made to Figure 5 for illustration, but it does not limit the present invention. Figure 5 FIG. is a schematic diagram of the execution of database construction for an embodiment of the general database construction method of the present invention. In the figure, the user submits a database construction task through the database construction platform. The database construction platform schedules and executes the task execution script according to the basic workflow selected by the user. The script starts the MapReduce task for database construction operations. When the MapReduce task executes, it reads the source data from HBase and stores the intermediate data of database construction in Alluxio, and the index data is saved to HDFS.

[0092] Further, for the sake of facilitating the user to timely understand the execution situation of the database construction task, after the step S30, it further includes: obtaining the database construction information during the database construction process of the database construction service; generating the database construction progress and database construction log of the database construction service according to the database construction information.

[0093] Further, for the sake of facilitating the user to timely handle problems and ensure the smooth progress of the database construction task, after the step S30, it further includes: when the database construction fails, generating an alarm message according to the database construction failure information; sending the alarm message to the management terminal corresponding to the database construction service.

[0094] For the sake of easy understanding, reference is made to Figure 5 for illustration, but it does not limit the present invention. Figure 5This is a schematic diagram of the execution of the database building process for an embodiment of the general database building method of the present invention. In the figure, during the task execution process, the database building platform will receive and record the execution status and logs of the task in real time. Once a task in a certain stage fails, the system will automatically send an alarm notification to the person in charge. Users can observe the task execution progress, task running logs and other information in real time through the database building platform, and can view historical log records. In addition, the database building platform also provides a friendly user interface, enabling users to easily manage and monitor the execution status of tasks. In addition, the database building platform also has high scalability and flexibility, and can be customized and extended according to the needs of users. At the same time, the database building platform also provides rich data analysis and visualization tools, enabling users to better understand and analyze the data during the task execution process.

[0095] The database building platform provides users with an efficient, reliable and professional database building solution, which can meet the needs and expectations of different users.

[0096] Further, in order to shorten the test deployment process, after step S30, the following steps are also included: after the database building is completed, the index library is tested to obtain a test result; after the test result is a pass, the index library is deployed.

[0097] It can be understood that in this embodiment, the general distributed database building architecture may further include a CICD module, and the CICD module can be used for automated testing. By automatically performing construction, testing and deployment after the code is submitted, the test deployment process is greatly shortened.

[0098] Further, in order to improve the test efficiency, the step of testing the index library to obtain a test result after the database building is completed includes: after the database building is completed, a part of the data is selected based on a preset gray scale rule; the index library is subjected to gray scale testing based on the part of the data to obtain a test result.

[0099] It should be understood that in this embodiment, the general distributed database building architecture may further include a gray scale database building module: before a new service is released or an existing service is changed, small batch data gray scale testing can be performed through the gray scale database building module.

[0100] Refer to Figure 6 , Figure 6 This is a schematic flow diagram of the second embodiment of the general database building method of the present invention. Based on the first embodiment shown above Figure 2 a second embodiment of the general database building method of the present invention is proposed.

[0101] In the second embodiment, step S30 includes:

[0102] Step S301: Obtain the configuration information of the database building service.

[0103] It should be understood that, in order to ensure the accuracy and efficiency of the database building task, in this embodiment, an index database corresponding to the database building service may also be established according to the configuration information of the database building service and the service workflow.

[0104] For ease of understanding, reference is made to Figure 7 for illustration, but the present invention is not limited thereto. Figure 7 FIG. is a schematic diagram of platform scheduling for an embodiment of the general database building method of the present invention. In the figure, the configuration center is responsible for maintaining the configuration information of each database building service. The configuration information includes, but is not limited to, service configuration, MapReduce parameter configuration, etc. These configuration information provide necessary parameters and strategies for task execution, ensuring the accuracy and efficiency of the database building task.

[0105] Step S302: Establish an index database corresponding to the database building service according to the configuration information and the service workflow.

[0106] It can be understood that establishing an index database corresponding to the database building service according to the configuration information and the service workflow may be to obtain task execution parameters and task execution strategies from the configuration information, and establish an index database corresponding to the database building service according to the task execution parameters, task execution strategies, and service workflow.

[0107] In this embodiment, an index database corresponding to the database building service is established according to the configuration information of the database building service and the service workflow, thereby ensuring the accuracy and efficiency of the database building task.

[0108] Further, in this embodiment, the database building configuration is externalized to the database building platform, and the user can conveniently modify the database building configuration on the database building platform to meet different service requirements. The step S302 includes: modifying the configuration information according to the user configuration operation to obtain the modified configuration information; establishing an index database corresponding to the database building service according to the modified configuration information and the service workflow.

[0109] Referring to Figure 8 , Figure 8 FIG. is a schematic flowchart of the third embodiment of the general database building method of the present invention. Based on the above embodiments, the third embodiment of the general database building method of the present invention is proposed.

[0110] In the third embodiment, the basic database building service includes a common database building service and a special database building service. The common database building service is used to represent the database building services required by multiple database building services, and the special database building service is used to represent the database building services required by special database building services. Before the step S10, it further includes:

[0111] Step S01: Split the multiple database building services required by the database building service into a common database building service and a special database building service.

[0112] It should be understood that in order to solve the problems in the prior art, such as a large variety of library types, cumbersome and inefficient manual operation and maintenance work, a long deployment process for new services that requires multiple steps of testing and verification, insufficient flexibility in the library building process, making it difficult to execute from a certain intermediate stage, which results in a long testing process and the need for multiple validations and adjustments, and manual deployment of the environment and testing for new business access, leading to slow business access and long iteration cycles.

[0113] In this embodiment, a general library building method is disclosed to achieve centralized management of services, configurations, and businesses, reduce the time and labor costs of business access, implement modular design and unit testing, and improve the efficiency and maintainability of the library building process.

[0114] To achieve the above object, in this embodiment, first, various library building services required for the library building business are split into modules and configuration extraction is performed to abstract out common modules (i.e., common library building services), and for the special logic of a certain library type, personalized modules (i.e., special library building services) are set; secondly, these modules are combined to form a basic workflow, and after the basic workflow is instantiated, a business workflow is formed, and the business workflow is directly used for the library building task.

[0115] Step S02: Combine the common library building service and the special library building service to obtain a basic workflow.

[0116] Further, the step S02 includes: receiving an interaction operation of the user for the common library building service and the special library building service; combining the common library building service and the special library building service according to the interaction operation to obtain a basic workflow.

[0117] For the sake of understanding, reference is made to Figure 7 for illustration, but it does not limit the present invention. Figure 7 It is a platform scheduling schematic diagram of an embodiment of the general library building method of the present invention. In the figure, the user configures basic modules on the library building platform, and each basic module includes the following parameters: module name, absolute path of script execution, concurrency, number of retries, dynamic concurrency policy, and configuration file. These basic modules can form a basic workflow, and the user can configure them by dragging on the page. When the basic workflow is instantiated, a business workflow will be formed, and the business workflow is directly used for the library building task.

[0118] When a user starts a task on the database building platform, the task scheduling center will execute the task according to the specified business workflow. The configuration center is responsible for maintaining the configuration information of each database building service, including business configuration, MapReduce parameter configuration, etc. This configuration information provides the necessary parameters and policies for task execution, ensuring the accuracy and efficiency of the database building task. In addition, the database building platform also provides rich monitoring and logging functions, allowing users to observe the execution status, progress, and log information of tasks in real time. Once a task at a certain stage fails, the system will automatically send an alarm notification to ensure that problems can be solved in a timely manner.

[0119] The user configures the basic module, starts the task, and the functions of the task scheduling center and the configuration center on the database building platform. These functions together constitute a complete database building solution, providing users with efficient, reliable, and professional services.

[0120] In this embodiment, the database building workflow is configured by the user's dragging method, the database building configuration is externalized to the platform, and the module is split to achieve business logic reuse, and the database building progress and logs are monitored in real time to ensure the accuracy and efficiency of the database building task.

[0121] Through the above optimization scheme, this embodiment can efficiently manage a large number of database types and multiple database building environments, significantly reducing the workload of manual operation and maintenance, reducing the time and labor costs of business access, realizing modular design and unit testing, shortening the deployment and testing processes of new services, and improving the availability and stability of the system.

[0122] In addition, in this embodiment, the database building service is split into multiple modules by the module splitting method, and each module implements specific business logic. This enables different services to reuse the same logic, thus greatly reducing the amount of code to be deployed.

[0123] In addition, referring to Figure 9 , an embodiment of the present invention also proposes a general database building device, and the general database building device includes:

[0124] An obtaining module 10, configured to obtain a basic workflow corresponding to a database building task, where the basic workflow is used to represent the database building process through the process relationship between basic database building services.

[0125] In a specific implementation, the general database building device can implement the above general database building method through a general distributed database building architecture. For ease of understanding, reference is made to Figure 3 for description, but it does not limit the present invention. Figure 3 FIG. is a diagram of a general distributed database building architecture for an embodiment of the general database building method of the present invention. In the figure, the general distributed database building architecture includes a database building platform, a business management layer, a fly task scheduling center, a database building service layer, and a storage layer.

[0126] Among them, the database building platform can be used to provide a friendly user interface, enabling users to implement the database building process through simple operations, reducing the time and labor costs for business access; the database building platform can also be used to manage multiple basic workflows, and users can select them at will according to their needs; the database building platform can also be used to set permissions for users, and each person can only see the tasks they have permissions for.

[0127] The business management layer can be used to uniformly manage each database building business according to the settings of the database building platform. Among them, the types of database building businesses include but are not limited to full-scale large search, image search, document library, large search increment, good doctor, and sky library, etc. This embodiment does not limit this.

[0128] The fly task scheduling center can be used to schedule the database building service according to the basic workflow configured by the user on the database building platform.

[0129] The database building service layer includes a variety of basic database building services (such as library scanning, general engine database building, vector database building, KV database building, and incremental database building, etc.). Among them, the basic database building services can be responsible for specific functions, can be deployed, extended, and tested separately. The basic database building services can be obtained by splitting the various database building services required for the existing database building business. For example, the basic database building services include public database building services and special database building services. The public database building services are used to represent the database building services required by multiple database building businesses, and the special database building services are used to represent the database building services required by special database building businesses. In this embodiment, a unified database building framework is provided, and the database building services required for the existing database building business are split into basic database building services, so that the database building process can be customized according to needs, improving the flexibility and customizability of the database building process.

[0130] The storage layer includes but is not limited to distributed databases such as HBase, Alluxio, and HDFS. Among them, HBase is used to store the source data for database building, Alluxio is used to store the intermediate data for database building, and HDFS is used to store index data.

[0131] Of course, in order to implement other functions, in this embodiment, the general distributed database building architecture can also include other modules. For example, the CICD module: automatically performs building, testing, and deployment after code submission, greatly shortening the test deployment process; the monitoring and alarm module: real-time monitors the database building progress and the database building completion rate, and sends an alarm to the person in charge when the database building fails; the log module: the front-end page collects the database building logs in real time and displays them to users; the reusable component module: realizes the same function by using encapsulated modules, facilitating the quick access of the business; the gray-scale database building module: before releasing a new service or changing an existing service, small-batch data gray-scale testing can be performed. This embodiment does not limit this.

[0132] It should be noted that the library building task can be initiated by the user through the library building platform or automatically initiated by the general library building device according to the requirements. This embodiment does not limit this. Similarly, the basic workflow corresponding to the library building task can be specified by the user through the library building platform or selected by the general library building device according to the library building task. This embodiment does not limit this.

[0133] The basic workflow may include a common library building service and a special library building service. The common library building service is used to represent the library building services required by multiple library building operations, and the special library building service is used to represent the library building services required by special library building operations. In this embodiment, the basic workflow can be obtained by combining the common library building service and the special library building service.

[0134] The instantiation module 20 is configured to instantiate the basic workflow according to the library building operation corresponding to the library building task to obtain a service workflow, and the service workflow is used to represent the library building process of the library building operation.

[0135] For ease of understanding, reference is made to Figure 4 for illustration, but it does not limit the present invention. Figure 4 FIG. is an instantiation schematic diagram of an embodiment of the general library building method of the present invention. In the figure, module is the basic module (i.e., the basic library building service). Assume that the basic workflow corresponding to the library building task specified by the user is module A → module B → module C, and the library building operations corresponding to the library building task are operation 1 and operation 2. Then, the process of instantiating the basic workflow according to the library building operations corresponding to the library building task to obtain the service workflow is as Figure 4 shown, where the service workflow can be directly used for the library building task.

[0136] The library building module 30 is configured to establish an index library corresponding to the library building operation according to the service workflow.

[0137] It can be understood that establishing an index library corresponding to the library building operation according to the service workflow may be to call a task execution script according to the service workflow to establish an index library corresponding to the library building operation. Among them, the task execution script can be preset. This embodiment does not limit this.

[0138] In this embodiment, a basic workflow corresponding to a database building task is disclosed. The basic workflow is used to represent the database building process through the process relationship between basic database building services. The basic workflow is instantiated according to the database building business instance corresponding to the database building task to obtain a business workflow. The business workflow is used to represent the database building process of the database building business, and an index database corresponding to the database building business is established according to the business workflow. Since a basic workflow for representing a general database building process is pre-configured in this embodiment, and the basic workflow forms a business workflow after instantiation, and the business workflow is directly used for the database building task, the database building process can be modularized, the database building efficiency can be improved, and thus the user experience can be enhanced.

[0139] For other embodiments or specific implementation manners of the general database building device of the present invention, reference may be made to the above method embodiments, which will not be elaborated here.

[0140] In addition, an embodiment of the present invention also proposes a storage medium, on which a general database building program is stored. When the general database building program is executed by a processor, the general database building method as described above is implemented.

[0141] It should be noted that in this article, the term "including", "comprising" or any other variant thereof is intended to cover non-exclusive inclusion, so that a process, method, article or system including a series of elements not only includes those elements, but also includes other elements not expressly listed, or further includes elements inherent to such process, method, article or system. Without further limitation, an element defined by the statement "including a..." does not exclude the existence of another identical element in the process, method, article or system including the element.

[0142] The serial numbers of the above embodiments of the present invention are only for description and do not represent the advantages or disadvantages of the embodiments.

[0143] Through the description of the above embodiments, those skilled in the art can clearly understand that the above embodiment methods can be implemented by means of software plus a necessary general hardware platform. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation manner. Based on such an understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product. The computer software product is stored in a storage medium (such as a Read Only Memory image (ROM) / Random Access Memory (RAM), magnetic disk, optical disc), and includes several instructions for causing a terminal device (which may be a mobile phone, a computer, a server, or a network device, etc.) to execute the methods described in the various embodiments of the present invention.

[0144] The above are only the preferred embodiments of the present invention, and do not limit the patent scope of the present invention accordingly. Any equivalent structure or equivalent process transformation made by using the specification and drawings of the present invention, or directly or indirectly applied in other related technical fields, shall be included in the patent protection scope of the present invention by the same token.

[0145] The present invention discloses A1. A general library building method, the general library building method comprising:

[0146] Obtaining a basic workflow corresponding to the library building task, the basic workflow being used to represent the library building process through the process relationship between basic library building services;

[0147] Instantiating the basic workflow according to the library building business instance corresponding to the library building task to obtain a business workflow, the business workflow being used to represent the library building process of the library building business;

[0148] Establishing an index library corresponding to the library building business according to the business workflow.

[0149] A2. The general library building method as described in A1, wherein establishing the index library corresponding to the library building business according to the business workflow comprises:

[0150] Obtaining the configuration information of the library building business;

[0151] Establishing an index library corresponding to the library building business according to the configuration information and the business workflow.

[0152] A3. The general library building method as described in A2, wherein establishing the index library corresponding to the library building business according to the configuration information and the business workflow comprises:

[0153] Modifying the configuration information according to the user configuration operation to obtain modified configuration information;

[0154] Establishing an index library corresponding to the library building business according to the modified configuration information and the business workflow.

[0155] A4. The general library building method as described in A1, wherein establishing the index library corresponding to the library building business according to the business workflow comprises:

[0156] Scheduling a task execution script according to the business workflow, and executing a distributed parallel task through the task execution script to establish an index library corresponding to the library building business. When executing the distributed parallel task, steps of reading library building source data from a first distributed database in parallel, processing the library building source data to generate library building intermediate data, and storing the library building intermediate data in a second distributed database, and generating index data and saving the index data to a third distributed database are executed in parallel.

[0157] A5. The general database building method as described in A4, wherein the first distributed database includes HBase, the second distributed database includes Alluxio, and the third distributed database includes HDFS.

[0158] A6. The general database building method as described in any one of A1 to A5, wherein the basic database building service includes a common database building service and a special database building service. The common database building service is used to represent the database building services required for multiple database building operations, and the special database building service is used to represent the database building services required for special database building operations. Before obtaining the basic workflow corresponding to the database building task, it further includes:

[0159] Splitting the multiple database building services required for the database building operation into a common database building service and a special database building service;

[0160] Combining the common database building service and the special database building service to obtain a basic workflow.

[0161] A7. The general database building method as described in A6, wherein combining the common database building service and the special database building service to obtain a basic workflow includes:

[0162] Receiving an interaction operation from the user for the common database building service and the special database building service;

[0163] Combining the common database building service and the special database building service according to the interaction operation to obtain a basic workflow.

[0164] A8. The general database building method as described in any one of A1 to A5, after establishing the index database corresponding to the database building operation according to the business workflow, it further includes:

[0165] Obtaining the database building information during the database building process of the database building operation;

[0166] Generating the database building progress and database building log of the database building operation according to the database building information.

[0167] A9. The general database building method as described in any one of A1 to A5, after establishing the index database corresponding to the database building operation according to the business workflow, it further includes:

[0168] When the database building fails, generating an alarm message according to the database building failure information;

[0169] Sending the alarm message to the management terminal corresponding to the database building operation.

[0170] A10. The general database building method as described in any one of A1 to A5, after establishing the index database corresponding to the database building operation according to the business workflow, it further includes:

[0171] After the database building is completed, test the index database to obtain test results;

[0172] After the test result is a pass, deploy the index database.

[0173] A11. The general database building method as described in A10, where after the database building is completed, testing the index database to obtain test results includes:

[0174] After the database building is completed, select some data based on a preset gray scale rule;

[0175] Perform a gray scale test on the index database based on the partial data to obtain test results.

[0176] The present invention also discloses B12. A general database building device, the general database building device includes:

[0177] An acquisition module, configured to acquire a basic workflow corresponding to a database building task, where the basic workflow is used to represent a database building process through a process relationship between basic database building services;

[0178] An instantiation module, configured to instantiate the basic workflow according to a database building business instance corresponding to the database building task to obtain a business workflow, where the business workflow is used to represent a database building process of the database building business;

[0179] A database building module, configured to build an index database corresponding to the database building business according to the business workflow.

[0180] B13. The general database building device as described in B12, where the database building module is further configured to acquire configuration information of the database building business; build an index database corresponding to the database building business according to the configuration information and the business workflow.

[0181] B14. The general database building device as described in B13, where the database building module is further configured to modify the configuration information according to a user configuration operation to obtain modified configuration information; build an index database corresponding to the database building business according to the modified configuration information and the business workflow.

[0182] B15. The general database building device as described in B12, where the database building module is further configured to schedule a task execution script according to the business workflow, and execute a distributed parallel task through the task execution script to build an index database corresponding to the database building business. When executing the distributed parallel task, the steps of reading database building source data from a first distributed database in parallel, processing the database building source data to generate database building intermediate data, and storing the database building intermediate data in a second distributed database, and generating index data and saving the index data to a third distributed database are executed in parallel.

[0183] B16. The general database building device as described in B15, wherein the first distributed database includes HBase, the second distributed database includes Alluxio, and the third distributed database includes HDFS.

[0184] B17. The general database building device as described in any one of B12 to B16, wherein the basic database building service includes a common database building service and a special database building service. The common database building service is used to represent the database building services required by multiple database building operations, and the special database building service is used to represent the database building services required by special database building operations. The general database building device further includes:

[0185] A workflow module, configured to split the multiple database building services required for a database building operation into a common database building service and a special database building service; and combine the common database building service and the special database building service to obtain a basic workflow.

[0186] B18. The general database building device as described in B17, wherein the workflow module is further configured to receive an interactive operation of a user on the common database building service and the special database building service; and combine the common database building service and the special database building service according to the interactive operation to obtain a basic workflow.

[0187] The present invention also discloses C19. A general database building device, including: a memory, a processor, and a general database building program stored on the memory and executable on the processor. When the general database building program is executed by the processor, it implements the general database building method as described above.

[0188] The present invention also discloses D20. A storage medium, on which a general database building program is stored. When the general database building program is executed by a processor, it implements the general database building method as described above.

Claims

1. A general library construction method, characterized in that, The general database building method includes: Obtaining a basic workflow corresponding to the database building task, where the basic workflow is used to represent the database building process through the process relationship between basic database building services; Instantiating the basic workflow according to the database building business instance corresponding to the database building task to obtain a business workflow, where the business workflow is used to represent the database building process of the database building business; Establishing an index database corresponding to the database building business according to the business workflow.

2. The general database building method according to claim 1, wherein The establishing an index database corresponding to the database building business according to the business workflow includes: Obtaining the configuration information of the database building business; Establishing an index database corresponding to the database building business according to the configuration information and the business workflow.

3. The general database building method according to claim 2, characterized in that The establishing an index database corresponding to the database building business according to the configuration information and the business workflow includes: Modifying the configuration information according to the user configuration operation to obtain the modified configuration information; Establishing an index database corresponding to the database building business according to the modified configuration information and the business workflow.

4. The general library building method according to claim 1, characterized in that, The establishing an index database corresponding to the database building business according to the business workflow includes: Scheduling a task execution script according to the business workflow, and executing a distributed parallel task through the task execution script to establish an index database corresponding to the database building business. When executing the distributed parallel task, the steps of reading database building source data from a first distributed database in parallel, processing the database building source data to generate database building intermediate data, and storing the database building intermediate data in a second distributed database, and generating index data and saving the index data to a third distributed database are executed in parallel.

5. The general library building method according to claim 4, wherein The first distributed database includes HBase, the second distributed database includes Alluxio, and the third distributed database includes HDFS.

6. The general library building method according to any one of claims 1 to 5, characterized in that The basic database building services include common database building services and special database building services. The common database building services are used to represent the database building services required by multiple database building businesses, and the special database building services are used to represent the database building services required by special database building businesses. Before obtaining the basic workflow corresponding to the database building task, it further includes: Splitting the multiple database building services required by the database building business into common database building services and special database building services; Combining the common database building services and the special database building services to obtain a basic workflow.

7. The general database building method according to claim 6, characterized in that The combining the common database building services and the special database building services to obtain a basic workflow includes: Receiving an interaction operation of the user for the common database building services and the special database building services; Combining the common database building services and the special database building services according to the interaction operation to obtain a basic workflow.

8. A general library building device, characterized in that, The general database building device includes: An obtaining module, configured to obtain a basic workflow corresponding to the database building task, where the basic workflow is used to represent the database building process through the process relationship between basic database building services; An instantiating module, configured to instantiate the basic workflow according to the database building business instance corresponding to the database building task to obtain a business workflow, where the business workflow is used to represent the database building process of the database building business; A database building module, configured to establish an index database corresponding to the database building business according to the business workflow.

9. A general library building device, characterized in that The general library building device includes: a memory, a processor, and a general library building program stored on the memory and executable on the processor. When the general library building program is executed by the processor, it implements the general library building method according to any one of claims 1 to 7.

10. A storage medium, characterized in that, A general library building program is stored on the storage medium. When the general library building program is executed by a processor, it implements the general library building method according to any one of claims 1 to 7.

Images