A method and device for storing search engine results

Abstract

The present invention provides a method and device for storing search engine results, the method comprising: in response to a pipeline configuration instruction, obtaining data pipelines respectively associated with at least one configured search task, and storage locations corresponding to each data pipeline, wherein data pipelines corresponding to different storage types are predefined; in response to a search engine instruction, performing data search, for data obtained from each search task, using the data pipeline corresponding to the search task, converting the obtained data into a storage type corresponding to the data pipeline, and storing the converted data in the storage location corresponding to the data pipeline. The method and device for storing search engine results provided by the present invention solve the problems of the existing crawler result storage method having few crawler result storage types and the need to change the storage type and medium of the stored data later.

Description

Technical Field

[0001] The present invention relates to the field of Internet data processing, and in particular to a method and device for storing search engine results. Background Art

[0002] With the rapid development of Internet technology, the data in the network is growing exponentially, the amount of data is getting larger and larger, and the data forms are becoming more and more diverse. In today's information age, it is particularly important to efficiently and accurately derive and store valuable information from large and complex network data.

[0003] At present, web crawler methods are usually used to obtain required data from massive network data. A web crawler is a search engine program that automatically extracts web pages and can automatically capture data on the network according to certain rules. Although the use of web crawlers can quickly obtain network data, the export and storage of these data are accompanied by the work. At present, most web crawler methods only support the storage of crawler results, that is, the obtained data, as Excel table types. A small number of web crawler methods can provide two to three data export and storage types, which are not enough to meet the current demand for network data storage types. As a result, the data stored according to the current crawler result storage method may not have the required storage type, so it is necessary to adjust the data storage type and storage medium separately, and the data storage efficiency is low.

[0004] Therefore, the current storage method of crawler results has the problem that there are few crawler result storage types, and the stored data needs to change the storage type and medium later. The crawler result storage process is complicated and the storage efficiency is low. Summary of the invention

[0005] The present invention provides a method and device for storing search engine results, which are used to solve the problems that the existing crawler result storage method has few crawler result storage types and the storage type and medium of the stored data need to be changed later.

[0006] According to a first aspect of an embodiment of the present invention, a method for storing search engine results is provided, the method comprising:

[0007] In response to the pipeline configuration instruction, obtaining the data pipelines respectively associated with at least one configured search task and the storage location corresponding to each data pipeline, wherein data pipelines corresponding to different storage types are predefined;

[0008] In response to the search engine instruction, data search is performed, and for the data obtained by each search task, the data pipeline corresponding to the search task is used to convert the obtained data into a storage type corresponding to the data pipeline, and the converted data is stored in the storage location corresponding to the data pipeline.

[0009] In some embodiments, the data pipelines corresponding to different storage types include at least one of the following:

[0010] At least one database type data pipeline corresponding to each of the database format data;

[0011] At least one file type data pipeline corresponding to each file format data;

[0012] At least one FTP type data pipeline corresponding to each of the FTP format data.

[0013] In some embodiments, before performing a data search in response to a search engine instruction, the method further includes:

[0014] Using each configured data pipeline, a data interface between search tasks associated with the data pipeline is established, and a connection with a storage location corresponding to the data pipeline is established.

[0015] In some embodiments, further comprising:

[0016] In response to the search task completion instruction, when it is determined that all converted data are stored in the storage location corresponding to the data pipeline, the connection with the storage location corresponding to the data pipeline is disconnected.

[0017] In some embodiments, converting the acquired data into a storage type corresponding to the data pipeline and storing the converted data into a storage location corresponding to the data pipeline includes:

[0018] Using the data pipeline corresponding to the search task, data is read from the temporary storage medium storing the data obtained by the search task, and converted into a storage type corresponding to the data pipeline, and the converted data is stored in a storage location corresponding to the permanent storage medium corresponding to the data pipeline.

[0019] In some embodiments, using each configured data pipeline, establishing a connection with a storage location corresponding to the data pipeline includes performing at least one of the following steps:

[0020] Use the configured database type data pipeline and the Java database connection JDBC method to establish a connection with the database corresponding to the data pipeline;

[0021] Using the configured file type data pipeline, a connection with the disk file corresponding to the data pipeline is established in the form of data stream;

[0022] Using the configured FTP type data pipeline, a connection with the server corresponding to the data pipeline is established through the FTP protocol.

[0023] In some embodiments, using the data pipeline corresponding to the search task to convert the acquired data into a storage type corresponding to the data pipeline includes at least one of the following steps:

[0024] Using a database type data pipeline, converting the acquired data into a SQL database language format;

[0025] Using a file type data pipeline, converting the acquired data into a workbook object or a binary stream format;

[0026] The acquired data is converted into a binary stream format using an FTP type data pipeline.

[0027] According to a second aspect of an embodiment of the present invention, there is provided a device for storing search engine results, comprising:

[0028] A pipeline configuration module, for obtaining, in response to the pipeline configuration instruction, data pipelines respectively associated with at least one configured search task and storage locations corresponding to each data pipeline, wherein data pipelines corresponding to different storage types are predefined;

[0029] The data storage module is used to perform data search in response to search engine instructions, and for the data obtained from each search task, the data pipeline corresponding to the search task is used to convert the obtained data into a storage type corresponding to the data pipeline, and the converted data is stored in a storage location corresponding to the data pipeline.

[0030] In some embodiments, the data pipelines corresponding to different storage types include at least one of the following:

[0031] At least one database type data pipeline corresponding to each of the database format data;

[0032] At least one file type data pipeline corresponding to each file format data;

[0033] At least one FTP type data pipeline corresponding to each of the FTP format data.

[0034] In some embodiments, before the data storage module performs data search in response to the search engine instruction, the data storage module further includes:

[0035] Using each configured data pipeline, a data interface between search tasks associated with the data pipeline is established, and a connection with a storage location corresponding to the data pipeline is established.

[0036] In some embodiments, the data storage module is further used to:

[0037] In response to the search task completion instruction, when it is determined that all converted data are stored in the storage location corresponding to the data pipeline, the connection with the storage location corresponding to the data pipeline is disconnected.

[0038] In some embodiments, the data storage module converts the acquired data into a storage type corresponding to the data pipeline, and stores the converted data in a storage location corresponding to the data pipeline, including:

[0039] Using the data pipeline corresponding to the search task, data is read from the temporary storage medium storing the data obtained by the search task, and converted into a storage type corresponding to the data pipeline, and the converted data is stored in a storage location corresponding to the permanent storage medium corresponding to the data pipeline.

[0040] In some embodiments, the data storage module uses each configured data pipeline to establish a connection with a storage location corresponding to the data pipeline, including performing at least one of the following steps:

[0041] Use the configured database type data pipeline and the Java database connection JDBC method to establish a connection with the database corresponding to the data pipeline;

[0042] Using the configured file type data pipeline, a connection with the disk file corresponding to the data pipeline is established in the form of data stream;

[0043] Using the configured FTP type data pipeline, a connection with the server corresponding to the data pipeline is established through the FTP protocol.

[0044] In some embodiments, the data storage module uses the data pipeline corresponding to the search task to convert the acquired data into a storage type corresponding to the data pipeline, including at least one of the following steps:

[0045] Using a database type data pipeline, converting the acquired data into a SQL database language format;

[0046] Using a file type data pipeline, converting the acquired data into a workbook object or a binary stream format;

[0047] The acquired data is converted into a binary stream format using an FTP type data pipeline.

[0048] According to a third aspect of an embodiment of the present invention, there is provided a device for storing search engine results, including:

[0049] In response to the pipeline configuration instruction, obtaining the data pipelines respectively associated with at least one configured search task and the storage location corresponding to each data pipeline, wherein data pipelines corresponding to different storage types are predefined;

[0050] In response to the search engine instruction, data search is performed, and for the data obtained by each search task, the data pipeline corresponding to the search task is used to convert the obtained data into a storage type corresponding to the data pipeline, and the converted data is stored in the storage location corresponding to the data pipeline.

[0051] In some embodiments, the data pipelines corresponding to different storage types include at least one of the following:

[0052] At least one database type data pipeline corresponding to each of the database format data;

[0053] At least one file type data pipeline corresponding to each file format data;

[0054] At least one FTP type data pipeline corresponding to each of the FTP format data.

[0055] In some embodiments, before the processor performs a data search in response to a search engine instruction, the processor further includes:

[0056] Using each configured data pipeline, a data interface between search tasks associated with the data pipeline is established, and a connection with a storage location corresponding to the data pipeline is established.

[0057] In some embodiments, the processor is further configured to:

[0058] In response to the search task completion instruction, when it is determined that all converted data are stored in the storage location corresponding to the data pipeline, the connection with the storage location corresponding to the data pipeline is disconnected.

[0059] In some embodiments, the processor converts the acquired data into a storage type corresponding to the data pipeline, and stores the converted data in a storage location corresponding to the data pipeline, including:

[0060] Using the data pipeline corresponding to the search task, data is read from the temporary storage medium storing the data obtained by the search task, and converted into a storage type corresponding to the data pipeline, and the converted data is stored in a storage location corresponding to the permanent storage medium corresponding to the data pipeline.

[0061] In some embodiments, the processor uses each configured data pipeline to establish a connection with a storage location corresponding to the data pipeline, including performing at least one of the following steps:

[0062] Use the configured database type data pipeline and the Java database connection JDBC method to establish a connection with the database corresponding to the data pipeline;

[0063] Using the configured file type data pipeline, a connection with the disk file corresponding to the data pipeline is established in the form of data stream;

[0064] Using the configured FTP type data pipeline, a connection with the server corresponding to the data pipeline is established through the FTP protocol.

[0065] In some embodiments, the processor uses the data pipeline corresponding to the search task to convert the acquired data into a storage type corresponding to the data pipeline, including at least one of the following steps:

[0066] Using a database type data pipeline, converting the acquired data into a SQL database language format;

[0067] Using a file type data pipeline, converting the acquired data into a workbook object or a binary stream format;

[0068] The acquired data is converted into a binary stream format using an FTP type data pipeline.

[0069] According to the fourth aspect of an embodiment of the present invention, a chip is provided, which is coupled to a memory in a user device so that the chip calls program instructions stored in the memory during operation to implement the above-mentioned various aspects of the embodiment of the present application and any possible design method involved in each aspect.

[0070] According to a fifth aspect of an embodiment of the present invention, a computer-readable storage medium is provided, which stores program instructions. When the computer-readable storage medium is run on a terminal device, the terminal device executes the above-mentioned various aspects of the embodiment of the present invention and any possible design method involved in each aspect.

[0071] According to a sixth aspect of an embodiment of the present invention, a computer program product is provided. When the computer program product is run on an electronic device, the electronic device executes a method for implementing the above-mentioned various aspects of the embodiment of the present application and any possible design involved in each aspect.

[0072] The method and device for storing search engine results provided by the present invention have the following beneficial effects:

[0073] The method and device for storing search engine results provided by the present invention predefines data pipelines corresponding to different storage types, associates the search task with the data pipeline of the required type when searching for data using the established search task, and uses the data pipeline corresponding to the search task to convert the data searched by the search task into the storage type corresponding to the data pipeline and then stores it in the corresponding storage medium. The data searched by the search task can be converted into the required storage format and then stored, which solves the problem of the existing crawler result storage method having few crawler result storage types and the need to change the storage type and medium of the stored data later. BRIEF DESCRIPTION OF THE DRAWINGS

[0074] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0075] Figure 1 A schematic diagram of a method for storing search engine results provided in an embodiment of the present invention;

[0076] Figure 2 A schematic diagram of a page for pipeline selection prompt and pipeline selection instruction reception provided in an embodiment of the present invention;

[0077] Figure 3 A schematic diagram of a page for pipeline configuration prompt and pipeline configuration instruction reception provided in an embodiment of the present invention;

[0078] Figure 4 An example diagram of using a data pipeline to store search result data provided in an embodiment of the present invention;

[0079] Figure 5 A schematic diagram of a process of storing data using a data pipeline in a search task provided in an embodiment of the present invention;

[0080] Figure 6 A schematic diagram of a process of storing data using a data pipeline provided in an embodiment of the present invention;

[0081] Figure 7 A schematic diagram of a device for storing search engine results provided in an embodiment of the present invention;

[0082] Figure 8 The present invention is a schematic diagram of a device structure for storing search engine results provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0083] In order to make the purpose, technical scheme and advantages of the present invention clearer, the present invention will be further described in detail below with reference to the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, rather than all the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0084] For ease of understanding, the terms involved in the embodiments of the present invention are explained below:

[0085] 1) Web crawler: It is a program that automatically obtains web page content and downloads web page data from the Internet for search engines. It is an important part of search engines. Traditional web crawlers start from the uniform resource locator URL of one or several initial web pages, obtain the URL on the initial web page, and in the process of crawling web page data, continuously extract new URLs from the current page and put them into the queue until a certain stop condition of the system is met, then stop crawling data.

[0086] Example 1

[0087] An embodiment of the present invention provides a method for storing search engine results. Based on the search engine, multiple data export types are provided, and the data export type and the storage medium are encapsulated as a data pipeline. When storing the search engine results, the corresponding data pipeline is selected as needed to store the search result data.

[0088] Since a web crawler method is generally used to crawl data from the network, the specific implementation of the search engine in the embodiment of the present invention may be a web crawler, and the search engine results are corresponding crawler results.

[0089] Reference Figure 1 , a method for storing search engine results provided by an embodiment of the present invention, as shown in the figure, the method includes:

[0090] Step S101, in response to the pipeline configuration instruction, obtaining the data pipelines respectively associated with at least one configured search task and the storage location corresponding to each data pipeline, wherein data pipelines corresponding to different storage types are predefined;

[0091] In an embodiment of the present invention, data pipelines corresponding to different storage types are predefined. When a search engine is used to search for data, a data pipeline of a required type is selected from the predefined data pipelines of different storage types and configured accordingly, so that the data pipeline is used to perform data type conversion and storage, wherein the search task can be a crawler task established using a web crawler; the data pipeline is used to move specified data from one storage location to another storage location for storage.

[0092] Reference Figure 2 , which is a schematic diagram of a page for pipeline selection prompt and pipeline selection instruction reception provided in an embodiment of the present invention.

[0093] As an optional implementation, in the embodiment of the present invention, when establishing a search task to search for data, output is as follows: Figure 2 The prompt information of whether to call the data pipeline is shown in the figure, and the data pipelines corresponding to the pre-defined different storage types are received, and the input pipeline selection instruction is received, and it is determined whether to call the data pipeline according to the instruction, and the data pipeline called when calling is determined.

[0094] After determining to call the data pipeline, by responding to the search task establishment instruction, when establishing the search task to search for data, the pipeline configuration prompt information for data pipeline configuration is output, and the input pipeline configuration instruction is received, and the data pipeline associated with at least one configured search task is obtained according to the pipeline configuration instruction; or, in response to the input pipeline configuration instruction, the data pipeline associated with at least one configured search task is obtained according to the pipeline configuration instruction.

[0095] Among them, the pipeline configuration instructions include the main configuration information such as pipeline type, storage medium information, such as data pipeline name, data pipeline type, IP address and port corresponding to the permanent storage medium, user name and password for connecting to the permanent storage medium, database name, table name, file storage path and other configuration information.

[0096] Reference Figure 3 , is a schematic diagram of a pipeline configuration prompt and pipeline configuration instruction receiving page provided by an embodiment of the present invention. As shown in the figure, when outputting pipeline configuration prompt information for data pipeline configuration, the output displays various configuration items for configuring the data pipeline, including the above-mentioned configuration information, and receives input configuration instructions for each configuration item.

[0097] In an embodiment of the present invention, data pipelines corresponding to different storage types include at least one of the following:

[0098] 1) at least one database type data pipeline corresponding to each of the database format data;

[0099] Among them, the database format data includes MySQL, for example, data stored in MySQL in JSON (Java Script Object Notation, JS object notation) format, MySQL_FIELD format, MPP (Massively Parallel Processing) format, HBase (Hadoop Database) database format, Mongo (distributed document storage database) format and other types of data.

[0100] When the data pipeline associated with at least one search task configured is a database type data pipeline, the above pipeline configuration instruction includes data connection information, such as pipeline name, database IP address and port number, user name, password, database name, table name and other configuration information, and the data pipeline is configured accordingly according to the configuration information.

[0101] 2) a file type data pipeline corresponding to at least one file format data;

[0102] The file format data includes data in Excel format, File format, and other types of data.

[0103] When the data pipeline associated with at least one search task configured is a file type data pipeline, the above pipeline configuration instruction includes file storage path information and name information, such as pipeline name, storage path and other configuration information, and the data pipeline is configured accordingly according to the configuration information.

[0104] 3) at least one FTP type data pipeline corresponding to each of the FTP formatted data.

[0105] When the data pipeline associated with at least one search task configured is a file type data pipeline, the above pipeline configuration instruction includes server connection and storage information, such as pipeline name, server connection path and other configuration information, and the data pipeline is configured accordingly according to the configuration information.

[0106] In an embodiment of the present invention, when obtaining data pipelines respectively associated with at least one configured search task in response to a pipeline configuration instruction, the at least one search task can be associated with the same data pipeline, and each search task can be associated with multiple data pipelines of different types.

[0107] Specifically, multiple search tasks can be executed simultaneously to search for data, and each search task can be associated with different types of data pipelines, and use the associated data pipelines to store the searched data as data of the storage type corresponding to each data pipeline. Among them, each data pipeline can be associated with multiple search tasks. When multiple search tasks are associated, the data pipeline can store the searched data of the search tasks associated with it and performing data search as the storage type corresponding to the data pipeline.

[0108] In an embodiment of the present invention, when data search is performed using a search task, the same data pipeline associated with multiple search tasks is configured only once according to the pipeline configuration instruction. After the configured data pipeline is obtained in response to the pipeline configuration instruction, the data obtained by the multiple search tasks is converted into a storage type corresponding to the data pipeline using the data pipeline, and the converted data is stored in the storage location corresponding to the data pipeline.

[0109] Step S102, performing data search in response to the search engine instruction, converting the data obtained by each search task into a storage type corresponding to the data pipeline using the data pipeline corresponding to the search task, and storing the converted data in the storage location corresponding to the data pipeline.

[0110] After obtaining the data pipelines associated with at least one search task of the above-mentioned configuration and the storage positions corresponding to each data pipeline, the data pipeline corresponding to the search task is used to read data from the temporary storage medium storing the data obtained by the search task, and convert the data into a storage type corresponding to the data pipeline, and the converted data is stored in the storage position corresponding to the permanent storage medium corresponding to the data pipeline.

[0111] Reference Figure 4 , which is an example diagram of using data pipelines to store search result data provided by an embodiment of the present invention. As shown in the figure, it is assumed that the data pipelines associated with a search task include three types of data pipelines, namely data pipeline 1, data pipeline 2, and data pipeline 3. After the search task is started, the search task performs data search and stores the searched data in a temporary storage medium. Then, the data pipelines 1-3 associated with the search task read the data obtained by the search task from the temporary storage medium, convert the data format of the data, and store them in the corresponding permanent storage medium. When the search task ends and all the data obtained by the search task are stored in the permanent storage medium, the data pipeline is closed to complete the data storage.

[0112] In the embodiment of the present invention, before performing data search, a connection with a corresponding storage medium is first established using a data pipeline. Specifically, each configured data pipeline is used to establish a data interface between search tasks associated with the data pipeline, and to establish a connection with a storage location corresponding to the data pipeline.

[0113] The data pipeline acquires the data stored in the temporary storage medium by the search task through the data interface established with the associated search task, and stores the data in the corresponding storage location by establishing a connection with the storage location corresponding to the data pipeline. After the search task is completed, in response to the received search task completion instruction, when it is determined that all the converted data are stored in the storage location corresponding to the data pipeline, the connection with the storage location corresponding to the data pipeline is disconnected.

[0114] In an embodiment of the present invention, when utilizing each configured data pipeline to establish a connection with a storage location corresponding to the data pipeline, and converting data acquired by an associated search task into a storage type corresponding to the data pipeline and storing it in a corresponding storage medium, the specific implementation methods corresponding to different types of data pipelines are different.

[0115] Specifically, when using each configured data pipeline to establish a connection with a storage location corresponding to the data pipeline, at least one of the following medium connection steps is included:

[0116] 1) Using the configured database type data pipeline, adopt the Java database connection JDBC method to establish a connection with the database corresponding to the data pipeline;

[0117] 2) Using the configured file type data pipeline, establish a connection with the disk file corresponding to the data pipeline in the form of data stream;

[0118] 3) Using the configured FTP type data pipeline, establish a connection with the server corresponding to the data pipeline through the FTP protocol.

[0119] When converting the data acquired by the associated search task into a storage type corresponding to the data pipeline, at least one of the following data conversion steps is included:

[0120] 1) using a database type data pipeline, converting the acquired data into a SQL database language format;

[0121] 2) using a file type data pipeline, converting the acquired data into a workbook object or a binary stream format;

[0122] 3) Using an FTP type data pipeline, convert the acquired data into a binary stream format.

[0123] Reference Figure 5 , is a flow chart of using a data pipeline to store data in a search task provided by an embodiment of the present invention.

[0124] Reference Figure 6 , which is a flow chart of storing data using a data pipeline provided by an embodiment of the present invention.

[0125] like Figure 5As shown, after establishing the search task and associating the storage medium, configure the data pipeline according to the pipeline configuration instruction, and then execute the corresponding medium connection step according to the data pipeline type to connect the data pipeline to the corresponding storage medium. Then, execute the above data conversion step for the data obtained by the search task and store it in the corresponding storage medium. After the search task is completed and the obtained data is stored in the corresponding storage medium, close the connection between the data pipeline and the storage medium, and close the data pipeline to complete the storage of the result data corresponding to this search task.

[0126] Among them, Figure 6 As shown, for different types of data pipelines, three different implementation methods are specifically included. For database type data pipelines, a database connection method, i.e., the above-mentioned medium connection step 1), is used to establish a connection between the data pipeline and the corresponding database, and then the original data obtained by the search task is converted into the SQL database language format and stored in the corresponding database; for file type data pipelines, a file stream connection method, i.e., the above-mentioned medium connection step 2), is used to establish a connection between the data pipeline and the corresponding disk file, and then the original data obtained by the search task is converted into a workbook object or a binary stream format and stored in the corresponding disk file location; for FTP type data pipelines, an FTP protocol connection method, i.e., the above-mentioned medium connection step 3), is used to establish a connection between the data pipeline and the corresponding server, and then the original data obtained by the search task is converted into a binary stream format and stored in the corresponding FTP storage location (server).

[0127] The above-mentioned method for storing search engine results provided by the embodiment of the present invention predefines data pipelines corresponding to different storage types. When searching for data using the established search task, the search task is associated with the data pipeline of the required type, and the data searched by the search task is converted into the storage type corresponding to the data pipeline using the data pipeline corresponding to the search task, and then stored in the corresponding storage medium. The types of data export storage are enriched, the process of search engine search data and data processing is simplified, and the efficiency of data storage is improved. The method provided by the embodiment of the present invention can be applied to the crawler result storage scenario, solving the problem that the existing crawler result storage method has few crawler result storage types and the storage type and medium of the stored data need to be changed later.

[0128] Example 2

[0129] Based on the same inventive concept, an embodiment of the present invention further provides a device for storing search engine results. Since the device is the device in the method in the embodiment of the present invention, and the principle of solving the problem by the device is similar to that of the method, the implementation of the device can refer to the implementation of the method, and the repeated parts will not be repeated.

[0130] See also Figure 7 , an embodiment of the present invention provides a device for storing search engine results, including:

[0131] The pipeline configuration module 701 is used to obtain data pipelines respectively associated with at least one configured search task and storage locations corresponding to each data pipeline in response to the pipeline configuration instruction, wherein data pipelines corresponding to different storage types are predefined;

[0132] The data storage module 702 is used to perform data search in response to search engine instructions, and for the data obtained from each search task, uses the data pipeline corresponding to the search task to convert the obtained data into a storage type corresponding to the data pipeline, and stores the converted data in the storage location corresponding to the data pipeline.

[0133] In some embodiments, the data pipelines corresponding to different storage types include at least one of the following:

[0134] At least one database type data pipeline corresponding to each of the database format data;

[0135] At least one file type data pipeline corresponding to each file format data;

[0136] At least one FTP type data pipeline corresponding to each of the FTP format data.

[0137] In some embodiments, before the data storage module performs data search in response to the search engine instruction, the data storage module further includes:

[0138] Using each configured data pipeline, a data interface between search tasks associated with the data pipeline is established, and a connection with a storage location corresponding to the data pipeline is established.

[0139] In some embodiments, the data storage module is further used to:

[0140] In response to the search task completion instruction, when it is determined that all converted data are stored in the storage location corresponding to the data pipeline, the connection with the storage location corresponding to the data pipeline is disconnected.

[0141] In some embodiments, the data storage module converts the acquired data into a storage type corresponding to the data pipeline, and stores the converted data in a storage location corresponding to the data pipeline, including:

[0142] Using the data pipeline corresponding to the search task, data is read from the temporary storage medium storing the data obtained by the search task, and converted into a storage type corresponding to the data pipeline, and the converted data is stored in a storage location corresponding to the permanent storage medium corresponding to the data pipeline.

[0143] In some embodiments, the data storage module uses each configured data pipeline to establish a connection with a storage location corresponding to the data pipeline, including performing at least one of the following steps:

[0144] Use the configured database type data pipeline and the Java database connection JDBC method to establish a connection with the database corresponding to the data pipeline;

[0145] Using the configured file type data pipeline, a connection with the disk file corresponding to the data pipeline is established in the form of data stream;

[0146] Using the configured FTP type data pipeline, a connection with the server corresponding to the data pipeline is established through the FTP protocol.

[0147] In some embodiments, the data storage module uses the data pipeline corresponding to the search task to convert the acquired data into a storage type corresponding to the data pipeline, including at least one of the following steps:

[0148] Using a database type data pipeline, converting the acquired data into a SQL database language format;

[0149] Using a file type data pipeline, converting the acquired data into a workbook object or a binary stream format;

[0150] The acquired data is converted into a binary stream format using an FTP type data pipeline.

[0151] Example 3

[0152] Based on the same inventive concept, an embodiment of the present invention further provides a device for storing search engine results. Since the device is the device in the method in the embodiment of the present invention, and the principle of solving the problem by the device is similar to that of the method, the implementation of the device can refer to the implementation of the method, and the repeated parts will not be repeated.

[0153] See also Figure 8 , which is a schematic diagram of the structure of a device for storing search engine results according to an embodiment of the present invention.

[0154] like Figure 8 As shown, the device 800 includes: a communication interface 810, a memory 820 and a processor 830. The communication interface 810, the memory 820 and the processor 830 may be connected via a bus 840. Those skilled in the art will appreciate that Figure 8 The structure of the device 800 shown in the figure does not constitute a limitation on the device 800. Figure 8 The device 800 shown is only an example and the device 800 may have more Figure 8More or fewer components may be shown, two or more components may be combined, or different component configurations may be provided. The various components shown in the figures may be implemented in hardware, software, or a combination of hardware and software including one or more signal processing and / or application specific integrated circuits. Figure 8 The components of the device 800 are described in detail:

[0155] The communication interface 810 may be used to communicate with a voice recognition device and a cloud server, for example, to receive elevator control instructions sent by the voice recognition device or the cloud server.

[0156] The memory 820 can be used to store data, programs and / or modules used when the device is running, such as program instructions and / or modules corresponding to the control method of the switch controller in the embodiment of the present application. The processor 830 executes various functional applications and data processing of the device 800 by running the programs and / or modules stored in the memory 820, such as the relevant control method of the elevator control system provided in the embodiment of the present application. The memory 820 can mainly include a program storage area and a data storage area, wherein the program storage area can store an operating system, at least one application application, etc.; the data storage area can store data created according to the use of the device 800, etc. In addition, the memory 820 can include a high-speed random access memory, and can also include a non-volatile memory, such as at least one disk storage device, a flash memory device, or other volatile solid-state storage devices.

[0157] The processor 830 is the control center of the device 800. It uses various interfaces and lines to connect various parts of the entire device 800. By running or executing software programs and / or modules stored in the memory 820, and calling data stored in the memory 820, it performs various functions of the device 800 and processes data, thereby monitoring the device 800 as a whole. In some embodiments, the processor 830 may include one or more processing units. The processor 830 in the embodiment of the present application can run an operating system, an application program, execute an operation instruction sent by a server, and the related control method of the embodiment of the present invention.

[0158] The specific connection medium between the memory 820, the processor 830 and the communication interface 810 is not limited in the embodiment of the present invention. Figure 8 In the embodiment, the memory 820, the processor 830 and the communication interface 810 are connected via a bus 840, and the connection between other components is only for illustrative purposes and is not intended to be limiting. The bus 840 can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Figure 8 Only one thick line is used in the diagram, but this does not mean that there is only one bus or only one type of bus.

[0159] In this embodiment, the processor is specifically used for:

[0160] In response to the pipeline configuration instruction, obtaining the data pipelines respectively associated with at least one configured search task and the storage location corresponding to each data pipeline, wherein data pipelines corresponding to different storage types are predefined;

[0161] In response to the search engine instruction, data search is performed, and for the data obtained by each search task, the data pipeline corresponding to the search task is used to convert the obtained data into a storage type corresponding to the data pipeline, and the converted data is stored in the storage location corresponding to the data pipeline.

[0162] In some embodiments, the data pipelines corresponding to different storage types include at least one of the following:

[0163] At least one database type data pipeline corresponding to each of the database format data;

[0164] At least one file type data pipeline corresponding to each file format data;

[0165] At least one FTP type data pipeline corresponding to each of the FTP format data.

[0166] In some embodiments, before the processor performs a data search in response to a search engine instruction, the processor further includes:

[0167] Using each configured data pipeline, a data interface between search tasks associated with the data pipeline is established, and a connection with a storage location corresponding to the data pipeline is established.

[0168] In some embodiments, the processor is further configured to:

[0169] In response to the search task completion instruction, when it is determined that all converted data are stored in the storage location corresponding to the data pipeline, the connection with the storage location corresponding to the data pipeline is disconnected.

[0170] In some embodiments, the processor converts the acquired data into a storage type corresponding to the data pipeline, and stores the converted data in a storage location corresponding to the data pipeline, including:

[0171] Using the data pipeline corresponding to the search task, data is read from the temporary storage medium storing the data obtained by the search task, and converted into a storage type corresponding to the data pipeline, and the converted data is stored in a storage location corresponding to the permanent storage medium corresponding to the data pipeline.

[0172] In some embodiments, the processor uses each configured data pipeline to establish a connection with a storage location corresponding to the data pipeline, including performing at least one of the following steps:

[0173] Use the configured database type data pipeline and the Java database connection JDBC method to establish a connection with the database corresponding to the data pipeline;

[0174] Using the configured file type data pipeline, a connection with the disk file corresponding to the data pipeline is established in the form of data stream;

[0175] Using the configured FTP type data pipeline, a connection with the server corresponding to the data pipeline is established through the FTP protocol.

[0176] In some embodiments, the processor uses the data pipeline corresponding to the search task to convert the acquired data into a storage type corresponding to the data pipeline, including at least one of the following steps:

[0177] Using a database type data pipeline, converting the acquired data into a SQL database language format;

[0178] Using a file type data pipeline, converting the acquired data into a workbook object or a binary stream format;

[0179] The acquired data is converted into a binary stream format using an FTP type data pipeline.

[0180] It should be understood that the technical solutions defined in the embodiments of the present application are merely illustrative descriptions of devices or control methods for storing search engine results and are not any limitations. The same or similar concepts or processes may not be repeated in some embodiments.

[0181] An embodiment of the present invention further provides a computer-readable storage medium, comprising instructions, which, when executed on a terminal device, enables the terminal device to execute the method for storing search engine results provided in the above embodiment.

[0182] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the above-described devices and modules can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0183] In the several embodiments provided in the present application, it should be understood that the disclosed systems, devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the modules is only a logical function division. There may be other division methods in actual implementation, such as multiple modules or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be an indirect coupling or communication connection through some interfaces, devices or modules, which can be electrical, mechanical or other forms.

[0184] The modules described as separate components may or may not be physically separated, and the components shown as modules may or may not be physical modules, that is, they may be located in one place or distributed on multiple network modules. Some or all of the modules may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0185] In addition, each functional module in each embodiment of the present application can be integrated into a processing module, or each module can exist physically separately, or two or more modules can be integrated into one module. The above-mentioned integrated module can be implemented in the form of hardware or in the form of a software functional module. If the integrated module is implemented in the form of a software functional module and sold or used as an independent product, it can be stored in a computer-readable storage medium.

[0186] The technical solution provided by the present application is introduced in detail above. The principles and implementation methods of the present application are explained by using specific examples in the present application. The description of the above embodiments is only used to help understand the method of the present application and its core idea. At the same time, for those skilled in the art, according to the idea of ​​the present application, there will be changes in the specific implementation methods and application scope. In summary, the content of this specification should not be understood as a limitation on the present application.

[0187] Obviously, those skilled in the art can make various changes and modifications to the present application without departing from the spirit and scope of the present application. Thus, if these modifications and variations of the present application fall within the scope of the claims of the present application and their equivalents, the present application is also intended to include these modifications and variations.

Claims

1. A method for storing search engine results, characterized in that: include: When creating a search task to search for data, output a prompt message on whether to call the data pipeline; After confirming to call the data pipeline, outputting pipeline configuration prompt information for configuring the data pipeline; In response to the pipeline configuration instruction, the data pipelines respectively associated with each search task and the storage location corresponding to each data pipeline are obtained, wherein data pipelines corresponding to different storage types are predefined, and each search task is associated with the same data pipeline, or each search task is associated with multiple data pipelines of different types; Establishing a data interface between search tasks associated with each configured data pipeline, and establishing a connection with a storage location corresponding to the data pipeline; In response to the search engine instruction, executing each search task to search for data; Acquire data acquired by the search task corresponding to the data pipeline through the established data interface between the search tasks associated with each data pipeline, convert the acquired data into a storage type corresponding to the data pipeline, and store the converted data in a storage location corresponding to the data pipeline; In response to the search task completion instruction, when it is determined that any data pipeline stores all converted data in a storage location corresponding to the data pipeline, the connection with the storage location corresponding to the data pipeline is disconnected.

2. The method according to claim 1, characterized in that The data pipelines corresponding to different storage types include at least one of the following: At least one database type data pipeline corresponding to each of the database format data; At least one file type data pipeline corresponding to each file format data; At least one FTP type data pipeline corresponding to each of the FTP format data.

3. The method according to claim 1 or 2, characterized in that: Converting the acquired data into a storage type corresponding to the data pipeline, and storing the converted data into a storage location corresponding to the data pipeline, including: Data is read from a temporary storage medium storing data obtained by the corresponding search task, and converted into a storage type corresponding to the data pipeline, and the converted data is stored in a storage location corresponding to the permanent storage medium corresponding to the data pipeline.

4. The method according to claim 1, characterized in that: Establishing a connection with a storage location corresponding to the data pipeline includes: If the data pipeline is a database type data pipeline, a Java database connection JDBC method is used to establish a connection with the database corresponding to the data pipeline; If the data pipeline is a file type data pipeline, a connection with the disk file corresponding to the data pipeline is established in the form of data stream; If the data pipeline is an FTP type data pipeline, a connection with a server corresponding to the data pipeline is established through the FTP protocol.

5. The method according to claim 2, characterized in that: Convert the acquired data into the storage type corresponding to the data pipeline, including: If the data pipeline is a database type data pipeline, converting the acquired data into a SQL database language format; If the data pipeline is a file type data pipeline, converting the acquired data into a workbook object or a binary stream format; If the data pipeline is an FTP type data pipeline, the acquired data is converted into a binary stream format.

6. A device for storing search engine results, characterized in that: include: The pipeline configuration module is used to output prompt information on whether to call the data pipeline when establishing a search task to search for data; After confirming to call the data pipeline, outputting pipeline configuration prompt information for configuring the data pipeline; In response to the pipeline configuration instruction, the data pipelines respectively associated with each search task and the storage location corresponding to each data pipeline are obtained, wherein data pipelines corresponding to different storage types are predefined, and each search task is associated with the same data pipeline, or each search task is associated with multiple different types of data pipelines; a data interface is established between the search tasks associated with each configured data pipeline, and a connection is established with the storage location corresponding to the data pipeline; A data storage module is used to respond to search engine instructions and execute various search tasks to perform data searches; obtain the data obtained by the search task corresponding to the data pipeline through the data interface between the search tasks associated with each data pipeline, convert the obtained data into a storage type corresponding to the data pipeline, and store the converted data in the storage location corresponding to the data pipeline; in response to a search task completion instruction, when it is determined that any data pipeline stores all the converted data in the storage location corresponding to the data pipeline, disconnect the connection with the storage location corresponding to the data pipeline.

7. A device for storing search engine results, characterized in that: include: A memory and a processor; wherein: The memory is used to store programs; The processor is used to execute the program in the memory to implement the steps of any one of the methods described in claims 1 to 5.

8. A computer program medium, characterized in that A computer program is stored thereon, characterized in that when the program is executed by a processor, the steps of the method as claimed in any one of claims 1 to 5 are implemented.

Images