Data crawling method, device, computer system and computer readable storage medium

Abstract

The present disclosure provides a data crawling method, a data crawling device, a computer system, a computer-readable storage medium, and a computer program product, which can be used in information security, the field of the Internet of Things, or other fields. The method includes: using a data stream engine to obtain a data set consisting of data fragments, wherein the data fragments are constructed based on the web page data pre-crawled by the client; crawling web page data from the web page according to the data fragments in the data set to generate target crawling data; and sending the target crawling data to the client, so that the client determines the pre-crawled web page data according to the target crawling data.

Description

Technical Field

[0001] The present disclosure relates to the fields of information security and Internet of Things technology, and more specifically, to a data crawling method, a data crawling device, a computer system, a computer-readable storage medium, and a computer program product. Background Art

[0002] With the rapid development of the Internet, the World Wide Web has become a carrier of a large amount of information. How to effectively extract and use this information has become a huge challenge. In response to this challenge, crawler technology that can capture relevant web resources in a targeted manner has emerged.

[0003] In the process of realizing the concept disclosed in the present invention, the inventors found that there are at least the following problems in the related technology. In the process of using a crawler to crawl data, it is inevitable that the crawling of data will be interrupted due to external reasons. At this time, it is usually necessary to re-crawl, which is time-consuming and labor-intensive, and may be interrupted again due to the same reason. Summary of the invention

[0004] In view of this, the present disclosure provides a data crawling method, a data crawling device, a computer system, a computer-readable storage medium, and a computer program product.

[0005] One aspect of the present disclosure provides a data crawling method, comprising: using a data flow engine to obtain a data set composed of data slices, wherein the data slices are constructed based on web page data pre-crawled by a client; crawling web page data from the web page according to the data slices in the data set to generate target crawled data; and sending the target crawled data to the client, so that the client determines the pre-crawled web page data based on the target crawled data.

[0006] According to an embodiment of the present disclosure, the data slices constructed according to the web page data pre-crawled by the client include multiple data slices, each of which is configured with a slice identifier, and the web page data is crawled from the web page according to the data slices in the data set, and the target crawled data is generated, including: in the event that the process of pre-crawl the web page data by the client is interrupted, obtaining a first target slice identifier corresponding to the partial web page data crawled by the client; obtaining all slice identifiers corresponding to multiple data slices in the data set; matching the first target slice identifier with the all slice identifiers to filter out a second target slice identifier; crawling from the web page according to the target data slice corresponding to the second target slice identifier to obtain the target web page data that was not crawled during the pre-crawling process of the client; and using the target web page data as the target crawled data.

[0007] According to an embodiment of the present disclosure, crawling web page data from the web page end according to the data slices in the data set, generating target crawling data also includes: detecting through a probe whether the data flow engine has established a connection with the web page end; when the data flow engine has established a connection with the web page end, obtaining from the web page end the download data corresponding to the target data slice and the download identifier returned for the target data slice; and using the target data slice and the download data and download identifier corresponding to the target data slice as the target crawling data.

[0008] According to an embodiment of the present disclosure, the above method also includes: combining the first target fragment identifier and the second target fragment identifier to obtain a final fragment identifier; and when the final fragment identifier is consistent with the fragment identifier stored on the client, determining the web page data pre-crawled by the client based on the partial web page data and the target web page data.

[0009] Another aspect of the present disclosure provides a data crawling method, including: obtaining data slices constructed based on web page data pre-crawled by a client; configuring slice identifiers for the data slices; storing the data slices configured with the slice identifiers in a data flow engine in the form of data sets, so that the data flow engine crawls web page data from the web page according to the data slices in the data set to generate target crawled data; and obtaining the target crawled data to determine the web page data pre-crawled by the client according to the target crawled data.

[0010] According to an embodiment of the present disclosure, the data fragments constructed according to the web page data pre-crawled by the client include multiple data fragments. Before obtaining the target crawled data to determine the web page data pre-crawled by the client according to the target crawled data, the method includes: in the event that the process of the client pre-crawls the web page data is interrupted, determining a first target fragment identifier corresponding to part of the web page data crawled by the client; obtaining all fragment identifiers of the data fragments; matching the first target fragment identifier with all fragment identifiers to filter out a second target fragment identifier; and determining a target data fragment in the data set corresponding to the second target fragment identifier according to the second target fragment identifier, so that the data flow engine crawls from the web page according to the target data fragment to obtain the target web page data that was not crawled during the pre-crawling process of the client, and using the target web page data as the target crawled data, wherein the target web page data includes download data corresponding to the target data fragment obtained from the web page and a download identifier returned for the target data fragment.

[0011] According to an embodiment of the present disclosure, the above method also includes: obtaining the second target fragment identifier and its corresponding target web page data; combining the first target fragment identifier and the second target fragment identifier to obtain a final fragment identifier; and when the final fragment identifier is consistent with the fragment identifier stored on the client, determining the web page data pre-crawled by the client based on the partial web page data and the target web page data.

[0012] Another aspect of the present disclosure provides a data crawling device, including: an acquisition module, used to use a data flow engine to acquire a data set composed of data slices, wherein the data slices are constructed based on web page data pre-crawled by a client; a generation module, used to crawl web page data from the web page according to the data slices in the data set to generate target crawled data; and a sending module, used to send the target crawled data to the client, so that the client determines the pre-crawled web page data based on the target crawled data.

[0013] Another aspect of the present disclosure provides a data crawling device, including: a first acquisition module, used to acquire data slices constructed according to web page data pre-crawled by a client; a configuration module, used to configure slice identifiers for the data slices; a storage module, used to store the data slices configured with the slice identifiers in a data flow engine in the form of data sets, so that the data flow engine crawls web page data from the web page according to the data slices in the data set and generates target crawled data; and a second acquisition module, used to acquire the target crawled data to determine the web page data pre-crawled by the client according to the target crawled data.

[0014] Another aspect of the present disclosure provides a computer system, including: one or more processors; and a memory for storing one or more programs, wherein when the one or more programs are executed by the one or more processors, the one or more processors implement the data crawling method as described above.

[0015] Another aspect of the present disclosure provides a computer-readable storage medium having computer-executable instructions stored thereon, which, when executed, are used to implement the data crawling method as described above.

[0016] Another aspect of the present disclosure provides a computer program product, which includes computer executable instructions, and when the instructions are executed, are used to implement the data crawling method as described above.

[0017] According to the embodiments of the present disclosure, a technical means is adopted by utilizing a data flow engine to obtain a data set composed of data slices, wherein the data slices are constructed based on the web page data pre-crawled by the client; crawling web page data from the web page according to the data slices in the data set to generate target crawled data; and sending the target crawled data to the client so that the client determines the pre-crawled web page data according to the target crawled data. Since the web page data that the client needs to crawl can be crawled according to the data set in the data flow engine, and the data set will not be affected by external interruptions, the technical problem of time-consuming and labor-intensive data crawling process caused by the need to re-crawl when crawling data is interrupted is at least partially overcome, thereby achieving the technical effect of simple and effective breakpoint resumption according to data slices. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] The above and other objects, features and advantages of the present disclosure will become more apparent through the following description of the embodiments of the present disclosure with reference to the accompanying drawings, in which:

[0019] Figure 1 An exemplary system architecture to which the data crawling method according to an embodiment of the present disclosure can be applied is schematically shown;

[0020] Figure 2 A flowchart of a data crawling method applied to a data stream engine according to an embodiment of the present disclosure is schematically shown;

[0021] Figure 3 A flowchart of a data crawling method applied to a client according to an embodiment of the present disclosure is schematically shown;

[0022] Figure 4 A system architecture diagram of a normal crawler scenario for implementing a data crawling method according to an embodiment of the present disclosure is schematically shown;

[0023] Figure 5 A system architecture diagram for implementing a breakpoint crawler scenario of a data crawling method according to an embodiment of the present disclosure is schematically shown;

[0024] Figure 6 A block diagram schematically shows a data crawling device applied to a data stream engine according to an embodiment of the present disclosure;

[0025] Figure 7 A block diagram schematically shows a data crawling device applied to a client according to an embodiment of the present disclosure; and

[0026] Figure 8 A block diagram of a computer system suitable for implementing the method described above according to an embodiment of the present disclosure is schematically shown. DETAILED DESCRIPTION

[0027] Hereinafter, embodiments of the present disclosure will be described with reference to the accompanying drawings. However, it should be understood that these descriptions are exemplary only and are not intended to limit the scope of the present disclosure. In the following detailed description, for ease of explanation, many specific details are set forth to provide a comprehensive understanding of the embodiments of the present disclosure. However, it is apparent that one or more embodiments may also be implemented without these specific details. In addition, in the following description, descriptions of known structures and technologies are omitted to avoid unnecessary confusion of the concepts of the present disclosure.

[0028] The terms used herein are only for describing specific embodiments and are not intended to limit the present disclosure. The terms "comprise", "include", etc. used herein indicate the existence of the features, steps, operations and / or components, but do not exclude the existence or addition of one or more other features, steps, operations or components.

[0029] All terms (including technical and scientific terms) used herein have the meanings commonly understood by those skilled in the art, unless otherwise defined. It should be noted that the terms used herein should be interpreted as having a meaning consistent with the context of this specification, and should not be interpreted in an idealized or overly rigid manner.

[0030] In the case of using expressions such as "at least one of A, B, and C, etc.", it should generally be interpreted in accordance with the meaning of the expression generally understood by those skilled in the art (for example, "a system having at least one of A, B, and C" should include but is not limited to a system having A alone, B alone, C alone, A and B, A and C, B and C, and / or A, B, C, etc.). In the case of using expressions such as "at least one of A, B, or C, etc.", it should generally be interpreted in accordance with the meaning of the expression generally understood by those skilled in the art (for example, "a system having at least one of A, B, or C" should include but is not limited to a system having A alone, B alone, C alone, A and B, A and C, B and C, and / or A, B, C, etc.).

[0031] In the process of realizing the concept disclosed in the present invention, the inventors found that with the continuous development of Internet technology, the amount of data is getting larger and larger, and it takes a long time to crawl a correspondingly large amount of data. A series of reasons may arise in the crawling process, such as the crash of the crawler program, or the interruption of the crawler due to system reasons, etc., which leads to the interruption of crawling data and the inability to download the data completely. At this time, it is necessary to download it again, but it is impossible to determine which breakpoint to download from, which is time-consuming and labor-intensive and very inconvenient. At the same time, crawling again will encounter the same problem, and it is also very time-consuming to locate and solve the problem.

[0032] The embodiments of the present disclosure provide a data crawling method, a data crawling device, a computer system, a computer-readable storage medium, and a computer program product. The method includes: at the data engine end, using a data flow engine to obtain a data set composed of data fragments, wherein the data fragments are constructed based on the web page data pre-crawled by the client; crawling web page data from the web page according to the data fragments in the data set to generate target crawled data; and sending the target crawled data to the client, so that the client determines the pre-crawled web page data according to the target crawled data. At the client, obtain the data fragments constructed based on the web page data pre-crawled by the client; configure fragment identifiers for the data fragments; store the data fragments configured with fragment identifiers in the data flow engine in the form of a data set, so that the data flow engine crawls the web page data from the web page according to the data fragments in the data set to generate target crawled data; and obtain the target crawled data to determine the pre-crawled web page data of the client according to the target crawled data.

[0033] Figure 1 The exemplary system architecture 100 to which the data crawling method according to the embodiment of the present disclosure can be applied is schematically shown. It should be noted that: Figure 1 What is shown is merely an example of a system architecture to which the embodiments of the present disclosure can be applied, in order to help those skilled in the art understand the technical content of the present disclosure, but it does not mean that the embodiments of the present disclosure cannot be used in other devices, systems, environments or scenarios.

[0034] like Figure 1 As shown, the system architecture 100 according to this embodiment may include terminal devices 101, 102, 103, a network 104 and a server 105. The network 104 is used to provide a medium for communication links between the terminal devices 101, 102, 103 and the server 105. The network 104 may include various connection types, such as wired and / or wireless communication links, etc.

[0035] Users can use terminal devices 101, 102, 103 to interact with server 105 through network 104 to receive or send messages, etc. Various communication client applications can be installed on terminal devices 101, 102, 103, such as shopping applications, web browser applications, search applications, instant messaging tools, email clients and / or social platform software, etc.

[0036] The terminal devices 101 , 102 , and 103 may be various electronic devices having a display screen and supporting web browsing, including but not limited to smart phones, tablet computers, laptop computers, and desktop computers.

[0037] The server 105 may be a server that provides various services, such as a background management server that provides support for websites browsed by users using the terminal devices 101, 102, and 103. The background management server may analyze and process received data such as user requests, and feed back processing results (such as web pages, information, or data obtained or generated according to user requests) to the terminal device.

[0038] It should be noted that the data crawling method provided in the embodiment of the present disclosure can generally be executed by the server 105. Accordingly, the data crawling device provided in the embodiment of the present disclosure can generally be set in the server 105. The data crawling method provided in the embodiment of the present disclosure can also be executed by a server or server cluster that is different from the server 105 and can communicate with the terminal devices 101, 102, 103 and / or the server 105. Correspondingly, the data crawling device provided in the embodiment of the present disclosure can also be set in a server or server cluster that is different from the server 105 and can communicate with the terminal devices 101, 102, 103 and / or the server 105. Alternatively, the data crawling method provided in the embodiment of the present disclosure can also be executed by the terminal devices 101, 102, or 103, or can also be executed by other terminal devices different from the terminal devices 101, 102, or 103. Correspondingly, the data crawling device provided in the embodiment of the present disclosure may also be set in the terminal device 101, 102, or 103, or in other terminal devices different from the terminal device 101, 102, or 103.

[0039] For example, the data set may be originally stored in the data flow engine of any one of the terminal devices 101, 102, or 103 (for example, the terminal device 101, but not limited thereto), or stored in the data flow engine of an external storage device or a client and may be imported into the data flow engine of the terminal device 101. Then, the data flow engine of the terminal device 101 may locally execute the data crawling method provided by the embodiment of the present disclosure, or send the data set to the data flow engine of other terminal devices, servers, or server clusters, and the data flow engines of other terminal devices, servers, or server clusters that receive the data set may execute the data crawling method provided by the embodiment of the present disclosure.

[0040] For another example, the data shards may be originally stored in any one of the terminal devices 101, 102, or 103 (for example, the terminal device 101, but not limited thereto), or stored in an external storage device and may be imported into the terminal device 101. Then, the terminal device 101 may locally execute the data crawling method provided in the embodiment of the present disclosure, or send the data shards to other terminal devices, servers, or server clusters, and the data crawling method provided in the embodiment of the present disclosure is executed by other terminal devices, servers, or server clusters that receive the data shards. In this embodiment, the terminal devices 101, 102, 103, the external storage device, the server, or the server cluster are all configured with a data flow engine to provide a basic environment for the above-mentioned data crawling method.

[0041] It should be understood that Figure 1 The number of terminal devices, networks and servers in the embodiment is only for illustration. Any number of terminal devices, networks and servers may be provided according to implementation requirements.

[0042] It should be noted that the data crawling method, data crawling device, computer system, computer-readable storage medium and computer program product disclosed in the present invention can be used in the fields of information security and Internet of Things technology, and can also be used in any field other than the fields of information security and Internet of Things technology. The application fields of the data crawling method, data crawling device, computer system, computer-readable storage medium and computer program product disclosed in the present invention are not limited.

[0043] Figure 2 The flowchart of the data crawling method applied to the data flow engine according to the embodiment of the present disclosure is schematically shown.

[0044] like Figure 2 As shown, the method includes operations S201 to S203.

[0045] In operation S201, a data set consisting of data slices is obtained by using a data stream engine, wherein the data slices are constructed based on web page data pre-crawled by a client.

[0046] In operation S202, web page data is crawled from the web page end according to the data slices in the data set to generate target crawled data.

[0047] In operation S203, the target crawled data is sent to the client, so that the client determines the pre-crawled web page data according to the target crawled data.

[0048] According to an embodiment of the present disclosure, the data stream engine may be, for example, a Flink engine (an open source stream processing framework developed by the Apache Software Foundation, the core of which is a distributed stream data stream engine written in Java and Scala), Flink is used for crawler data stream analysis, and crawler data streams may be recorded through Flink. The data set may be, for example, a set obtained by processing the Flink data stream in a dataset (an independent data set that does not depend on a database) manner.

[0049] According to an embodiment of the present disclosure, the communication information corresponding to the process of the client pre-crawls the web page data, for example, can be stored in the Flink engine (i.e., the data flow engine) in the form of a dataset, and the data shards can be obtained according to the communication information, wherein the communication information can include information of pre-crawled data, client information, and web page information, etc. The method for implementing the data crawling can include, for example, direct crawling from the web page by the client, and crawling from the web page by the data flow engine, and sending the crawled data to the client, so that the client can finally obtain the web page data that it wants to crawl according to the crawled data and / or the data received from the data flow engine.

[0050] Through the above-mentioned embodiments of the present disclosure, since the dataset is still available when the data link is disconnected or the database is closed, a method for crawling data from the path of the data flow engine is added by adding a data set obtained based on the dataset. This can effectively solve the problem of inability to completely crawl data due to interruptions in the process of the client directly crawling data from the web page.

[0051] According to an embodiment of the present disclosure, the data slices constructed according to the web page data pre-crawled by the client may include, for example, multiple data slices, each of which may be configured with a corresponding slice identifier. The operation S202 may include, for example: in the event that the process of pre-crawl of web page data by the client is interrupted, obtaining a first target slice identifier corresponding to a portion of the web page data crawled by the client; obtaining all slice identifiers corresponding to multiple data slices in the data set; matching the first target slice identifier with all slice identifiers to obtain a second target slice identifier by screening; crawling from the web page the target data slice corresponding to the second target slice identifier to obtain the target web page data that was not crawled during the pre-crawl process of the client; and using the target web page data as the target crawled data.

[0052] According to an embodiment of the present disclosure, the communication information corresponding to the process of the client pre-crawling web page data can be divided into one or more data slices, and each data slice can be provided with a corresponding slice identifier, for example. The data slice and its corresponding slice identifier (that is, all the slice identifiers) can be stored in the client and the data flow engine at the same time. The first target slice identifier is used to indicate the identifier of the data slice corresponding to the data crawled by the client before the interruption (that is, the above-mentioned partial web page data), and the second target slice identifier is used to indicate the identifier of the data slice corresponding to the data crawled by the data flow engine after the interruption (that is, the above-mentioned target web page data).

[0053] Through the above-mentioned embodiments of the present disclosure, even if an interruption occurs between the client and the web page, the data required by the client can still be crawled from the web page through the data fragments in the data set of the data flow engine, effectively solving the problem of high difficulty in resuming breakpoint transmission.

[0054] According to an embodiment of the present disclosure, in the process of executing the above operation S202, for example, it can also include: detecting through a probe whether a connection is established between the data flow engine and the web page; when a connection is established between the data flow engine and the web page, obtaining the download data corresponding to the target data segment and the download identifier returned for the target data segment from the web page; and using the target data segment and the download data and download identifier corresponding to the target data segment as target crawling data.

[0055] According to an embodiment of the present disclosure, due to the existence of an interruption, the data corresponding to some data segments have been downloaded but the data corresponding to another part of the data segments have not been downloaded. In this case, different data segments may correspond to different download states. To distinguish the download states of different data segments, for example, it can be achieved by configuring a download identifier. The data segment that has completed the data download is configured with the download identifier, and the data segment that has not completed the data download is not configured with the download identifier. Based on this, by judging whether the download identifier is configured, the data segments corresponding to the data that the client has crawled and the data segments corresponding to the data that the client has not crawled (i.e., the above-mentioned target web page data) can be distinguished. After the data stream engine completes the crawling of the data that the client has not crawled (i.e., obtains the above-mentioned download data), the data segments corresponding to the data that the client has not crawled can be configured with a download identifier, so as to provide a basis for judging whether the crawling of the web page data pre-crawled by the client has been completed based on the data segment configured with the download identifier.

[0056] According to an embodiment of the present disclosure, in order to realize the judgment of the crawling results of web page data and to finally obtain the crawling results, the above-mentioned data crawling method may also include, for example: combining the first target shard identifier and the second target shard identifier to obtain a final shard identifier; when the final shard identifier is consistent with the shard identifier stored on the client, determining the web page data pre-crawled by the client based on the partial web page data and the target web page data.

[0057] According to an embodiment of the present disclosure, by matching the combination of the first target shard identifier and the second target shard identifier with the aforementioned all shard identifiers, it can be determined that all the web page data pre-crawled by the client has been downloaded, and the web page data can be determined by combining the download data corresponding to the data slices corresponding to the first target shard identifier and the second target shard identifier.

[0058] Through the above-mentioned embodiments of the present disclosure, the Flink data flow engine and the dataset data set are introduced, which effectively solves the problem of data crawling difficulties caused by crawling interruptions and provides an effective solution for breakpoint resumption.

[0059] Figure 3 The flowchart of the data crawling method applied to the client according to an embodiment of the present disclosure is schematically shown.

[0060] like Figure 3 As shown, the method includes operations S301 to S304.

[0061] In operation S301, data fragments are obtained based on the web page data pre-crawled by the client.

[0062] In operation S302, a slice identifier is configured for the data slice.

[0063] In operation S303, the data slices configured with the slice identifiers are stored in the data flow engine in the form of data sets, so that the data flow engine crawls web page data from the web page end according to the data slices in the data set to generate target crawled data.

[0064] In operation S304, target crawling data is obtained to determine the web page data pre-crawled by the client according to the target crawling data.

[0065] Through the above-mentioned embodiments of the present disclosure, a method for crawling data from the path of the data flow engine is added, which can effectively solve the problem of inability to completely crawl data due to interruptions during the process of the client directly crawling data from the web page.

[0066] According to an embodiment of the present disclosure, the data fragments constructed based on the web page data pre-crawled by the client include multiple data fragments, and before executing the above operation S304, the following are included: in the event that the process of pre-crawl of web page data by the client is interrupted, determining the first target fragment identifier corresponding to the partial web page data crawled by the client; obtaining all fragment identifiers of the data fragments; matching the first target fragment identifier with all fragment identifiers to screen and obtain the second target fragment identifier; and determining the target data fragment corresponding to the second target fragment identifier in the data set according to the second target fragment identifier, so that the data flow engine can crawl from the web page according to the target data fragment to obtain the target web page data that was not crawled during the pre-crawling process of the client, and use the target web page data as the target crawling data, wherein the target web page data includes the download data corresponding to the target data fragment obtained from the web page and the download identifier returned for the target data fragment.

[0067] According to an embodiment of the present disclosure, the above-mentioned data crawling method may also include, for example: obtaining the second target shard identifier shown and its corresponding target web page data; combining the first target shard identifier and the second target shard identifier to obtain a final shard identifier; when the final shard identifier is consistent with the shard identifier stored on the client, determining the web page data pre-crawled by the client based on the partial web page data and the target web page data.

[0068] Through the above-mentioned embodiments of the present disclosure, the problem of data crawling difficulty caused by crawling interruption can be effectively solved, and a simple and effective solution for breakpoint resumption can be provided.

[0069] It should be noted that the data crawling method part applied to the client in the embodiments of the present disclosure corresponds to the data crawling method part applied to the data flow engine in the embodiments of the present disclosure. The description of the data crawling method part applied to the client specifically refers to the data crawling method part applied to the data flow engine, which will not be repeated here.

[0070] It should also be noted that the above-mentioned data flow engine can be set inside the above-mentioned client or outside the above-mentioned client. The difference is that when the data flow engine is set outside the client, it is necessary to maintain a communication relationship between the data flow engine and the client.

[0071] Reference below Figure 4-5 , combined with specific embodiments Figure 2 and Figure 3 The method shown is further explained.

[0072] Figure 4 A system architecture diagram of a normal crawler scenario for implementing a data crawling method according to an embodiment of the present disclosure is schematically shown.

[0073] like Figure 4 As shown in the figure, a normal crawler scenario is depicted, and based on this scenario, the overall process of the client resuming crawling the server (for example, the above-mentioned web page) without crawling interruption is shown. This system includes three participants, namely, the local Python development of the scrapy framework client (such as Figure 4 The client in ), Flink (that is, the above-mentioned data flow engine, which can be expressed as follows Figure 4 Flink in the code) and the Web front-end website (i.e. the web page mentioned above, which can be expressed as follows Figure 4 In this scenario, the overall process of data crawling implemented based on the system may include operations S401 to S406.

[0074] In operation S401, the client pre-crawls Web site data.

[0075] In operation S402, data slices are constructed according to the read data. The data slices obtained in this operation may be first stored in the client, for example.

[0076] In operation S403, the client may further transmit the pre-read data shards to the Flink dataset.

[0077] In operation S404, when using Flink to crawl data from a web page, Flink may first detect through a probe whether it is connected to the web site, and then perform subsequent operations after returning a successful connection message.

[0078] In operation S405, the data can be downloaded through Flink's dataset, and the content returned by the web page includes a download identifier for distinguishing whether the corresponding data slice has completed data download.

[0079] In operation S406, the downloaded data obtained in operation S405 can be stored in Flink in advance in the form of a dataset. By parsing the dataset and sending a download identifier (such as the above download identifier) ​​to the client, it can be determined that the process of using Flink to implement data crawling is completed.

[0080] Figure 5 The system architecture diagram of a breakpoint crawler scenario for implementing a data crawling method according to an embodiment of the present disclosure is schematically shown.

[0081] like Figure 5As shown, a breakpoint crawler scenario is depicted, and based on this scenario, the overall process of the client continuing to crawl the server (for example, the above-mentioned web page) when there is a crawling interruption is shown. Assuming that any node between the Web, the client and Flink is disconnected, here, for example, the interruption occurs between the two nodes of the Web and the client. Since the client can obtain the corresponding data fragments when pre-crawls the Web data and can transmit them to Flink, Flink has already stored the fragment identifiers of the data fragments corresponding to the pre-crawled Web data. In this scenario, the overall process of data crawling implemented based on the system may include operations S501 to S504.

[0082] In operation S501, the client's data shard (configured with a shard identifier) ​​is read and sent to Flink's dataset, and the shard identifiers already stored in Flink (for example, all the shard identifiers mentioned above) are matched with the shard identifiers corresponding to the data shards downloaded locally by the client (for example, the first target shard identifier mentioned above), and the shard identifiers that the client has not downloaded are filtered out.

[0083] In operation S502, the data shards that need to be activated in the Flink dataset are determined according to the shard identifiers that have not been downloaded, and the data shards are sent to the Web end.

[0084] In operation S503, the Web downloads data according to the data slices received from Flink, and feeds back an identified downloading degree (such as the above-mentioned download identifier), and the fed-back identifier is sent to Flink.

[0085] In operation S504, the shard identifier of the data shard corresponding to the downloaded data received by Flink is combined with the shard identifier of the data shard corresponding to the data downloaded by the client, and the data corresponding to the relevant data shard is reassembled, and the data is parsed and sent to the client. At the same time, the downloaded identifier (such as the above download identifier) ​​is sent to match the download identifier of the client. After the matching is completed without any problem (that is, the shard identifier of the data shard configured with the downloaded identifier is completely consistent with the above all shard identifiers), the download is completed.

[0086] It should be noted that interruptions may also occur between the client and Flink, or between Flink and the Web. In this case, the client only needs to crawl data directly from the Web.

[0087] Through the above-mentioned embodiments of the present disclosure, an asynchronous crawler method based on Flink breakpoint resumption is proposed, and the design work of breakpoint resumption is realized through the interaction between Python and Flink. Specifically, the Scrapy framework developed in Python (a fast, high-level screen crawling and web crawling framework for crawling web sites and extracting structured data from pages) and Flink stream data dataset processing are adopted to record asynchronous breakpoints in Flink, and automatically resume crawling through breakpoint records. Among them, the Scrapy framework is mainly used in crawlers, and Flink is used to analyze crawler data streams. The crawler data stream is recorded through Flink. When a breakpoint occurs in the crawler, the crawler can be resumed by reading the data in the Flink crawler data stream. This solution can be applied to crawler scenarios in any scenario.

[0088] Figure 6 A block diagram of a data crawling device applied to a data stream engine according to an embodiment of the present disclosure is schematically shown.

[0089] like Figure 6 As shown, the data crawling device 600 includes an acquisition module 610, a generation module 620 and a sending module 630.

[0090] The acquisition module 610 is used to use the data flow engine to acquire a data set composed of data slices, wherein the data slices are constructed based on the web page data pre-crawled by the client.

[0091] The generation module 620 is used to crawl web page data from the web page end according to the data slices in the data set to generate target crawled data.

[0092] The sending module 630 is used to send the target crawled data to the client, so that the client determines the pre-crawled web page data according to the target crawled data.

[0093] According to the above-mentioned embodiments of the present disclosure, a technical means is adopted by using a data flow engine to obtain a data set composed of data slices, wherein the data slices are constructed according to the web page data pre-crawled by the client; crawling web page data from the web page according to the data slices in the data set to generate target crawled data; and sending the target crawled data to the client so that the client determines the pre-crawled web page data according to the target crawled data. Since the web page data that the client needs to crawl can be crawled according to the data set in the data flow engine, and the data set will not be affected by external interruptions, the technical problem of time-consuming and labor-intensive data crawling process caused by the need to re-crawl when crawling data is interrupted is at least partially overcome, thereby achieving the technical effect of simple and effective breakpoint resumption according to data slices.

[0094] According to an embodiment of the present disclosure, the data slices constructed based on the web page data pre-crawled by the client include multiple data slices, each data slice is configured with a slice identifier, and the generation module 620 includes a first acquisition unit, a second acquisition unit, a screening unit, a crawling unit and a first definition unit.

[0095] The first acquisition unit is used to acquire a first target fragment identifier corresponding to a portion of the web page data crawled by the client when a process of pre-crawl of web page data by the client is interrupted.

[0096] The second acquisition unit is used to acquire all shard identifiers corresponding to multiple data shards in the data set.

[0097] The screening unit is used to match the first target fragment identifier with all fragment identifiers to obtain the second target fragment identifier through screening.

[0098] The crawling unit is used to crawl from the web page end according to the target data segment corresponding to the second target segment identifier to obtain the target web page data that has not been crawled in the pre-crawling process of the client.

[0099] The first definition unit is used to take the target web page data as the target crawling data.

[0100] According to an embodiment of the present disclosure, the generation module 620 further includes a detection unit, a third acquisition unit and a second definition unit.

[0101] The detection unit is used to detect whether a connection is established between the data flow engine and the web page end through a probe.

[0102] The third acquisition unit is used to acquire the download data corresponding to the target data segment and the download identifier returned for the target data segment from the web page when the data flow engine establishes a connection with the web page.

[0103] The second definition unit is used to use the target data segment and the download data and download identifier corresponding to the target data segment as target crawling data.

[0104] According to an embodiment of the present disclosure, the above-mentioned data crawling device 600 also includes a combining module and a determining module.

[0105] The combining module is used to combine the first target fragment identifier and the second target fragment identifier to obtain a final fragment identifier.

[0106] The determination module is used to determine the web page data pre-crawled by the client based on the partial web page data and the target web page data when the final shard identifier is consistent with the shard identifier stored by the client.

[0107] Figure 7A block diagram of a data crawling device applied to a client according to an embodiment of the present disclosure is schematically shown.

[0108] like Figure 7 As shown, the data crawling device 700 includes a first acquisition module 710, a configuration module 720, a storage module 730 and a second acquisition module 740.

[0109] The first acquisition module 710 is used to acquire data fragments constructed based on the web page data pre-crawled by the client.

[0110] The configuration module 720 is used to configure a slice identifier for the data slice.

[0111] The storage module 730 is used to store the data fragments configured with fragment identifiers in the data flow engine in the form of data sets, so that the data flow engine can crawl web page data from the web page end according to the data fragments in the data set to generate target crawled data.

[0112] The second acquisition module 740 is used to acquire target crawled data to determine the web page data pre-crawled by the client according to the target crawled data.

[0113] Through the above-mentioned embodiments of the present disclosure, a method for crawling data from the path of the data flow engine is added, which can effectively solve the problem of inability to completely crawl data due to interruptions during the process of the client directly crawling data from the web page.

[0114] According to an embodiment of the present disclosure, the data fragments constructed based on the web page data pre-crawled by the client include multiple data fragments, and the data crawling device 700 also includes a first determination module, a third acquisition module, a screening module and a second determination module.

[0115] The first determination module is used to determine the first target fragment identifier corresponding to the partial web page data crawled by the client when the process of pre-crawl of web page data by the client is interrupted.

[0116] The third acquisition module is used to obtain all the fragment identifiers of the data fragments.

[0117] The screening module is used to match the first target fragment identifier with all fragment identifiers to screen and obtain the second target fragment identifier.

[0118] The second determination module is used to determine the target data slice corresponding to the second target slice identifier in the data set according to the second target slice identifier, so that the data flow engine can crawl from the web page end according to the target data slice to obtain the target web page data that was not crawled in the pre-crawling process of the client, and use the target web page data as the target crawling data, wherein the target web page data includes the download data corresponding to the target data slice obtained from the web page end and the download identifier returned for the target data slice.

[0119] According to an embodiment of the present disclosure, the above-mentioned data crawling device 700 also includes a fourth acquisition module, a combination module and a third determination module.

[0120] The fourth acquisition module is used to acquire the second target segment identifier and its corresponding target web page data.

[0121] The combining module is used to combine the first target fragment identifier and the second target fragment identifier to obtain a final fragment identifier.

[0122] The third determination module is used to determine the web page data pre-crawled by the client according to the partial web page data and the target web page data when the final fragment identifier is consistent with the fragment identifier stored by the client.

[0123] According to the modules and units of the embodiments of the present disclosure, any multiple or at least part of the functions of any multiple thereof can be implemented in one module. According to the modules and units of the embodiments of the present disclosure, any one or more can be split into multiple modules for implementation. According to the modules and units of the embodiments of the present disclosure, any one or more can be at least partially implemented as a hardware circuit, such as a field programmable gate array (FPGA), a programmable logic array (PLA), a system on a chip, a system on a substrate, a system on a package, an application-specific integrated circuit (ASIC), or can be implemented by hardware or firmware of any other reasonable way of integrating or packaging the circuit, or implemented in any one of the three implementation methods of software, hardware and firmware or in any appropriate combination of any of them. Alternatively, according to the modules and units of the embodiments of the present disclosure, one or more can be at least partially implemented as a computer program module, and when the computer program module is run, the corresponding function can be executed.

[0124] For example, any of the acquisition module 610, the generation module 620 and the sending module 630, or the first acquisition module 710, the configuration module 720, the storage module 730 and the second acquisition module 740 can be combined in one module / unit for implementation, or any of the modules / units can be split into multiple modules / units. Alternatively, at least part of the functions of one or more of these modules / units can be combined with at least part of the functions of other modules / units and implemented in one module / unit. According to an embodiment of the present disclosure, at least one of the acquisition module 610, the generation module 620 and the sending module 630, or the first acquisition module 710, the configuration module 720, the storage module 730 and the second acquisition module 740 can be at least partially implemented as a hardware circuit, such as a field programmable gate array (FPGA), a programmable logic array (PLA), a system on a chip, a system on a substrate, a system on a package, an application specific integrated circuit (ASIC), or can be implemented by hardware or firmware such as any other reasonable way of integrating or packaging the circuit, or by any one of the three implementation methods of software, hardware and firmware or by a suitable combination of any of them. Alternatively, the acquisition module 610, the generation module 620 and the sending module 630, or at least one of the first acquisition module 710, the configuration module 720, the storage module 730 and the second acquisition module 740 can be at least partially implemented as a computer program module, which can perform corresponding functions when the computer program module is executed.

[0125] It should be noted that the data crawling device part in the embodiment of the present disclosure corresponds to the data crawling method part in the embodiment of the present disclosure. The description of the data crawling device part specifically refers to the data crawling method part, which will not be repeated here.

[0126] Figure 8 A block diagram of a computer system suitable for implementing the method described above according to an embodiment of the present disclosure is schematically shown. Figure 8 The computer system shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present disclosure.

[0127] like Figure 8As shown, the computer system 800 according to the embodiment of the present disclosure includes a processor 801, which can perform various appropriate actions and processes according to the program stored in the read-only memory (ROM) 802 or the program loaded from the storage part 808 to the random access memory (RAM) 803. The processor 801 may include, for example, a general-purpose microprocessor (such as a CPU), an instruction set processor and / or a related chipset and / or a special-purpose microprocessor (for example, an application-specific integrated circuit (ASIC)), etc. The processor 801 may also include an onboard memory for caching purposes. The processor 801 may include a single processing unit or multiple processing units for performing different actions of the method flow according to the embodiment of the present disclosure.

[0128] In RAM 803, various programs and data required for the operation of system 800 are stored. Processor 801, ROM 802 and RAM 803 are connected to each other through bus 804. Processor 801 performs various operations of the method flow according to the embodiment of the present disclosure by executing the program in ROM 802 and / or RAM 803. It should be noted that the program can also be stored in one or more memories other than ROM 802 and RAM 803. Processor 801 can also perform various operations of the method flow according to the embodiment of the present disclosure by executing the program stored in the one or more memories.

[0129] According to an embodiment of the present disclosure, the system 800 may further include an input / output (I / O) interface 805, which is also connected to the bus 804. The system 800 may further include one or more of the following components connected to the I / O interface 805: an input section 806 including a keyboard, a mouse, etc.; an output section 807 including a cathode ray tube (CRT), a liquid crystal display (LCD), etc., and a speaker, etc.; a storage section 808 including a hard disk, etc.; and a communication section 809 including a network interface card such as a LAN card, a modem, etc. The communication section 809 performs communication processing via a network such as the Internet. A drive 810 is also connected to the I / O interface 805 as needed. A removable medium 811, such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, etc., is installed on the drive 810 as needed, so that a computer program read therefrom is installed into the storage section 808 as needed.

[0130] According to an embodiment of the present disclosure, the method flow according to an embodiment of the present disclosure can be implemented as a computer software program. For example, an embodiment of the present disclosure includes a computer program product, which includes a computer program carried on a computer-readable storage medium, and the computer program contains a program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from the network through the communication part 809, and / or installed from the removable medium 811. When the computer program is executed by the processor 801, the above-mentioned functions defined in the system of the embodiment of the present disclosure are executed. According to an embodiment of the present disclosure, the system, equipment, device, module, unit, etc. described above can be implemented by a computer program module.

[0131] The present disclosure also provides a computer-readable storage medium, which may be included in the device / apparatus / system described in the above embodiments; or may exist independently without being assembled into the device / apparatus / system. The above computer-readable storage medium carries one or more programs, and when the above one or more programs are executed, the method according to the embodiment of the present disclosure is implemented.

[0132] According to an embodiment of the present disclosure, the computer-readable storage medium may be a non-volatile computer-readable storage medium. For example, it may include, but is not limited to: a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination thereof. In the present disclosure, a computer-readable storage medium may be any tangible medium containing or storing a program that may be used by or in combination with an instruction execution system, apparatus, or device.

[0133] For example, according to an embodiment of the present disclosure, the computer-readable storage medium may include the ROM 802 and / or the RAM 803 described above and / or one or more memories other than the ROM 802 and the RAM 803 .

[0134] An embodiment of the present disclosure also includes a computer program product, which includes a computer program, which contains program code for executing the method provided by the embodiment of the present disclosure. When the computer program product runs on an electronic device, the program code is used to enable the electronic device to implement the data crawling method provided by the embodiment of the present disclosure.

[0135] When the computer program is executed by the processor 801, the above functions defined in the system / device of the embodiment of the present disclosure are executed. According to the embodiment of the present disclosure, the system, device, module, unit, etc. described above can be implemented by a computer program module.

[0136] In one embodiment, the computer program may rely on tangible storage media such as optical storage devices, magnetic storage devices, etc. In another embodiment, the computer program may also be transmitted and distributed in the form of signals on a network medium, and downloaded and installed through the communication part 809, and / or installed from a removable medium 811. The program code contained in the computer program may be transmitted using any appropriate network medium, including but not limited to: wireless, wired, etc., or any suitable combination of the above.

[0137] According to an embodiment of the present disclosure, the program code for executing the computer program provided by the embodiment of the present disclosure can be written in any combination of one or more programming languages. Specifically, these computing programs can be implemented using high-level process and / or object-oriented programming languages, and / or assembly / machine languages. Programming languages ​​include, but are not limited to, Java, C++, python, "C" language or similar programming languages. The program code can be executed entirely on the user computing device, partially on the user device, partially on the remote computing device, or entirely on the remote computing device or server. In the case of a remote computing device, the remote computing device can be connected to the user computing device through any type of network, including a local area network (LAN) or a wide area network (WAN), or can be connected to an external computing device (for example, using an Internet service provider to connect through the Internet).

[0138] The flowcharts and block diagrams in the accompanying drawings illustrate the possible architecture, functions and operations of the systems, methods and computer program products according to various embodiments of the present disclosure. In this regard, each box in the flowchart or block diagram may represent a module, a program segment, or a part of a code, and the above-mentioned module, program segment, or a part of the code contains one or more executable instructions for implementing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box may also occur in an order different from that marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram or flowchart, and the combination of boxes in the block diagram or flowchart, can be implemented with a dedicated hardware-based system that performs the specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions. It can be understood by those skilled in the art that the features recorded in the various embodiments and / or claims of the present disclosure can be combined and / or combined in a variety of ways, even if such a combination or combination is not explicitly recorded in the present disclosure. In particular, without departing from the spirit and teaching of the present disclosure, the features described in the various embodiments and / or claims of the present disclosure may be combined and / or combined in a variety of ways. All of these combinations and / or combinations fall within the scope of the present disclosure.

[0139] The embodiments of the present disclosure are described above. However, these embodiments are only for illustrative purposes and are not intended to limit the scope of the present disclosure. Although the embodiments are described above separately, this does not mean that the measures in the various embodiments cannot be used in combination to advantage. The scope of the present disclosure is defined by the attached claims and their equivalents. Without departing from the scope of the present disclosure, those skilled in the art may make a variety of substitutions and modifications, which should all fall within the scope of the present disclosure.

Claims

1. A data crawling method, comprising: Using a data stream engine to obtain a data set consisting of data slices, wherein the data slices are constructed based on web page data pre-crawled by a client, and the data slices constructed based on the web page data pre-crawled by the client include a plurality of data slices, each of which is configured with a slice identifier; Crawling web page data from the web page end according to the data slices in the data set to generate target crawled data, including: When the process of pre-crawl of the web page data by the client is interrupted, obtaining a first target fragment identifier corresponding to a portion of the web page data crawled by the client; Obtain all shard identifiers corresponding to multiple data shards in the data set; Matching the first target fragment identifier with the all fragment identifiers to obtain a second target fragment identifier; crawling from the web page end the target web page data that was not crawled during the pre-crawling process of the client according to the target data slice corresponding to the second target slice identifier; and Using the target webpage data as the target crawling data; and The target crawled data is sent to the client, so that the client determines the pre-crawled web page data according to the target crawled data.

2. The method according to claim 1, wherein: Crawling web page data from the web page end according to the data slices in the data set, and generating target crawled data further comprises: Detecting whether a connection is established between the data stream engine and the web page end by a probe; When the data stream engine establishes a connection with the web page, obtaining download data corresponding to the target data segment and a download identifier returned for the target data segment from the web page; and The target data segment and the download data and download identifier corresponding to the target data segment are used as the target crawling data.

3. The method according to claim 1 or 2, further comprising: Combining the first target fragment identifier and the second target fragment identifier to obtain a final fragment identifier; as well as When the final fragment identifier is consistent with the fragment identifier stored in the client, the web page data pre-crawled by the client is determined according to the partial web page data and the target web page data.

4. A data crawling method, comprising: Acquire data fragments constructed based on web page data pre-crawled by the client, wherein the data fragments constructed based on the web page data pre-crawled by the client include a plurality of data fragments; Configuring a slice identifier for the data slice; The data slices configured with the slice identifiers are stored in the data flow engine in the form of a data set, so that the data flow engine crawls web page data from the web page end according to the data slices in the data set to generate target crawled data; In the case that the process of the client pre-crawls the web page data is interrupted, determining a first target fragment identifier corresponding to a portion of the web page data crawled by the client; Obtain all shard identifiers of the data shards; Matching the first target fragment identifier with the all fragment identifiers to obtain a second target fragment identifier; Determine the target data shard corresponding to the second target shard identifier in the data set according to the second target shard identifier, so that the data flow engine crawls from the web page end according to the target data shard to obtain the target web page data that was not crawled in the pre-crawling process of the client, and uses the target web page data as the target crawled data; as well as The target crawling data is obtained to determine the web page data pre-crawled by the client according to the target crawling data.

5. The method according to claim 4, wherein: The target web page data includes download data corresponding to the target data segment obtained from the web page end and a download identifier returned for the target data segment.

6. The method according to claim 5, further comprising: Obtain the second target segment identifier and its corresponding target web page data; Combining the first target fragment identifier and the second target fragment identifier to obtain a final fragment identifier; as well as When the final fragment identifier is consistent with the fragment identifier stored in the client, the web page data pre-crawled by the client is determined according to the partial web page data and the target web page data.

7. A data crawling device, comprising: An acquisition module, used to acquire a data set composed of data slices by using a data stream engine, wherein the data slices are constructed based on web page data pre-crawled by a client, and the data slices constructed based on the web page data pre-crawled by the client include a plurality of data slices, each of which is configured with a slice identifier; A generation module is used to crawl web page data from the web page end according to the data slices in the data set to generate target crawled data, and the generation module includes: A first acquisition unit, configured to acquire, when a process in which the client pre-crawls the web page data is interrupted, a first target fragment identifier corresponding to a portion of the web page data crawled by the client; A second acquisition unit, used to acquire all shard identifiers corresponding to the multiple data shards in the data set; A screening unit, configured to match the first target fragment identifier with the all fragment identifiers to obtain a second target fragment identifier through screening; A crawling unit, configured to crawl from the web page end, according to the target data slice corresponding to the second target slice identifier, target web page data that has not been crawled in the pre-crawling process of the client; and A first definition unit, configured to use the target webpage data as the target crawling data; and A sending module is used to send the target crawled data to the client, so that the client determines the pre-crawled web page data according to the target crawled data.

8. A data crawling device, comprising: A first acquisition module is used to acquire data fragments constructed based on the web page data pre-crawled by the client, wherein the data fragments constructed based on the web page data pre-crawled by the client include a plurality of data fragments; A configuration module, used to configure a slice identifier for the data slice; A storage module, used for storing the data fragments configured with the fragment identifiers in the data flow engine in the form of a data set, so that the data flow engine crawls web page data from the web page end according to the data fragments in the data set to generate target crawled data; A first determination module, configured to determine, when a process in which the client pre-crawls the web page data is interrupted, a first target fragment identifier corresponding to a portion of the web page data crawled by the client; A third acquisition module is used to obtain all the fragment identifiers of the data fragments; A screening module, used for matching the first target fragment identifier with the all fragment identifiers to screen and obtain a second target fragment identifier; A second determination module is used to determine the target data shard corresponding to the second target shard identifier in the data set according to the second target shard identifier, so that the data flow engine crawls from the web page end according to the target data shard to obtain the target web page data that has not been crawled in the pre-crawling process of the client, and uses the target web page data as the target crawled data; as well as The second acquisition module is used to acquire the target crawled data to determine the web page data pre-crawled by the client according to the target crawled data.

9. A computer system comprising: one or more processors; a memory for storing one or more programs, Wherein, when the one or more programs are executed by the one or more processors, the one or more processors implement the method of any one of claims 1 to 3 or 4 to 6.

10. A computer-readable storage medium having executable instructions stored thereon, which, when executed by a processor, enables the processor to implement the method according to any one of claims 1 to 3 or 4 to 6.

11. A computer program product comprising computer executable instructions, which when executed are used to implement the method of any one of claims 1 to 3 or 4 to 6.

Images