Map data acquisition method, computer readable storage medium and terminal device

Abstract

The present invention provides a map data acquisition method, a computer-readable storage medium and a terminal device, including: acquiring the boundary of a target area; determining the crawling area of ​​a web crawler according to the boundary of the target area, the crawling area being a closed area composed of multiple line segments, and the crawling area including the target area; determining the linear function of the straight line corresponding to any line segment of the multiple line segments in the map coordinate system corresponding to the third-party map; establishing the boundary constraint condition for the web crawler to crawl the map data of the third-party map according to the linear function corresponding to the multiple line segments; calling the interface of the third-party map, and crawling the map data of the third-party map through the web crawler according to the boundary constraint condition. Crawling the latest map data through the web crawler improves the real-time performance of acquiring map data, and by determining the boundary of the crawling area of ​​the web crawler, it is avoided that the crawler crawls a large amount of invalid data, and the calculation amount of the server is reduced.

Description

Technical Field

[0001] The present invention belongs to the field of computer technology, and in particular relates to a map data acquisition method, a computer-readable storage medium and a terminal device. Background Art

[0002] With the rapid development of computer network technology and geographic information technology, geographic information data, as an important basic information data resource, has been widely used not only in national construction fields such as urban planning, traffic navigation, disaster monitoring, national defense and military, but also has become increasingly closely linked to people's daily lives. For example, from mobile phone maps and car navigation maps used in people's daily travel to high-speed railways, airports, urban planning and construction in infrastructure, all rely on the support of geographic information.

[0003] In many scenarios, it is necessary to obtain map data with higher real-time performance. However, in the existing technology, geographic information can only be obtained through geographic information service providers, and offline maps can only be obtained through the official download channels provided by the service providers. Offline maps have high latency and are quite different from the actual environment when the information is obtained. Therefore, how to obtain map data with higher real-time performance is a problem that needs to be solved urgently. Summary of the invention

[0004] In view of this, an embodiment of the present invention provides a map data acquisition method, a computer-readable storage medium, and a terminal device to solve the problem of poor real-time performance of map data acquired in the prior art.

[0005] A first aspect of an embodiment of the present invention provides a method for obtaining map data, comprising:

[0006] Get the boundary of the target area;

[0007] Determine a crawling area of ​​the network crawler according to the boundary of the target area, wherein the crawling area is a closed area formed by a plurality of line segments, and the crawling area includes the target area;

[0008] For a straight line corresponding to any line segment among the plurality of line segments, determining a linear function of the straight line in a map coordinate system corresponding to the third-party map;

[0009] Establishing boundary constraint conditions for a web crawler to crawl map data of the third-party map according to the linear functions corresponding to the plurality of line segments;

[0010] The interface of the third-party map is called, and according to the boundary constraint condition, the map data of the third-party map is crawled by a web crawler.

[0011] A second aspect of an embodiment of the present invention provides a distributed web crawler system, the system comprising a control server and multiple data acquisition servers, the control server being connected to each data acquisition server respectively, and the system being used for:

[0012] The control server obtains the boundary of the target area;

[0013] The control server determines a crawling area of ​​the network crawler according to the boundary of the target area, wherein the crawling area is a closed area formed by a plurality of line segments, and the crawling area includes the target area;

[0014] The control server determines, for a straight line corresponding to any line segment among the plurality of line segments, a linear function of the straight line in a map coordinate system corresponding to a third-party map;

[0015] The control server establishes boundary constraint conditions for the web crawler to crawl the map data of the third-party map according to the linear functions corresponding to the plurality of line segments;

[0016] The data collection server calls the interface of the third-party map and crawls the map data of the third-party map through a web crawler according to the boundary constraint condition.

[0017] A third aspect of an embodiment of the present invention provides a computer-readable storage medium, wherein the computer-readable storage medium stores computer-readable instructions, and when the computer-readable instructions are executed by a processor, the following steps are implemented:

[0018] Get the boundary of the target area;

[0019] Determine a crawling area of ​​the network crawler according to the boundary of the target area, wherein the crawling area is a closed area formed by a plurality of line segments, and the crawling area includes the target area;

[0020] For a straight line corresponding to any line segment among the plurality of line segments, determining a linear function of the straight line in a map coordinate system corresponding to the third-party map;

[0021] Establishing boundary constraint conditions for a web crawler to crawl map data of the third-party map according to the linear functions corresponding to the plurality of line segments;

[0022] The interface of the third-party map is called, and according to the boundary constraint condition, the map data of the third-party map is crawled by a web crawler.

[0023] A fourth aspect of an embodiment of the present invention provides a terminal device, including a memory, a processor, and a computer-readable instruction stored in the memory and executable on the processor, wherein the processor implements the following steps when executing the computer-readable instruction:

[0024] Get the boundary of the target area;

[0025] Determine a crawling area of ​​the network crawler according to the boundary of the target area, wherein the crawling area is a closed area formed by a plurality of line segments, and the crawling area includes the target area;

[0026] For a straight line corresponding to any line segment among the plurality of line segments, determining a linear function of the straight line in a map coordinate system corresponding to the third-party map;

[0027] Establishing boundary constraint conditions for a web crawler to crawl map data of the third-party map according to the linear functions corresponding to the plurality of line segments;

[0028] The interface of the third-party map is called, and according to the boundary constraint condition, the map data of the third-party map is crawled by a web crawler.

[0029] The present invention provides a map data acquisition method, a computer-readable storage medium and a terminal device, which crawl the latest map data through a network crawler, thereby improving the real-time performance of acquiring map data, and by determining the boundary of the crawling area of ​​the network crawler, avoids the crawler from crawling a large amount of invalid data, thereby reducing the computing power of the server. BRIEF DESCRIPTION OF THE DRAWINGS

[0030] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.

[0031] Figure 1 A schematic diagram of a flow chart of a method for obtaining map data provided by an embodiment of the present invention;

[0032] Figure 2 A schematic diagram of a target area provided by an embodiment of the present invention;

[0033] Figure 3 A schematic diagram of a network crawler crawling area provided by an embodiment of the present invention;

[0034] Figure 4a A schematic diagram of a map grid provided by an embodiment of the present invention;

[0035] Figure 4b A schematic diagram of another network crawler crawling area provided by an embodiment of the present invention;

[0036] Figure 5A schematic flow chart of a method for crawling map data of a third-party map by using a web crawler provided in an embodiment of the present invention;

[0037] Figure 6 A schematic flow chart of another method for crawling map data of a third-party map by using a web crawler provided in an embodiment of the present invention;

[0038] Figure 7 A structural block diagram of a distributed network crawler system provided by an embodiment of the present invention;

[0039] Figure 8 A schematic diagram of a terminal device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0040] In the following description, specific details such as specific system structures, technologies, etc. are provided for the purpose of illustration rather than limitation, so as to provide a thorough understanding of the embodiments of the present invention. However, it should be clear to those skilled in the art that the present invention may be implemented in other embodiments without these specific details. In other cases, detailed descriptions of well-known systems, devices, circuits, and methods are omitted to prevent unnecessary details from obstructing the description of the present invention.

[0041] In order to illustrate the technical solution of the present invention, a specific embodiment is provided below for illustration.

[0042] The embodiment of the present invention provides a method for obtaining map data. Figure 1 , the method comprising:

[0043] S101, obtaining the boundary of the target area.

[0044] By entering the name of the target area through a third-party geographic information data service website, you can obtain the boundaries of the target area. Figure 2 The figure shows the boundary of a target area. The longitude and latitude of each point on the boundary of the target area can be obtained by clicking.

[0045] S102: Determine a crawling area of ​​the network crawler according to the boundary of the target area, wherein the crawling area is a closed area formed by a plurality of line segments, and the crawling area includes the target area.

[0046] like Figure 1 As shown, the boundary of the target area is irregular and cannot be limited by a function. To traverse geographic information based on spatial location through a web crawler, the crawling area of ​​the web crawler must first be determined, and the boundary of the crawling area can be represented by a function obtained by calculation.

[0047] Specifically, the embodiments of the present invention provide two methods for determining the crawling area of ​​a network crawler according to the boundary of the target area:

[0048] The first method is: selecting multiple points on the boundary of the target area, connecting any two adjacent points among the multiple points, and obtaining a closed area, which is the crawling area of ​​the web crawler.

[0049] like Figure 3 As shown, seven points A to G are selected on the boundary of the target area, and A→B→C→D→E→F→G→A are linked in sequence. The resulting closed area is the crawling area of ​​the web crawler.

[0050] The second method is to establish a grid on a third-party map, wherein the grid is composed of a plurality of continuous but non-overlapping rectangular areas; determine a plurality of rectangular areas in the grid according to the boundaries of the grid and the target area, wherein, for any rectangular area, the rectangular area has an overlapping portion with the target area; and determine the crawling area of ​​the web crawler according to the plurality of rectangular areas.

[0051] Specifically, Figure 4a As shown, a grid is established on the third map, and the grid size can be customized. For example, each small rectangular area in the grid represents an area of ​​1km×1km, and the area of ​​each small rectangular area in the grid is the same. According to the grid establishment rules, the longitude and latitude of the four boundaries of each small rectangular area can be determined.

[0052] Determine the boundaries of the crawling area in the grid. Optionally, determine all small rectangular areas in the grid that overlap with the target area, such as Figure 4b As shown, Figure 4b The area formed by all the small rectangular areas that overlap with the target area is the crawling area of ​​the web crawler.

[0053] S103: For a straight line corresponding to any line segment among the plurality of line segments, determine a linear function of the straight line in a map coordinate system corresponding to the third-party map.

[0054] Optionally, if the first method for determining the crawling area in step S102 is adopted, since the longitude and latitude of multiple points selected in the target area can be directly obtained through a third-party map, such as the longitude and latitude of point A to point G are known, the linear function of the straight line corresponding to the two points can be obtained when the map coordinate system is determined, such as the linear function used to represent the straight line AB, the linear function used to represent the straight line BC, etc.

[0055] Alternatively, if the second method of determining the crawling area in step S102 is adopted, when the map coordinate system is determined, since the longitude and latitude corresponding to each small rectangular area in the grid are determined, the linear function of each corresponding straight line in the boundary of the crawling area is also determined.

[0056] S104: establishing boundary constraint conditions for a web crawler to crawl map data of the third-party map according to the linear functions corresponding to the plurality of line segments.

[0057] Optionally, if the first method of determining the crawling area in step S102 is adopted, at this time, the boundary constraint condition for the web crawler to crawl the third-party map data is the closed area corresponding to the multiple straight lines corresponding to the boundary of the crawling area determined in step S103;

[0058] If the second method for determining the crawling area in step S102 is adopted, at this time, the boundary constraint conditions of the map data of the third-party map crawled by the web crawler are multiple straight lines corresponding to the boundary of the crawling area and the intersection points of the straight lines.

[0059] S105, calling the interface of the third-party map, and crawling the map data of the third-party map through a web crawler according to the boundary constraint condition.

[0060] In an embodiment of the present invention, two methods are provided for a web crawler to crawl map data of a third-party map:

[0061] Combination Figure 5 ,The first method of crawling map data of third-party maps includes:

[0062] S1051, create a building list, a road list and an intersection list.

[0063] S1052, selecting a starting point in the target area, and crawling all buildings, roads and intersections within a circle with a radius of a preset length and the starting point as the center through a web crawler.

[0064] S1053, storing the crawled buildings into the building list, storing the roads into the road list, and storing the intersections into the intersection list.

[0065] S1054, traversing the building list, for any building in the building list, crawling all buildings, roads and intersections within the range of a circle with the building as the center and the preset length as the radius through a web crawler.

[0066] S1055, store the newly crawled buildings that do not belong to the building list into the building list, store the newly crawled roads that do not belong to the road list into the road list, and store the newly crawled intersections that do not belong to the intersection list into the intersection list, until no new buildings appear in the building list and each building in the building list is traversed once.

[0067] Specifically, this method is mainly used to crawl building information, road information and intersection information. This type of information is mainly discrete data, including data type information and data information. The web crawler crawls data according to the data type. In an embodiment of the present invention, three information lists are preset, namely, a building list, a road list and an intersection list, and the initial states of these three lists are empty. A starting point for the web crawler to crawl map information is selected in the target area. Optionally, the starting point is a point near the center of the target area.

[0068] The web crawler crawls all buildings, roads and intersections within the range of a circle with the starting point as the center and a preset length as the radius. For example, the starting point is point A, and the web crawler crawls all buildings, roads and intersections within the range of a circle with the starting point A as the center and a preset length, such as 1 km, as the radius, and stores the crawled buildings in the building list, the roads in the road list, and the intersections in the intersection list.

[0069] Afterwards, the building list is traversed, and for any building in the building list, all buildings, roads and intersections within the range of a circle with the building as the center and the preset length as the radius are crawled by a web crawler. For example, all buildings within the range of a circle with A as the center and 1 km as the radius are crawled by a web crawler to obtain 5 buildings, and the information of these 5 buildings is put into the building list. Afterwards, all buildings, roads and intersections within the range of 5 circles with the 5 buildings as the center and 1 km as the radius are crawled by a web crawler. The acquired new buildings are put into the building list, the new roads are put into the road list, and the new intersections are put into the intersection list.

[0070] After that, the newly added building is used as the center of the circle and the preset distance is used as the radius to continue crawling buildings. For roads and intersections, the newly crawled buildings that do not belong to the building list are stored in the building list, the newly crawled roads that do not belong to the road list are stored in the road list, and the newly crawled intersections that do not belong to the intersection list are stored in the intersection list, until no new buildings appear in the building list and each building in the building list is traversed once.

[0071] Combination Figure 6 ,The second method of crawling map data of third-party maps includes:

[0072] S1056, creating a search word library, wherein the search word library contains a plurality of search words, and the search words are used to represent crawling targets of the web crawler.

[0073] Specifically, for the Chinese word library, it can be classified and refined according to the commonly used search terms. For example, the search content can be divided into multiple categories, such as catering, shopping, accommodation, travel, cultural and sports entertainment, life services, education, medical care, enterprises and institutions, public service facilities, real estate, etc. Each category is subdivided into keywords, and the subdivided keywords of real estate are building, office building, community, apartment, etc.

[0074] S1057, for any search term, call a third-party map interface, input the search term, and obtain a uniform resource locator URL list.

[0075] Specifically, for any search term in the search term library, call the third-party map interface, enter the search term, and the third-party map displays all relevant geographic information obtained based on the search term to obtain a list. Through the list, obtain the URL (Uniform Resource Locator) corresponding to each geographic information to obtain a uniform resource locator list.

[0076] S1058: crawl the map data in the URL list through a web crawler according to the boundary constraint condition.

[0077] According to the URLs in the URL list, the map data corresponding to each URL is crawled in turn by a web crawler.

[0078] The embodiment of the present invention provides a method for obtaining map data. The method crawls the latest map data through a network crawler, thereby improving the real-time performance of obtaining map data. By determining the boundary of the crawling area of ​​the network crawler, the crawler is prevented from crawling a large amount of invalid data, thereby reducing the computational workload of the server.

[0079] Combination Figure 7 The embodiment of the present invention provides a distributed web crawler system, which includes: a control server 71 and multiple data acquisition servers 72, wherein the control server 71 is connected to each data acquisition server 72, and the system is used to:

[0080] The control server 71 obtains the boundary of the target area;

[0081] The control server 71 determines the crawling area of ​​the web crawler according to the boundary of the target area, wherein the crawling area is a closed area formed by a plurality of line segments, and the crawling area includes the target area;

[0082] The control server 71 determines, for a straight line corresponding to any line segment among the plurality of line segments, a linear function of the straight line in a map coordinate system corresponding to the third-party map;

[0083] The control server 71 establishes boundary constraint conditions for the web crawler to crawl the map data of the third-party map according to the linear functions corresponding to the plurality of line segments;

[0084] The data collection server 72 calls the interface of the third-party map and crawls the map data of the third-party map through a web crawler according to the boundary constraint condition.

[0085] Further, the control server 71 is also used to: divide the crawling area into N sub-areas, and determine the boundary constraint conditions of each sub-area in turn, select N data acquisition servers 72, and send a subtask request message to each data acquisition server 72 in turn, wherein the subtask request message includes a boundary constraint condition of a sub-area, and the boundary constraint conditions of the sub-area carried in the subtask request messages received by any two data acquisition servers 72 are different, and N is a positive integer greater than or equal to 2;

[0086] For any data acquisition server 72 among the N data acquisition servers 72, the data acquisition server 72 is used to: call the interface of the third-party map, and crawl the map data corresponding to the sub-area in the third-party map through a web crawler according to the boundary constraints of the sub-area in the sub-task request message.

[0087] Further, the control server 71 is also used to: divide the search terms in the search term database into N groups, each group contains at least one search term, and no duplicate search terms are contained between any two groups; select N data acquisition servers 72, and send a subtask request message to each data acquisition server 72 in turn, wherein the subtask request message contains a group of search terms and boundary constraints corresponding to the crawling area; the search terms carried in the subtask request messages received by any two data acquisition servers 72 are different, and N is a positive integer greater than or equal to 2;

[0088] For any one of the N data acquisition servers 72, the data acquisition server 72 is used to: call the interface of the third-party map, and crawl the map data corresponding to the search term in the third-party map through a web crawler according to the search term in the subtask request message and the boundary constraints corresponding to the crawling area.

[0089] The embodiment of the present invention provides a distributed network crawler system for implementing Figure 1 , Figure 6 and Figure 7 In the method of the corresponding embodiment, the system crawls the latest map data through a web crawler, thereby improving the real-time performance of obtaining map data, and by determining the boundary of the crawler crawling area, the crawler is prevented from crawling a large amount of invalid data, thereby reducing the amount of calculation of the server. In addition, the system controls the server to distribute tasks, so that multiple data collection servers crawl map data at the same time, thereby improving the efficiency of the web crawler crawling map data.

[0090] Figure 8 This is a schematic diagram of a map data acquisition terminal device provided in Embodiment 4 of the present invention. Figure 8 As shown, the map data acquisition terminal device 8 of this embodiment includes: a processor 80, a memory 81, and a computer program 82 stored in the memory 81 and executable on the processor 80, such as a map data acquisition program. When the processor 80 executes the computer program 82, the steps in the above-mentioned map data acquisition method embodiments are implemented, such as Figure 1 Steps 101 to 105 shown, or Figure 5 Steps 1051 to 1055 shown, or Figure 6 1056 to 1058 shown.

[0091] Exemplarily, the computer program 82 may be divided into one or more modules / units, which are stored in the memory 81 and executed by the processor 80 to implement the present invention. The one or more modules / units may be a series of computer program instruction segments capable of implementing specific functions, which are used to describe the execution process of the computer program 82 in the map data acquisition terminal device 8.

[0092] The map data acquisition terminal device 8 may be a computing device such as a desktop computer, a notebook, a PDA, or a cloud server. The map data acquisition terminal device may include, but is not limited to, a processor 80 and a memory 81. Those skilled in the art will appreciate that Figure 8It is only an example of the map data acquisition terminal device 8 and does not constitute a limitation of the map data acquisition terminal device 8. It may include more or fewer components than shown in the figure, or a combination of certain components, or different components. For example, the map data acquisition terminal device may also include input and output devices, network access devices, buses, etc.

[0093] The processor 80 may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or any conventional processor, etc.

[0094] The memory 81 may be an internal storage unit of the map data acquisition terminal device 8, such as a hard disk or memory of the map data acquisition terminal device 8. The memory 81 may also be an external storage device of the map data acquisition terminal device 8, such as a plug-in hard disk, a smart memory card (SmartMedia Card, SMC), a secure digital (Secure Digital, SD) card, a flash card (Flash Card), etc. equipped on the map data acquisition terminal device 8. Further, the memory 81 may also include both an internal storage unit and an external storage device of the map data acquisition terminal device 8. The memory 81 is used to store the computer program and other programs and data required by the map data acquisition terminal device. The memory 81 may also be used to temporarily store data that has been output or is to be output.

[0095] An embodiment of the present invention further provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps of the map data acquisition method described in any of the above embodiments are implemented.

[0096] In addition, each functional unit in each embodiment of the present invention may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0097] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product. The computer software product is stored in a storage medium, including several instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk, etc. Various media that can store program codes.

[0098] The embodiments described above are only used to illustrate the technical solutions of the present invention, rather than to limit the same. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that the technical solutions described in the aforementioned embodiments may still be modified, or some of the technical features may be replaced by equivalents. Such modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention, and should all be included in the protection scope of the present invention.

Claims

1. A method for obtaining map data, characterized in that: The method includes: Get the boundary of the target area; Determine a crawling area of ​​the network crawler according to the boundary of the target area, wherein the crawling area is a closed area formed by a plurality of line segments, and the crawling area includes the target area; For a straight line corresponding to any line segment among the plurality of line segments, determining a linear function of the straight line in a map coordinate system corresponding to the third-party map; Establishing boundary constraint conditions for a web crawler to crawl map data of the third-party map according to the linear functions corresponding to the plurality of line segments; Calling the interface of the third-party map, and crawling the map data of the third-party map through a web crawler according to the boundary constraint condition; The map data of the third-party map crawled by a web crawler includes: Create a list of buildings, a list of roads, and a list of intersections; Select a starting point in the target area, and crawl all buildings, roads and intersections within a circle with a radius of a preset length and the starting point as the center through a web crawler; The crawled buildings are stored in the building list, the roads are stored in the road list, and the intersections are stored in the intersection list, and the initial state of each list is empty; Traversing the building list, for any building in the building list, crawling all buildings, roads and intersections within the range of a circle with the building as the center and the preset length as the radius through a web crawler; The newly crawled buildings that do not belong to the building list are stored in the building list, the newly crawled roads that do not belong to the road list are stored in the road list, and the newly crawled intersections that do not belong to the intersection list are stored in the intersection list, until no new buildings appear in the building list and each building in the building list is traversed once; Alternatively, the search terms in the search term library are divided into N groups, each group contains at least one search term, and no repeated search terms are contained between any two groups. N data acquisition servers are selected, and a subtask request message is sent to each data acquisition server in turn, wherein the subtask request message contains a group of search terms and boundary constraints corresponding to the crawling area, and the search terms carried in the subtask request messages received by any two data acquisition servers are different, and N is a positive integer greater than or equal to 2; for any data acquisition server among the N data acquisition servers, the data acquisition server is used to: call the interface of the third-party map, and crawl the map data corresponding to the search terms in the third-party map through a web crawler according to the search terms in the subtask request message and the boundary constraints corresponding to the crawling area.

2. The map data acquisition method according to claim 1, characterized in that: Determining the crawling area of ​​the network crawler according to the boundary of the target area includes: Selecting a plurality of points on the boundary of the target area; Any two adjacent points among the multiple points are connected to obtain a closed area, and the closed area is the crawling area of ​​the web crawler.

3. The map data acquisition method according to claim 1, characterized in that: Determining the crawling area of ​​the network crawler according to the boundary of the target area includes: Creating a grid on the third-party map, the grid consisting of a plurality of continuous but non-overlapping rectangular areas; Determine a plurality of rectangular areas in the grid according to the boundaries of the grid and the target area, wherein for any rectangular area, the rectangular area has an overlapping portion with the target area; A crawling area of ​​the network crawler is determined according to the multiple rectangular areas.

4. A distributed web crawler system, characterized in that: The distributed web crawler system comprises a control server and multiple data acquisition servers, wherein the control server is connected to each data acquisition server respectively, and the system is used for: The control server obtains the boundary of the target area; The control server determines a crawling area of ​​the network crawler according to the boundary of the target area, wherein the crawling area is a closed area formed by a plurality of line segments, and the crawling area includes the target area; The control server determines, for a straight line corresponding to any line segment among the plurality of line segments, a linear function of the straight line in a map coordinate system corresponding to a third-party map; The control server establishes boundary constraint conditions for the web crawler to crawl the map data of the third-party map according to the linear functions corresponding to the plurality of line segments; The data collection server calls the interface of the third-party map and crawls the map data of the third-party map through a web crawler according to the boundary constraint condition; Wherein, crawling the map data of the third-party map by a web crawler includes: Create a list of buildings, a list of roads, and a list of intersections; Select a starting point in the target area, and crawl all buildings, roads and intersections within a circle with a radius of a preset length and the starting point as the center through a web crawler; The crawled buildings are stored in the building list, the roads are stored in the road list, and the intersections are stored in the intersection list, and the initial state of each list is empty; Traversing the building list, for any building in the building list, crawling all buildings, roads and intersections within the range of a circle with the building as the center and the preset length as the radius through a web crawler; The newly crawled buildings that do not belong to the building list are stored in the building list, the newly crawled roads that do not belong to the road list are stored in the road list, and the newly crawled intersections that do not belong to the intersection list are stored in the intersection list, until no new buildings appear in the building list and each building in the building list is traversed once; Alternatively, the control server is further used to: divide the search terms in the search term library into N groups, each group containing at least one search term, and no repeated search terms are contained between any two groups; select N data acquisition servers, and send a subtask request message to each data acquisition server in turn, wherein the subtask request message contains a group of search terms and boundary constraints corresponding to the crawling area, the search terms carried in the subtask request messages received by any two data acquisition servers are different, and N is a positive integer greater than or equal to 2; for any data acquisition server among the N data acquisition servers, the data acquisition server is used to: call the interface of the third-party map, and crawl the map data corresponding to the search terms in the third-party map through a web crawler according to the search terms in the subtask request message and the boundary constraints corresponding to the crawling area.

5. The distributed web crawler system according to claim 4, characterized in that: The control server is also used for: The crawling area is divided into N sub-areas, and the boundary constraint condition of each sub-area is determined in turn, N data acquisition servers are selected, and a subtask request message is sent to each data acquisition server in turn, wherein the subtask request message includes a boundary constraint condition of a sub-area, and the boundary constraint conditions of the sub-area carried in the subtask request messages received by any two data acquisition servers are different, and N is a positive integer greater than or equal to 2; For any data acquisition server among the N data acquisition servers, the data acquisition server is used to: call the interface of the third-party map, and crawl the map data corresponding to the sub-area in the third-party map through a network crawler according to the boundary constraints of the sub-area in the sub-task request message.

6. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 3 are implemented.

7. A terminal device, characterized in that: The terminal device includes a memory and a processor, wherein the memory stores a computer program that can be run on the processor, and the processor implements the steps of the method according to any one of claims 1 to 3 when executing the computer program.

Images