A detection and response method and device for geographic information tile crawler program

Abstract

The present invention discloses a detection and response method and device for a geographic information tile crawler program, the method comprising: S1, constructing a tile request response program; S2, checking the historical records of user access to tiles; S3, time aggregation behavior discrimination; S4, space aggregation behavior discrimination; S5, suspending the output of available tiles and generating warnings. The present invention comprehensively utilizes time and space aggregation, user behavior detection, access behavior restriction, noise interference crawling and other technologies, and can detect abnormal behaviors of users performing high-frequency intensive access and spatial contiguous access to geographic information tiles. Since crawlers can be discovered in a timely and accurate manner, and measures such as suspending the provision of correct tiles, generating warning tile information, and adding noise to interfere with the client extracting available tiles are taken for crawler program users, the rights and interests of geographic information data providers are protected.

Description

Technical Field

[0001] The present invention belongs to the field of geographic information technology, and more specifically, relates to a detection and response method and device for a geographic information tile crawler program. Background Art

[0002] Geographic information is a type of data with geographic spatial locations. Tiling is a discrete data organization method formed by spatial segmentation of continuous geographic information. A geographic information tile dataset is a data set of discrete geographic information tiles. Geographic information tiles divide the geographic information data range into a two-dimensional array of multiple tile units. For a tile set with K levels, M columns, and N rows, it is recorded as (L K， C M， R N ). A specific geographic information tile is identified by (L, C, R), where L represents the scale of the level, and C and R represent the vertical column number and horizontal row number of the tile unit starting from the origin, respectively.

[0003] Providing geographic information tile services is one of the main ways to share geographic information in the HTTP protocol in Web application scenarios. Since geographic information tiles are a kind of data resource, they also have production factor value. There are inconsistent purposes between the service providers and data users of geographic information data: data service providers hope that the content of geographic information data is "available but not taken", "available but not redistributed", and "local area data can be used instead of downloading all data"; data users may want to use as much data as possible, perhaps to exceed the original purpose of the data service provider, and even download and redistribute data. Therefore, data users may adopt crawling behavior, that is, using crawlers to continuously request tiles from the client through HTTP, and obtain spatially continuous tiles from the Web server in a way of accumulating small amounts. Such behavior of data users is contrary to the expectations of data service providers and may cause damage to the interests of data providers.

[0004] For geographic information tilesets (L K， C M， R N ), the larger K is, the larger the map scale is, the higher the accuracy of the geographic information of the tile content at this level is, and the higher the value of the geographic information of its continuous range is.

[0005] The behavior of crawling geographic information tiles generally refers to the behavior of continuously requesting (reaching a certain time span) geographic information online tiles through computer programs to obtain spatially continuous (reaching a certain area) geographic information data. Such programs that continuously crawl geographic information data are defined as crawler programs. If such crawling behavior is not authorized by the data provider, it is defined as unauthorized crawling.

[0006] In order to maintain the original expectations of geographic information data service providers and protect the interests of data service providers, it is necessary to diagnose and discover the crawling behavior of data users who continuously request tiles from the client through HTTP, prevent a larger range of data from being crawled, and impose restrictions and warnings on the client.

[0007] The current status of relevant technical means in online geographic information services is as follows:

[0008] 1) Conventional geographic information service basic software does not include user access behavior feature detection. Conventional geographic information service basic software (such as SuperMap, ArcGIS Server, GeoServer, etc.) can publish geographic information tiles, but does not have user behavior feature detection technology. Therefore, if the data user (client) crawls the geographic information tiles published by the above software (SuperMap, etc.) through a crawler program, the data provider cannot detect the crawler program and its source, nor can it effectively respond to the behavior of the crawler program.

[0009] 2) Use geographic information tile watermark and its detection technology. Add additional information with characteristics (such as the ownership unit of the data, application purpose, and user unit) to the geographic information tile. This additional information is called a watermark. When the data service user requests the tile from the Web server, the watermark is sent to the client along with the tile. If the data provider believes that the data is used unexpectedly (including being crawled), the data obtained by the data user can be detected. If the watermark information characteristics added by the data provider are detected, it proves that the data obtained by the data provider comes from the data service provider. However, watermark technology is a "post-monitoring behavior" after crawling, and requires data users to accept the detection of suspected unauthorized data.

[0010] 3) Use a control method that limits the number of accesses within a time period. For example, online map service platforms such as Baidu Maps and Tiandi Maps limit the number of tile accesses per day to authorized users, and the number of times a user can access a tile per day is usually set at 10,000 or tens of thousands. The data service program only restricts access after reaching the number of accessible times. Therefore, if the data user uses it to obtain continuous geographic information tiles, data may still be crawled. 10,000 requests may result in 10,000 geographic information tiles being crawled and stored on the client. At the same time, the purpose of limiting the number of user accesses confuses users with high access volume and crawling data behavior. Users are restricted from accessing, which may not be the behavior of crawling geographic information tiles, but just a large number of visits. Crawling geographic information tiles in continuous space may lead to a large number of visits, but a large number of visits does not necessarily mean that there is a behavior of crawling tiles. Therefore, the main defect of this technology is that it does not take into account the continuity of the crawled geographic information tiles in space.

[0011] In summary, the detection of geographic information tile crawler programs in the prior art mainly has the following defects:

[0012] 1) The crawling behavior of the crawler cannot be discovered;

[0013] 2) The evidence collection conditions for crawling behavior are too complex;

[0014] 3) Failure to detect crawling behavior in a timely manner;

[0015] 4) To prevent tiles from being crawled, clients with high traffic may be mistakenly restricted;

[0016] 5) The crawled tile data can be easily used on the client. Summary of the invention

[0017] Purpose of the invention: The purpose of the present invention is to address the deficiencies in the prior art and to provide a detection and response method and device for a geographic information tile crawler program, which can timely detect the behavior of users crawling geographic information tiles in the process of providing geographic information services, and restrict the behavior of users crawling geographic information tiles without authorization, thereby protecting the interests of data providers and ensuring the availability of geographic information tile services.

[0018] Technical solution: A detection and response method for a geographic information tile crawler program according to the present invention comprises:

[0019] S1, construct the request response program of the tile;

[0020] S2, check the history of user access to tiles;

[0021] S3, temporal aggregation behavior discrimination;

[0022] S4, spatial aggregation behavior discrimination;

[0023] S5. Pause outputting available tiles and generate warnings.

[0024] In some implementations, step S1 specifically includes:

[0025] The application is built based on the Web server. After receiving the user's HTTP request information, it reads the tile from the tile file set, writes the response content and returns it to the client. At the same time, an identity is established for the requesting user, and a historical record queue of the user's access to the tile is built based on the in-memory database, which supports querying the historical record of tile access by user.

[0026] In some implementations, step S2 specifically includes:

[0027] The request response program intercepts user U's request for geographic information tiles, and writes each tile request from each user into the tile access record queue; a query is performed in the tile access queue according to user U and the level L of the requested tile. If the query result is empty, it is considered that the crawler program has not been detected and the detection ends; if the query result of the tile access is not empty, proceed to step S3.

[0028] In some implementations, step S3 specifically includes:

[0029] For user U's tile access records of level L in the most recent time window T, time is aggregated and the time is discretized into a time step Step. In this time window T, it is determined whether there is a slice access behavior in each time step and the total number of slices accessed reaches the upper limit T. t If yes, it is determined that the time meets the crawler behavior characteristics and the process goes to step S4; if no, it is considered that the crawler program is not detected and the detection ends.

[0030] In some implementations, step S4 specifically includes:

[0031] Spatial aggregation is performed on the tile access records of user U to level L within the time window T. Based on the spatial adjacency of the accessed slices, the DBSCAN method is used to perform spatial contiguous aggregation, find the cluster with the largest number of aggregated tiles, and determine whether the number of slices contained in the largest cluster has reached the upper limit T. s , and determine whether the sum of the ellipsoid areas represented by the slice set contained in the largest cluster exceeds the upper limit S. If the upper limit T s If the space meets the crawler characteristics at the same time as the upper line S, it is considered that the space meets the crawler characteristics and proceeds to step S5.

[0032] In some embodiments, the process of calculating the area of ​​the ellipsoid includes:

[0033] The tile corner point coordinate operator is used to calculate the longitude and latitude coordinates of the four corner points of the tile, and then the ellipsoid area calculation operator is used to calculate the ellipsoid area of ​​each tile according to the longitude and latitude coordinates of the four corner points of the tile. The areas of each tile are summarized and accumulated to obtain the total ellipsoid area of ​​the largest cluster.

[0034] In some implementations, step S5 specifically includes:

[0035] Stop providing correct tiles and output warning tiles instead, construct a blank image according to the normal tile image size, and write text warning content directly in the image file.

[0036] In some implementations, step S6 is further included: adding noise to interfere with the client using the crawled tiles:

[0037] Pretend to continue providing available geographic information tiles to the client, so that the crawler program thinks it is still obtaining normal tiles; based on the warning tiles, random noise is added to the data in the tile image to form camouflage tiles.

[0038] On the other hand, the present invention also discloses a detection and response device for a geographic information tile crawler program, comprising:

[0039] Tile request response program module: after receiving the user's HTTP request information, read the tile from the tile file set, write the response content and return it to the client; at the same time, establish an identity for the requesting user and build a historical record queue of the user's access to the tile based on the in-memory database;

[0040] User access tile history module: The request response program intercepts user U's request for geographic information tiles, and writes each tile request of each user into the tile access record queue; query is performed in the tile access queue according to user U and the level L of the requested tile;

[0041] Time aggregation behavior identification module: Time aggregation is performed on the tile access records of level L by user U in the most recent time window T, and the time is discretized into a time step Step. In this time window T, it is determined whether there is a slice access behavior in each time step, and the total number of slices accessed reaches the upper limit T. t ;

[0042] Spatial aggregation behavior identification module: Spatial aggregation is performed on the tile access records of user U to level L within the time window T. Based on the spatial proximity of the accessed slices, the DBSCAN method is used to perform spatial contiguous slice aggregation, find the cluster with the largest number of aggregated tiles, and determine whether the number of slices contained in the largest cluster has reached the upper limit T. s , and determine whether the total area of ​​the ellipsoid represented by the slice set contained in the largest cluster exceeds the upper limit S;

[0043] Warning module: Stop providing correct tiles and output warning tiles instead, construct a blank image according to the normal tile image size, and write text warning content directly in the image file;

[0044] Camouflage tile generation module: Pretends to continue to provide available geographic information tiles to the client, so that the crawler program thinks it is still obtaining normal tiles; based on the warning tile, random noise is added to the data in the tile image to form a camouflage tile.

[0045] On the other hand, the present invention further discloses an electronic device, including a memory and a processor, wherein the memory stores a computer program that can be run on the processor, and the processor implements the above method when executing the computer program.

[0046] Beneficial effects: The present invention comprehensively utilizes technologies such as time-space aggregation, user behavior detection, access behavior restriction, and noise interference crawling, and can detect abnormal behaviors of users performing high-frequency and intensive access to geographic information tiles and spatially continuous access. Since crawlers can be discovered in a timely and accurate manner, and measures such as suspending the provision of correct tiles, generating warning tile information, and adding noise to interfere with the client's extraction of available tiles are taken against crawler program users, the rights and interests of geographic information data providers are protected. The beneficial effects are detailed as follows:

[0047] (1) It is possible to detect crawlers, the cost of detecting crawlers is low, and crawler behavior can be dealt with and restricted after it is discovered.

[0048] Compared with conventional geographic information service basic software, the present invention has a detection function for discovering the behavior of users crawling geographic information tiles, and can detect crawlers. Compared with the "watermark" retrospective method and the requirement to collect evidence on the client site, the tile request response program in the present invention is autonomously controllable by the data provider on the server side, so the cost of discovering crawlers is low. After discovering crawler behavior, the data provider can immediately restrict the crawler program on the server side, such as suspending the response output of correct tiles and instead outputting warning tiles.

[0049] (2) Able to detect crawlers in a timely manner.

[0050] The time consumption of finding a crawler in the present invention is related to the set time window T, that is, the time consumption of finding a crawler is ≤ 2T. For example, T is usually set to 2 minutes, and it takes up to 4 minutes to find a crawler. The speed of finding crawling behavior within 4 minutes is fast and timely. Compared with the way of limiting the number of tile requests per day such as Baidu Maps and Tiandi Maps, the time period of crawler program detection is greatly shortened.

[0051] (3) Accurately restrict crawlers rather than restricting users with large traffic.

[0052] Some active users may have a large number of tile requests with daily access needs. If Baidu Maps, Tiandi Maps, etc. are used to limit the number of daily tile requests, users with large access volume may be restricted due to the number of requests. Simply increasing the limit on the number of daily requests is similar to the situation where no crawler detection is performed. The present invention is oriented to crawler programs, focusing on the temporal aggregation and spatial aggregation behavior characteristics of crawlers on geographic information tiles, and is specifically aimed at crawler program detection and restriction, rather than limiting clients with large access volume.

[0053] (4) The scope of crawled data is relatively small, and it cannot be easily used for unintended purposes on the client side.

[0054] Assuming that the crawler client crawls a small number of available tiles in two consecutive time windows (2T), the server can still interfere with the client's extraction and utilization of the crawled tiles by using noise camouflage tiles. The server adds random noise to the tile image data based on warning tiles. When the user of the crawler program finds the intentionally added warning information and tries to extract useful tiles from the tile set, it will be found that useful tiles (tiles without noise) and noisy tiles are mixed together, which is difficult to classify by extracting image features. Therefore, even if the tile set obtained by the crawler contains some useful tiles, it is difficult to extract them through program automation. BRIEF DESCRIPTION OF THE DRAWINGS

[0055] Figure 1 This is a flow chart of a detection and response method according to an embodiment of the present invention;

[0056] Figure 2 A flow chart of a detection and response method according to another embodiment of the present invention;

[0057] Figure 3 A schematic diagram of the specific internal relationships of a detection and response method according to an embodiment of the present invention. DETAILED DESCRIPTION

[0058] The technical solution of the present invention will be described clearly and completely below in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0059] In the description of the present invention, it should be noted that the directions or positional relationships indicated by terms such as "center", "up", "down", "left", "right", "inside" and "outside" are the directions or positional relationships shown, which are only for the convenience of describing the present invention and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific direction, be constructed and operated in a specific direction, and therefore cannot be understood as a limitation on the present invention.

[0060] In the description of the present invention, it should be noted that, unless otherwise clearly specified and limited, the terms "installed", "connected", and "connected" should be understood in a broad sense, for example, it can be a fixed connection, a detachable connection, or an integral connection; it can be a mechanical connection or an electrical connection; it can be a direct connection, or it can be indirectly connected through an intermediate medium, or it can be the internal communication of two components. For ordinary technicians in this field, the specific meanings of the above terms in the present invention can be understood according to specific circumstances.

[0061] The present invention is further described in detail below through specific implementation examples in conjunction with the accompanying drawings. Example

[0062] like Figure 1 As shown, a detection and response method for a geographic information tile crawler program includes:

[0063] S1. Construct tile request response program: Build an application based on the Web server. After receiving the user's HTTP request information, read the (L, C, R) tile from the tile file set, write the response content and return it to the client. On this basis, establish an identity (IP identity or token identity) for the requesting user, build a historical record queue of user access to tiles based on the memory database, and support querying the historical record of access to tiles by user.

[0064] S2. Check the history of user access to tiles: The request response program intercepts user U's request for geographic information tiles (L, C, R), and writes each tile request of each user into the tile access record queue. According to user U and the level L of the requested tile, a query is performed in the tile access queue. If the query result is empty, it is considered that no crawler program is detected and the detection ends. If the query result of the tile access is not empty, go to the next step.

[0065] S3, Time aggregation behavior judgment: Through the time aggregation operator, the tile access records of level L of user U in the most recent time window T (such as the last 2 minutes) are time aggregated, and the time is discretized into a time step Step (such as 5 seconds). In this time window T, it is judged whether there is a slice access behavior in each time step, and the total number of slices accessed reaches the upper limit T t If the number of crawlers is 400, then the time is considered to be consistent with the crawler behavior characteristics, and the next step is carried out. If the number is not, then it is considered that no crawler program is detected, and the detection ends.

[0066] Normal access to geographic information tiles is temporally random and intermittent, while crawlers are temporally continuous and intensive. Considering the peak and trough periods of network channel transmission rate, even if the crawler continuously and intensively accesses tiles, it is difficult to directly determine whether it is a crawler by the instantaneous tile access rate (such as the number of tile accesses per second). This method discretizes time into time windows by step length, and continuously checks whether each step length has a certain intensive row access behavior, which can smooth out the fluctuation of network channel transmission rate. It is more accurate and robust in monitoring the temporal continuity and access intensive characteristics of crawlers crawling tiles, and has statistical and regular significance for detecting crawler time characteristics.

[0067] S4. Spatial aggregation behavior judgment: Through the spatial aggregation operator, spatial aggregation is performed on the tile access records of user U to level L within the time window T. According to the spatial adjacency of the accessed slices, the DBSCAN method is used to perform spatial contiguous aggregation to find the cluster with the largest number of aggregated tiles and determine whether the number of slices contained in the largest cluster has reached the upper limit T. s (Condition 1), and determine whether the total area of ​​the ellipsoid represented by the slice set contained in the largest cluster exceeds the upper limit S (Condition 2). If both Condition 1 and Condition 2 are met, it is considered that the space meets the crawler characteristics and proceeds to the next step. s It can be set to a fixed value (such as 200), or a corresponding value can be determined according to different levels of L.

[0068] Among them, the process of ellipsoid area calculation is: use the tile corner point coordinate operator to calculate the longitude and latitude coordinates of the four corner points of the (L, C, R) tile, and then use the ellipsoid area calculation operator to calculate the longitude and latitude coordinates of the four corner points of the tile to obtain the ellipsoid area of ​​each tile. The areas of each tile are summarized and accumulated to obtain the total ellipsoid area of ​​the largest cluster.

[0069] Normal access to geographic information tiles is spatially random and discrete, while crawler access to tiles is spatially adjacent and continuous. The DBSCAN clustering algorithm can aggregate adjacent discrete points with proximity relationships into continuous surface boundaries in two-dimensional space. The geographic information tile set (L K ,C M ,R N ) data structure and its access history have this two-dimensionality and proximity, specifically: if the row and column numbers (integer values) of the tiles are numerically close, then the tiles are also spatially adjacent.

[0070] The density-reachable and density-connected rules of the DBSCAN algorithm are reflected in this method as the numerical adjacency and spatial adjacency of tiles in row and column numbers; the results (clusters) of the DBSCAN algorithm in this method correspond to the maximum set of adjacent tiles visited. The number of tiles in the largest cluster aggregated by the DBSCAN algorithm is positively correlated with the area of ​​the crawled tiles. The larger the number of tiles in the largest cluster, the larger the area of ​​geographic information data crawled in pieces, which also means that the value loss of geographic information data of the data provider is greater. The clusters aggregated by the DBSCAN algorithm are irregular in geometry (the shape of the cluster is not limited to circles, rectangles, regular polygons, etc.), which can adapt to the irregularity of the spatial boundaries of the tiles crawled in pieces by the crawler.

[0071] S5. Pause the output of available tiles and generate warnings: After reaching this stage, it is considered that the user U's access behavior to the geographic information tiles of the L-level meets the crawler characteristics in terms of time aggregation and spatial aggregation. At this time, the provision of correct tiles is immediately suspended and warning tiles are output instead. That is, a blank image is constructed according to the normal tile image size, and the text warning content (such as "tile access is too intensive") is directly written in the image file. Example

[0072] like Figure 2 As shown, a detection and response method for a geographic information tile crawler program includes:

[0073] S1. Construct tile request response program: Build an application based on the Web server. After receiving the user's HTTP request information, read the (L, C, R) tile from the tile file set, write the response content and return it to the client. On this basis, establish an identity (IP identity or token identity) for the requesting user, build a historical record queue of user access to tiles based on the memory database, and support querying the historical record of access to tiles by user.

[0074] like Figure 3 As shown, an application is built based on a Web server to realize the user request information from the ⑨ tile file set (a tile set with K levels, M columns, and N rows, denoted as (L K ,C M ,R N ), each tile is an image file in PNG or JPG format) (L ， C ， R) Basic functionality for tiles to write response content.

[0075] S2. Check the history of user access to tiles: The request response program intercepts user U's request for geographic information tiles (L, C, R), and writes each tile request of each user into the tile access record queue. According to user U and the level L of the requested tile, a query is performed in the tile access queue. If the query result is empty, it is considered that no crawler program is detected and the detection ends. If the query result of the tile access is not empty, go to the next step.

[0076] like Figure 3 As shown, for the tile request from ①, an identity is established for each user (such as an IP identifier or a token identifier), and a user access tile history record queue is built based on the memory database. The queue supports querying the history of tile access by user.

[0077] S3. Time aggregation behavior judgment: Through the time aggregation operator: Time aggregation is performed on the tile access records of level L for user U in the most recent time window T (such as the last 2 minutes), and the time is discretized into a time step Step (such as 5 seconds). In this time window T, it is determined whether there is a slice access behavior in each time step, and the total number of slices accessed reaches the upper limit T t If the number of crawlers is 400, then the time is considered to be consistent with the crawler behavior characteristics, and the next step is carried out. If the number is not, then it is considered that no crawler program is detected, and the detection ends.

[0078] like Figure 3 As shown in the figure, based on the query results from step ②, it is determined whether the user's recent slice access behavior meets the crawler characteristics in terms of temporal continuity and density. Through the above time aggregation calculation, it is determined whether the user has made continuous and high-frequency intensive accesses to the tiles of level L in the most recent time window T. If so, it is considered to have the temporal behavior characteristics of a crawler.

[0079] S4. Spatial aggregation behavior judgment: Through the spatial aggregation operator: spatially aggregate the tile access records of user U to level L within the time window T, and use the DBSCAN method to perform spatial contiguous aggregation based on the spatial adjacency of the accessed slices, find the cluster with the largest number of aggregated tiles, and determine whether the number of slices contained in the largest cluster has reached the upper limit T s (Condition 1), and determine whether the total area of ​​the ellipsoid represented by the slice set contained in the largest cluster exceeds the upper limit S (Condition 2). If both Condition 1 and Condition 2 are met, it is considered that the space meets the crawler characteristics and proceeds to the next step. s It can be set to a fixed value (such as 200), or a corresponding value can be determined according to different levels of L.

[0080] like Figure 3 As shown in the figure, based on the query results from ②, it is determined whether the user's recent slice access behavior meets the characteristics of a crawler in terms of the continuity of the tile geographic space. Through spatial aggregation calculation (DBSCAN algorithm), it is determined whether the user has made spatially continuous accesses to tile access records of level L within a recent time window T, and the contiguous area (calculated by ⑤⑥) has reached the limit. If so, it is considered to have the spatial behavior characteristics of a crawler.

[0081] Among them, the process of ellipsoid area calculation is: use the tile corner point coordinate operator to calculate the longitude and latitude coordinates of the four corner points of the (L, C, R) tile, and then use the ellipsoid area calculation operator to calculate the longitude and latitude coordinates of the four corner points of the tile to obtain the ellipsoid area of ​​each tile. The areas of each tile are summarized and accumulated to obtain the total ellipsoid area of ​​the largest cluster.

[0082] Combination Figure 3, Tile corner coordinate calculation: For a given (L, C, R) tile, calculate the longitude and latitude coordinates of the four corners of the tile (lower left, lower right, upper right, upper left) according to the slice origin, image resolution, number of tile pixels, and coordinate projection information. Ellipsoid area calculation: According to the longitude and latitude coordinates of the four corners of the tile, the ellipsoid area of ​​each tile is calculated by the differential arc length formula of the meridian, the differential arc length formula of the latitude, and the integral formula of the spherical trapezoid formed by the four corners of the tile.

[0083] S5. Pause outputting available tiles and generate warnings: After reaching this stage, it is considered that user U's access behavior to the geographic information tiles at level L meets the crawler characteristics in both temporal aggregation and spatial aggregation. At this time, the correct tiles are immediately suspended and warning tiles are output instead, that is, a blank image is constructed according to the normal tile image size, and the text warning content (such as "tile access is too intensive") is directly written into the image file. Or, for example, among every 4 warning tiles, 2 tiles are selected to be directly written into the response stream output, and the remaining 2 tiles are transferred to the next step.

[0084] Combination Figure 3 , the tile crawling behaviors that meet the time behavior characteristics of ③ and the spatial behavior characteristics of ④ are considered to be crawler programs, so the normal tile response output is suspended and a warning tile image is generated to the client instead. A blank image is constructed with reference to the size of a normal tile image, and the text warning content is directly written in the image file.

[0085] S6. Add noise to interfere with the client's use of crawled tiles: Pretend to the client that it can continue to provide available tiles with geographic information, so that the crawler program thinks it is still acquiring normal tiles. Based on the warning tile, random noise is added to the data in the tile image to form a camouflaged tile. Specifically, color component values ​​are randomly generated in the RGB color space and combined into pixel color values, and noise is written into the image of the camouflaged tile according to the color value.

[0086] Combination Figure 3 , based on the warning tile generated in ⑦, random noise is added to the tile image data. Then the tiles crawled by the client contain three kinds of tiles, one is the available tile, one is the warning tile, and one is the camouflage tile containing noise. The three kinds of tiles are mixed together, so that the client cannot easily extract the crawled available tiles locally. Example

[0087] A detection and response device for a geographic information tile crawler program, comprising:

[0088] Tile request response program module: after receiving the user's HTTP request information, read the tile from the tile file set, write the response content and return it to the client; at the same time, establish an identity for the requesting user and build a historical record queue of the user's access to the tile based on the in-memory database;

[0089] User access tile history module: The request response program intercepts user U's request for geographic information tiles, and writes each tile request of each user into the tile access record queue; query is performed in the tile access queue according to user U and the level L of the requested tile;

[0090] Time aggregation behavior identification module: Time aggregation is performed on the tile access records of level L by user U in the most recent time window T, and the time is discretized into a time step Step. In this time window T, it is determined whether there is a slice access behavior in each time step, and the total number of slices accessed reaches the upper limit T. t ;

[0091] Spatial aggregation behavior identification module: Spatial aggregation is performed on the tile access records of user U to level L within the time window T. Based on the spatial proximity of the accessed slices, the DBSCAN method is used to perform spatial contiguous slice aggregation, find the cluster with the largest number of aggregated tiles, and determine whether the number of slices contained in the largest cluster has reached the upper limit T. s , and determine whether the total area of ​​the ellipsoid represented by the slice set contained in the largest cluster exceeds the upper limit S;

[0092] Warning module: Stop providing correct tiles and output warning tiles instead, construct blank images according to the normal tile image size, and write text warning content directly into the image file.

[0093] Camouflage tile generation module: Pretend to continue to provide geographic information tiles to the client, so that the crawler program thinks it is still acquiring normal tiles. Based on the warning tile, random noise is added to the tile image data to form a camouflage tile. Specifically, color component values ​​are randomly generated in the RGB color space and combined into pixel color values, and noise is written into the camouflage tile image according to the color value. Example

[0094] An electronic device includes a memory and a processor, wherein the memory stores a computer program that can be run on the processor, and when the processor executes the computer program, the method described in Example 1 or Example 2 is implemented.

[0095] The present invention extracts the spatial continuity and temporal continuity behavior characteristics of crawled geographic information tiles, calculates whether the current user's tile access behavior in a recent period of time conforms to the spatial continuity and temporal continuity characteristics, and determines the algorithm rules for judging spatial continuity and the rules for judging temporal continuity.

[0096] The present invention records the information, time point and user identification of the user's request to access the tile, and constructs a queryable queue. According to the tile request record queried by the user, a queue is constructed in the memory to improve the algorithm detection performance.

[0097] The present invention uses tile camouflage and interference technology to deal with crawler programs. After discovering the crawler program, it stops responding to the correct tile content and adds random noise and warning information to the tile, with the position and color of the noise being random.

[0098] The above description is only a preferred embodiment of the present invention and does not constitute any form of limitation to the present invention. Although the present invention has been disclosed as a preferred embodiment as above, it is not intended to limit the present invention. Any technician familiar with the profession can make some changes or modifications to equivalent embodiments of equivalent changes using the technical contents disclosed above without departing from the scope of the technical solution of the present invention. However, any simple modification, equivalent change and modification made to the above embodiments based on the technical essence of the present invention without departing from the content of the technical solution of the present invention still falls within the scope of the technical solution of the present invention.

Claims

1. A detection and response method for a geographic information tile crawler program, characterized in that: include: S1, construct the request response program of the tile; S2, check the history of users accessing tiles; S3, temporal aggregation behavior discrimination; S4, spatial aggregation behavior discrimination; S5, suspend output of available tiles and generate a warning; The detection and response device includes: Tile request response program module: after receiving the user's HTTP request information, read the tile from the tile file set, write the response content and return it to the client; at the same time, establish an identity for the requesting user and build a historical record queue of the user's access to the tile based on the in-memory database; User access tile history module: The request response program intercepts user U's request for geographic information tiles, and writes each tile request of each user into the tile access record queue; query is performed in the tile access queue according to user U and the level L of the requested tile; Time aggregation behavior identification module: Time aggregation is performed on the tile access records of level L by user U in the most recent time window T, and the time is discretized into a time step Step. In this time window T, it is determined whether there is a slice access behavior in each time step, and the total number of slices accessed reaches the upper limit T. t ; Spatial aggregation behavior identification module: Spatial aggregation is performed on the tile access records of user U to level L within the time window T. Based on the spatial proximity of the accessed slices, the DBSCAN method is used to perform spatial contiguous slice aggregation, find the cluster with the largest number of aggregated tiles, and determine whether the number of slices contained in the largest cluster has reached the upper limit T. s , and determine whether the total area of ​​the ellipsoid represented by the slice set contained in the largest cluster exceeds the upper limit S; Warning module: Stop providing correct tiles and output warning tiles instead, construct a blank image according to the normal tile image size, and write text warning content directly in the image file; Camouflage tile generation module: Pretends to continue to provide available geographic information tiles to the client, so that the crawler program thinks it is still obtaining normal tiles; based on the warning tile, random noise is added to the data in the tile image to form a camouflage tile.

2. A detection and response method for a geographic information tile crawler program according to claim 1, characterized in that: Step S1 specifically includes: The application is built based on the Web server. After receiving the user's HTTP request information, it reads the tile from the tile file set, writes the response content and returns it to the client. At the same time, an identity is established for the requesting user, and a historical record queue of the user's access to the tile is built based on the in-memory database, which supports querying the historical record of tile access by user.

3. A detection and response method for a geographic information tile crawler program according to claim 1, characterized in that: Step S2 specifically includes: The request response program intercepts user U's request for geographic information tiles, and writes each tile request from each user into the tile access record queue; a query is performed in the tile access queue according to user U and the level L of the requested tile. If the query result is empty, it is considered that the crawler program has not been detected and the detection ends; if the query result of the tile access is not empty, proceed to step S3.

4. A detection and response method for a geographic information tile crawler program according to claim 1, characterized in that: Step S3 specifically includes: For user U's tile access records of level L in the most recent time window T, time is aggregated and the time is discretized into a time step Step. In this time window T, it is determined whether there is a slice access behavior in each time step and the total number of slices accessed reaches the upper limit T. t If yes, it is determined that the time meets the crawler behavior characteristics and the process goes to step S4; if no, it is considered that the crawler program is not detected and the detection ends.

5. A detection and response method for a geographic information tile crawler program according to claim 1, characterized in that: Step S4 specifically includes: Spatial aggregation is performed on the tile access records of user U to level L within the time window T. Based on the spatial adjacency of the accessed slices, the DBSCAN method is used to perform spatial contiguous aggregation, find the cluster with the largest number of aggregated tiles, and determine whether the number of slices contained in the largest cluster has reached the upper limit T. s , and determine whether the sum of the ellipsoid areas represented by the slice set contained in the largest cluster exceeds the upper limit S. If the upper limit T s If the space meets the crawler characteristics at the same time as the upper line S, it is considered that the space meets the crawler characteristics and proceeds to step S5.

6. A detection and response method for a geographic information tile crawler program according to claim 5, characterized in that: The calculation process of the ellipsoid area includes: The tile corner point coordinate operator is used to calculate the longitude and latitude coordinates of the four corner points of the tile, and then the ellipsoid area calculation operator is used to calculate the ellipsoid area of ​​each tile according to the longitude and latitude coordinates of the four corner points of the tile. The areas of each tile are summarized and accumulated to obtain the total ellipsoid area of ​​the largest cluster.

7. A detection and response method for a geographic information tile crawler program according to claim 1, characterized in that: Step S5 specifically includes: Stop providing correct tiles and output warning tiles instead, construct a blank image according to the normal tile image size, and write text warning content directly in the image file.

8. A detection and response method for a geographic information tile crawler program according to claim 1, characterized in that: The step S6 is also included: adding noise to interfere with the client using the crawled tiles: Pretend to continue providing available geographic information tiles to the client, so that the crawler program thinks it is still obtaining normal tiles; based on the warning tiles, random noise is added to the data in the tile image to form camouflage tiles.

9. An electronic device comprising a memory and a processor, wherein the memory stores a computer program that can be run on the processor, wherein: When the processor executes the computer program, the method according to any one of claims 1 to 8 is implemented.

Images