Website identification method and device
Through the combination of automated test examples and large language models, the existing URL recognition methods are solved, and the accurate recognition of complex web page content is achieved.
Patent Information
- Application Number
- CN202510554189.7
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-04-28
- Publication Date
- 2025-08-22
AI Technical Summary
The existing URL recognition methods are inefficient and susceptible to subjective factors, and are less applicable to simple keyword matching, making it difficult to accurately identify URLs of complex web page content.
Through automated test examples, simulate the browser to access the URL, obtain the target page, extract the web page content and perform screenshot processing, analyze the page pictures in combination with the large language model to determine the URL type.
It improves the accuracy and applicability of URL recognition, and achieves a comprehensive understanding and in-depth mining of the target page.
Smart Images
Figure CN120528811A_ABST
Abstract
Description
Technical Field
[0001] The embodiments of the present application relate to the field of information security technology, and in particular to a website identification method, apparatus, computer equipment, computer-readable storage medium, and computer program product. Background Art
[0002] In the era of Internet information explosion, URL recognition technology is particularly important. Traditional URL recognition methods mainly rely on manual analysis of the webpage content corresponding to the URL or simple keyword matching to directly identify the URL.
[0003] However, manual analysis of web content corresponding to URLs is often inefficient when dealing with complex webpages and is easily influenced by subjective factors, resulting in inaccurate recognition results. While simple keyword matching can quickly identify relevant URLs, it requires users to maintain a whitelist of URLs, making it less practical.
[0004] In view of this, there is an urgent need for an accurate and universal method for identifying various types of URLs to meet the challenges of URL identification in the current Internet environment.
[0005] It should be noted that the above content is not necessarily prior art, nor is it intended to limit the scope of patent protection of this application. Summary of the Invention
[0006] The embodiments of the present application provide a website identification method, apparatus, computer device, computer-readable storage medium, and computer program product to solve or alleviate one or more of the technical problems raised above.
[0007] One aspect of an embodiment of the present application provides a method for identifying a website, the method comprising: Get the URL to be identified; Through the automated test instance, simulate the browser to access the URL to be identified to obtain the target page; Extracting webpage content from the source code corresponding to the target page, and taking a screenshot of the target page to obtain a page image; Analyze the page image using a large language model to obtain a page analysis result; Based on the webpage content and the page analysis result, it is determined whether the URL to be identified is a URL of a preset type.
[0008] Optionally, simulating a browser to access the to-be-identified website through an automated test instance to obtain a target page includes: Simulate a browser to access the URL to be identified through an automated test instance; If the target page corresponding to the URL to be identified is not obtained within the preset time, a null value is returned; When the target page corresponding to the URL to be identified is obtained after the preset time period, the target page is returned in an asynchronous manner.
[0009] Optionally, the method further includes: Creating an automated test instance pool, wherein the automated test instance pool includes automated test instances of multiple levels; The step of simulating a browser to access the URL to be identified through an automated test instance to obtain a target page includes: Selecting a target automated test instance from the automated test instances of the multiple levels; The target automated test instance simulates a browser accessing the URL to be identified to obtain a target page.
[0010] Optionally, the method further includes: After selecting the target automated test instance, locking the target automated test instance; After simulating a browser to access the to-be-identified website through the target automated test instance, an unlocking operation is performed on the target automated test instance.
[0011] Optionally, selecting a target automated test instance from the automated test instances of the multiple levels includes: Obtaining priority information of the to-be-identified website; and selecting an idle automated test instance from the multiple levels of automated test instances as a target automated test instance based on the priority information.
[0012] Optionally, the method further includes: When the URL to be identified is a URL of a preset type, the URL to be identified is intercepted by a URL interception system.
[0013] Optionally, determining whether the to-be-identified URL is a URL of a preset type based on the webpage content and the page analysis result includes: When the webpage content includes content of a preset type and the page analysis result is a preset page analysis result, it is determined that the URL to be identified is a URL of the preset type.
[0014] Another aspect of an embodiment of the present application provides a device for identifying a website address, the device comprising: The acquisition module is used to obtain the URL to be identified; An access module is used to simulate a browser accessing the URL to be identified through an automated test instance to obtain a target page; An extraction module is used to extract webpage content from the source code corresponding to the target page, and to take a screenshot of the target page to obtain a page image; An analysis module, configured to analyze the page image using a large language model to obtain a page analysis result; The determination module is used to determine whether the URL to be identified is a URL of a preset type based on the webpage content and the page analysis result.
[0015] Another aspect of an embodiment of the present application provides a computer device, including: at least one processor; and a memory communicatively coupled to the at least one processor; Wherein: the memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the method described above.
[0016] Another aspect of an embodiment of the present application provides a computer-readable storage medium, wherein the computer-readable storage medium stores computer instructions, and when the computer instructions are executed by a processor, the method described above is implemented.
[0017] Another aspect of an embodiment of the present application provides a computer program product, including a computer program, which implements the above-mentioned method when executed by a processor.
[0018] The embodiment of the present application adopts the above-mentioned technical solution, which may include the following advantages: obtaining the URL to be identified, simulating the browser to automatically access the URL to be identified through an automated test instance (for example, a selenium instance), and utilizing the browser's ability to redirect to automatically obtain the target page to which any type of URL ultimately jumps. Afterwards, the web page content is extracted from the source code corresponding to the target page, and the target page is screenshotted to obtain a page image. Finally, the web page content and the analysis results of the page image corresponding to the target page are combined with the large language model to identify whether the URL is a preset type of URL, thereby achieving a comprehensive understanding and in-depth mining of the target page, thereby improving the accuracy and applicability of URL recognition. BRIEF DESCRIPTION OF THE DRAWINGS
[0019] The accompanying drawings illustrate exemplary embodiments and constitute a part of the specification. Together with the description of the specification, they serve to explain exemplary implementation of the embodiments. The illustrated embodiments are for illustrative purposes only and do not limit the scope of the claims. Throughout the drawings, the same reference numerals designate similar, but not necessarily identical, elements.
[0020] Figure 1The following diagram schematically illustrates an operating environment of the URL identification method according to the first embodiment of the present application; Figure 2 The flowchart of the website identification method according to the first embodiment of the present application is schematically shown; Figure 3 Schematically shows Figure 1 Flowchart of sub-steps of step S202; Figure 4 A schematic diagram shows a detailed flow chart of the steps of simulating a browser accessing the URL to be identified through an automated test instance to obtain a target page; Figure 5 Schematically shows a newly added flow chart of the URL identification method according to the first embodiment of the present application; Figure 6 A block diagram schematically shows a URL identification device according to the second embodiment of the present application; and Figure 7 The following schematically shows a hardware architecture diagram of a computer device according to the third embodiment of the present application. DETAILED DESCRIPTION
[0021] In order to make the purpose, technical solutions and advantages of this application more clear, the present application is further described in detail below with reference to the accompanying drawings and examples. It should be understood that the specific embodiments described herein are only used to explain this application and are not intended to limit this application. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of this application.
[0022] It should be noted that the descriptions of "first", "second", etc. in the embodiments of the present application are for descriptive purposes only and should not be understood as indicating or implying their relative importance or implicitly indicating the number of the indicated technical features. Therefore, the features defined as "first" or "second" may explicitly or implicitly include at least one of such features. In addition, the technical solutions between the various embodiments can be combined with each other, but this must be based on the fact that they can be implemented by ordinary technicians in this field. When the combination of technical solutions is contradictory or cannot be implemented, it should be deemed that such a combination of technical solutions does not exist and is not within the scope of protection required by this application.
[0023] In the description of this application, it should be understood that the numerical labels before the steps do not indicate the order in which the steps are executed. They are only used to facilitate the description of this application and to distinguish each step. Therefore, they cannot be understood as limitations on this application.
[0024] First, an explanation of the terms used in this application is provided: Special URLs: URLs with redirects. Redirects mean that when you access a webpage through a browser, the webpage will automatically jump to another URL. This may happen multiple times, which is called multiple redirects.
[0025] To facilitate those skilled in the art to understand the technical solutions provided in the embodiments of the present application, the following describes the relevant technologies: In the prior art, the URL is mainly identified by manual analysis of the webpage content corresponding to the URL or simple keyword matching.
[0026] However, manual analysis of web content corresponding to URLs is often inefficient when dealing with complex webpages and is easily influenced by subjective factors, resulting in inaccurate recognition results. While simple keyword matching can quickly identify relevant URLs, it requires users to maintain a whitelist of URLs, making it less practical.
[0027] To this end, an embodiment of the present application provides a technical solution for URL identification. In this technical solution, the URL to be identified is obtained, and an automated test instance (for example, a selenium instance) is used to simulate a browser to automatically access the URL to be identified, thereby utilizing the browser's ability to redirect to automatically obtain the target page to which any type of URL ultimately jumps. Afterwards, the web page content is extracted from the source code corresponding to the target page, and the target page is screenshoted to obtain a page image. Finally, the analysis results of the page image corresponding to the target page are combined with the web page content and the large language model to identify whether the URL is a preset type of URL, thereby achieving a comprehensive understanding and in-depth mining of the target page, thereby improving the accuracy and applicability of URL identification. See below for details.
[0028] Finally, for ease of understanding, an exemplary operating environment is provided below.
[0029] like Figure 1 As shown, the environment diagram includes a service platform 2, a network 4, and a client 6, wherein: The service platform 2 can be comprised of a single or multiple computing devices. These multiple computing devices can include virtualized computing instances. Virtualized computing instances can include virtual machines, such as emulations of computer systems, operating systems, servers, and the like. A computing device can load a virtual machine based on a virtual image and / or other data defining the specific software (e.g., operating system, specialized application, server) used for the emulation. As demand for different types of processing services changes, different virtual machines can be loaded and / or terminated on one or more computing devices. A hypervisor can be implemented to manage the use of different virtual machines on the same computing device.
[0030] The service platform 2 can be configured to communicate with clients 6 and the like via a network 4. The network 4 includes various network devices, such as routers, switches, multiplexers, hubs, modems, bridges, repeaters, firewalls, proxy devices, and / or the like. The network 4 can include physical links, such as coaxial cable links, twisted pair cable links, fiber optic links, combinations thereof, or wireless links, such as cellular links, satellite links, Wi-Fi links, and the like.
[0031] The service platform 2 can provide storage, reading, writing, querying, deleting and other services, such as providing web page access services for clients.
[0032] Client 6 can be an electronic device running an operating system such as Windows, Android™, or iOS, such as a smartphone, tablet, laptop, virtual reality device, gaming device, set-top box, in-car terminal, or smart TV. Based on these operating systems, various applications, such as URL recognition applications, can be run.
[0033] The client 6 may provide / configure a user access page for manipulating the service platform 2 or uploading objects, etc.
[0034] It should be noted that the above devices are exemplary, and the number and type of devices can be adjusted in different scenarios or according to different needs.
[0035] The technical solutions of the present application are described below through a number of embodiments. It should be noted that these embodiments can be implemented in a variety of different forms and should not be construed as being limited to the embodiments described herein.
[0036] Example 1 Figure 2 The flowchart of the URL identification method according to the first embodiment of the present application is schematically shown.
[0037] like Figure 2 As shown, the URL identification method may include steps S200 to S208, wherein: Step S200: Obtain the URL to be identified.
[0038] Step S202: simulating a browser to access the URL to be identified through an automated test instance to obtain a target page.
[0039] Step S204: extracting webpage content from the source code corresponding to the target page, and taking a screenshot of the target page to obtain a page image.
[0040] Step S206: Use a large language model to analyze the page image to obtain a page analysis result.
[0041] Step S208: determining whether the URL to be identified is a URL of a preset type based on the webpage content and the page analysis result.
[0042] The URL identification method provided in this embodiment obtains a URL to be identified and, through an automated test instance (e.g., a Selenium instance), simulates a browser automatically accessing the URL to be identified. This method leverages the browser's ability to redirect to automatically obtain the target page to which any type of URL ultimately redirects. Subsequently, the webpage content is extracted from the source code corresponding to the target page, and a screenshot of the target page is taken to obtain a page image. Finally, the webpage content and the analysis results of the page image corresponding to the target page by a large language model are combined to determine whether the URL is of a preset type. This achieves a comprehensive understanding and in-depth exploration of the target page, thereby improving the accuracy and applicability of URL identification.
[0043] The following combination Figure 2 , each step in steps S200~S208 and other optional steps are described in detail.
[0044] Step S200 , get the URL to be identified.
[0045] The URL to be identified may be a common URL (a URL without redirection) or a special URL (a URL with redirection).
[0046] Step S202 , simulate the browser access to the URL to be identified through the automated test instance to obtain the target page.
[0047] The automated test instance is an instance in which an automated test tool or framework is used to simulate a browser automatically accessing the URL to be identified. In a specific implementation process, the automated test instance first constructs a browser window and then accesses the URL to be identified in the browser window.
[0048] The automated test instance may be a Selenium instance, or a Playwright instance, a Puppeteer instance, or the like.
[0049] Selenium is a collection of open-source tools for automated testing of web applications. A Selenium instance is one that can be used to automatically build a browser window and access a URL to be identified in that browser window.
[0050] Playwright is a modern automated testing tool developed by Microsoft, specifically designed for modern web applications. It supports multiple programming languages and major browsers. Its powerful functionality allows it to be used not only for automated testing but also for crawling dynamic content, debugging, and analyzing web applications. A Playwright instance is a browser environment created using the Playwright tool. It simulates user browser behavior for automated testing, web crawling, and other related tasks. Using a Puppeteer instance, you can automatically build a browser window and access the URL to be identified within it.
[0051] Puppeteer is a Node.js library that provides an API for controlling Chrome or Chromium browsers, enabling automated operations. A Puppeteer instance is an instance that uses the Puppeteer library for browser automation. This instance automatically creates a browser window and accesses the URL to be identified within it.
[0052] The target page is the page to which the to-be-identified website finally jumps.
[0053] In this embodiment, the browser environment can be simulated through the automated test instance to access the URL to be identified, thereby utilizing the browser's ability to redirect to directly obtain the target page after the website finally jumps.
[0054] Step S204 , extracting web page content from the source code corresponding to the target page, and taking a screenshot of the target page to obtain a page image.
[0055] In this embodiment, different automated testing tools extract webpage content and take screenshots of target pages in different ways.
[0056] In one embodiment, when the automated testing tool is Selenium, after obtaining the target page, the driver.page_source method can be used to obtain the webpage source code, and then the required webpage content can be extracted using a parsing tool. After obtaining the target page, the driver.save_screenshot("screenshot path") or driver.get_screenshot_as_file("screenshot path") method can also be used to take a screenshot of the target page to obtain a page image.
[0057] In another embodiment, when the automated testing tool is Puppeteer, after obtaining the target page, the page.content() method can be used to obtain the webpage source code and then parse it to obtain the webpage content. After obtaining the target page, the page.screenshot({ path: "screenshot path", fullPage: true}) method can also be used to capture a screenshot of the entire webpage.
[0058] The webpage content may be at least one of text content, image content or video content contained in the webpage.
[0059] Step S206 , use a large language model to analyze the page image and obtain a page analysis result.
[0060] The page analysis result may be whether the image contains any content that does not conform to the specification. In one embodiment, to facilitate the large language model to analyze the page image, an OCR tool (such as Tesseract, Google Vision API) may be used to extract text from the page image before analysis.
[0061] In one embodiment, in order to improve the OCR accuracy, before the OCR tool extracts the text in the page image, the page image may be subjected to noise reduction, binarization and other processing.
[0062] After the text is extracted, the text and the image metadata of the page image (such as size and color distribution) can be input into the large language model, and the page image can be analyzed by the large language model to obtain the image analysis result.
[0063] Step S208 , based on the web page content and the page analysis result, determine whether the URL to be identified is a preset type of URL.
[0064] The preset type of URL refers to the URL type set by the user according to actual conditions, which generally refers to a type of URL that does not comply with the standards, and may include URLs that do not comply with information security standards and URLs that do not comply with network security standards.
[0065] In an optional embodiment, determining whether the to-be-identified URL is a URL of a preset type based on the webpage content and the page analysis result includes: When the webpage content includes content of a preset type and the page analysis result is a preset page analysis result, it is determined that the URL to be identified is a URL of the preset type.
[0066] Among them, the preset type of content refers to the content type set by the user according to actual conditions, which generally refers to a type of non-standard content, which may include content that does not comply with information security specifications and content that does not comply with network security specifications.
[0067] In this embodiment, after obtaining the webpage content of the webpage, it is possible to determine whether the webpage content contains content of a preset type. If the webpage content contains content of the preset type, it is possible to preliminarily determine that the URL to be identified is a URL of the preset type. After preliminarily determining that the URL to be identified is a URL of the preset type, it is possible to further determine whether the page analysis result is a preset page analysis result. If the page analysis result is also a preset page analysis result, it is possible to determine whether the URL to be identified is a URL of the preset type.
[0068] In one embodiment, if the webpage content does not include a preset type of content, or the page analysis result is not a preset page analysis result, it can be determined that the URL to be identified is a normal URL (compliant URL).
[0069] In this embodiment, when it is identified that the webpage content contains content of a preset type and the page analysis result is the preset page analysis result, the URL to be identified is determined to be a URL of the preset type, and the URL to be identified can be used as a URL posted by a non-ordinary user. When it is identified that the webpage content does not contain content of the preset type or the page analysis result is not the preset page analysis result, the URL to be identified is determined to be a legitimate URL, and the URL to be identified can be used as a URL posted by an ordinary user. In this way, URLs posted by ordinary users and non-ordinary users can be accurately distinguished.
[0070] It should be noted that the non-ordinary users mentioned above refer to users who publish content of preset types.
[0071] In another embodiment, when it is identified that the webpage content contains unexpected content and the page analysis result is a preset page analysis result, it is determined that the URL to be identified is a URL of a preset type.
[0072] The expected content refers to the content output according to preset standards, specifications, and requirements.
[0073] In this embodiment, the browser's redirection capability is leveraged to obtain the target page to which any type of URL ultimately redirects. Subsequently, the webpage content is extracted from the target page's source code and a screenshot of the target page is taken to obtain a page image. Finally, the webpage content and the analysis results of the target page image corresponding to the large language model are combined to identify whether the URL is of a preset type. This achieves a comprehensive understanding and in-depth exploration of the target page, thereby improving the accuracy and applicability of URL recognition.
[0074] In an optional embodiment, if Figure 3 As shown, step S202 may include: Step S300: simulating a browser access to the URL to be identified through an automated test instance.
[0075] Step S302: When the target page corresponding to the URL to be identified is not obtained within a preset time period, a null value is returned.
[0076] Step S304: When a target page corresponding to the URL to be identified is obtained after the preset time period, the target page is returned asynchronously.
[0077] The preset time length is a time length set in advance according to actual conditions, for example, the preset time length is 2 seconds.
[0078] In this embodiment, after simulating a browser access to the URL to be identified through an automated test instance, if the target page corresponding to the URL to be identified is not obtained within a preset time period, a null value will be returned to avoid affecting the user experience. If the target page corresponding to the URL to be identified is obtained after the preset time period, the target page will be returned asynchronously. This method allows users to obtain access results without having to wait for a long time, thereby improving the user experience.
[0079] In an optional embodiment, the method further includes: An automated test instance pool is created, where the automated test instance pool includes automated test instances of multiple levels.
[0080] Multiple levels of automated test instances in the automated test instance pool can run independently without affecting each other. Each level of automated test instance can include one or more automated test instances.
[0081] In this embodiment, by pre-creating multiple levels of automated test instances, high concurrency and prioritized identification requirements can be achieved.
[0082] Correspondingly, see Figure 4, simulate the browser access to the URL to be identified through the automated test instance to obtain the target page, including: Step S400: Select a target automated test instance from the automated test instances of the multiple levels.
[0083] Step S402: simulating a browser to access the URL to be identified through the target automated test instance to obtain a target page.
[0084] In this embodiment, when selecting a target automated test instance from the multiple levels of automated test instances, priority information of the URL to be identified can be first determined. Then, an automated test instance of a level that matches the priority information can be selected from the multiple levels of automated test instances. Thereafter, the target automated test instance can be selected from one or more automated test instances included in the automated test instances of that level.
[0085] When selecting a target automated test instance from one or more automated test instances included in a certain level of automated test instances, the target automated test instance may be selected randomly, or an idle automated test instance may be selected as the target automated test instance.
[0086] It should be noted that when a target automated test instance is selected randomly, if the target automated test instance is not currently idle, it is necessary to wait until the automated test instance is idle before simulating a browser through the automated test instance to access the URL to be identified and obtain the target page.
[0087] In one embodiment, to distinguish whether each automated test instance is an idle automated test instance, the current status of each automated test instance can be marked when it is used, and its current status can be marked again after use is completed. In this way, the marking information of the automated test instance can accurately distinguish whether the current automated test instance is an idle automated test instance or a non-idle automated test instance.
[0088] In this embodiment, a target automated test instance is selected from the automated test instances of the multiple levels to simulate a browser accessing the URL to be identified to obtain a target page, thereby enabling identification of multiple URLs to be identified of different priorities at the same time.
[0089] In an optional embodiment, selecting a target automated test instance from the automated test instances of the multiple levels includes: Obtaining priority information of the website to be identified; An idle automated test instance is selected from the automated test instances of the multiple levels as a target automated test instance based on the priority information.
[0090] In this embodiment, the automated test instances corresponding to the URLs to be identified of different priorities are different. The higher the priority of the URL to be identified, the more levels of automated test instances that can be selected. For example, when the priority information of the URL to be identified is "high priority", an automated test instance can be selected from the high-level, medium-level and low-level automated test instances as the target automated test instance. When the priority information of the URL to be identified is "medium priority", the target automated test instance can be selected from the medium-level and low-level automated test instances. When the priority information of the URL to be identified is "low priority", the target automated test instance can only be selected from the low-level automated test instances.
[0091] In a specific scenario, when the priority information of the URL to be identified is "high priority", it is possible to query whether there are idle automated test instances in the "high-level" automated test instances. If so, the idle automated test instances can be used as target automated test instances. If there are no idle automated test instances in the "high-level" automated test instances, it is possible to further query whether there are idle automated test instances in the "medium-level" automated test instances. If so, the idle automated test instances can be used as target automated test instances. If there are no idle automated test instances in the "medium-level" automated test instances, it is possible to further query whether there are idle automated test instances in the "low-level" automated test instances. If so, the idle automated test instances can be used as target automated test instances.
[0092] In a specific scenario, when the priority information of the URL to be identified is "medium priority", it is possible to query whether there are idle automated test instances in the "medium-level" automated test instances. If so, the idle automated test instance can be used as the target automated test instance. If there are no idle automated test instances in the "medium-level" automated test instances, it is possible to further query whether there are idle automated test instances in the "low-level" automated test instances. If so, the idle automated test instance can be used as the target automated test instance.
[0093] In a specific scenario, when the priority information of the URL to be identified is "low priority", you can only query whether there is an idle automated test instance in the "low-level" automated test instance. If so, the idle automated test instance can be used as the target automated test instance. If not, an automated test instance can be randomly selected from one or more automated test instances included in the "low-level" automated test instance as the target automated test instance.
[0094] In this embodiment, the above-mentioned method can ensure that the high-priority URL to be identified is identified and processed in the shortest possible time.
[0095] In an alternative embodiment, see Figure 5 , the method further comprises: Step S500: After the target automated test instance is selected, a locking operation is performed on the target automated test instance.
[0096] Step S502: After simulating a browser access to the to-be-identified website through the target automated test instance, unlocking the target automated test instance.
[0097] In this embodiment, after the target automated test instance is selected, the target automated test instance is locked so that when another URL needs to be identified, the selected target automated test instance cannot be used to simulate a browser to access the URL, thereby avoiding conflicts. After the target automated test instance simulates a browser to access the URL to be identified, the target automated test instance is unlocked so that the target automated test instance can be released in a timely manner, so that the target automated test instance can be used again to simulate a browser to access other URLs.
[0098] In an optional embodiment, the method further includes: When the URL to be identified is a URL of a preset type, the URL to be identified is intercepted by a URL interception system.
[0099] In this embodiment, when the URL to be identified is identified as a URL of a preset type, the URL to be identified is intercepted by the URL interception system to prevent the webpage content corresponding to the URL to be identified from being obtained by the user.
[0100] Example 2 Figure 6The block diagram of the URL identification device 600 according to the second embodiment of the present application is schematically shown. The device can be divided into one or more program modules, one or more program modules are stored in a storage medium and executed by one or more processors to complete the embodiment of the present application. The program module referred to in the embodiment of the present application refers to a series of computer program instruction segments that can complete specific functions. The following description will specifically introduce the functions of each program module in this embodiment. Figure 6 As shown, the website identification device 600 may include: an acquisition module 610, an access module 620, an extraction module 630, an analysis module 640 and a determination module 650, wherein: An acquisition module 610 is used to acquire a URL to be identified; An access module 620 is configured to simulate a browser accessing the URL to be identified through an automated test instance to obtain a target page; An extraction module 630 is configured to extract webpage content from the source code corresponding to the target page and to take a screenshot of the target page to obtain a page image; An analysis module 640 is configured to analyze the page image using a large language model to obtain a page analysis result; The determination module 650 is configured to determine whether the URL to be identified is a URL of a preset type based on the webpage content and the page analysis result.
[0101] As an optional embodiment, simulating a browser to access the URL to be identified through an automated test instance to obtain a target page includes: Simulate a browser to access the URL to be identified through an automated test instance; If the target page corresponding to the URL to be identified is not obtained within the preset time, a null value is returned; When the target page corresponding to the URL to be identified is obtained after the preset time period, the target page is returned in an asynchronous manner.
[0102] As an optional embodiment, the website address identification device 600 further includes a creation module.
[0103] The creation module is used to create an automated test instance pool, which includes automated test instances of multiple levels.
[0104] The step of simulating a browser to access the URL to be identified through an automated test instance to obtain a target page includes: Selecting a target automated test instance from the automated test instances of the multiple levels; The target automated test instance simulates a browser accessing the URL to be identified to obtain a target page.
[0105] As an optional embodiment, the website identification device 600 further includes a locking module and an unlocking module.
[0106] The locking module is used to lock the target automated test instance after the target automated test instance is selected; The unlocking module is used to unlock the target automated test instance after simulating a browser to access the to-be-identified website through the target automated test instance.
[0107] As an optional embodiment, the selecting a target automated test instance from the automated test instances of the multiple levels includes: Obtaining priority information of the to-be-identified website; and selecting an idle automated test instance from the multiple levels of automated test instances as a target automated test instance based on the priority information.
[0108] As an optional embodiment, the website address identification device 600 further includes an interception module.
[0109] The interception module is used to intercept the URL to be identified through a URL interception system when the URL to be identified is a URL of a preset type.
[0110] As an optional embodiment, determining whether the to-be-identified URL is a URL of a preset type based on the webpage content and the page analysis result includes: When the webpage content includes content of a preset type and the page analysis result is a preset page analysis result, it is determined that the URL to be identified is a URL of the preset type.
[0111] Example 3 Figure 7 The following schematically shows a hardware architecture diagram of a computer device 10000 suitable for implementing a URL identification method according to the third embodiment of the present application. In some embodiments, the computer device 10000 may be a terminal device such as a smartphone, a wearable device, a tablet computer, a personal computer, a vehicle-mounted terminal, a game console, a virtual device, a workbench, a digital assistant, a set-top box, a robot, etc. In other embodiments, the computer device 10000 may be a rack server, a blade server, a tower server, or a cabinet server (including an independent server or a server cluster composed of multiple servers). Figure 7 As shown, the computer device 10000 includes but is not limited to: a memory 10010, a processor 10020, and a network interface 10030 that can communicate with each other via a system bus. Memory 10010 includes at least one type of computer-readable storage medium, including flash memory, a hard disk, a multimedia card, card-type memory (e.g., SD or DX memory), random access memory (RAM), static random access memory (SRAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), programmable read-only memory (PROM), magnetic storage, a magnetic disk, an optical disk, and the like. In some embodiments, memory 10010 may be an internal storage module of computer device 10000, such as a hard disk or memory of computer device 10000. In other embodiments, memory 10010 may also be an external storage device of computer device 10000, such as a plug-in hard disk, a Smart Media Card (SMC), a Secure Digital (SD) card, a flash memory card, and the like equipped on computer device 10000. Of course, memory 10010 may also include both internal storage modules and external storage devices of computer device 10000. In this embodiment, the memory 10010 is generally used to store the operating system and various application software installed on the computer device 10000, such as the program code of the website identification method, etc. In addition, the memory 10010 can also be used to temporarily store various data that has been output or is to be output.
[0112] In some embodiments, processor 10020 may be a central processing unit (CPU), a controller, a microcontroller, a microprocessor, or other chips. Processor 10020 is typically used to control the overall operation of computer device 10000, such as performing control and processing related to data exchange or communication with computer device 10000. In this embodiment, processor 10020 is used to execute program code stored in memory 10010 or process data.
[0113] Network interface 10030 may include a wireless network interface or a wired network interface. Network interface 10030 is typically used to establish a communication link between computer device 10000 and other computer devices. For example, network interface 10030 is used to connect computer device 10000 to an external terminal via a network, establishing a data transmission channel and a communication link between computer device 10000 and the external terminal. The network may be a wireless or wired network such as an intranet, the Internet, the Global System of Mobile Communications (GSM), Wideband Code Division Multiple Access (WCDMA), a 4G network, a 5G network, Bluetooth, or Wi-Fi.
[0114] It should be pointed out that Figure 7 Only a computer device having components 10010 - 10030 is shown, but it should be understood that implementing all of the shown components is not a requirement, and more or fewer components may alternatively be implemented.
[0115] In this embodiment, the URL identification method stored in the memory 10010 can also be divided into one or more program modules and executed by one or more processors (such as processor 10020) to complete the embodiment of the present application.
[0116] Example 4 An embodiment of the present application further provides a computer-readable storage medium having a computer program stored thereon, wherein when the computer program is executed by a processor, the steps of the URL identification method in the embodiment are implemented.
[0117] In this embodiment, computer-readable storage media include flash memory, hard disks, multimedia cards, card-type memories (e.g., SD or DX memories), random access memory (RAM), static random access memory (SRAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), programmable read-only memory (PROM), magnetic storage, magnetic disks, optical disks, and the like. In some embodiments, the computer-readable storage medium may be an internal storage unit of a computer device, such as the computer device's hard disk or memory. In other embodiments, the computer-readable storage medium may also be an external storage device of the computer device, such as a plug-in hard disk, a Smart Media Card (SMC), a Secure Digital (SD) card, a flash memory card, and the like. Of course, the computer-readable storage medium may also include both the internal storage unit and external storage devices of the computer device. In this embodiment, the computer-readable storage medium is typically used to store the operating system and various application software installed on the computer device, such as the program code of the website address identification method described in the embodiment. In addition, the computer-readable storage medium may also be used to temporarily store various types of data that has been output or is about to be output.
[0118] Example 5 An embodiment of the present application further provides a computer program product, including a computer program, which implements the method in the above embodiment when executed by a processor.
[0119] Obviously, those skilled in the art should understand that the modules or steps of the above-mentioned embodiments of the present application can be implemented using general-purpose computer devices. They can be concentrated on a single computer device or distributed on a network composed of multiple computer devices. Alternatively, they can be implemented using program codes executable by the computer device, so that they can be stored in a storage device and executed by the computer device. In some cases, the steps shown or described can be performed in a different order than herein, or they can be made into individual integrated circuit modules, or multiple modules or steps therein can be made into a single integrated circuit module for implementation. Thus, the embodiments of the present application are not limited to any specific combination of hardware and software.
[0120] It should be noted that the above are only preferred embodiments of the present application and do not limit the scope of patent protection of the present application. Any equivalent structure or equivalent process transformation made using the contents of the description and drawings of this application, or directly or indirectly applied in other related technical fields, are also included in the scope of patent protection of the present application.
Claims
1. A method for identifying a website, characterized in that: The method comprises: Get the URL to be identified; Through the automated test instance, simulate the browser to access the URL to be identified to obtain the target page; Extracting webpage content from the source code corresponding to the target page, and taking a screenshot of the target page to obtain a page image; Analyze the page image using a large language model to obtain a page analysis result; Based on the webpage content and the page analysis result, it is determined whether the URL to be identified is a URL of a preset type.
2. The method according to claim 1, characterized in that The step of simulating a browser to access the URL to be identified by an automated test instance to obtain a target page includes: Simulate a browser to access the URL to be identified through an automated test instance; If the target page corresponding to the URL to be identified is not obtained within the preset time, a null value is returned; When the target page corresponding to the URL to be identified is obtained after the preset time period, the target page is returned in an asynchronous manner.
3. The method according to claim 1, characterized in that The method further comprises: Creating an automated test instance pool, wherein the automated test instance pool includes automated test instances of multiple levels; The step of simulating a browser to access the URL to be identified through an automated test instance to obtain a target page includes: Selecting a target automated test instance from the automated test instances of the multiple levels; The target automated test instance simulates a browser accessing the URL to be identified to obtain a target page.
4. The method according to claim 3, characterized in that The method further comprises: After selecting the target automated test instance, locking the target automated test instance; After simulating a browser to access the to-be-identified website through the target automated test instance, an unlocking operation is performed on the target automated test instance.
5. The method according to claim 3, characterized in that The selecting a target automated test instance from the automated test instances of the multiple levels includes: Obtaining priority information of the website to be identified; An idle automated test instance is selected from the automated test instances of the multiple levels as a target automated test instance based on the priority information.
6. The method according to any one of claims 1 to 5, characterized in that The method further comprises: When the URL to be identified is a URL of a preset type, the URL to be identified is intercepted by a URL interception system.
7. The method according to any one of claims 1 to 5, characterized in that Determining whether the to-be-identified URL is a URL of a preset type based on the webpage content and the page analysis result includes: When the webpage content includes content of a preset type and the page analysis result is a preset page analysis result, it is determined that the URL to be identified is a URL of the preset type.
8. A website identification device, characterized in that: The device comprises: The acquisition module is used to obtain the URL to be identified; An access module is used to simulate a browser accessing the URL to be identified through an automated test instance to obtain a target page; An extraction module is used to extract webpage content from the source code corresponding to the target page, and to take a screenshot of the target page to obtain a page image; An analysis module, configured to analyze the page image using a large language model to obtain a page analysis result; The determination module is used to determine whether the URL to be identified is a URL of a preset type based on the webpage content and the page analysis result.
9. A computer device, characterized in that: include: at least one processor; and a memory communicatively coupled to the at least one processor; wherein: The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the method according to any one of claims 1 to 7.
10. A computer-readable storage medium, characterized in that The computer-readable storage medium stores computer instructions, which, when executed by a processor, implement the method according to any one of claims 1 to 7.
11. A computer program product comprising a computer program, characterized in that When the computer program is executed by a processor, the steps of the method according to claims 1 to 7 are implemented.