Scalable rendering method and device, equipment and storage medium

Abstract

The invention belongs to the field of computers, and discloses a scaling rendering method and device, equipment and a storage medium. The method comprises the following steps: receiving target object data corresponding to a to-be-crawled network resource; obtaining rendering parameters selected by a user based on the target object data; and performing telescopic rendering on the target object data according to the rendering parameters. The rendering parameters selected by the user based on the target object data are acquired; and performing telescopic rendering on target object data corresponding to the to-be-crawled network resources according to the rendering parameters. Compared with an existing mode that all data corresponding to the to-be-crawled network resources are rendered and displayed, rendering can be conducted according to the rendering parameters selected by the user, the data expected to be obtained by the user is visually displayed, and the user experience feeling is improved.

Description

Technical Field

[0001] The present invention relates to the field of computer technology, and in particular to a scalable rendering method, device, equipment and storage medium. Background Art

[0002] With the development of Internet technology, most websites nowadays are dynamically rendered and generated. Ordinary GET can no longer obtain the complete website content. However, if the crawler system wants to obtain the complete content, it must use the method of simulating browser rendering to obtain the complete HTML. There are many ways to realize the function of simulating browsers. The more commonly used one is phantomjs, which allows the program to simulate the browser to open the web page without an interface, obtain the real page content, and then render and display the crawled content for users to view. The rendering process generally involves file parsing, style calculation, building DOM tree and rendering tree and other steps. The final result can only be obtained through programmatic viewing, which is troublesome to debug and has low rendering efficiency.

[0003] The above contents are only used to assist in understanding the technical solution of the present invention and do not constitute an admission that the above contents are prior art. Summary of the invention

[0004] The main purpose of the present invention is to provide a scalable rendering method, device, equipment and storage medium, aiming to solve the technical problem of low data rendering efficiency in the prior art due to the large amount of crawled data when crawling data.

[0005] To achieve the above object, the present invention provides a scalable rendering method, which comprises the following steps:

[0006] Receive target object data corresponding to the network resources to be crawled;

[0007] Acquire rendering parameters selected by a user based on the target object data;

[0008] The target object data is rendered in a scalable manner according to the rendering parameters.

[0009] Optionally, after the step of receiving the target object data corresponding to the network resource to be crawled, the method further includes:

[0010] Determine whether the target data corresponding to the network resource to be crawled has been loaded according to the target object data;

[0011] If the target data has not been loaded, then the loading time of the target object data is obtained;

[0012] When the loading time is longer than a preset loading time, the loading of the target data corresponding to the network resource to be crawled is stopped.

[0013] Optionally, the step of judging whether the target data corresponding to the network resource to be crawled has been loaded according to the target object data includes:

[0014] Start the preset timer to start timing;

[0015] Determine whether the DOM number corresponding to the target object data changes within a preset time range;

[0016] If yes, reset the preset timer and return to the step of determining whether the DOM number corresponding to the target object data changes within the preset time range;

[0017] If not, it is determined that the target data corresponding to the to-be-crawled network resource has been loaded.

[0018] Optionally, the step of receiving target object data corresponding to the network resource to be crawled includes:

[0019] Receive web crawling tasks;

[0020] Determine the network resources to be crawled according to the network crawling task;

[0021] The target object data corresponding to the network resources to be crawled is obtained through a preset browser running in incognito mode.

[0022] Optionally, the step of receiving a network crawling task includes:

[0023] Monitor the number of currently rendered pages and resource utilization through the preset monitoring thread;

[0024] Determine whether the preset conditions are met according to the current number of rendered pages, resource utilization and hardware configuration information;

[0025] If not satisfied, accept the web crawling task.

[0026] Optionally, after the step of receiving the target object data corresponding to the network resource to be crawled, the method further includes:

[0027] Obtain a target service for rendering the target object data in a distributed cluster;

[0028] Obtaining an operation interface corresponding to the target service;

[0029] The operation interface is displayed so that the user can input rendering parameters based on the operation interface.

[0030] Optionally, the step of performing scalable rendering on the target object data according to the rendering parameters includes:

[0031] The rendering parameters are input into a preset Lua script, so that the preset Lua script performs scaling rendering on the target object data based on the rendering parameters.

[0032] Optionally, after the step of inputting the rendering parameters into a preset Lua script, the method further includes:

[0033] Obtain identification information of the preset Lua script based on the rendering parameter feedback;

[0034] When the identification information meets a preset condition, the rendering parameter is sent to the target terminal, so that the target terminal updates the preset Lua script according to the rendering parameter.

[0035] Optionally, after the step of performing scaling rendering on the target object data according to the rendering parameters, the step further includes:

[0036] When the current service running time reaches the preset time, obtain the current memory resource usage information;

[0037] Determine whether there is a memory leak according to the current memory resource usage information;

[0038] When there is a memory leak, restart the current service.

[0039] Optionally, the step of determining whether there is a memory leak according to the current memory resource usage information includes:

[0040] Determine a current memory allocation frequency and a current memory release frequency according to the current memory resource usage information;

[0041] Whether there is a memory leak is determined according to the current memory allocation frequency and the current memory release frequency.

[0042] Optionally, after the step of receiving the target object data corresponding to the network resource to be crawled, the method further includes:

[0043] Get the test task;

[0044] According to the test task, the target object data is tested based on a preset Lua script to obtain a test result, wherein the test result includes whether the network resource to be crawled is successfully crawled.

[0045] In addition, to achieve the above-mentioned purpose, the present invention also provides a scalable rendering device, the device comprising:

[0046] A receiving module, used to receive target object data corresponding to the network resources to be crawled;

[0047] An acquisition module, used for acquiring rendering parameters selected by a user based on the target object data;

[0048] A rendering module is used to perform scaling rendering on the target object data according to the rendering parameters.

[0049] In addition, to achieve the above-mentioned purpose, the present invention also proposes a telescopic rendering device, which includes: a memory, a processor, and a telescopic rendering program stored in the memory and executable on the processor, wherein the telescopic rendering program is configured to implement the steps of the telescopic rendering method described above.

[0050] In addition, to achieve the above objective, the present invention further proposes a storage medium, on which a scalable rendering program is stored, and when the scalable rendering program is executed by a processor, the steps of the scalable rendering method described above are implemented.

[0051] The present invention receives target object data corresponding to the network resources to be crawled; obtains rendering parameters selected by the user based on the target object data; and performs scalable rendering on the target object data according to the rendering parameters. Since the present invention obtains rendering parameters selected by the user based on the target object data; and performs scalable rendering on the target object data corresponding to the network resources to be crawled according to the rendering parameters. Compared with the existing method of rendering and displaying all data corresponding to the network resources to be crawled, the above method of the present invention can render according to the rendering parameters selected by the user, intuitively displays the data that the user expects to obtain, and improves the user experience. BRIEF DESCRIPTION OF THE DRAWINGS

[0052] Figure 1 It is a structural diagram of a scalable rendering device in a hardware operating environment involved in an embodiment of the present invention;

[0053] Figure 2 It is a flowchart of a first embodiment of a scalable rendering method of the present invention;

[0054] Figure 3 A schematic diagram of a rendering control process of a first embodiment of a scalable rendering method according to the present invention;

[0055] Figure 4 It is a flowchart of a second embodiment of the scalable rendering method of the present invention;

[0056] Figure 5 It is a flowchart of a third embodiment of the scalable rendering method of the present invention;

[0057] Figure 6 FIG. 4 is a structural block diagram of a first embodiment of a scalable rendering device according to the present invention.

[0058] The realization of the purpose, functional features and advantages of the present invention will be further explained in conjunction with embodiments and with reference to the accompanying drawings. DETAILED DESCRIPTION

[0059] It should be understood that the specific embodiments described herein are only used to explain the present invention, and are not used to limit the present invention.

[0060] Reference Figure 1 , Figure 1 The figure is a schematic diagram of the structure of a scalable rendering device in a hardware operating environment involved in an embodiment of the present invention.

[0061] like Figure 1 As shown, the telescopic rendering device may include: a processor 1001, such as a central processing unit (CPU), a communication bus 1002, a user interface 1003, a network interface 1004, and a memory 1005. Among them, the communication bus 1002 is used to realize the connection and communication between these components. The user interface 1003 may include a display screen (Display), an input unit such as a keyboard (Keyboard), and the optional user interface 1003 may also include a standard wired interface and a wireless interface. The network interface 1004 may optionally include a standard wired interface and a wireless interface (such as a wireless fidelity (Wireless-Fidelity, WI-FI) interface). The memory 1005 may be a high-speed random access memory (Random Access Memory, RAM), or a stable non-volatile memory (Non-Volatile Memory, NVM), such as a disk storage. The memory 1005 may also be a storage device independent of the aforementioned processor 1001.

[0062] Those skilled in the art will understand that Figure 1 The structure shown in the figure does not constitute a limitation on the telescopic rendering device, and may include more or fewer components than shown in the figure, or combine certain components, or arrange the components differently.

[0063] like Figure 1 As shown, the memory 1005 as a storage medium may include an operating system, a network communication module, a user interface module, and a scalable rendering program.

[0064] exist Figure 1 In the illustrated telescopic rendering device, the network interface 1004 is mainly used for data communication with a network server; the user interface 1003 is mainly used for data interaction with a user; the processor 1001 and the memory 1005 in the telescopic rendering device of the present invention can be set in the telescopic rendering device, and the telescopic rendering device calls the telescopic rendering program stored in the memory 1005 through the processor 1001, and executes the telescopic rendering method provided in the embodiment of the present invention.

[0065] Based on the above-mentioned scalable rendering device, an embodiment of the present invention provides a scalable rendering method, referring to Figure 2 , Figure 2 FIG. 4 is a flow chart of a first embodiment of a scalable rendering method according to the present invention.

[0066] In this embodiment, the scalable rendering method includes the following steps:

[0067] Step S10: Receive target object data corresponding to the network resources to be crawled.

[0068] It should be noted that the execution subject of this embodiment can be a computing service device with data processing, network communication and program running functions, such as a mobile phone, a tablet computer, a personal computer, etc., or an electronic device or a scalable rendering device that can realize the above functions. The following takes the scalable rendering device as an example to illustrate this embodiment and the following embodiments.

[0069] It should be noted that the network resources to be crawled may be network resources to be crawled in the data crawling instruction sent by the user, such as web page links. The data crawling instruction may be a command sent by the user to crawl network resources. The target object data may be all web page data corresponding to the network resources to be crawled, or may be the homepage network resource of the network resources to be crawled. The target object data corresponding to the network resources to be crawled may be crawled by the existing data crawling method.

[0070] Furthermore, in order to avoid obtaining target object data that is not updated locally, the step S10 may include:

[0071] Step S101: receiving a web crawling task;

[0072] Step S102: determining the network resources to be crawled according to the network crawling task;

[0073] Step S103: obtaining target object data corresponding to the network resources to be crawled through a preset browser running in the incognito mode.

[0074] It should be noted that the web crawling task can be a data crawling task generated by a user. The web resources to be crawled may include web links, logos, etc. of the data to be crawled. The preset browser may be a browser such as Chrome for obtaining web resources to be crawled. Incognito Mode (also known as privacy mode or private browsing) is a browser function that allows users to browse the web without leaving any traces. When a user browses a website in incognito mode, the browser will not save access records, search records, cached files, download records, form filling data, etc. Once the incognito window is closed, the relevant information will be automatically cleared.

[0075] Step S20: obtaining rendering parameters selected by the user based on the target object data.

[0076] It should be noted that the rendering parameters may be related parameters of the information that needs to be rendered and displayed based on the target object data selected by the user. For example, if the network resource to be crawled is website A, and the user wants to obtain pictures in website A, then the rendering parameters may be pictures. Then, when rendering the web page content in website A that has been crawled, it is not necessary to render all the crawled data, but to extract pictures from the web page content in website A for rendering. If the target object data is the homepage network resource of the network resource to be crawled, the rendering parameters may also be the target resource selected by the user based on a link or jump page in the homepage network resource. When the rendering parameters are the target resource selected by the user based on the homepage network resource, the page corresponding to the target resource is rendered.

[0077] Step S30: performing scaling rendering on the target object data according to the rendering parameters.

[0078] It should be noted that performing scalable rendering on the target object data according to the rendering parameters may be extracting the data to be rendered from the target object data according to the rendering parameters, and then rendering the data to be rendered.

[0079] Furthermore, for various problems that may arise during the rendering process, an intuitive interactive platform is usually required to complete the testing and verification process. In the prior art, the HTML content is checked after the program is executed to check whether the result is abnormal. This method has low operating efficiency. Therefore, in order to improve the rendering efficiency, the step S30 may include: inputting the rendering parameters into a preset Lua script, so that the preset Lua script performs scalable rendering on the target object data based on the rendering parameters.

[0080] It should be noted that Lua is a lightweight, high-level scripting language, which is mainly used to be embedded in other applications to provide scalability and flexibility. The preset Lua script can be a script program written based on the user's crawling and rendering requirements that can perform scalable rendering on the target object data according to rendering parameters.

[0081] Furthermore, in order to meet different rendering requirements of users, after the step of inputting the rendering parameters into the preset Lua script, it also includes:

[0082] Obtain identification information of the preset Lua script based on the rendering parameter feedback;

[0083] When the identification information meets a preset condition, the rendering parameter is sent to the target terminal, so that the target terminal updates the preset Lua script according to the rendering parameter.

[0084] It should be noted that, since the rendering parameters can be specific rendering conditions input by the user according to the requirements, it is inevitable that the preset Lua script cannot be recognized. Therefore, when the preset Lua script cannot recognize the rendering parameters based on the identification information fed back by the rendering parameters, the rendering parameters can be sent to the target terminal so that the target terminal updates the preset Lua script according to the rendering parameters.

[0085] Furthermore, in order to support simultaneous rendering of multiple pages, just like opening multiple tabs in a browser, and to enable resources to be effectively controlled, after step S10, the method further includes: obtaining a target service for rendering the target object data in a distributed cluster;

[0086] Obtaining an operation interface corresponding to the target service;

[0087] The operation interface is displayed so that the user can input rendering parameters based on the operation interface.

[0088] It should be noted that the specific Figure 3 , Figure 3 This is a schematic diagram of the rendering control process of the first embodiment of the scalable rendering method of the present invention; in order to improve the user experience, the scalable rendering device can use different services to render multiple crawled target object data at the same time in a distributed cluster manner. Figure 3 As shown, the scalable rendering device may correspond to multiple rendering services. When the scalable rendering device receives a web crawling task, it may allocate the web crawling task based on the current resource utilization of each rendering service. The target service may be a rendering service responsible for rendering the target object data. The rendering service may be further divided into multiple rendering subsystems. When the rendering service is further divided into multiple rendering subsystems, the target service may be the rendering subsystem responsible for rendering the target object data. For example, when the scalable rendering device receives a web crawling task A, it allocates the web crawling task A to rendering service 1 for rendering. Then the target service is rendering service 1. When the scalable rendering device receives a web crawling task B, it allocates the web crawling task B to rendering subsystem 1 in rendering service 2 for rendering. Then the target service is rendering subsystem 1 in rendering service 2. As Figure 3 As shown, each rendering service is configured with an interface operation service, which can allow users to input rendering parameters by providing an operation interface.

[0089] Furthermore, in order to avoid memory leakage, after step S30, the method further includes: when the current service running time reaches a preset time, obtaining current memory resource usage information;

[0090] Determine whether there is a memory leak according to the current memory resource usage information;

[0091] When there is a memory leak, restart the current service.

[0092] It should be noted that, since the risk of memory leak is greater after a long running time, when the running time of the current service reaches a preset time, it starts to determine whether there is a memory leak. The preset time can be a pre-set running time to start determining whether there is a memory leak. The current service can be a rendering service that performs scalable rendering on target object data, or it can be a rendering subsystem in a rendering service. The determination of whether there is a memory leak based on the current memory resource usage information can be determined by the following aspects:

[0093] Continuously growing memory usage: If the current memory resource usage information shows that the current memory usage continues to grow and is not released even after the rendering task is completed, it is determined that there is a memory leak;

[0094] Memory release mismatch: If the current memory resource usage information shows that the frequency of memory allocation is much greater than the frequency of memory release, it is determined that there is a memory leak. You can also use memory leak detection tools such as Valgrind, LeakTracer, and AddressSanitizer to monitor memory usage to help discover and locate potential memory leaks.

[0095] This embodiment receives target object data corresponding to the network resources to be crawled; obtains rendering parameters selected by the user based on the target object data; and performs scalable rendering on the target object data according to the rendering parameters. Since this embodiment obtains rendering parameters selected by the user based on the target object data; and performs scalable rendering on the target object data corresponding to the network resources to be crawled according to the rendering parameters. Compared with the existing method of rendering and displaying all data corresponding to the network resources to be crawled, the above method of this embodiment can render according to the rendering parameters selected by the user, intuitively displays the data that the user expects to obtain, and improves the user experience.

[0096] refer to Figure 4 , Figure 4 FIG. 4 is a flow chart of a second embodiment of a scalable rendering method according to the present invention.

[0097] Based on the first embodiment above, in this embodiment, after step S10, the following steps are further included:

[0098] Step S101: judging whether the target data corresponding to the network resource to be crawled has been loaded according to the target object data.

[0099] It should be noted that the target data may be a data resource that needs to be obtained from a server or locally corresponding to the network resource to be crawled. For example, a certain picture in the network resource to be crawled, a certain subpage, etc. The determination of whether the target data corresponding to the network resource to be crawled has been loaded based on the target object data may be to determine whether the target data corresponding to the network resource to be crawled has been loaded based on a preset monitoring event. For example, the window.onload event or the DOMContentLoaded event or the MutationObserver may be used to determine whether the target data corresponding to the network resource to be crawled has been loaded.

[0100] Furthermore, in order to accurately determine whether the target data has been loaded, the step S101 may include: starting a preset timer to start timing;

[0101] Determine whether the DOM number corresponding to the target object data changes within a preset time range;

[0102] If yes, reset the preset timer and return to the step of determining whether the DOM number corresponding to the target object data changes within the preset time range;

[0103] If not, it is determined that the target data corresponding to the to-be-crawled network resource has been loaded.

[0104] It should be noted that the preset time range may be a preset time period for determining whether the target data has been loaded. For example, a period of 0S-2S, when the timing time of the preset timer is within 0S-2S, the DOM number changes, and it is determined that the target data has not been loaded. If the timing time of the preset timer is equal to 2S, the DOM number does not change, and it is determined that the target data has been loaded.

[0105] Step S102: If the target data has not been loaded, the loading time of the target object data is obtained.

[0106] It should be noted that the loading time of the target object data may include the total loading time of the target object data and / or the loading time of a certain data in the target object data. For example, if the target data corresponding to the network resource to be crawled includes multiple web pages, the loading time may be the loading time of loading the multiple web pages at the same time, or the loading time of each web page in the target data to be loaded.

[0107] Step S103: When the loading time is longer than a preset loading time, the loading of the target data corresponding to the network resource to be crawled is stopped.

[0108] It should be noted that the preset loading time can be a preset maximum loading time for loading the target data. Different preset loading times can also be set for different data that need to be loaded in the target data. For example, the preset loading time of page 1 in the target data is 1s, and the preset loading time of page 2 in the target data is 2s. In order to prevent resources from being occupied and unable to be released, when the loading time is greater than the preset loading time, the loading of the target data corresponding to the network resource to be crawled is stopped. Only the loaded target object data is displayed.

[0109] Furthermore, if the user crawls the network resources in order to obtain specific target data, then in order to ensure that the target data is obtained, after step S10, the step further includes: obtaining a test task;

[0110] According to the test task, the target object data is tested based on a preset Lua script to obtain a test result, wherein the test result includes whether the network resource to be crawled is successfully crawled.

[0111] It should be noted that the test task may be a test requirement for testing whether the acquired target object data contains the target data expected by the user. The testing of the target object data based on the preset Lua script according to the test task may be to query the target object data through the preset Lua script whether the target data in the test task exists. If the target data in the test task exists in the target object data, it is determined that the crawling of the network resource to be crawled is successful. At this time, even if other data corresponding to the network resource to be crawled has not been crawled, the crawling is terminated and the target data in the test task is rendered.

[0112] This embodiment determines whether the target data corresponding to the network resource to be crawled has been loaded according to the target object data; if the target data has not been loaded, the loading time of the target object data is obtained; when the loading time is greater than the preset loading time, the loading of the target data corresponding to the network resource to be crawled is stopped. This can prevent resources from being occupied and unable to be released, and improve data crawling efficiency.

[0113] refer to Figure 5 , Figure 5 FIG. 4 is a flow chart of a third embodiment of a scalable rendering method according to the present invention.

[0114] Based on the above embodiments, in this embodiment, step S101 includes:

[0115] Step S1011: monitor the number of currently rendered pages and resource utilization through a preset monitoring thread.

[0116] It should be noted that the preset monitoring thread may be a thread for monitoring the number of currently rendered pages and resource utilization of the scalable rendering device. The resource utilization may include the usage of resources such as CPU, memory and / or hard disk.

[0117] Step S1012: determining whether a preset condition is currently met based on the current number of rendered pages, resource utilization, and hardware configuration information.

[0118] It should be noted that the hardware configuration information may be the processor (CPU), memory (RAM), storage device (hard disk), network interface or GPU of the telescopic rendering device. The preset condition may be the service upper limit of the telescopic rendering device calculated based on the hardware configuration information, the current number of rendered pages and / or resource utilization. For example, based on the current number of rendered pages, resource utilization and hardware configuration information, it is calculated that the current service can also receive web crawling tasks. It is then determined that the current service does not meet the preset conditions. If it is calculated based on the current number of rendered pages, resource utilization and hardware configuration information that the current service pressure is too high and cannot receive web crawling tasks. It is then determined that the current service meets the preset conditions.

[0119] Step S1013: If not satisfied, receive the network crawling task.

[0120] This embodiment monitors the current number of rendered pages and resource utilization through a preset monitoring thread; determines whether the preset conditions are currently met based on the current number of rendered pages, resource utilization, and hardware configuration information; if not, receives a web crawling task. This embodiment monitors the current number of rendered pages and resource utilization through a preset monitoring thread, and determines whether the preset conditions are currently met based on the current number of rendered pages, resource utilization, and hardware configuration information, which can avoid the rendering service from being stuck due to excessive pressure and improve rendering efficiency.

[0121] Reference Figure 6 , Figure 6 FIG. 4 is a structural block diagram of a first embodiment of a scalable rendering device according to the present invention.

[0122] like Figure 6 As shown, the scalable rendering device proposed in the embodiment of the present invention includes:

[0123] A receiving module 10 is used to receive target object data corresponding to the network resources to be crawled;

[0124] An acquisition module 20, used to acquire rendering parameters selected by a user based on the target object data;

[0125] The rendering module 30 is used to perform scaling rendering on the target object data according to the rendering parameters.

[0126] This embodiment receives target object data corresponding to the network resources to be crawled; obtains rendering parameters selected by the user based on the target object data; and performs scalable rendering on the target object data according to the rendering parameters. Since this embodiment obtains rendering parameters selected by the user based on the target object data; and performs scalable rendering on the target object data corresponding to the network resources to be crawled according to the rendering parameters. Compared with the existing method of rendering and displaying all data corresponding to the network resources to be crawled, the above method of this embodiment can render according to the rendering parameters selected by the user, intuitively displays the data that the user expects to obtain, and improves the user experience.

[0127] It should be noted that the workflow described above is merely illustrative and does not limit the scope of protection of the present invention. In practical applications, technicians in this field can select part or all of them according to actual needs to achieve the purpose of the present embodiment, and no limitation is made here.

[0128] In addition, for technical details not fully described in this embodiment, reference may be made to the scalable rendering method provided in any embodiment of the present invention, and will not be repeated here.

[0129] Based on the first embodiment of the scalable rendering device of the present invention, a second embodiment of the scalable rendering device of the present invention is proposed.

[0130] In this embodiment, the receiving module 10 is further used to determine whether the target data corresponding to the to-be-crawled network resource has been loaded according to the target object data;

[0131] If the target data has not been loaded, then the loading time of the target object data is obtained;

[0132] When the loading time is longer than a preset loading time, the loading of the target data corresponding to the network resource to be crawled is stopped.

[0133] Furthermore, the receiving module 10 is also used to start a preset timer to start timing;

[0134] Determine whether the DOM number corresponding to the target object data changes within a preset time range;

[0135] If yes, reset the preset timer and return to the step of determining whether the DOM number corresponding to the target object data changes within the preset time range;

[0136] If not, it is determined that the target data corresponding to the to-be-crawled network resource has been loaded.

[0137] Furthermore, the receiving module 10 is also used to receive a network crawling task;

[0138] Determine the network resources to be crawled according to the network crawling task;

[0139] The target object data corresponding to the network resources to be crawled is obtained through a preset browser running in incognito mode.

[0140] Furthermore, the receiving module 10 is also used to monitor the number of currently rendered pages and resource utilization through a preset monitoring thread;

[0141] Determine whether the preset conditions are met according to the current number of rendered pages, resource utilization and hardware configuration information;

[0142] If not satisfied, accept the web crawling task.

[0143] Furthermore, the receiving module 10 is also used to obtain a target service for rendering the target object data in a distributed cluster;

[0144] Obtaining an operation interface corresponding to the target service;

[0145] The operation interface is displayed so that the user can input rendering parameters based on the operation interface.

[0146] Furthermore, the rendering module 30 is further configured to input the rendering parameters into a preset Lua script, so that the preset Lua script performs scalable rendering on the target object data based on the rendering parameters.

[0147] Furthermore, the rendering module 30 is further used to obtain identification information fed back by the preset Lua script based on the rendering parameters;

[0148] When the identification information meets a preset condition, the rendering parameter is sent to the target terminal, so that the target terminal updates the preset Lua script according to the rendering parameter.

[0149] Furthermore, the rendering module 30 is also used to obtain current memory resource usage information when the current service running time reaches a preset time;

[0150] Determine whether there is a memory leak according to the current memory resource usage information;

[0151] When there is a memory leak, restart the current service.

[0152] Furthermore, the rendering module 30 is further used to determine a current memory allocation frequency and a current memory release frequency according to the current memory resource usage information;

[0153] Whether there is a memory leak is determined according to the current memory allocation frequency and the current memory release frequency.

[0154] Furthermore, the receiving module 10 is also used to obtain a test task;

[0155] According to the test task, the target object data is tested based on a preset Lua script to obtain a test result, and the test result includes whether the network resource to be crawled is successfully crawled.

[0156] Other embodiments or specific implementations of the scalable rendering device of the present invention may refer to the above-mentioned method embodiments, which will not be described in detail here.

[0157] In addition, an embodiment of the present invention further provides a storage medium, on which a scalable rendering program is stored. When the scalable rendering program is executed by a processor, the steps of the scalable rendering method described above are implemented.

[0158] It should be noted that, in this article, the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, article or system including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or system. In the absence of further restrictions, an element defined by the sentence "comprises a ..." does not exclude the existence of other identical elements in the process, method, article or system including the element.

[0159] The serial numbers of the above embodiments of the present invention are only for description and do not represent the advantages or disadvantages of the embodiments.

[0160] Through the description of the above implementation methods, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of software plus a necessary general hardware platform, and of course can also be implemented by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present invention, or the part that contributes to the prior art, can be embodied in the form of a software product, which is stored in a storage medium (such as a read-only memory / random access memory, a magnetic disk, or an optical disk), and includes a number of instructions for enabling a terminal device (which can be a mobile phone, a computer, a server, an air conditioner, or a network device, etc.) to execute the methods described in each embodiment of the present invention.

[0161] The above are only preferred embodiments of the present invention, and are not intended to limit the patent scope of the present invention. Any equivalent structure or equivalent process transformation made using the contents of the present invention specification and drawings, or directly or indirectly applied in other related technical fields, are also included in the patent protection scope of the present invention.

[0162] The present invention discloses A1, a telescopic rendering method, the telescopic rendering method comprising the following steps:

[0163] Receive target object data corresponding to the network resources to be crawled;

[0164] Acquire rendering parameters selected by a user based on the target object data;

[0165] The target object data is rendered in a scalable manner according to the rendering parameters.

[0166] A2. The scalable rendering method as described in A1, after the step of receiving the target object data corresponding to the network resource to be crawled, further comprising:

[0167] Determine whether the target data corresponding to the network resource to be crawled has been loaded according to the target object data;

[0168] If the target data has not been loaded, then the loading time of the target object data is obtained;

[0169] When the loading time is longer than a preset loading time, the loading of the target data corresponding to the network resource to be crawled is stopped.

[0170] A3. In the telescopic rendering method described in A2, the step of determining whether the target data corresponding to the network resource to be crawled has been loaded according to the target object data comprises:

[0171] Start the preset timer to start timing;

[0172] Determine whether the DOM number corresponding to the target object data changes within a preset time range;

[0173] If yes, reset the preset timer and return to the step of determining whether the DOM number corresponding to the target object data changes within the preset time range;

[0174] If not, it is determined that the target data corresponding to the to-be-crawled network resource has been loaded.

[0175] A4. In the scalable rendering method described in A1, the step of receiving target object data corresponding to the network resource to be crawled comprises:

[0176] Receive web crawling tasks;

[0177] Determine the network resources to be crawled according to the network crawling task;

[0178] The target object data corresponding to the network resources to be crawled is obtained through a preset browser running in incognito mode.

[0179] A5. In the scalable rendering method as described in A4, the step of receiving a network crawling task comprises:

[0180] Monitor the number of currently rendered pages and resource utilization through the preset monitoring thread;

[0181] Determine whether the preset conditions are met according to the current number of rendered pages, resource utilization and hardware configuration information;

[0182] If not satisfied, accept the web crawling task.

[0183] A6. The scalable rendering method as described in A1, after the step of receiving the target object data corresponding to the network resource to be crawled, further comprising:

[0184] Obtain a target service for rendering the target object data in a distributed cluster;

[0185] Obtaining an operation interface corresponding to the target service;

[0186] The operation interface is displayed so that the user can input rendering parameters based on the operation interface.

[0187] A7. The scalable rendering method as described in A1, wherein the step of performing scalable rendering on the target object data according to the rendering parameters comprises:

[0188] The rendering parameters are input into a preset Lua script, so that the preset Lua script performs scaling rendering on the target object data based on the rendering parameters.

[0189] A8. The scalable rendering method as described in A7, after the step of inputting the rendering parameters into a preset Lua script, further comprising:

[0190] Obtain identification information of the preset Lua script based on the rendering parameter feedback;

[0191] When the identification information meets a preset condition, the rendering parameter is sent to the target terminal, so that the target terminal updates the preset Lua script according to the rendering parameter.

[0192] A9. The scalable rendering method according to any one of A1 to A8, further comprising, after the step of performing scalable rendering on the target object data according to the rendering parameters:

[0193] When the current service running time reaches the preset time, obtain the current memory resource usage information;

[0194] Determine whether there is a memory leak according to the current memory resource usage information;

[0195] When there is a memory leak, restart the current service.

[0196] A10. In the scalable rendering method described in A9, the step of determining whether there is a memory leak according to the current memory resource usage information comprises:

[0197] Determine a current memory allocation frequency and a current memory release frequency according to the current memory resource usage information;

[0198] Whether there is a memory leak is determined according to the current memory allocation frequency and the current memory release frequency.

[0199] A11. The scalable rendering method as described in any one of A1 to A8, after the step of receiving the target object data corresponding to the network resource to be crawled, further comprising:

[0200] Get the test task;

[0201] According to the test task, the target object data is tested based on a preset Lua script to obtain a test result, wherein the test result includes whether the network resource to be crawled is successfully crawled.

[0202] The present invention discloses B12, a telescopic rendering device, the telescopic rendering device comprising:

[0203] A receiving module, used to receive target object data corresponding to the network resources to be crawled;

[0204] An acquisition module, used for acquiring rendering parameters selected by a user based on the target object data;

[0205] A rendering module is used to perform scaling rendering on the target object data according to the rendering parameters.

[0206] B13, the retractable rendering device as described in B12, wherein the receiving module is further used to determine whether the target data corresponding to the network resource to be crawled is loaded according to the target object data;

[0207] If the target data has not been loaded, then the loading time of the target object data is obtained;

[0208] When the loading time is longer than a preset loading time, the loading of the target data corresponding to the network resource to be crawled is stopped.

[0209] B14. The scalable rendering device as described in B13, wherein the receiving module is further used to start a preset timer to start timing;

[0210] Determine whether the DOM number corresponding to the target object data changes within a preset time range;

[0211] If yes, reset the preset timer and return to the step of determining whether the DOM number corresponding to the target object data changes within the preset time range;

[0212] If not, it is determined that the target data corresponding to the to-be-crawled network resource has been loaded.

[0213] B15. The scalable rendering device as described in B12, wherein the receiving module is further used to receive a network crawling task;

[0214] Determine the network resources to be crawled according to the network crawling task;

[0215] The target object data corresponding to the network resources to be crawled is obtained through a preset browser running in incognito mode.

[0216] B16. The scalable rendering device as described in B15, wherein the receiving module is further used to monitor the number of currently rendered pages and resource utilization through a preset monitoring thread;

[0217] Determine whether the preset conditions are met according to the current number of rendered pages, resource utilization and hardware configuration information;

[0218] If not satisfied, accept the web crawling task.

[0219] B17, the scalable rendering device as described in B12, wherein the receiving module is further used to obtain a target service for rendering the target object data in a distributed cluster;

[0220] Obtaining an operation interface corresponding to the target service;

[0221] The operation interface is displayed so that the user can input rendering parameters based on the operation interface.

[0222] B18. The scalable rendering device as described in B12, wherein the rendering module is further used to obtain identification information of the preset Lua script based on the rendering parameter feedback;

[0223] When the identification information meets a preset condition, the rendering parameter is sent to the target terminal, so that the target terminal updates the preset Lua script according to the rendering parameter.

[0224] The present invention discloses C19, a telescopic rendering device, the device comprising: a memory, a processor, and a telescopic rendering program stored in the memory and executable on the processor, the telescopic rendering program being configured to implement the steps of the telescopic rendering method described in any one of A1 to A11.

[0225] The present invention discloses D20, a storage medium, on which a telescopic rendering program is stored. When the telescopic rendering program is executed by a processor, the steps of the telescopic rendering method as described in any one of A1 to A11 are implemented.

Claims

1. A telescopic rendering method, characterized in that: The telescopic rendering method comprises the following steps: Receive target object data corresponding to the network resources to be crawled; Acquire rendering parameters selected by a user based on the target object data; The target object data is rendered in a scalable manner according to the rendering parameters.

2. The telescopic rendering method according to claim 1, wherein: After the step of receiving the target object data corresponding to the network resource to be crawled, the method further includes: Determine whether the target data corresponding to the network resource to be crawled has been loaded according to the target object data; If the target data has not been loaded, then the loading time of the target object data is obtained; When the loading time is longer than a preset loading time, the loading of the target data corresponding to the network resource to be crawled is stopped.

3. The telescopic rendering method according to claim 2, wherein: The step of judging whether the target data corresponding to the network resource to be crawled has been loaded according to the target object data comprises: Start the preset timer to start timing; Determine whether the DOM number corresponding to the target object data changes within a preset time range; If yes, reset the preset timer and return to the step of determining whether the DOM number corresponding to the target object data changes within the preset time range; If not, it is determined that the target data corresponding to the to-be-crawled network resource has been loaded.

4. The telescopic rendering method according to claim 1, wherein: The step of receiving target object data corresponding to the network resource to be crawled includes: Receive web crawling tasks; Determine the network resources to be crawled according to the network crawling task; The target object data corresponding to the network resources to be crawled is obtained through a preset browser running in incognito mode.

5. The telescopic rendering method according to claim 4, wherein: The step of receiving a web crawling task includes: Monitor the number of currently rendered pages and resource utilization through the preset monitoring thread; Determine whether the preset conditions are met according to the current number of rendered pages, resource utilization and hardware configuration information; If not satisfied, accept the web crawling task.

6. The telescopic rendering method according to claim 1, wherein: After the step of receiving the target object data corresponding to the network resource to be crawled, the method further includes: Obtain a target service for rendering the target object data in a distributed cluster; Obtaining an operation interface corresponding to the target service; The operation interface is displayed so that the user can input rendering parameters based on the operation interface.

7. The telescopic rendering method according to claim 1, wherein: The step of performing scalable rendering on the target object data according to the rendering parameters comprises: The rendering parameters are input into a preset Lua script, so that the preset Lua script performs scaling rendering on the target object data based on the rendering parameters.

8. A telescopic rendering device, characterized in that: The telescopic rendering device comprises: A receiving module, used to receive target object data corresponding to the network resources to be crawled; An acquisition module, used for acquiring rendering parameters selected by a user based on the target object data; A rendering module is used to perform scaling rendering on the target object data according to the rendering parameters.

9. A scalable rendering device, characterized in that: The device comprises: a memory, a processor, and a scalable rendering program stored in the memory and executable on the processor, wherein the scalable rendering program is configured to implement the steps of the scalable rendering method according to any one of claims 1 to 7.

10. A storage medium, characterized in that: The storage medium stores a scalable rendering program, and when the scalable rendering program is executed by the processor, the steps of the scalable rendering method according to any one of claims 1 to 7 are implemented.

Images