Data acquisition method and device, computer equipment and computer readable storage medium
By analyzing and classifying the data packets returned by the client device on the server device, data collection for all applications is realized, and the problem of large-scale custom logic requirements in the existing technology is solved, and the universality of large-scale data collection is realized. efficiency.
Patent Information
- Application Number
- CN202411008321.6
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-07-25
- Publication Date
- 2024-11-19
AI Technical Summary
The existing APP data acquisition methods require customization of logic for each application, resulting in a large development workload and are not suitable for large-scale data acquisition.
By sending data acquisition requests to client devices on the server device, receiving and analyzing data packets, and performing text classification processing to obtain structured data, it realizes common data acquisition for all applications without customizing the acquisition logic for each application.
It greatly reduces the development workload and is suitable for large-scale data acquisition, achieving general data acquisition capabilities for all applications.
Smart Images

Figure CN118981406A_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the technical field of data collection, and in particular to a data collection method, device, computer equipment and computer-readable storage medium. Background Art
[0002] With the popularity of smart terminals, applications (APPs) have become the main medium for people to obtain information. Therefore, APP data collection has become particularly important. There are two main existing APP data collection methods: (1) cracking the APP through reverse engineering and simulating the APP to send requests to obtain data; (2) analyzing the interface layout of the APP, using automated collection tools to locate interface elements, and obtaining the text content corresponding to the interface elements. However, the above two APP data collection methods require customized data collection logic for each APP, which requires a large development workload and is not suitable for large-scale collection. Summary of the invention
[0003] The embodiments of the present application provide a data collection method, apparatus, computer device, and computer-readable storage medium, which do not require customized collection logic for each application, can significantly reduce development workload, and are suitable for large-scale data collection.
[0004] The technical solution adopted by the present invention to solve the problem is as follows:
[0005] On the one hand, the present application provides a data collection method, which is applied to a server device of a data collection system, comprising:
[0006] Sending a data collection request to a client device, and receiving a data packet of a target application returned by the client device based on the data collection request;
[0007] Parse the data packet to obtain the original data of the target application;
[0008] Perform text classification on the original data to obtain structured data for the target application.
[0009] In some implementation schemes of the present application, text classification processing is performed on the original data to obtain structured data of the target application, including:
[0010] Get the application type of the target application;
[0011] Based on the application type, determine the text classification model corresponding to the target application;
[0012] Use the text classification model to perform text classification processing on multiple block texts in the original data to obtain the text category corresponding to each block text;
[0013] Based on the multiple block texts and the text category corresponding to each block text, structured data of the target application is determined.
[0014] In some implementation schemes of the present application, the data packet is parsed to obtain the original data of the target application, including:
[0015] Get the protocol type of the data packet;
[0016] Parse the data packet according to the protocol type to obtain the original data of the target application.
[0017] In some embodiments of the present application, the data collection system includes multiple client devices, and sending a data collection request to the client device includes:
[0018] Get load information of multiple client devices;
[0019] determining a target client device from a plurality of client devices based on the load information;
[0020] Send a data collection request to the target client device.
[0021] In a second aspect, the present application provides a data collection method, which is applied to a client device of a data collection system, comprising:
[0022] Receive a data collection request sent by a server device; the data collection request includes application information of a target application and maximum level interface information of the target application;
[0023] Traversing the interface of the target application based on the application information and the maximum level interface information, and capturing the data packet of the target application through a data capture program;
[0024] The data packet of the target application is sent to the server device, so that the server device parses and classifies the data packet to obtain the structured data of the target application.
[0025] In some implementation schemes of the present application, the interface of the target application is traversed based on the application information and the maximum level interface information, including:
[0026] Launch the target application based on the application information;
[0027] Use automated operation tools to simulate user operations to traverse each level of the target application interface;
[0028] Determine whether the currently traversed level interface is the maximum level interface of the target application based on the maximum level interface information;
[0029] If the currently traversed hierarchical interface is the maximum hierarchical interface or the currently traversed hierarchical interface has no lower-level interface, return to the upper-level interface of the currently traversed hierarchical interface, and continue to execute the steps of traversing each hierarchical interface of the target application by simulating user operations through the automated operation tool until each hierarchical interface has been traversed or the traversal time of each hierarchical interface reaches the preset time.
[0030] In a third aspect, an embodiment of the present invention further provides a data acquisition device, which is applied to a server device of a data acquisition system, comprising:
[0031] The data collection module is used to send a data collection request to the client device and receive a data packet of the target application returned by the client device based on the data collection request;
[0032] The data parsing module is used to parse the data packet to obtain the original data of the target application;
[0033] The text classification module is used to perform text classification processing on the original data to obtain structured data for the target application.
[0034] In a fourth aspect, an embodiment of the present invention further provides a data acquisition device, which is applied to a client device of a data acquisition system, comprising:
[0035] The request receiving module is used to receive the data collection request sent by the server device; the data collection request includes application information of the target application and the maximum level interface information of the target application;
[0036] An interface traversal module, used to traverse the interface of the target application based on the application information and the maximum level interface information, and to capture the data packet of the target application through a data capture program;
[0037] The data sending module is used to send the data packet of the target application to the server device, so that the server device parses and classifies the data packet to obtain the structured data of the target application.
[0038] In a fifth aspect, the present application further provides a computer device, the computer device comprising:
[0039] one or more processors;
[0040] Memory; and
[0041] One or more applications, wherein the one or more applications are stored in the memory and configured to be executed by the processor to implement the data collection method of any one of the first aspect or the second aspect.
[0042] In a sixth aspect, an embodiment of the present application provides a computer-readable storage medium having a computer program stored thereon, and the computer program is loaded by a processor to execute the steps in the data acquisition method of any one of the first aspect or the second aspect.
[0043] The beneficial effects of the present invention are as follows: a data collection request is sent to a client device through a server device, and a data packet of a target application returned by the client device based on the data collection request is received, the data packet is parsed to obtain the original data of the target application, and the original data is subjected to text classification processing to obtain the structured data of the target application. The data collection method is applicable to all applications, and there is no need to customize the collection logic for each application, which can greatly reduce the development workload and is suitable for large-scale data collection. BRIEF DESCRIPTION OF THE DRAWINGS
[0044] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.
[0045] Figure 1 It is a flow chart of a data acquisition system provided by an embodiment of the present invention;
[0046] Figure 2 It is a flowchart of an embodiment of a data collection method provided by an embodiment of the present invention;
[0047] Figure 3 is a flow chart of another embodiment of the data collection method provided by an embodiment of the present invention;
[0048] Figure 4 is a principle block diagram of an embodiment of a data acquisition device provided by an embodiment of the present invention;
[0049] Figure 5 is a principle block diagram of another embodiment of the data acquisition device provided by an embodiment of the present invention;
[0050] Figure 6 It is a schematic diagram of the structure of a computer device provided in an embodiment of the present invention. DETAILED DESCRIPTION
[0051] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative work are within the scope of protection of this application.
[0052] In the description of this application, the terms "first", "second", "third", etc. are used for descriptive purposes only and should not be understood as indicating or implying relative importance or implicitly indicating the number of the indicated technical features. Thus, features defined as "first", "second", "third", etc. may explicitly or implicitly include one or more of the features. In the description of this application, "plurality" means two or more, unless otherwise clearly and specifically defined.
[0053] In this application, the word "exemplary" is used to mean "used as an example, illustration, or description." Any embodiment described in this application as "exemplary" is not necessarily to be construed as being preferred or advantageous over other embodiments. The following description is given to enable any technician in the field to implement and use the present application. In the following description, details are listed for the purpose of explanation. It should be understood that a person of ordinary skill in the art can recognize that the present application can be implemented without using these specific details. In other instances, well-known structures and processes will not be elaborated in detail to avoid obscuring the description of the present application with unnecessary details. Therefore, the present application is not intended to be limited to the embodiments shown, but is consistent with the widest scope consistent with the principles and features disclosed in the present application.
[0054] It should be noted that since the method of the embodiment of the present application is executed in a computer device, the processing objects of each computer device exist in the form of data or information. For example, time is actually time information. It can be understood that if size, quantity, position, etc. are mentioned in subsequent embodiments, they are all corresponding data for processing by the computer device. The details will not be repeated here.
[0055] The embodiments of the present application provide a data collection method, apparatus, computer device, and computer-readable storage medium, which are described in detail below.
[0056] See also Figure 1 , Figure 1 A schematic diagram of a scenario of a data acquisition system provided in an embodiment of the present application. The data acquisition system may include a server device and a client device.
[0057] In a specific embodiment, the server device is mainly used to send a data collection request to the client device, and receive a data packet of the target application returned by the client device based on the data collection request; parse the data packet to obtain the original data of the target application; perform text classification processing on the original data to obtain structured data of the target application. The data collection method is common to all applications, and there is no need to customize the collection logic for each application, which can greatly reduce the development workload and is suitable for large-scale data collection.
[0058] In the embodiments of the present application, the server device may be an independent server, or a server network or server cluster composed of servers. For example, the computer device described in the embodiments of the present application includes but is not limited to a computer, a network host, a single network server, a plurality of network server sets or a cloud server composed of a plurality of servers. The cloud server is composed of a large number of computers or network servers based on cloud computing.
[0059] It is understandable that the server device used in the embodiments of the present application may be a device that includes both receiving and transmitting hardware, that is, a device that has receiving and transmitting hardware capable of performing two-way communication on a two-way communication link. Such a device may include: a cellular or other communication device having a single-line display or a multi-line display or a cellular or other communication device without a multi-line display. The specific server device may be a desktop terminal or a mobile terminal, and the server device may also be one of a mobile phone, a tablet computer, a laptop computer, etc.
[0060] In a specific embodiment, the client device is mainly used to receive a data collection request sent by a server device; the data collection request includes application information of the target application and maximum-level interface information of the target application; the interface of the target application is traversed based on the application information and the maximum-level interface information, and the data packet of the target application is captured by a data capture program; the data packet of the target application is sent to the server device, so that the server device parses and performs text classification processing on the data packet to obtain structured data of the target application. The data collection method is applicable to all applications, and there is no need to customize the collection logic for each application, which can greatly reduce the development workload and is suitable for large-scale data collection.
[0061] In the embodiment of the present application, the client device is a terminal device for installing the target application. The client device may specifically be a desktop terminal or a mobile terminal. The client device may specifically be a television, a mobile phone, a tablet computer, a laptop computer, etc., which is not limited in this embodiment.
[0062] Those skilled in the art will understand that Figure 1The application environment shown in the figure is only one application scenario of the present application solution and does not constitute a limitation on the application scenario of the present application solution. Other application environments may also include Figure 1 More or less computer equipment as shown in Figure 1 Only one server device and one client device are shown. It can be understood that the data acquisition system can also include one server device and multiple client devices or multiple server devices and multiple client devices, which is not specifically limited here.
[0063] In addition, if Figure 1 As shown, the data collection system may also include a memory for storing data, such as data of the target application, such as raw data, structured data, etc., such as data collection requests, such as application name, maximum level interface information, etc.
[0064] It should be noted that Figure 1 The scenario diagram of the data acquisition system shown is only an example. The data acquisition system and scenario described in the embodiment of the present application are intended to more clearly illustrate the technical solution of the embodiment of the present application, and do not constitute a limitation on the technical solution provided in the embodiment of the present application. Ordinary technicians in this field can know that with the evolution of the data acquisition system and the emergence of new business scenarios, the technical solution provided in the embodiment of the present application is also applicable to similar technical problems.
[0065] First, a data collection method, device, computer equipment and computer-readable storage medium are provided in an embodiment of the present application. The executor of the data collection method is a data collection device, and the data collection device is applied to a server-side device of a data collection system. The data collection method includes: sending a data collection request to a client device, and receiving a data packet of a target application returned by the client device based on the data collection request; parsing the data packet to obtain the original data of the target application; and performing text classification processing on the original data to obtain structured data of the target application.
[0066] like Figure 2 As shown, it is a flow chart of an embodiment of a data collection method in an embodiment of the present application. The data collection method is applied to a server device of a data collection system. The data collection method may include the following steps S201 to S203, which are as follows:
[0067] Step S201: Send a data collection request to a client device, and receive a data packet of a target application returned by the client device based on the data collection request.
[0068] In a specific embodiment, the data collection request is a data collection instruction for a target application sent by a server device to a client device, and the data collection request may carry application information of the target application and maximum level interface information of the target application. The application information is information related to the target application, and the application information may uniquely identify the target application. For example, the application information may be the name information of the target application, or the identification information of the target application, which is not limited in this embodiment.
[0069] Furthermore, after receiving the data collection request, the client device can traverse the interface of the target application based on the application information and maximum level interface information in the data collection request, and capture the data packet of the target application through the data capture program, and then send the data packet of the target application to the server device.
[0070] In a specific embodiment, the data capture program may use an eBPF data capture program. eBPF (Extended Berkeley Packet Filter) is a kernel-based lightweight virtual machine that allows users to run custom programs in the kernel to achieve real-time monitoring and analysis of system events.
[0071] In a specific embodiment, the data acquisition system includes multiple client devices, and the step of sending a data acquisition request to the client device in the above step S201 specifically includes: obtaining load information of the multiple client devices; determining a target client device from the multiple client devices based on the load information; and sending a data acquisition request to the target client device.
[0072] For example, the data collection system includes client device A, client device B, and client device C. Based on the load information, it is determined that the load of client device A is the smallest, and then a data collection request is sent to client device A.
[0073] Step S202: parse the data packet to obtain the original data of the target application.
[0074] In a specific embodiment, the data packet of the target application returned by the client device based on the data collection request may include data packets of various protocols, for example, data packets of various protocols such as http protocol, https protocol, websocket protocol, udp protocol, etc. The steps of parsing the data packet to obtain the original data of the target application specifically include: obtaining the protocol type of the data packet; parsing the data packet according to the protocol type to obtain the original data of the target application.
[0075] Step S203: Perform text classification processing on the original data to obtain structured data of the target application.
[0076] In a specific embodiment, the original data includes multiple text blocks, and the original data is subjected to text classification processing, and each text block can be subjected to text classification to extract structured data from the original data. For example, the data of an e-commerce APP can extract structured data including product title, original price, discount price, merchant name, merchant rating, number of comments, and product specification fields.
[0077] In this embodiment, the server device receives the data packet of the target application returned by the client device based on the data collection request, and parses and performs text classification processing on the data packet to obtain structured data of the target application. The data collection method is applicable to all applications, and there is no need to customize the collection logic for each application. The development workload can be greatly reduced, and it is suitable for large-scale data collection.
[0078] Considering that different categories of APPs have different text categories, for example, the text categories of e-commerce APPs include product title, original price, discount price, merchant name, merchant rating, number of comments, product specifications, etc., and the text categories of information APPs include title, publisher, release date, text, tags, etc. In order to implement text classification processing of raw data corresponding to different categories of APPs, this embodiment pre-sets different text classification models for different categories of APPs. For example, text classification model A is used to perform text classification processing on raw data of e-commerce APPs, and text classification model B is used to perform text classification processing on raw data of information APPs.
[0079] Correspondingly, the above-mentioned step of performing text classification processing on the original data to obtain structured data of the target application specifically includes: obtaining the application type of the target application; based on the application type, determining the text classification model corresponding to the target application; using the text classification model to perform text classification processing on multiple block texts in the original data to obtain the text category corresponding to each block text; based on the multiple block texts and the text category corresponding to each block text, determine the structured data of the target application.
[0080] Optionally, based on the application type, the step of determining the text classification model corresponding to the target application specifically includes: obtaining model association information; the model association information represents the correspondence between the application category and the text classification model; based on the application category and the model association information, determining the text classification model corresponding to the target application. For example, e-commerce apps correspond to text classification model A, and information apps correspond to text classification model B. If the application type of the target application is an e-commerce app, then the text classification model corresponding to the target application is determined to be text classification model A.
[0081] In a specific embodiment, the text classification model corresponding to the target application can be obtained by training a preset network model using a training sample set, the training sample set includes multiple sample texts and the real text category corresponding to each sample text, and the process of training the preset network model using the training sample set is: input multiple sample texts into the preset network model, and output the predicted text category corresponding to each sample text through the preset network model; determine the loss value based on the real text category, the predicted text category and the loss function of the preset network model; if the loss value does not meet the preset condition, train the preset network model based on the preset parameter learning rate, and continue to perform the steps of inputting multiple sample texts into the preset network model, and outputting the predicted text category corresponding to each sample text through the preset network model, until the loss value meets the preset condition. Wherein, the loss value meeting the preset condition can be that the loss value is less than the preset first threshold value or the difference between the loss values obtained twice before and after is less than the preset second threshold value.
[0082] Optionally, the text classification model can adopt a machine learning model or a deep learning model. For example, the text classification model can adopt a machine learning algorithm such as Naive Bayes, KNN, etc., and the text classification model can also adopt a deep learning model such as TextCNN, FastText, TextRNN, etc., which is not limited in this embodiment.
[0083] like Figure 3 As shown, it is a flow chart of another embodiment of the data collection method in the embodiment of the present application. The data collection method is applied to the client device of the data collection system. The data collection method may include the following steps S301 to S303, which are as follows:
[0084] Step S301: receiving a data collection request sent by a server device; the data collection request includes application information of a target application and maximum level interface information of the target application.
[0085] In a specific embodiment, the data collection request is a data collection instruction for a target application sent by a server device to a client device, and the data collection request may carry application information of the target application and maximum level interface information of the target application. The application information is information related to the target application, and the application information may uniquely identify the target application. For example, the application information may be the name information of the target application, or the identification information of the target application, which is not limited in this embodiment.
[0086] Step S302: traverse the interface of the target application based on the application information and the maximum level interface information, and capture the data packet of the target application through a data capture program.
[0087] In a specific embodiment, after receiving the data collection request, the client device can traverse the interface of the target application based on the application information and the maximum level interface information in the data collection request, and capture the data packet of the target application through the data capture program.
[0088] Optionally, the data capture program may adopt an eBPF data capture program. eBPF (Extended Berkeley Packet Filter) is a kernel-based lightweight virtual machine that allows users to run custom programs in the kernel to achieve real-time monitoring and analysis of system events.
[0089] In a specific embodiment, the step of traversing the interface of the target application based on the application information and the maximum-level interface information specifically includes: starting the target application based on the application information; traversing the various level interfaces of the target application by simulating user operations through an automated operation tool; determining whether the currently traversed level interface is the maximum level interface of the target application based on the maximum-level interface information; if the currently traversed level interface is the maximum level interface or the currently traversed level interface has no lower level interface, returning to the previous level interface of the currently traversed level interface, and continuing to execute the step of traversing the various level interfaces of the target application by simulating user operations through an automated operation tool until the traversal of each level interface has been completed or the traversal time of each level interface reaches a preset time.
[0090] Optionally, the automation tool may adopt an existing automation tool, such as UIAutomator / AutoJS, etc. The automation tool may simulate a user operation to randomly click on the interface of the target application, thereby traversing various levels of interfaces of the target application.
[0091] Step S303: Send the data packet of the target application to the server device, so that the server device parses and performs text classification processing on the data packet to obtain structured data of the target application.
[0092] In this embodiment, after the client device captures the data packet of the target application through the data capture program, it sends the data packet of the target application to the server device. After the server device receives the data packet of the target application, it can extract structured data from the data packet of the target application based on the aforementioned steps S202 and S203.
[0093] In this embodiment, the interface of the target application is traversed based on the application information and the maximum level interface information, and the data packet of the target application is captured by the data capture program. The data packet is then sent to the server device for parsing and text classification. The data collection method is common to all applications. There is no need to customize the collection logic for each application, which can greatly reduce the development workload and is suitable for large-scale data collection.
[0094] In order to better implement the data collection method in the embodiment of the present application, based on the data collection method, a data collection device is also provided in the embodiment of the present application. The data collection device is applied to the server device of the data collection system, such as Figure 4 As shown, the data acquisition device 600 includes:
[0095] The data collection module 610 is used to send a data collection request to the client device and receive a data packet of the target application returned by the client device based on the data collection request;
[0096] The data analysis module 620 is used to analyze the data packet to obtain the original data of the target application;
[0097] The text classification module 630 is used to perform text classification processing on the original data to obtain structured data of the target application.
[0098] In an embodiment of the present application, a data collection request is sent to a client device through a server device, and a data packet of a target application returned by the client device based on the data collection request is received, the data packet is parsed to obtain the original data of the target application, and the original data is subjected to text classification processing to obtain the structured data of the target application. The data collection method is common to all applications, and there is no need to customize the collection logic for each application. The development workload can be greatly reduced, and it is suitable for large-scale data collection.
[0099] In some embodiments of the present application, the data collection system includes multiple client devices, and the data collection module 610 sends a data collection request to the client device, including:
[0100] Get load information of multiple client devices;
[0101] determining a target client device from a plurality of client devices based on the load information;
[0102] Send a data collection request to the target client device.
[0103] In some embodiments of the present application, the data parsing module 620 parses the data packet to obtain the original data of the target application, including:
[0104] Get the protocol type of the data packet;
[0105] Parse the data packet according to the protocol type to obtain the original data of the target application.
[0106] In some embodiments of the present application, the text classification module 630 performs text classification processing on the original data to obtain structured data of the target application, including:
[0107] Get the application type of the target application;
[0108] Based on the application type, determine the text classification model corresponding to the target application;
[0109] Use the text classification model to perform text classification processing on multiple block texts in the original data to obtain the text category corresponding to each block text;
[0110] Based on the multiple block texts and the text category corresponding to each block text, structured data of the target application is determined.
[0111] The present application also provides a data acquisition device in an embodiment, which is applied to a server device of a data acquisition system, such as Figure 5 As shown, the data acquisition device 700 includes:
[0112] The request receiving module 710 is used to receive a data collection request sent by a server device; the data collection request includes application information of a target application and maximum level interface information of the target application;
[0113] The interface traversal module 720 is used to traverse the interface of the target application based on the application information and the maximum level interface information, and to capture the data packet of the target application through the data capture program;
[0114] The data sending module 730 is used to send the data packet of the target application to the server device, so that the server device parses and performs text classification processing on the data packet to obtain structured data of the target application.
[0115] In an embodiment of the present application, a client device traverses the interface of a target application based on application information and maximum-level interface information, captures data packets of the target application through a data capture program, and sends the data packets of the target application to a server device. The server device then parses and performs text classification processing on the data packets to obtain structured data of the target application. The data collection method is common to all applications, and there is no need to customize the collection logic for each application. This can significantly reduce development workload and is suitable for large-scale data collection.
[0116] In some embodiments of the present application, the interface traversal module 720 traverses the interface of the target application based on the application information and the maximum level interface information, including:
[0117] Launch the target application based on the application information;
[0118] Use automated operation tools to simulate user operations to traverse each level of the target application interface;
[0119] Determine whether the currently traversed level interface is the maximum level interface of the target application based on the maximum level interface information;
[0120] If the currently traversed hierarchical interface is the maximum hierarchical interface or the currently traversed hierarchical interface has no lower-level interface, return to the upper-level interface of the currently traversed hierarchical interface, and continue to execute the steps of traversing each hierarchical interface of the target application by simulating user operations through the automated operation tool until each hierarchical interface has been traversed or the traversal time of each hierarchical interface reaches the preset time.
[0121] The embodiment of the present application further provides a computer device, which integrates any one of the data acquisition devices provided in the embodiment of the present application, and the computer device includes:
[0122] one or more processors;
[0123] Memory; and
[0124] One or more applications, wherein the one or more applications are stored in the memory and are configured to execute, by the processor, the steps of the data collection method in any of the above data collection method embodiments.
[0125] The present application also provides a computer device that integrates any data acquisition device provided in the present application. Figure 6 As shown, it shows a schematic diagram of the structure of the computer device involved in the embodiment of the present application, specifically:
[0126] The computer device may include one or more processing core processors 801, one or more computer-readable storage media memories 802, a power supply 803, an input unit 804 and other components. Those skilled in the art will appreciate that Figure 6 The computer device structure shown in the figure does not constitute a limitation on the computer device, and may include more or fewer components than shown in the figure, or combine certain components, or arrange the components differently. Among them:
[0127] The processor 801 is the control center of the computer device. It uses various interfaces and lines to connect various parts of the entire computer device. By running or executing software programs and / or modules stored in the memory 802 and calling data stored in the memory 802, it executes various functions of the computer device and processes data, thereby monitoring the computer device as a whole. Optionally, the processor 801 may include one or more processing cores; preferably, the processor 801 may integrate an application processor and a modem processor, wherein the application processor mainly processes the operating system, user interface, and application programs, and the modem processor mainly processes wireless communications. It is understandable that the above-mentioned modem processor may not be integrated into the processor 801.
[0128] The memory 802 can be used to store software programs and modules. The processor 801 executes various functional applications and data processing by running the software programs and modules stored in the memory 802. The memory 802 may mainly include a program storage area and a data storage area, wherein the program storage area may store an operating system, an application required for at least one function (such as a sound playback function, an image playback function, etc.), etc.; the data storage area may store data created according to the use of the computer device, etc. In addition, the memory 802 may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one disk storage device, a flash memory device, or other volatile solid-state storage devices. Accordingly, the memory 802 may also include a memory controller to provide the processor 801 with access to the memory 802.
[0129] The computer device also includes a power supply 803 for supplying power to each component. Preferably, the power supply 803 can be logically connected to the processor 801 through a power management system, so that the power management system can manage charging, discharging, power consumption and other functions. The power supply 803 can also include one or more DC or AC power supplies, recharging systems, power failure detection circuits, power converters or inverters, power status indicators and other arbitrary components.
[0130] The computer device may further include an input unit 804, which may be used to receive input digital or character information and generate keyboard, mouse, joystick, optical or trackball signal input related to user settings and function control.
[0131] Although not shown, the computer device may further include a display unit, etc., which will not be described in detail herein. Specifically in this embodiment, the processor 801 in the computer device will load the executable files corresponding to the processes of one or more application programs into the memory 802 according to the following instructions, and the processor 801 will run the application programs stored in the memory 802, thereby realizing various functions, as follows:
[0132] Sending a data collection request to a client device, and receiving a data packet of a target application returned by the client device based on the data collection request;
[0133] Parse the data packet to obtain the original data of the target application;
[0134] Perform text classification on the original data to obtain structured data for the target application.
[0135] A person of ordinary skill in the art will appreciate that all or part of the steps in the various methods of the above embodiments may be completed by instructions, or by controlling related hardware through instructions. The instructions may be stored in a computer-readable storage medium and loaded and executed by a processor.
[0136] To this end, the embodiment of the present application provides a computer-readable storage medium, which may include: a read-only memory (ROM), a random access memory (RAM), a disk or an optical disk, etc. A computer program is stored thereon, and the computer program is loaded by a processor to execute the steps in any data acquisition method provided in the embodiment of the present application. For example, the computer program loaded by the processor can execute the following steps:
[0137] Sending a data collection request to a client device, and receiving a data packet of a target application returned by the client device based on the data collection request;
[0138] Parse the data packet to obtain the original data of the target application;
[0139] Perform text classification on the original data to obtain structured data for the target application.
[0140] In the above embodiments, the description of each embodiment has its own emphasis. For parts that are not described in detail in a certain embodiment, please refer to the detailed description of other embodiments above, and will not be repeated here.
[0141] In specific implementation, the above units or structures can be implemented as independent entities, or can be arbitrarily combined to be implemented as the same or several entities. The specific implementation of the above units or structures can refer to the previous method embodiments, which will not be repeated here.
[0142] The specific implementation of the above operations can be found in the previous embodiments, which will not be described in detail here.
[0143] The above is a detailed introduction to a data acquisition method, device, computer equipment and computer-readable storage medium provided in the embodiments of the present application. Specific examples are used in this article to illustrate the principles and implementation methods of the present application. The description of the above embodiments is only used to help understand the method of the present application and its core idea; at the same time, for technical personnel in this field, according to the idea of the present application, there will be changes in the specific implementation method and application scope. In summary, the content of this specification should not be understood as a limitation on the present application.
Claims
1. A data collection method, characterized in that: The server equipment used in the data acquisition system includes: Sending a data collection request to a client device, and receiving a data packet of a target application returned by the client device based on the data collection request; Parsing the data packet to obtain original data of the target application; The original data is subjected to text classification processing to obtain structured data of the target application.
2. The method according to claim 1, characterized in that The performing text classification processing on the original data to obtain structured data of the target application includes: Obtaining the application type of the target application; Based on the application type, determining a text classification model corresponding to the target application; Using the text classification model to perform text classification processing on multiple block texts in the original data to obtain a text category corresponding to each block text; Based on the multiple block texts and the text category corresponding to each block text, the structured data of the target application is determined.
3. The method according to claim 1, characterized in that The parsing of the data packet to obtain the original data of the target application includes: Obtaining the protocol type of the data packet; The data packet is parsed according to the protocol type to obtain original data of the target application.
4. The method according to claim 1, characterized in that: The data collection system includes a plurality of client devices, and sending a data collection request to the client device includes: Obtaining load information of a plurality of the client devices; determining a target client device from the plurality of client devices based on the load information; The data collection request is sent to the target client device.
5. A data collection method, characterized in that: Client devices used in data acquisition systems include: Receiving a data collection request sent by a server device; the data collection request includes application information of a target application and maximum level interface information of the target application; Traversing the interface of the target application based on the application information and the maximum level interface information, and capturing data packets of the target application through a data capture program; The data packet of the target application is sent to the server device, so that the server device parses and performs text classification processing on the data packet to obtain structured data of the target application.
6. The method according to claim 5, characterized in that The traversing the interface of the target application based on the application information and the maximum level interface information includes: Starting the target application based on the application information; Simulating user operations through an automated operation tool to traverse each level interface of the target application; Determine whether the currently traversed hierarchical interface is the maximum hierarchical interface of the target application based on the maximum hierarchical interface information; If the hierarchical interface currently traversed is the maximum hierarchical interface or the hierarchical interface currently traversed has no lower-level interface, return to the upper-level interface of the hierarchical interface currently traversed, and continue to execute the step of traversing each hierarchical interface of the target application by simulating user operations through the automated operation tool until each hierarchical interface has been traversed or the traversal time of each hierarchical interface reaches the preset time.
7. A data acquisition device, characterized in that: The server equipment used in the data acquisition system includes: A data collection module, used to send a data collection request to a client device, and receive a data packet of a target application returned by the client device based on the data collection request; A data parsing module, used to parse the data packet to obtain the original data of the target application; The text classification module is used to perform text classification processing on the original data to obtain structured data of the target application.
8. A data acquisition device, characterized in that: Client devices used in data acquisition systems include: A request receiving module, used to receive a data collection request sent by a server device; the data collection request includes application information of a target application and maximum level interface information of the target application; An interface traversal module, used to traverse the interface of the target application based on the application information and the maximum level interface information, and to capture data packets of the target application through a data capture program; The data sending module is used to send the data packet of the target application to the server device, so that the server device parses and performs text classification processing on the data packet to obtain the structured data of the target application.
9. A computer device, characterized in that: The computer device comprises: one or more processors; Memory; and One or more application programs, wherein the one or more application programs are stored in the memory and configured to be executed by the processor to implement the data collection method according to any one of claims 1 to 6.
10. A computer-readable storage medium, characterized in that: A computer program is stored thereon, and the computer program is loaded by a processor to execute the steps in the data collection method according to any one of claims 1 to 6.
