Data processing method and device, equipment and storage medium

Abstract

The invention is applicable to the technical field of data processing, and provides a data processing method, device and equipment and a storage medium, the method is applied to equipment provided with a Gawan system, and the method comprises the following steps: obtaining an original request method for requesting original response data, the original request method comprises a hyper text transfer protocol (http) request method and / or a hyper text transfer security protocol (https) request method, and determining a target request method according to the original request method and a newly added request logic, the newly added request logic comprises at least one of log recording logic, request parameter modification logic, data acquisition execution logic and request data transmission logic; the target request method is adopted to intercept and cover the original request method, the target request method is executed to obtain target response data, the target response data is plaintext data, and the content of the target response data comprises the content of the original response data. The scheme is beneficial to flow collection of wide-gap application.

Description

Technical Field

[0001] The present application relates to the field of data processing technology, and in particular to a data processing method, device, equipment and storage medium. Background Art

[0002] HarmonyOS is a new generation operating system designed to enhance the technical connectivity and user experience between devices. Among them, traffic collection technology is particularly important, especially in scenarios such as security monitoring, performance optimization and data analysis.

[0003] Traditional traffic collection technology generally uses server-side traffic mirroring. This technology usually copies traffic data through network devices (such as routers or switches) and sends it to the traffic analysis system. However, this method cannot directly obtain data on terminal devices installed with the Hongmeng system, which limits the comprehensiveness of the collected information; and due to the encryption characteristics of the HTTPS protocol, the traffic content is encrypted during transmission, and the server-side traffic mirroring technology cannot obtain the plaintext content of the data, and cannot obtain richer contextual data. Summary of the invention

[0004] The embodiments of the present application provide a data processing method, apparatus, device and storage medium, which can solve the technical problem of how to efficiently collect traffic data of Hongmeng applications.

[0005] In a first aspect, an embodiment of the present application provides a data processing method, which is applied to a device installed with a Hongmeng system, including:

[0006] The original request method used to request the original response data is obtained, where the original request method includes a hypertext transfer protocol (http) request method and / or a hypertext transfer security protocol (https) request method.

[0007] The target request method is determined according to the original request method and the newly added request logic, wherein the newly added request logic includes at least one of the following: log recording logic, request parameter modification logic, data collection execution logic, and request data transmission logic.

[0008] The target request method is used to intercept and overwrite the original request method, and the target request method is executed to obtain target response data. The target response data is plain text data, and the content of the target response data includes the content of the original response data.

[0009] In one implementation, determining a target request method according to an original request method and a newly added request logic includes:

[0010] Get the new request method and copy the original request method to the new request method;

[0011] The newly added request logic is set by writing the code corresponding to the newly added request logic in the newly created request method to obtain the target request method. The code corresponding to the newly added request logic includes the newly added request information, and the newly added request information includes the request host name and / or request path.

[0012] In one implementation, the method further includes: backing up the original request method to obtain a backup request method; and when executing the target request method, calling and executing the backup request method to obtain original response data.

[0013] In one implementation, when executing the target request method, calling and executing the backup request method to obtain original response data includes:

[0014] A calling function is set in the newly created request method, wherein the parameters of the calling function include at least one of the following: uniform resource locator information, request method type information, and request header information;

[0015] When executing the calling function, the backup request method is called, and the original response data is obtained by executing the backup request method.

[0016] In one implementation, the method further includes:

[0017] Process the target response data using data events and / or end events to obtain target response information, the target response information including text information and / or image information, the data event is used to receive and splice the target response data, and the end event is used to parse the target response data;

[0018] According to the type of target response information, text information and / or image information are transmitted and stored respectively.

[0019] In one implementation, text information and / or image information are transmitted and stored respectively according to the type of target response information, including:

[0020] When the target response information is text information, the text information is transmitted using a distributed streaming media platform and / or a server log service, and the text information is stored using a structured data storage method and / or a semi-structured data storage method;

[0021] When the target response information is image information, the image information is transmitted using a web socket protocol and / or a preset interface, and an unstructured data storage method is used to store the image information.

[0022] In one implementation, executing a target request method to obtain target response data includes:

[0023] Setting sampling logic in the target request method, the sampling logic including at least one of the following information: sampling rate, sampling frequency, request type, and sampling interface;

[0024] According to the sampling logic, the target request method is executed to obtain the target response data.

[0025] In a second aspect, an embodiment of the present application provides a data processing device, which has the function of implementing the method in the first aspect or any possible implementation thereof. Specifically, the device includes a unit for implementing the method in the first aspect or any possible implementation thereof.

[0026] In one embodiment, the device is applied to the Hongmeng system, including:

[0027] The acquisition unit is used to acquire the original request method used to request the original response data, where the original request method includes an http request method and / or an https request method.

[0028] The processing unit is used to determine the target request method according to the original request method and the newly added request logic, wherein the newly added request logic includes at least one of the following: log recording logic, request parameter modification logic, data collection execution logic, and request data transmission logic.

[0029] The processing unit is also used to intercept and overwrite the original request method using the target request method, execute the target request method to obtain target response data, the target response data is plain text data, and the content of the target response data includes the content of the original response data.

[0030] In a third aspect, an embodiment of the present application provides a computer device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, the computer device implements a method according to any one of the implementation modes of the first aspect.

[0031] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium, which stores a computer program. When the computer program is executed by a computer device, the computer device implements the method of any one of the implementation methods of the first aspect above.

[0032] In a fifth aspect, an embodiment of the present application provides a computer program product. When the computer program product is run on a computer device, the computer device executes a method of any one of the implementation modes of the first aspect above.

[0033] Compared with the prior art, the embodiments of the present application have the following beneficial effects: by obtaining the original request method, it is ensured that all requests issued can be tracked; the target request method is determined based on the original request method obtained and the newly added request logic (such as logging, modifying request parameters, executing data collection, transmitting request data, etc.); through the newly added request logic, the system can flexibly add functions without changing the original request, which enables traffic collection to be customized according to different needs, ensuring that the collected request information is more comprehensive. By intercepting the original request and response, the system can obtain the target response data in plain text without relying on the complexity of encrypted transmission. BRIEF DESCRIPTION OF THE DRAWINGS

[0034] Figure 1 It is a flowchart of a data processing method provided in an embodiment of the present application.

[0035] Figure 2 It is a flowchart of a data processing method provided in an embodiment of the present application.

[0036] Figure 3 It is a system architecture diagram provided in an embodiment of the present application.

[0037] Figure 4 It is a structural diagram of a data processing device provided in an embodiment of the present application.

[0038] Figure 5 It is a schematic diagram of the structure of a computer device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0039] In Hongmeng system, many applications use HTTPS protocol for secure communication, and all data is encrypted during transmission. Collecting such encrypted data requires decryption, which makes traffic analysis complicated.

[0040] Traditional traffic collection technology generally uses server-side traffic mirroring. However, this method cannot directly obtain data on terminal devices installed with the Hongmeng system, nor can it obtain the plaintext content of the data, and it cannot obtain richer contextual data.

[0041] In response to the above problems, this application proposes a data processing method that can efficiently and comprehensively collect traffic for Hongmeng applications.

[0042] In order to further illustrate the technical solution of the present application, specific embodiments are provided below.

[0043] Figure 1 It is a flowchart of a data processing method provided in an embodiment of the present application.

[0044] like Figure 1As shown, the above method is applied to a device installed with the Hongmeng system, including the following steps S101 to S103.

[0045] S101. Obtain an original request method for requesting original response data.

[0046] The original request method includes a hypertext transfer protocol (http) request method and / or a hypertext transfer security protocol (https) request method.

[0047] It can be understood that the original request method refers to the standard method for initiating HTTP or HTTPS requests, such as GET, POST, PUT, DELETE, etc. These methods define the intent and operation type of the request. HTTP is a protocol for transmitting data on the Web, while HTTPS is a secure version of HTTP that uses SSL / TLS encryption to ensure data security.

[0048] The original request method is used to request original response data, and the original data includes at least one of the following: response status code data, response header data, response body data, context data, and network metadata.

[0049] The response status code data is used to indicate the result of the request, such as success, failure, or redirection, etc. For example, common status codes include: 200 OK, indicating a successful request; 404 Not Found, indicating that the requested resource was not found; 500 Internal Server Error, indicating that a server error occurred.

[0050] The response header data contains metadata information about the server, the response content, and the request. For example, the response header data may include: Content-Type, indicating the type of the response content, such as application / json, text / html, application / xml, etc.; Content-Length, indicating the byte size of the response body; Set-Cookie, indicating the cookie information set by the server; Cache-Control, indicating the instructions of the cache mechanism.

[0051] The response body data is the actual data content returned by the server, and the specific content depends on the requested resource. For example, for an API request, the response body may contain data in JSON format, such as user information, product list, etc.; when requesting a web page, the response body may be HTML document code; when requesting a file resource, the response body may be a binary file (such as an image, PDF file, etc.).

[0052] Context data refers to additional context information that can be obtained as needed when using the original request method. For example, context information such as user ID, request source dimension, etc. can be passed by setting a custom request header in the request options, which may be returned during response processing.

[0053] Network metadata refers to the use of some network monitoring tools to obtain metadata about requests, such as: the start time, end time and duration of the request; data transmission rate, request size and response size, etc.

[0054] It can be understood that by calling the original request method, you can obtain the response status code, response header, response body content, error information, and related context and network data. This information can be used in scenarios such as debugging, error handling, and data analysis. The specific original request method varies depending on the actual application scenario and is not limited here.

[0055] S102: Determine the target request method according to the original request method and the newly added request logic.

[0056] The newly added request logic includes at least one of the following: logging logic, request parameter modification logic, data collection execution logic, and request data transmission logic.

[0057] It can be understood that the target request method is to add additional logic functions, such as logging, modifying request parameters, etc., so as to ensure the flexibility and controllability of the request. The following is an exemplary introduction to the aforementioned several newly added request logics.

[0058] Logging logic: Each time a request is initiated, the request time, user ID, and request parameters are recorded for subsequent analysis and review.

[0059] Modify request parameter logic: Before sending a request, dynamically adjust request parameters according to business needs, such as adding or modifying a custom HTTP header in the request.

[0060] Execute data collection logic: collect some data such as user behavior and system status during the request process to support data analysis or performance monitoring.

[0061] Transmit request data logic: Transmit request data to other services through the network for real-time monitoring or analysis.

[0062] It can be understood that through the newly added request logic introduced above, combined with the original request method, the target request method can be determined and generated, that is, the actual data collection requirements can be determined.

[0063] S103: Adopt a target request method to intercept and overwrite the original request method, and execute the target request method to obtain target response data.

[0064] The target response data is plain text data, and the content of the target response data includes the content of the original response data.

[0065] Interception refers to the control and processing of requests and responses through middleware or proxy technology during the process of request initiation or response reception. Through interception, data can be reviewed and modified without affecting the original request process.

[0066] Overriding means replacing the original request method with the target request method so that subsequent network interactions are based on the new request logic. This ensures that the new logic is applied and all subsequent requests automatically follow the updated policy.

[0067] Exemplarily, assuming that the original request method is a GET request and the original response data is the basic information of the user, through step S102, the logic of logging and data collection is added, and then the original GET request is overwritten, and finally the new request is executed, so as to obtain the target response data including the user authority information.

[0068] By executing the target request method, the target response data is collected, and finally rich plaintext target response data is obtained for subsequent use.

[0069] By obtaining the original request method, it is ensured that all requests issued can be tracked; the target request method is determined based on the original request method obtained and the newly added request logic (such as logging, modifying request parameters, performing data collection, transmitting request data, etc.); by adding request logic, the system can flexibly add functions without changing the original request, which enables traffic collection to be customized according to different needs and ensures that the collected request information is more comprehensive. By intercepting the original request and response, the system can obtain the target response data in plain text without relying on the complexity of encrypted transmission.

[0070] In one implementation, a target request method is determined based on an original request method and a newly added request logic, including: obtaining a newly created request method, and copying the original request method to the newly created request method; setting the newly added request logic by writing a code corresponding to the newly added request logic in the newly created request method to obtain the target request method, wherein the code corresponding to the newly added request logic includes newly added request information, and the newly added request information includes a request host name and / or a request path.

[0071] The new request method can be understood as a newly created request object, and then the original request method is copied to the new request method to carry the new request logic to be applied. This can preserve the structure and information of the original request method and ensure that key request parameters and settings are not lost.

[0072] New request logic refers to the additional logic code to be executed during the request process, such as the request parameter modification logic, log recording logic, and data collection logic mentioned above.

[0073] The new request information is part of the new request logic, specifically pointing to the new data that needs to be added to the request, such as the requested host name, path, header information, etc.

[0074] For example, a new https.request method may be created to intercept the original https.request method, and the corresponding request content may be added to the new https.request method, which may be represented by the code: log('HTTP Request URL:', options.hostname+options.path).

[0075] By combining the original request method with the newly added request logic, a new request object with rich functions, namely the target request object, is generated, which adds additional management, monitoring and data collection capabilities to the original request method, which is conducive to the realization of traffic data collection of Hongmeng system applications.

[0076] In one implementation, the original request method is backed up to obtain a backup request method; when executing the target request method, the backup request method is called and executed to obtain original response data.

[0077] It can be understood that backing up the original request method means copying the state and configuration of the original request method to protect the original request method so that it can be restored or called again when needed. The request object generated after backing up the original request method will retain the same properties and settings as the original request. While executing the target request method, call the backup request method independently, send the original request method, and obtain the original response data.

[0078] As an example and not a limitation, you can create a function or method that is responsible for receiving the original request object and returning a backup request object. Use a deep copy (for example, using JSON serialization and deserialization) to copy the contents of the original request object into the new object. Make sure to include all key information such as request headers, parameters, request body, etc. The code that may be involved is as follows:

[0079]

[0080]

[0081] It can be understood that while the target request method executes the newly added request logic, it continues to obtain the original response data corresponding to the original request method, which can be used for subsequent comparison, rollback or verification. In other words, the main purpose of this step is to maintain the integrity of the original request. Once the target request method has a problem or needs to be compared, the original request method can be quickly restored without affecting subsequent target requests.

[0082] In one example, the backup original request method may include the URL of the API, such as api.openweathermap.org, the request method such as Get, a request header (such as an API key), and other required parameters.

[0083] In other words, backing up the original request method can ensure that subsequent logic modifications will not affect the normal application logic, and the acquisition process of the original request method can still be executed normally after injecting the data collection logic.

[0084] In one implementation, when executing the target request method, calling and executing the backup request method to obtain the original response data also includes: setting a calling function in the newly created request method, the parameters of the calling function include at least one of the following: uniform resource locator information, request method type information, request header information; when executing the calling function, calling the backup request method, and obtaining the original response data by executing the backup request method.

[0085] Uniform Resource Locator (URL) is used to specify the target address of a network request.

[0086] The request method type information is used to indicate the type of HTTP request, such as GET, POST, etc.

[0087] The request header information is used to contain metadata about the request, such as content type, authentication information, etc.

[0088] For example, in the new request method, a call function is defined, which accepts the URL, request method and request header as parameters. With these parameters, the request can be flexibly constructed and sent. A possible code representation is as follows:

[0089]

[0090]

[0091] It can be understood that in different application scenarios, the code corresponding to the calling function is also different. The specific code corresponding to the calling function can be set according to the actual situation and is not limited here.

[0092] It can be understood that while executing the target request method, the program will call the backup request method in parallel, that is, it can obtain the original response data corresponding to the original request method. In other words, in the method of executing the target request method, by calling the backup request method to extract the original response data, it is ensured that no matter what the result of the target request method is, the response data of the original request method can be accessed and used.

[0093] When the target request is ready, the function is called to get the URL, method type, and request header information of the original request, and then the backup request method is called to resend the original request. By calling this function during execution, the system can resend the backup request while the target request is executing, ensuring that the original response data is obtained. This effectively ensures that the response to the original request can be retained regardless of whether the result of the target request meets expectations.

[0094] In one implementation, target response data is processed using data events and / or end events to obtain target response information, which includes text information and / or image information. Data events are used to receive and splice target response data, and end events are used to parse target response data. According to the type of target response information, text information and / or image information are transmitted and stored respectively.

[0095] Exemplarily, by utilizing the data event and the end event of the response object, the target response data can be obtained and parsed into the JSON format.

[0096] The response object is the response received after initiating an HTTP request. This is an object generated when a network request is made through the HTTP or HTTPS module of Node.js. It contains various information and data returned by the server, usually including the following parts:

[0097] Status code (for example: 200, 404, etc.): indicates the processing result of the request.

[0098] Response header: Contains meta information about the response, such as content type, date, server information, etc.

[0099] Response body: The actual data content, usually in HTML, JSON or other formats.

[0100] In Node.js, the response object is a stream object that allows the response data to be read step by step in a stream manner.

[0101] The data event is one of the events emitted by the response object, which is triggered when the response data is available. Whenever a chunk of data is received, Node.js will trigger the data event. At this point, you can process this chunk of data through a callback function.

[0102] The data event may be triggered multiple times because the response data may be large and is sent in multiple chunks.

[0103] Each time the data event is triggered, the callback function receives the data block as a parameter, and the response data can be gradually accumulated and processed in this way.

[0104] The end event is triggered when the response is completely received. It indicates that there is no more data to read and all response data blocks have been sent. When the end event is triggered, it means that all accumulated data can be safely processed (e.g. parsed, stored).

[0105] When processing HTTP responses, the final data parsing or logic processing is usually performed in the callback function of the end event, such as parsing the response body into a JSON object.

[0106] The end event is used to avoid processing while data is still being received, thus ensuring integrity.

[0107] In other words, the response object is used to represent the HTTP response received from the server, including the status, headers, and data. The data event is used to receive parts of the response data step by step, and is triggered when a new data block arrives. The end event is triggered after all data has been completely received, which is suitable for final data processing.

[0108] The following is an introduction with specific examples.

[0109] When the HTTP request returns successfully, the response object triggers the data event. This event will be called multiple times when the response data arrives, and each time it is called, a part of the data chunk will be passed. This method is suitable for processing large data responses, because the data may not be returned in full at one time, but will be transmitted multiple times. In the callback function of the data event, the received data chunks are spliced ​​together. Since each triggered data event only returns a part of the data, a variable is needed to accumulate all the data chunks from the data event.

[0110] When all data blocks have been transferred, the response object triggers the end event. At this point, it means that the entire response is complete and you can start processing the accumulated data. In the callback function of the end event, parse the accumulated data into JSON format (if the data is in JSON format). Use the JSON.parse() method to parse the complete response data string into a JavaScript object for subsequent data manipulation and use.

[0111] This can be expressed in code as follows:

[0112] Here is a code example based on the above explanation:

[0113]

[0114] By using the data event and end event of the response object, you can gradually receive all the data from the HTTP response. When the data event is triggered, you can receive and splice the response data block; when the end event is triggered, it means that all data has been received. At this time, you can parse the accumulated response data into JSON format for subsequent processing and use. This method can effectively handle large-scale data responses and ensure that no part is missed.

[0115] In one implementation, text information and / or image information are transmitted and stored separately according to the type of target response information, including: when the target response information is text information, the text information is transmitted using a distributed streaming media platform and / or a server log service, and the text information is stored using a structured data storage method and / or a semi-structured data storage method; when the target response information is image information, the image information is transmitted using a web socket protocol and / or a preset interface, and the image information is stored using an unstructured data storage method.

[0116] Text information includes readable text data, such as JSON, XML, HTML, plain text, log information, etc.

[0117] Image information includes files containing image data, such as pictures in JPEG, PNG, GIF and other formats.

[0118] It can be understood that after obtaining the target response information, the Content-Type header of the response can be checked to determine the type of the response information. According to the corresponding type classification, the text information and the image information are processed separately. By classifying the response information, the system can adopt appropriate transmission and storage methods for different types of data to ensure the efficiency and accuracy of data processing.

[0119] It can be expressed in code as follows:

[0120]

[0121] Distributed Streaming Platform (Kafka) is a distributed streaming platform developed and open sourced by LinkedIn. It is mainly used to build real-time data pipelines and stream processing applications. Kafka has the characteristics of high throughput, high scalability and persistent storage, and can handle large amounts of real-time data streams. It supports the publish / subscribe messaging model and is very suitable for real-time transmission and processing of data.

[0122] Server Log Service (SLS) is a service used to record and store server operation status and request information.

[0123] Structured data storage refers to storing data in a database with a fixed format, such as a relational database (for example, MySQL, PostgreSQL).

[0124] Semi-structured data storage refers to storing data in a database with a flexible format, such as a NoSQL database (e.g. MongoDB, Cassandra).

[0125] Exemplarily, when transmitting text information, a distributed streaming platform (such as Apache Kafka) is used to stream text data to a target location, or a server log service can be used to record the text information.

[0126] When storing text information, you can use structured data storage to store text information in a relational database for easy subsequent query and analysis. You can also use semi-structured data storage to store data in a NoSQL database to support flexible query requirements.

[0127] A possible code could be represented as follows:

[0128] / / Transmit text information

[0129] sendToStreamPlatform(textData);

[0130] / / Store text information

[0131] storeInDatabase(textData,'structured'); / / or'semi-structured'.

[0132] This approach ensures that text information can be transmitted quickly and reliably and stored in a suitable format to facilitate subsequent data processing, query and analysis.

[0133] WebSocket is a network protocol that provides full-duplex, real-time communication between clients and servers. Unlike the traditional HTTP request / response model, WebSocket allows for two-way data stream interaction over a persistent connection. This is very suitable for application scenarios that require real-time data updates, such as online chat, real-time gaming, real-time data monitoring, etc. WebSocket is widely used in Web applications to achieve more efficient real-time communication.

[0134] A preset interface refers to an API interface designed for specific data transmission, usually used to upload or download files.

[0135] Unstructured data storage refers to storing data in its original format without any fixed format constraints, usually using a file system or object storage (such as AWS S2).

[0136] Exemplarily, when transmitting image information, the WebSocket protocol is used to transmit image data in real time between the client and the server, or the image data is uploaded to the server through a preset interface (such as a REST API).

[0137] When storing image information, image data is stored in an unstructured manner in object storage, allowing flexible access and management.

[0138] A possible code could be represented as follows:

[0139] / / Transmit image information

[0140] sendImageOverWebSocket(imageData); / / or

[0141] uploadImageToAPI(imageData);

[0142] / / Store image information

[0143] storeInObjectStorage(imageData).

[0144] This approach ensures that image information can be transmitted efficiently and stored in a flexible way for subsequent access and processing. At the same time, using WebSocket can achieve real-time transmission, which is suitable for application scenarios that require fast response.

[0145] This implementation ensures the efficiency and flexibility of data processing by adopting different transmission and storage strategies according to the type of response information (text or image). Text information uses a distributed streaming platform and structured / semi-structured storage methods, which is suitable for fast query and analysis; while image information is processed through WebSocket and unstructured storage methods, which adapts to the characteristics of image data. This classification processing method can improve the overall performance and scalability of the system.

[0146] In one implementation, executing a target request method to obtain target response data includes: setting a sampling logic in the target request method, the sampling logic including at least one of the following information: a sampling rate, a sampling frequency, a request type, and a sampling interface; and executing the target request method according to the sampling logic to obtain the target response data.

[0147] Sampling logic refers to selecting part of the requested sample according to specific rules during the request process to reduce the amount of response data to be processed. Sampling logic can include the following information:

[0148] The sampling rate indicates the ratio of samples selected from the data. For example, if the sampling rate is 0.1, it means that one sample is selected from every 10 requests.

[0149] The sampling frequency indicates the time interval of the sampling operation, which can be every second, every minute, etc. For example, data is sampled every 5 minutes.

[0150] The request type specifies the type of request for sampling (such as GET, POST, etc.) to correspond to different data sources.

[0151] The sampling interface specifies the data interface path or method used for sampling, such as a specific API endpoint.

[0152] For example, a sampling logic object is set in the target request method, including various sampling parameters. It can be expressed in code as follows:

[0153]

[0154] As an example and not a limitation, before executing the target request, a decision is made based on the sampling logic whether to execute the request. This process can be set to perform sampling at each request or at a fixed time interval, and then call the target request method.

[0155] By executing requests and controlling them according to sampling logic, it is possible to efficiently collect the required data samples to generate meaningful statistics and analysis results. This approach helps understand the behavior of the system without adding excessive burden, while allowing better management of resources under high load conditions.

[0156] Combine the following Figure 2 To comprehensively introduce the data processing methods mentioned above.

[0157] Figure 2 It is a flowchart of a data processing method provided in an embodiment of the present application.

[0158] S201. Create a new monitor.ets file in the src / main / ets directory.

[0159] It can be understood that Hongmeng applications are written in ArkTS language. The main source code is generally contained in this directory, and the collection logic is implemented in the monitor.ets file.

[0160] S202. Import http and https libraries.

[0161] S203: Back up the original request method.

[0162] S204, intercept http and https request methods, and obtain relevant request content.

[0163] S205: Obtain specific response information from the request data packet and the response data packet.

[0164] S206. Configure relevant transport layer components according to data characteristics, and select log, file or image format for data transmission.

[0165] If the amount of data is large or there is a business need, you can also add sampling logic here.

[0166] S207. Execute the original business logic to ensure that the original business process is not affected.

[0167] S208. After the Monitor.ets file is inserted and the relevant logic is implemented, the entire application is repackaged, released, and deployed.

[0168] After deployment, the application has the ability to collect traffic and implement in-depth data exploration capabilities.

[0169] The method mentioned above is introduced below with reference to specific examples.

[0170] For example, for a Hongmeng application that obtains weather data through the https weather API, the overall collection method is as follows:

[0171] 1) Import the https library via import https from 'https';.

[0172] 2) Back up the original request method, which may include the API URL, such as api.openweathermap.org, the request method such as Get, the request header (such as the API key), and other required parameters. Backing up the original request method can ensure that subsequent logic modifications will not affect the normal application logic, and the weather data acquisition process can still be performed normally after injecting the data collection logic.

[0173] 3) Intercept the https request method and obtain the relevant request content. Create a new https.request method to intercept the original https.request method, and add the corresponding request content to the new https.request method, such as log('HTTP Request URL:', options.hostname+options.path).

[0174] 4) Get specific weather information from the response package, such as response.statusCode, JSON (response.text).temperature.

[0175] 5) Text data can be transmitted to the subsequent processing platform through Kafka, SLS, etc. Image data can be transmitted to the storage system through WebSocket or through a specific https interface after compression and format conversion.

[0176] The sampling logic can be formulated according to the needs, including the sampling rate (for example, 1:100), the sampling frequency (once per minute), the request type for sampling, such as sampling for GET requests, the sampling interface, such as sampling 10 items per interface, etc. Since the amount of weather data is not large, it can be set to be sampled once per minute.

[0177] 6) Use the backed-up original request method originalHttpsRequest.request to execute the original weather API acquisition logic to ensure that the temperature, rainfall and other information can be displayed normally on the APP page.

[0178] Combine the following Figure 3 To comprehensively introduce the data processing methods mentioned above.

[0179] Figure 3 It is a system architecture diagram provided in an embodiment of the present application.

[0180] like Figure 3 As shown, the entire system is divided into three parts: client, transport layer and storage.

[0181] The client is a mobile phone, tablet or other device equipped with Hongmeng system and related applications. The transport layer is mainly a series of devices that open up the data link between the client and the storage. The storage is mainly responsible for storing the collected data.

[0182] In the client's ArkTS application, execute steps S201 to S208 mentioned above, and then compile and package. The transport layer mainly performs corresponding data link adaptation according to different network environments and application scenarios, including the selection of transport protocols (kafka, http, websocket, etc.), the adaptation of network environments (intranet, Internet, dedicated lines, etc.), and the adaptation of servers (specific IP, domain name, cloud environment, etc.). The configuration of these transport links can be sent to the collection engine in the form of configuration files.

[0183] The storage layer sets different storage units for storage according to the different structural characteristics of the collected data, including structured database storage, semi-structured log storage, and unstructured image storage. The storage time can be set to different durations such as 3 days, one week, one month, and 3 months according to business needs, which is convenient for subsequent data analysis.

[0184] The above mainly introduces the method of the embodiment of the present application in conjunction with the accompanying drawings. It should be understood that although the various steps in the flowcharts involved in the embodiments described above are shown in sequence, these steps are not necessarily performed in sequence in the order shown in the figure. Unless there is a clear description in this article, the execution of these steps is not strictly limited in order, and these steps can be performed in other orders. Moreover, at least a part of the steps in the flowcharts involved in the embodiments described above may include multiple steps or multiple stages, and these steps or stages are not necessarily performed at the same time, but can be performed at different times, and the execution order of these steps or stages is not necessarily performed in sequence, but can be performed in turn or alternately with other steps or at least a part of the steps or stages in other steps. A device of an embodiment of the present application is introduced below in conjunction with the accompanying drawings. For simplicity, when introducing the device below, it will be appropriately omitted, and the relevant content can refer to the relevant description in the method above, and will not be repeated.

[0185] Figure 4 It is a structural diagram of a data processing device provided in an embodiment of the present application.

[0186] like Figure 4 As shown, the device 1000 includes the following units.

[0187] The acquisition unit 1001 is used to acquire an original request method for requesting original response data, where the original request method includes an http request method and / or an https request method.

[0188] The processing unit 1002 is used to determine the target request method according to the original request method and the newly added request logic, wherein the newly added request logic includes at least one of the following: logging logic, request parameter modification logic, data collection execution logic, and request data transmission logic.

[0189] The processing unit 1002 is also used to intercept and overwrite the original request method with a target request method, execute the target request method to obtain target response data, the target response data is plain text data, and the content of the target response data includes the content of the original response data.

[0190] In one implementation, the device 1000 further includes a storage unit 1003, which can be used to store instructions and / or data, so as to implement the method in the above embodiment.

[0191] In one implementation, the processing unit 1002 is also used to obtain the target request method using the original request method and the newly added request logic, including: obtaining the newly created request method, copying the original request method to the newly created request method; setting the newly added request logic by writing the code corresponding to the newly added request logic in the newly created request method to obtain the target request method, the code corresponding to the newly added request logic includes the newly added request information, and the newly added request information includes the request host name and / or the request path.

[0192] By combining the original request method with the newly added request logic, a new request object with rich functions, namely the target request object, is generated, which adds additional management, monitoring and data collection capabilities to the original request method, which is conducive to the realization of traffic data collection of Hongmeng system applications.

[0193] In one implementation, the processing unit 1002 is further configured to back up the original request method to obtain a backup request method; and when executing the target request method, call and execute the backup request method to obtain original response data.

[0194] The main purpose of this step is to maintain the integrity of the original request. Once a problem occurs in the target request method or a comparison is required, the original request method can be quickly restored without affecting subsequent target requests.

[0195] In one implementation, the processing unit 1002 is also used to, when executing the target request method, call and execute the backup request method to obtain the original response data, including: setting a calling function in a newly created request method, wherein the parameters of the calling function include at least one of the following: uniform resource locator information, request method type information, and request header information; when executing the calling function, calling the backup request method, and obtaining the original response data by executing the backup request method.

[0196] In the method that executes the target request method, the original response data is extracted by calling the backup request method, ensuring that the response data of the original request method can be accessed and used regardless of the result of the target request method.

[0197] In one implementation, the processing unit 1002 is also used to process the target response data using data events and / or end events to obtain target response information, where the target response information includes text information and / or image information. The data event is used to receive and splice the target response data, and the end event is used to parse the target response data. According to the type of the target response information, the text information and / or image information are transmitted and stored respectively.

[0198] By using the data event and end event of the response object, you can gradually receive all the data from the HTTP response. When the data event is triggered, you can receive and splice the response data block; when the end event is triggered, it means that all data has been received. At this time, you can parse the accumulated response data into JSON format for subsequent processing and use. This method can effectively handle large-scale data responses and ensure that no part is missed.

[0199] In one implementation, the processing unit 1002 is also used to transmit and store text information and / or image information respectively according to the type of target response information, including: when the target response information is text information, the text information is transmitted using a distributed streaming media platform and / or a server log service, and the text information is stored using a structured data storage method and / or a semi-structured data storage method; when the target response information is image information, the image information is transmitted using a web socket protocol and / or a preset interface, and the image information is stored using an unstructured data storage method.

[0200] According to the corresponding type classification, text information and image information are processed separately. By classifying the response information, the system can adopt appropriate transmission and storage methods for different types of data to ensure the efficiency and accuracy of data processing.

[0201] In one implementation, the processing unit 1002 is also used to execute the target request method to obtain target response data, including: setting sampling logic in the target request method, the sampling logic including at least one of the following information: sampling rate, sampling frequency, request type, sampling interface; according to the sampling logic, executing the target request method to obtain target response data.

[0202] By executing requests and controlling them according to sampling logic, it is possible to efficiently collect the required data samples to generate meaningful statistics and analysis results. This approach helps understand the behavior of the system without adding excessive burden, while allowing better management of resources under high load conditions.

[0203] It should be noted that the information interaction, execution process and other contents between the above-mentioned units are based on the same concept as the method embodiment of the present application. Their specific functions and technical effects can be found in the method embodiment part and will not be repeated here.

[0204] Figure 5 Schematic diagram of the structure of the computer device provided in the embodiment of the present application. Figure 5 As shown, the computer device 3000 of this embodiment includes: at least one processor 3100 ( Figure 5 Only one) processor, memory 3200, and a computer program 3210 stored in the memory 3200 and executable on at least one processor 3100 are shown. When the processor 3100 executes the computer program 3210, the computer device implements the steps in the above embodiments.

[0205] The processor 3100 may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or the processor may be any conventional processor, etc.

[0206] The memory 3200 may be an internal storage unit of the computer device 3000 in some embodiments, such as a hard disk or memory of the computer device 3000. The memory 3200 may also be an external storage device of the computer device 3000 in other embodiments, such as a plug-in hard disk, a smart memory card (Smart Media Card, SMC), a secure digital (Secure Digital, SD) card, a flash card (Flash Card), etc. equipped on the computer device 3000. Further, the memory 3200 may also include both an internal storage unit of the computer device 3000 and an external storage device. The memory 3200 is used to store an operating system, an application program, a boot loader (Boot Loader) data, and other programs, such as a program code of a computer program, etc. The memory 3200 may also be used to temporarily store data that has been output or is to be output.

[0207] Those skilled in the art can clearly understand that, for the convenience and simplicity of description, only the division of the above-mentioned functional units is used as an example for illustration. In practical applications, the above-mentioned function allocation can be completed by different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above. The functional units in the embodiment can be integrated into a processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The above-mentioned integrated unit can be implemented in the form of hardware or in the form of software functional units. In addition, the specific names of the functional units are only for the convenience of distinguishing each other, and are not used to limit the scope of protection of this application. The specific working process of the units in the above-mentioned system can refer to the corresponding process in the aforementioned method embodiment, which will not be repeated here.

[0208] The embodiment of the present application further provides a computer-readable storage medium, which stores a computer program. When the computer program is executed by a computer device, the computer device can implement the steps in the above-mentioned method embodiments.

[0209] The embodiment of the present application provides a computer program product. When the computer program product is executed on a computer device, the computer device can implement the above-mentioned methods.

[0210] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the present application implements all or part of the processes in the above-mentioned embodiment method, which can be completed by instructing the relevant hardware through a computer program. The computer program can be stored in a computer-readable storage medium. When the computer program is executed by the processor, the computer device can implement the steps of the above-mentioned various method embodiments. Among them, the computer program includes computer program code, and the computer program code can be in source code form, object code form, executable file or some intermediate form. The computer-readable medium may at least include: any entity or device that can carry the computer program code to the camera / terminal device, a recording medium, a computer memory, a read-only memory (ROM), a random access memory (RAM), an electric carrier signal, a telecommunication signal, and a software distribution medium. For example, a USB flash drive, a mobile hard disk, a magnetic disk or an optical disk. In some jurisdictions, according to legislation and patent practice, computer-readable media cannot be electric carrier signals and telecommunication signals.

[0211] It should be understood that the size of the sequence number of each step in the above-described embodiment does not mean the order of execution, and the order of execution of each process should be determined by its function and inherent logic, and should not constitute any limitation to the implementation process of the embodiment of the present application. In the description, for the purpose of explanation rather than for limitation, specific details such as specific system structure, technology and the like are proposed, so as to thoroughly understand the embodiment of the present application. However, it should be clear to those skilled in the art that the present application can also be implemented in other embodiments without these specific details. In other cases, the detailed description of well-known systems, devices, circuits and methods is omitted to prevent unnecessary details from hindering the description of the present application.

[0212] It should be understood that when used in the present specification and the appended claims, the term "comprising" indicates the presence of described features, wholes, steps, operations, elements and / or components, but does not exclude the presence or addition of one or more other features, wholes, steps, operations, elements, components and / or combinations thereof.

[0213] It should also be understood that the term “and / or” used in the specification and appended claims refers to any and all possible combinations of one or more of the associated listed items, and includes these combinations.

[0214] Additionally, in the specification of the present application and the description of the appended claims, the terms “include,” “comprise,” “have,” and their variations all mean “including but not limited to,” unless otherwise specifically emphasized.

[0215] In the above embodiments, the description of each embodiment has its own emphasis. For parts that are not described or recorded in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0216] Those of ordinary skill in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.

[0217] In the embodiments provided in the present application, it should be understood that the disclosed devices, computer equipment and methods can be implemented in other ways. For example, the device and computer equipment embodiments described above are only schematic. For example, the division of units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0218] The embodiments described above are only used to illustrate the technical solutions of the present application, rather than to limit them. Although the present application has been described in detail with reference to the aforementioned embodiments, a person skilled in the art should understand that the technical solutions described in the aforementioned embodiments may still be modified, or some of the technical features may be replaced by equivalents. Such modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present application, and should all be included in the protection scope of the present application.

Claims

1. A data processing method, characterized in that: Applicable to devices with Hongmeng system installed, including: Obtaining an original request method for requesting original response data, wherein the original request method includes a hypertext transfer protocol (http) request method and / or a hypertext transfer security protocol (https) request method, Determine the target request method according to the original request method and the newly added request logic, wherein the newly added request logic includes at least one of the following: log recording logic, request parameter modification logic, data collection execution logic, and request data transmission logic; The target request method is used to intercept and overwrite the original request method, and the target request method is executed to obtain target response data, the target response data is plain text data, and the content of the target response data includes the content of the original response data.

2. The method according to claim 1, characterized in that The determining the target request method according to the original request method and the newly added request logic includes: Obtain a new request method, and copy the original request method to the new request method; The newly added request logic is set by writing the code corresponding to the newly added request logic in the newly created request method to obtain the target request method. The code corresponding to the newly added request logic includes newly added request information, and the newly added request information includes the request host name and / or request path.

3. The method according to claim 2, characterized in that The method further comprises: Backing up the original request method to obtain a backup request method; When executing the target request method, the backup request method is called and executed to obtain the original response data.

4. The method according to claim 3, characterized in that In the case of executing the target request method, calling and executing the backup request method to obtain the original response data includes: Setting a calling function in the newly created request method, wherein the parameters of the calling function include at least one of the following: uniform resource locator information, request method type information, and request header information; When the calling function is executed, the backup request method is called, and the original response data is obtained by executing the backup request method.

5. The method according to any one of claims 1 to 4, characterized in that: The method further comprises: Processing the target response data using a data event and / or an end event to obtain target response information, wherein the target response information includes text information and / or image information, the data event is used to receive and splice the target response data, and the end event is used to parse the target response data; According to the type of the target response information, the text information and / or the image information are transmitted and stored respectively.

6. The method according to claim 5, characterized in that The transmitting and storing the text information and / or the image information respectively according to the type of the target response information includes: When the target response information is text information, the text information is transmitted using a distributed streaming media platform and / or a server log service, and the text information is stored using a structured data storage method and / or a semi-structured data storage method; When the target response information is image information, the image information is transmitted using a web socket protocol and / or a preset interface, and the image information is stored in an unstructured data storage manner.

7. The method according to any one of claims 1 to 4 or 6, characterized in that: The executing the target request method to obtain target response data includes: Setting a sampling logic in the target request method, wherein the sampling logic includes at least one of the following information: a sampling rate, a sampling frequency, a request type, and a sampling interface; According to the sampling logic, the target request method is executed to obtain the target response data.

8. A data processing device, characterized in that: Applied to Hongmeng system, including: An acquiring unit, used for acquiring an original request method for requesting original response data, wherein the original request method includes an http request method and / or an https request method; A processing unit, configured to determine a target request method according to the original request method and a newly added request logic, wherein the newly added request logic includes at least one of the following: a log recording logic, a request parameter modification logic, a data collection execution logic, and a request data transmission logic; The processing unit is also used to adopt the target request method to intercept and overwrite the original request method, execute the target request method to obtain target response data, the target response data is plain text data, and the content of the target response data includes the content of the original response data.

9. A computer device, characterized in that: The computer device comprises a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, the computer device implements the method according to any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a computer device, the method according to any one of claims 1 to 7 is implemented.

Images