Data acquisition method and device, equipment and storage medium
Through the authentication of the distributed data return platform architecture and the transfer module, combined with the data acquisition strategy of the classification forwarding module, the problems of low efficiency and poor security of data acquisition in the external network are solved, and efficient acquisition and secure transmission of multimodal data are achieved .
Patent Information
- Application Number
- CN202411087858.6
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-08-08
- Publication Date
- 2024-11-29
AI Technical Summary
The data acquisition in the prior art of China and foreign networks has problems of high risks and poor freedom, especially in multimodal data acquisition scenarios, where data acquisition efficiency is low and security cannot be guaranteed.
Adopt a distributed data return platform architecture, authenticate through the transit module, and send data acquisition requests to the corresponding data acquisition service module using the classification forwarding module, obtain multiple types of multimodal data according to the preset strategy, and reduce resources Take up and improve efficiency.
It realizes rapid acquisition and aggregation of large-scale server data, supports data acquisition requests with large concurrent and large data volumes, improves the security and efficiency of data acquisition, and reduces the operational risk of frequent logging into the external network.
Smart Images

Figure CN119051918A_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the field of computer technology, and in particular to a data acquisition method, device, equipment and storage medium. Background Art
[0002] The efficiency of problem repair in the official server is crucial to the stable operation of the game and the player experience. Server problem verification relies on log data and game server save data, but most log data and game server save data are not saved in real time. In addition, data such as temporary variables in memory are basically in a black box state and cannot be obtained through conventional means.
[0003] In the prior art, the intranet can obtain data from the external network by combining hot update with logs. However, there are risks in obtaining data in this way, and there is a problem of poor freedom in data acquisition. Summary of the invention
[0004] The purpose of this application is to provide a data acquisition method, device, equipment and storage medium to address the deficiencies in the above-mentioned prior art, so as to solve the problems of high risk and poor freedom in the prior art of multimodal data acquisition for external networks.
[0005] To achieve the above objectives, the technical solutions adopted in this application are as follows:
[0006] In a first aspect, the present application provides a data acquisition method, which is applied to a data acquisition platform in a data acquisition system, wherein the data acquisition system includes an intranet platform and multiple data acquisition platforms, each of the data acquisition platforms is communicatively connected to the intranet platform, and the data acquisition platform includes multiple data acquisition service modules, each of the data acquisition service modules is used to acquire a type of data, and the method includes:
[0007] Perform identity authentication based on the data acquisition request of the intranet platform and obtain the identity authentication result;
[0008] If the identity authentication result indicates that the authentication is passed, then based on the data type indicated by the data acquisition request, the data acquisition request is sent to a corresponding data acquisition service module;
[0009] The data acquisition service module acquires target data according to the data acquisition request based on a preset data acquisition strategy;
[0010] The target data is sent to the intranet platform according to a preset sending strategy.
[0011] In a second aspect, the present application provides a data acquisition device, applied to a data acquisition platform, the device comprising:
[0012] The identity authentication module is used to perform identity authentication based on the data acquisition request of the intranet platform and obtain the identity authentication result;
[0013] A request forwarding module, configured to send the data acquisition request to a corresponding data acquisition service module based on the data type indicated by the data acquisition request if the identity authentication result indicates that the authentication is passed;
[0014] A data acquisition module, configured to acquire target data according to the data acquisition request based on a preset data acquisition strategy;
[0015] The sending module is used to send the target data to the intranet according to a preset sending strategy.
[0016] In a third aspect, an embodiment of the present application further provides an electronic device, comprising: a processor, a storage medium and a bus, wherein the storage medium stores machine-readable instructions executable by the processor, and when the electronic device is running, the processor communicates with the storage medium through the bus, and the processor executes the machine-readable instructions to perform the steps of the data acquisition method as described in any one of the first aspects.
[0017] In a fourth aspect, an embodiment of the present application further provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the steps of the data acquisition method as described in any one of the first aspects are executed.
[0018] The beneficial effects of the present application are: through the distributed data backhaul platform architecture, it can support large-scale server data rapid acquisition and rapid aggregation, and meet data acquisition requests with large concurrency and large data volumes. The data acquisition request is authenticated by the transfer module, which can improve the security of data acquisition. The data acquisition request is sent to the corresponding data acquisition service module based on the data type indicated by the data acquisition request through the classification forwarding module, which can achieve the refinement of multi-modal data acquisition from multiple sources and multiple types. The data acquisition service module acquires the target data based on the preset data acquisition strategy, which can reduce the resources occupied by acquiring the target data and improve the efficiency of data acquisition. In addition, in the present application, when the intranet platform acquires the target data, the data acquisition platform can acquire the data through the data acquisition service module. The intranet platform can complete the data acquisition by sending a request through the data acquisition platform, without the need to frequently log in to the server of the external network, which improves the efficiency of data acquisition and reduces the operational risks brought by logging in to the external network.
[0019] In order to make the above-mentioned objects, features and advantages of the present application more obvious and easy to understand, preferred embodiments are specifically cited below and described in detail with reference to the attached drawings. BRIEF DESCRIPTION OF THE DRAWINGS
[0020] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings required for use in the embodiments will be briefly introduced below. It should be understood that the following drawings only show certain embodiments of the present application and therefore should not be regarded as limiting the scope. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without paying creative work.
[0021] Figure 1 A schematic diagram of the architecture of a data acquisition system provided in an embodiment of the present application is shown;
[0022] Figure 2 A flow chart of a data acquisition method provided by an embodiment of the present application is shown;
[0023] Figure 3 A flowchart for identity authentication provided by an embodiment of the present application is shown;
[0024] Figure 4 A flow chart of source and authority verification provided by an embodiment of the present application is shown;
[0025] Figure 5 A flowchart for performing permission verification provided by an embodiment of the present application is shown;
[0026] Figure 6 A flow chart for obtaining target data provided by an embodiment of the present application is shown;
[0027] Figure 7 A flowchart of a multi-level acquisition of target data provided by an embodiment of the present application is shown;
[0028] Figure 8 A flow chart of a method for acquiring data in an intranet platform provided in an embodiment of the present application is shown;
[0029] Fig. 9 A schematic diagram of the structure of a data acquisition device provided in an embodiment of the present application is shown;
[0030] Fig.10 A schematic diagram showing the structure of the modules in the intranet platform provided in the embodiment of the present application is shown;
[0031] Fig.11 A schematic structural diagram of an electronic device provided in an embodiment of the present application is shown. DETAILED DESCRIPTION
[0032] To make the purpose, technical scheme and advantages of the embodiments of the present application clearer, the technical scheme in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, rather than all of the embodiments. The components of the embodiments of the present application generally described and shown in the drawings here can be arranged and designed in various different configurations. Therefore, the following detailed description of the embodiments of the present application provided in the drawings is not intended to limit the scope of the application claimed for protection, but merely represents the selected embodiments of the present application. Based on the embodiments of the present application, all other embodiments obtained by those skilled in the art without making creative work belong to the scope of protection of the present application.
[0033] It should be noted that the term "comprising" will be used in the embodiments of the present application to indicate the existence of the features declared thereafter, but does not exclude the addition of other features.
[0034] The server verification of the external network relies on log data and game server saved data, but most of the data is not saved in real time. In addition, the diverse data sources, diverse data modes, and network isolation between the external network and the internal network platform have also become factors affecting the acquisition of external network data.
[0035] Specifically, the sources of external network data include disk data and memory data. Memory data is divided into data in the game process and shared memory data in the memory space. The data from different source channels are divided into text data, binary compressed packages, attribute information, binlog data, running memory variables, mapping data in shared memory, etc. Therefore, the sources of data acquisition are diverse and the types of data are also diverse.
[0036] In the prior art, in an emergency, memory data can be obtained through hot update and log. However, this method of obtaining data has a high acquisition risk on the one hand, and on the other hand, changing data requires hot update again. For scenarios where multi-modal and multi-source data is obtained, this method of data acquisition is very inefficient.
[0037] In addition, due to the network isolation between the external network and the internal network platform, the efficiency and security of data acquisition cannot be guaranteed.
[0038] Based on this, the present application proposes a data acquisition method, which is applied to a data acquisition platform. Figure 1 This is a schematic diagram of the architecture of a data acquisition system provided in this application, referring to Figure 1 The data acquisition system includes: a data acquisition platform, an external network and an internal network (also called an office network). The terminal devices in the internal network can obtain the data of the client devices in the external network through the data acquisition platform.
[0039] Continue to refer to Figure 1 , a transfer module and a classified forwarding module can be deployed in the data acquisition platform. A data acquisition platform can be deployed in each external network environment. Each data acquisition platform is connected to the intranet platform for receiving data acquisition requests from the intranet platform and forwarding the data acquisition requests to the corresponding data acquisition service module to obtain data of the corresponding type through the data acquisition service module. The classified forwarding module classifies and forwards the data acquisition request to obtain data through the corresponding data acquisition module. For multi-modal and multi-source data acquisition scenarios, the efficiency of data acquisition can be improved, and the present application can also improve the security of data acquisition.
[0040] Continue to refer to Figure 1 Data forwarding servers and data receiving and processing servers can also be deployed in the intranet. Developers can initiate data acquisition requests through terminal devices in the intranet. The data forwarding server can filter the data acquisition requests and forward them to the corresponding data acquisition platform.
[0041] After receiving the data acquisition request, the data acquisition platform can authenticate the identity through the transfer module and send the successfully authenticated data acquisition request to the classification forwarding module. The classification forwarding module forwards the data acquisition request to the corresponding data acquisition service module, and the data acquisition service module acquires the target data based on the data acquisition request. The transfer module can be a message transfer plug-in, service or program, such as Link-Agent, which is used to authenticate and forward the data acquisition request. The data acquisition service module can be a plug-in or service in the data acquisition platform, and each data acquisition service module is used to acquire a type of data.
[0042] Among them, the functions implemented by the relay module and the classified forwarding module can also be implemented through the proxy relay service and the classified forwarding service. For example, the proxy relay service and the classified forwarding service can be deployed on the server, and the identity authentication and forwarding of the request can be performed through the proxy service, and the classified forwarding of the request can be performed through the classified forwarding service. The specific deployment method is not limited in this application.
[0043] Figure 2 This is a flow chart of a data acquisition method provided in this application, which can be applied to Figure 1 In the data acquisition platform shown, refer to Figure 2 , the method comprising:
[0044] Step 201: Perform identity authentication based on the data acquisition request of the intranet platform to obtain an identity authentication result.
[0045] The intranet platform includes multiple terminal devices. After the terminal device sends a data acquisition request, the intranet platform can forward the data acquisition request, determine the data acquisition platform where the data indicated by the data acquisition request is located, and send the data acquisition request to the transit module in the corresponding data acquisition platform. The transit module authenticates the data acquisition request.
[0046] Step 202: If the identity authentication result indicates that the authentication is successful, the data acquisition request is sent to a corresponding data acquisition service module based on the data type indicated by the data acquisition request.
[0047] The identity authentication result may indicate whether the user initiating the data acquisition request is an intranet user and whether the user has the authority to acquire the data indicated by the data acquisition request.
[0048] If the user initiating the data acquisition request is a user of the authenticated intranet platform, and the user initiating the data acquisition request has the authority to obtain the data indicated by the data acquisition request, the identity verification result is passed. On the contrary, if the user initiating the data acquisition request is not an intranet user, and / or the user initiating the data acquisition request does not have the authority to obtain the data indicated by the data acquisition request, the identity verification result is failed.
[0049] The transfer module is in communication connection with the classification forwarding module. After the transfer module verifies the data acquisition request, it can forward the data acquisition request to the classification forwarding module. The classification forwarding module can be a plug-in, service or program for secondary authentication and forwarding of the request.
[0050] As a possible implementation method, the classified forwarding module can also perform secondary signature authentication on the data acquisition request to improve the security of request forwarding.
[0051] The data acquisition platform includes multiple data acquisition service modules, each of which is used to acquire data from different data sources and different data types. As a possible implementation method, different data acquisition service modules can be used to acquire data from different source channels, or different data acquisition service modules can be used to acquire data from different source channels and different data types.
[0052] The classification forwarding module can determine the data acquisition service module corresponding to the data requested by the data acquisition request based on the data type and data source requested by the data acquisition request, and forward the data acquisition request to the corresponding data acquisition service module.
[0053] Step 203: The data acquisition service module acquires target data according to the data acquisition request based on a preset data acquisition strategy.
[0054] Optionally, the data acquisition module may be a data acquisition hook, through which data may be directly acquired from a client device on an external network.
[0055] The target data may be data indicated by the data acquisition request, the data source of the target data may include: disk data, memory data, and the data type of the target data may include at least one of the following: text data, binary compression package, attribute information, database log file, running memory variable, and mapping data in shared memory. After the data acquisition service module acquires the target data, it may send the target data to the classification forwarding module, and the classification forwarding module may then send the target data to the transfer module.
[0056] It is worth noting that when acquiring data based on the prior art, each type of data needs to be acquired separately through hot update and log methods. Therefore, for multimodal target data, it is necessary to perform hot update again after acquiring one type of data, which undoubtedly reduces the efficiency of data acquisition. In this application, different types of target data are acquired through corresponding data acquisition service modules respectively, so it is possible to acquire multiple types of data at the same time, reducing the complexity of data acquisition and improving the efficiency of data acquisition.
[0057] In the process of acquiring the target data, the data acquisition service module can first determine the data volume of the target data, and adopt different data acquisition strategies based on the data volume of the target data. In the first implementation method, if the data volume of the target data is greater than the preset value, the target data can be streamlined, and the streamlined target data can be sent to the transfer module via the classification forwarding module. The streamlined data can include key information of the data. At this time, the client device that initiates the data acquisition request can initiate a secondary request based on the streamlined target data to obtain the specific data that needs to be viewed based on the key information of the data. If the data volume of the target data is less than the preset value, the target data can be directly sent to the transfer module.
[0058] In the second implementation method, the data acquisition service module can also acquire the target data in segments, and send the multiple data segments of the acquired target data to the transfer module in sequence via the classification forwarding module. At this time, the data segment may include an identifier to indicate the target data to which the data segment belongs and the order of each data segment.
[0059] Step 204: Send the target data to the intranet platform according to the preset sending strategy.
[0060] Optionally, the transfer module may send the target data to the intranet platform according to a preset sending strategy, so that the intranet platform displays the target data on the terminal device according to a preset format.
[0061] Exemplarily, a terminal device that initiates a data acquisition request in an intranet platform may first convert the target data into a preset format and then present it to the user on the front end.
[0062] In an embodiment of the present application, the transit module performs identity authentication based on the data acquisition request of the intranet platform and obtains an identity authentication result. If the identity authentication result indicates that the verification is successful, the transit module sends the data acquisition request to the classification forwarding module. The classification forwarding module sends the data acquisition request to the corresponding data acquisition service module based on the data type indicated by the data acquisition request. The data acquisition service module acquires the target data according to the data acquisition request based on a preset data acquisition strategy, and sends the target data to the transit module via the classification forwarding module. The transit module sends the target data to the intranet platform.
[0063] Through the distributed data backhaul platform architecture, it is possible to support rapid acquisition and rapid aggregation of large-scale server data, and meet data acquisition requests with large concurrency and large data volumes. By authenticating the data acquisition request through the transit module, the security of data acquisition can be improved. By sending the data acquisition request to the corresponding data acquisition service module based on the data type indicated by the data acquisition request through the classification forwarding module, the refinement of multimodal data acquisition from multiple sources and multiple types can be achieved. The data acquisition service module acquires the target data based on the preset data acquisition strategy, which can reduce the resources occupied by acquiring the target data and improve the efficiency of data acquisition. In addition, in the present application, when the intranet platform acquires the target data, the data acquisition platform can acquire the data through the data acquisition service module. The intranet platform can complete the data acquisition by sending a request through the data acquisition platform, without the need to frequently log in to the server of the external network, thereby improving the efficiency of data acquisition and reducing the operational risks brought by logging in to the external network.
[0064] Next, the steps of authenticating the above-mentioned data acquisition request based on the intranet and obtaining the authentication result are described. This step can be performed by the transfer module, such as Figure 3 As shown, the above step 201 includes:
[0065] Step 301: Based on the request parameters of the data acquisition request, determine the source verification result and the authority verification result of the data acquisition request. The source verification result is used to indicate whether the source of the data acquisition request meets the source requirements, and the authority verification result is used to indicate whether the data acquisition request meets the authority requirements.
[0066] The request parameters of the data acquisition request may include the source information of the data acquisition request and the user information of the initiating user, such as the user ID. Based on the source of the data acquisition request, the source verification result of the data acquisition request may be determined, and based on the user ID in the request parameters, the authority verification result of the data acquisition request may be determined.
[0067] The source of the data acquisition request meets the source requirements. The source of the data acquisition request can be a device on the intranet that has been authenticated by the data acquisition platform. If the source of the data acquisition request is other unauthenticated devices, the source of the data acquisition request does not meet the source requirements.
[0068] The data acquisition request satisfies the permission requirement if the user corresponding to the user information has the permission to view the data indicated by the data acquisition request. Specifically, the user information can be compared with the user permission information table pre-saved in the transfer module to determine the permission of the user corresponding to the user information. If the permission of the user includes the permission to view the data indicated by the data acquisition request, the data acquisition request is considered to meet the permission requirement; if the permission of the user does not include the permission to view the data indicated by the data acquisition request, the data acquisition request is considered to not meet the permission requirement.
[0069] If the data acquisition request does not meet the permission requirements or the data acquisition request does not meet the source requirements, the transfer module may not forward the data acquisition request and return a prompt message to prompt that the user who initiated the data acquisition request does not have viewing permissions or is not an intranet authenticated user.
[0070] Step 302: Determine a legal verification result of the data acquisition request based on the signature data of the data acquisition request.
[0071] The data acquisition request includes signature data, which is used to verify the final legitimacy of the source of the data acquisition request. It can be calculated by the sha256 code composed of urs, data type, and time. The transit module can calculate the signature data based on the data acquisition request, and compare the calculated signature data with the signature data in the data acquisition request to confirm whether the data acquisition request has been tampered with during the transmission process. If the signature data is consistent, the legal verification result is passed, and if it is inconsistent, the legal verification result is failed.
[0072] Step 303: Obtain identity verification results based on the source verification results, authority verification results, and legality verification results.
[0073] If the source verification result indicates that the source of the data acquisition request meets the source requirements, and the permission verification result indicates that the data acquisition request meets the permission requirements, and the legal verification result is passed, then the identity authentication result of the data acquisition request is passed, and the transit module can forward the data acquisition request.
[0074] If the source verification result indicates that the source of the data acquisition request does not meet the source requirements, or the permission verification result indicates that the data acquisition request does not meet the permission requirements, or the legal verification result is failure, then the identity authentication result of the data acquisition request is failure. At this time, the transit module can intercept the data acquisition request and send a prompt message to the terminal device that initiated the data acquisition request.
[0075] Furthermore, the process of determining the source verification result and the authority verification result of the data acquisition request based on the request parameters of the data acquisition request is as follows: Figure 4 As shown, including:
[0076] Step 401: Determine data source information of the data acquisition request based on request parameters of the data acquisition request.
[0077] As a possible implementation method, the transfer module can obtain user information from the cookie of the data acquisition request to obtain data source information of the data acquisition request.
[0078] Exemplarily, the data source information may be an IP segment of a data acquisition request, which is used to indicate a network environment to which a device initiating the data acquisition request belongs.
[0079] Step 402: Determine the source verification result based on the data source information and the preset legal user list.
[0080] The preset legal user list may be the device information of the intranet that has been authenticated in the data acquisition platform. The transfer module can determine the source verification result by comparing the data source information with the authenticated device information of the intranet.
[0081] Specifically, if the IP of the data source information is within the IP segment of the authenticated device information, then it can be confirmed that the data acquisition request meets the source requirement. Conversely, if the IP of the data source information is not within the IP segment of the authenticated device information, then the data acquisition request does not meet the source requirement.
[0082] Step 403: Determine the authority verification result based on the user identifier in the request parameter.
[0083] The user identification may be a user ID that logs into the terminal device and initiates a data acquisition request, and the transfer module may determine whether the account that initiates the request has the authority to view the requested data based on the user ID.
[0084] If the user ID is an account with permission to view the requested data, the data acquisition request meets the permission requirement. Otherwise, if the user ID is an account without permission to view the requested data, the data acquisition request does not meet the permission requirement.
[0085] The following is a further explanation of the above-mentioned determination of the authorization verification result based on the user ID in the request parameter. Figure 5 As shown, the above step 403 includes:
[0086] Step 501: Determine user authority information corresponding to the user identifier based on the user identifier and a pre-built user authority information table.
[0087] The user authority information table records the user ID of the authenticated user and the authority the user has. As a possible implementation method, both the user authority and the data authority can be graded, and different authority levels have the authority to view the corresponding level of data. In this case, the user authority information table can record the user ID of the authenticated user and the user's authority level.
[0088] The transfer module can search in the user authority information table based on the user identification to determine the user authority corresponding to the user identification, or the user authority level corresponding to the user identification.
[0089] Step 502: Determine the authority verification result based on the user authority information and the authority information of the data indicated by the data acquisition request.
[0090] In the first implementation method, the permission information of the data indicated by the data acquisition request can be determined first, such as the permission level of the data or the list of people who can view the data, and whether the user permission information includes the permission to view the data indicated by the data acquisition request can be checked. If so, the data acquisition request meets the permission requirements.
[0091] In the second implementation, the permission verification result can be determined based on the permission level in the user permission information and the permission level of the data indicated by the data acquisition request. Specifically, if the permission level in the user permission information is equal to or higher than the permission level of the data indicated by the data acquisition request, it can be determined that the data acquisition request meets the permission requirement.
[0092] In the embodiment of the present application, the security of the data acquisition process can be improved by verifying the source, authority and legitimacy of the data acquisition request.
[0093] The following is a further description of the data acquisition service module acquiring target data according to the data acquisition request based on the preset data acquisition strategy. Figure 6 As shown, the above step 204 includes:
[0094] Step 601: The data acquisition service module determines the amount of data to be requested based on the data acquisition request.
[0095] The data to be requested may be data requested by the data acquisition request. It should be understood that the data acquired based on different data acquisition strategies are also different, and the target data may be part of the data to be requested, or the target data may be the data to be requested.
[0096] Step 602: The data acquisition service module acquires target data based on the data acquisition strategy according to the data volume of the requested data.
[0097] It is worth noting that if the amount of target data is too large, the data acquisition module directly acquiring the target data may cause memory overflow. Therefore, before acquiring the data, the amount of data to be requested can be determined first, the amount of data to be requested can be compared with the preset value, and different data acquisition strategies can be selected based on the comparison results.
[0098] In the first implementation, Figure 7 As shown, the process of the data acquisition service module acquiring target data according to the amount of data to be requested includes:
[0099] Step 701: If the amount of the data to be requested is greater than a preset value, the data acquisition service module extracts key information of the data to be requested and sends the key information to the intranet platform through the transfer module.
[0100] The data to be requested may be in the form of a key-value pair, and the key information of the data to be requested may be the key of all the data to be requested. If the amount of the data to be requested is greater than a preset value, the data acquisition service module may extract the key of the data to be requested, and send the key of all the data to be requested to the intranet platform through the transfer module.
[0101] Step 702: The data acquisition service module receives the filtered key information sent by the intranet platform through the transfer module, obtains the data corresponding to the filtered key information, and uses the data corresponding to the filtered key information as the target data.
[0102] The data acquisition module can send key information to the intranet devices in the intranet platform via the classification forwarding module and the transfer module. The intranet devices can filter based on the received key information, determine the data that needs to be viewed, and send the filtered key information to the transfer module, and then send it to the data acquisition service module via the classification forwarding module.
[0103] The data acquisition service module can acquire the value of the data based on the filtered key information, and use the value of the filtered key data as the target data.
[0104] Exemplarily, if the data to be requested is the memory data of the client device, after the data acquisition service module judges the data to be requested and determines that the data volume of the data to be requested is greater than the preset value, the key information of the memory data A can be returned first, including: A0 variable data of the running process, A1 directional memory data, A2 discrete memory data, A3 external memory data and A4 server status information. After receiving the key information, the device that initiates the request in the intranet platform can filter based on the key information to determine the filtered key information [A][A3] and obtain the target data to be viewed, that is, the external memory data.
[0105] In another example, if the requested data is disk data and the data volume is greater than the preset value, the key information of the disk data B can be returned first: B1 UDB data, B2 discrete data, B3 data recovery log. After the data acquisition service module receives the filtered key information [B][B1], it can use the UDB data as the target data.
[0106] In the second implementation, the step of obtaining target data by the data acquisition service module according to the amount of data to be requested further includes:
[0107] If the amount of the data to be requested is less than or equal to the preset value, the data to be requested is used as the target data.
[0108] If the amount of data to be requested is less than a preset value, the data to be requested can be directly used as target data and sent to the transfer module via the classification forwarding module, and the transfer module directly sends the target data to the intranet platform.
[0109] In an embodiment of the present application, by determining the data volume of the requested data according to a preset value and adopting different data acquisition strategies to acquire data, small data volumes are directly returned, and large data volumes of requested data are streamlined, thereby achieving multi-level screening and acquisition of large data volumes, thereby breaking through the limitation of string length, reducing the overhead of variable serialization, and solving the problem of acquiring ultra-large memory data.
[0110] Furthermore, the intranet platform in the above data acquisition system includes: a data forwarding server and a data receiving and processing server.
[0111] The intranet platform can send a request to the data acquisition platform to obtain data from the external network, and when the data acquisition platform returns the data, it will display the data in a preset format. Figure 8 As shown, after the data acquisition platform sends the target data to the intranet platform based on the preset sending strategy, the method of the present application further includes:
[0112] Step 801: The data forwarding server obtains a data acquisition request.
[0113] Step 802: The data forwarding server determines a target data acquisition platform corresponding to the data acquisition request, and sends the data acquisition request to the target data acquisition platform.
[0114] The data forwarding server may receive a data acquisition request initiated by a terminal device in the intranet platform, and the data acquisition request is used to acquire data from a client device in the external network.
[0115] The data forwarding server can determine the target data acquisition platform connected to the client device based on the client device to which the data indicated by the data acquisition request belongs. When multiple data acquisition platforms are deployed in a distributed manner, the client device can be connected to multiple data acquisition platforms. At this time, the data forwarding server can send the data acquisition request to all data acquisition platforms connected to the client device.
[0116] As a possible implementation method, the data acquisition request can be used to acquire data from client devices in multiple external networks. When the multiple client devices are in different external networks, the data forwarding server can determine the data acquisition platforms connected to each external network based on the external network to which the data requested by the data acquisition request belongs, and forward the data acquisition request to each data acquisition platform.
[0117] Step 803: The data receiving and processing server receives the target data sent by the target data acquisition platform, and outputs the target data in a preset format.
[0118] The data receiving and processing server is used to receive the data returned by the data acquisition platform and process the data accordingly according to the data type, such as formatting memory data and disk data, backing up log data, storing log data and status data based on a preset format, etc.
[0119] After receiving the data, the data receiving and processing server can format the data, for example, convert the data into a uniform KV visual format, and then present it to the user on the front end.
[0120] Based on the same inventive concept, a data acquisition device corresponding to the data acquisition method is also provided in the embodiment of the present application. Since the principle of solving the problem by the device in the embodiment of the present application is similar to the above-mentioned data acquisition method in the embodiment of the present application, the implementation of the device can refer to the implementation of the method, and the repeated parts will not be repeated.
[0121] Fig. 9 The structure diagram of a data acquisition device provided in an embodiment of the present application is shown. The device is applied to a data acquisition platform, referring to Fig. 9 , the device comprises:
[0122] The identity authentication module 901 is used to perform identity authentication based on the data acquisition request of the intranet platform and obtain an identity authentication result;
[0123] The request forwarding module 902 is used to send the data acquisition request to the corresponding data acquisition service module based on the data type indicated by the data acquisition request if the identity authentication result indicates that the authentication is passed;
[0124] A data acquisition module 903 is used to acquire target data according to a data acquisition request based on a preset data acquisition strategy;
[0125] The sending module 904 is used to send the target data to the intranet according to a preset sending strategy.
[0126] In a feasible implementation scheme, the identity verification module 901 is specifically used for:
[0127] Determine, based on the request parameters of the data acquisition request, a source verification result and an authority verification result of the data acquisition request, wherein the source verification result is used to indicate whether the source of the data acquisition request meets the source requirement, and the authority verification result is used to indicate whether the data acquisition request meets the authority requirement;
[0128] Determine the legality verification result of the data acquisition request based on the signature data of the data acquisition request;
[0129] The identity verification result is obtained based on the source verification result, the authority verification result and the legality verification result.
[0130] In a feasible implementation scheme, the identity verification module 901 is specifically used for:
[0131] Determine data source information of the data acquisition request based on request parameters of the data acquisition request;
[0132] Determine the source verification result based on the data source information and the preset legal user list;
[0133] The authorization verification result is determined based on the user ID in the request parameter.
[0134] In a feasible implementation scheme, the identity verification module 901 is specifically used for:
[0135] Determine the user authority information corresponding to the user identity based on the user identity and the pre-built user authority information table;
[0136] Based on the user authority information and the authority information of the data indicated by the data acquisition request, an authority verification result is determined.
[0137] In a feasible implementation scheme, the data acquisition module 903 is specifically used for:
[0138] Determine the amount of data to be requested based on the data acquisition request;
[0139] According to the amount of data to be requested, the target data is acquired based on the data acquisition strategy.
[0140] In a feasible implementation scheme, the data acquisition module 903 is specifically used for:
[0141] If the amount of data to be requested is greater than the preset value, the key information of the data to be requested is extracted and sent to the intranet platform through the transfer module;
[0142] The filtered key information sent by the intranet platform is received through the transfer module, the data corresponding to the filtered key information is obtained, and the data corresponding to the filtered key information is used as the target data.
[0143] In a feasible implementation scheme, the data acquisition module 903 is specifically used for:
[0144] If the amount of the data to be requested is less than or equal to the preset value, the data to be requested is used as the target data.
[0145] Fig.10 In one feasible implementation, the intranet platform includes a request acquisition module 1001 , a request sending module 1002 and a data receiving module 1003 .
[0146] The request acquisition module 1001 is used to acquire a data acquisition request;
[0147] A request sending module 1002 is used to determine a target data acquisition platform corresponding to the data acquisition request and send the data acquisition request to the target data acquisition platform;
[0148] The data receiving module 1003 is used to receive the target data sent by the target data acquisition platform and output the target data according to a preset format.
[0149] In a feasible implementation scheme, the data sources of the target data include: disk data, memory data, and the data types of the target data include at least one of the following: text data, binary compressed package, attribute information, database log file, running memory variable, and mapping data in shared memory.
[0150] The embodiment of the present application can support large-scale rapid acquisition and rapid aggregation of server data through a distributed data backhaul platform architecture, and meet data acquisition requests with large concurrency and large data volumes. The data acquisition request is authenticated by the transfer module, which can improve the security of data acquisition. The data acquisition request is sent to the corresponding data acquisition service module based on the data type indicated by the data acquisition request through the classification forwarding module, which can achieve the refinement of multi-modal data acquisition from multiple sources and multiple types. The data acquisition service module acquires the target data based on the preset data acquisition strategy, which can reduce the resources occupied by acquiring the target data and improve the efficiency of data acquisition. In addition, in the present application, when the intranet platform acquires the target data, the data acquisition platform can acquire the data through the data acquisition service module. The intranet platform can complete the data acquisition by sending a request through the data acquisition platform, without the need to frequently log in to the server of the external network, which improves the efficiency of data acquisition and reduces the operational risks brought by logging in to the external network.
[0151] Fig.11 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application is shown, including: a processor 1101, a storage medium 1102 and a bus 1103, wherein the storage medium 1102 stores machine-readable instructions executable by the processor 1101. When the electronic device runs a data acquisition method in the embodiment, the processor 1101 communicates with the storage medium 1102 through the bus 1103, and the processor 1101 executes the machine-readable instructions. The processor 1101 performs the following steps in the preamble of the method item:
[0152] Perform identity authentication based on the data acquisition request of the intranet platform and obtain the identity authentication result;
[0153] If the identity authentication result indicates that the authentication is passed, then based on the data type indicated by the data acquisition request, the data acquisition request is sent to the corresponding data acquisition service module;
[0154] The data acquisition service module acquires target data according to the data acquisition request based on the preset data acquisition strategy;
[0155] The transfer module sends the target data to the intranet according to the preset sending strategy.
[0156] In a feasible implementation scheme, when the processor 1101 performs identity authentication on the intranet-based data acquisition request and obtains the identity authentication result, it is further configured to:
[0157] Determine, based on the request parameters of the data acquisition request, a source verification result and an authority verification result of the data acquisition request, wherein the source verification result is used to indicate whether the source of the data acquisition request meets the source requirement, and the authority verification result is used to indicate whether the data acquisition request meets the authority requirement;
[0158] Determine the legality verification result of the data acquisition request based on the signature data of the data acquisition request;
[0159] The identity verification result is obtained based on the source verification result, the authority verification result and the legality verification result.
[0160] In a feasible implementation, when the processor 1101 determines the source verification result and the authority verification result of the data acquisition request based on the request parameters of the data acquisition request, it is further configured to:
[0161] Determine data source information of the data acquisition request based on request parameters of the data acquisition request;
[0162] Determine the source verification result based on the data source information and the preset legal user list;
[0163] The authorization verification result is determined based on the user ID in the request parameter.
[0164] In a feasible implementation, when the processor 1101 determines the authority verification result based on the user identifier in the request parameter, it is further configured to:
[0165] Determine the user authority information corresponding to the user identity based on the user identity and the pre-built user authority information table;
[0166] Based on the user authority information and the authority information of the data indicated by the data acquisition request, an authority verification result is determined.
[0167] In a feasible implementation, when the processor 1101 executes the acquisition of target data based on the data acquisition request, it is further configured to:
[0168] Determine the amount of data to be requested based on the data acquisition request;
[0169] According to the amount of data to be requested, the target data is acquired based on the data acquisition strategy.
[0170] In a feasible implementation manner, when the processor 1101 acquires the target data based on the data acquisition strategy according to the data amount of the data to be requested, it is further configured to:
[0171] If the amount of data to be requested is greater than the preset value, the key information of the data to be requested is extracted and sent to the intranet platform through the transfer module;
[0172] The filtered key information sent by the intranet platform is received through the transfer module, the data corresponding to the filtered key information is obtained, and the data corresponding to the filtered key information is used as the target data.
[0173] In a feasible implementation manner, when the processor 1101 acquires the target data based on the data acquisition strategy according to the data amount of the data to be requested, it is further configured to:
[0174] If the amount of the data to be requested is less than or equal to the preset value, the data to be requested is used as the target data.
[0175] In a feasible implementation scheme, after executing the preset sending strategy to send the target data to the intranet platform, the processor 1101 is specifically configured to:
[0176] Get data acquisition request;
[0177] Determine a target data acquisition platform corresponding to the data acquisition request, and send the data acquisition request to the target data acquisition platform;
[0178] Receive the target data sent by the target data acquisition platform and output the target data in a preset format.
[0179] In a feasible implementation scheme, the data sources of the target data include: disk data, memory data, and the data types of the target data include at least one of the following: text data, binary compressed package, attribute information, database log file, running memory variable, and mapping data in shared memory.
[0180] The embodiment of the present application can support large-scale rapid acquisition and rapid aggregation of server data through a distributed data backhaul platform architecture, and meet data acquisition requests with large concurrency and large data volumes. The data acquisition request is authenticated by the transfer module, which can improve the security of data acquisition. The data acquisition request is sent to the corresponding data acquisition service module based on the data type indicated by the data acquisition request through the classification forwarding module, which can achieve the refinement of multi-modal data acquisition from multiple sources and multiple types. The data acquisition service module acquires the target data based on the preset data acquisition strategy, which can reduce the resources occupied by acquiring the target data and improve the efficiency of data acquisition. In addition, in the present application, when the intranet platform acquires the target data, the data acquisition platform can acquire the data through the data acquisition service module. The intranet platform can complete the data acquisition by sending a request through the data acquisition platform, without the need to frequently log in to the server of the external network, which improves the efficiency of data acquisition and reduces the operational risks brought by logging in to the external network.
[0181] The embodiment of the present application further provides a computer-readable storage medium, on which a computer program is stored. The computer program is executed when a processor is running, and the processor performs the following steps:
[0182] Perform identity authentication based on the data acquisition request of the intranet platform and obtain the identity authentication result;
[0183] If the identity authentication result indicates that the authentication is passed, then based on the data type indicated by the data acquisition request, the data acquisition request is sent to the corresponding data acquisition service module;
[0184] The data acquisition service module acquires target data according to the data acquisition request based on the preset data acquisition strategy;
[0185] Send the target data to the intranet according to the preset sending strategy.
[0186] In a feasible implementation scheme, when the processor performs identity authentication on an intranet-based data acquisition request and obtains an identity authentication result, it is also used to:
[0187] Determine, based on the request parameters of the data acquisition request, a source verification result and an authority verification result of the data acquisition request, wherein the source verification result is used to indicate whether the source of the data acquisition request meets the source requirement, and the authority verification result is used to indicate whether the data acquisition request meets the authority requirement;
[0188] Determine the legality verification result of the data acquisition request based on the signature data of the data acquisition request;
[0189] The identity verification result is obtained based on the source verification result, the authority verification result and the legality verification result.
[0190] In a feasible implementation scheme, when the processor determines the source verification result and the authority verification result of the data acquisition request based on the request parameters of the data acquisition request, it is also used to:
[0191] Determine data source information of the data acquisition request based on request parameters of the data acquisition request;
[0192] Determine the source verification result based on the data source information and the preset legal user list;
[0193] The authorization verification result is determined based on the user ID in the request parameter.
[0194] In a feasible implementation scheme, when the processor determines the authority verification result based on the user identifier in the request parameter, it is also used to:
[0195] Determine the user authority information corresponding to the user identity based on the user identity and the pre-built user authority information table;
[0196] Based on the user authority information and the authority information of the data indicated by the data acquisition request, an authority verification result is determined.
[0197] In a feasible implementation manner, when the processor executes the acquisition of target data based on the data acquisition request, it is further configured to:
[0198] Determine the amount of data to be requested based on the data acquisition request;
[0199] According to the amount of data to be requested, the target data is acquired based on the data acquisition strategy.
[0200] In a feasible implementation scheme, when the processor acquires the target data based on the data acquisition strategy according to the data amount of the data to be requested, it is further configured to:
[0201] If the amount of data to be requested is greater than the preset value, the key information of the data to be requested is extracted and sent to the intranet platform through the transfer module;
[0202] The filtered key information sent by the intranet platform is received through the transfer module, the data corresponding to the filtered key information is obtained, and the data corresponding to the filtered key information is used as the target data.
[0203] In a feasible implementation scheme, when the processor acquires the target data based on the data acquisition strategy according to the data amount of the data to be requested, it is further configured to:
[0204] If the amount of the data to be requested is less than or equal to the preset value, the data to be requested is used as the target data.
[0205] In one possible embodiment, the processor is further configured to:
[0206] Get data acquisition request;
[0207] Determine a target data acquisition platform corresponding to the data acquisition request, and send the data acquisition request to the target data acquisition platform;
[0208] Receive the target data sent by the target data acquisition platform and output the target data in a preset format.
[0209] In a feasible implementation scheme, the data sources of the target data include: disk data, memory data, and the data types of the target data include at least one of the following: text data, binary compressed package, attribute information, database log file, running memory variable, and mapping data in shared memory.
[0210] The embodiment of the present application can support large-scale rapid acquisition and rapid aggregation of server data through a distributed data backhaul platform architecture, and meet data acquisition requests with large concurrency and large data volumes. The data acquisition request is authenticated by the transfer module, which can improve the security of data acquisition. The data acquisition request is sent to the corresponding data acquisition service module based on the data type indicated by the data acquisition request through the classification forwarding module, which can achieve the refinement of multi-modal data acquisition from multiple sources and multiple types. The data acquisition service module acquires the target data based on the preset data acquisition strategy, which can reduce the resources occupied by acquiring the target data and improve the efficiency of data acquisition. In addition, in the present application, when the intranet platform acquires the target data, the data acquisition platform can acquire the data through the data acquisition service module. The intranet platform can complete the data acquisition by sending a request through the data acquisition platform, without the need to frequently log in to the server of the external network, which improves the efficiency of data acquisition and reduces the operational risks brought by logging in to the external network.
[0211] In the embodiment of the present application, the computer program can also execute other machine-readable instructions when run by the processor to execute other methods described in the embodiment. For the specific execution method steps and principles, please refer to the description of the embodiment, which will not be repeated here.
[0212] In the embodiments provided in the present application, it should be understood that the disclosed devices and methods can be implemented in other ways. The device embodiments described above are merely schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some communication interfaces, and the indirect coupling or communication connection of the devices or units can be electrical, mechanical or other forms.
[0213] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.
[0214] In addition, each functional unit in the embodiments provided in the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit.
[0215] If the functions are implemented in the form of software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art or the part of the technical solution, can be embodied in the form of a software product, which is stored in a storage medium and includes several instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the methods described in each embodiment of the present application. The aforementioned storage medium includes: various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk.
[0216] It should be noted that similar numbers and letters represent similar items in the following figures. Therefore, once an item is defined in one figure, it does not need to be further defined and explained in subsequent figures. In addition, the terms "first", "second", "third", etc. are only used to distinguish the description and are not to be understood as indicating or implying relative importance.
[0217] Finally, it should be noted that the above-described embodiments are only specific implementation methods of the present application, which are used to illustrate the technical solutions of the present application, rather than to limit them. The protection scope of the present application is not limited thereto. Although the present application is described in detail with reference to the above-mentioned embodiments, ordinary technicians in the field should understand that any technician familiar with the technical field can still modify the technical solutions recorded in the above-mentioned embodiments within the technical scope disclosed in the present application, or can easily think of changes, or make equivalent replacements for some of the technical features therein; and these modifications, changes or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present application. They should all be included in the protection scope of the present application. Therefore, the protection scope of the present application shall be based on the protection scope of the claims.
Claims
1. A data acquisition method, characterized in that: A data acquisition platform applied to a data acquisition system, wherein the data acquisition system includes an intranet platform and multiple data acquisition platforms, each of the data acquisition platforms is communicatively connected to the intranet platform, the data acquisition platform includes multiple data acquisition service modules, each of the data acquisition service modules is used to acquire a type of data, and the method includes: Perform identity authentication based on the data acquisition request of the intranet platform and obtain the identity authentication result; If the identity authentication result indicates that the authentication is passed, then based on the data type indicated by the data acquisition request, the data acquisition request is sent to a corresponding data acquisition service module; The data acquisition service module acquires target data according to the data acquisition request based on a preset data acquisition strategy; The target data is sent to the intranet platform according to a preset sending strategy.
2. The method according to claim 1, characterized in that The data acquisition request based on the intranet platform performs identity authentication to obtain an identity authentication result, including: Determine, based on the request parameters of the data acquisition request, a source verification result and an authority verification result of the data acquisition request, wherein the source verification result is used to indicate whether the source of the data acquisition request meets the source requirement, and the authority verification result is used to indicate whether the data acquisition request meets the authority requirement; Determining a legal verification result of the data acquisition request based on the signature data of the data acquisition request; The identity verification result is obtained according to the source verification result, the authority verification result and the legality verification result.
3. The method according to claim 2, characterized in that The determining, based on the request parameters of the data acquisition request, a source verification result and an authority verification result of the data acquisition request includes: Determining data source information of the data acquisition request based on request parameters of the data acquisition request; Determine the source verification result according to the data source information and a preset legal user list; The authority verification result is determined based on the user identifier in the request parameter.
4. The method according to claim 3, characterized in that The determining the authority verification result based on the user identifier in the request parameter includes: Based on the user identifier and a pre-built user authority information table, determining the user authority information corresponding to the user identifier; The authority verification result is determined based on the user authority information and the authority information of the data indicated by the data acquisition request.
5. The method according to claim 1, characterized in that The data acquisition service module acquires target data according to the data acquisition request based on a preset data acquisition strategy, including: The data acquisition service module determines the data amount of the data to be requested based on the data acquisition request; The data acquisition service module acquires the target data based on the data acquisition strategy according to the data volume of the data to be requested.
6. The method according to claim 5, characterized in that The data acquisition platform also includes: a transfer module; The data acquisition service module acquires the target data according to the data volume of the data to be requested and based on the data acquisition strategy, including: If the amount of the data to be requested is greater than a preset value, the data acquisition service module extracts key information of the data to be requested, and sends the key information to the intranet platform through the transfer module; The data acquisition service module receives the filtered key information sent by the intranet platform through the transfer module, obtains the data corresponding to the filtered key information, and uses the data corresponding to the filtered key information as the target data.
7. The method according to claim 5, characterized in that The data acquisition service module acquires the target data according to the data volume of the data to be requested and based on the data acquisition strategy, including: If the data size of the to-be-requested data is less than or equal to a preset value, the to-be-requested data is used as the target data.
8. The method according to claim 1, characterized in that The intranet platform includes: a data forwarding server and a data receiving and processing server; After sending the target data to the intranet platform according to the preset sending strategy, the method further includes: The data forwarding server obtains a data acquisition request; The data forwarding server determines a target data acquisition platform corresponding to the data acquisition request, and sends the data acquisition request to the target data acquisition platform; The data receiving and processing server receives the target data sent by the target data acquisition platform, and outputs the target data in a preset format.
9. The method according to any one of claims 1 to 8, characterized in that: The data sources of the target data include: disk data, memory data, and the data types of the target data include at least one of the following: text data, binary compression package, attribute information, database log file, running memory variable, and mapping data in shared memory.
10. A data acquisition device, characterized in that: Applied to a data acquisition platform, the device comprises: The identity authentication module is used to perform identity authentication based on the data acquisition request of the intranet platform and obtain the identity authentication result; A request forwarding module, configured to send the data acquisition request to a corresponding data acquisition service module based on the data type indicated by the data acquisition request if the identity authentication result indicates that the authentication is passed; A data acquisition module, configured to acquire target data according to the data acquisition request based on a preset data acquisition strategy; The sending module is used to send the target data to the intranet according to a preset sending strategy.
11. An electronic device, characterized in that: include: A processor, a storage medium and a bus, wherein the storage medium stores machine-readable instructions executable by the processor, and when the electronic device is running, the processor and the storage medium communicate via the bus, and the processor executes the machine-readable instructions to perform the steps of the data acquisition method as described in any one of claims 1 to 9.
12. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps of the data acquisition method according to any one of claims 1 to 9 are executed.
