Data writing method, apparatus, and electronic device

Abstract

Embodiments of the present application provide a data writing method and device and electronic equipment, which are applied to a log processing end in an object storage system, the object storage system further comprises a proxy cluster, the log processing end communicates with the proxy cluster, and the method comprises the following steps: continuously receiving original information corresponding to each data request sent by the proxy cluster, wherein the original information corresponding to each data request is information generated by the proxy cluster based on the received data request and used for generating log entry data; for each data request, generating log entry data corresponding to the data request based on the original information corresponding to the data request, wherein each log entry data comprises field data, and the field identifier of each field data is a preset field identifier; and writing the log entry data corresponding to each data request into a log database. Through the present scheme, the performance of the proxy cluster can be prevented from being reduced.

Description

Technical Field

[0001] This invention relates to the field of data processing technology, and in particular to data writing methods, apparatus and electronic devices. Background Technology

[0002] Object storage systems provide users with network-based data access services. Through object storage systems, users can easily store and retrieve various data files such as images, audio, video, and text over the network. Specifically, object storage systems can organize user-uploaded data files into storage spaces (buckets) and then provide addresses (domain names) to users for sharing or downloading the data files.

[0003] The aforementioned object storage systems typically implement data access based on a proxy cluster (such as Nginx). In existing technologies, the proxy cluster is used to receive data requests from clients and distribute them to the object storage server to obtain the corresponding request results. Simultaneously, the proxy cluster is also used to generate log entries for the received data requests and is responsible for writing the generated log entries to the log database in the log server.

[0004] However, in high-concurrency scenarios, the performance of the proxy cluster degrades significantly because it needs to generate and store log data for each data request while receiving and distributing a large number of data requests. Summary of the Invention

[0005] The purpose of this invention is to provide a data writing method, apparatus, and electronic device to avoid performance degradation in proxy clusters. The specific technical solution is as follows:

[0006] In a first aspect, embodiments of the present invention provide a data writing method applied to a log processing terminal in an object storage system, wherein the object storage system further includes a proxy cluster, the log processing terminal communicating with the proxy cluster, and the method comprising:

[0007] The system continuously receives the original information corresponding to each data request sent by the proxy cluster. The original information corresponding to each data request is: information generated by the proxy cluster based on the received data request, which is used to generate log entry data.

[0008] For each data request, based on the original information corresponding to the data request, log entry data corresponding to the data request is generated. Each log entry data includes various field data, and the field identifier of each field data is a preset field identifier.

[0009] Write the log entry data corresponding to each data request to the log database.

[0010] Optionally, writing the log entry data corresponding to each data request into the log database includes:

[0011] Log entries corresponding to each data request are written to at least one log file. Each log file is used to write log entries that meet predetermined aggregation conditions. The predetermined aggregation conditions include: the generation time belongs to the same time granularity, or the receiving time of the corresponding data request belongs to the same time granularity.

[0012] The log database stores the at least one log file.

[0013] Optionally, before storing the at least one log file in the log database, the method further includes:

[0014] The log entries in the log file are corrected to obtain the corrected log file.

[0015] The correction process includes:

[0016] If the field content of the target field data in the log entry data is different from the actual content corresponding to the target field data, the field content of the target field data shall be modified to the storage content corresponding to the target field data.

[0017] The actual content corresponding to the target field data is the storage content of the data requested by the log entry data to which the target field data belongs in the object storage system.

[0018] Optionally, the step of correcting the data of each log entry in the log file to obtain a corrected log file includes:

[0019] By using at least one running instance, the data of each log entry in the log file is corrected to obtain a corrected log file.

[0020] Optionally, the number of the at least one running instance is multiple, and different running instances correspond to different entry position ranges;

[0021] The step of correcting the log entries in the log file using at least one running instance to obtain a corrected log file includes:

[0022] By using each of the multiple running instances, the log entry data in the log file corresponding to the entry position range of that running instance is corrected to obtain the corrected log file.

[0023] Optionally, the method further includes:

[0024] Receive a statistical request for the stored log entries in the log database, wherein the statistical request carries a user identifier, statistical fields, and filtering rules;

[0025] According to the filtering rules, the log entries corresponding to the data requests from the user represented by the user identifier in the log database are filtered.

[0026] The field data belonging to the statistical field in the filtered log entries;

[0027] Based on the statistical field data, a response result is generated for the statistical request.

[0028] Optionally, the method further includes:

[0029] Receive an index request for stored log entry data in the log database, wherein the index request carries an index identifier;

[0030] From the log database, query the log entry data represented by the index identifier;

[0031] Based on the queried log entry data, a response result is generated for the index request.

[0032] Secondly, embodiments of the present invention provide a data writing device applied to a log processing terminal in an object storage system, wherein the object storage system further includes a proxy cluster, the log processing terminal communicating with the proxy cluster, and the device comprising:

[0033] The information receiving module is used to continuously receive the original information corresponding to each data request sent by the agent cluster. The original information corresponding to each data request is: the information generated by the agent cluster based on the received data request, which is used to generate log entry data.

[0034] The data generation module is used to generate log entry data corresponding to each data request based on the original information corresponding to the data request. Each log entry data includes various field data, and the field identifier of each field data is a preset field identifier.

[0035] The data writing module is used to write the log entry data corresponding to each data request into the log database.

[0036] Optionally, the data writing module is specifically used to write the log entry data corresponding to each data request into at least one log file, wherein each log file is used to write log entry data that meets predetermined summarization conditions, the predetermined summarization conditions including: the generation time belongs to the same time granularity, or the receiving time of the corresponding data request belongs to the same time granularity; at least one log file is stored in the log database.

[0037] Optionally, the data writing module is further configured to correct the log entry data in the log file before storing at least one log file in the log database, thereby obtaining a corrected log file; wherein the correction process includes: if the field content of the target field data in the log entry data is different from the actual content corresponding to the target field data, modifying the field content of the target field data to the storage content corresponding to the target field data; wherein the actual content corresponding to the target field data is: the storage content in the object storage system of the data targeted by the data request corresponding to the log entry data to which the target field data belongs.

[0038] Optionally, the data writing module is specifically used to correct the data of each log entry in the log file through at least one running instance to obtain a corrected log file.

[0039] Optionally, at least one running instance may be multiple, with different running instances corresponding to different entry location ranges;

[0040] The data writing module is specifically used to correct the log entry data in the log file corresponding to the entry position range of the running instance through each of the multiple running instances, so as to obtain the corrected log file.

[0041] Optionally, the device further includes:

[0042] The statistics module receives statistical requests for stored log entries in the log database. The statistical request carries a user identifier, statistical fields, and filtering rules. According to the filtering rules, it filters the log entry data in the log database corresponding to the data request from the user represented by the user identifier. It then counts the field data belonging to the statistical fields in the filtered log entry data. Based on the counted field data, it generates a response result for the statistical request.

[0043] Optionally, the device further includes:

[0044] The indexing module is used to receive index requests for stored log entries in the log database, wherein the index request carries an index identifier; query the log entry data represented by the index identifier from the log database; and generate a response result for the index request based on the queried log entry data.

[0045] Thirdly, embodiments of the present invention provide an electronic device, including a processor, a communication interface, a memory, and a communication bus, wherein the processor, the communication interface, and the memory communicate with each other through the communication bus;

[0046] Memory, used to store computer programs;

[0047] When a processor executes a program stored in memory, it implements any of the steps described in the first aspect.

[0048] Fourthly, embodiments of the present invention provide a computer-readable storage medium storing a computer program, wherein the computer program, when executed by a processor, implements the steps of the method described in any of the first aspects.

[0049] Beneficial effects of the embodiments of the present invention:

[0050] In the data writing method provided by this invention, since the log processing terminal in the object storage system can receive the raw information generated by the proxy cluster based on the received data request, and generate log entry data based on the raw information, and then write the log entry data into the log database, the proxy cluster can avoid generating log entry data for the received data request, thereby reducing the resource consumption of the proxy cluster. Therefore, this solution can avoid performance degradation of the proxy cluster.

[0051] Of course, implementing any product or method of the present invention does not necessarily require achieving all of the advantages described above at the same time. Attached Figure Description

[0052] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other embodiments can be obtained based on these drawings without creative effort.

[0053] Figure 1 A flowchart of the data writing method provided in an embodiment of the present invention;

[0054] Figure 2 This is another flowchart of the data writing method provided in an embodiment of the present invention;

[0055] Figure 3 This is another flowchart of the data writing method provided in an embodiment of the present invention;

[0056] Figure 4This is another flowchart of the data writing method provided in an embodiment of the present invention;

[0057] Figure 5 This is another flowchart of the data writing method provided in an embodiment of the present invention;

[0058] Figure 6 This is a schematic diagram of the structure of the object storage system provided in an embodiment of the present invention;

[0059] Figure 7 This is a schematic diagram of the architecture of the data writing device provided in an embodiment of the present invention;

[0060] Figure 8 This is a schematic diagram of the data writing device provided in an embodiment of the present invention;

[0061] Figure 9 This is a schematic diagram of the structure of the electronic device provided in an embodiment of the present invention. Detailed Implementation

[0062] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0063] To avoid performance degradation of the proxy cluster, embodiments of the present invention provide a data writing method, apparatus, and electronic device.

[0064] The following section will first introduce a data writing method provided by an embodiment of the present invention from the perspective of the log processing end in an object storage system.

[0065] It should be noted that the object storage system provided in this embodiment of the invention also includes a proxy cluster, and the log processing terminal communicates with the proxy cluster. The proxy cluster is responsible for receiving / distributing data requests from clients in the object storage system. Simultaneously, it generates raw information for generating log entry data based on the received data requests and sends this raw information to the log processing terminal. The log processing terminal then generates and stores log entry data based on the raw information, while the proxy cluster does not need to generate or store log entry data.

[0066] The aforementioned log processing terminal can be a server or other device with data processing capabilities. Furthermore, the data writing method provided in this embodiment of the invention can be implemented through software, hardware, or a combination of both.

[0067] One embodiment of the present invention provides a data writing method applied to the log processing end of an object storage system, which may include the following steps:

[0068] The system continuously receives the original information corresponding to each data request sent by the proxy cluster. The original information corresponding to each data request is: information generated by the proxy cluster based on the received data request, which is used to generate log entry data.

[0069] For each data request, based on the original information corresponding to the data request, log entry data corresponding to the data request is generated. Each log entry data includes various field data, and the field identifier of each field data is a preset field identifier.

[0070] Write the log entry data corresponding to each data request to the log database.

[0071] In the data writing method provided by this invention, since the log processing terminal in the object storage system can receive the raw information generated by the proxy cluster based on the received data request, and generate log entry data based on the raw information, and then write the log entry data into the log database, the proxy cluster can avoid generating log entry data for the received data request, thereby reducing the resource consumption of the proxy cluster. Therefore, this solution can avoid performance degradation of the proxy cluster.

[0072] The following is a detailed description of a data writing method provided by an embodiment of the present invention, with reference to the accompanying drawings.

[0073] like Figure 1 As shown in the figure, an embodiment of the present invention provides a data writing method applied to the log processing end of an object storage system, which may include the following steps:

[0074] S101, continuously receive the original information corresponding to each data request sent by the agent cluster. The original information corresponding to each data request is: the information generated by the agent cluster based on the received data request, which is used to generate log entry data.

[0075] Upon receiving each data request, the agent cluster can generate raw information based on that request and send it to the log processing end. The content of the raw information can be determined based on actual needs.

[0076] Optionally, in one implementation, the proxy cluster can use the received data request as the raw information. Alternatively, after receiving a data request, the proxy cluster can record information such as the reception time, data volume, and request type, and use this recorded information to generate the raw information. Alternatively, the proxy cluster can obtain data information of the requested data, such as data identifier, data size, storage type, and billing traffic, and use this obtained data information as the raw information. Alternatively, the proxy cluster can package the received data request, the recorded information, and / or the obtained data information into a single raw information package.

[0077] After generating the raw information, the proxy cluster can send it to the log processing end. Optionally, the proxy cluster sends the raw information corresponding to each data request it receives to the log processing end, so that the log processing end can continuously receive the raw information corresponding to each data request sent by the proxy cluster.

[0078] S102, for each data request, based on the original information corresponding to the data request, generate log entry data corresponding to the data request, wherein each log entry data includes various field data, and the field identifier of each field data is a preset field identifier;

[0079] After receiving the raw information sent by the proxy cluster, the log processing end can process the raw information corresponding to each data request in the order in which the raw information is received, and generate the log entry data corresponding to that data request, since it is necessary to generate log entry data based on the received raw information.

[0080] It is understandable that, in order to generate the required log entry data, preset field identifiers can be pre-set. Each preset field identifier is the field identifier of the field data in the log entry data to be generated. The above field identifiers can be field names, and can include at least one of the following: request time, data identifier, data size, connection duration, transmitted bytes, storage type, request type, billing traffic, etc.

[0081] When it is necessary to generate log entry data based on the original information, the field data with the field identifier set to the preset field identifier can be selected from the original information according to each preset field identifier.

[0082] For example, the preset field identifiers are: request time and data identifier. The original information corresponding to data request 1 includes: {(request time: November 12, 2020), (data identifier: file 1), (request type: modification type), (transfer bytes: 1024KB)}. The log processing end can then select the field data (request time: November 12, 2020) and (data identifier: file 1) to generate log entry data as shown in Table 1.

[0083] Table 1

[0084] Request time Data Identifier November 12, 2020 Document 1

[0085] The log entry data shown in Table 1 is the log entry data corresponding to data request 1.

[0086] S103, Write the log entry data corresponding to each data request into the log database;

[0087] After generating log entry data corresponding to each data request, the log processing client can write the generated log entry data to the log database for persistent storage. Optionally, the log processing client can initiate a write request to the log database and send the generated log entry data to the log database to complete the writing of the log entry data.

[0088] In the solution provided in this embodiment, since the log processing terminal in the object storage system can receive the raw information generated by the proxy cluster based on the received data requests, and generate log entry data based on the raw information, and then write the log entry data to the log database, the proxy cluster can avoid generating log entry data for the received data requests, thereby reducing the resource consumption of the proxy cluster. Therefore, this solution can avoid performance degradation of the proxy cluster.

[0089] Optionally, in another embodiment of the present invention, after writing the directory entry data into the log database, the log processing terminal can also receive various operation requests for the log entry data already stored in the log database, and perform corresponding response operations on the log entry data already stored in the log database based on the received operation requests, so as to enable users to manage the log entry data.

[0090] For example, the above operation request can be a statistics request, in which case, based on Figure 1 Examples, such as Figure 2 As shown, another embodiment of the data writing method provided by the present invention may further include, after step S103:

[0091] S104, Receive a statistical request for the stored log entries in the log database, wherein the statistical request carries a user identifier, statistical fields and filtering rules;

[0092] For example, the statistics request received by the log processing terminal may be sent by the proxy cluster. In one implementation, the proxy cluster receives the statistics request from the user and forwards the received statistics request to the log processing terminal after receiving it.

[0093] For example, the statistical requests received by the log processing terminal can also be sent by any electronic device that communicates with it. For instance, the statistical request can be a request sent directly from the client device to the log processing terminal through the log processing terminal's interface, or a request sent by other servers that manage log data.

[0094] The statistics request received by the log processing end carries a user identifier, statistical fields, and filtering rules. The user identifier can be a user account, username, or other information that identifies the user. The statistical fields are the fields to be statistically analyzed. The filtering rules are used to filter log entries from the log data to identify those relevant to the statistics request.

[0095] For example, if a statistics request needs to count the number of write requests received from user A between January 1, 2020 and January 2, 2020, then the user identifier can be user A, the statistics field can be write type, and the filtering rule can be: filter out the log entries corresponding to data requests from January 1, 2020 to January 2, 2020.

[0096] For another example, if based on billing requirements, it is necessary to count the amount of billed traffic used by user A from January 1, 2020 to January 2, 2020, then the user identifier can be user A, the statistical field can be the billed traffic of the requested data, and the filtering rule is: filter out the log entries corresponding to data requests of the request type of write request from January 1, 2020 to January 2, 2020.

[0097] Optionally, the above filtering rules can also filter field data corresponding to abnormal data requests for users, allowing statistics to be performed only on normal data requests.

[0098] S105, According to the filtering rules, filter the log entry data corresponding to the data request from the user represented by the user identifier in the log database;

[0099] Optionally, log entries corresponding to data requests from the user represented by the user identifier can be determined from the stored log entry data based on the user identifier. The determined log entry data can then be further filtered according to filtering rules.

[0100] For example, as shown in Table 1, there are 6 log entries. Each row represents a log entry for a data request. Taking number 1 as an example, the object storage system received a write request from user A on January 1, 2020, to store a 5MB file named "2" in the object storage system.

[0101] Table 2

[0102]

[0103]

[0104] The log processing terminal receives statistics request 1, which requests statistics on the amount of write data received from user A during the period from January 1st to January 2nd, 2020. Statistics request 1 carries the user identifier of user A and the filtering rule: filter log entries corresponding to data requests of type write type received during the period from January 1st to January 2nd, 2020. Based on user A, log entries numbered 1, 2, and 3 are identified. Then, according to the filtering rule, log entry number 1 is selected.

[0105] S106, Statistically analyze the field data that belongs to the statistical field in the filtered log entries;

[0106] After filtering out the log entries, you can count the data in the fields that belong to the statistical fields.

[0107] For example, in the example above, if the statistical field in statistical request 1 is data size, then the data in the field belonging to data size in the filtered log entry data is determined to be 5M.

[0108] S107, based on the statistical field data, generates a response result for the statistical request.

[0109] Optionally, the above response results can include various information such as frequency, number of times, magnitude, and duration. The determined response results will vary depending on different statistical requirements.

[0110] For example, in the example above, the data to be counted is the size of the written data. Therefore, the filtered field value of 5M represents the size of the written data to be counted. Optionally, if multiple fields are selected, such as field 1 having a value of 5M, field 2 having a value of 6M, and field 3 having a value of 10M, then the selected results can be summed: 5M + 6M + 10M = 11M, which will be the response result.

[0111] To give another example, if the goal is to statistically analyze the change in the amount of data stored by users in the object storage system within a preset time period, we can filter out the data sizes corresponding to write data requests and delete data requests within that time period, and then perform the calculation. For instance, if the write data sizes are 5MB and 6MB respectively, and the delete data size is 4MB, then the result would be 5MB + 6MB - 4MB = 7MB, which would be the response.

[0112] To give another example, if the goal is to count the number of times a user sends a write request within a preset time period, then the log entries corresponding to each write request within the preset time period can be filtered out, and the number of the filtered log entries can be counted as the response result.

[0113] In the solution provided in this embodiment, since the log processing terminal in the object storage system can receive the raw information generated by the proxy cluster based on the received data requests, and generate log entry data based on the raw information, and then write the log entry data to the log database, the proxy cluster can avoid generating log entry data for the received data requests, thereby reducing the resource consumption of the proxy cluster. Therefore, this solution can avoid performance degradation of the proxy cluster.

[0114] In addition, the log processing terminal can perform statistics on the log entries stored in the log database based on the received statistical requests, thereby further enabling users to manage the log entry data.

[0115] For example, the above operation request can be an index request, in which case, based on Figure 1 Examples, such as Figure 3 As shown, another embodiment of the data writing method provided by the present invention may further include, after step S103:

[0116] S108, receive an index request for stored log entry data in the log database, wherein the index request carries an index identifier;

[0117] In one implementation, the index request received by the log processing terminal can be sent by the proxy cluster. The proxy cluster receives the index request from the user and forwards it to the log processing terminal upon receipt. Of course, the index request received by the log processing terminal can also be sent by any electronic device communicating with it. For example, the index request can be a request sent directly to the log processing terminal by a client device through its interface, or a request sent by another server used to manage log data.

[0118] Optionally, in one implementation of the present invention, the index request received by the log processing terminal can be a full-text index request. The full-text index request can enable indexing of the full-text content of the log directory data, and its index identifier can be field content, supporting the querying of all field content in the log directory data.

[0119] For example, as shown in Table 3:

[0120] Table 3

[0121] serial number Request time Data Identifier Data size Request duration 1 January 1, 2020 Document 1 5M 0.1s 2 January 2, 2020 Document 1 5M 0.5s 3 January 3, 2020 Document 2 5M 0.4s

[0122] In the table above, blank cells indicate that the log processing end did not select the field content of that field from the original information. Therefore, the index identifier for the full-text index request can include any field content from Table 3, such as file1 or file1+0.5s.

[0123] Optionally, in another implementation of the present invention, the index request received by the log processing terminal can be a segment index request. The segment index request can specify a field identifier, which can be the field content. For example, when the field identifier specified in the segment index request is a data identifier, the index identifier can be file 2.

[0124] Optionally, the field type identified by the above index can be set as needed, such as including text type, JSON (JavaScript Object Notation) type, Double type, etc. Optionally, all the above requests can be implemented based on SQL (Structured Query Language) query syntax or API (Application Programming Interface) call commands.

[0125] S109, Query the log entry data represented by the index identifier from the log database;

[0126] This allows querying log entry data represented by an index identifier from the log entry data already stored in the log database.

[0127] For example, in the example in Table 3, if the index request is a full-text index request, and the index identifier is file 1, then log entry data numbered 1 and 2 will be retrieved. If the index identifier is file 1 + 0.5s, then log entry data numbered 2 will be retrieved. If the index request is a segment index request, the field identifier indicated by the segment index request is the data size. When the index identifier is 5M, then log entry data numbered 1, 2, and 3 will be retrieved. The field identifier indicated by the segment index request is the data identifier; if the index identifier is file 2, then log entry data numbered 3 will be retrieved.

[0128] S110, based on the queried log entry data, generates a response result for the index request.

[0129] The response can be either the queried log entries or sorted according to specified criteria.

[0130] In the solution provided in this embodiment, since the log processing terminal in the object storage system can receive the raw information generated by the proxy cluster based on the received data requests, and generate log entry data based on the raw information, and then write the log entry data to the log database, the proxy cluster can avoid generating log entry data for the received data requests, thereby reducing the resource consumption of the proxy cluster. Therefore, this solution can avoid performance degradation of the proxy cluster.

[0131] In addition, the log processing end can index the log entry data already stored in the log database based on the received index request, thereby further enabling users to manage the log entry data.

[0132] based on Figure 1 Examples, such as Figure 4 As shown, another embodiment of the present invention provides a data writing method, wherein step S103 includes:

[0133] S103A, write the log entry data corresponding to each data request to at least one log file, wherein each log file is used to write log entry data that meets the predetermined summary conditions, the predetermined summary conditions include: the generation time belongs to the same time granularity, or the receiving time of the corresponding data request belongs to the same time granularity.

[0134] At least one log file can be at least one CSV (Comma-Separated Values) file. The time granularity can be configured as needed. For example, the time granularity can be configured to 1 hour or 1 minute. It should be noted that, generally speaking, the longer the time granularity, the larger the generated log file and the longer the log entry data is stored. Conversely, the smaller the granularity, the smaller the generated log file and the shorter the log entry data is stored.

[0135] Optionally, in one implementation, the generation time of log entry data can be used as the basis for writing to the log file. For example, if the time granularity is 1 hour, log entry data generated between 11:00 AM and 12:00 PM on January 1, 2020, will be written to the same log file. After 12:00 PM on January 1, 2020, a new log file will be generated to write log entry data generated later.

[0136] Alternatively, in another implementation, the receipt time of the data request corresponding to the log entry data can be used as the basis for writing to the log file. Since there is a time difference between receiving a data request and generating the log entry data, and to facilitate management of log entry data based on request receipt time, the receipt time of the data request corresponding to the log entry data can be used as the basis for writing to the log file. For example, if the time granularity is 1 hour, then the log entry data corresponding to data requests received between 11:00 AM and 12:00 PM on January 1, 2020, will be written to the same log file.

[0137] Optionally, in one implementation, log entry data that meets predetermined aggregation conditions can be cached first, and then the cached log entry data can be written to the log file simultaneously. Optionally, in another implementation, a log entry can be generated for each received data request, that is, the log entry data can be written to the corresponding log file, until a time granularity is filled, and then the next log entry data is written to the log file of the next time granularity.

[0138] S103B, In the log database, at least one log file is stored.

[0139] This involves sending the log file to the log database so that the log database can store the log file.

[0140] In the solution provided in this embodiment, since the log processing terminal in the object storage system can receive the raw information generated by the proxy cluster based on the received data requests, and generate log entry data based on the raw information, and then write the log entry data to the log database, the proxy cluster can avoid generating log entry data for the received data requests, thereby reducing the resource consumption of the proxy cluster. Therefore, this solution can avoid performance degradation of the proxy cluster.

[0141] based on Figure 4 Examples, such as Figure 5 As shown, another embodiment of the data writing method provided by the present invention further includes, before step S103B:

[0142] S111, correct the data of each log entry in the log file to obtain the corrected log file.

[0143] The aforementioned correction process includes: if the field content of the target field data in the log entry data is different from the actual content corresponding to the target field data, modifying the field content of the target field data to the storage content corresponding to the target field data; wherein, the actual content corresponding to the target field data is: the storage content of the data targeted by the data request corresponding to the log entry data to which the target field data belongs in the object storage system.

[0144] It should be noted that in object storage systems, the data identifiers of the stored data may need to be rewritten to conform to object storage rules. Therefore, to avoid discrepancies between the field content in the stored log entries and the actual content stored in object storage, the fields need to be corrected.

[0145] For example, as shown in Table 4:

[0146] Table 4

[0147] serial number User ID Request time Data Identifier Data size 1 User A January 1, 2020 Document 1 5M 2 User B January 2, 2020 Document 2 3M

[0148] Log entries numbered 1 and 2 are generated by the log processing end. For log entry data 1 (numbered 1), the field content corresponding to the data identifier is "File 1". The content of "File 1" for data request 1 corresponding to log entry data 1 in the object storage system is: Data Identifier: File 1a. Since the field content of the data identifier in log entry data 1 (File 1) differs from the actual content of the field data (File 1a), the field content corresponding to the data identifier of log entry data 1 (numbered 1) needs to be corrected to "File 1a". For log entry data 2 (numbered 2), the field content corresponding to the data identifier is "File 2". The content of "File 2" for data request 2 corresponding to log entry data 2 in the object storage system is: Data Identifier: File 2. Since the field content of the data identifier in log entry data 2 (File 2) is the same as the actual content of the field data (File 2), no correction is needed. The final corrected log entry data is shown in Table 5.

[0149] Table 5

[0150] serial number User ID Request time Data Identifier Data size 1 User A January 1, 2020 Document 1a 5M 2 User B January 2, 2020 Document 2 3M

[0151] Optionally, in one implementation, step S111 may include:

[0152] By using at least one running instance, the data of each log entry in the log file is corrected to obtain the corrected log file.

[0153] The number of running instances can be determined based on their memory and CPU (Central Processing Unit) consumption. At least one running instance can correct the log entries in the log file according to the correction process described above, resulting in a corrected log file.

[0154] Optionally, at least one running instance may be multiple, with different running instances corresponding to different entry position ranges. In this case, step S111 may include:

[0155] By modifying the log entries in the log file corresponding to the specified entry range in each of the multiple running instances, a corrected log file is obtained.

[0156] Optionally, the number of running instances can be multiple. However, to avoid resource waste, the maximum number of running instances is the same as the number of entry position ranges divided for the log entry data in the log file.

[0157] For example, a log file contains 100 log entries, divided into five entry ranges: Range 1 (lines 1-20), Range 2 (lines 21-40), Range 3 (lines 41-60), Range 4 (lines 61-80), and Range 5 (lines 81-100).

[0158] A maximum of five running instances can be started: Instance 1, Instance 2, Instance 3, Instance 4, and Instance 5. Instance 1 corresponds to log entries 1-20; Instance 2 to lines 21-40; Instance 3 to lines 41-60; Instance 4 to lines 61-80; and Instance 5 to lines 81-100.

[0159] If the number of running instances is less than the number of entry position ranges, then after each running instance has processed the log entry data for its corresponding entry position range, it will sequentially obtain the log entry data for the next entry position range that has not yet been processed.

[0160] Optionally, in one embodiment of the present invention, at least one scheduler may also record the position of the log entry data it is processing while processing the log entry data, so as to ensure that the processing of the log entry data can continue from that position after an unexpected stop.

[0161] In the solution provided in this embodiment, since the log processing terminal in the object storage system can receive the raw information generated by the proxy cluster based on the received data requests, and generate log entry data based on the raw information, and then write the log entry data to the log database, the proxy cluster can avoid generating log entry data for the received data requests, thereby reducing the resource consumption of the proxy cluster. Therefore, this solution can avoid performance degradation of the proxy cluster.

[0162] To more clearly demonstrate the data writing method provided in the embodiments of the present invention, such as Figure 6 The diagram shows a schematic of an object storage system structure provided in an embodiment of the present invention. The data writing method provided in this embodiment is applied to... Figure 6The proxy cluster is the log processing end in the object storage system. It is responsible for receiving / distributing data requests from clients and generating raw information for generating log entry data based on the received data requests. The generated raw information is then sent to the log processing end, which generates and stores log entry data based on the raw information. The proxy cluster does not need to generate or store log entry data.

[0163] In the solution provided in this embodiment, since the log processing terminal in the object storage system can receive the raw information generated by the proxy cluster based on the received data requests, and generate log entry data based on the raw information, and then write the log entry data to the log database, the proxy cluster can avoid generating log entry data for the received data requests, thereby reducing the resource consumption of the proxy cluster. Therefore, this solution can avoid performance degradation of the proxy cluster.

[0164] Optional, such as Figure 7 This is a schematic diagram illustrating the architecture of a data writing device provided in one embodiment of the present invention. The data source acquisition module, data processing module, information storage module, and query and analysis module are all included in the log processing terminal of the object storage system.

[0165] The aforementioned data source acquisition module is used to continuously receive the original information corresponding to each data request sent by the proxy cluster, namely... Figure 7 The source Logstore (log storage) in the database; and based on the original information corresponding to the data request, generate the log entry data corresponding to the data request;

[0166] The aforementioned data processing module includes a scheduler, consumer groups, and dimension tables. For each preset processing rule, the scheduler launches one or more running instances. These processing rules are used to correct log entry data. Each running instance acts as a consumer in the consumer group, streaming log entry data from the data source acquisition module. Specifically, each consumer streams log entry data from the data source acquisition module according to a preset time granularity. Figure 7 The diagram shows shards, where each shard represents a log entry read by the consumer from the data source acquisition module according to a preset time granularity. Simultaneously, the consumer processes the read log entries according to processing rules and generates corresponding log files. The dimension table in the data processing module records the consumption position of the shard, ensuring that log entry reading can resume from the breakpoint after an unexpected stop.

[0167] The aforementioned information storage module is used to persist the log files generated by the data processing module to disk, enabling persistent storage and sharing between systems. The data storage utilizes object storage, allowing for massive, infinitely scalable storage and handling high-concurrency access scenarios.

[0168] The aforementioned query and analysis module is used to receive statistical requests for stored log entries in the log database, wherein the statistical request carries a user identifier, statistical fields, and filtering rules; according to the filtering rules, it filters the log entry data in the log database corresponding to the data requests from the user represented by the user identifier; it statistically analyzes the field data belonging to the statistical fields in the filtered log entry data; and based on the statistical field data, it generates a response result for the statistical request. It is also used to receive index requests for stored log entries in the log database, wherein the index request carries an index identifier; it queries the log entry data represented by the index identifier from the log database; and based on the queried log entry data, it generates a response result for the index request.

[0169] Corresponding to the methods provided above, such as Figure 8 As shown, this embodiment of the invention also provides a data writing device applied to a log processing end in an object storage system. The object storage system further includes a proxy cluster, and the log processing end communicates with the proxy cluster. The device includes:

[0170] The information receiving module 801 is used to continuously receive the original information corresponding to each data request sent by the agent cluster. The original information corresponding to each data request is: the information generated by the agent cluster based on the received data request, which is used to generate log entry data.

[0171] The data generation module 802 is used to generate log entry data corresponding to each data request based on the original information corresponding to the data request. Each log entry data includes various field data, and the field identifier of each field data is a preset field identifier.

[0172] The data writing module 803 is used to write the log entry data corresponding to each data request into the log database.

[0173] Optionally, the data writing module is specifically used to write the log entry data corresponding to each data request to at least one log file, wherein each log file is used to write log entry data that meets predetermined summarization conditions, the predetermined summarization conditions including: the generation time belongs to the same time granularity, or the receiving time of the corresponding data request belongs to the same time granularity; at least one log file is stored in the log database.

[0174] Optionally, the data writing module is further configured to correct the log entry data in the log file before storing at least one log file in the log database, thereby obtaining a corrected log file. The correction process includes: if the field content of the target field data in the log entry data is different from the actual content corresponding to the target field data, modifying the field content of the target field data to the storage content corresponding to the target field data. The actual content corresponding to the target field data is the storage content in the object storage system of the data requested by the log entry data to which the target field data belongs.

[0175] Optionally, a data writing module is used to correct the data of each log entry in the log file through at least one running instance to obtain a corrected log file.

[0176] Optionally, at least one running instance may be multiple, with different running instances corresponding to different entry location ranges;

[0177] The data writing module is specifically used to correct the log entry data in the log file corresponding to the entry position range of the running instance through each of the multiple running instances, so as to obtain the corrected log file.

[0178] Optionally, the device also includes:

[0179] The statistics module receives statistical requests for stored log entries in the log database. The statistical request carries a user identifier, statistical fields, and filtering rules. According to the filtering rules, it filters the log entry data in the log database corresponding to the data request from the user represented by the user identifier. It then counts the field data belonging to the statistical fields in the filtered log entry data. Based on the counted field data, it generates a response result for the statistical request.

[0180] Optionally, the device also includes:

[0181] The indexing module is used to receive index requests for stored log entries in the log database, wherein the index request carries an index identifier; query the log entry data represented by the index identifier from the log database; and generate a response result for the index request based on the queried log entry data.

[0182] This invention also provides an electronic device, such as... Figure 9 As shown, it includes a processor 901, a communication interface 902, a memory 903, and a communication bus 904, wherein the processor 901, the communication interface 902, and the memory 903 communicate with each other through the communication bus 904.

[0183] Memory 903 is used to store computer programs;

[0184] The processor 901, when executing the program stored in the memory 903, implements the method steps provided above.

[0185] The communication bus mentioned in the above electronic devices can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. This communication bus can be divided into address bus, data bus, control bus, etc. For ease of illustration, only one thick line is used to represent it in the diagram, but this does not indicate that there is only one bus or one type of bus.

[0186] The communication interface is used for communication between the aforementioned electronic devices and other devices.

[0187] The memory may include random access memory (RAM) or non-volatile memory (NVM), such as at least one disk storage device. Optionally, the memory may also be at least one storage device located remotely from the aforementioned processor.

[0188] The processors mentioned above can be general-purpose processors, including central processing units (CPUs), network processors (NPs), etc.; they can also be digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components.

[0189] In another embodiment of the present invention, a computer-readable storage medium is also provided, which stores a computer program that, when executed by a processor, implements the steps of any of the above-described data writing methods.

[0190] In another embodiment of the present invention, a computer program product containing instructions is also provided, which, when run on a computer, causes the computer to execute any of the data writing methods described above.

[0191] In the above embodiments, implementation can be achieved entirely or partially through software, hardware, firmware, or any combination thereof. When implemented using software, it can be implemented entirely or partially in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the processes or functions described in the embodiments of the present invention are generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wired (e.g., coaxial cable, fiber optic, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that integrates one or more available media. The available medium can be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., DVD), or a semiconductor medium (e.g., solid state disk (SSD)).

[0192] It should be noted that, in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0193] The various embodiments in this specification are described in a related manner. Similar or identical parts between embodiments can be referred to mutually. Each embodiment focuses on describing the differences from other embodiments. In particular, the device and apparatus embodiments are basically similar to the method embodiments, so the descriptions are relatively simple; relevant parts can be referred to the descriptions of the method embodiments.

[0194] The above description is merely a preferred embodiment of the present invention and is not intended to limit the scope of protection of the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention are included within the scope of protection of the present invention.

Claims

1. A data writing method, characterized in that, A log processing endpoint applied in an object storage system, wherein the object storage system further includes a proxy cluster, the log processing endpoint communicating with the proxy cluster, and the log processing endpoint being a device with data processing capabilities; the method includes: The system continuously receives the original information corresponding to each data request sent by the proxy cluster. The original information corresponding to each data request is: information generated by the proxy cluster based on the received data request for generating log entry data, and the content to be included in the original information is determined based on actual needs. For each data request, based on the original information corresponding to the data request, log entry data corresponding to the data request is generated. Each log entry data includes various field data, and the field identifier of each field data is a preset field identifier. The method for generating log entry data corresponding to any data request is as follows: according to each preset field identifier, field data with the field identifier set to the preset field identifier is selected from the original information corresponding to the data request. Log entries corresponding to each data request are written to at least one log file. Each log file is used to write log entries that meet predetermined aggregation conditions. The predetermined aggregation conditions include: the generation time belongs to the same time granularity, or the receiving time of the corresponding data request belongs to the same time granularity. The log database stores the at least one log file; The method further includes: The system receives a statistical request from the proxy cluster for stored log entries in the log database. The statistical request carries a user identifier, statistical fields, and filtering rules. The statistical request is forwarded to the log processing terminal by the proxy cluster after receiving a statistical request from a user. According to the filtering rules, the log entries corresponding to the data requests from the user represented by the user identifier in the log database are filtered. The field data belonging to the statistical field in the filtered log entries; Based on the statistical field data, a response result is generated for the statistical request.

2. The method according to claim 1, characterized in that, Before storing the at least one log file in the log database, the method further includes: The log entries in the log file are corrected to obtain the corrected log file. The correction process includes: If the field content of the target field data in the log entry data is different from the actual content corresponding to the target field data, the field content of the target field data shall be modified to the storage content corresponding to the target field data. The actual content corresponding to the target field data is the storage content of the data requested by the log entry data to which the target field data belongs in the object storage system.

3. The method according to claim 2, characterized in that, The step of correcting the data of each log entry in the log file to obtain a corrected log file includes: By using at least one running instance, the data of each log entry in the log file is corrected to obtain a corrected log file.

4. The method according to claim 3, characterized in that, The number of the at least one running instance is multiple, and different running instances correspond to different entry position ranges; The step of correcting the log entries in the log file using at least one running instance to obtain a corrected log file includes: By using each of the multiple running instances, the log entry data in the log file corresponding to the entry position range of that running instance is corrected to obtain the corrected log file.

5. The method according to any one of claims 1-4, characterized in that, The method further includes: Receive an index request for stored log entry data in the log database, wherein the index request carries an index identifier; From the log database, query the log entry data represented by the index identifier; Based on the queried log entry data, a response result is generated for the index request.

6. A data writing device, characterized in that, A log processing terminal applied in an object storage system, wherein the object storage system further includes a proxy cluster, the log processing terminal communicates with the proxy cluster, and the log processing terminal is a device with data processing capabilities; the device includes: The information receiving module is used to continuously receive the original information corresponding to each data request sent by the agent cluster. The original information corresponding to each data request is: the information generated by the agent cluster based on the received data request, which is used to generate log entry data. The data generation module is used to generate log entry data corresponding to each data request based on the original information corresponding to the data request. Each log entry data includes various field data, and the field identifier of each field data is a preset field identifier. The method for generating log entry data corresponding to any data request is as follows: according to each preset field identifier, select the field data with the field identifier of the preset field identifier from the original information corresponding to the data request. The data writing module is used to write the log entry data corresponding to each data request to at least one log file. Each log file is used to write log entry data that meets predetermined summarization conditions. The predetermined summarization conditions include: the generation time belongs to the same time granularity, or the receiving time of the corresponding data request belongs to the same time granularity. The log database stores the at least one log file; The device further includes: The statistics module is used to receive statistics requests sent by the proxy cluster for stored log entries in the log database. The statistics request carries a user identifier, statistics fields, and filtering rules. The statistics request is forwarded to the log processing terminal by the proxy cluster after receiving the statistics request from the user. According to the filtering rules, the log entries corresponding to the data requests from the user represented by the user identifier in the log database are filtered. The field data belonging to the statistical field in the filtered log entries; Based on the statistical field data, a response result is generated for the statistical request.

7. An electronic device, characterized in that, It includes a processor, a communication interface, a memory, and a communication bus, wherein the processor, the communication interface, and the memory communicate with each other through the communication bus; Memory, used to store computer programs; A processor, when executing a program stored in memory, implements the steps of the method described in any one of claims 1-5.

8. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, implements the steps of the method described in any one of claims 1-5.

Images