Data statistical method, device, equipment and storage medium

Abstract

The application discloses a data statistics method and device, equipment and storage medium, relates to the technical field of data processing, and is used for improving the efficiency of data statistics. Including: obtaining log information generated when accessing the storage cluster each time, and determining first information corresponding to each file in a plurality of files based on the log information, the first information including at least one of the following: data volume of the file, access type; send a plurality of first information to a kafka cluster, determine the second information corresponding to each storage space in a plurality of storage spaces included in the storage cluster through a plurality of first information, the second information including at least one of the following: memory change of the storage space, file quantity included in the storage space; based on the second information corresponding to each storage space in the plurality of storage spaces included in the storage cluster, determine the third information corresponding to the storage cluster, the third information is used for indicating at least one of the following: memory usage of the storage cluster, total number of files included in the storage cluster.

Description

Data statistical methods, apparatus, equipment and storage media Technical Field

[0001] This application relates to the field of data processing technology, and in particular to a data statistics method, apparatus, device and storage medium. Background Technology

[0002] With the continuous development of information technology, the volume of information and data is increasing, and data storage methods are becoming more and more diversified. Among related technologies, distributed object storage clusters can be used to store data. In a distributed object storage cluster, the number of stored objects and the usage of storage space are key data that need to be statistically analyzed in a timely manner during data storage. Currently, the main method for measuring distributed object storage clusters is to periodically scan each bucket in the background to determine the number of stored objects and the usage of storage space.

[0003] In the methods described above, when the data volume is large (e.g., the number of stored objects in the storage space exceeds 1 billion), the metadata volume is too large, each scan consumes a significant amount of computing resources, and it takes a long time to obtain the scan results. This makes it impossible to promptly determine the actual number of stored objects and the usage of storage space, and it also affects the operation of normal business processes. Therefore, the current method for statistically analyzing files stored in the storage space is inefficient. Summary of the Invention

[0004] This application provides a data statistics method, apparatus, device, and storage medium for improving the efficiency of data statistics.

[0005] To achieve the above objectives, this application adopts the following technical solution:

[0006] Firstly, a data statistics method is provided, comprising: acquiring log information generated each time a storage cluster is accessed, and determining first information corresponding to each file in multiple files based on the log information, wherein the storage cluster includes multiple storage spaces, the multiple files are the files corresponding to the access to the storage cluster, one file in the multiple files corresponds to one piece of first information, one piece of first information corresponds to one storage space in the multiple storage spaces, and one piece of first information includes at least one of the following: the data volume of the file and the access type; sending the multiple pieces of first information to a Kafka cluster, and determining second information corresponding to each storage space in the multiple storage spaces included in the storage cluster through the multiple pieces of first information, the second information including at least one of the following: the memory change of the storage space and the number of files included in the storage space; and determining third information corresponding to the storage cluster based on the second information corresponding to each storage space in the multiple storage spaces included in the storage cluster, the third information indicating at least one of the following: the memory usage of the storage cluster and the total number of files included in the storage cluster.

[0007] In one design, a Kafka cluster includes multiple Kafka partitions, and one of the Kafka partitions corresponds to one of the multiple storage spaces. Sending multiple pieces of first information to the Kafka cluster includes: for any one piece of first information, determining the target storage space corresponding to any one piece of first information; and sending any one piece of first information to the target Kafka partition corresponding to the target storage space in the Kafka cluster.

[0008] In one design, second information corresponding to each of the multiple storage spaces included in the storage cluster is determined by multiple first pieces of information, including: for a target Kafka partition among multiple Kafka partitions, determining the second information corresponding to the target storage space based on at least one first piece of information corresponding to the target Kafka partition; sending the second information corresponding to each of the multiple storage spaces to the management Kafka, and the management Kafka corresponding to the storage cluster.

[0009] In one design, based on the second information corresponding to each of the multiple storage spaces included in the storage cluster, the third information corresponding to the storage cluster is determined, including: summarizing the second information corresponding to each of the multiple storage spaces included in the management of Kafka to obtain the third information corresponding to the storage cluster.

[0010] Secondly, a data statistics device is provided, comprising: an acquisition unit, a determination unit, and a sending unit; the acquisition unit is used to acquire log information generated each time a storage cluster is accessed; the determination unit is used to determine first information corresponding to each file in multiple files based on the log information, the storage cluster including multiple storage spaces, the multiple files being the files corresponding to the access to the storage cluster, one file in the multiple files corresponding to one piece of first information, one piece of first information corresponding to one storage space in the multiple storage spaces, and one piece of first information including at least one of the following: the data volume of the file, access type; the sending unit is used to send the multiple pieces of first information to a Kafka cluster; the determination unit is used to determine second information corresponding to each storage space in the multiple storage spaces included in the storage cluster through the multiple pieces of first information, the second information including at least one of the following: the memory change of the storage space, the number of files included in the storage space; the determination unit is used to determine third information corresponding to the storage cluster based on the second information corresponding to each storage space in the multiple storage spaces included in the storage cluster, the third information indicating at least one of the following: the memory usage of the storage cluster, the total number of files included in the storage cluster.

[0011] In one design, a Kafka cluster includes multiple Kafka partitions, and one of the Kafka partitions corresponds to one of the multiple storage spaces; a determining unit is used to determine the target storage space corresponding to any one of the multiple first pieces of information; and a sending unit is used to send any one of the first pieces of information to the target Kafka partition corresponding to the target storage space in the Kafka cluster.

[0012] In one design, a determining unit is used to determine, for a target Kafka partition among multiple Kafka partitions, second information corresponding to a target storage space based on at least one first piece of information corresponding to the target Kafka partition; and a sending unit is used to send the second information corresponding to each of the multiple storage spaces to the management Kafka, and the management Kafka corresponds to the storage cluster.

[0013] In one design, a unit is defined to summarize the second information corresponding to each of the multiple storage spaces included in the management of Kafka, in order to obtain the third information corresponding to the storage cluster.

[0014] Thirdly, an electronic device is provided, comprising: a processor and a memory; wherein the memory is used to store one or more programs, the one or more programs including computer-executable instructions, and when the electronic device is running, the processor executes the computer-executable instructions stored in the memory to cause the electronic device to perform a data statistics method as described in the first aspect.

[0015] Fourthly, a computer-readable storage medium is provided for storing one or more programs, the one or more programs including instructions that, when executed by a computer, cause the computer to perform a data statistics method as described in the first aspect.

[0016] This application provides a data statistics method, apparatus, device, and storage medium applied to scenarios involving determining the number of files in a storage cluster. First, log information generated during each access to the storage cluster, which includes multiple storage spaces, is acquired. Based on this log information, first information, including the file's data size and access type, is determined for each of the multiple files accessed during the storage cluster access. This first information is then sent to a Kafka cluster to determine second information, including the storage space's memory usage change and the number of files within that storage space, for each of the multiple storage spaces in the storage cluster. Furthermore, based on the second information for each storage space in the storage cluster, third information indicating the storage cluster's memory usage and the total number of files within the storage cluster is determined. This method enables real-time statistics of relevant data in the storage cluster, improving the efficiency of data statistics, by determining the first information based on the log information generated during each access to the storage cluster and summarizing this first information to obtain the third information indicating the storage cluster's memory usage and the number of files within the storage cluster. Attached Figure Description

[0017] Figure 1 is a schematic diagram of a data statistics system structure provided in an embodiment of this application;

[0018] Figure 2 is a schematic flowchart of a data statistics method provided in an embodiment of this application;

[0019] Figure 3 is a schematic diagram of a data statistics method provided in an embodiment of this application;

[0020] Figure 4 is a schematic flowchart of a data statistics method provided in an embodiment of this application;

[0021] Figure 5 is a schematic flowchart of a data statistics method provided in an embodiment of this application;

[0022] Figure 6 is a schematic flowchart of a data statistics method provided in an embodiment of this application;

[0023] Figure 7 is a schematic diagram of a data statistics device structure provided in an embodiment of this application;

[0024] Figure 8 is a schematic diagram of an electronic device structure provided in an embodiment of this application. Detailed Implementation

[0025] The technical solutions in the embodiments of this application will now be described with reference to the accompanying drawings.

[0026] In the description of this application, unless otherwise stated, " / " means "or". For example, A / B can mean A or B. "And / or" in this document is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A alone, A and B simultaneously, and B alone. Furthermore, "at least one" and "multiple" refer to two or more. The terms "first," "second," etc., do not limit the quantity or order of execution, and "first," "second," etc., do not necessarily imply differences.

[0027] This application provides a data statistics method applicable to data statistics systems. Figure 1 shows a schematic diagram of one structure of the data statistics system. As shown in Figure 1, the data statistics system 20 includes: a server 21 and an electronic device 22. The server 21 is connected to the electronic device 22.

[0028] The data statistics system 20 can be used for the Internet of Things. The data statistics system 20 may include hardware such as multiple central processing units (CPUs), multiple memories, and storage devices storing multiple operating systems.

[0029] Server 21 can be a server corresponding to the storage cluster, used to store data. For example, server 21 can store log information generated when a user accesses the storage cluster and send the log information to electronic device 22 so that electronic device 22 can process the data.

[0030] Electronic device 22 can be used in the Internet of Things to process data. For example, electronic device 22 can interact with the second base station server 21 to obtain log information generated each time it accesses the storage cluster, and then determine the third information corresponding to the storage cluster through the log information.

[0031] The following description, in conjunction with the accompanying drawings, describes a data statistics method provided by an embodiment of this application.

[0032] As shown in Figure 2, an embodiment of this application provides a data statistics method, including steps S201-S203:

[0033] S201. Obtain the log information generated each time the storage cluster is accessed, and determine the first information corresponding to each file in multiple files based on the log information.

[0034] The storage cluster includes multiple storage spaces, and multiple files are the files corresponding to accessing the storage cluster. One file in the multiple files corresponds to a first piece of information, and one first piece of information corresponds to one storage space in the multiple storage spaces. One first piece of information includes at least one of the following: the data volume of the file and the access type.

[0035] Optionally, the first information may also include information such as call access type, uplink traffic, and downlink traffic.

[0036] It should be noted that the access type refers to the type of access, such as file upload, file download, file modification, etc.; uplink traffic refers to the traffic information of uploaded files, and downlink traffic refers to the traffic information of downloaded files.

[0037] Optionally, in the storage service, log files containing log information can be generated based on information such as the application program interface (API) access information, lifecycle deletion information, and storage space access log (bucket logging) information corresponding to each access to the storage cluster.

[0038] It should be noted that a log file corresponds to an action taken by a user in the storage cluster.

[0039] For example, when a user uploads a file to the storage cluster, log information corresponding to that file upload is generated, resulting in a log file.

[0040] It should be noted that when a user processes multiple files in batches (such as batch uploading files, batch deleting files, etc.), multiple log files will be generated. Each log file corresponds to the action of one file (such as uploading a file, deleting a file, etc.), resulting in multiple pieces of first information.

[0041] Optionally, the first information can be obtained by parsing the relevant parameters in the log information.

[0042] S202. Send multiple first pieces of information to the Kafka cluster, and use the multiple first pieces of information to determine the second pieces of information corresponding to each storage space among the multiple storage spaces included in the storage cluster.

[0043] The second piece of information includes at least one of the following: the amount of memory change in the storage space, and the number of files included in the storage space.

[0044] Optionally, for any one of the multiple storage spaces included in the storage cluster, all first information corresponding to that storage space can be determined, and second information corresponding to that storage space can be determined based on all the first information corresponding to that storage space.

[0045] Optionally, the second information corresponding to any storage space can be obtained by statistically summarizing all the first information corresponding to any storage space.

[0046] S203. Based on the second information corresponding to each of the multiple storage spaces included in the storage cluster, determine the third information corresponding to the storage cluster.

[0047] The third piece of information is used to indicate at least one of the following: the memory usage of the storage cluster and the total number of files included in the storage cluster.

[0048] Optionally, the third information corresponding to the storage cluster can be obtained by statistically summarizing the second information corresponding to each of the multiple storage spaces included in the storage cluster.

[0049] Optionally, the third information may also include the number of times the user accesses the API, uplink traffic, and downlink traffic.

[0050] In this embodiment, log information generated each time the storage cluster, which includes multiple storage spaces, is accessed is first obtained. Based on the log information, first information, including the data volume and access type of each file in the multiple files corresponding to the access to the storage cluster, is determined. This first information is then sent to the Kafka cluster to determine second information, including the memory change and the number of files in each storage space within the storage cluster. Furthermore, based on the second information for each storage space, third information indicating the memory usage and the total number of files in the storage cluster is determined. This method allows for the determination of first information based on the log information generated each time the storage cluster is accessed, and by summarizing the first information, third information indicating the memory usage and the number of files in the storage cluster is obtained. This enables real-time statistics of relevant data in the storage cluster and improves the efficiency of data statistics.

[0051] In one design, the Kafka cluster includes multiple Kafka partitions, and one Kafka partition corresponds to one storage space among multiple storage spaces, as shown in Figure 3. In a data statistics method provided in this application embodiment, the "sending multiple first pieces of information to the Kafka cluster" in S202 above includes S301-S302:

[0052] S301. For any one of the multiple first pieces of information, determine the target storage space corresponding to any one of the first pieces of information.

[0053] It should be noted that in a distributed object storage cluster, there are multiple storage spaces (buckets). When a user accesses the storage cluster, a single access (to a single log file) involves only one storage space.

[0054] Optionally, the storage space corresponding to the log file can be determined by the identification information of the log file, thereby obtaining the target storage space corresponding to the first information of the log file.

[0055] Optionally, the number of Kafka partitions can be adjusted based on the amount of storage space in the storage cluster.

[0056] For example, when the storage cluster includes 10 storage spaces, it can be configured with 10 Kafka partitions.

[0057] S302. Send any first information to the target Kafka partition corresponding to the target storage space in the Kafka cluster.

[0058] Optionally, each Kafka partition is responsible for recording information about its corresponding storage space.

[0059] Optionally, a corresponding identifier can be set for each Kafka partition, and the storage space corresponding to the Kafka partition can be determined by the corresponding identifier of the Kafka partition.

[0060] Optionally, multiple first messages can be sent asynchronously to the Kafka partition corresponding to each of the multiple first messages, thereby achieving distributed message processing.

[0061] In this embodiment of the application, different Kafka partitions are used to perform statistics on the first information corresponding to different storage spaces in the storage cluster, and fine-grained distributed processing is achieved through asynchronous messages, thereby improving the efficiency of data statistics.

[0062] In one design, as shown in Figure 4, in a data statistics method provided by an embodiment of this application, the "determining the second information corresponding to each storage space among the multiple storage spaces included in the storage cluster through multiple first information" in S202 above includes S401-S402:

[0063] S401. For a target Kafka partition among multiple Kafka partitions, determine the second information corresponding to the target storage space based on at least one first piece of information corresponding to the target Kafka partition.

[0064] Optionally, based on the message type, the second information corresponding to the target storage space can be obtained by statistically analyzing the memory change of the target storage space and the number of files included in the target storage space according to the first information.

[0065] Optionally, statistics can also be compiled on the number of API accesses, uplink traffic, and downlink traffic corresponding to the target storage space.

[0066] Optionally, the change in memory usage can be understood as the change in the amount of storage space used.

[0067] Optionally, the first information corresponding to the target Kafka partition can be summarized at preset intervals (such as 10 seconds, 1 minute, etc.) to obtain the second information corresponding to the target storage space.

[0068] Optionally, the first information can be continuously summarized, that is, each time the first information is received, the summarized second information is updated based on the first information.

[0069] S402. Send the second information corresponding to each of the multiple storage spaces to the management Kafka.

[0070] This involves managing the corresponding Kafka storage cluster.

[0071] Optionally, you can manage Kafka, or aggregate Kafka, which is used to aggregate messages from multiple Kafka partitions.

[0072] Optionally, the second message can be sent to the management Kafka at preset intervals (such as 30 seconds, 5 minutes, etc.).

[0073] Optionally, when the number of first messages exceeds a preset threshold, the second message, which is now aggregated, is sent directly to the management Kafka.

[0074] For example, when the target Kafka partition records a total of 100 first information entries, the second information obtained by aggregating these first information entries is directly sent to the management Kafka.

[0075] Optionally, when a certain piece of information in the second information exceeds a preset threshold, the second information summarized at this time can be directly sent to the management Kafka.

[0076] For example, when the difference between the number of files included in the storage space in the second information and the number of files included in the storage space in the previous statistics is greater than 100, the second information summarized at this time can be sent to the management Kafka.

[0077] In this embodiment, by summarizing and statistically analyzing the first information in the Kafka partition, the file data information of multiple storage spaces can be determined through multiple second information corresponding to multiple Kafka partitions, thereby improving the accuracy of data statistics.

[0078] In one design, as shown in Figure 5, in a data statistics method provided by an embodiment of this application, the above-mentioned S203 includes S501:

[0079] S501. Summarize the second information corresponding to each of the multiple storage spaces included in the management of Kafka to obtain the third information corresponding to the storage cluster.

[0080] Optionally, the second information can be summarized to obtain the third information, and the third information can be written to the preset database at regular intervals (a certain period of time). The preset database is used to indicate the capacity information of the current storage cluster, such as the storage space usage, number of files, number of API accesses, uplink traffic, downlink traffic, etc.

[0081] Optionally, when the amount of second information exceeds a preset threshold, the third information summarized at this time is directly written into a preset database.

[0082] For example, when there are a total of 1,000 pieces of second information corresponding to all storage spaces in multiple storage spaces, the third information obtained by summarizing these second pieces of information is directly written to disk and then written to the database.

[0083] Optionally, the data can be written to the database, or the third-party information can be written to the disk corresponding to the preset database to prevent data loss due to restart or unexpected interruption.

[0084] Optionally, the second information can be summarized at preset time intervals, or it can be accumulated into the currently summarized third information each time the second information is received.

[0085] Optionally, when a certain piece of information in the third information exceeds a preset threshold, the third information summarized at this time can be directly written into a preset database.

[0086] For example, when the difference between the number of files included in the storage space in the third information and the number of files included in the storage space in the previous statistics is greater than 500, the third information summarized at this time can be written into the preset database.

[0087] It should be noted that, in the embodiments of this application, the specific values ​​of the period and the threshold can be determined according to the specific usage requirements, and the period and threshold used in different steps do not need to be consistent.

[0088] For example, the second message can be sent once every 1 minute, and the third message can be sent once every 2 minutes.

[0089] Optionally, in this embodiment of the application, the processing of information can be achieved through metering services.

[0090] For example, the metering service can determine the first information corresponding to the log file based on the log file, and the first information corresponding to multiple Kafka partitions can be consumed by multiple metering services to obtain the second information corresponding to multiple storage spaces. The third information in the aggregated Kafka can be consumed by the metering aggregation service and written to the preset database.

[0091] Optionally, the metering process can be made lock-free to enhance functionality, timeliness, and robustness.

[0092] In one design, when determining the second and third pieces of information, a corresponding consumption offset value can be recorded to indicate the current information consumption progress. This allows information consumption to resume directly from the recorded offset position after the metering service restarts, thus ensuring the accuracy of information consumption.

[0093] For example, as shown in Figure 6, in this embodiment of the application, the usage change, uplink and downlink traffic and other information of this call are first determined based on lifecycle call information, API call information and bucket logging call information to obtain first information, and the first information is sent to the Kafka cluster. The message is then sent asynchronously to multiple Kafka partitions to achieve distributed message processing.

[0094] For example, if there are 16 storage spaces in the storage cluster, then the Kafka cluster will have 16 Kafka partitions. The first message corresponding to each storage space will be sent to the corresponding Kafka partition. Further, the first message in each Kafka partition will be consumed to obtain incremental information (second message), which will be sent periodically, and the current consumption offset will be recorded.

[0095] The messages from the 16 Kafka partitions are merged, the current metering value (change) is calculated, the summary information (third information) is obtained, and the consumption offset of the summary information is recorded.

[0096] The foregoing mainly describes the solutions provided by the embodiments of this application from a methodological perspective. To achieve the above functions, it includes corresponding hardware structures and / or software modules for executing each function. Those skilled in the art should readily recognize that, in conjunction with the units and algorithm steps of the various examples described in the embodiments disclosed herein, the embodiments of this application can be implemented in hardware or a combination of hardware and computer software. Whether a function is executed in hardware or by computer software driving hardware depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0097] This application embodiment can divide a data statistics device into functional modules based on the above method example. For example, each function can be divided into its own functional modules, or two or more functions can be integrated into one processing module. The integrated module can be implemented in hardware or as a software functional module. Optionally, the module division in this application embodiment is illustrative and only represents one logical functional division; other division methods may be used in actual implementation.

[0098] Figure 7 is a schematic diagram of a data statistics device provided in an embodiment of this application. As shown in Figure 7, the data statistics device 40 is used to improve the efficiency of data statistics, for example, to execute a data statistics method shown in Figure 2. The data statistics device 40 includes: an acquisition unit 401, a determination unit 402, and a sending unit 403.

[0099] The acquisition unit 401 is used to acquire log information generated each time the storage cluster is accessed.

[0100] The determining unit 402 is used to determine the first information corresponding to each file in multiple files based on log information. The storage cluster includes multiple storage spaces, and the multiple files are the files corresponding to accessing the storage cluster. One file in the multiple files corresponds to one piece of first information, and one piece of first information corresponds to one storage space in the multiple storage spaces. One piece of first information includes at least one of the following: the data volume of the file and the access type.

[0101] Sending unit 403 is used to send multiple first messages to the Kafka cluster.

[0102] The determining unit 402 is used to determine the second information corresponding to each of the multiple storage spaces included in the storage cluster through multiple first information. The second information includes at least one of the following: the amount of memory change of the storage space and the number of files included in the storage space.

[0103] The determining unit 402 is used to determine the third information corresponding to the storage cluster based on the second information corresponding to each of the multiple storage spaces included in the storage cluster. The third information is used to indicate at least one of the following: the memory usage of the storage cluster and the total number of files included in the storage cluster.

[0104] In one design, a Kafka cluster consists of multiple Kafka partitions, and one Kafka partition corresponds to one storage space among multiple storage spaces.

[0105] The determining unit 402 is used to determine the target storage space corresponding to any first information among a plurality of first information.

[0106] The sending unit 403 is used to send any first information to the target Kafka partition corresponding to the target storage space in the Kafka cluster.

[0107] In one design, a determining unit 402 is used to determine, for a target Kafka partition among multiple Kafka partitions, second information corresponding to a target storage space based on at least one first piece of information corresponding to the target Kafka partition.

[0108] The sending unit 403 is used to send the second information corresponding to each of the multiple storage spaces to the management Kafka, which corresponds to the storage cluster.

[0109] In one design, a unit is defined to summarize the second information corresponding to each of the multiple storage spaces included in the management of Kafka, in order to obtain the third information corresponding to the storage cluster.

[0110] In the case of implementing the functions of the integrated modules described above in hardware, this application provides another possible structural schematic diagram of the electronic device involved in the above embodiments. As shown in FIG8, an electronic device 70 is used to improve the efficiency of data statistics, for example, for executing a data statistics method shown in FIG2. The electronic device 70 includes a processor 701, a memory 702, and a bus 703. The processor 701 and the memory 702 can be connected via the bus 703.

[0111] Processor 701 is the control center of the communication device. It can be a single processor or a collective term for multiple processing elements. For example, processor 701 can be a general-purpose central processing unit (CPU) or other general-purpose processors. Among them, the general-purpose processor can be a microprocessor or any conventional processor.

[0112] As one embodiment, processor 701 may include one or more CPUs, such as CPU 0 and CPU 1 shown in FIG8.

[0113] The memory 702 may be a read-only memory (ROM) or other type of static storage device capable of storing static information and instructions, random access memory (RAM) or other type of dynamic storage device capable of storing information and instructions, or electrically erasable programmable read-only memory (EEPROM), disk storage medium or other magnetic storage device, or any other medium capable of carrying or storing desired program code in the form of instructions or data structures and accessible by a computer, but is not limited thereto.

[0114] As one possible implementation, the memory 702 can exist independently of the processor 701. The memory 702 can be connected to the processor 701 via a bus 703 and is used to store instructions or program code. When the processor 701 calls and executes the instructions or program code stored in the memory 702, it can implement a data statistics method provided in this application embodiment.

[0115] In another possible implementation, the memory 702 can also be integrated with the processor 701.

[0116] Bus 703 can be an Industry Standard Architecture (ISA) bus, a Peripheral Component Interconnect (PCI) bus, or an Extended Industry Standard Architecture (EISA) bus. This bus can be divided into address bus, data bus, control bus, etc. For ease of illustration, only one thick line is used in Figure 8, but this does not indicate that there is only one bus or one type of bus.

[0117] It should be noted that the structure shown in FIG8 does not constitute a limitation on the electronic device 70. In addition to the components shown in FIG8, the electronic device 70 may include more or fewer components than shown, or combine certain components, or have different component arrangements.

[0118] As an example, referring to Figure 7, the functions of the acquisition unit 401, determination unit 402, and sending unit 403 in the data statistics device 40 are the same as those of the processor 701 in Figure 8.

[0119] Optionally, as shown in FIG8, the electronic device 70 provided in the embodiments of this application may further include a communication interface 704.

[0120] Communication interface 704 is used to connect with other devices via a communication network. This communication network can be Ethernet, a wireless access network, a wireless local area network (WLAN), etc. Communication interface 704 may include a receiving unit for receiving data and a transmitting unit for transmitting data.

[0121] In one design, the communication interface in the electronic device provided in this application embodiment can also be integrated into the processor.

[0122] Through the above description of the embodiments, those skilled in the art will clearly understand that, for the sake of convenience and brevity, only the division of the above functional units is used as an example. In practical applications, the above functions can be assigned to different functional units as needed, that is, the internal structure of the device can be divided into different functional units to complete all or part of the functions described above. The specific working process of the system, device, and unit described above can be referred to the corresponding process in the foregoing method embodiments, and will not be repeated here.

[0123] This application also provides a computer-readable storage medium storing instructions. When a computer executes these instructions, the computer performs each step of the method flow shown in the above-described method embodiments.

[0124] Embodiments of this application provide a computer program product containing instructions that, when executed on a computer, cause the computer to perform a data statistics method as described in the above method embodiments.

[0125] The computer-readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of computer-readable storage media (a non-exhaustive list) include: an electrical connection having one or more wires, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM), registers, hard disks, optical fibers, compact disc read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof, or any other form of computer-readable storage medium in the art. An exemplary storage medium is coupled to a processor, enabling the processor to read information from and write information to the storage medium. Of course, the storage medium may also be a component of the processor. The processor and the storage medium may reside in an application-specific integrated circuit (ASIC). In the embodiments of this application, the computer-readable storage medium can be any tangible medium that contains or stores a program that can be used by or in conjunction with an instruction execution system, apparatus, or device.

[0126] Since the electronic devices, computer-readable storage media, and computer program products in the embodiments of this application can be applied to the above methods, the technical effects they can achieve can also be referred to the above method embodiments. The embodiments of this application will not be repeated here.

[0127] The above are merely specific embodiments of this application, but the scope of protection of this application is not limited thereto. Any changes or substitutions within the technical scope disclosed in this application should be covered within the scope of protection of this application.

Claims

1. A data statistics method, characterized in that, The method includes: acquiring log information generated each time the storage cluster is accessed, and determining first information corresponding to each file in multiple files based on the log information. The storage cluster includes multiple storage spaces, and the multiple files are the files corresponding to the access to the storage cluster. One file in the multiple files corresponds to one piece of first information, and one piece of first information corresponds to one storage space in the multiple storage spaces. One piece of first information includes at least one of the following: the data volume of the file and the access type; the access type includes file modification; for any piece of first information, determining the target storage space corresponding to the first piece of first information; and sending the first piece of first information to the target storage space corresponding to the target storage space in the Kafka cluster. The Kafka cluster comprises multiple Kafka partitions, and one of the Kafka partitions corresponds to one of the multiple storage spaces. Second information corresponding to each storage space in the storage cluster is determined using the multiple first pieces of information. The second information includes at least one of the following: the amount of memory change in the storage space, and the number of files included in the storage space. Based on the second information corresponding to each storage space in the storage cluster, third information corresponding to the storage cluster is determined. The third information indicates at least one of the following: the memory usage of the storage cluster, and the total number of files included in the storage cluster.

2. The data statistics method according to claim 1, characterized in that, The step of determining the second information corresponding to each of the multiple storage spaces included in the storage cluster through the multiple first information includes: for the target Kafka partition among the multiple Kafka partitions, determining the second information corresponding to the target storage space based on at least one of the first information corresponding to the target Kafka partition; and sending the second information corresponding to each of the multiple storage spaces to the management Kafka, wherein the management Kafka corresponds to the storage cluster.

3. The data statistics method according to claim 2, characterized in that, The step of determining the third information corresponding to the storage cluster based on the second information corresponding to each of the multiple storage spaces included in the storage cluster includes: summarizing the second information corresponding to each of the multiple storage spaces included in the management Kafka to obtain the third information corresponding to the storage cluster.

4. A data statistics device, characterized in that, The device includes: an acquisition unit, a determination unit, and a sending unit; the acquisition unit is used to acquire log information generated each time the storage cluster is accessed; the determination unit is used to determine first information corresponding to each file in a plurality of files based on the log information, the storage cluster includes a plurality of storage spaces, the plurality of files are files corresponding to accessing the storage cluster, one file in the plurality of files corresponds to one piece of first information, one piece of first information corresponds to one storage space in the plurality of storage spaces, and one piece of first information includes at least one of the following: the data volume of the file and the access type; the access type includes file modification; the sending unit is used to send the plurality of first information to a Kafka cluster; the Kafka cluster includes a plurality of Kafka partitions, one Kafka partition in the plurality of Kafka partitions corresponds to one storage space in the plurality of storage spaces. The storage cluster includes a storage space; the determining unit is configured to determine a target storage space corresponding to any one of the plurality of first information; the sending unit is configured to send the any one of the first information to the target Kafka partition corresponding to the target storage space in the Kafka cluster; the determining unit is configured to determine second information corresponding to each of the plurality of storage spaces included in the storage cluster based on the plurality of first information, wherein the second information includes at least one of the following: the amount of memory change of the storage space, and the number of files included in the storage space; the determining unit is configured to determine third information corresponding to the storage cluster based on the second information corresponding to each of the plurality of storage spaces included in the storage cluster, wherein the third information indicates at least one of the following: the memory usage of the storage cluster, and the total number of files included in the storage cluster.

5. The data statistics device according to claim 4, characterized in that, The determining unit is configured to, for the target Kafka partition among the plurality of Kafka partitions, determine the second information corresponding to the target storage space based on at least one piece of first information corresponding to the target Kafka partition; The sending unit is used to send the second information corresponding to each of the plurality of storage spaces to the management Kafka, wherein the management Kafka corresponds to the storage cluster.

6. The data statistics device according to claim 5, characterized in that, The determining unit is used to summarize the second information corresponding to each of the multiple storage spaces included in the managed Kafka to obtain the third information corresponding to the storage cluster.

7. An electronic device, characterized in that, include: A processor and a memory; wherein the memory is used to store one or more programs, the one or more programs including computer execution instructions, and when the electronic device is running, the processor executes the computer execution instructions stored in the memory to cause the electronic device to perform a data statistics method according to any one of claims 1-3.

8. A computer-readable storage medium for storing one or more programs, characterized in that, The one or more programs include instructions that, when executed by a computer, cause the computer to perform a data statistics method according to any one of claims 1-3.

Images