Data acquisition method and device, and storage medium

Abstract

The application provides a data collection method and device and a storage medium, relates to the technical field of data collection, and can solve the problem of low efficiency during data collection. The method comprises the following steps: acquiring service information of a plurality of target services; the target service is a service subscribed by a client; determining a message queue according to the service information of the plurality of target services; the message queue is used for storing the service information of the plurality of target services; respectively instructing a plurality of listening units to perform a data collection operation according to the order of the service information in the message queue, so as to obtain traffic data of the plurality of target services. The application can improve the efficiency of data collection.

Description

Technical Field

[0001] This application relates to the field of data acquisition technology, and in particular to a data acquisition method, apparatus and storage medium. Background Technology

[0002] A cloud management platform is an operational platform that provides standard cloud services such as customer management, product management, order management, instance management, billing management, and statistical analysis. It has the capability to handle and host the sale, integration, statistics, and operation of various products and services listed on the cloud management platform. With the development of various cloud products and services, the method of billing based on the actual resource usage of cloud products is becoming increasingly common in cloud management platforms.

[0003] With the surge in data volume from cloud products in a short period of time, the limited number of servers available for data collection on the cloud management platform leads to excessive server load and data collection failures in the existing data collection and billing system, resulting in low data collection efficiency of the cloud management platform's data collection system. Summary of the Invention

[0004] This application provides a data acquisition method, apparatus, and storage medium, which solves the problem of low efficiency in data acquisition and can improve the efficiency of data acquisition.

[0005] To achieve the above objectives, this application adopts the following technical solution:

[0006] In a first aspect, this application provides a data acquisition method, which includes: acquiring business information of multiple target services; the target services being services subscribed to by customers; determining a message queue based on the business information of the multiple target services; the message queue being used to store the business information of the multiple target services; and instructing multiple listening units to perform data acquisition operations according to the order of the business information in the message queue, thereby obtaining traffic data of the multiple target services.

[0007] The above solution offers at least the following advantages: Based on the above technical solution, the data acquisition method provided in this application first involves the data acquisition device obtaining service information subscribed to by multiple customers, and then determining a message queue based on the service information. Since the message queue stores service information for multiple target services, and the data acquisition device instructs multiple listening units to perform data acquisition operations according to the order of the service information in the message queue, traffic data for multiple target services is obtained. Compared to the existing technology where the server pressure is too high and data cannot be collected when the data acquisition volume is large, the technical solution provided in this application, through message queue queuing and instructing multiple listening units to perform data acquisition, can effectively alleviate server pressure and improve data acquisition efficiency.

[0008] In conjunction with the first aspect above, in one possible implementation, the method further includes: the service information includes at least one of acquisition information or first service data information; the acquisition information is used to instruct the monitoring unit to acquire traffic data of the plurality of target services; the first service data information is the initial information of the traffic data of the target services.

[0009] In conjunction with the first aspect above, in one possible implementation, the method further includes: when the business information is the collection information, acquiring the collection information of each target business among multiple target businesses; the collection information of each target business includes the target account information and target business identification information of the corresponding target business; and adding the collection information of each target business to a message queue.

[0010] In conjunction with the first aspect above, in one possible implementation, the method further includes: when the business information is the first business data information, obtaining the first business data information of each of the multiple target businesses; performing data processing on the first business data information to obtain second business data information; the second business data information is the first business data information with complete fields; and adding the second business data information to the message queue.

[0011] In conjunction with the first aspect above, in one possible implementation, the method further includes: when the billing type of the target service is peak billing, extracting traffic data of the target service from the target database according to the target service identification information; calculating peak data of the target service based on the traffic data of the target service; and storing the peak data of the traffic data of the target service into the target database.

[0012] In conjunction with the first aspect mentioned above, in one possible implementation, the method further includes: taking the traffic data of the target service with the maximum value within the peak period of the target service's traffic data as the peak data of the target service.

[0013] Secondly, this application provides a data acquisition device, which includes: a communication unit and a processing unit; the communication unit is used to acquire service information of multiple target services; the target services are services subscribed by customers; the processing unit is used to determine a message queue based on the service information of the multiple target services; the message queue is used to store the service information of the multiple target services; the processing unit is also used to instruct multiple listening units to perform data acquisition operations according to the order of the service information in the message queue, so as to obtain traffic data of the multiple target services.

[0014] In conjunction with the second aspect above, in one possible implementation, the business information includes at least one of acquisition information or first business data information; the acquisition information is used to instruct the monitoring unit to acquire traffic data of multiple target services; the first business data information is the initial information of the traffic data of the target services.

[0015] In conjunction with the second aspect above, in one possible implementation, when the service information is collected information, the communication unit is further configured to acquire the collected information of each of the multiple target services; the collected information of each target service includes the target account information and target service identification information of the corresponding target service; the processing unit is further configured to add the collected information of each target service to the message queue.

[0016] In conjunction with the second aspect above, in one possible implementation, when the business information is first business data information, the communication unit is further configured to acquire the first business data information of each of the multiple target services; the processing unit is further configured to perform data processing on the first business data information to obtain second business data information; the second business data information is the first business data information with complete fields; the processing unit is further configured to add the second business data information to the message queue.

[0017] In conjunction with the second aspect above, in one possible implementation, when the billing type of the target service is peak billing, the processing unit is further configured to extract the traffic data of the target service from the target database based on the target service identification information; the processing unit is further configured to calculate the peak data of the target service based on the traffic data of the target service; and the processing unit is further configured to store the peak data of the traffic data of the target service into the target database.

[0018] In conjunction with the second aspect above, in one possible implementation, the processing unit is further configured to: take the traffic data of the target service with the maximum value within the peak period as the peak data of the target service, based on the peak period of the traffic data of the target service.

[0019] Thirdly, this application provides a data acquisition device, which includes: a processor and a communication interface; the communication interface and the processor are coupled, and the processor is used to run computer programs or instructions to implement the data acquisition method as described in the first aspect and any possible implementation of the first aspect.

[0020] Fourthly, this application provides a computer-readable storage medium storing instructions that, when executed on a terminal, cause the terminal to perform the data acquisition method described in the first aspect and any possible implementation thereof.

[0021] Fifthly, this application provides a computer program product containing instructions that, when run on a data acquisition device, cause the data acquisition device to perform the data acquisition method as described in the first aspect and any possible implementation thereof.

[0022] In a sixth aspect, this application provides a chip including a processor and a communication interface, the communication interface being coupled to the processor, the processor being used to run computer programs or instructions to implement the data acquisition method as described in the first aspect and any possible implementation thereof.

[0023] Specifically, the chip provided in this application also includes a memory for storing computer programs or instructions.

[0024] It should be noted that the aforementioned computer instructions may be stored, in whole or in part, on a computer-readable storage medium. This computer-readable storage medium may be packaged together with the processor of the device, or it may be packaged separately from the processor of the device; this application does not impose any limitation on this.

[0025] In a seventh aspect, this application provides a data acquisition system, comprising: a data acquisition device and a terminal device, wherein the data acquisition device is used to perform the data acquisition method as described in the first aspect and any possible implementation thereof.

[0026] The descriptions of aspects two through seven in this application can be referenced to the detailed description of aspect one; and the beneficial effects of the descriptions of aspects two through seven can be referenced to the analysis of the beneficial effects of aspect one, which will not be repeated here.

[0027] In this application, the names of the aforementioned data acquisition devices do not limit the devices or functional modules themselves. In actual implementation, these devices or functional modules may appear under other names. As long as the functions of each device or functional module are similar to those in this application, they fall within the scope of the claims of this application and their equivalents.

[0028] These or other aspects of this application will become more readily apparent in the following description. Attached Figure Description

[0029] Figure 1 This is a schematic diagram of the architecture of a data acquisition system provided in an embodiment of this application;

[0030] Figure 2 A schematic diagram of a data acquisition process provided in an embodiment of this application;

[0031] Figure 3 A schematic diagram illustrating another data acquisition process provided in an embodiment of this application;

[0032] Figure 4 A schematic diagram illustrating another data acquisition process provided in an embodiment of this application;

[0033] Figure 5 A schematic diagram illustrating another data acquisition process provided in an embodiment of this application;

[0034] Figure 6 A flowchart illustrating a data acquisition method provided in this application embodiment;

[0035] Figure 7 A flowchart illustrating another data acquisition method provided in this application embodiment;

[0036] Figure 8 A flowchart illustrating another data acquisition method provided in this application embodiment;

[0037] Figure 9 A flowchart illustrating another data acquisition method provided in this application embodiment;

[0038] Figure 10 This is a schematic diagram of the structure of a data acquisition device provided in an embodiment of this application;

[0039] Figure 11 This is a schematic diagram of another data acquisition device provided in an embodiment of this application. Detailed Implementation

[0040] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0041] In this article, the term "and / or" is merely a description of the relationship between related objects, indicating that there can be three relationships. For example, A and / or B can represent three situations: A exists alone, A and B exist simultaneously, and B exists alone.

[0042] The terms "first" and "second," etc., used in the specification and drawings of this application are used to distinguish different objects or to distinguish different treatments of the same object, rather than to describe a specific order of objects.

[0043] Furthermore, the terms "comprising" and "having," and any variations thereof, used in the description of this application are intended to cover non-exclusive inclusion. For example, a process, method, system, product, or apparatus that includes a series of steps or units is not limited to the steps or units listed, but may optionally include other steps or units not listed, or may optionally include other steps or units inherent to such process, method, product, or apparatus.

[0044] It should be noted that in the embodiments of this application, the words "exemplary" or "for example" are used to indicate examples, illustrations, or explanations. Any embodiment or design scheme described as "exemplary" or "for example" in the embodiments of this application should not be construed as being more preferred or advantageous than other embodiments or design schemes. Specifically, the use of the words "exemplary" or "for example" is intended to present the relevant concepts in a specific manner.

[0045] In the description of this application, unless otherwise stated, "a plurality of" means two or more.

[0046] A cloud management platform is an operational platform that provides standard cloud services such as customer management, product management, order management, instance management, billing management, and statistical analysis. It has the capability to handle and host the sale, integration, statistics, and operation of various products and services listed on the cloud management platform. With the development of various cloud products and services, the method of billing based on the actual resource usage of cloud products is becoming increasingly common in cloud management platforms.

[0047] Existing cloud platform data collection and billing systems develop unique data collection and billing methods for each product's resource type. When new products are integrated into the billing system, product interface development is required, resulting in inflexible and non-standardized systems. Furthermore, existing data collection and billing systems are designed to perform billing after data collection, leading to tight coupling between data collection and billing. If the data collection interface is not working, it will affect the operation of the billing module.

[0048] With the surge in data volume from cloud products in a short period of time, the limited number of servers available for data collection on the cloud management platform leads to excessive server load and data collection failures in the existing data collection and billing system, resulting in low data collection efficiency of the cloud management platform's data collection system.

[0049] Therefore, the data acquisition method provided in this application first involves a data acquisition device obtaining service information subscribed to by multiple customers, and then determining a message queue based on the service information. Since the message queue stores service information for multiple target services, and the data acquisition device instructs listening units to perform data acquisition operations based on the message queue, it obtains traffic data for multiple target services. Therefore, the data acquisition device can instruct multiple listening units to perform data acquisition operations according to the order of the service information in the message queue, thereby obtaining traffic data for multiple target services. Compared to the existing technology where existing data collection and billing systems suffer from excessive server pressure and are unable to collect data when the data collection volume is large, the technical solution provided in this application, through message queue queuing and instructing multiple listening units to perform data acquisition, can effectively alleviate server pressure and improve data acquisition efficiency.

[0050] The embodiments of this application will now be described in detail with reference to the accompanying drawings.

[0051] Figure 1 This is an architecture diagram of a data acquisition system 10 provided in an embodiment of this application. Figure 1 As shown, the data acquisition system 10 includes a data acquisition device 101 and a terminal device 102.

[0052] The data acquisition device 101 and the terminal device 102 can be one or more, for ease of understanding. Figure 1 Only one is shown in the image.

[0053] The data acquisition device 101 and the terminal device 102 are connected via a communication link. This communication link can be a wired communication link or a wireless communication link, and this application does not limit it in this regard.

[0054] The aforementioned data acquisition device 101 includes:

[0055] The processor can be a general-purpose central processing unit (CPU), a microprocessor, an application-specific integrated circuit (ASIC), or one or more integrated circuits used to control the execution of the program in this application.

[0056] A transceiver can be any type of transceiver used to communicate with other devices or communication networks, such as Ethernet, radio access network (RAN), wireless local area network (WLAN), etc.

[0057] Memory can be read-only memory (ROM) or other types of static storage devices capable of storing static information and instructions, random access memory (RAM) or other types of dynamic storage devices capable of storing information and instructions, electrically erasable programmable read-only memory (EEPROM), compact disc read-only memory (CD-ROM) or other optical disc storage, optical disc storage (including compressed discs, laser discs, optical discs, universal optical discs, Blu-ray discs, etc.), magnetic disk storage media or other magnetic storage devices, or any other medium capable of carrying or storing desired program code in the form of instructions or data structures and accessible by a computer, but is not limited thereto. Memory can exist independently and be connected to the processor via communication lines. Memory can also be integrated with the processor.

[0058] Terminal device 102 is a device with wireless communication capabilities that can be deployed on land, including indoors or outdoors, handheld or vehicle-mounted. It can also be deployed on water (such as on ships) or in the air (e.g., on airplanes, balloons, and satellites). A terminal, also known as user equipment (UE), mobile station (MS), mobile terminal (MT), or other terminal equipment, is a device that provides voice and / or data connectivity to a user. For example, terminals include handheld devices and vehicle-mounted devices with wireless connectivity. Currently, terminals can be: mobile phones, tablets, laptops, PDAs, mobile internet devices (MIDs), wearable devices (such as smartwatches, smart bracelets, pedometers, etc.), in-vehicle devices (such as cars, bicycles, electric vehicles, airplanes, ships, trains, high-speed trains, etc.), virtual reality (VR) devices, augmented reality (AR) devices, wireless terminals in industrial control, smart home devices (such as refrigerators, televisions, air conditioners, electricity meters, etc.), intelligent robots, workshop equipment, wireless terminals in self-driving, wireless terminals in remote medical surgery, wireless terminals in smart grids, wireless terminals in transportation safety, wireless terminals in smart cities, or wireless terminals in smart homes, and flying equipment (such as intelligent robots, hot air balloons, drones, airplanes), etc. In one possible application scenario of this application, the terminal device is a terminal device that frequently operates on the ground, such as an in-vehicle device. In this application, for ease of description, the chip deployed in the above-mentioned device, such as a system-on-a-chip (SOC), a baseband chip, or other chip with communication functions, may also be referred to as a terminal.

[0059] In one possible implementation, the data acquisition device 101 can acquire data from the terminal device 102.

[0060] For example, the traffic data of terminal device 102 can be public network bandwidth data, Internet Protocol Version 6 (IPv6) data, Content Delivery Network (CDN) data, object storage data, distributed site service data, video surveillance data, application development and deployment service (aPaaS) data, Network Attached Storage (NAS) data, image service data, etc. The timing data of terminal device 102 can be Elastic Compute Service (ECS) data, public network bandwidth duration data, cloud disk data, etc.

[0061] One possible implementation is, such as Figure 2 As shown, the CDN data, object storage data, public network bandwidth data, etc. of terminal device 102 are processed by the data acquisition device 101 through timed active collection and processing, message reception collection and processing, interface reception collection and processing, file collection and processing, data verification processing, format conversion processing, collection retry processing, peak processing, and batch pricing notification processing. The processed data can then be used for billing management, specifically for scenarios such as pay-as-you-go pricing and pay-as-you-go billing. When the data acquisition device 101 processes the data, the data can be stored in a message queue, a distributed file storage database (MongoDB), a MySQL database, or a Redis database.

[0062] One possible implementation is that the data acquisition device 101 actively acquires traffic data periodically through the Hyper Text Transfer Protocol (HTTP) interface based on the target service identification information.

[0063] One possible implementation, as shown in Table 1, is that the data acquisition device 101 provides a Hypertext Transfer Protocol (HTTP) request format.

[0064] Table 1. One HTTP request format

[0065]

[0066] For example, the Hypertext Transfer Protocol (HTTP) interface provided by the data acquisition device 101 includes an HTTP request format. Based on the data from the acquisition source, the data acquisition device 101 collects traffic data from the acquisition source according to the fields in the HTTP request format. The fields in the HTTP request format may include the main account ID, start time (milliseconds), end time (milliseconds), acquisition interval (60 minutes), resource instance ID list, resource instance type, etc. The fields of the Hypertext Transfer Protocol request format provided by the data acquisition device 101 can be added or removed according to different data from the acquisition source.

[0067] One possible implementation, as shown in Table 2, is that the data acquisition device 101 provides an HTTP response format.

[0068] Table 2. One HTTP response format

[0069]

[0070]

[0071] For example, the Hypertext Transfer Protocol (HTTP) interface provided by the data acquisition device 101 includes an HTTP response format. Based on the data from the acquisition source and the HTTP request format, the data acquisition device 101 acquires data from the fields of the HTTP response format in the data from the acquisition source. The fields in the HTTP response format may include status codes: 200 for success, 500 for exception, result, main account ID, start time (milliseconds), end time (milliseconds), acquisition interval (60 minutes), resource instance ID, resource instance type, resource pool, cloud zone, billing factor, billing factor source marker, billing factor unit, total acquisition usage, and extended attributes of acquisition details data, etc. The fields of the Hypertext Transfer Protocol response format provided by the data acquisition device 101 can be added or removed according to different data from the acquisition source.

[0072] For example, such as Figure 3As shown, after the configured scheduled task execution time is reached, the data acquisition device initiates an active acquisition task to obtain the acquisition attribute configuration information of the target service. If the acquisition type in the acquisition attribute configuration information is duration-based acquisition, the usage duration within the acquisition period is directly calculated based on the duration information in the on-demand resource instance table. If the acquisition type in the acquisition attribute configuration information is usage-based acquisition, an HTTP acquisition message is sent to the acquisition queue based on the target account information in the on-demand resource instance table. Then, multiple consumers are started to concurrently listen to the HTTP acquisition processing queue, send HTTP requests, and call the corresponding interface service based on the acquisition protocol interface address and other information in the acquisition attribute configuration information, receiving the returned results. Data from duration-based acquisition or received return results is validated and format-converted. The acquired data is stored in a MongoDB database. When the service type of the acquired data is peak-billing, the acquisition attribute configuration information of the acquisition data of peak-billing type is recorded, peak processing is performed, and a batch pricing message is sent.

[0073] In one possible implementation, the data acquisition device 101 receives and saves traffic data in HTTP response format by listening to traffic data added to a message queue from the terminal device 102. The message queue can be at least one of a distributed publish-subscribe messaging system (Kafka) or a RabbitMQ message queue (MQ). The traffic data is stored in the message queue in the form of messages.

[0074] One possible implementation is that the data acquisition device 101 receives traffic data from the terminal device 102 through an HTTP receive and response format interface, and returns the results to the terminal device 102.

[0075] For example, such as Figure 4As shown, the data acquisition device 101 receives traffic data from the terminal device 102 via an interface. The traffic data undergoes format verification. If the traffic data contains complete fields, the data acquisition device 101 sends the traffic data to an internal message queue and sends a successful data reception instruction to the smart terminal 102. Correspondingly, the smart terminal 102 receives the successful data reception instruction from the data acquisition device 101. If the traffic data contains incomplete fields, the data acquisition device 101 sends a data error instruction to the smart terminal 102. Correspondingly, the smart terminal 102 receives the data error instruction from the data acquisition device 101. Then, multiple consumers are started to concurrently listen to the traffic data in the internal message queue and the traffic data directly sent by the smart terminal 102 to the external data queue. The data acquisition device 101 receives the traffic data in the message queue and performs data verification and format conversion. The collected traffic data is stored in a MongoDB database. When the service type of the collected traffic data is peak billing, the collection attribute configuration information of the traffic data of peak billing type is recorded, peak processing is performed, and batch pricing messages are sent.

[0076] One possible implementation is that the data acquisition device 101 logs into the Secure File Transfer Protocol (SFTP) to retrieve the traffic data file stored by the terminal device 102. The data acquisition device 101 then parses the retrieved file and stores it in a database.

[0077] One possible implementation is that the data acquisition device 101 verifies the format of the acquired traffic data to ensure it is complete and that the data is not duplicated, records abnormal data, and converts the correct data into a standard format and saves it to the MongoDB database.

[0078] One possible implementation is that the data acquisition device 101 sends the traffic data as a message to the subsequent billing system for pricing and billing processing. When the traffic data is billed as peak-based, after the configured peak calculation period is reached, the peak data corresponding to each resource is calculated based on the traffic data of the peak calculation period, and a pricing notification is sent.

[0079] For example, such as Figure 5As shown, after the configured scheduled task execution time is reached, the data acquisition device starts the peak processing task, collecting the peak-billed traffic data acquisition attribute configuration information and the target account information from the on-demand resource instance table. It retrieves traffic data from the database, calculates the corresponding peak data, stores the peak data in the database, and sends a batch pricing message. The peak data includes hourly peaks, daily peaks, monthly peaks, or monthly 95% peaks. The monthly 95% peak is calculated by taking traffic data every 5 minutes within a calendar month, sorting it in descending order by the main account ID, removing the top 5% of the traffic data, and then selecting the largest traffic data from the remaining 95%.

[0080] One possible implementation is that for data that fails to be acquired, the data acquisition device 101 periodically performs retry processing. For timed tasks that are not executed, the data acquisition device 101 reacquires data.

[0081] It should be noted that the various embodiments of this application can be referenced or learned from each other. For example, the same or similar steps, method embodiments, system embodiments and device embodiments can be referenced from each other without limitation.

[0082] Figure 6 This is a flowchart illustrating a data acquisition method provided in an embodiment of this application. Figure 6 As shown, the method includes the following steps:

[0083] S601, The data acquisition device acquires business information for multiple target services.

[0084] The target business refers to the services that the customer subscribes to.

[0085] It should be noted that the business information also includes at least one of the following: collection information or first business data information. Collection information is used to instruct the monitoring unit to acquire traffic data from multiple target services. First business data information is the initial information of the traffic data for the target services. For example, the collection information for each target service may include the target service identifier, target account information, collection period, collection type, collection protocol, interface address, and whether peak-rate billing applies. The first business data information for the target services may include information such as whether peak-rate billing applies to the target service identifier.

[0086] For example, the services subscribed by a customer may include public network bandwidth data, Internet Protocol Version 6 (IPv6) data, Content Delivery Network (CDN) data, object storage data, distributed site service data, video surveillance data, application development and deployment service (aPaaS) data, network attached storage (NAS) data, image service data, etc.

[0087] S602, The data acquisition device determines the message queue based on the business information of multiple target services.

[0088] The message queue is used to store business information for multiple target services.

[0089] For example, when the business information is collected information, the data acquisition device adds the collected information of multiple target businesses to the message queue. When the business information is first business data information, the data acquisition device adds the first business data information of multiple businesses to the message queue.

[0090] S603. The data acquisition device instructs multiple listening units to perform data acquisition operations according to the order of service information in the message queue, thereby obtaining traffic data of multiple target services.

[0091] One possible implementation involves using business information from multiple target services in a message queue. A first number of listening units sequentially collect data from the first number of target services according to the order of the business information in the message queue, thereby obtaining the traffic data of the first number of target services.

[0092] For example, if the target services corresponding to the service information in the message queue are public network bandwidth service, object storage service, and content delivery network service, and the number of listening units is two, the two listening units simultaneously collect data from public network bandwidth and object storage respectively, obtaining traffic data for public network bandwidth service and object storage service respectively. If the listening unit handling public network bandwidth service completes data collection before the listening unit handling object storage service, then the listening unit handling public network bandwidth service will collect data from content delivery network service, obtaining traffic data for content delivery network service.

[0093] One possible implementation is that, given that the service information for each target service is in a first standard format, the data acquisition device obtains the corresponding traffic data for the target service in a second standard format based on the service information in the first standard format. Therefore, for different types of target services, the data acquisition device can standardize the field information of the collected traffic data for each target service.

[0094] For example, the first standard format can include fields such as resource instance ID, main account ID, start time, end time, collection interval, resource instance ID list, and resource instance type for each target service. The second standard format can include fields such as collection result, collection status, main account ID, start time, end time, collection interval, resource instance ID, resource instance type, resource pool, cloud zone, billing factor, billing factor source marker, billing factor unit, total collection usage, and attributes of the collected data. The first and second standard formats can be added or removed based on the specific needs of the business; this application does not impose any limitations on this. Compared to existing cloud platform collection and billing systems, which develop unique collection and billing methods for each product's resource type, new products need to develop product interfaces when integrating with the billing system, resulting in non-standard data formats collected by the system. The above method can standardize the field formats of the collected traffic data.

[0095] One possible implementation involves the data acquisition device collecting traffic data from multiple target services. The device then performs field format validation and duplicate verification on the traffic data. Complete traffic data is converted to the target standard format. Finally, both complete and incomplete / duplicate traffic data are saved to the target database.

[0096] Based on the above technical solution, the data acquisition method provided in this application first involves a data acquisition device obtaining service information subscribed to by multiple customers, and then determining a message queue based on the service information. Since the message queue stores service information for multiple target services, and the data acquisition device instructs listening units to perform data acquisition operations based on the message queue, traffic data for multiple target services is obtained. Therefore, the data acquisition device can instruct multiple listening units to perform data acquisition operations according to the order of the service information in the message queue, thereby obtaining traffic data for multiple target services. Compared to the existing technology where server pressure is too high and data cannot be collected when the data acquisition volume is large, the technical solution provided in this application, through message queue queuing and instructing multiple listening units to perform data acquisition, can effectively alleviate server pressure and improve data acquisition efficiency.

[0097] As one possible embodiment of this application, combined with Figure 6 ,like Figure 7 As shown, when the business information is collected information, the above S602 can also be implemented through the following S701-S702.

[0098] S701, The data acquisition device acquires the acquisition information of each target service among multiple target services.

[0099] The information collected for each target service includes the target account information and the target service identifier information.

[0100] One possible implementation involves the data acquisition device collecting traffic data for metering-type data. When the target service's data collection type is metering, the data acquisition device obtains the corresponding target service's target account information and target service identifier information.

[0101] For example, taking the target account information of the target business as the main account ID and the target business identifier information as the resource instance ID as an example, the data acquisition device obtains the corresponding target business's resource instance ID, main account ID, and other information.

[0102] Another possible implementation is that, for duration-based data, the data acquisition device can also collect duration data. Specifically, when the target service's collection type is duration-based, the data acquisition device obtains the corresponding target service's target account information, target service identifier information, target service duration information, etc.

[0103] For example, when the target service's data collection type is duration-based, the data collection device collects information such as resource instance ID, main account ID, resource activation time, resource expiration time, target service start time, and target service end time. The difference between the target service start time and the target service end time is used as the duration data of the target service.

[0104] S702, The data acquisition device adds the acquisition information for each target service to the message queue.

[0105] For example, consider collecting information including the resource instance ID and main account ID of the corresponding target service. The data collection device adds the collection information of multiple target services to a message queue. The collection information for each target service includes the resource instance ID and main account ID of the corresponding target service.

[0106] Based on the above technical solution, the data acquisition device obtains the acquisition information for each of the multiple target services and adds the acquisition information for each target service to a message queue. Since the acquisition information for each target service includes the target account information and target service identifier information, the data acquisition device can accurately collect the traffic data for each target service based on the corresponding target account information and target service identifier information. Furthermore, by adding the acquisition information for each target service to the message queue, the server load can be effectively reduced, and the efficiency of data acquisition can be improved.

[0107] As one possible embodiment of this application, combined with Figure 6 ,like Figure 8 As shown, when the business information is the first business data information, the above S602 can also be implemented by the following S801-S803.

[0108] S801, The data acquisition device acquires the first service data information of each of the multiple target services.

[0109] For example, taking the initial information of traffic data for the target service as the first data information, the data acquisition device receives the initial information of traffic data for the target service through a data receiving interface. The data receiving interface is configured with a second standard format; when the data acquisition device uses this interface to acquire data, it can obtain traffic data for the target service in the second standard format. Therefore, for different types of target services, the data acquisition device can standardize the field information of the collected traffic data for the target service.

[0110] For example, the first business data information may include fields such as collection result, collection status, main account ID, start time, end time, collection interval time, resource instance ID, resource instance type, resource pool, cloud zone, billing factor, billing factor source marker, billing factor unit, total collection usage, and attributes of the collected data.

[0111] S802, The data acquisition device processes the first business data information to obtain the second business data information.

[0112] The second business data information is the first business data information with all fields complete.

[0113] For example, the data acquisition device performs field format validation on the first business data information. The first business data information with complete fields is then used as the second business data information.

[0114] S803, the data acquisition device adds the second business data information to the message queue.

[0115] For example, the data acquisition device adds second business data information from multiple services to a message queue.

[0116] Based on the above technical solution, the data acquisition device can directly acquire the first business data information of each of multiple target businesses. Since the data acquisition device processes the first business data information to obtain the second business data information, it can obtain the second business data information in a complete format. The data acquisition device adds the second business data information to a message queue, which can effectively alleviate the server pressure and improve the efficiency of data acquisition.

[0117] As one possible embodiment of this application, such as Figure 9 As shown, when the billing type of the target service is peak billing, the above method also includes the following S901-S903 implementations.

[0118] S901, The data acquisition device extracts the traffic data of the target service from the target database based on the target service identification information.

[0119] For example, taking the target service identification information as the resource instance ID, the Chujiu data collection device extracts the corresponding target service traffic data from the target database based on the target service's resource instance ID.

[0120] S902, The data acquisition device calculates the peak data of the target service based on the traffic data of the target service.

[0121] One possible implementation is that the data acquisition device takes the maximum value of the target service's traffic data within the peak period as the peak data of the target service.

[0122] For example, taking an hourly peak period for the traffic data of the target service as an example, the data acquisition device takes the highest traffic data of the target service within one hour as the peak data of the target service within one hour.

[0123] S903, The data acquisition device stores the peak data of the target service into the target database.

[0124] For example, taking the peak data of the target business as the peak data of the target business within one hour as an example, the data acquisition device stores the peak data of the target business within one hour into the target database. The target database can be a distributed file storage database (MongoDB), a MySQL database, or a Redis database, etc.

[0125] Based on the above technical solution, the data acquisition device extracts the traffic data of the target service from the target database according to the target service identification information. Then, based on the traffic data of the target service and the target account information of the target service, the peak data of the target service is calculated and stored in the target database. Therefore, the above technical solution can calculate the peak data of the target service when the billing method for traffic data is peak-based billing.

[0126] This application embodiment can divide the data acquisition device into functional modules or functional units according to the above method examples. For example, each function can be divided into its own functional modules or functional units, or two or more functions can be integrated into one processing module. The integrated module can be implemented in hardware or in software functional modules or functional units. The module or unit division in this application embodiment is illustrative and represents only one logical functional division; other division methods may be used in actual implementation.

[0127] like Figure 10 The diagram shown is a structural schematic of a data acquisition device 100 provided in an embodiment of this application. The device includes a communication unit 1001 and a processing unit 1002.

[0128] Communication unit 1001 is used to acquire service information of multiple target services; the target services are services subscribed by customers; processing unit 1002 is used to determine a message queue based on the service information of multiple target services; the message queue is used to store the service information of multiple target services; processing unit 1002 is also used to instruct multiple listening units to perform data acquisition operations according to the order of the service information in the message queue, so as to obtain traffic data of multiple target services.

[0129] The business information includes at least one of the collected information or the first business data information; the collected information is used to instruct the monitoring unit to acquire traffic data of multiple target services; the first business data information is the initial information of the traffic data of the target services.

[0130] When the service information is the collected information, the communication unit 1001 is further configured to acquire the collected information of each target service among multiple target services; the collected information of each target service includes the target account information and target service identification information of the corresponding target service; the processing unit 1002 is further configured to add the collected information of each target service to the message queue.

[0131] When the business information is the first business data information, the communication unit 1001 is further used to acquire the first business data information of each target business among multiple target businesses; the processing unit 1002 is further used to process the first business data information to obtain the second business data information; the second business data information is the first business data information with complete fields; the processing unit 1002 is further used to add the second business data information to the message queue.

[0132] When the billing type of the target service is peak billing, the processing unit 1002 is further configured to extract the traffic data of the target service from the target database according to the target service identification information; the processing unit 1002 is further configured to calculate the peak data of the target service based on the traffic data of the target service; and the processing unit 1002 is further configured to store the peak data of the traffic data of the target service into the target database.

[0133] The processing unit 1002 is also used to take the traffic data of the target service with the maximum value within the peak period as the peak data of the target service, based on the peak period of the traffic data of the target service.

[0134] In one possible implementation, the data acquisition device 10 may further include a storage unit 1003. Figure 10 (shown in dashed box) The storage unit 1003 stores a program or instruction. When the processing unit 1002 executes the program or instruction, the data acquisition device 10 can perform the data acquisition method described in the above method embodiment.

[0135] When implemented in hardware, the communication unit 1001 in this embodiment can be integrated onto the communication interface, and the processing unit 1002 can be integrated onto the processor. Specific implementation methods are as follows: Figure 11 As shown.

[0136] Figure 11 A schematic diagram of another possible structure of the data acquisition device involved in the above embodiments is shown. The data acquisition device includes a processor 1102 and a communication interface 1101. The processor 1102 is used to control and manage the operation of the data acquisition device, for example, executing the steps performed by the processing unit 1002, and / or performing other processes of the technology described herein. The communication interface 1101 is used to support communication between the data acquisition device and other network entities, for example, executing the steps performed by the communication unit 1001. The data acquisition device may also include a memory 1103 and a bus 1104, the memory 1103 being used to store the program code and data of the data acquisition device.

[0137] The memory 1103 may be a memory in a data acquisition device, and the memory may include volatile memory, such as random access memory; the memory may also include non-volatile memory, such as read-only memory, flash memory, hard disk or solid-state drive; the memory may also include a combination of the above types of memory.

[0138] The processor 1102 described above can implement or execute various exemplary logic blocks, modules, and circuits described in conjunction with the disclosure of this application. The processor can be a central processing unit, a general-purpose processor, a digital signal processor, an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable logic devices, transistor logic devices, hardware components, or any combination thereof. It can implement or execute various exemplary logic blocks, modules, and circuits described in conjunction with the disclosure of this application. The processor can also be a combination that implements computational functions, such as including one or more microprocessor combinations, a combination of a DSP and a microprocessor, etc.

[0139] Bus 1104 can be an Extended Industry Standard Architecture (EISA) bus, etc. Bus 1104 can be divided into address bus, data bus, control bus, etc. For ease of representation, Figure 11 The bus is represented by a single thick line, but this does not mean that there is only one bus or one type of bus.

[0140] Figure 11 The data acquisition device in the middle can also be a chip. The chip includes one or more processors 1102 and a communication interface 1101.

[0141] In some embodiments, the chip further includes a memory 1103, which may include read-only memory and random access memory, and provides operation instructions and data to the processor 1102. A portion of the memory 1103 may also include non-volatile random access memory (NVRAM).

[0142] In some implementations, memory 1103 stores elements such as execution modules or data structures, or subsets thereof, or extended sets thereof.

[0143] In this embodiment of the application, the corresponding operation is executed by calling the operation instructions stored in the memory 1103 (the operation instructions can be stored in the operating system).

[0144] Through the above description of the embodiments, those skilled in the art will clearly understand that, for the sake of convenience and brevity, only the division of the above functional modules is used as an example. In practical applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the device can be divided into different functional modules to complete all or part of the functions described above. The specific working process of the system, device, and unit described above can be referred to the corresponding process in the foregoing method embodiments, and will not be repeated here.

[0145] This application provides a computer program product containing instructions that, when run on a computer, cause the computer to execute the data acquisition method described in the above method embodiments.

[0146] This application also provides a computer-readable storage medium storing instructions that, when executed on a computer, cause the computer to perform the data acquisition method in the method flow shown in the above method embodiments.

[0147] The computer-readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples (a non-exhaustive list) of computer-readable storage media include: electrical connections having one or more wires; portable computer disks; hard disks; random access memory (RAM); read-only memory (ROM); erasable programmable read-only memory (EPROM); registers; hard disks; optical fibers; portable compact disc read-only memory (CD-ROM); optical storage devices; magnetic storage devices; or any suitable combination thereof; or any other form of computer-readable storage medium known in the art. An exemplary storage medium is coupled to a processor, enabling the processor to read information from and write information to the storage medium. Of course, the storage medium may also be a component of the processor. The processor and the storage medium may reside in an application-specific integrated circuit (ASIC). In the embodiments of this application, the computer-readable storage medium can be any tangible medium that contains or stores a program that can be used by or in conjunction with an instruction execution system, apparatus, or device.

[0148] Since the data acquisition device, computer-readable storage medium, and computer program product in the embodiments of this application can be applied to the above method, the technical effects that can be obtained can also be referred to the above method embodiments. The embodiments of this application will not be repeated here.

[0149] In the several embodiments provided in this application, it should be understood that the disclosed systems, devices, and methods can be implemented in other ways. For example, the device embodiments described above are merely illustrative; for instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the mutual coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between devices or units may be electrical, mechanical, or other forms.

[0150] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0151] In addition, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit.

[0152] The above are merely specific embodiments of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions within the technical scope disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. A data acquisition method, characterized in that, The method includes: Obtain business information for multiple target services; the target services are those subscribed to by the customer. A message queue is determined based on the service information of the multiple target services; the message queue is used to store the service information of the multiple target services; the service information includes at least one of collection information or first service data information; the collection information is used to instruct the monitoring unit to acquire traffic data of the multiple target services; the first service data information is the initial information of the traffic data of the target services; When the business information is the collected information, determining the message queue based on the business information of the plurality of target businesses includes: obtaining the collected information of each target business among the plurality of target businesses; the collected information of each target business includes the target account information and target business identification information of the corresponding target business; and adding the collected information of each target business to the message queue. When the business information is the first business data information, determining the message queue based on the business information of the plurality of target businesses includes: obtaining the first business data information of each target business among the plurality of target businesses; performing data processing on the first business data information to obtain the second business data information; the second business data information is the first business data information with complete fields; and adding the second business data information to the message queue. According to the order of the service information in the message queue, multiple listening units are instructed to perform data collection operations to obtain the traffic data of the multiple target services.

2. The method according to claim 1, characterized in that, When the billing type for the target service is peak billing, the method further includes: Based on the target service identification information, extract the traffic data of the target service from the target database; Based on the traffic data of the target service, calculate the peak data of the target service; The peak data of the target service is stored in the target database.

3. The method according to claim 2, characterized in that, The calculation of peak data for the target service based on the traffic data of the target service and the target account information of the target service includes: Based on the peak period of the target service's traffic data, the traffic data of the target service with the highest value within the peak period is taken as the peak data of the target service.

4. A data acquisition device, characterized in that, The device includes: a communication unit and a processing unit; The communication unit is used to acquire service information of multiple target services; the target services are services subscribed to by the customer. The processing unit is configured to determine a message queue based on the service information of the plurality of target services; the message queue is configured to store the service information of the plurality of target services; the service information includes at least one of collection information or first service data information; the collection information is configured to instruct the monitoring unit to acquire traffic data of the plurality of target services; the first service data information is the initial information of the traffic data of the target services; When the service information is the collection information, the communication unit is further configured to acquire the collection information of each target service among the plurality of target services; the collection information of each target service includes the target account information and target service identification information of the corresponding target service; the processing unit is further configured to add the collection information of each target service to a message queue; When the service information is the first service data information, the communication unit is further configured to acquire the first service data information of each of the plurality of target services; the processing unit is further configured to perform data processing on the first service data information to obtain second service data information; the second service data information is the first service data information with complete fields; the processing unit is further configured to add the second service data information to the message queue. The processing unit is also configured to instruct multiple listening units to perform data collection operations according to the order of the service information in the message queue, so as to obtain the traffic data of the multiple target services.

5. The apparatus according to claim 4, characterized in that, When the billing type for the target service is peak billing, The processing unit is further configured to extract traffic data of the target service from the target database based on the target service identification information; The processing unit is also configured to calculate the peak data of the target service based on the traffic data of the target service; The processing unit is also used to store the peak data of the traffic data of the target service into the target database.

6. The apparatus according to claim 5, characterized in that, The processing unit is also used for: Based on the peak period of the target service's traffic data, the traffic data of the target service with the highest value within the peak period is taken as the peak data of the target service.

7. A data acquisition device, characterized in that, include: A processor and a communication interface; the communication interface is coupled to the processor, the processor being used to run computer programs or instructions to implement the data acquisition method as described in any one of claims 1-3.

8. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores instructions that, when executed by a computer, perform the data acquisition method as described in any one of claims 1-3.

Images