Batch data transmission method and device and electronic equipment

Abstract

The invention provides a batch data transmission method and device and electronic equipment, and relates to the technical field of data processing. The method comprises the steps that received data are written into a current to-be-written file within the current period of batch data receiving, and the current to-be-written file is any to-be-written file in a plurality of preset to-be-written files; detecting whether the current to-be-written file meets a preset transmission condition or not; if the current to-be-written file meets the preset transmission condition, determining the current to-be-written file as a to-be-read file; and before the current period is ended, selecting any file to be written from the plurality of files to be written as a new current file to be written, and continuously executing the operation of writing the received data into the current file to be written. According to the embodiment of the invention, the occupied resources of the cloud server are reduced, so that the cloud server can uniformly and stably transmit data, enough resources are provided to support the processing of other services, and the performance of the cloud server is kept stable.

Description

Technical Field

[0001] The present disclosure belongs to the technical field of data processing, and particularly relates to a method, an apparatus, and an electronic device for transmitting batch data. Background Art

[0002] In the scenario where a cloud server manages a device cluster, the data of each device in the device cluster is managed by the cloud server. In one implementation scenario, each device in the device cluster usually uploads data to the cloud server at a specific moment, causing the amount of data to be processed by the cloud server to increase explosively in a short period of time.

[0003] As a result, on the one hand, it is difficult for the cloud server to quickly process this data, resulting in a large delay. On the other hand, the explosively growing data occupies the resources of the cloud server, making the cloud server unable to process other services, thereby causing the performance of the cloud service to be unstable. Summary of the Invention

[0004] The present disclosure provides a method, an apparatus, and an electronic device for transmitting batch data. By presetting a plurality of files to be written and the transmission conditions corresponding to the files to be written, during the period of batch receiving data, the received data is written into the files to be written in batches, and then the data in the files to be written is read for transmission, so as to cache and transmit the explosively growing data in batches, which is beneficial to reducing the resources occupied by the cloud server, enabling the cloud server to transmit data evenly and stably, and having sufficient resources to support the processing of other services, so that the performance of the cloud server remains stable.

[0005] A first aspect embodiment of the present disclosure provides a method for transmitting batch data, the method including:

[0006] During the current period of batch receiving data, write the received data into the current file to be written, where the current file to be written is any one of the preset plurality of files to be written;

[0007] Detect whether the current file to be written meets the preset transmission conditions, where the preset transmission conditions include at least one of the following: the amount of data contained in the current file to be written reaches a quantity threshold, the writing duration of the current file to be written reaches a transmission time interval; the quantity threshold or the transmission time interval is preset in advance, the quantity threshold is greater than or equal to the product of the transmission time interval and the unit data transmission amount, and the unit data transmission amount refers to the amount of data that should be transmitted in a preset unit duration within the duration of the current period;

[0008] If the current file to be written meets the preset transmission conditions, determine the current file to be written as the file to be read to transmit the data in the file to be read;

[0009] Before the end of the current cycle, select any one of the multiple files to be written as the new current file to be written, and continuously perform the operation of writing the received data into the current file to be written.

[0010] In the embodiments of the present disclosure, the detecting whether the current file to be written meets the preset transmission condition includes:

[0011] Judging whether the amount of data contained in the current file to be written reaches the quantity threshold;

[0012] If the amount of data contained in the current file to be written reaches the quantity threshold, determine that the current file to be written meets the preset transmission condition;

[0013] If the amount of data contained in the current file to be written does not reach the quantity threshold, judge whether the writing duration of the current file to be written reaches the transmission time interval;

[0014] If the writing duration of the current file to be written reaches the transmission time interval, determine that the current file to be written meets the preset transmission condition.

[0015] In the embodiments of the present disclosure, before the detecting whether the current file to be written meets the preset transmission condition, it further includes:

[0016] Determine the total amount of data of the data to be batch-received in the current cycle;

[0017] Use the total amount of data and the duration of the current cycle except to obtain the unit data transmission amount;

[0018] Determine the quantity threshold and the transmission time interval according to the unit data transmission amount.

[0019] In the embodiments of the present disclosure, the determining the quantity threshold and the transmission time interval according to the unit data transmission amount includes:

[0020] If the preset file data amount is determined as the quantity threshold, determine the integer part of the quotient of dividing the quantity threshold by the unit data transmission amount as the transmission time interval; or,

[0021] If the preset transmission period is determined as the transmission time interval, determine the product of the transmission time interval and the unit data transmission amount as the quantity threshold.

[0022] In the embodiments of the present disclosure, the determining the total amount of data of the data to be batch-received in the current cycle includes:

[0023] If the current cycle is the first cycle of performing batch data reception, determine the rated data amount corresponding to the network card as the total amount of data;

[0024] If the current cycle is the i-th cycle for batch receiving data, determine the total amount of data received in the (i - 1)-th cycle as the total amount of data, where i is an integer greater than or equal to 2.

[0025] In an embodiment of the present disclosure, the writing the received data into the current file to be written includes:

[0026] Generate a data packet from the received data according to a preset format, where the data packet includes the source device identifier of the data, the destination device identifier of the data, and the data;

[0027] Write the data packet into the current file to be written.

[0028] In an embodiment of the present disclosure, it further includes:

[0029] If there is a file to be read currently stored, read the data packets in the file to be read;

[0030] Forward the data included in the data packet to the device indicated by the destination device identifier according to the destination device identifier included in the data packet;

[0031] After all the data in the file to be read is transmitted, delete the file to be read.

[0032] In an embodiment of the present disclosure, it further includes:

[0033] Detect the number of remaining files to be written among the multiple files to be written;

[0034] If the number of remaining files to be written is less than or equal to a first preset number of files, create a new file to be written so that the total number of files to be written reaches a second preset number of files.

[0035] An embodiment of the second aspect of the present disclosure provides a device for transmitting batch data, where the device includes:

[0036] A writing module, configured to write the received data into the current file to be written during the current cycle of batch receiving data, where the current file to be written is any one of a preset multiple files to be written;

[0037] A detection module, configured to detect whether the current file to be written meets a preset transmission condition, where the preset transmission condition includes at least one of the following: the amount of data included in the current file to be written reaches a quantity threshold, and the writing duration of the current file to be written reaches a transmission time interval; the quantity threshold or the transmission time interval is preset in advance, the quantity threshold is greater than or equal to the product of the transmission time interval and the unit data transmission amount, and the unit data transmission amount refers to the amount of data that should be transmitted during a preset unit duration within the duration of the current period;

[0038] A determination module, configured to, if the current file to be written meets the preset transmission condition, determine the current file to be written as a file to be read, so as to transmit the data in the file to be read;

[0039] A selection module, configured to, before the end of the current period, select any one of the multiple files to be written as the new current file to be written, and continuously execute the operation of writing the received data into the current file to be written.

[0040] An embodiment of the third aspect of the present disclosure provides an electronic device, including a memory, a processor, and a computer program stored on the memory and executable on the processor, where the processor runs the computer program to implement the method described in the first aspect above.

[0041] An embodiment of the fourth aspect of the present disclosure provides a computer-readable storage medium, on which a computer program is stored, and the program is executed by a processor to implement the method described in the first aspect above.

[0042] The technical solution provided in the embodiments of the present disclosure has at least the following technical effects or advantages:

[0043] In the embodiments of the present disclosure, a cloud server pre-sets multiple files to be written. During the current period of batch receiving data, the transmission conditions of the files to be written can be determined according to the total amount of data received in batch. The transmission conditions include that the amount of data included in the file to be written reaches a quantity threshold, and the writing duration of the file to be written reaches a transmission time interval. Furthermore, the file to be written that meets the transmission condition is used as the file to be read, and the data in the file to be read is transmitted. In this way, it is supported to write the batch-received data into files in batches and transmit them in batches by reading the files, so as to deploy the transmission process of the batch data throughout the duration of the period, thereby reducing the instantaneous storage and processing pressure of the cloud server, enabling the cloud server to transmit data evenly and stably, and having sufficient resources to support the processing of other services, so that the performance of the cloud server remains stable.

[0044] The additional aspects and advantages of the present disclosure will be partially given in the following description, partially become obvious from the following description, or be understood through the practice of the present disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0045] By reading the following detailed description of the preferred embodiments, various other advantages and benefits will become clear to those of ordinary skill in the art. The drawings are only for the purpose of showing the preferred embodiments and are not considered to be a limitation of the present disclosure. Also, throughout the drawings, the same reference numerals are used to represent the same components.

[0046] In the drawings:

[0047] Figure 1A The structural schematic diagram of the cloud system provided by an embodiment of the present disclosure is shown;

[0048] Figure 1B The schematic diagram of the time-series change of the amount of data to be processed by the cloud server provided by an embodiment of the present disclosure is shown;

[0049] Figure 2 The schematic diagram of the method flow of a method for transmitting batch data provided by an embodiment of the present disclosure is shown;

[0050] Figure 3 The schematic diagram of the exemplary data flow of the method for transmitting batch data provided by an embodiment of the present disclosure is shown;

[0051] Figure 4A Shows Figure 3 An exemplary method flow diagram of writing data in;

[0052] Figure 4B Shows Figure 3 An exemplary method flow diagram of reading data in;

[0053] Figure 5 The structural schematic diagram of the device for transmitting batch data provided by an embodiment of the present disclosure is shown;

[0054] Figure 6 The structural schematic diagram of an electronic device provided by an embodiment of the present disclosure is shown;

[0055] Figure 7 The schematic diagram of a storage medium provided by an embodiment of the present disclosure is shown. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0056] The exemplary embodiments of the present disclosure will be described in more detail below with reference to the drawings. Although the exemplary embodiments of the present disclosure are shown in the drawings, it should be understood that the present disclosure can be implemented in various forms and should not be limited by the embodiments set forth herein. On the contrary, these embodiments are provided so that the present disclosure can be more thoroughly understood and the scope of the present disclosure can be fully communicated to those skilled in the art.

[0057] It should be noted that, unless otherwise specified, the technical terms or scientific terms used in this disclosure shall have the ordinary meanings understood by those skilled in the art to which this disclosure pertains.

[0058] The implementation scenarios and related technologies involved in the embodiments of this disclosure are introduced below.

[0059] The technical solution of this disclosure relates to a cloud system. Refer to Figure 1A , Figure 1A FIG. shows a schematic structural diagram of a cloud system provided by an embodiment of this disclosure. The cloud system may include a cloud server 1000 and a device cluster 2000. The cloud server 1000 manages the data of each device in the device cluster 2000. For example, each device in the device cluster 2000 may forward data via the cloud server 1000; for another example, any device in the device cluster 2000 may forward data to other devices via the cloud server 1000. Among them, the cloud server 1000 may be an independent physical server, or a server cluster, a distributed system or a cloud platform composed of multiple physical servers. The cloud server 1000 may deploy resources and computing power for network content detection, etc. The device cluster 2000 may include electronic devices such as mobile terminal devices, smart home devices, Internet of Things devices, sensors, and monitoring cameras.

[0060] In a conventional implementation, each device in the device cluster 2000 uploads data to the cloud server 1000 centrally at a specific moment. The specific moment may be a moment with 0 or 5 as the mantissa, such as moments every 5 minutes like 10:00, 10:05, 10:10, 10:15... etc. The device cluster 2000 usually contains numerous devices. These devices uploading data centrally at the same moment will cause the cloud server 1000 to store a large amount of data in a short time. In this case of explosive growth of the data volume, the performance of the cloud server 1000 cannot support the rapid processing of this data, resulting in the data being stored in the memory for a long time, not only occupying the memory resources of the cloud server 1000, but also occupying the overhead of the central processing unit (CPU) of the cloud server 1000, making the cloud server 1000 unable to process other services.

[0061] To solve the problems brought about by the explosive growth of the data volume, a commonly used processing method is to improve the processing performance of the cloud service server or increase the number of cloud services. However, referring to Figure 1B the time-series change of the data volume to be processed by the cloud server shown can be obtained. The situation of explosive growth and backlog of the data volume only exists in a short time. Most of the time, the performance of the cloud server can meet the usage. It can be seen that the methods of improving the processing performance of the cloud service server and increasing the number of cloud services will cause waste of the performance of the cloud service.

[0062] In view of this, in the technical solution proposed by the embodiments of the present disclosure, the cloud server presets a plurality of files to be written and the corresponding transmission conditions of the files to be written. During the period of batch receiving data, the received data is written into the files to be written in batches, and then the data in the files to be written is read for transmission, so as to cache and transmit the explosively growing data in batches, which is beneficial to reducing the resources occupied by the cloud server, enabling the cloud server to transmit data evenly and stably, and having sufficient resources to support the processing of other services, so that the performance of the cloud server remains stable.

[0063] The following describes a method, device, electronic device, and storage medium for transmitting batch data according to the embodiments of the present disclosure. The technical solution of the present disclosure will be described in detail below with specific embodiments. These specific embodiments may be combined with each other, and the same or similar concepts or processes may not be repeated in some embodiments. The embodiments of the present disclosure will be described below with reference to the drawings.

[0064] See Figure 2 , Figure 2 shows the method for transmitting batch data provided by the present disclosure, and this method can be applied to a cloud server. The method includes:

[0065] In step S11, during the current period of batch receiving data, the received data is written into the current file to be written.

[0066] Wherein, the current file to be written is any one of the preset plurality of files to be written.

[0067] It should be noted that in the embodiments of the present disclosure, after the cloud server is started and before batch receiving data, a file pool to be written can be created. The file pool to be written can include a second preset number of files to be written, and these files to be written can be empty files without any data. Any file to be written is used to write the data received by the cloud server.

[0068] In some embodiments, the second preset number of files can be 2. In this way, during the current period of batch receiving data and at the stage of starting to batch receive data, the cloud server can open any file to be written to write data without creating a file to be written, improving the efficiency of writing data.

[0069] It should be understood that if the cloud server directly receives data, the received data is usually written into the memory, and if the data is stored through a file, the file can usually be stored on the hard disk or disk. It can be seen that by adopting this implementation method, storing data through a file can store batch data on the hard disk, thereby being able to relieve the long-term occupation of a large amount of data on the memory and being beneficial to supporting the stable operation of the CPU.

[0070] Combined with Figure 1A As can be seen from the technical scenario shown, the cloud server receives data from multiple devices, and the destination addresses of different data are also different. To facilitate the identification and forwarding of this data, the cloud server can generate data packets according to a preset format for the received data. The data packet includes the source device identifier of the data, the destination device identifier of the data, and the data. Then, the data packet is written into the current file to be written.

[0071] Exemplarily, the data packet combined according to the preset format can be the source device identifier, delimiter, destination device identifier, delimiter, and data combined in sequence. Among them, the source device identifier is, for example, the device identifier of the data sending device, the destination device identifier is, for example, the device identifier of the data receiving device, and the delimiter can be any symbol, such as "#" or "*". For example, if the data reported by device D1 is Data1, the device identifier of device D1 is ID123, and the destination device identifier of Data1 is ID567, then the data packet of Data1 generated according to the preset format is: ID123#ID567#Data1.

[0072] In step S12, it is detected whether the current file to be written meets the preset transmission conditions.

[0073] Among them, the preset transmission conditions include at least one of the following: the amount of data contained in the current file to be written reaches a quantity threshold, the writing duration of the current file to be written reaches a transmission time interval. The quantity threshold or the transmission time interval is set in advance. The quantity threshold is greater than or equal to the product of the transmission time interval and the unit data transmission amount. The unit data transmission amount refers to the amount of data that should be transmitted in a preset unit duration within the duration of the current period.

[0074] Combined with the description of the cycle of the device cluster uploading data centrally mentioned above, the cycle of the device cluster uploading data centrally has a certain duration. And the embodiments of the present disclosure aim to transmit the batch of data received this time in batches and evenly within the duration of this cycle by means of file storage. Exemplarily, the total amount of data batch-received by the cloud server is 1 Gbps (Giga Bits Per Second), and the time interval between two adjacent data upload times is, for example, 5 minutes (i.e., the cycle duration is 5 minutes). Then, if the 1 Gbps of data is sent in batches and evenly within these 5 minutes, for example, 0.2 Gbps of data can be sent every 1 minute.

[0075] In view of this, before detecting whether the current file to be written meets the preset transmission conditions, the cloud server may determine the total amount of data to be batch-received in the current period, and obtain the unit data transmission amount by using the total amount of data and the duration of the current period except for the current period, and determine the quantity threshold and the transmission time interval according to the unit data transmission amount.

[0076] Wherein, the unit data transmission amount refers to the amount of data that should be transmitted within a preset unit duration in the current period, that is, the data transmission rate of the cloud server to data, and the preset unit duration may be seconds.

[0077] It should be noted that although the total amount of data uploaded by each device in the device cluster in different periods may be different, however, in the case where the number of devices included in the device cluster is large, the total amount of data uploaded by the device cluster in each period is relatively close. Based on this, if the current period described in step S11 is the first period for batch-receiving data, the cloud server may determine the rated data amount corresponding to the network card as the total amount of data. And if the current period described in step S11 is the i-th period for batch-receiving data, the total amount of data received in the (i - 1)-th period is determined as the total amount of data, where i is an integer greater than or equal to 2.

[0078] In an actual implementation scenario, during the process of transmitting data in batches, the amount of data transmitted each time and the transmission time interval between two adjacent transmissions should be considered. For example, in order to ensure that the cloud server does not occupy too much overhead during the process of transmitting data for each file, the amount of data written in each file should not be too large. In addition, to ensure that all the data received this time is transmitted before the next batch-receiving of data, the amount of data written in each file should not be too small, and the transmission time interval between the data in two adjacent files should not be too large. That is, the quantity threshold and the transmission time interval are related.

[0079] In some embodiments, the quantity threshold may be specified in advance. Furthermore, the cloud server may calculate the transmission time interval according to the quantity threshold specified in advance. In other embodiments, the transmission time interval may be specified in advance. Furthermore, the cloud server may calculate the quantity threshold according to the transmission time interval specified in advance. Exemplarily, if the preset file data amount is determined as the quantity threshold, the integer part of the quotient obtained by dividing the quantity threshold by the unit data transmission amount is determined as the transmission time interval. If the preset transmission period is determined as the transmission time interval, the product of the transmission time interval and the unit data transmission amount is determined as the quantity threshold.

[0080] Exemplarily, the total amount of data to be received corresponding to the current period is, for example, Size (e.g., 1 Gbps), the period duration is Duration, and the data transmission rate Speed of the cloud server can satisfy: Speed = Size / Duration. Further, the quantity threshold Filesize of the file to be written and the transmission time interval Time can be determined according to the corresponding relationship Filesize = Time * Speed. Among them, Filesize or Time can be specified in advance.

[0081] In some embodiments, detecting whether the current file to be written meets the preset transmission conditions can be implemented as: determining whether the amount of data contained in the current file to be written reaches the quantity threshold. If the amount of data contained in the current file to be written reaches the quantity threshold, it is determined that the current file to be written meets the preset transmission conditions; if the amount of data contained in the current file to be written does not reach the quantity threshold, it can be considered that the current file to be written does not meet the preset transmission conditions, and data can continue to be written to the current file to be written.

[0082] In some other embodiments, the current file to be written can be the last written file before the next batch of data reception. In this scenario, the amount of data that can be written to the current file to be written may never reach the quantity threshold. To transmit the data in the current file to be written in a timely manner, if the amount of data contained in the current file to be written does not reach the quantity threshold, the cloud server can determine whether the writing duration of the current file to be written reaches the transmission time interval. If the writing duration of the current file to be written reaches the transmission time interval, it is determined that the current file to be written meets the preset transmission conditions. If the writing duration of the current file to be written does not reach the transmission time interval, the timing of the writing duration of the current file to be written can continue.

[0083] In step S13, if the current file to be written meets the preset transmission conditions, the current file to be written is determined as the file to be read.

[0084] Among them, the file to be read refers to a file that contains data to be transmitted, and the cloud server should read and forward this data to be transmitted. In some embodiments, the file to be read can be stored in a preset file pool to be read, and the cloud server reads and forwards the file in the file pool to be read. The file pool to be read can be created during the stage when the cloud server creates the file pool to be written, and the file pool to be read can be empty in the initial state.

[0085] Exemplarily, the file pool to be written and the file pool to be read can be implemented in the form of a folder, and the embodiments of the present disclosure do not make specific limitations on this.

[0086] In some embodiments, the cloud server may detect whether there is a file to be read currently stored. If there is a file to be read currently stored, the cloud server reads data packets in the file to be read, and then forwards the data included in the data packets to the device indicated by the destination device identifier included in the data packets according to the destination device identifier. After all the data in the file to be read has been transmitted, the file to be read may be deleted.

[0087] It should be understood that for the above processes of writing data and reading data, the cloud server may execute them with two threads, and these two threads may be executed in parallel.

[0088] In step S14, before the end of the current cycle, any one of the multiple files to be written is selected as the new current file to be written, and step S11 is continued to be executed.

[0089] Among them, after storing the current file to be written as a file to be read in the file pool to be read, before the end of the current cycle, the cloud server may obtain a new file to be written from the multiple files to be written as the current file to be written, and continue to execute step S11 until all the batch data in the current cycle has been received.

[0090] As can be seen from the above embodiments, the number of files to be written in the file pool to be written is continuously decreasing. To avoid insufficient files to be written during the process of receiving data, during the execution of the above embodiments, the cloud server may detect the number of remaining files to be written among the multiple files to be written. If the number of the remaining files to be written is less than or equal to the first preset number of files, a new file to be written is created so that the total number of files to be written reaches the second preset number of files. The first preset number of files may be 1, for example.

[0091] It can be seen that by adopting the technical solution of the embodiments of the present disclosure, the batch-received data is stored in a hard disk or a disk in the form of files, which can relieve the long-term occupation of memory by a large amount of data. In addition, by writing data into files to be written in batches and using the files to be written that meet the transmission conditions as files to be read, it is beneficial to achieve the effect of fast forward and slow output of batch data, which is thus beneficial to supporting the stable operation of the CPU and having sufficient resources to support the processing of other services, so that the performance of the cloud server remains stable.

[0092] The technical solution of the embodiments of the present disclosure will be described below with reference to examples.

[0093] Refer to Figure 3 , according to the execution time sequence, after establishing a communication connection with the device cluster and before receiving the first centralized upload of data from the device cluster, the cloud server may create a file pool to be written and a file pool to be read in the hard disk, and may create 10 files to be written in the file pool to be written.

[0094] After creating the pool of files to be written and the pool of files to be read, for any scenario of batch receiving data from a device cluster, after determining the total amount of data to be received this time, the cloud server can calculate the quantity threshold Filesize and the transmission time interval Time. Furthermore, referring again to Figure 3 In one execution thread, the cloud server obtains the current file to be written from the pool of files to be written, and writes data into the current file to be written. When the current file to be written meets the conditions corresponding to the quantity threshold Filesize and the transmission time interval Time, the current file to be written is stored as a file to be read in the pool of files to be read. In another execution thread, the cloud server obtains the file to be read from the pool of files to be read, and forwards the data in the file to be read.

[0095] For example, if the total amount of data to be received this time is 10 Mbp and the Duration is 5 minutes, the Filesize of each file to be written can be set to 1 Mbp, and the maximum writing time of the file to be written is 1 minute.

[0096] It should be noted that Figure 3 In the illustrated scenario, the processing process of the cloud server for the pool of files to be written and the processing process for the pool of files to be read are independent of each other and have no interaction.

[0097] Regarding the processing process for the pool of files to be written, referring to Figure 4A as shown, the process of writing data by the cloud server may include steps S411 to S415.

[0098] Step S411: In response to receiving data, obtain the current file to be written from the pool of files to be written.

[0099] Step S412: Write data into the current file to be written.

[0100] Taking the data uploaded by device 1 as an example, the identifier of device 1 is ID13148, the destination of the data reported by device 1 is the device with the identifier DID13, and device 1 continuously reports data as Data1 to Data10. The length of each Data is, for example, 100 kbp. The cloud server can, for example, encapsulate Data1 into a data packet with the format ID13148#DID13#Data1 and write it into the current file to be written.

[0101] Step S413: Determine whether the data size in the current file to be written is greater than or equal to 1 Mbp; if so, execute step S415; if not, execute step S414;

[0102] Step S414: Determine whether the writing time of the current file to be written is greater than or equal to 1 minute; if so, execute step S415; if not, continue to execute step S412;

[0103] Step S415: Store the current file to be written into the file pool to be read, and obtain a new file to be written from the file pool to be read as the current file to be written.

[0104] For example, if the data in the current file to be written is 9900 kbp and the writing duration of the current file to be written is 9 seconds at this time, it can be determined that the data in the current file to be written is less than 10 Mbp and the writing duration is also less than 1 minute. Then, the cloud server can continue to write data into the current file to be written.

[0105] For another example, after writing data into the current file to be written, if the data in the current file to be written is greater than 10 Mbp, the cloud server can close the file to be written and store the file in the file pool to be read.

[0106] In addition, it should be noted that after step S415, the cloud server can determine whether the number of files to be written in the file pool to be written is less than or equal to 2. If the number of files to be written in the file pool to be written is less than or equal to 2, 8 files to be written can be created and added to the file pool to be written.

[0107] Regarding the processing procedure for the file pool to be read, refer to Figure 4B As shown, the process of the cloud server reading data may include step S421 to step S423.

[0108] Step S421: Detect whether there is a file to be read in the file pool to be read.

[0109] Step S422: If there is a file to be read in the file pool to be read, read and forward the data in the file to be read item by item.

[0110] For example, taking ID13148#DID13#Data1 as an example, the cloud server can parse out the source device identifier ID13148, the destination device identifier DID13, and the data Data1, and forward the data Data1 at a speed of 1 Mbps.

[0111] Step S423: After all the data in the file to be read has been forwarded, delete the file to be read from the file pool to be read.

[0112] In summary, in the embodiments of the present disclosure, the cloud server pre-sets multiple files to be written. During the current cycle of batch receiving data, the transmission conditions of the files to be written can be determined according to the total amount of data received in batch. The transmission conditions include that the amount of data contained in the file to be written reaches a quantity threshold, and the writing duration of the file to be written reaches a transmission time interval. Furthermore, the file to be written that meets the transmission conditions is used as the file to be read, and the data in the file to be read is transmitted. In this way, it is supported to write the batch-received data into files in batches and transmit it in batches by reading the files, so as to deploy the transmission process of the batch data throughout the cycle duration, thereby reducing the instantaneous storage and processing pressure of the cloud server, enabling the cloud server to transmit data evenly and stably, and having sufficient resources to support the processing of other services, so that the performance of the cloud server remains stable.

[0113] Corresponding to the batch data transmission method illustrated in the above embodiments, the embodiments of the present disclosure further provide a batch data transmission device, which can be deployed on the cloud server side and is used to execute the above Figures 2 to 4B operations of the batch data transmission method of any one of the embodiments, as Figure 5 shown, the device includes:

[0114] A writing module 501, configured to write the received data into the current file to be written during the current cycle of batch receiving data, where the current file to be written is any one of the pre-set multiple files to be written;

[0115] A detection module 502, configured to detect whether the current file to be written meets the preset transmission conditions, where the preset transmission conditions include at least one of the following: the amount of data contained in the current file to be written reaches a quantity threshold, and the writing duration of the current file to be written reaches a transmission time interval; the quantity threshold or the transmission time interval is pre-set, and the quantity threshold is greater than or equal to the product of the transmission time interval and the unit data transmission amount, where the unit data transmission amount refers to the amount of data that should be transmitted during a preset unit duration within the duration of the current cycle;

[0116] A determination module 503, configured to, if the current file to be written meets the preset transmission conditions, determine the current file to be written as the file to be read to transmit the data in the file to be read;

[0117] A selection module 504, configured to select any one of the multiple files to be written as the new current file to be written before the end of the current cycle, and continuously execute the operation of writing the received data into the current file to be written.

[0118] In the embodiments of the present disclosure, the detection module 502 is further configured to determine whether the amount of data contained in the current file to be written reaches the quantity threshold;

[0119] If the amount of data included in the current file to be written reaches the quantity threshold, it is determined that the current file to be written meets the preset transmission condition;

[0120] If the amount of data included in the current file to be written does not reach the quantity threshold, it is judged whether the writing duration of the current file to be written reaches the transmission time interval;

[0121] If the writing duration of the current file to be written reaches the transmission time interval, it is determined that the current file to be written meets the preset transmission condition.

[0122] In an embodiment of the present disclosure, the device further includes: a calculation module. The determination module 503 is further configured to determine the total amount of the data to be batch-received in the current period; the calculation module is configured to obtain the unit data transmission amount by using the total amount of the data and dividing by the duration of the current period; the determination module 503 is further configured to determine the quantity threshold and the transmission time interval according to the unit data transmission amount.

[0123] In an embodiment of the present disclosure, the determination module 503 is further configured to, if a preset file data amount is determined as the quantity threshold, determine the integer part of the quotient obtained by dividing the quantity threshold by the unit data transmission amount as the transmission time interval; the determination module 503 is further configured to, if a preset transmission period is determined as the transmission time interval, determine the product of the transmission time interval and the unit data transmission amount as the quantity threshold.

[0124] In an embodiment of the present disclosure, the determination module 503 is further configured to, if the current period is the first period for performing batch data reception, determine the rated data amount corresponding to the network card as the total amount of the data; the determination module 503 is further configured to, if the current period is the i-th period for performing batch data reception, determine the total amount of the data received in the (i - 1)-th period as the total amount of the data, where i is an integer greater than or equal to 2.

[0125] In an embodiment of the present disclosure, the writing module 501 is further configured to generate a data packet in a preset format according to the received data, where the data packet includes the source device identifier of the data, the destination device identifier of the data, and the data; and write the data packet into the current file to be written.

[0126] In an embodiment of the present disclosure, the device further includes: a reading module, a forwarding module, and a deleting module. The reading module is configured to read the data packet in the file to be read if there is a file to be read stored currently; the forwarding module is configured to forward the data included in the data packet to the device indicated by the destination device identifier according to the destination device identifier included in the data packet; the deleting module is configured to delete the file to be read after all the data in the file to be read is transmitted.

[0127] In an embodiment of the present disclosure, the device further includes: a file creation module. The detection module 502 is further configured to detect the number of remaining files to be written among the multiple files to be written; the file creation module is configured to create a new file to be written if the number of remaining files to be written is less than or equal to a first preset number of files, so that the total number of files to be written reaches a second preset number of files.

[0128] The batch data transmission device provided in the above embodiment of the present disclosure and the batch data transmission method provided in the embodiment of the present disclosure are based on the same inventive concept and have the same beneficial effects as the methods adopted, run or implemented by the application programs stored therein.

[0129] The present disclosure also provides an electronic device to execute the above batch data transmission method. Please refer to Figure 6 , which shows a schematic diagram of an electronic device provided in some embodiments of the present disclosure. As Figure 6 shown, the electronic device 6 includes: a processor 600, a memory 601, a bus 602, and a communication interface 603. The processor 600, the communication interface 603, and the memory 601 are connected through the bus 602; a computer program that can run on the processor 600 is stored in the memory 601, and when the processor 600 runs the computer program, it executes the batch data transmission method provided in any of the foregoing embodiments of the present disclosure.

[0130] Among them, the memory 601 may include a high-speed random access memory (RAM: Random Access Memory), and may also include a non-volatile memory, such as at least one disk memory. The communication connection between the system virtual devices is realized through at least one communication interface 603 (which can be wired or wireless), and the Internet, wide area network, local area network, metropolitan area network, etc. can be used.

[0131] The bus 602 may be an ISA bus, a PCI bus, an EISA bus, etc. The bus can be divided into an address bus, a data bus, a control bus, etc. Among them, the memory 601 is used to store a program, and after receiving an execution instruction, the processor 600 executes the program. The batch data transmission method disclosed in any of the foregoing embodiments of the present disclosure can be applied to the processor 600 or implemented by the processor 600.

[0132] The processor 600 may be an integrated circuit chip with the ability to process signals. In the implementation process, each step of the above method may be completed by the integrated logic circuit of the hardware in the processor 600 or the instructions in the form of software. The above-mentioned processor 600 may be a general-purpose processor, including a central processing unit (CPU for short), a network processor (NP for short), etc.; it may also be a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components. It can implement or execute each method, step and logic block diagram disclosed in the embodiments of the present disclosure. The general-purpose processor may be a microprocessor or the processor may also be any conventional processor, etc. The steps of the method disclosed in combination with the embodiments of the present disclosure may be directly embodied as being executed and completed by a hardware decoding processor, or executed and completed by a combination of the hardware and software modules in the decoding processor. The software module may be located in a mature storage medium in the art such as a random access memory, a flash memory, a read-only memory, a programmable read-only memory or an electrically erasable programmable memory, a register, etc. This storage medium is located in the memory 601, and the processor 600 reads the content in the memory 601 and combines its hardware to complete the steps of the above method.

[0133] The electronic device provided by the embodiments of the present disclosure and the method for transmitting batch data provided by the embodiments of the present disclosure are based on the same inventive concept and have the same beneficial effects as the method adopted, run or implemented by it.

[0134] The embodiments of the present disclosure also provide a computer-readable storage medium corresponding to the method for transmitting batch data provided by the foregoing embodiments. Please refer to Figure 7 which shows that the computer-readable storage medium is an optical disc 30, on which a computer program (i.e., a program product) is stored. When the computer program is run by a processor, it will execute the method for transmitting batch data provided by any of the foregoing embodiments.

[0135] It should be noted that examples of the computer-readable storage medium may also include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other optical and magnetic storage media, which will not be elaborated here one by one.

[0136] The computer-readable storage medium provided by the above embodiments of the present disclosure and the method for transmitting batch data provided by the embodiments of the present disclosure are based on the same inventive concept and have the same beneficial effects as the methods adopted, run, or implemented by the application programs stored therein.

[0137] Although the optional embodiments of the present disclosure have been described, those skilled in the art can make additional changes and modifications to these embodiments once they learn the basic creative concept. Therefore, the appended claims are intended to be construed as including the preferred embodiments and all changes and modifications falling within the scope of the present disclosure.

[0138] The specific embodiments described above further elaborate on the purpose, technical solutions, and beneficial effects of the present disclosure. It should be understood that the above description is only the specific embodiments of the present disclosure and is not used to limit the protection scope of the present disclosure. Any modifications, equivalent replacements, improvements, etc. made on the basis of the technical solutions of the present disclosure shall be included in the protection scope of the present invention.

Claims

1. A method for transmitting batch data, characterized in that: The method comprises: In a current cycle of receiving data in batches, the received data is written into a current file to be written, where the current file to be written is any file to be written among a plurality of preset files to be written; Detecting whether the current file to be written meets a preset transmission condition, where the preset transmission condition includes at least one of the following: the amount of data contained in the current file to be written reaches a quantity threshold, and the writing duration of the current file to be written reaches a transmission time interval; the quantity threshold or the transmission time interval is preset, and the quantity threshold is greater than or equal to the product of the transmission time interval and a unit data transmission amount, where the unit data transmission amount refers to the amount of data that should be transmitted in a preset unit time interval within the duration of the current cycle; If the current file to be written meets the preset transmission condition, the current file to be written is determined as a file to be read, so as to transmit the data in the file to be read; Before the current cycle ends, any file to be written is selected from the multiple files to be written as a new current file to be written, and the operation of writing the received data into the current file to be written is continuously performed.

2. The method according to claim 1, characterized in that The detecting whether the current file to be written meets the preset transmission condition includes: Determine whether the amount of data contained in the current file to be written reaches the amount threshold; If the amount of data contained in the current file to be written reaches the amount threshold, determining that the current file to be written meets the preset transmission condition; If the amount of data contained in the current file to be written does not reach the amount threshold, determining whether the writing time of the current file to be written reaches the transmission time interval; If the writing time length of the current file to be written reaches the transmission time interval, it is determined that the current file to be written meets the preset transmission condition.

3. The method according to claim 1 or 2, characterized in that: Before detecting whether the current file to be written meets the preset transmission condition, the method further includes: Determine the total amount of data to be received in batches in the current week; The unit data transmission amount is obtained by dividing the total amount of data by the duration of the current cycle; The quantity threshold and the transmission time interval are determined according to the unit data transmission volume.

4. The method according to claim 3, characterized in that The determining the quantity threshold and the transmission time interval according to the unit data transmission amount includes: If the preset file data volume is determined as the quantity threshold, the integer part of the quotient of the quantity threshold divided by the unit data transmission volume is determined as the transmission time interval; or, If the preset transmission period is determined as the transmission time interval, the product of the transmission time interval and the unit data transmission amount is determined as the quantity threshold.

5. The method according to claim 3, characterized in that: The determining the total amount of data to be received in batches in the current week includes: If the current cycle is a cycle for the first execution of receiving data in batches, the rated data volume corresponding to the network card is determined as the total data volume; If the current cycle is the i-th cycle for executing batch data reception, the total amount of data received in the i-1-th cycle is determined as the total amount of data, where i is an integer greater than or equal to 2.

6. The method according to claim 1, characterized in that The step of writing the received data into the current file to be written comprises: Generate a data packet from the received data according to a preset format, the data packet including a source device identifier of the data, a destination device identifier of the data and the data; The data packet is written into the current file to be written.

7. The method according to claim 6, characterized in that Also includes: If there is a file to be read currently stored, read the data packet in the file to be read; According to the destination device identifier contained in the data packet, forwarding the data contained in the data packet to the device indicated by the destination device identifier; After all the data in the file to be read is transmitted, the file to be read is deleted.

8. The method according to claim 1, characterized in that Also includes: Detecting the number of remaining files to be written among the multiple files to be written; If the number of the remaining files to be written is less than or equal to the first preset number of files, a new file to be written is created so that the total number of the files to be written reaches the second preset number of files.

9. A device for transmitting batch data, characterized in that: The device comprises: A writing module, used for writing the received data into a current file to be written in a current cycle of receiving data in batches, wherein the current file to be written is any file to be written among a plurality of preset files to be written; A detection module, used to detect whether the current file to be written meets a preset transmission condition, wherein the preset transmission condition includes at least one of the following: the amount of data contained in the current file to be written reaches a quantity threshold, and the writing time of the current file to be written reaches a transmission time interval; the quantity threshold or the transmission time interval is preset, and the quantity threshold is greater than or equal to the product of the transmission time interval and a unit data transmission amount, and the unit data transmission amount refers to the amount of data that should be transmitted in a preset unit time within the duration of the current cycle; A determination module, configured to determine the current file to be written as a file to be read if the current file to be written meets the preset transmission condition, so as to transmit the data in the file to be read; The selection module is used to select any file to be written from the multiple files to be written as a new current file to be written before the current cycle ends, and continuously execute the operation of writing the received data into the current file to be written.

10. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: The processor executes the computer program to implement the method according to any one of claims 1 to 8.

Images