Data stream fragmentation dynamic transmission method, device, storage medium and electronic device

Abstract

The present application relates to a data stream segmentation dynamic transmission method, device, storage medium and electronic device, and relates to the field of data transmission technology, wherein the method includes: obtaining the actual data length and source address of the target data stream that needs to be uploaded this time; determining the upload configuration information structure corresponding to the target data stream, and assigning the actual data length to the data length variable of the upload configuration information structure, and obtaining the updated upload configuration information structure; according to the updated upload configuration information structure and the source address, calling the preset segmentation upload interface, the segmentation upload interface encapsulates the segmentation processing logic of the target data stream and the upload interface in the libcurl library; based on the segmentation upload interface and the target buffer area, the target data stream is segmented, the segmentation result of the target data stream is obtained, and the segmentation result is uploaded. The present application can improve the effect of data stream segmentation upload in resource-constrained devices.

Description

Technical Field

[0001] The present application relates to the technical field of data transmission, and in particular to a method, device, storage medium and electronic device for dynamic transmission of data stream fragments. Background Art

[0002] A data stream refers to a series of continuous data packets or data units transmitted in a computer network. A data stream can contain various types of information, such as text, pictures, audio, video, etc. Data stream is one of the basic forms of data transmission in a computer network, and it is of great significance for achieving efficient network communications and applications. Fragmented transmission is a data transmission technology that divides larger data into smaller fragments, that is, transmits them in fragments. At present, in the development of Linux-related security services, it is usually necessary to continuously collect traffic and generate traffic files, and finally upload them to a file server. In other words, in this scenario, there is a need to continuously upload heterogeneous types and large data volumes to a file server in the form of files. Therefore, how to effectively upload data streams has become an urgent problem to be solved.

[0003] At present, the common method for uploading data streams is to use Amazon Simple Storage Service (AWS S3) to upload files. However, this method is only compatible with existing files as upload input sources. When the data volume of the data stream is large, if a complete file is constructed for fragmented upload, the memory occupied will be high. For some devices with limited resources, such as small available disk space and memory, the file size exceeds the disk space or memory limit, causing the device to crash, resulting in poor effect of data stream fragmented upload in resource-constrained devices. Summary of the invention

[0004] In order to improve the effect of uploading data stream fragments in resource-constrained devices, the present application provides a data stream fragment dynamic transmission method, device, storage medium and electronic device.

[0005] In a first aspect of the present application, a method for dynamic transmission of data stream fragments is provided, which specifically includes:

[0006] Get the actual data length and source address of the target data stream that needs to be uploaded this time;

[0007] Determine an upload configuration information structure corresponding to the target data stream, and assign the actual data length to a data length variable of the upload configuration information structure to obtain an updated upload configuration information structure, wherein the upload configuration information structure includes a variable assigned with an address of a target buffer area corresponding to the target data stream, wherein the target buffer area is a buffer area for storing the target data stream and has a capacity equal to a preset fragment size;

[0008] According to the updated upload configuration information structure and the source address, a preset slice upload interface is called, wherein the slice upload interface encapsulates the slice processing logic of the target data stream and the upload interface in the libcurl library;

[0009] Based on the slice upload interface and the target buffer area, the target data stream is sliced ​​to obtain a slice result of the target data stream, and the slice result is uploaded, wherein the data size of the slice result is equal to the preset slice size;

[0010] When the uploading of the fragmentation results is completed and there is currently a remaining data stream that has not been uploaded in the target data stream, the entire uploading of the target data stream is completed based on the comparison result of the remaining size of the current remaining data stream in the target data stream and the preset fragmentation size.

[0011] By adopting the above technical solution, after obtaining the actual data length and source address of the target data stream, the updated upload configuration information structure is determined, thereby realizing the record of the data size of the target data stream to be uploaded this time, the input source uploaded to the file server, and the current data length in the target buffer area. Furthermore, the updated upload configuration information structure and the source address are used as parameters to call the shard upload interface, and the shard processing operation and the upload operation are performed on the target data stream in sequence, thereby reducing the coupling between the two. Finally, through this shard upload interface, the target data stream is sharded respectively, and the obtained shard results are uploaded until the entire target data stream is uploaded. In this solution, the shard upload interface is called by using the source address as one of the parameters, so that the shard processing logic is called at the source of the target data source, without the need to perform shard processing on the actual existing files. In addition, the updated upload configuration information structure contains the address of the target cache area, which is used as the input source for the upload operation, thereby achieving the purpose of calling the upload logic at the source of the target data stream, and better avoiding the crash of resource-constrained devices due to excessively large uploaded files, thereby improving the effect of data stream segmented upload in resource-constrained devices to a certain extent.

[0012] Optionally, the determining of the upload configuration information structure corresponding to the target data stream specifically includes:

[0013] Determine a preset buffer area as a target buffer area, wherein the buffer area is used to store the data stream to be uploaded;

[0014] Initialize an initial structure corresponding to the target data stream, wherein the initial structure is a structure recording upload configuration information variables of the target data stream;

[0015] Assign the address of the target buffer area and the address of the data length variable in the target buffer area to the corresponding variables in the initial structure to obtain an assigned structure;

[0016] The data length variable in the assigned structure is initialized to 0 to obtain the upload configuration information structure corresponding to the target data stream.

[0017] By adopting the above technical solution, the pre-created cache area is determined as the target cache area for storing the target data stream, and then the address of the target cache area and the data length variable in the target cache area are assigned to the corresponding variables in the initial structure, and the internal data length variable is initialized to 0, so that the upload configuration information structure can record the input source uploaded to the file server and the current data length in the target cache area for the target data stream, thereby facilitating subsequent implementation of not being limited to the input source being a fragmented upload of an actual existing file.

[0018] Optionally, the performing segmentation processing on the target data stream based on the segment upload interface and the target buffer area to obtain the segmentation result of the target data stream includes:

[0019] Determine the current data length of the target buffer area according to the variable in the updated upload configuration information structure to which the address of the data length variable in the target buffer area is assigned;

[0020] Summing the actual data length and the current data length to obtain a data sum, and comparing the data sum with the preset fragment size;

[0021] If the sum of the data is greater than or equal to the preset fragmentation size, the target buffer area is filled with data based on the target data stream to obtain a fragmentation result of the target data stream;

[0022] If the data sum is smaller than the preset segment size, the current data length variable in the target buffer is updated to the data sum, and the target data stream is written into the target buffer.

[0023] By adopting the above technical solution, the combined data size of the target data stream and the data existing in the target cache area is characterized by the data sum, and then the data sum is compared with the preset segment size. If the data sum is greater than or equal to the preset segment size, it means that the target data stream can be uploaded with at least one segment, and the target cache area is filled with data to make up the preset segment size that can be uploaded; if the data sum is less than the preset segment size, it means that the combined data size of the target data stream and the data existing in the target cache area is less than the size of one segment, and then it is temporarily not uploaded, and only the current data length variable in the target cache area is updated to the data sum, thereby realizing reasonable segmentation processing of the target data stream.

[0024] Optionally, the filling of the target buffer area with data based on the target data stream to obtain a slicing result of the target data stream specifically includes:

[0025] Extracting a portion of the target data stream through the source address;

[0026] The partial data stream is written into the target buffer area, and the data in the target buffer area is supplemented by the partial data stream to obtain a segmentation result of the target data stream.

[0027] By adopting the above technical solution, the target data stream is extracted in real time through the source address, and part of the data stream is written to the target cache area, thereby completing it into a single block fragment, realizing fragmentation processing of the target data stream, and finally obtaining the fragmentation result. The fragmentation result is subsequently uploaded, which can break the restriction that the input source uploaded to the file server must be an actual existing file in the traditional upload operation. The upload logic can be called directly at the source of the target data stream through the source address of the target data stream, which can better avoid the crash of resource-constrained devices due to excessively large uploaded files, thereby improving the effect of data stream fragmentation upload in resource-constrained devices to a certain extent.

[0028] Optionally, the completing the entire uploading of the target data stream based on the comparison result of the remaining size of the current remaining data stream in the target data stream and the preset fragment size specifically includes:

[0029] If the comparison result of the remaining size of the current remaining data stream in the target data stream and the preset fragment size is that the remaining size is greater than the preset fragment size, then based on the fragment upload interface, the data of the preset fragment size in the current remaining data stream is written into the target buffer area to obtain the next fragment result of the target data stream;

[0030] Upload the next fragmentation result, determine the remaining data stream after the current fragmentation is uploaded as the current remaining data stream, and repeat the step of comparing the remaining size of the current remaining data stream in the target data stream with the preset fragmentation size and finding that the remaining size is greater than the preset fragmentation size;

[0031] If the comparison result of the remaining size of the current remaining data stream in the target data stream and the preset fragment size is that the remaining size is smaller than the preset fragment size, the current remaining data stream is written into the target buffer area, and the current data length variable in the target buffer area is updated to the remaining size to wait for the next data upload in the same upload task as the target data stream;

[0032] If the comparison result of the remaining size of the current remaining data stream in the target data stream and the preset fragment size is that the remaining size is equal to the preset fragment size, the current remaining data stream is written into the target buffer area, and the final fragmentation result of the target data stream is obtained and uploaded.

[0033] By adopting the above technical solution, if the remaining size is greater than the preset fragment size, it means that the data size of the current remaining data stream is greater than the size of a single-block fragment. In this case, the data of the preset fragment size is directly written from the current remaining data stream to the target cache area for subsequent upload operations, thereby realizing fragment processing by writing into the target cache area; if the remaining size is less than the preset fragment size, the current remaining data stream is insufficient for the fragment size of a single block and cannot be uploaded as a single-block fragment. In this case, the upload operation is not performed, and the current remaining data stream is directly written into the target cache area, thereby waiting for the next data upload in the same upload task, and then supplementing it as a single-block fragment upload, thereby improving transmission efficiency.

[0034] Optionally, after the current data length variable in the target buffer is updated to the remaining size to wait for the next data upload in the same upload task as the target data stream, the method further includes:

[0035] Determine whether there is the next data upload. If there is no next data upload, call the end upload interface in the libcurl library;

[0036] The current remaining data stream is taken as a complete segment and uploaded through the end upload interface.

[0037] By adopting the above technical solution, if there is no next data upload, it means that there is no new data stream to upload in the upload task to which the target data stream belongs. In order to ensure that the target data stream can be uploaded smoothly, even if the remaining size is less than the preset segment size, it is uploaded as a complete segment, thereby completing the entire upload of the target data stream.

[0038] Optionally, uploading the slicing result specifically includes:

[0039] Determine whether the fragmentation result is the last fragment to be uploaded;

[0040] If not, the file name in the updated upload configuration information structure is determined as the upload file name, the address of the target buffer area is determined as the upload input source, and the length of the target buffer area is determined as the upload data length, and the interface configuration data form is determined according to the upload file name, the upload input source and the upload data length;

[0041] According to the interface configuration data form, the upload interface in the libcurl library is called to upload the slicing result.

[0042] By adopting the above technical solution, if the fragmentation result is not the last fragment to be uploaded, the interface configuration data form is determined according to the upload file name, upload input source and upload data length, so as to facilitate subsequent better management and calling of the upload interface. Finally, the upload interface in the libcurl library is called accordingly, and the data corresponding to the upload input source is uploaded to the file server according to the upload data length, so as to realize the fragmentation transmission of the target data stream, thereby improving the transmission efficiency.

[0043] In a second aspect of the present application, a data stream segment dynamic transmission device is provided, which specifically includes:

[0044] The information acquisition module is used to obtain the actual data length and source address of the target data stream that needs to be uploaded this time;

[0045] an assignment update module, used to determine an upload configuration information structure corresponding to the target data stream, and assign the actual data length to a data length variable of the upload configuration information structure to obtain an updated upload configuration information structure, wherein the upload configuration information structure includes a variable assigned with an address of a target buffer area corresponding to the target data stream, wherein the target buffer area is a buffer area for storing the target data stream and has a capacity equal to a preset fragment size;

[0046] An interface calling module, used to call a preset slice upload interface according to the updated upload configuration information structure and the source address, wherein the slice upload interface encapsulates the slice processing logic of the target data stream and the upload interface in the libcurl library;

[0047] A first uploading module is used to perform segmentation processing on the target data stream based on the segmentation upload interface and the target buffer area, obtain a segmentation result of the target data stream, and upload the segmentation result, wherein the data size of the segmentation result is equal to the preset segmentation size;

[0048] The second uploading module is used to complete the entire uploading of the target data stream based on the comparison result of the remaining size of the current remaining data stream in the target data stream and the preset segment size when the segmentation result upload is completed and there is currently a remaining data stream that has not been uploaded in the target data stream.

[0049] By adopting the above technical solution, after the information acquisition module obtains the actual data length and the source address, the assignment update module then updates the upload configuration information structure according to the actual data length to obtain the updated upload configuration information structure. The interface call module calls the fragment upload interface, and then the first upload module performs fragment processing to obtain the fragmentation result and upload it. Finally, the second upload module completes the entire upload of the target data stream based on the comparison result.

[0050] In a third aspect of the present application, a computer-readable storage medium is provided, in which a computer program is stored. When the computer program is loaded and executed by a processor, the method steps described in any one of the first aspects are performed.

[0051] In a fourth aspect of the present application, an electronic device is provided, specifically comprising:

[0052] A processor, a memory, and a computer program stored in the memory and capable of running on the processor, wherein the processor is used to load and execute the computer program stored in the memory so that the electronic device performs the method as described in any one of the first aspects.

[0053] In summary, the present application includes at least one of the following beneficial technical effects: by using the source address as one of the parameters to call the shard upload interface, the shard processing logic is called at the source of the target data source, without the need to perform shard processing on the actual existing files. In addition, the address of the target cache area is included in the updated upload configuration information structure, which is used as the input source for the upload to perform the upload operation, thereby achieving the purpose of calling the upload logic at the source of the target data stream, better avoiding the crash of resource-constrained devices due to excessively large uploaded files, and thus improving the effect of data stream shard upload in resource-constrained devices to a certain extent. BRIEF DESCRIPTION OF THE DRAWINGS

[0054] Figure 1 It is a flowchart of a data stream segmentation dynamic transmission method provided in an embodiment of the present application;

[0055] Figure 2 It is a flowchart of another method for dynamic transmission of data stream fragments provided in an embodiment of the present application;

[0056] Figure 3It is a structural schematic diagram of a data stream segmentation dynamic transmission device provided in an embodiment of the present application;

[0057] Figure 4 It is a structural schematic diagram of another data stream fragmentation dynamic transmission device provided in an embodiment of the present application.

[0058] Explanation of the accompanying drawings: 11. Information acquisition module; 12. Value assignment and update module; 13. Interface calling module; 14. First upload module; 15. Second upload module; 16. End upload module. DETAILED DESCRIPTION

[0059] In order to enable those skilled in the art to better understand the technical solutions in this specification, the technical solutions in the embodiments of this specification will be clearly and completely described below in conjunction with the drawings in the embodiments of this specification. Obviously, the described embodiments are only part of the embodiments of this application, not all of the embodiments.

[0060] In the description of the embodiments of the present application, words such as "illustrative", "for example" or "for example" are used to indicate examples, illustrations or descriptions. Any embodiment or design described as "illustrative", "for example" or "for example" in the embodiments of the present application should not be interpreted as being more preferred or more advantageous than other embodiments or designs. Specifically, the use of words such as "illustrative", "for example" or "for example" is intended to present related concepts in a concrete way.

[0061] In the description of the embodiments of the present application, the term "and / or" is only a kind of association relationship describing the associated objects, indicating that there may be three kinds of relationships, for example, A and / or B, which can represent: A exists alone, B exists alone, and A and B exist at the same time. In addition, unless otherwise specified, the meaning of the term "multiple" refers to two or more. For example, multiple systems refer to two or more systems, and multiple screen terminals refer to two or more screen terminals. In addition, the terms "first" and "second" are only used for descriptive purposes, and cannot be understood as indicating or implying relative importance or implicitly indicating the indicated technical features. Thus, the features defined as "first" and "second" can explicitly or implicitly include one or more of the features. The terms "include", "comprise", "have" and their variations all mean "including but not limited to", unless otherwise specifically emphasized.

[0062] See also Figure 1 The present application embodiment discloses a flow chart of a data stream segmentation dynamic transmission method, which can be implemented by a computer program or run on a data stream segmentation dynamic transmission device based on the von Neumann system. The computer program can be integrated into an application or run as an independent tool application, specifically including:

[0063] S101: Acquire the actual data length and source address of the target data stream to be uploaded this time.

[0064] Specifically, in an embodiment of the present application, the target data stream is a series of continuous data packets that need to be uploaded this time, and the data corresponding to the target data stream can specifically be a flow file constructed by the flow continuously collected by the information collection interface, and finally these flow files need to be uploaded to the file server in the form of data streams. Among them, the flow is the amount of data transmitted through the network. In order to analyze and optimize network performance, it is necessary to monitor and collect network traffic, and the collected traffic data can be saved in a flow file for subsequent processing and analysis. Furthermore, the actual data length refers to the total number of bytes or bits contained in this target data stream. The source address represents the starting position or source position of the data corresponding to the target data stream.

[0065] Further, a feasible way to obtain the source address of the target data stream is to obtain the source address through a preset information collection interface. For example, when the information collection interface captures network traffic, the source address is specifically obtained by parsing the header information of the data packet. This is a prior art and will not be described in detail here. In addition, the actual data length of the target data stream can be determined by the iftop tool built into the Linux system. In other embodiments, the actual data length of the target data stream can also be obtained by the preset Wireshark tool.

[0066] It should be noted that a data stream fragmentation dynamic transmission method disclosed in the present application is applied to resource-constrained devices. In an embodiment of the present application, the resource-constrained device is a Linux embedded device, that is, a device with a Linux operating system installed, which can be a smart phone, a personal computer or a router, etc.

[0067] S102: Determine the upload configuration information structure corresponding to the target data stream, and assign the actual data length to the data length variable of the upload configuration information structure to obtain an updated upload configuration information structure.

[0068] Specifically, the upload configuration information structure includes a variable assigned with the address of the target buffer area corresponding to the target data stream, and the target buffer area is a buffer area for storing the target data stream and having a capacity equal to the preset slice size. Wherein, the structure is a structured data composed of a batch of data, and each data constituting the structured data is called a "member" of the structured data. The structure type is not defined by the system, but needs to be defined by the programmer himself. Wherein, the upload configuration information structure contains all the necessary information required for uploading the target data stream, and can be used as a global variable, or as a parameter carrying configuration information to be passed to all places related to this transmission. In the embodiment of the present application, a feasible way to determine the upload configuration information structure is: determine the pre-created buffer area as the target buffer area corresponding to the target data stream to be uploaded this time, wherein the capacity of the buffer area is the preset slice size, which is mainly used to store the data stream that needs to be uploaded, and it may have uploaded other data streams in the same upload task before uploading the target data stream this time. A feasible way to create a buffer area is: create a buffer area by calling the malloc interface, that is, reserve a storage space in the memory for storing the area of ​​the data stream data that needs to be uploaded. Then, the capacity of this buffer is set to the preset shard size, the size of a shard, and a corresponding data length variable is created to record the current data length in the buffer in real time. The initial value of the data length variable is set to 0, and the data length variable can be an integer variable. It should be noted that the size of the data temporarily stored in the target buffer each time is the preset shard size. In addition, when the target data stream is uploaded, it is first written to this target buffer, and then the data is uploaded from the target buffer.

[0069] Further, the initial structure corresponding to the target data stream is obtained by initialization, that is, a plurality of variables recording the upload configuration information of the target data stream are defined in the structure to obtain the initial structure. Wherein, in the embodiment of the present application, the upload configuration information refers to a series of parameters required to ensure that the target data stream can be correctly and efficiently uploaded to the file server. Further, the address of this target buffer is obtained using a pointer, and the address of this target buffer is used as the upload configuration information, that is, the input source uploaded to the file server, and assigned to the corresponding variable in the initial structure. At the same time, the address of the data length variable of the target buffer is obtained using a pointer, and the & operator is added to the variable name of the data length variable to obtain its address. The address of the data length variable of the target buffer is assigned to the corresponding variable in the initial structure to obtain the assigned structure. Further, the data length variable of the assigned structure is initialized to 0, wherein the data length variable of the assigned structure is a variable that records the data length of the data stream required to be uploaded this time, and finally the upload configuration information structure corresponding to the target data stream is obtained.

[0070] Furthermore, the actual data length corresponding to the target data stream that actually needs to be uploaded this time is assigned to the data length variable in the upload configuration information structure, and finally the updated upload configuration information structure is obtained, thereby realizing the record of the data size of the target data stream that needs to be uploaded this time, the input source uploaded to the file server, and the current data length in the target cache area.

[0071] S103: According to the updated upload configuration information structure and the source address, a preset segment upload interface is called, where the segment upload interface encapsulates the segment processing logic of the target data stream and the upload interface in the libcurl library.

[0072] Specifically, after determining the updated upload configuration information structure, the updated upload configuration information structure and the source address of the target data stream are used as parameters to call the preset fragment upload interface, which encapsulates the fragment processing logic of the target data stream and the upload interface in the libcurl library, so as to separate the fragment processing logic and the upload logic of the target data stream. The fragment processing process does not occur inside the upload interface, which reduces the coupling of the two links to a certain extent, and provides convenience for the transformation and replacement of the upload method. In addition, the libcurl library is an open source, cross-platform network transmission library, which provides a set of easy-to-use application programming interfaces (Application Programming Interface, API) for realizing data transmission of protocols such as HTTP, HTTPS, FTP, FTPS, SCP, SFTP, LDAP, SMTP, etc. The upload interface in the libcurl library supports the streaming mode of data, which better meets the needs of continuous transmission of unknown or uncontrollable data. In addition, the fragment processing logic of the target data stream, that is, the logic of dividing the target data stream into relatively small fragments. For details, please refer to steps S104 and S105 below, which will not be repeated here.

[0073] S104: Based on the fragment upload interface and the target buffer area, fragment the target data stream to obtain a fragment result of the target data stream, and upload the fragment result, wherein the data size of the fragment result is equal to the preset fragment size.

[0074] Specifically, based on the called segmented upload interface, the segmentation processing logic of the target data stream is first executed, and the target data stream is segmented to obtain the segmentation result. In an embodiment of the present application, a feasible implementation method is: by updating the variable in the upload configuration information structure that is assigned the address of the data length variable in the target cache area, the current data length in the target cache area is determined. Then, the actual data length and current data length of the target data stream to be uploaded are summed to obtain the data sum, thereby characterizing the comprehensive data size of the target data stream and the data existing in the target buffer area. Then, the data sum is compared with the preset fragment size. If the data sum is greater than or equal to the preset fragment size, it means that at least one fragment can be uploaded. Then, based on the target data stream, the target buffer area is filled with data to obtain a single fragment result. The data size of the fragment result is equal to the preset fragment size. The specific process is: by calling the source address in the parameters of the fragment upload interface, a partial data stream in the target data stream is extracted, and this partial data stream is written to the target buffer area, wherein the sum of the data size of the partial data stream and the data size of the data existing in the target buffer area is equal to the preset fragment size, thereby realizing the partial data stream to supplement the data in the target buffer area and obtain the fragment result. Finally, the fragment result is uploaded, and the upload logic in the fragment upload interface is specifically executed, that is, the fragment result is uploaded through the upload interface in the libcurl library. Furthermore, if the total data is smaller than the preset segment size, it means that the combined data size of the target data stream and the data in the target buffer area itself is less than the size of a segment. In this case, the data will not be uploaded temporarily. Only the current data length variable in the target buffer area will be updated to the total data, and the target data stream will be directly written into the target buffer area, so as to facilitate the subsequent final upload as the last segment, or wait for the next data upload process in the upload task to which the target data stream belongs, and complete the upload with the target data stream as a complete segment.

[0075] S105: When the fragmentation result upload is completed and there is a remaining data stream that has not been uploaded in the target data stream, based on the comparison result of the remaining size of the current remaining data stream in the target data stream and the preset fragmentation size, the entire upload of the target data stream is completed.

[0076] Specifically, after a single fragment result is uploaded in step S104, if there is a remaining data stream that has not been uploaded in the target data stream, it means that the target data stream that needs to be uploaded this time has not been uploaded yet. Since there is no data in the target buffer area at this time, the data size of the current remaining data stream in the target data stream, that is, the remaining size, is compared with the preset fragment size to obtain a comparison result. If the comparison result is that the remaining size is greater than the preset fragment size, it means that the data size of the current remaining data stream is greater than the size of a single fragment. Then, based on the fragment upload interface, data of the preset fragment size is written from the current remaining data stream to the target buffer area through the source address to obtain the next fragment result, that is, the next single fragment that needs to be uploaded. Further, the next fragment result is uploaded through the upload interface in the libcurl library. After the upload is completed, the remaining data stream after this upload is re-determined as the current remaining data stream, and the step of comparing the remaining size of the current remaining data stream in the target data stream with the preset fragment size is repeated. If the remaining size is greater than the preset fragment size, the fragment upload process is performed again.

[0077] If the comparison result is that the remaining size is smaller than the preset fragment size, the current remaining data stream is less than the fragment size of a single block and cannot be uploaded as a fragment of a single block, then the upload operation will not be performed, and the current remaining data stream will be directly written into the target buffer according to the source address, and the current data length variable in the target buffer will be updated to the remaining size of the current remaining data stream, waiting for the next data upload in the same upload task as the target data stream, that is, waiting for the next data stream to be uploaded in the same upload task, so as to form a fragmentation result together with the next data stream and upload it in fragments.

[0078] In other embodiments, if there is no new data stream to be uploaded in the subsequent upload task to which the target data stream belongs, then it is determined that the current remaining data stream that is smaller than the preset fragment size cannot wait for the next data upload, and even if it does not reach the preset fragment size, it is uploaded as a complete fragment, thereby completing the entire upload of the target data stream. Further, the end upload interface in the libcurl library is called, and a flag is configured to indicate that the fragment uploaded this time is the last fragment, and finally the current remaining data stream is uploaded as a complete fragment through the end upload interface.

[0079] If the remaining size is equal to the preset shard size, the target data stream is directly written into the target buffer area, the final shard result of the target data stream is obtained, and the upload operation is performed on it, and finally the entire upload of the target data stream is completed. As a result, during the entire shard upload process of the target data stream, the restriction that the input source uploaded to the file server must be an actual existing file in the traditional upload operation is broken, and the upload logic can be directly called at the source of the target data stream through the source address of the target data stream, which better avoids the crash of resource-constrained devices due to the upload of too large files, thereby improving the effect of data stream shard upload in resource-constrained devices to a certain extent.

[0080] See also Figure 2 The present application embodiment discloses a flowchart of another method for dynamic transmission of data stream slicing, which can be implemented by a computer program or run on a data stream slicing dynamic transmission device based on the von Neumann system. The computer program can be integrated into an application or run as an independent tool application, specifically including:

[0081] S201: Obtain the actual data length and source address of the target data stream that needs to be uploaded this time.

[0082] S202: Determine the upload configuration information structure corresponding to the target data stream, and assign the actual data length to the data length variable of the upload configuration information structure to obtain an updated upload configuration information structure.

[0083] S203: According to the updated upload configuration information structure and the source address, a preset segment upload interface is called, where the segment upload interface encapsulates the segment processing logic of the target data stream and the upload interface in the libcurl library.

[0084] S204: Based on the fragment upload interface and the target buffer area, fragment the target data stream to obtain a fragment result of the target data stream.

[0085] For details, please refer to steps S101-S104, which will not be described in detail here. In one embodiment, the historical time period in which the network bandwidth fluctuation occurs in the historical upload data of the resource-constrained device is obtained, and the number of occurrences of each historical time period is counted. The larger the number of occurrences, the more likely the network bandwidth fluctuation occurs in the corresponding historical time period, which affects the upload effect. According to the order of the number of occurrences from large to small, the historical time period with the first number is selected from each historical time period and determined as the easy-to-fluctuate time period, that is, the time period in which the network bandwidth fluctuation is likely to occur. Further, the historical fragment size when the resource-constrained device uploads when the network bandwidth fluctuates and is unstable in a single easy-to-fluctuate time period is obtained, the transmission success rate when uploading with a single historical fragment size is counted, and the number of uploads of the historical fragment size with a transmission success rate exceeding a preset success rate threshold is counted. The higher the number of uploads, the easier it is to have a higher transmission success rate with the corresponding historical fragment size in the easy-to-fluctuate time period, and the better the upload effect. The historical fragment size with the second number is selected from each historical fragment size to be determined as the easy-to-transmit fragment size of the corresponding easy-to-fluctuate time period, that is, the fragment size with a better upload effect in the easy-to-fluctuate time period.

[0086] Further, the first weight of each easy-to-fluctuate time period and the second weight of the corresponding easy-to-transmit fragment size are calculated, the first weight is the ratio of the number of occurrences of each easy-to-fluctuate time period to the sum of the number of occurrences of all easy-to-fluctuate time periods, and the second weight is the ratio of the number of uploads of a single easy-to-transmit fragment size corresponding to the easy-to-fluctuate time period to the sum of the number of uploads of all the corresponding easy-to-transmit fragment sizes. Further, the current time when the target data stream is ready to be uploaded is obtained, the easy-to-fluctuate time period at the current time is determined as the key fluctuation time period, and the first weight of the key fluctuation time period and the product of the second weights of the corresponding easy-to-transmit fragment sizes are calculated, that is, the first weight product. The larger the first weight product is, the easier it is to obtain a better transmission effect by uploading the corresponding easy-to-transmit fragment size once the network bandwidth fluctuates. The easy-to-transmit fragment size corresponding to the maximum first weight product among the first weight products is determined as the final fragment size, and is set as the preset fragment size when uploading the target data stream fragments.

[0087] If the ratio of the data size of the target data stream to the final fragment size is greater than the preset ratio threshold, it means that the final fragment size is too small, and it is easy for the target data stream to be uploaded too many times, affecting the upload efficiency. Then determine the appropriate fragment size according to the data size of the target data stream. The easy-to-transmit fragment size after the current moment of this appropriate fragment size is determined as the final fluctuation period, and the product of the first weight of each final fluctuation period and the second weight of the corresponding appropriate fragment size is calculated. This product is recorded as: the second weight product. The maximum second weight product is selected from each second weight product, and the final fluctuation period corresponding to this maximum second weight product is determined as the final upload period of the target data stream.

[0088] S205: Determine whether the fragmentation result is the last fragment to be uploaded.

[0089] S206: If not, the file name in the updated upload configuration information structure is determined as the upload file name, the address of the target buffer area is determined as the upload input source, and the data length of the target buffer area is determined as the upload data length, and the interface configuration data form is determined according to the upload file name, the upload input source and the upload data length.

[0090] S207: According to the interface configuration data form, call the upload interface in the libcurl library to upload the sharding result.

[0091] Specifically, a feasible way to determine whether the fragmentation result is the last fragment to be uploaded is to determine whether there is a new data stream that needs to be uploaded in the upload task to which the target data stream belongs. If so, it means that this fragmentation result is not the last fragment to be uploaded, then the file name in the updated upload configuration information structure is determined as the upload file name, and the address of the target buffer area is determined as the upload input source, that is, the input source uploaded to the file server, and then the length of the target buffer area is determined as the upload data length, wherein the length of the target buffer area can be understood as the capacity of the target buffer area, that is, the preset fragment size, and finally the upload file name, upload input source and upload data length are added to the configuration data form of the upload interface to obtain the interface configuration data form. Among them, the configuration data form refers to a data structure that defines and manages information related to the upload interface. The upload interface can be more conveniently managed and called through the configuration data form.

[0092] Furthermore, through this interface configuration data form, call the upload interface in the libcurl library to upload the data corresponding to the upload input source to the file server with the upload data length. In addition, in order to prevent the libcurl library upload interface from automatically printing the return response information, it is also necessary to add an empty write callback logic to replace the default callback.

[0093] S208: When the fragmentation result upload is completed and there is a remaining data stream that has not been uploaded in the target data stream, based on the comparison result of the remaining size of the current remaining data stream in the target data stream and the preset fragmentation size, complete the entire upload of the target data stream.

[0094] For details, please refer to step S105, which will not be described in detail here.

[0095] The implementation principle of a data stream sharding dynamic transmission method in the embodiment of the present application is: by using the source address as one of the parameters to call the sharding upload interface, the sharding processing logic is called at the source of the target data source, without the need to perform sharding processing on the actual existing files. In addition, the address of the target cache area is included in the updated upload configuration information structure, which is used as the input source for the upload to perform the upload operation, thereby achieving the purpose of calling the upload logic at the source of the target data stream, better avoiding the crash of resource-constrained devices due to the excessive size of the uploaded files, and thus improving the effect of data stream sharding upload in resource-constrained devices to a certain extent.

[0096] The following are device embodiments of the present application, which can be used to execute the method embodiments of the present application. For details not disclosed in the device embodiments of the present application, please refer to the method embodiments of the present application.

[0097] See also Figure 3 , is a schematic diagram of the structure of a data stream slicing dynamic transmission device provided in an embodiment of the present application. The data stream slicing dynamic transmission device can be implemented as all or part of the device through software, hardware or a combination of both. The device includes an information acquisition module 11, a value assignment update module 12, an interface call module 13, a first upload module 14 and a second upload module 15.

[0098] The information acquisition module 11 is used to obtain the actual data length and source address of the target data stream that needs to be uploaded this time;

[0099] The assignment update module 12 is used to determine the upload configuration information structure corresponding to the target data stream, and assign the actual data length to the data length variable of the upload configuration information structure to obtain the updated upload configuration information structure, wherein the upload configuration information structure includes a variable assigned with the address of the target buffer area corresponding to the target data stream, and the target buffer area is a buffer area for storing the target data stream and having a capacity equal to the preset fragment size;

[0100] The interface calling module 13 is used to call the preset segment upload interface according to the updated upload configuration information structure and the source address, and the segment upload interface encapsulates the segment processing logic of the target data stream and the upload interface in the libcurl library;

[0101] The first uploading module 14 is used to perform slicing processing on the target data stream based on the slicing upload interface and the target buffer area, obtain the slicing result of the target data stream, and upload the slicing result, wherein the data size of the slicing result is equal to the preset slicing size;

[0102] The second uploading module 15 is used to complete the entire upload of the target data stream based on the comparison result of the remaining size of the current remaining data stream in the target data stream and the preset segment size when the segmentation result upload is completed and there is currently a remaining data stream in the target data stream that has not been uploaded.

[0103] Optionally, the assignment update module 12 is specifically used for:

[0104] Determine a preset buffer area as a target buffer area, where the buffer area is used to store the data stream to be uploaded;

[0105] Initialize the initial structure corresponding to the target data stream, which is a structure that records the upload configuration information variables of the target data stream;

[0106] Assign the address of the target buffer and the address of the data length variable in the target buffer to the corresponding variables in the initial structure to obtain the assigned structure;

[0107] Initialize the data length variable in the assigned structure to 0 to obtain the upload configuration information structure corresponding to the target data stream.

[0108] Optionally, the first uploading module 14 is specifically used for:

[0109] Determine the current data length of the target buffer area according to the variable assigned with the address of the data length variable in the target buffer area in the updated upload configuration information structure;

[0110] The actual data length is added to the current data length to obtain the total data, and the total data is compared with the preset shard size;

[0111] If the total amount of data is greater than or equal to the preset shard size, the target buffer is filled with data based on the target data stream to obtain the shard result of the target data stream;

[0112] If the total data is less than the preset segment size, the current data length variable in the target buffer is updated to the total data, and the target data stream is written into the target buffer.

[0113] Optionally, the first uploading module 14 is specifically used for:

[0114] Extract part of the data stream in the target data stream through the source address;

[0115] Part of the data stream is written into the target buffer area, and the data in the target buffer area is supplemented by the part of the data stream to obtain the slicing result of the target data stream.

[0116] Optionally, the second uploading module 15 is specifically used for:

[0117] If the comparison result of the remaining size of the current remaining data stream in the target data stream and the preset fragment size is that the remaining size is greater than the preset fragment size, then based on the fragment upload interface, the data of the preset fragment size in the current remaining data stream is written into the target buffer area to obtain the next fragment result of the target data stream;

[0118] Upload the next fragmentation result, and determine the remaining data stream after the current fragmentation is uploaded as the current remaining data stream, and repeat the step if the comparison result between the remaining size of the current remaining data stream in the target data stream and the preset fragmentation size is that the remaining size is greater than the preset fragmentation size;

[0119] If the comparison result between the remaining size of the current remaining data stream in the target data stream and the preset fragment size is that the remaining size is smaller than the preset fragment size, the current remaining data stream is written into the target buffer area, and the current data length variable in the target buffer area is updated to the remaining size to wait for the next data upload in the same upload task as the target data stream;

[0120] If the comparison result of the remaining size of the current remaining data stream in the target data stream and the preset fragment size is that the remaining size is equal to the preset fragment size, the current remaining data stream is written into the target buffer area, the final fragmentation result of the target data stream is obtained and uploaded.

[0121] Optional, such as Figure 4 As shown, the device also includes an end uploading module 16, which is specifically used for:

[0122] Determine whether there is the next data upload. If there is no next data upload, call the end upload interface in the libcurl library;

[0123] The remaining data stream is treated as a complete segment and uploaded through the end upload interface.

[0124] Optionally, the first uploading module 14 is specifically used for:

[0125] Determine whether the fragmentation result is the last fragment to be uploaded;

[0126] If not, the file name in the updated upload configuration information structure is determined as the upload file name, the address of the target buffer area is determined as the upload input source, and the length of the target buffer area is determined as the upload data length, and the interface configuration data form is determined according to the upload file name, the upload input source and the upload data length;

[0127] According to the interface configuration data form, call the upload interface in the libcurl library to upload the sharding results.

[0128] It should be noted that the data stream segmentation dynamic transmission device provided in the above embodiment only uses the division of the above functional modules as an example when executing the data stream segmentation dynamic transmission method. In actual applications, the above functional distribution can be completed by different functional modules as needed, that is, the internal structure of the device is divided into different functional modules to complete all or part of the functions described above. In addition, the data stream segmentation dynamic transmission device provided in the above embodiment and the data stream segmentation dynamic transmission method embodiment belong to the same concept. The implementation process is detailed in the method embodiment and will not be repeated here.

[0129] An embodiment of the present application further discloses a computer-readable storage medium, and the computer-readable storage medium stores a computer program, wherein when the computer program is executed by a processor, a data stream fragmentation dynamic transmission method of the above embodiment is adopted.

[0130] Among them, the computer program can be stored in a computer-readable medium, the computer program includes computer program code, the computer program code can be in the form of source code, object code, executable file or certain middleware, etc. The computer-readable medium includes any entity or device that can carry the computer program code, recording medium, USB flash drive, mobile hard disk, magnetic disk, optical disk, computer memory, read-only memory (ROM), random access memory (RAM), electrical carrier signal, telecommunication signal and software distribution medium, etc. It should be noted that the computer-readable medium includes but is not limited to the above-mentioned components.

[0131] Among them, through this computer-readable storage medium, a data stream fragmentation dynamic transmission method of the above embodiment is stored in a computer-readable storage medium, and is loaded and executed on a processor to facilitate the storage and application of the above method.

[0132] An embodiment of the present application further discloses an electronic device, in which a computer program is stored in a computer-readable storage medium. When the computer program is loaded and executed by a processor, the above-mentioned data stream segmentation dynamic transmission method is adopted.

[0133] The electronic device may be a desktop computer, a laptop computer, a cloud server or other electronic device, and the electronic device includes but is not limited to a processor and a memory. For example, the electronic device may also include input and output devices, a network access device, and a bus.

[0134] Among them, the processor can adopt a central processing unit (CPU). Of course, according to actual usage, other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. can also be adopted. The general-purpose processor can adopt a microprocessor or any conventional processor, etc., and this application does not impose any restrictions on this.

[0135] Among them, the memory can be an internal storage unit of the electronic device, such as a hard disk or memory of the electronic device, or it can be an external storage device of the electronic device, such as a plug-in hard disk, a smart memory card (SMC), a secure digital card (SD) or a flash memory card (FC) equipped on the electronic device. Moreover, the memory can also be a combination of an internal storage unit and an external storage device of the electronic device. The memory is used to store computer programs and other programs and data required by the electronic device. The memory can also be used to temporarily store data that has been output or is to be output, and this application does not impose any restrictions on this.

[0136] Among them, through this electronic device, a data stream fragmentation dynamic transmission method of the above embodiment is stored in the memory of the electronic device, and is loaded and executed on the processor of the electronic device for easy use.

[0137] The above is only an exemplary embodiment of the present disclosure and cannot be used to limit the scope of the present disclosure. That is, any equivalent changes and modifications made according to the teachings of the present disclosure are still within the scope of the present disclosure. This application is intended to cover any variation, use or adaptive change of the present disclosure, which follows the general principles of the present disclosure and includes common knowledge or customary technical means in the technical field that are not recorded in the present disclosure. The description and examples are only regarded as exemplary, and the scope and spirit of the present disclosure are defined by the claims.

Claims

1. A data stream fragmentation dynamic transmission method, characterized in that: The method comprises: Get the actual data length and source address of the target data stream that needs to be uploaded this time; Determine an upload configuration information structure corresponding to the target data stream, and assign the actual data length to a data length variable of the upload configuration information structure to obtain an updated upload configuration information structure, wherein the upload configuration information structure includes a variable assigned with an address of a target buffer area corresponding to the target data stream, wherein the target buffer area is a buffer area for storing the target data stream and has a capacity equal to a preset fragment size; According to the updated upload configuration information structure and the source address, a preset slice upload interface is called, wherein the slice upload interface encapsulates the slice processing logic of the target data stream and the upload interface in the libcurl library; Based on the fragment upload interface and the target buffer area, the target data stream is fragmented to obtain a fragmentation result of the target data stream, and the fragmentation result is uploaded, wherein the data size of the fragmentation result is equal to the preset fragmentation size; When the upload of the fragmentation result is completed and there is a remaining data stream that has not been uploaded in the target data stream, based on the comparison result of the remaining size of the current remaining data stream in the target data stream and the preset fragmentation size, the entire upload of the target data stream is completed; wherein, based on the fragmentation upload interface and the target buffer area, the target data stream is fragmented to obtain the fragmentation result of the target data stream, including: Determine the current data length of the target buffer area according to the variable in the updated upload configuration information structure to which the address of the data length variable in the target buffer area is assigned; Summing the actual data length and the current data length to obtain a data sum, and comparing the data sum with the preset fragment size; If the sum of the data is greater than or equal to the preset fragmentation size, the target buffer area is filled with data based on the target data stream to obtain a fragmentation result of the target data stream; If the data sum is smaller than the preset slice size, the current data length variable in the target buffer is updated to the data sum, and the target data stream is written into the target buffer; The step of filling the target buffer area with data based on the target data stream to obtain a slicing result of the target data stream specifically includes: Extracting a portion of the target data stream through the source address; The partial data stream is written into the target buffer area, and the data in the target buffer area is supplemented by the partial data stream to obtain a segmentation result of the target data stream.

2. The data stream fragmentation dynamic transmission method according to claim 1 is characterized in that: The step of determining the upload configuration information structure corresponding to the target data stream specifically includes: Determine a preset buffer area as a target buffer area, wherein the buffer area is used to store the data stream to be uploaded; Initialize an initial structure corresponding to the target data stream, wherein the initial structure is a structure recording upload configuration information variables of the target data stream; Assign the address of the target buffer area and the address of the data length variable in the target buffer area to the corresponding variables in the initial structure to obtain an assigned structure; The data length variable in the assigned structure is initialized to 0 to obtain the upload configuration information structure corresponding to the target data stream.

3. The data stream fragmentation dynamic transmission method according to claim 1 is characterized in that: The step of completing the entire upload of the target data stream based on the comparison result of the remaining size of the current remaining data stream in the target data stream and the preset fragment size specifically includes: If the comparison result of the remaining size of the current remaining data stream in the target data stream and the preset fragment size is that the remaining size is greater than the preset fragment size, then based on the fragment upload interface, the data of the preset fragment size in the current remaining data stream is written into the target buffer area to obtain the next fragment result of the target data stream; Upload the next fragmentation result, determine the remaining data stream after the current fragmentation is uploaded as the current remaining data stream, and repeat the step of comparing the remaining size of the current remaining data stream in the target data stream with the preset fragmentation size and finding that the remaining size is greater than the preset fragmentation size; If the comparison result of the remaining size of the current remaining data stream in the target data stream and the preset fragment size is that the remaining size is smaller than the preset fragment size, the current remaining data stream is written into the target buffer area, and the current data length variable in the target buffer area is updated to the remaining size to wait for the next data upload in the same upload task as the target data stream; If the comparison result of the remaining size of the current remaining data stream in the target data stream and the preset fragment size is that the remaining size is equal to the preset fragment size, the current remaining data stream is written into the target buffer area, and the final fragmentation result of the target data stream is obtained and uploaded.

4. The data stream fragmentation dynamic transmission method according to claim 3 is characterized in that: After the current data length variable in the target buffer is updated to the remaining size to wait for the next data upload in the same upload task as the target data stream, the method further includes: Determine whether there is the next data upload. If there is no next data upload, call the end upload interface in the libcurl library; The current remaining data stream is taken as a complete segment and uploaded through the end upload interface.

5. The data stream fragmentation dynamic transmission method according to claim 1, characterized in that: The uploading of the fragmentation result specifically includes: Determine whether the fragmentation result is the last fragment to be uploaded; If not, the file name in the updated upload configuration information structure is determined as the upload file name, the address of the target buffer area is determined as the upload input source, and the length of the target buffer area is determined as the upload data length, and the interface configuration data form is determined according to the upload file name, the upload input source and the upload data length; According to the interface configuration data form, the upload interface in the libcurl library is called to upload the slicing result.

6. A data stream fragmentation dynamic transmission device, used to implement the data stream fragmentation dynamic transmission method according to any one of claims 1 to 5, characterized in that: include: An information acquisition module (11) is used to acquire the actual data length and source address of the target data stream to be uploaded this time; An assignment update module (12) is used to determine an upload configuration information structure corresponding to the target data stream, and assign the actual data length to a data length variable of the upload configuration information structure to obtain an updated upload configuration information structure, wherein the upload configuration information structure includes a variable assigned with an address of a target buffer area corresponding to the target data stream, wherein the target buffer area is a buffer area for storing the target data stream and has a capacity equal to a preset slice size; An interface calling module (13) is used to call a preset slice upload interface according to the updated upload configuration information structure and the source address, wherein the slice upload interface encapsulates the slice processing logic of the target data stream and the upload interface in the libcurl library; A first uploading module (14) is used to perform segmentation processing on the target data stream based on the segmentation upload interface and the target buffer area, obtain a segmentation result of the target data stream, and upload the segmentation result, wherein the data size of the segmentation result is equal to the preset segmentation size; The second uploading module (15) is used to complete the entire upload of the target data stream based on the comparison result of the remaining size of the current remaining data stream in the target data stream and the preset segment size when the segmentation result upload is completed and there is currently a remaining data stream that has not been uploaded in the target data stream.

7. A computer-readable storage medium, wherein a computer program is stored in the computer-readable storage medium, characterized in that: When the computer program is loaded and executed by a processor, the method according to any one of claims 1 to 5 is adopted.

8. An electronic device comprising a memory, a processor, and a computer program stored in the memory and capable of running on the processor, characterized in that: When the processor loads and executes the computer program, the method according to any one of claims 1 to 5 is adopted.

Images