File transmission method, storage service allocation method, device, equipment and medium

By using the coordinated work of file transfer service unit and service management device in the video surveillance platform, and using preset storage service allocation rules, the problem of uneven storage service in storage caused by uneven time distribution of video files and different file sizes is solved, and the storage service imbalance is achieved. Improved storage performance and reliability.

CN114338654BActive Publication Date: 2024-11-12CHINA TELECOM CORP LTD
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202111664298.2
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2021-12-31
Publication Date
2024-11-12
Estimated Expiration
2041-12-31

AI Technical Summary

Technical Problem

In video surveillance platforms, due to the uneven time distribution of video files and the different file sizes, the use of storage services is unbalanced, which in turn reduces storage performance and reliability, which may lead to video loss.

Method used

Through the coordinated work of the file transfer service unit and the service management device, preset storage service allocation rules are used to determine candidate storage service units based on the storage performance parameters of the storage service unit to ensure the load balancing and reliability of file transfer tasks, including file transfer queues Management, storage service list update, exception handling and other mechanisms.

Benefits of technology

It realizes load balancing and performance optimization of storage services, improves storage reliability, avoids video loss due to uneven video file transfer, and enhances the high availability of the system.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN114338654B_ABST
    Figure CN114338654B_ABST
Patent Text Reader

Abstract

The file transfer method, storage service allocation method, device, equipment and medium provided in the embodiments of the present application, the service device collects and calculates the capacity performance parameters of the storage service unit, and extracts the corresponding transmission performance parameters based on the notification completed by the storage service unit, and combines the storage performance parameters of the accurate storage service unit performance to determine each storage service unit through relevant preset storage service allocation rules to provide them to the file transfer service unit. The file transfer service unit queries the storage service unit that matches the transferred file in the candidate storage service list, thereby completing the execution of the file transfer task, and can form a notification to feedback to the service device to form a closed loop, so that the effect of balanced storage service allocation is continuously strengthened, storage performance is optimized, and conditions such as file transfer failure and high availability guarantee can also be used to exclude unreliable storage service units, thereby improving the reliability of storage services.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present application relates to the field of storage management technology, and in particular to a file transmission method, a storage service allocation method, a device, a equipment and a medium. Background Art

[0002] Video surveillance platforms mostly use distributed storage systems as video storage services, but object storage (OSS) is increasingly favored by surveillance video platforms as a video service due to its reliability and high cost-effectiveness. In addition, streaming video servers may load hundreds or thousands of devices at the same time. If only one storage service is connected, a large amount of video may be lost in the event of a failure. Therefore, surveillance platforms often consider allocating multiple storage services to the same storage server.

[0003] However, due to the characteristics of video files, the time distribution of generation may not be uniform, and the file sizes vary, which may cause some storage services to be busy while others to be idle, resulting in a large difference in the use of storage space. Storage services without space cannot be used, resulting in reduced storage bandwidth, data loss, and reduced storage performance and reliability. Summary of the invention

[0004] In view of the above-mentioned shortcomings of the prior art, the present application provides a file transmission method, a storage service allocation method, an apparatus, a device and a medium, which optimize the performance and reliability of storage services by balancing the storage services.

[0005] The first aspect of the present application provides a file transfer method, which is applied to a file transfer service unit to transfer a video file to a storage service unit assigned by a service management device; the method comprises: the file transfer service unit puts the transferred file into a file transfer queue according to the file transfer task; the file transfer service unit determines the destination storage service unit for storing the transferred file according to a candidate storage service list; wherein the candidate storage service list comprises a plurality of storage service units; each of the storage service units is determined by the service management device from each available storage service unit according to the storage performance parameters of each storage service unit using a preset storage service allocation rule, and is provided to the file transfer service unit; the file transfer service unit completes the file transfer task using the destination storage service unit; in response to the completion of the file transfer task, the file transfer service unit sends a transmission task execution success notification to the service management device, the transmission task execution success notification comprising the transmission performance parameters of the corresponding destination storage service unit, which is used as a reference for calculating the storage performance parameters when re-determining the storage service unit.

[0006] In some embodiments, the file transfer service unit determines the destination storage service unit for storing the transferred file based on the candidate storage service list, including: determining whether the candidate storage service list meets preset conditions; wherein the preset conditions include: there are enough available storage service units in the candidate storage service list; and / or the non-update time of the candidate storage service list exceeds a preset time threshold; in response to meeting the preset conditions, the file transfer service unit determines the storage service unit in the candidate storage service list that matches the transferred file as the destination storage service unit; in response to not meeting the preset conditions, the file transfer service unit sends a candidate storage service allocation request to the service management device to request the allocation of a storage service unit; and, based on the candidate storage service list updated corresponding to the allocation, determines the destination storage service unit.

[0007] In some embodiments, the file transfer service unit determines the destination storage service unit for storing the transferred file based on the candidate storage service list, including: avoiding selecting a storage service unit that has failed within the previous n times in the candidate storage service list as the destination storage service unit, n≥1.

[0008] In some embodiments, the file transfer method further includes: performing an exception handling action based on the statistical results of the transmission failure event.

[0009] In some embodiments, the performing of an exception handling action based on the statistical results of the transmission failure event includes at least one of the following:

[0010] 1) In response to the number of file transmission failures exceeding a first preset number threshold, the file is removed from the file transmission queue, added to the file transmission failure queue, and a file transmission failure notification is sent to the service management device;

[0011] 2) In response to the number of consecutive file transmission failures that have occurred in the candidate storage service unit exceeding a second preset number threshold, the candidate storage service unit is removed from the candidate storage service list, and a storage service abnormality notification is sent to the service management device.

[0012] In some embodiments, the file transfer method includes: excluding files to be transferred whose size exceeds a preset capacity threshold from the file transfer queue; wherein the preset capacity threshold is related to the maximum capacity of the storage service unit.

[0013] A second aspect of the present application provides a storage service allocation method, which is applied to allocate storage service units for file transfer by a file transfer service unit; the method includes: the service management device collects storage usage information of each storage service unit and calculates capacity performance parameters; the service management device extracts the transmission performance parameters of the corresponding destination storage service unit based on the transmission task execution success notification received from the file transfer service unit regarding the completion of the file transfer task; the destination storage service unit is used to store the video files after the transmission is completed; the service management device calculates the storage performance parameters based on the transmission performance parameters and capacity performance parameters corresponding to each storage service unit; the service management device uses preset allocation rules to determine candidate storage service units from each available storage service unit based on the storage performance parameters of each storage service unit; the service management device responds to the candidate service allocation request and sends the information of the determined storage service unit to the file transfer service unit.

[0014] In some embodiments, the service management device calculates the storage performance parameters based on the transmission performance parameters and capacity performance parameters corresponding to each storage service unit, including: calculating a weighted result of the transmission performance parameters and capacity performance parameters.

[0015] In some embodiments, the service management device uses preset allocation rules to determine the storage service unit based on the storage performance parameters of each storage service unit, including: the service management device selects a preset allocation number of storage service units with higher storage performance parameters to determine the storage service unit from them.

[0016] In some embodiments, the service management device uses preset allocation rules to determine storage service units based on storage performance parameters of each storage service unit, and also includes: excluding storage service units whose capacity performance parameters exceed a preset utilization threshold from allocation.

[0017] In some embodiments, the storage service allocation method includes: in response to a candidate storage service allocation request of a file transfer service unit, the service management device determines a storage service unit to respond to the candidate storage service allocation request.

[0018] In some embodiments, the candidate storage service allocation request is triggered by the failure to meet a preset condition, wherein the preset condition includes: the candidate storage service in the candidate storage service list of the file transfer service unit is sufficient to complete the file transfer task; and / or the candidate storage service list has not been updated for a time period exceeding a preset time threshold.

[0019] In some embodiments, the storage service allocation method includes: in response to the number of available storage service units being lower than a preset allocation number, the service management device uses each of the available storage service units as the candidate storage service unit.

[0020] In some embodiments, the storage service allocation method includes: in response to a storage anomaly notification indicating an abnormality in a storage service unit, the service management device marks the corresponding storage service unit as abnormal so that allocation is restricted; or, in response to the storage service unit indicating the abnormality returning to normal, eliminating the abnormal mark.

[0021] According to a third aspect of the present application, there is provided a file transfer device, which is applied to a file transfer service unit to transfer a video file to a storage service unit assigned by a service management device; the device comprises: a transmission task execution module, which is used to put the transferred file into a file transfer queue according to the file transfer task; a destination storage determination module, which is used to determine the destination storage service unit for storing the transferred file according to a candidate storage service list; wherein the candidate storage service list comprises a plurality of storage service units; each of the storage service units is determined by the service management device from each available storage service unit according to the storage performance parameters of each storage service unit using a preset storage service allocation rule, and is provided to the file transfer service unit; the transmission task execution module is used to complete the file transfer task using the destination storage service unit; an execution result reporting module is used to send a transmission task execution success notification to the service management device in response to the completion of the file transfer task, wherein the transmission task execution success notification comprises the transmission performance parameters of the corresponding destination storage service unit, which is used as a reference for calculating the storage performance parameters when re-determining the storage service unit.

[0022] In a fourth aspect, the present application provides a storage service allocation device, which is used to allocate storage service units for file transfer by a file transfer service unit; the device includes: a storage information collection module, which is used to collect storage usage information of each storage service unit and calculate capacity performance parameters; an information extraction module, which is used to extract the transmission performance parameters of the corresponding target storage service unit based on the transmission task execution success notification received from the file transfer service unit regarding the completion of the file transfer task; a storage performance calculation module, which is used to calculate the storage performance parameters based on the transmission performance parameters and capacity performance parameters corresponding to each storage service unit; an information generation module, which is used to determine candidate storage service units from each available storage service unit based on the storage performance parameters of each storage service unit using preset allocation rules; and a sending module, which is used to send the information of the determined storage service unit to the file transfer service unit in response to a candidate service allocation request, so as to determine the target storage service unit for storing the transferred files in the file transfer task.

[0023] The fifth aspect of the present application provides a computer device, including: a communicator, a memory and a processor; the communicator is used to communicate with the outside; the memory is used to store program instructions; the processor is used to run the program instructions to execute the file transfer method as described in any one of the first aspects; or, execute the storage service allocation method as described in any one of the second aspects.

[0024] In a sixth aspect of the present application, a computer-readable storage medium stores program instructions, and when the program instructions are executed, the file transfer method as described in any one of the first aspect is executed; or, the storage service allocation method as described in any one of the second aspect is executed.

[0025] As described above, the file transfer method, storage service allocation method, device, equipment and medium provided in the embodiments of the present application, the service device collects and calculates the capacity performance parameters of the storage service unit, and extracts the corresponding transmission performance parameters according to the notification completed by the storage service unit, and combines the storage performance parameters of the storage service unit performance to obtain accurate storage service unit performance, and determines each storage service unit through the relevant preset storage service allocation rules to provide to the file transfer service unit, and the file transfer service unit queries the storage service unit that matches the transferred file in the candidate storage service list, thereby completing the execution of the file transfer task, and can form a notification to feedback to the service device to form a closed loop, so that the effect of balanced storage service allocation continues to be strengthened, and storage performance is optimized, and file transfer failure, high availability guarantee and other conditions can also be used to exclude unreliable storage service units, and improve storage service reliability. In this way, it can effectively deal with the problem of uneven use of storage services caused by the characteristics of different time distribution and different sizes of video files in the related technology. BRIEF DESCRIPTION OF THE DRAWINGS

[0026] Figure 1 A schematic diagram showing an application scenario in an embodiment of the present application.

[0027] Figure 2 A flowchart of a storage service allocation method in one embodiment of the present application is shown.

[0028] Figure 3 A flowchart showing a method for transferring stored files in one embodiment of the present application is shown.

[0029] Figure 4 An interactive schematic diagram showing the file transfer process in an application example of the present application.

[0030] Figure 5 A schematic diagram showing the process of allocating storage services by a service management device in an application example of the present application.

[0031] Figure 6 A module schematic diagram of a file transmission device in an embodiment of the present application is shown.

[0032] Figure 7 A module schematic diagram of a storage service device in one embodiment of the present application is shown.

[0033] Figure 8 A schematic diagram showing the structure of a computer device in one embodiment of the present application is shown. DETAILED DESCRIPTION

[0034] The following is an explanation of the implementation of the present application by specific examples, and those skilled in the art can easily understand other advantages and effects of the present application from the information disclosed in the present application. The present application can also be implemented or applied through other different specific implementations, and the details in the present application can also be modified or changed in various ways according to different viewpoints and application systems without departing from the spirit of the present application. It should be noted that the embodiments in the present application and the features in the embodiments can be combined with each other without conflict.

[0035] The following is a detailed description of the embodiments of the present application with reference to the accompanying drawings so that those skilled in the art can easily implement the present application. The present application can be embodied in a variety of different forms and is not limited to the embodiments described herein.

[0036] In the representations of the present application, the representations with reference to the terms "one embodiment", "some embodiments", "examples", "specific examples", or "some examples" etc. mean that the specific features, structures, materials or characteristics represented in conjunction with the embodiment or example are included in at least one embodiment or example of the present application. Moreover, the represented specific features, structures, materials or characteristics may be combined in any one or more embodiments or examples in a suitable manner. In addition, those skilled in the art may combine and combine the different embodiments or examples represented in the present application and the features of the different embodiments or examples, unless they contradict each other.

[0037] In addition, the terms "first" and "second" are only used to indicate the target, and cannot be understood as indicating or implying relative importance or implicitly indicating the number of the indicated technical features. Therefore, the features defined as "first" and "second" may explicitly or implicitly include at least one of the features. In the representation of this application, the meaning of "plurality" is two or more, unless otherwise clearly and specifically defined.

[0038] In order to clearly describe the present application, components irrelevant to the description are omitted, and the same reference symbols are given to the same or similar components throughout the specification.

[0039] Throughout the specification, when a device is said to be "connected" to another device, this includes not only the case of "direct connection" but also the case of "indirect connection" by placing other elements therebetween. In addition, when a device is said to "include" a certain component, unless otherwise stated, it does not exclude other components, but means that other components may be included.

[0040] Although the terms first, second, etc. are used to represent various elements in this article in some examples, these elements should not be limited by these terms. These terms are only used to distinguish one element from another element. For example, the first interface and the second interface, etc. are represented. Moreover, as used in this article, the singular forms "one", "one" and "the" are intended to also include plural forms, unless there is an opposite indication in the context. It should be further understood that the terms "comprising" and "including" indicate that there are the described features, steps, operations, elements, modules, projects, kinds, and / or groups, but do not exclude the existence, occurrence or addition of one or more other features, steps, operations, elements, modules, projects, kinds, and / or groups. The terms "or" and "and / or" used herein are interpreted as inclusive, or mean any one or any combination. Therefore, "A, B or C" or "A, B and / or C" means "any of the following: A; B; C; A and B; A and C; B and C; A, B and C". Only when the combination of elements, functions, steps or operations is inherently mutually exclusive in some way, will there be an exception to this definition.

[0041] The technical terms used herein are only used to refer to specific embodiments and are not intended to limit the present application. The singular form used herein also includes the plural form unless the sentence clearly indicates the contrary meaning. The meaning of "including" used in the specification is to specify specific characteristics, regions, integers, steps, operations, elements and / or components, and does not exclude the existence or addition of other characteristics, regions, integers, steps, operations, elements and / or components.

[0042] Although not defined differently, all terms, including technical and scientific terms used herein, have the same meaning as those generally understood by those skilled in the art to which this application belongs. Terms defined in commonly used dictionaries are additionally interpreted as having meanings consistent with relevant technical literature and the message of the present disclosure, and as long as they are not defined, they should not be overly interpreted as ideal or very formal meanings.

[0043] At present, in the video security related technologies, multiple storage services are used to serve the upload of video files. However, due to the diversity of scenes corresponding to video security, the generation time distribution of video files is uneven, and the file size is uneven, which causes the storage resource usage of the storage service to be extremely uneven, thus causing the storage performance and reliability of the multiple storage services to be greatly affected.

[0044] In view of this, a corresponding solution may be provided in an embodiment of the present application to balance the use of storage services.

[0045] It should be noted that the "units" involved below, such as "storage service unit", "file upload service unit", etc., represent functional units that may be formed by a part of software and / or hardware resources in a hardware device (such as a server). For example, the storage service unit includes storage resources (such as cache) for file storage. Possibly, the "units" may also have task threads responsible for executing corresponding tasks configured to realize their respective functions.

[0046] In addition, the "transmission" referred to in this article, such as "transferred file", "file transfer service unit", etc., although shown as an "upload" example, this is because the identities of the "terminal" and the "server" are different, but it can be understood that in other embodiments it may also be applied to the "download" example, or it may be just a transmission between two ends with the same identity, etc., and is not limited to the example.

[0047] like Figure 1 As shown, a schematic diagram showing an application scenario in an embodiment of the present application is shown.

[0048] exist Figure 1 In the example, n streaming service units are shown, namely, streaming service unit #1 to streaming service unit #n. In a possible embodiment, the n streaming service units #1 may be distributed on multiple hardware devices. In addition, n storage service units are shown, namely, storage service unit #1 to storage service unit #n.

[0049] Each streaming media service unit may include a video storage service unit, a file transfer queue, a file transfer service unit, and a video file cache. The structure of streaming media service #1 is exemplarily represented in the figure. Exemplarily, the video storage service unit is used to receive the video stream sent by the monitoring device and save the generated video file. It can save the video file in the video file cache, and can add the file information (such as identification information, feature information) of the video file to the file transfer queue on behalf of the video file. Exemplarily, the file transfer service unit is used to upload the video files in each video file cache to the remote end according to the order of the file information in the file transfer queue and store them in the storage service unit to complete the file transfer. Optionally, the file transfer service unit can also perform exception processing according to the transmission failure situation and notify the service management device, for example, put the file that failed to be transmitted multiple times into the file transfer failure queue and send the service management device an exception notification of the file transfer failure, or also perform an exception notification on the storage service unit that failed to transmit multiple times in a row. Among them, the file transfer service unit can maintain a candidate storage service list, in which the information of each storage service unit is stored for selection as the destination storage service unit for executing the file transfer. The storage service unit is determined by the service management device and provides information to the file transfer service unit.

[0050] In some embodiments, the transmission data volume of each file transfer task may be part of a video file, a complete video file, parts of multiple video files, multiple complete video files, or one or more complete video files and parts of one or more video files, etc., which can be selected according to actual needs.

[0051] Figure 1 The service management device is also exemplarily shown in FIG. In some embodiments, the service management device can be implemented as a central service management server or server group.

[0052] In some embodiments, the service management device can respond to the candidate storage service allocation request of the file transfer service unit, determine the information of each candidate storage service unit for the next cycle from storage service units #1 to #n and provide it to the file transfer service unit, so that the file transfer service unit can update the candidate storage service list, and when executing file transfer, select the destination storage service unit in the list that matches the video file as the "destination" for storage after the file transfer to complete the file transfer task.

[0053] The service management device may determine a plurality of storage service units from each storage service unit according to a preset storage service allocation rule. Exemplarily, the basis for predicting the storage service allocation rule may be simply referring to the performance of the storage service unit, or may take into account the storage performance and reliability (i.e., high availability) of the storage service unit.

[0054] On the one hand, the service management device can collect and evaluate the storage performance of each storage service unit, and give priority to the storage service unit with better storage performance under the load balancing principle. For example, the storage performance can be calculated by periodically collecting the capacity usage information of each storage service unit to calculate the capacity performance parameter, and on the other hand, the storage performance parameter of each storage service unit can be obtained by using the transmission performance of the video files received by each storage service unit, such as using the transmission performance parameter as the evaluation parameter, and weighted calculation based on the capacity performance parameter and the transmission performance parameter to obtain the storage performance parameter of each storage service unit, and selecting a preset number of storage service units with relatively high storage performance parameters for selecting the storage service unit.

[0055] On the other hand, the service management device receives requests or notifications sent by each file transfer service unit and performs corresponding processing. In a possible example, the service management device may receive one or more of, for example, notifications of successful execution of transmission tasks reported by each file transfer service unit (which may indicate that all video file data corresponding to a task have been successfully uploaded), exception notifications related to file exceptions (which may indicate that multiple file transfers have failed), and storage service unit requests related to storage service exceptions (for example, the available storage service units are insufficient to complete the file transfer, and the information of the storage service unit is unreliable). The service management device may refer to the information in the "abnormal" notification to exclude inappropriate storage service units as storage service units in accordance with the principle of balance to meet reliability (i.e., high availability) requirements.

[0056] It should be noted that Figure 1 The video file transmission and storage architecture shown in the application scenario and the above notifications are only examples and can be changed according to actual needs. For example, when storage performance or reliability is given priority, the selection criteria for storage service units (i.e., the content of the "preset storage service allocation rules") can be changed. Considering the system operation performance, the number of "units" and the number of corresponding hardware devices can also be selected according to actual needs.

[0057] like Figure 2 FIG. 2 is a flow chart showing a storage service allocation method in an embodiment of the present application. The storage service allocation method can be applied to a service management device.

[0058] The process includes:

[0059] Step S201: The service management device collects storage usage information of each storage service unit and calculates capacity performance parameters.

[0060] In some embodiments, the service management device can periodically (for example, T1 is the period length) collect information such as total capacity and used capacity of each storage service unit, and then calculate the storage utilization rate as a capacity performance parameter, for example, obtained by dividing the used capacity by the total capacity.

[0061] Step S202: the service management device extracts the transmission performance parameters of the corresponding destination storage service unit according to the transmission task execution success notification received from the file transmission service unit regarding the completion of the file transmission task.

[0062] In order to more accurately evaluate the storage performance of the storage service unit during load balancing, the capacity performance parameter can be used as one of the references for evaluating the storage performance of the storage service unit, and the transmission performance of the storage service unit in the execution of the file transfer task can also be used as another reference for evaluating its storage performance, and the transmission performance is represented by the transmission performance adoption number. Exemplarily, the transmission performance parameter can be the file transfer rate; optionally, the file transfer rate can be the average transmission rate of each destination storage service unit used to complete the entire file transfer task.

[0063] Step S203: The service management device calculates storage performance parameters based on the transmission performance parameters and capacity performance parameters corresponding to each storage service unit.

[0064] In some embodiments, the storage performance parameter can be obtained by calculating the weighted sum of the transmission performance parameter and the capacity performance parameter. It can be understood that the capacity performance parameter is between 0 and 1, and the transmission performance parameter is in the order of MB / s. By reasonably setting their weight coefficients, the weighted result of the two can accurately reflect the accurate contribution of the two to the storage performance parameter.

[0065] In some embodiments, before calculating the storage performance parameters, the storage service units that are not suitable for use can be excluded as storage service units by using the capacity performance parameters, which meets the requirement of "high availability". For example, if the capacity performance parameter of a storage service unit is greater than a preset utilization threshold (such as 70%, 80%, 90%, etc.), it indicates that the capacity performance parameter of the storage service unit is too high and is no longer suitable for storing video files.

[0066] Step S204: The service management device determines a candidate storage service unit from each available storage service unit based on the storage performance parameters of each storage service unit by using a preset allocation rule.

[0067] In some embodiments, the preset allocation rule may include a load balancing rule related to the performance of the storage service unit. Accordingly, after calculating each storage service unit, the service management device may select a preset allocation number of storage service units with higher storage performance parameters to determine the storage service unit. In a possible example, the storage performance parameters may be sorted from high to low, and the preset allocation number of storage service units with the highest ranking may be selected.

[0068] In some embodiments, the preset allocation rules may also include reliability rules related to the reliability (i.e., high availability) of the storage service unit. The service management device may mark the corresponding previously determined storage service unit as abnormal so as to be allocated in the next round according to the storage abnormality notification caused by the file transfer failure sent by the file transfer service unit. Alternatively, when the storage service unit receiving the abnormality returns to normal (for example, triggered by manual repair), the abnormal mark is removed and allocation is restored.

[0069] Step S205: the service management device sends the information of the determined storage service unit to the file transfer service unit in response to the candidate service allocation request.

[0070] The service management device can assign storage service units to file transfer service units passively and be triggered by preset conditions. When the storage service units in the candidate storage service list cannot meet the needs of the file transfer service unit, for example, the storage resource capacity corresponding to the storage service unit is less than the amount of file data to be transferred, or the candidate service list has not been updated for a period of time exceeding the preset time threshold, the reliability of the storage service units therein cannot be guaranteed, and the file transfer service unit can submit a candidate storage service allocation request to the service management device to request the allocation of storage service units. The service management device can execute the above-mentioned storage service allocation method to determine a new storage service unit to allocate to the file transfer service unit that initiated the request based on the candidate storage service allocation request. In some other embodiments, the service management device can also periodically and actively execute the storage service allocation method to update its candidate storage service list for the file transfer service unit.

[0071] In some embodiments, the storage service allocation method may also include: in response to a storage anomaly notification indicating an abnormality in a storage service unit, the service management device marks the corresponding storage service unit as abnormal so that it can be restricted in allocation, which may be ignored when selecting the next round of storage service units; or, in response to the storage service unit indicating the abnormality returning to normal, eliminating the abnormal mark.

[0072] The "abnormality" may be generated based on the statistical results of the video file upload failure event. In some examples, if file upload failures occur continuously, there may be a problem with the file itself, such as the same file repeatedly failing to upload more than a first preset number of times; or there may be a problem with the storage service unit, such as a storage service unit failing in multiple consecutive file transfers and exceeding a second preset number of times or other preset time.

[0073] According to the above content, in some specific embodiments, the service management device may exemplarily have the following functions:

[0074] (1) The service management device sorts the storage service units according to the storage performance parameters, excludes the storage services whose storage utilization exceeds the threshold, and selects candidate service unit information to provide to the file transfer service unit. Optionally, the service management device may also form a second candidate storage service list based on the selected candidate storage service units, so as to update the local first candidate storage service list of the file transfer service unit.

[0075] (2) The service management device receives a candidate storage service allocation request from the file transfer service unit; for example, the file transfer service unit is triggered by the fact that there are not enough candidate storage service units in the local list or the local list has not been updated for a long time.

[0076] (3) The service management device receives a storage service abnormality notification from the file transfer service unit, marks the abnormal storage service unit, and cancels the allocation of the storage service unit with the abnormal mark when responding to the candidate storage service acquisition request of the file transfer service unit next time, and it cannot be selected as a candidate storage service unit. If possible, an alarm can be processed and wait for manual processing, and the abnormal mark can be canceled after confirmation of recovery to normal. In normal times, storage service units with abnormal marks that have not been restored can be checked to exclude them from being candidate storage service units;

[0077] (4) If the available storage service units are more than the preset number of candidate storage units, the service management device may extract the file transfer rate according to the successful task execution notification of the file transfer service, and perform weighted calculation on the storage utilization calculated according to the collected storage usage information to obtain the storage performance parameter, and select the preset number of storage units with higher storage performance parameters as candidate storage service units to be allocated to the file transfer service that initiated the request. If the available storage service units are less than the preset number of storage units, they are directly allocated as candidate storage service units;

[0078] (5) The service management device receives a file abnormality notification from the file transfer service unit (triggered by the number of file transfer failures exceeding a first preset threshold), and performs alarm processing;

[0079] (6) The service management device receives the notification of successful task execution from the file transfer service unit and records it for calculating storage performance parameters; or, performs performance evaluation on other objects, such as evaluating the performance of the file transfer service unit, so as to perform load balancing, etc.

[0080] It should be noted that the above functions are only shown for example and can be modified according to actual needs, but are not limited to this.

[0081] For example Figure 3 As shown, a flowchart of the file transmission method in an embodiment of the present application is shown.

[0082] Exemplarily, the file transmission method may be applied to a file transmission service unit to transmit a video file to a storage service unit allocated by a service management device.

[0083] The process of the file transfer method includes:

[0084] Step S301: the file transfer service unit puts the file to be transferred into a file transfer queue according to the file transfer task.

[0085] In some embodiments, files whose size exceeds a preset capacity threshold can be excluded from the file transfer queue; wherein the preset capacity threshold is related to the maximum capacity of the storage service unit. For example, if the size of a video file exceeds the upper limit of the storage service unit, it will not be successfully transferred and needs to be excluded from the queue.

[0086] Step S302: The file transfer service unit determines a destination storage service unit for storing the transferred file according to the candidate storage service list.

[0087] You can refer to Figure 2 The principle of generating the candidate storage service list in the storage service allocation method in the embodiment will not be repeated here.

[0088] In some embodiments, the file transfer service unit can determine the destination storage service unit by matching the file size of the video file and the available capacity of the storage service unit. In the list, each storage service unit can be arranged from high to low according to the storage performance parameter to indicate the allocation priority. When performing the matching, the first storage service unit in the list that meets the size of the video file will be prioritized as the destination storage service unit. In an optional example, file transfer failure can also be used as a reference to exclude storage service units with relatively poor reliability as destination storage service units, for example, avoiding selecting a storage service unit that has failed within the previous n times in the candidate storage service list as the destination storage service unit, n≥1; specifically, for example, skipping the storage service unit that failed to upload the previous file when it is matched in the list.

[0089] The file transfer service unit selects the destination storage service unit based on the available storage service units in the local candidate storage service list. However, considering that as the storage service units are used, there will eventually be a shortage, and it is necessary to request new storage service units to be supplemented from the service management device, in some examples, before step S302, it can be performed first: determine whether the candidate storage service list meets the preset conditions; wherein the preset conditions include: the available storage service units in the candidate storage service list are sufficient, which can be judged by the number of storage service units n in the list being lower than the preset allocation number threshold (for example, n<1 or n<2), or it can also be judged by whether the capacity of the remaining candidate storage services is sufficient to meet the current video file transfer or file transfer task; and / or, the non-update time of the candidate storage service list exceeds the preset time threshold (indicating that it is expired and unreliable), so as to perform the next step.

[0090] It is understandable that the preset condition may be constructed based on whether the storage service units are sufficient or not, or based on whether the candidate storage service list is expired or not, or based on whether the storage service units are sufficient or not and whether the candidate storage service list is expired or not.

[0091] Taking the preset condition that the storage service units are sufficient and the candidate storage service list is not expired as an example, step S302 may include:

[0092] In response to satisfying the preset condition, i.e., there are enough storage service units and the candidate storage service list is not expired, the file transfer service unit determines a storage service unit in the candidate storage service list that matches the file to be transferred as the destination storage service unit. Regarding the matching, for example, for example, capacity matching means that the capacity of the destination storage service unit is sufficient to store the transferred video file; however, if the capacity of each storage service unit is sufficient, it can be randomly selected as the destination storage service unit; or, it can also be selected in combination with the storage performance parameter in a high to low order.

[0093] Alternatively, in response to not meeting the preset conditions (i.e., insufficient storage service units, or an expired list of candidate storage services), the file transfer service unit sends a candidate storage service allocation request to the service management device to request the allocation of a storage service unit; and, based on the candidate storage service list updated corresponding to the allocation, determines the destination storage service unit through the matching method.

[0094] Step S303: The file transfer service unit uses the destination storage service unit to complete the file transfer task.

[0095] Step S304: In response to the completion of the file transfer task, the file transfer service unit sends a transfer task execution success notification to the service management device.

[0096] You can refer to Figure 2 In an embodiment, the task execution success notification may include transmission performance parameters as a basis for calculating storage performance parameters, and the storage performance parameters are used as a basis for determining the allocable storage service units. In a possible example, the task execution success notification may record the information of the destination storage service unit, the size of the transmitted video file or data, the transmission duration, etc. Alternatively, in some embodiments, the task execution success notification may also directly include the calculated file transfer rate, and may also include the average file transfer rate of each destination storage service unit in the task. For example, the task execution success notification includes: file name, file size, destination storage service unit, upload time, average upload rate, etc.

[0097] When a file transfer fails, the failure information can be recorded for use in determining abnormal files and abnormal storage service units. In a possible example, the recorded failure information may include: the file name, file size, destination storage service, failure information, failure time, etc. of the video file that failed to be transferred.

[0098] In some embodiments, the file transfer service unit can perform an exception handling action based on the statistical results of the transmission failure event to exclude unreliable video file or storage service units and avoid congestion caused by continuous transmission failure caused by anomalies. The execution of the exception handling action based on the statistical results of the transmission failure event may include at least one of the following:

[0099] 1) In response to the number of file transmission failures exceeding a first preset number threshold, the file transmission service unit removes the file from the file transmission queue, adds the file to the file transmission failure queue, and sends a file transmission failure notification to the service management device.

[0100] For details, please refer to Figure 1 The file transfer failure queue in the file transfer queue is used as a shunt queue for abnormal files in the file transfer queue, thereby avoiding the transmission congestion caused by abnormal files in the file transfer queue. The abnormal alarm can be referred to by the service management device.

[0101] 2) In response to the number of consecutive file transfer failures that have occurred in the candidate storage service unit exceeding a second preset number threshold, the file transfer service unit removes the candidate storage service unit from the candidate storage service list and sends a storage service abnormality notification to the service management device.

[0102] Specifically, if the number of consecutive transmission failures is too large, the storage service unit may be abnormal, so the candidate storage service list can be excluded and the abnormality can be notified to the service management device. Figure 2 In an embodiment, the service management device may mark the storage service unit corresponding to the storage service abnormality notification as abnormal, and not participate in the allocation; and participate in the allocation again when the storage service returns to normal.

[0103] You can also refer to Figure 4 , showing a flow chart of a file transfer method in an application example of the present application.

[0104] exist Figure 4 The process includes:

[0105] Step S401: Calculate the time interval E from the last acquisition of the storage service unit to the current time;

[0106] Step S402: determine if E>T2 or the number of storage service units n<1, that is, determine if there are not enough available storage service units or the candidate storage service list has not been updated for more than a threshold time; if so, proceed to step S403; if not, proceed to step S404;

[0107] Step S403: Sending a candidate storage service allocation request to the service management device;

[0108] Step S404: determine whether the file size is abnormal, that is, exceeds the preset capacity threshold; if so, exclude the large file exceeding the preset capacity threshold from the file transfer queue, and report the abnormality for processing; if not, proceed to step S405;

[0109] Step S405: according to the file transfer task, the transferred file is placed into the file transfer queue;

[0110] Step S406: matching the target storage service unit for the transferred file in the candidate storage service list; wherein, the transferred file is avoided from being assigned to the storage service unit to which the last file transfer failed, and the remaining storage service units can be randomly assigned to the transferred file;

[0111] Step S407: executing the file transfer task according to the matched target storage service unit; wherein the transfer result of each file may be recorded;

[0112] Step S408: When all files are successfully transferred, the task is completed, a task execution success notification is sent to the service management device, and then the next file transfer task is started. The task execution success notification may include: file name, file size, destination storage service unit, upload time, average upload rate, etc.; this information can be obtained based on the previous transfer results of each file.

[0113] Step S409: When a file transfer fails, the failure information may be recorded, including the file name, file size, destination storage service unit, failure information, failure time, etc., and then the process may return to step S407 to try to retransmit the failed file.

[0114] Step S410: determining whether the number of transmission failures of the transmitted file exceeds a first preset number threshold value X;

[0115] If yes, proceed to step S411: remove the file from the file upload queue, and notify the service management device of the file transfer failure;

[0116] If not, proceed to step S412: determine whether the number of consecutive transmission failures of the layer storage service unit exceeds a second preset number threshold Y;

[0117] If yes, then proceed to step S413, remove the storage service unit from the candidate storage list, and send a storage service abnormality notification to the service management device; and wait for the next file transfer task;

[0118] If not, proceed to step S414. Optionally, if the candidate storage list is sorted by, for example, storage performance parameters from high to low to give priority to storage with higher performance parameters for file transfer, then in step S414, the position of this candidate storage service unit can be adjusted to the last position in the sort.

[0119] refer to Figure 5 , showing an interactive schematic diagram of the file transfer process in an application example of this application.

[0120] Figure 5 The file transfer service unit, service management device, storage service unit #1, storage service unit #2, and storage service unit #3 are displayed.

[0121] See the "Storage Capacity Collection" box, which indicates that the service management device collects storage usage information from storage service unit #1, storage service unit #2, and storage service unit #3, respectively, as shown in the "Storage Capacity Collection Request" and "Storage Capacity Collection Response" pairs. "Loop" indicates a cyclic execution, for example, here the storage capacity usage information is periodically collected according to the period T1.

[0122] In the box below, the file transfer service unit periodically determines whether the number of storage service units is insufficient (less than 2), or the time interval from the last acquisition (that is, the time when the candidate storage service list has not been updated) is greater than the preset time threshold T2 (that is, expired). If it meets the requirements, it requests the service management device to "obtain candidate storage", that is, to obtain the candidate storage service unit.

[0123] The two boxes below indicate the execution of "file upload processing", which is divided into two situations: "file upload successful" and "file upload failed". In this example, successful file upload means that all files are uploaded successfully, that is, the file transfer task is successfully completed. It can be seen that by matching each file of the file transfer task, i.e., file #1, file #2, ... file #n, in the candidate storage service list, such as matching the capacity and excluding previous upload failure records to determine the destination storage service unit; or, skipping the storage service unit with previous upload failure records and randomly assigning the remaining units, etc. After matching, file #1, file #2, ... file #n are uploaded to their respective destination storage service units, i.e., storage #1, storage #2, ..., storage #n.

[0124] If there is a "file upload failure", the figure schematically shows that file #2 fails to upload to the target storage service unit, i.e., storage #2. Among them, if "the number of file upload failures exceeds X", X is the first preset number threshold; if it exceeds X, the file transfer service unit can delete the file from the file transfer queue, and report the file transfer failure notification to the service management device to prompt the file abnormality. If "the storage service has failed to upload Y times in a row on this service", Y is the second preset number threshold; if it exceeds Y, the file transfer service unit can remove the storage service unit from the candidate storage service list, such as storage #2, and report the storage service abnormality notification to the service management device to prompt the abnormality of storage #2. The service management device can further mark it as abnormal without parameterizing the next allocation.

[0125] like Figure 6 , which shows a schematic diagram of a module of a file transmission device in an embodiment of the present application. The implementation of the file transmission device can refer to the file transmission method of the file transmission service unit in the previous embodiment, so the same technical features will not be repeated in this embodiment.

[0126] The file transmission device 600 includes:

[0127] The transmission task execution module 601 is used to put the transferred file into the file transmission queue according to the file transmission task;

[0128] The destination storage determination module 602 is used to determine the destination storage service unit for storing the transferred file according to the candidate storage service list; wherein the candidate storage service list includes a plurality of storage service units; each of the storage service units is determined by the service management device from each available storage service unit according to the storage performance parameters of each storage service unit using a preset storage service allocation rule, and provided to the file transfer service unit;

[0129] The transmission task execution module 601 is used to complete the file transmission task using the destination storage service unit;

[0130] The execution result reporting module 603 is used to send a transmission task execution success notification to the service management device in response to the completion of the file transfer task. The transmission task execution success notification includes the transmission performance parameters of the corresponding destination storage service unit, which is used as a reference for calculating the storage performance parameters when redetermining the storage service unit.

[0131] In some embodiments, the destination storage determination module 602 includes:

[0132] A condition determination module, used to determine whether the candidate storage service list meets a preset condition; wherein the preset condition includes: sufficient available storage service units in the candidate storage service list; and / or, the non-update time of the candidate storage service list exceeds a preset time threshold;

[0133] A first determining module, configured to determine, in response to satisfying the preset condition, a storage service unit in the candidate storage service list that matches the file to be transferred as a destination storage service unit;

[0134] A storage service application module, in response to not meeting the preset conditions, sends a candidate storage service allocation request to the service management device to request the allocation of a storage service unit; and a second determination module, for determining the target storage service unit based on a candidate storage service list updated corresponding to the allocation.

[0135] In some embodiments, the target storage determination module 602 includes: an avoidance module, used to avoid selecting a storage service unit that has failed in the previous n times in the candidate storage service list as the target storage service unit, n≥1.

[0136] In some embodiments, the file transmission device 600 further includes: an exception handling module, configured to perform exception handling actions based on statistical results of transmission failure events.

[0137] In some embodiments, the exception handling module includes at least one of the following modules:

[0138] 1) a file exception processing module, configured to remove the file from the file transfer queue, add the file to the file transfer failure queue, and send a file transfer failure notification to the service management device in response to the number of file transfer failures exceeding a first preset number threshold;

[0139] 2) A storage service exception processing module, for, in response to a number of consecutive file transfer failures occurring in a candidate storage service unit exceeding a second preset number threshold, removing the candidate storage service unit from the candidate storage service list and sending a storage service exception notification to a service management device.

[0140] In some embodiments, the file transfer device 600 includes: a file exclusion module, used to exclude transferred files whose size exceeds a preset capacity threshold from the file transfer queue; wherein the preset capacity threshold is related to the maximum capacity of the storage service unit.

[0141] like Figure 7, which shows a module diagram of a storage service allocation device in an embodiment of the present application. The implementation of the storage service allocation device can refer to the storage service allocation method of the service management device in the previous embodiment, so the same technical features will not be repeated in this embodiment.

[0142] The storage service allocation device 700 includes:

[0143] Storage information collection module 701, used to collect storage usage information of each storage service unit and calculate capacity performance parameters;

[0144] An information extraction module 702 is used to extract the transmission performance parameters of the corresponding destination storage service unit according to the transmission task execution success notification received from the file transmission service unit regarding the completion of the file transmission task;

[0145] The storage performance calculation module 703 is used to calculate the storage performance parameters based on the transmission performance parameters and capacity performance parameters corresponding to each storage service unit;

[0146] The information generating module 704 is used to determine a candidate storage service unit from each available storage service unit based on the storage performance parameters of each storage service unit by using a preset allocation rule;

[0147] The sending module 705 is used to send the information of the determined storage service unit to the file transfer service unit in response to the candidate service allocation request, so as to determine the destination storage service unit for storing the transferred file in the file transfer task.

[0148] In some embodiments, the storage performance calculation module 703 is used to calculate a weighted result of the transmission performance parameter and the capacity performance parameter.

[0149] In some embodiments, the information generation module 704 selects a preset allocated number of storage service units having higher storage performance parameters for determining the storage service unit therefrom.

[0150] In some embodiments, the information generation module 704 further includes: a first exclusion module, configured to exclude storage service units whose capacity performance parameters exceed a preset utilization threshold from allocation.

[0151] In some embodiments, the storage service allocation device 700 includes: an allocation module, which is used to respond to the candidate storage service allocation request of the file transfer service unit and determine the storage service unit to respond to the candidate storage service allocation request.

[0152] In some embodiments, the candidate storage service allocation request is triggered by the failure to meet a preset condition, wherein the preset condition includes: the candidate storage service in the candidate storage service list of the file transfer service unit is sufficient to complete the file transfer task; and / or the candidate storage service list has not been updated for a time period exceeding a preset time threshold.

[0153] In some embodiments, the information generation module 704 is further configured to, in response to the number of available storage service units being lower than a preset allocation number, use each of the available storage service units as the candidate storage service unit.

[0154] In some embodiments, the storage service allocation device 700 includes: an exception handling module, which is used to respond to a storage exception notification indicating an abnormality in a storage service unit, and the service management device marks the corresponding storage service unit as abnormal so that it can be restricted in allocation; or, in response to the storage service unit indicating the abnormality returning to normal, eliminate the abnormal mark.

[0155] It should be noted that in Figure 6 , Figure 7 Each functional module in the embodiment can be implemented in whole or in part by software, hardware, firmware or any combination thereof. When implemented using software, it can be implemented in whole or in part in the form of a program instruction product. The program instruction product includes one or more program instructions. When loading and executing program instructions on a computer, the process or function according to the present application is generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network or other programmable devices. The program instructions can be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium.

[0156] and, Figure 6 , Figure 7 The device disclosed in the embodiment can be implemented by other module division methods. The device embodiments shown above are only schematic. For example, the module division is only a logical function division. There may be other division methods in actual implementation, such as multiple modules or modules can be combined or dynamically transferred to another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, and the indirect coupling or communication connection of devices or modules can be electrical or other forms.

[0157] in addition, Figure 6 , Figure 7Each functional module and submodule in the embodiment can be dynamically in a processing component, or each module can exist physically separately, or two or more modules can be dynamically in one component. The above-mentioned dynamic components can be implemented in the form of hardware or in the form of software functional modules. If the above-mentioned dynamic components are implemented in the form of software functional modules and sold or used as independent products, they can also be stored in a computer-readable storage medium. The storage medium can be a read-only memory, a disk or an optical disk, etc.

[0158] It should be noted that the process or method represented by the flowchart of the above embodiment of the present application can be understood as a module, fragment or part of a code that represents one or more executable instructions for implementing the steps of a specific logical function or process. And the scope of the preferred embodiment of the present application includes other implementations, in which the functions may not be performed in the order shown or discussed, including performing the functions in a substantially simultaneous manner or in a reverse order according to the functions involved.

[0159] For example, Figure 2 , Figure 3 , Figure 4 The order of the steps in the embodiments may be changed in specific scenarios and is not limited to the above.

[0160] like Figure 8 As shown, a schematic diagram of the circuit structure of a computer device in one embodiment of the present application is shown.

[0161] In some embodiments, the computer device 800 can be implemented in, for example, a server, a server group, a desktop computer, etc.

[0162] The computer device 800 includes a bus 801, a processor 802, a memory 803, and a communicator 804. The processor 802 and the memory 803 can communicate with each other through the bus 801. The memory 803 can store program instructions. The processor 802 implements the file transfer service unit or service management device in the embodiment of the present application by running the program instructions in the memory 803 to execute the steps in the file transfer method or storage service allocation method.

[0163] The bus 801 may be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus. The bus may be divided into an address bus, a data bus, a control bus, etc. For ease of representation, although Figure 1 Only one thick line is used in the diagram, but this does not mean that there is only one bus or only one type of bus.

[0164] In some embodiments, the processor 802 may be implemented as a central processing unit (CPU), a microprocessing unit (MCU), a system on chip (System On Chip), or a field programmable logic array (FPGA). The memory 803 may include a volatile memory (Volatile Memory) for temporary storage of data when running a program, such as a random access memory (Random Access Memory, RAM).

[0165] The memory 803 may also include a non-volatile memory (non-volatile memory) for data storage, such as a read-only memory (ROM), a flash memory, a hard disk drive (HDD), or a solid-state disk (SSD).

[0166] The communicator 804 is used to communicate with the outside. In a specific example, the communicator 804 may include one or more wired and / or wireless communication circuit modules. For example, the wired communication circuit module may include one or more of a wired network card, a USB module, a serial interface module, etc. For another example, the wireless communication protocol followed by the wireless communication module includes: for example, near field communication (NFC) technology, infrared (IR) technology, global system for mobile communications (GSM), general packet radio service (GPRS), code division multiple access (CDMA), wideband code division multiple access (WCDMA), time division code division multiple access (TD-SCDMA), long term evolution (LTE), Bluetooth (BT), global navigation satellite system (GNSS), etc. One or more.

[0167] In an embodiment of the present application, a computer-readable storage medium may be provided, storing program instructions, which, when executed, execute the file transmission method or storage service allocation method in the previous embodiment.

[0168] That is, the method steps in the above embodiments are implemented as software or computer code that can be stored in a recording medium (such as a CD ROM, RAM, floppy disk, hard disk or magneto-optical disk), or are implemented as computer code originally stored in a remote recording medium or a non-temporary machine-readable medium downloaded via a network and to be stored in a local recording medium, so that the method represented herein can be stored in such software processing on a recording medium using a general-purpose computer, a dedicated processor, or programmable or dedicated hardware (such as an ASIC or FPGA).

[0169] In summary, the file transfer method, storage service allocation method, device, equipment and medium provided in the embodiments of the present application, the service device collects and calculates the capacity performance parameters of the storage service unit, and extracts the corresponding transmission performance parameters according to the notification completed by the storage service unit, and combines the storage performance parameters of the accurate storage service unit performance to determine each storage service unit through relevant preset storage service allocation rules to provide them to the file transfer service unit. The file transfer service unit queries the storage service unit that matches the transferred file in the candidate storage service list, thereby completing the execution of the file transfer task, and can form a notification to feedback to the service device to form a closed loop, so that the effect of balanced storage service allocation continues to be strengthened, storage performance is optimized, and file transfer failures, high availability guarantees and other conditions can also be used to exclude unreliable storage service units, thereby improving the reliability of storage services.

[0170] It should be noted that some features of the solution in the embodiments of the present application need to be paid attention to:

[0171] 1) Decoupling storage service users and storage services

[0172] Storage users can obtain storage services for actual use from the service management device only when they need to use the storage services (insufficient or expired), rather than having the service management device forcibly allocate them before use, thereby achieving the decoupling and facilitating storage capacity expansion; in addition, the service management device can also schedule and load balance the storage services to optimize storage performance.

[0173] 2) Storage scheduling and load balancing control take into account file transfer rate

[0174] Not only does it take into account the balance of storage utilization of storage capacity, but it also refers to the file transfer rates of each destination storage unit used by the file transfer service unit to transfer files recently during scheduling, so as to ensure balanced use of capacity while giving priority to candidate storage service units with faster transmission speeds.

[0175] 3) Multiple candidate storage and adjustment exit mechanism

[0176] One of the main purposes of equipping multiple storage service units is to ensure high availability of the system. When allocating storage units in the embodiments of the present application, multiple candidate storage service units are allocated to ensure high availability, and an adjustment and exit mechanism for candidate storage service units is provided. In addition, the feature of using multiple threads to upload files to multiple storage service units concurrently is retained. While ensuring high availability, storage performance is improved as much as possible.

[0177] The above embodiments are merely illustrative of the principles and effects of the present application and are not intended to limit the present application. Anyone familiar with the technology may modify or change the above embodiments without violating the spirit and scope of the present application. Therefore, all equivalent modifications or changes made by a person of ordinary skill in the art without departing from the spirit and technical ideas disclosed in the present application shall still be covered by the claims of the present application.

Claims

1. A file transmission method, characterized in that: The method is applied to a file transmission service unit to transmit a video file to a storage service unit allocated by a service management device; the method comprises: The file transfer service unit puts the transferred file into the file transfer queue according to the file transfer task; The file transfer service unit determines a destination storage service unit for storing the transferred file according to a candidate storage service list; wherein the candidate storage service list includes a plurality of storage service units; each storage service unit is determined by the service management device from each available storage service unit according to storage performance parameters of each storage service unit using a preset storage service allocation rule, and provided to the file transfer service unit; The file transfer service unit uses the destination storage service unit to complete the file transfer task; In response to the completion of the file transfer task, the file transfer service unit sends a notification of successful execution of the transfer task to the service management device. The notification of successful execution of the transfer task includes the transmission performance parameters of the corresponding destination storage service unit, which is used as a reference for calculating the storage performance parameters when redetermining the storage service unit. The storage performance parameters are calculated by the service management device based on the transmission performance parameters and the collected capacity performance parameters.

2. The file transmission method according to claim 1, characterized in that: The file transfer service unit determines a destination storage service unit for storing the transferred file according to the candidate storage service list, including: Determine whether the candidate storage service list meets preset conditions; wherein the preset conditions include: sufficient available storage service units in the candidate storage service list; and / or, the non-update time of the candidate storage service list exceeds a preset time threshold; In response to satisfying the preset condition, the file transfer service unit determines a storage service unit in the candidate storage service list that matches the transferred file as a destination storage service unit; In response to not meeting the preset condition, the file transfer service unit sends a candidate storage service allocation request to the service management device to request allocation of a storage service unit; and determines a target storage service unit based on a candidate storage service list updated corresponding to the allocation.

3. The file transmission method according to claim 1, characterized in that: The file transfer service unit determines a destination storage service unit for storing the transferred file according to the candidate storage service list, including: Avoid selecting a storage service unit that has failed in the previous n times in the candidate storage service list as the target storage service unit, where n≥1.

4. The file transmission method according to claim 1, characterized in that: Also includes: Execute exception handling actions based on the statistical results of transmission failure events.

5. The file transmission method according to claim 4, characterized in that: The performing of an exception handling action based on the statistical result of the transmission failure event includes at least one of the following: 1) In response to the number of file transmission failures exceeding a first preset number threshold, the file is removed from the file transmission queue, added to the file transmission failure queue, and a file transmission failure notification is sent to the service management device; 2) In response to the number of consecutive file transmission failures that have occurred in the candidate storage service unit exceeding a second preset number threshold, the candidate storage service unit is removed from the candidate storage service list, and a storage service abnormality notification is sent to the service management device.

6. The file transmission method according to claim 1, characterized in that: include: Excluding the transferred files whose sizes exceed a preset capacity threshold from the file transfer queue; The preset capacity threshold is related to the maximum capacity of the storage service unit.

7. A storage service allocation method, characterized in that: The method is applied to a service management device to allocate a storage service unit for a file transmission service unit; the method comprises: The service management device collects storage usage information of each storage service unit and calculates capacity performance parameters; The service management device extracts the transmission performance parameters of the corresponding destination storage service unit according to the transmission task execution success notification received from the file transmission service unit regarding the completion of the file transmission task; The destination storage service unit is used to store the video files after the transmission is completed; The service management device calculates storage performance parameters based on the transmission performance parameters and capacity performance parameters corresponding to each storage service unit; The service management device determines a candidate storage service unit from each available storage service unit based on the storage performance parameters of each storage service unit by using a preset allocation rule; The service management device sends the information of the determined storage service unit to the file transfer service unit in response to the candidate service allocation request.

8. The storage service allocation method according to claim 7, characterized in that: The service management device calculates the storage performance parameter based on the transmission performance parameter and the capacity performance parameter corresponding to each storage service unit, including: A weighted result of the transmission performance parameter and the capacity performance parameter is calculated.

9. The storage service allocation method according to claim 7, characterized in that: The service management device determines the storage service unit based on the storage performance parameters of each storage service unit by using a preset allocation rule, including: The service management device selects a preset allocated number of storage service units with higher storage performance parameters for determining the storage service unit therefrom.

10. The storage service allocation method according to claim 7 or 9, characterized in that: The service management device determines the storage service unit based on the storage performance parameters of each storage service unit by using a preset allocation rule, and further includes: Storage service units whose capacity performance parameters exceed a preset utilization threshold are excluded from allocation.

11. The storage service allocation method according to claim 7, characterized in that: include: In response to a candidate storage service allocation request of a file transfer service unit, the service management device determines a storage service unit to respond to the candidate storage service allocation request.

12. The storage service allocation method according to claim 11, characterized in that: The candidate storage service allocation request is triggered by a preset condition not being met, wherein the preset condition includes: the candidate storage service in the candidate storage service list of the file transfer service unit is sufficient to complete the file transfer task; And / or, the candidate storage service list has not been updated for a period exceeding a preset time threshold.

13. The storage service allocation method according to claim 7, characterized in that: include: In response to the number of available storage service units being lower than a preset allocation number, the service management device uses each of the available storage service units as the candidate storage service unit.

14. The storage service allocation method according to claim 7, characterized in that: include: In response to a storage anomaly notification indicating that a storage service unit is abnormal, the service management device marks the corresponding storage service unit as abnormal so as to be restricted from allocation; Alternatively, in response to the storage service unit indicating the abnormality returning to normal, the mark of the abnormality is eliminated.

15. A file transmission device, characterized in that: The device is applied to a file transmission service unit to transmit a video file to a storage service unit allocated by a service management device; the device comprises: The transmission task execution module is used to put the transferred files into the file transmission queue according to the file transmission task; a destination storage determination module, configured to determine a destination storage service unit for storing the transferred file according to a candidate storage service list; wherein the candidate storage service list includes a plurality of storage service units; each of the storage service units is determined by the service management device from among the available storage service units according to storage performance parameters of each storage service unit using a preset storage service allocation rule, and provided to the file transfer service unit; The transmission task execution module is used to complete the file transmission task using the destination storage service unit; An execution result reporting module is used to send a transmission task execution success notification to a service management device in response to the completion of a file transfer task. The transmission task execution success notification includes the transmission performance parameters of the corresponding destination storage service unit, which are used as a reference for calculating the storage performance parameters when redetermining the storage service unit. The storage performance parameters are calculated by the service management device based on the transmission performance parameters and the collected capacity performance parameters.

16. A storage service allocation device, characterized in that: A service management device is used to allocate a storage service unit for file transmission for a file transmission service unit; the device comprises: The storage information collection module is used to collect storage usage information of each storage service unit and calculate capacity performance parameters; An information extraction module, used to extract the transmission performance parameters of the corresponding destination storage service unit according to the transmission task execution success notification received from the file transmission service unit regarding the completion of the file transmission task; A storage performance calculation module, used to calculate storage performance parameters based on the transmission performance parameters and capacity performance parameters corresponding to each storage service unit; An information generation module, configured to determine a candidate storage service unit from each available storage service unit based on storage performance parameters of each storage service unit by using a preset allocation rule; The sending module is used to send the information of the determined storage service unit to the file transfer service unit in response to the candidate service allocation request, so as to determine the target storage service unit for storing the transferred file in the file transfer task.

17. A computer device, characterized in that: include: Communicator, memory and processor; The communicator is used for communicating with the outside; The memory is used to store program instructions; the processor is used to run the program instructions to execute the file transmission method according to any one of claims 1 to 6; Alternatively, execute the storage service allocation method as described in any one of claims 7 to 14.

18. A computer-readable storage medium, characterized in that: Program instructions are stored, and when the program instructions are executed, the file transmission method according to any one of claims 1 to 6 is executed; or the storage service allocation method according to any one of claims 7 to 14 is executed.