Front-end large file uploading method and device

Abstract

The embodiment of the invention provides a front-end large file uploading method and device, and the method comprises the steps: receiving a large file uploaded by a user through a front end, carrying out the dynamic fragmentation of the large file according to a set dynamic fragmentation model, determining each corresponding fragment, carrying out the priority judgment of each fragment, and determining a corresponding priority order, the method comprises the following steps: determining a corresponding concurrent uploading fragment quantity according to a network bandwidth, a load and a preset concurrent quantity rule, carrying out concurrent uploading operation on each fragment according to a priority sequence and the concurrent uploading fragment quantity according to a preset multiplexing technology, judging whether concurrent uploading of each fragment is successful or not, and if the concurrent uploading fails, executing the concurrent uploading operation on each fragment. According to the method and the device, file fragmentation and concurrent uploading can be carried out on the basis of dynamically adjusting the sizes of the slices and the concurrent quantity, and the efficiency and the flexibility of uploading the large file are improved. The method and the device have the advantages that the file fragmentation and the concurrent uploading can be carried out on the basis of dynamically adjusting the sizes of the slices and the concurrent quantity, and the uploading efficiency and the uploading flexibility of the large file are improved.

Description

Technical Field

[0001] This application relates to the field of data processing, and particularly to a front-end large file upload method and device. Background Art

[0002] In existing large file upload technologies, files are usually segmented in a way with a fixed slice size, and these slices are uploaded to the server one by one. However, their upload speed and network resource utilization rate are often limited.

[0003] In response to this situation, there is a way to improve the upload speed and network resource utilization rate through concurrent upload. However, a fixed number of concurrent upload slices is adopted. This method may perform well in a static or relatively stable network environment, but it has obvious deficiencies in a dynamically changing network environment.

[0004] The network environment is highly dynamic, and its bandwidth, latency, and stability change over time and location. When uploading a large file, if the network environment suddenly deteriorates (such as network congestion, bandwidth decrease, etc.), the fixed number of concurrent upload slices may cause some upload requests to fail or time out, thereby increasing the number of retries and upload time and reducing the upload efficiency.

[0005] Therefore, there is an urgent need for a front-end large file upload method to improve the efficiency and flexibility of front-end large file upload in a dynamic network environment. Summary of the Invention

[0006] Aiming at the problems in the prior art, this application provides a front-end large file upload method and device, which can perform file sharding and concurrent upload based on dynamically adjusting the slice size and concurrency, improving the efficiency and flexibility of large file upload.

[0007] To solve at least one of the above problems, this application provides the following technical solutions: In a first aspect, this application provides a front-end large file upload method, including: The front end receives a large file uploaded by a user, dynamically segments the large file according to the network data during the upload of the large file and a set dynamic sharding model to determine the corresponding shards, performs content detection on the shards, judges the content features included in each shard, sorts the shards according to the content features to determine the corresponding priority order; Obtain the network bandwidth and load of the server in real time, determine the corresponding number of concurrent upload shards according to the network bandwidth, the load, and a preset concurrency rule, and perform a concurrent upload operation on the shards according to the priority order and the number of concurrent upload shards by using a preset multiplexing technology; Determine whether the concurrent upload of each slice is successful. If it fails, retransmit the slice that failed to upload according to the breakpoint resumption technology. If it is successful, trigger the backend merging operation to splice the slices into a complete file, and verify the complete file according to the preset hash algorithm to obtain the complete large file after upload.

[0008] Further, before dynamically slicing the large file according to the network data during the upload of the large file and the set dynamic slicing model to determine the corresponding slices, it includes: Determine the corresponding initial dual-network model according to the preset main network and the preset target network; Perform model training on the initial dual-network model according to the set experience buffer to determine the corresponding dynamic slicing model, where the experience buffer is used to store historical experiences.

[0009] Further, before performing model training on the initial dual-network model according to the set experience buffer to determine the corresponding dynamic slicing model, it includes: Collect historical network data, extract the network bandwidth change trend in the historical network data according to the sliding window technology, construct the corresponding state space according to the network bandwidth change trend, the preset network bandwidth, and the preset slice size, define the action space, and adjust the slice change amplitude of the action space according to the preset slice dynamic adjustment technology to determine the corresponding continuous action space; The preset main network selects an action for the current state in the state space according to the greedy algorithm, and selects the action corresponding to the current state within the continuous action space, where the action is the action with the smallest Q value; Determine the corresponding reward and new state according to the execution result of the action, and construct an experience buffer according to the current state, the action, the reward, and the new state.

[0010] Further, performing model training on the initial dual-network model according to the set experience buffer to determine the corresponding dynamic slicing model includes: Select an action for the new state in the experience buffer according to the main network in the initial dual-network model to determine the corresponding optimal action; Evaluate the optimal action according to the target network in the initial dual-network model to determine the optimal target Q value, and update the main network parameters according to the optimal target Q value to obtain the corresponding dynamic slicing model.

[0011] Further, dynamically slicing the large file according to the network data during the upload of the large file and the set dynamic slicing model to determine the corresponding slices includes: Dynamically fragment the large file according to the network data during the upload of the large file and the set dynamic fragmentation model, and determine the corresponding fragment size; Perform a slicing operation on the large file according to the preset file.slice method and the fragment size, and determine the corresponding fragments.

[0012] Further, the prioritizing the respective fragments according to the content characteristics to determine the corresponding priority order includes: If it is a video feature, identify key frames for the video feature fragments according to the preset frame difference method to determine the corresponding priority order; if it is a text feature, extract keywords for the text feature fragments according to the preset natural language processing technology to determine the corresponding priority order; if it is an image feature, identify the main body for the image feature fragments according to the preset saliency detection algorithm to determine the corresponding priority order.

[0013] Further, the prioritizing the respective fragments according to the content characteristics to determine the corresponding priority order further includes: If it is a video feature, identify the scene change key frames of the video feature fragments according to the optical flow method to determine the fragments with key frames; Divide the fragments with key frames into high-priority fragments to determine the corresponding priority order.

[0014] In a second aspect, the present application provides a front-end large file upload device, including: A large file fragmentation module, configured to receive a large file uploaded by a user at the front end, dynamically fragment the large file according to the network data during the upload of the large file and the set dynamic fragmentation model, determine the corresponding fragments, perform content detection on the respective fragments, judge the content characteristics included in each fragment, prioritize the respective fragments according to the content characteristics, and determine the corresponding priority order; A concurrent upload module, configured to obtain the network bandwidth and load of the server in real time, determine the corresponding number of concurrently uploaded fragments according to the network bandwidth, the load, and the preset concurrency rule, and perform a concurrent upload operation on the respective fragments according to the preset multiplexing technology in accordance with the priority order and the number of concurrently uploaded fragments; An upload verification module, configured to judge whether the concurrent upload of the respective fragments is successful. If it fails, retransmit the failed slices according to the breakpoint resumption technology. If it is successful, trigger a back-end merging operation, splice the respective slices into a complete file, and verify the complete file according to the preset hash algorithm to obtain the complete large file after upload.

[0015] In a third aspect, the present application provides an electronic device, including a memory, a processor, and a computer program stored on the memory and executable on the processor. When the processor executes the program, the steps of the front-end large file uploading method are implemented.

[0016] In a fourth aspect, the present application provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the steps of the front-end large file uploading method are implemented.

[0017] In a fifth aspect, the present application provides a computer program product, including a computer program / instructions. When the computer program / instructions are executed by a processor, the steps of the front-end large file uploading method are implemented.

[0018] As can be seen from the above technical solutions, the present application provides a front-end large file uploading method and device. By receiving a large file uploaded by a user at the front end, dynamically slicing the large file according to a set dynamic slicing model, determining the corresponding slices, determining the corresponding priority order for each slice, determining the corresponding number of concurrently uploaded slices according to network bandwidth, load, and a preset concurrency rule, performing a concurrent upload operation on each slice according to the priority order and the number of concurrently uploaded slices by using a preset multiplexing technology, determining whether the concurrent upload of each slice is successful. If it fails, retransmit the slice that fails to upload according to the breakpoint resumption technology. If it is successful, trigger a back-end merging operation to obtain the complete large file after uploading. Thus, it is possible to perform file slicing and concurrent upload based on dynamically adjusting the slice size and concurrency, improving the efficiency and flexibility of large file uploading. Description of the Drawings

[0019] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the following will briefly introduce the drawings required for use in the description of the embodiments or the prior art. Obviously, the drawings in the following description are some embodiments of the present application. For those of ordinary skill in the art, other drawings can be obtained based on these drawings without creative efforts.

[0020] Figure 1 It is one of the flow diagrams of the front-end large file uploading method in the embodiments of the present application; Figure 2 It is another flow diagram of the front-end large file uploading method in the embodiments of the present application; Figure 3 It is yet another flow diagram of the front-end large file uploading method in the embodiments of the present application; Figure 4 It is still another flow diagram of the front-end large file uploading method in the embodiments of the present application; Figure 5It is the fifth flowchart of the front-end large file upload method in the embodiments of the present application; Figure 6 It is the sixth flowchart of the front-end large file upload method in the embodiments of the present application; Figure 7 It is the seventh flowchart of the front-end large file upload method in the embodiments of the present application; Figure 8 It is the structural diagram of the front-end large file upload device in the embodiments of the present application; Figure 9 It is the structural schematic diagram of the electronic device in the embodiments of the present application.

[0021] Reference numerals: Electronic device 9600, central processing unit 9100, memory 9140, communication module 9110, input unit 9120, audio processor 9130, display 9160, power supply 9170, buffer memory 9141, application / function storage unit 9142, data storage unit 9143, driver program storage unit 9144, antenna 9111, speaker 9131, microphone 9132. Detailed implementation manners

[0022] To make the objectives, technical solutions and advantages of the embodiments of the present application clearer, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings in the embodiments of the present application. Apparently, the described embodiments are some, but not all, of the embodiments of the present application. All other embodiments obtained by those of ordinary skill in the art based on the embodiments of the present application without making creative efforts shall fall within the scope of protection of the present application.

[0023] In the technical solutions of the present application, the acquisition, storage, use, processing, etc. of data all comply with the relevant provisions of national laws and regulations.

[0024] Considering the problem that in the existing large file upload technology, the file is usually segmented in a way of fixed slice size and these slices are uploaded to the server one by one, and its upload speed and network resource utilization rate are often limited. This application provides a front-end large file upload method and device. The front-end receives a large file uploaded by the user, dynamically segments the large file according to a set dynamic segmentation model, determines the corresponding segments, determines the priority of each segment, determines the corresponding priority order, determines the corresponding number of concurrent upload segments according to the network bandwidth, load and preset concurrency rules, performs concurrent upload operations on each segment according to the preset multiplexing technology in the priority order and the number of concurrent upload segments, determines whether the concurrent upload of each segment is successful. If it fails, the failed slice is retransmitted according to the breakpoint resumption technology. If it is successful, the back-end merging operation is triggered to obtain the complete large file after upload. Thus, file segmentation and concurrent upload can be performed based on dynamically adjusting the slice size and concurrency, improving the efficiency and flexibility of large file upload.

[0025] In order to perform file segmentation and concurrent upload based on dynamically adjusting the slice size and concurrency, and improve the efficiency and flexibility of large file upload, this application provides an embodiment of a front-end large file upload method. Refer to Figure 1 , the specific content of the front-end large file upload method includes the following: Step S101: The front-end receives a large file uploaded by the user, dynamically segments the large file according to the network data when the large file is uploaded and a set dynamic segmentation model, determines the corresponding segments, performs content detection on the segments, determines the content characteristics included in each segment, and sorts the segments according to the content characteristics to determine the corresponding priority order; Optionally, in this embodiment, this step is a process of dynamically segmenting the large file uploaded by the user and determining the priority order of each segment.

[0026] Specifically, in this embodiment, a dynamic segmentation model is set for dynamically segmenting the large file uploaded by the user.

[0027] Optionally, in this embodiment, the dynamic segmentation model intelligent agent is constructed.

[0028] In order to be able to dynamically adjust the slice size according to the network environment and improve the user experience, this step models the slice size adjustment problem as a Markov decision process (MDP). An intelligent agent including a state space and an action space is constructed, and then the intelligent agent is trained through the model so that it can automatically adjust the slice size in different network environments to maximize the upload efficiency.

[0029] Specifically, the state space describes the current environmental state of the system, including: Current network bandwidth: The network bandwidth measured in real time (unit: MB / s).

[0030] Current slice size: The slice size currently in use (unit: MB).

[0031] Upload progress: The proportion of the file that has been uploaded (0% to 100%).

[0032] Network latency: The latency time of the current network (unit: ms).

[0033] Packet loss rate: The packet loss rate of the current network (unit: %).

[0034] Preferably, for the state space, the historical bandwidth change trend is innovatively introduced. The bandwidth change trend over a certain period of time (such as the bandwidth change in the past 5 seconds) is statistically calculated through the sliding window technique. The historical bandwidth change trend is calculated through the sliding window mechanism to ensure that the state space can reflect the latest network environment. At the same time, the historical bandwidth change trend is used as a dimension of the state space, enabling the model to capture the dynamic changes in network bandwidth, thereby making more accurate slice adjustment decisions.

[0035] Specifically, the action space describes the slice adjustment strategies that the system can adopt, including: Increase slice size: Increase the slice size by a certain proportion (such as 20%).

[0036] Decrease slice size: Decrease the slice size by a certain proportion (such as 20%).

[0037] Keep slice size: Do not change the current slice size.

[0038] Preferably, for the action space, on the basis of the fixed action space, a continuous action space is introduced, enabling the model to output any size of slice adjustment ratio instead of a fixed few options; combined with the continuous action space, a dynamic adjustment ratio is further introduced, enabling the model to dynamically adjust the change range of the slice size according to the current network environment. For example, when the network bandwidth is high, the proportion of increasing the slice size can be larger; when the network bandwidth is low, the proportion of decreasing the slice size can be smaller. By introducing the dynamic adjustment ratio, the model can dynamically adjust the change range of the slice size according to the current network environment, enhancing the flexibility of the model.

[0039] Specifically, the reward function is used to evaluate the effect of each action. Designing a reasonable reward function can guide the model to learn the optimal strategy.

[0040] In this embodiment, the reward function introduces a multi-objective reward function, enabling the model to simultaneously optimize multiple objectives such as upload speed, success rate, and energy consumption. It includes: Upload speed reward: The faster the upload speed, the higher the reward.

[0041] Success rate reward: The higher the success rate of slice upload, the higher the reward.

[0042] Energy consumption penalty: If the slice adjustment causes an increase in device energy consumption, a penalty is given.

[0043] Stability reward: If the upload speed remains stable after the slice size is adjusted, a reward is given.

[0044] Optionally, the reward function uses an adaptive reward weight, which dynamically adjusts the weight of the reward function according to the current network environment and upload progress. For example, in the initial stage of upload, more attention can be paid to the upload speed; in the later stage of upload, more attention can be paid to the success rate. By introducing a multi-objective reward function, the model can optimize multiple objectives such as upload speed, success rate, and energy consumption at the same time, improving the overall efficiency of the upload process.

[0045] After defining the state space, action space, and reward function, an agent for model training can be obtained.

[0046] Optionally, in this embodiment, the dynamic sharding model is defined.

[0047] When defining the DQN model, when DQN calculates the target Q value through the target network, it directly uses the maximum Q value output by it as the update benchmark. This design where the same network is responsible for both "selecting actions" and "evaluating Q values" easily leads to an over-preference for high-reward actions, and then causes a systematic overestimation of Q values. Therefore, in this embodiment, two independent Q networks are introduced to solve the problem of Q value estimation bias.

[0048] Specifically, a main network and a target network are introduced. The main network is used to select actions, and the target network is used to evaluate the value of actions. By separating action selection and action evaluation, Double DQN can reduce the bias of Q value estimation, thereby improving the stability and performance of the model.

[0049] First, initialize the main network and the target network. They have the same structure but independent parameters. The parameters of the target network are synchronized from the main network every once in a while (such as every 100 steps). Then, construct an experience buffer according to the main network. At each time step, use the main network to select actions for the states in the state space. Specifically, input the current state into the main network, output the Q value of each action, and select the action with the largest Q value.

[0050] For example, assume the current state is: network bandwidth = 2MB / s, slice size = 5MB, upload progress = 30%, network latency = 50ms, packet loss rate = 0.1%, network bandwidth change trend = -0.5% The system inputs the current state into the main network, and the main network outputs the Q-values for each action: increasing the slice size (Q-value = 0.8), decreasing the slice size (Q-value = 0.9), and maintaining the slice size (Q-value = 0.7).

[0051] The system selects the action with the largest Q-value: decreasing the slice size The system executes the action of decreasing the slice size, adjusting the slice size from 5MB to 4MB.

[0052] The system observes the new state: network bandwidth = 2.5MB / s, slice size = 4MB, upload progress = 35%, network latency = 40ms, packet loss rate = 0.05%, network bandwidth change trend = 0.1% The system stores the experience (current state, action, reward, new state) in the initial experience replay buffer, and repeats the above steps to obtain the constructed experience buffer.

[0053] After the above steps, the initial double network model is obtained according to the defined double network and the constructed experience buffer, and then model training is carried out.

[0054] Optionally, in this embodiment, dynamic sharding model training.

[0055] Specifically, during the training process, the model samples a batch of experiences from the constructed experience buffer. Preferably, a prioritized experience replay mechanism is introduced to enable the model to preferentially learn important experiences (such as experiences with higher rewards), thereby accelerating the model's learning process.

[0056] For each sampled experience, the main network is used to select the optimal action in the new state, but the target network is used to evaluate the Q-value of this action.

[0057] Calculate the target Q-value: target Q-value = reward + discount factor × Q-value evaluated by the target network.

[0058] Use the target Q-value to update the parameters of the main network, minimizing the loss of the Q-value function, thereby obtaining the dynamic sharding model.

[0059] Every once in a while (such as every 100 steps), copy the parameters of the main network to the target network to ensure that the parameters of the target network are synchronized with the main network.

[0060] By introducing Double DQN, the system can reduce the bias of Q-value estimation, thereby improving the stability and performance of the model. In the scenario of large file upload, Double DQN can more accurately dynamically adjust the slice size, adapt to complex network environments, and significantly improve the upload efficiency and stability.

[0061] Optionally, in this embodiment, dynamic sharding model application.

[0062] By monitoring network bandwidth, large files uploaded by users are segmented based on a trained dynamic segmentation model and the current network conditions, dynamically calculating the optimal segment size. For good network conditions, the segment size can be increased to reduce the number of requests; for poor network conditions, the segment size can be reduced. Once the segment size is determined, the large file is segmented accordingly, and each segment is uniquely identified.

[0063] Optionally, in this embodiment, content detection and priority order division are performed on each obtained fragment.

[0064] Specifically, depending on the type of the large file, the large file may include a video file, an image file, and a text file.

[0065] For large files with video features, a priority control technology is to identify key frames (I frames) in the video file, classify the video fragments with I frames as high-priority fragments, and the remaining fragments as ordinary fragments. When the network environment changes, high-priority fragments are uploaded first to ensure user experience.

[0066] Another priority control technology is to prioritize slices based on the file structure. This technology identifies keyframe locations by parsing the file format's metadata. Open-source multimedia libraries (such as FFmpeg, GStreamer, and OpenCV) are used to parse video files, obtain keyframe timestamps or frame numbers, and then classify video slices containing keyframes as high-priority slices, while the remaining slices are classified as normal slices. When network conditions change, high-priority slices are uploaded first to ensure a better user experience.

[0067] Another priority control technology is content-based slicing priority setting. This technology analyzes video content and identifies keyframes that change scenes. This technology uses the frame difference method, a computer vision technology, or alternatively, the optical flow method, which detects scene changes and identifies keyframes. Slices containing keyframes are then classified as high-priority slices, while the remaining slices are classified as standard slices. When network conditions change, high-priority slices are uploaded first to ensure a better user experience.

[0068] Another priority control technology is to identify key slices based on the hash value of the slice, use MD5 to calculate the hash for each slice, identify slices with large content changes, obtain key frames, and divide the video slices with key frames into high-priority slices, and the remaining slices into ordinary slices. When the network environment changes, high-priority slices are uploaded first to ensure user experience.

[0069] For large files with image features, feature extraction is performed on the images based on computer vision technology to obtain their edge and texture features. Then, the main object or significant region in the image is identified, the importance of the content is evaluated, and high priority is set for the slices with important content. When the network environment changes, high-priority slices are preferentially uploaded to ensure the user experience.

[0070] For large files with text features, natural language processing (NLP) technology is used to extract features from the text to obtain keywords, titles in the document, or chart features. According to the changes and importance of the content, the priority of the slices is dynamically adjusted to ensure that important content is uploaded first.

[0071] The above is the slicing priority division rule when there is only one type of large file.

[0072] When the large file is a mixed large file, for example, assume that the user uploads a mixed large file containing video, document, and image: Video part: Use the frame difference method to detect scene changes and identify key frames.

[0073] Mark the slices containing key frames as high priority.

[0074] Document part: Use NLP technology to extract keywords and titles.

[0075] Mark the pages containing keywords or titles as high priority.

[0076] Image part: Use object detection technology to identify the main object.

[0077] Mark the regions containing the main object as high priority.

[0078] During the upload process, high-priority slices, pages, and regions are preferentially uploaded to ensure that users can quickly preview important content.

[0079] It can be understood that through the above steps, we have achieved dynamically determining the slice size for large file slicing according to the network environment, and dividing priorities for the slices, ensuring that high-priority slices are preferentially uploaded during each batch of slice uploads, and guaranteeing the efficiency and flexibility of the upload.

[0080] Step S102: Real-time obtain the network bandwidth and load of the server, determine the corresponding concurrent upload slice quantity according to the network bandwidth, the load, and the preset concurrency rule, and perform concurrent upload operations on the respective slices according to the priority order and the concurrent upload slice quantity by using the preset multiplexing technology; Optionally, in this embodiment, this step is a process of dynamically adjusting the concurrency data by detecting the network status and performing concurrent uploads in the order of concurrency and priority.

[0081] Specifically, the upload bandwidth of the current network is measured in real time through the Network Information API of front-end JavaScript. The server returns the current load status to the front end through the API interface, and the front end adjusts the number of concurrent uploads according to the load situation.

[0082] The concurrency rule is that if the network bandwidth is high and the server load is low, the number of concurrent uploads is increased. If the network bandwidth is low or the server load is high, the number of concurrent uploads is decreased.

[0083] Among them, the server load factor is a value between 0 and 1, indicating the load situation of the server (0 means no load, 1 means full load).

[0084] The front end regularly (such as every 10 seconds) obtains the current network bandwidth and server load, and calculates the current number of concurrent uploads according to the preset concurrency rule. If the calculated number of concurrences is inconsistent with the currently used number of concurrences, the number of shards for concurrent upload is dynamically adjusted.

[0085] On the basis of obtaining the concurrency, the HTTP / 2 multiplexing technology is used to perform concurrent uploads according to the priority and concurrency of the shards. HTTP / 2 allows multiple request and response streams (Streams) to be transmitted simultaneously on a single TCP connection. Each stream can transmit data independently and will not block other streams.

[0086] The complete shard upload process is as follows: Accept the large file uploaded by the user, and calculate the shard size according to the network status and the dynamic sharding model; The large file is split into multiple shards according to the shard size (for example, each shard size is 1MB).

[0087] Generate a unique identifier (such as a hash value) for each shard, and set the upload priority based on the priority control technology.

[0088] According to the dynamically adjusted number of concurrent uploads, the shards are concurrently uploaded to the server according to the upload priority through the HTTP / 2 multiplexing technology.

[0089] After the server receives the shards, it stores them in a temporary location and records the order and identifier of the shards.

[0090] Step S103: Determine whether the concurrent uploads of the respective slices are successful. If the upload fails, retransmit the slices that failed to upload according to the breakpoint resumption technology. If the upload is successful, trigger a backend merging operation to splice the respective slices into a complete file, and verify the complete file according to a preset hashing algorithm to obtain the complete large file after upload.

[0091] Optionally, in this embodiment, in this step, it is a process of verifying the uploaded slices.

[0092] Specifically, the breakpoint resumption technology means that during the upload process, if a certain slice fails to upload, the system can automatically retry and support resuming the upload from the breakpoint instead of uploading the entire file again.

[0093] Generate a unique identifier (such as a hash value) for each slice and record the failed slices when the upload fails. When resuming the upload, only upload the slices that have not been successfully uploaded instead of uploading all slices again.

[0094] This example shows how this embodiment dynamically adjusts the large file slice size based on a reinforcement learning model to obtain large file slices and assign slice priorities, and then uploads the slices through concurrent uploads to ensure the efficiency and flexibility of large file uploads.

[0095] As can be seen from the above description, the front-end large file upload method provided by the embodiments of the present application can receive a large file uploaded by a user through the front end, dynamically slice the large file according to a set dynamic slicing model to determine the respective corresponding slices, determine the corresponding priority order by performing a priority determination on the respective slices, determine the corresponding number of concurrently uploaded slices according to the network bandwidth, load, and preset concurrency rules, perform a concurrent upload operation on the respective slices in the priority order and the number of concurrently uploaded slices according to a preset multiplexing technology, determine whether the concurrent uploads of the respective slices are successful, if the upload fails, retransmit the slices that failed to upload according to the breakpoint resumption technology, if the upload is successful, trigger a backend merging operation to obtain the complete large file after upload, thereby enabling file slicing and concurrent uploads based on dynamic adjustment of the slice size and concurrency, and improving the efficiency and flexibility of large file uploads.

[0096] In an embodiment of the front-end large file upload method of the present application, refer to Figure 2 , and it may specifically include the following content: Step S201: Determine a corresponding initial dual-network model according to a preset main network and a preset target network; Step S202: Train the initial dual-network model according to a set experience buffer to determine a corresponding dynamic slicing model, where the experience buffer is used to store historical experiences.

[0097] Optionally, in this embodiment, this step is a process of constructing and training a network model.

[0098] Specifically, when defining the DQN model, when DQN calculates the target Q value through the target network, it directly uses the maximum Q value output by it as the update benchmark. This design where the same network is responsible for both "selecting actions" and "evaluating Q values" easily leads to an over-preference for high-reward actions, and then causes a systematic overestimation of Q values. Therefore, in this embodiment, two independent Q networks are introduced to solve the problem of Q value estimation bias.

[0099] Specifically, a main network and a target network are introduced. The main network is used to select actions, and the target network is used to evaluate the value of actions. By separating action selection and action evaluation, Double DQN can reduce the bias of Q value estimation, thereby improving the stability and performance of the model.

[0100] Specifically, a buffer is set up. An experience buffer is constructed according to the main network. At each time step, the main network is used to select actions for the states in the state space to construct the experience buffer.

[0101] Specifically, during the training process, the model samples a batch of experiences from the constructed experience buffer. Preferably, a prioritized experience replay mechanism is introduced to enable the model to preferentially learn important experiences (such as experiences with higher rewards), thereby accelerating the learning process of the model.

[0102] For each sampled experience, the main network is used to select the optimal action in the new state, but the target network is used to evaluate the Q value of this action.

[0103] Calculate the target Q value: Target Q value = Reward + Discount factor × Q value evaluated by the target network.

[0104] Use the target Q value to update the parameters of the main network, minimize the loss of the Q value function, thereby obtaining the dynamic sharding model.

[0105] Through step S202, in this embodiment, by introducing Double DQN, the system can reduce the bias of Q value estimation, thereby improving the stability and performance of the model.

[0106] In an embodiment of the front-end large file upload method of the present application, refer to Figure 3 , it may further specifically include the following content: Step S301: Collect historical network data, extract the network bandwidth change trend in the historical network data according to the sliding window technique, construct a corresponding state space according to the network bandwidth change trend, preset network bandwidth, and preset slice size, define the action space, and adjust the slice change amplitude of the action space according to the preset slice dynamic adjustment technique to determine the corresponding continuous action space; Step S302: The preset main network selects an action for the current state in the state space according to the greedy algorithm, and selects an action corresponding to the current state in the continuous action space, where the action is the action with the minimum Q value; Step S303: Determine the corresponding reward and new state according to the execution result of the action, and construct an experience buffer according to the current state, the action, the reward, and the new state.

[0107] Optionally, in this embodiment, this step is the construction process of the experience buffer.

[0108] Specifically, the state space describes the current environmental state of the system, including: Current network bandwidth: The network bandwidth measured in real time (unit: MB / s).

[0109] Current slice size: The slice size currently used (unit: MB).

[0110] Upload progress: The ratio of the uploaded file (0% to 100%).

[0111] Network latency: The latency time of the current network (unit: ms).

[0112] Packet loss rate: The packet loss rate of the current network (unit: %).

[0113] Preferably, for the state space, the historical bandwidth change trend is innovatively introduced, and the bandwidth change trend in the past period of time (such as the bandwidth change in the past 5 seconds) is statistically analyzed through the sliding window technology. The historical bandwidth change trend is calculated through the sliding window mechanism to ensure that the state space can reflect the latest network environment. At the same time, the historical bandwidth change trend is used as a dimension of the state space, enabling the model to capture the dynamic changes of the network bandwidth, so as to make more accurate slice adjustment decisions.

[0114] Specifically, the action space describes the slice adjustment strategies that the system can adopt, including: Increase the slice size: Increase the slice size by a certain proportion (such as 20%).

[0115] Decrease the slice size: Decrease the slice size by a certain proportion (such as 20%).

[0116] Keep the slice size: Do not change the current slice size.

[0117] Preferably, for the action space, based on the fixed action space, a continuous action space is introduced, enabling the model to output slice adjustment ratios of arbitrary sizes instead of a fixed number of options; further introducing a dynamic adjustment ratio in combination with the continuous action space enables the model to dynamically adjust the change range of the slice size according to the current network environment. For example, when the network bandwidth is high, the ratio of increasing the slice size can be larger; when the network bandwidth is low, the ratio of decreasing the slice size can be smaller. By introducing the dynamic adjustment ratio, the model can dynamically adjust the change range of the slice size according to the current network environment, enhancing the flexibility of the model.

[0118] Specifically, the reward function is used to evaluate the effect of each action. Designing a reasonable reward function can guide the model to learn the optimal strategy.

[0119] In this embodiment, the reward function introduces a multi-objective reward function, enabling the model to optimize multiple objectives such as upload speed, success rate, and energy consumption simultaneously. It includes: Upload speed reward: The faster the upload speed, the higher the reward.

[0120] Success rate reward: The higher the success rate of slice upload, the higher the reward.

[0121] Energy consumption penalty: If the slice adjustment causes an increase in device energy consumption, a penalty is given.

[0122] Stability reward: If the upload speed remains stable after the slice size adjustment, a reward is given.

[0123] Optionally, the reward function uses an adaptive reward weight to dynamically adjust the weight of the reward function according to the current network environment and upload progress. For example, in the initial stage of upload, more attention can be paid to the upload speed; in the later stage of upload, more attention can be paid to the success rate. By introducing the multi-objective reward function, the model can optimize multiple objectives such as upload speed, success rate, and energy consumption simultaneously, improving the overall efficiency of the upload process.

[0124] After defining the state space, action space, and reward function, an agent for model training can be obtained.

[0125] The main network and the target network are introduced. An experience buffer is constructed based on the main network. At each time step, the main network is used to select actions for the states in the state space. Specifically, the current state is input into the main network, the Q-values of each action are output, and the action with the largest Q-value is selected.

[0126] Illustrate with an example. Assume the current state is: network bandwidth = 2MB / s, slice size = 5MB, upload progress = 30%, network latency = 50ms, packet loss rate = 0.1%, network bandwidth change trend = -0.5% The system inputs the current state into the main network, and the main network outputs the Q-values for each action: increasing the slice size (Q-value = 0.8), decreasing the slice size (Q-value = 0.9), and maintaining the slice size (Q-value = 0.7).

[0127] The system selects the action with the largest Q-value: decreasing the slice size The system executes the action of decreasing the slice size, adjusting the slice size from 5MB to 4MB.

[0128] The system observes the new state: network bandwidth = 2.5MB / s, slice size = 4MB, upload progress = 35%, network latency = 40ms, packet loss rate = 0.05%, network bandwidth change trend = 0.1% The system stores the experience (current state, action, reward, new state) in the initial experience replay buffer, and repeats the above steps to obtain the constructed experience buffer.

[0129] Through step S303, this embodiment successfully constructs an experience buffer, laying a foundation for subsequent model training.

[0130] In an embodiment of the front-end large file upload method of the present application, refer to Figure 4 , and it may specifically include the following content: Step S401: Select an action for the new state in the experience buffer according to the main network in the initial dual network model to determine the corresponding optimal action; Step S402: Evaluate the optimal action according to the target network in the initial dual network model to determine the optimal target Q-value, and update the main network parameters according to the optimal target Q-value to obtain the corresponding dynamic sharding model.

[0131] Optionally, in this embodiment, this step is the process of model training.

[0132] Specifically, during the training process, the model samples a batch of experiences from the constructed experience buffer. Preferably, a prioritized experience replay mechanism is introduced to enable the model to preferentially learn important experiences (such as experiences with higher rewards), thereby accelerating the learning process of the model.

[0133] For each sampled experience, the main network is used to select the optimal action in the new state, but the target network is used to evaluate the Q-value of this action.

[0134] Calculate the target Q-value: target Q-value = reward + discount factor × Q-value evaluated by the target network.

[0135] Use the target Q-value to update the parameters of the main network, minimizing the loss of the Q-value function, thereby obtaining the dynamic sharding model.

[0136] At regular intervals (e.g., every 100 steps), copy the parameters of the main network to the target network to ensure that the parameters of the target network are synchronized with the main network.

[0137] Through step S402, this embodiment successfully trains a dynamic sharding model, laying a foundation for subsequent model applications.

[0138] In an embodiment of the front-end large file upload method of this application, refer to Figure 5 , it may specifically include the following content: Step S501: Dynamically shard the large file according to the network data during the upload of the large file and the set dynamic sharding model to determine the corresponding shard size; Step S502: Perform slicing operations on the large file according to the preset file.slice method and the shard size to determine the corresponding individual shards.

[0139] Optionally, in this step, the dynamic sharding model obtains the optimal shard size based on the network data. Subsequently, when creating file slices, use the file.slice method to shard the file according to the "optimal shard size" to generate a list of Blob objects of the slices.

[0140] Through step S502, this embodiment successfully determines each shard, laying a foundation for subsequent determination of the priority order of each shard and concurrent upload.

[0141] In an embodiment of the front-end large file upload method of this application, refer to Figure 6 , it may specifically include the following content: Step S601: If it is a video feature, identify key frames for the video feature shards according to the preset frame difference method to determine the corresponding priority order; Step S602: If it is a text feature, extract keywords from the text feature shards according to the preset natural language processing technology to determine the corresponding priority order; Step S603: If it is an image feature, identify the main body of the image feature shards according to the preset saliency detection algorithm to determine the corresponding priority order. Optionally, in this embodiment, for a large file with video features, one priority control technology is to identify the key frames (I-frames) in the video file, divide the video shards with I-frames into high-priority shards, and the remaining shards are ordinary shards. When the network environment changes, give priority to uploading high-priority shards to ensure the user experience.

[0142] For large files with image features, feature extraction is performed on the images based on computer vision technology to obtain their edge and texture features. Then, the main object or significant region in the image is identified, the importance of the content is evaluated, and high priorities are set for the slices with important content. When the network environment changes, the high-priority slices are uploaded first to ensure the user experience.

[0143] For large files with text features, natural language processing (NLP) technology is used to extract features from the text to obtain keywords, titles in the document, or chart features. According to the changes and importance of the content, the priorities of the slices are dynamically adjusted to ensure that important content is uploaded first.

[0144] Through step S603, in this embodiment, the slice priorities are successfully set based on the file structure, laying a foundation for subsequent concurrent uploads in the order of priorities.

[0145] In an embodiment of the front-end large file upload method of the present application, refer to Figure 7 , and it may specifically include the following content: Step S701: If it is a video feature, identify the key frames of the scene change of the video feature slice according to the optical flow method, and determine the slices with key frames. Step S702: Divide the slices with key frames into high-priority slices and determine the corresponding priority order.

[0146] Optionally, in this embodiment, for large files with video features, a priority control technology is to set the slice priorities based on the content. By analyzing the video content, the key frames of the scene change are identified. The optical flow method of computer vision technology is used to detect the scene change to obtain the key frames. The video slices with key frames are divided into high-priority slices, and the remaining slices are ordinary slices. When the network environment changes, the high-priority slices are uploaded first to ensure the user experience.

[0147] Through step S702, in this embodiment, the slice priorities are successfully set based on the content, laying a foundation for subsequent concurrent uploads in the order of priorities.

[0148] In order to perform file slicing and concurrent upload based on dynamically adjusting the slice size and the number of concurrencies, and improve the efficiency and flexibility of large file uploads, the present application provides an embodiment of a front-end large file upload device for implementing all or part of the content of the front-end large file upload method. Refer to Figure 8 , and the front-end large file upload device specifically includes the following content: The large file slicing module 10 is used for the front end to receive a large file uploaded by a user, dynamically slice the large file according to the network data during the upload of the large file and a set dynamic slicing model, determine the corresponding slices, perform content detection on the slices, judge the content characteristics included in each slice, perform priority sorting on the slices according to the content characteristics, and determine the corresponding priority order; The concurrent upload module 20 is used to obtain the network bandwidth and load of the server in real time, determine the corresponding number of concurrent upload slices according to the network bandwidth, the load and a preset concurrent volume rule, and perform a concurrent upload operation on the slices according to the preset multiplexing technology in accordance with the priority order and the number of concurrent upload slices; The upload verification module 30 is used to judge whether the concurrent upload of the slices is successful. If it fails, the slices that fail to be uploaded are retransmitted according to the breakpoint resumption technology. If it is successful, a back-end merging operation is triggered to splice the slices into a complete file, and the complete file is verified according to a preset hash algorithm to obtain the complete large file after upload.

[0149] As can be seen from the above description, the front-end large file upload device provided by the embodiment of the present application can receive a large file uploaded by a user through the front end, dynamically slice the large file according to a set dynamic slicing model, determine the corresponding slices, perform priority determination on the slices, determine the corresponding priority order, determine the corresponding number of concurrent upload slices according to the network bandwidth, load and preset concurrent volume rule, perform a concurrent upload operation on the slices according to the preset multiplexing technology in accordance with the priority order and the number of concurrent upload slices, judge whether the concurrent upload of the slices is successful. If it fails, the slices that fail to be uploaded are retransmitted according to the breakpoint resumption technology. If it is successful, a back-end merging operation is triggered to obtain the complete large file after upload. Thus, file slicing and concurrent upload can be performed based on dynamic adjustment of the slice size and the concurrent volume, improving the efficiency and flexibility of large file upload.

[0150] From the hardware level, in order to perform file slicing and concurrent upload based on dynamic adjustment of the slice size and the concurrent volume, and improve the efficiency and flexibility of large file upload, the embodiment of the present application provides an electronic device for implementing all or part of the content in the front-end large file upload method. The electronic device specifically includes the following content: A processor, a memory, a communications interface, and a bus; wherein, the processor, the memory, and the communications interface complete communication with each other through the bus; the communications interface is used to implement information transmission between the front-end large file upload method and related devices such as a core business system, a user terminal, and a related database, etc.; this logic controller can be a desktop computer, a tablet computer, a mobile terminal, etc., and this embodiment is not limited thereto. In this embodiment, this logic controller can be implemented with reference to the embodiments of the front-end large file upload method in the embodiments, as well as the embodiments of the front-end large file upload method, and the content is incorporated herein, and the repeated parts will not be elaborated.

[0151] It can be understood that the user terminal may include a smart phone, a tablet electronic device, a network set-top box, a portable computer, a desktop computer, a personal digital assistant (PDA), a vehicle-mounted device, a smart wearable device, etc. Among them, the smart wearable device may include smart glasses, a smart watch, a smart bracelet, etc.

[0152] In practical applications, part of the front-end large file upload method can be executed on the electronic device side as described above, or all operations can be completed in the client device. Specifically, it can be selected according to the processing capacity of the client device and the limitations of the user usage scenario, etc. This application does not make any limitations in this regard. If all operations are completed in the client device, the client device may further include a processor.

[0153] The above-mentioned client device may have a communication module (i.e., a communication unit), and can be communicatively connected to a remote server to achieve data transmission with the server. The server may include a server on the task scheduling center side, and in other implementation scenarios, it may also include a server of an intermediate platform, such as a server of a third-party server platform communicatively linked to the task scheduling center server. The server may include a single computer device, or may include a server cluster composed of multiple servers, or a server structure of a distributed device.

[0154] Figure 9 This is a schematic block diagram of the system composition of the electronic device 9600 according to an embodiment of the present application. As Figure 9 shown, the electronic device 9600 may include a central processing unit 9100 and a memory 9140; the memory 9140 is coupled to the central processing unit 9100. It should be noted that this Figure 9 is exemplary; other types of structures can also be used to supplement or replace this structure to implement telecommunication functions or other functions.

[0155] In one embodiment, the function of the front-end large file upload method can be integrated into the central processing unit 9100. Among them, the central processing unit 9100 can be configured to perform the following controls: Step S101: The front-end receives a large file uploaded by the user, dynamically slices the large file according to the network data during the upload of the large file and the set dynamic slicing model, determines the corresponding slices, performs content detection on the slices, judges the content features included in each slice, and sorts the slices according to the content features to determine the corresponding priority order; Step S102: Obtain the network bandwidth and load of the server in real time, determine the corresponding number of concurrent upload slices according to the network bandwidth, the load, and the preset concurrency rule, and perform concurrent upload operations on the slices according to the preset multiplexing technology in the priority order and the number of concurrent upload slices; Step S103: Judge whether the concurrent upload of the slices is successful. If it fails, retransmit the slices that failed to upload according to the breakpoint resumption technology. If it is successful, trigger the back-end merging operation, splice the slices into a complete file, and perform verification on the complete file according to the preset hash algorithm to obtain the complete large file after upload.

[0156] As can be seen from the above description, the electronic device provided in the embodiment of the present application receives a large file uploaded by the user through the front-end, dynamically slices the large file according to the set dynamic slicing model, determines the corresponding slices, determines the corresponding priority order by performing priority determination on the slices, determines the corresponding number of concurrent upload slices according to the network bandwidth, the load, and the preset concurrency rule, performs concurrent upload operations on the slices according to the preset multiplexing technology in the priority order and the number of concurrent upload slices, judges whether the concurrent upload of the slices is successful. If it fails, retransmit the slices that failed to upload according to the breakpoint resumption technology. If it is successful, trigger the back-end merging operation to obtain the complete large file after upload. Thus, it is possible to perform file slicing and concurrent upload based on dynamically adjusting the slice size and concurrency, improving the efficiency and flexibility of large file upload.

[0157] In another embodiment, the front-end large file upload method can be separately configured from the central processing unit 9100. For example, the front-end large file upload method can be configured as a chip connected to the central processing unit 9100, and the function of the front-end large file upload method can be realized through the control of the central processing unit.

[0158] As Figure 9 shown, the electronic device 9600 may further include: a communication module 9110, an input unit 9120, an audio processor 9130, a display 9160, and a power supply 9170. It should be noted that the electronic device 9600 does not necessarily have to includeFigure 9 all components shown in; in addition, the electronic device 9600 may further include Figure 9 components not shown in, reference may be made to the prior art.

[0159] Such as Figure 9 As shown, the central processing unit 9100, sometimes also referred to as a controller or operation control, may include a microprocessor or other processor device and / or logic device. The central processing unit 9100 receives inputs and controls the operation of the various components of the electronic device 9600.

[0160] Among them, the memory 9140, for example, may be one or more of a buffer, a flash memory, a hard drive, a removable medium, a volatile memory, a non-volatile memory, or other suitable devices. The above information related to failures can be stored, and in addition, programs for executing relevant information can also be stored. And the central processing unit 9100 can execute the programs stored in the memory 9140 to implement information storage or processing, etc.

[0161] The input unit 9120 provides inputs to the central processing unit 9100. The input unit 9120 is, for example, a key or a touch input device. The power supply 9170 is used to supply power to the electronic device 9600. The display 9160 is used to display display objects such as images and texts. The display may be, for example, an LCD display, but is not limited thereto.

[0162] The memory 9140 may be a solid-state memory, for example, a read-only memory (ROM), a random access memory (RAM), a SIM card, etc. It may also be such a memory that stores information even when powered off, can be selectively erased and has more data. Examples of such a memory are sometimes referred to as EPROM, etc. The memory 9140 may also be some other type of device. The memory 9140 includes a buffer memory 9141 (sometimes referred to as a buffer). The memory 9140 may include an application / function storage section 9142, which is used to store application programs and function programs or the processes for operating the electronic device 9600 through the central processing unit 9100.

[0163] The memory 9140 may also include a data storage section 9143, which is used to store data, such as contacts, digital data, pictures, sounds, and / or any other data used by the electronic device. The driver storage section 9144 of the memory 9140 may include various drivers of the electronic device for communication functions and / or for performing other functions of the electronic device (such as a messaging application, an address book application, etc.).

[0164] The communication module 9110 is a transmitter / receiver that sends and receives signals via the antenna 9111. The communication module 9110 is coupled to the central processor 9100 to provide input signals and receive output signals, which can be the same as in the case of a conventional mobile communication terminal.

[0165] Based on different communication technologies, multiple communication modules 9110 can be provided in the same electronic device, such as a cellular network module, a Bluetooth module, and / or a wireless local area network module, etc. The communication module 9110 is also coupled to the speaker 9131 and the microphone 9132 via the audio processor 9130 to provide an audio output via the speaker 9131 and receive an audio input from the microphone 9132, thereby implementing the usual telecommunication functions. The audio processor 9130 can include any suitable buffers, decoders, amplifiers, etc. In addition, the audio processor 9130 is also coupled to the central processor 9100, so that recording can be performed on the local machine through the microphone 9132, and the sound stored on the local machine can be played through the speaker 9131.

[0166] Embodiments of the present application also provide a computer-readable storage medium capable of implementing all steps in the front-end large file upload method where the execution subject in the above embodiments is a server or a client. A computer program is stored on the computer-readable storage medium, and when the computer program is executed by a processor, all steps in the front-end large file upload method where the execution subject in the above embodiments is a server or a client are implemented. For example, when the processor executes the computer program, the following steps are implemented: Step S101: The front end receives a large file uploaded by a user, dynamically slices the large file according to the network data at the time of uploading the large file and a set dynamic slicing model, determines the corresponding slices, performs content detection on the slices, judges the content characteristics included in each slice, sorts the slices according to the content characteristics to determine the corresponding priority order; Step S102: Real-time obtain the network bandwidth and load of the server, determine the corresponding number of concurrently uploaded slices according to the network bandwidth, the load, and a preset concurrency rule, and perform a concurrent upload operation on the slices according to the priority order and the number of concurrently uploaded slices by using a preset multiplexing technology; Step S103: Judge whether the concurrent upload of the slices is successful. If it fails, retransmit the slices that have failed to be uploaded according to the breakpoint resumption technology. If it is successful, trigger a back-end merging operation, splice the slices into a complete file, and verify the complete file according to a preset hash algorithm to obtain the complete large file after uploading.

[0167] As can be seen from the above description, the computer-readable storage medium provided by the embodiments of the present application receives a large file uploaded by a user at the front end, dynamically slices the large file according to a set dynamic slicing model to determine the corresponding slices, determines the corresponding priority order by judging the priority of each slice, determines the corresponding number of concurrently uploaded slices according to the network bandwidth, load, and preset concurrency rules, performs a concurrent upload operation on each slice according to the preset multiplexing technology in the priority order and the number of concurrently uploaded slices, determines whether the concurrent upload of each slice is successful, if it fails, retransmits the slice that fails to upload according to the breakpoint resumption technology, and if it is successful, triggers a backend merging operation to obtain the complete large file after upload. Thus, file slicing and concurrent upload can be performed based on dynamically adjusting the slice size and concurrency, improving the efficiency and flexibility of large file uploads.

[0168] An embodiment of the present application also provides a computer program product that can implement all steps of the front-end large file upload method in which the execution subject in the above embodiment is a server or a client. When the computer program / instructions are executed by a processor, the steps of the front-end large file upload method are implemented. For example, the computer program / instructions implement the following steps: Step S101: The front end receives a large file uploaded by a user, dynamically slices the large file according to the network data when the large file is uploaded and a set dynamic slicing model to determine the corresponding slices, performs content detection on each slice, determines the content characteristics included in each slice, and sorts the priority of each slice according to the content characteristics to determine the corresponding priority order; Step S102: Real-time obtains the network bandwidth and load of the server, determines the corresponding number of concurrently uploaded slices according to the network bandwidth, the load, and preset concurrency rules, and performs a concurrent upload operation on each slice according to the preset multiplexing technology in the priority order and the number of concurrently uploaded slices; Step S103: Determine whether the concurrent upload of each slice is successful. If it fails, retransmit the slice that fails to upload according to the breakpoint resumption technology. If it is successful, trigger a backend merging operation, splice each slice into a complete file, and verify the complete file according to a preset hash algorithm to obtain the complete large file after upload.

[0169] As can be seen from the above description, the computer program product provided by the embodiments of the present application receives a large file uploaded by a user through the front end, dynamically slices the large file according to a set dynamic slicing model to determine the corresponding slices, determines the corresponding priority order by performing a priority determination on each slice, determines the corresponding number of concurrently uploaded slices according to the network bandwidth, load, and preset concurrency rules, performs a concurrent upload operation on each slice according to the preset multiplexing technology in the priority order and the number of concurrently uploaded slices, determines whether the concurrent upload of each slice is successful. If it fails, the failed slice is retransmitted according to the breakpoint resumption technology. If it is successful, a backend merging operation is triggered to obtain the complete large file after upload. Thus, it is possible to perform file slicing and concurrent upload based on dynamically adjusting the slice size and concurrency, improving the efficiency and flexibility of large file uploads.

[0170] Those skilled in the art should understand that the embodiments of the present invention can be provided as a method, apparatus, or computer program product. Therefore, the present invention can take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware aspects. Moreover, the present invention can take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) that contain computer-usable program code.

[0171] The present invention is described with reference to the flowcharts and / or block diagrams of methods, devices (apparatuses), and computer program products according to the embodiments of the present invention. It should be understood that each flow and / or block in the flowchart and / or block diagram, and the combination of flows and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to the processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing devices to generate a machine, such that the instructions executed by the processor of the computer or other programmable data processing devices generate a device for implementing the functions specified in Figure 1 one or more of the flows Figure 1 or multiple flows and / or blocks

[0172] These computer program instructions can also be stored in a computer-readable memory that can direct a computer or other programmable data processing device to work in a specific manner, such that the instructions stored in the computer-readable memory generate a manufactured article including an instruction device that implements the functions specified in Figure 1 one or more of the flows Figure 1 or multiple flows and / or blocks

[0173] These computer program instructions may also be loaded onto a computer or other programmable data processing apparatus, so that a series of operation steps are performed on the computer or other programmable apparatus to produce a computer-implemented process, thereby providing instructions for implementing the steps of the function specified in one process or a plurality of processes and / or blocks Figure 1 one process or a plurality of processes and / or blocks Figure 1 steps of the function specified in one block or a plurality of blocks.

[0174] Specific embodiments are used in the present invention to elaborate on the principles and implementation manners of the present invention. The description of the above embodiments is only used to help understand the method and its core idea of the present invention; at the same time, for those of ordinary skill in the art, according to the idea of the present invention, there will be changes in the specific implementation manners and application scopes. In summary, the content of this specification should not be construed as a limitation on the present invention.

Claims

1. A method for uploading large front-end files, characterized in that, The method includes: The front end receives a large file uploaded by a user, dynamically slices the large file according to the network data during the upload of the large file and a set dynamic slicing model, determines the corresponding slices, performs content detection on the slices, judges the content features included in each slice, sorts the slices according to the content features, and determines the corresponding priority order; Obtain the network bandwidth and load of the server in real time, determine the corresponding number of concurrently uploaded slices according to the network bandwidth, the load, and a preset concurrency rule, and perform a concurrent upload operation on the slices according to a preset multiplexing technology in the priority order and the number of concurrently uploaded slices; Judge whether the concurrent upload of the slices is successful. If it fails, retransmit the slices that failed to be uploaded according to the breakpoint resumption technology. If it is successful, trigger a back-end merging operation, splice the slices into a complete file, and verify the complete file according to a preset hash algorithm to obtain the complete large file after upload.

2. The front-end large file upload method according to claim 1, characterized in that Before dynamically slicing the large file according to the network data during the upload of the large file and a set dynamic slicing model and determining the corresponding slices, it includes: Determine a corresponding initial dual-network model according to a preset main network and a preset target network; Train the initial dual-network model according to a set experience buffer to determine a corresponding dynamic slicing model, where the experience buffer is used to store historical experiences.

3. The front-end large file upload method according to claim 2, wherein Before training the initial dual-network model according to a set experience buffer to determine a corresponding dynamic slicing model, it includes: Collect historical network data, extract the network bandwidth change trend in the historical network data according to the sliding window technology, construct a corresponding state space according to the network bandwidth change trend, a preset network bandwidth, and a preset slice size, define an action space, and adjust the slice change amplitude of the action space according to a preset slice dynamic adjustment technology to determine a corresponding continuous action space; The preset main network selects an action for the current state in the state space according to the greedy algorithm, and selects an action corresponding to the current state in the continuous action space, where the action is the action with the smallest Q value; Determine the corresponding reward and new state according to the execution result of the action, and construct an experience buffer according to the current state, the action, the reward, and the new state.

4. The front-end large file upload method according to claim 2, wherein Training the initial dual-network model according to a set experience buffer to determine a corresponding dynamic slicing model includes: Select an action for the new state in the experience buffer according to the main network in the initial dual-network model to determine the corresponding optimal action; Evaluate the optimal action according to the target network in the initial dual-network model to determine the optimal target Q value, and update the main network parameters according to the optimal target Q value to obtain a corresponding dynamic slicing model.

5. The front-end large file upload method according to claim 1, characterized in that Dynamically slicing the large file according to the network data during the upload of the large file and a set dynamic slicing model and determining the corresponding slices includes: Dynamically slice the large file according to the network data during the upload of the large file and the set dynamic slicing model, and determine the corresponding slice size; Perform a slicing operation on the large file according to the preset file.slice method and the slice size, and determine the corresponding slices.

6. The front-end large file upload method according to claim 1, wherein The prioritizing the slices according to the content features to determine the corresponding priority order includes: If it is a video feature, identify key frames for the video feature slices according to the preset frame difference method, and determine the corresponding priority order; If it is a text feature, extract keywords from the text feature slices according to the preset natural language processing technology, and determine the corresponding priority order; If it is an image feature, identify the main body of the image feature slices according to the preset saliency detection algorithm, and determine the corresponding priority order.

7. The front-end large file upload method according to claim 1, characterized in that, The prioritizing the slices according to the content features to determine the corresponding priority order further includes: If it is a video feature, identify the scene change key frames of the video feature slices according to the optical flow method, and determine the slices with key frames; Divide the slices with key frames into high-priority slices, and determine the corresponding priority order.

8. A front-end large file upload device, characterized in that, The device includes: A large file slicing module, configured to receive a large file uploaded by a user at the front end, dynamically slice the large file according to the network data during the upload of the large file and the set dynamic slicing model, determine the corresponding slices, perform content detection on the slices, determine the content features included in each slice, prioritize the slices according to the content features, and determine the corresponding priority order; A concurrent upload module, configured to obtain the network bandwidth and load of the server in real time, determine the corresponding number of concurrent upload slices according to the network bandwidth, the load, and the preset concurrent volume rule, and perform a concurrent upload operation on the slices according to the preset multiplexing technology in accordance with the priority order and the number of concurrent upload slices; An upload verification module, configured to determine whether the concurrent upload of the slices is successful. If it fails, retransmit the failed slices according to the breakpoint resumption technology. If it is successful, trigger a backend merging operation, splice the slices into a complete file, and verify the complete file according to the preset hash algorithm to obtain the complete large file after upload.

9. An electronic device, comprising a memory, a processor, and a computer program stored on the memory and executable on the processor, characterized in that, When the processor executes the program, it implements the steps of the front-end large file upload method according to any one of claims 1 to 7.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the steps of the front-end large file upload method according to any one of claims 1 to 7.

Images