Front-end large file uploading method and device

Abstract

An embodiment of the present application provides a front-end large file uploading method and device, the method comprising: receiving a large file uploaded by a user through a front-end, dynamically sharding the large file according to a set dynamic sharding model, determining corresponding shards, performing priority determination on each shard, determining a corresponding priority order, determining a corresponding number of concurrently uploaded shards according to network bandwidth, load, and preset concurrency rules, concurrently uploading each shard according to the priority order and the number of concurrently uploaded shards according to a preset multiplexing technology, judging whether the concurrent uploading of each shard is successful, and if failed, retransmitting the failed upload shards according to the breakpoint resume technology, and if successful, triggering a back-end merge operation to obtain a complete large file after upload. The present application can perform file sharding and concurrent uploading based on dynamically adjusting the slice size and concurrency, thereby improving the efficiency and flexibility of large file uploading.

Description

Technical Field

[0001] The present application relates to the field of data processing, and specifically to a method and device for uploading large files to a front end. Background Art

[0002] In existing large file uploading technologies, files are usually divided into fixed slice sizes and these slices are uploaded to the server one by one, which often limits the upload speed and network resource utilization.

[0003] To address this situation, there is a way to improve upload speed and network resource utilization through concurrent uploading, but it uses a fixed number of concurrent upload slices. This method may perform well in a static or relatively stable network environment, but it has obvious shortcomings in a dynamically changing network environment.

[0004] Network environments are highly dynamic, with bandwidth, latency, and stability varying over time and location. When uploading large files, if the network environment suddenly deteriorates (e.g., due to network congestion or bandwidth drops), a fixed number of concurrent uploads may cause some upload requests to fail or time out, increasing the number of retries and upload time, reducing upload efficiency.

[0005] Therefore, there is an urgent need for a front-end large file uploading method to improve the efficiency and flexibility of front-end large file uploading in a dynamic network environment. Summary of the Invention

[0006] In response to the problems in the prior art, the present application provides a front-end large file uploading method and device, which can perform file segmentation and concurrent upload based on dynamically adjusting the slice size and concurrency, thereby improving the efficiency and flexibility of large file uploading.

[0007] In order to solve at least one of the above problems, the present application provides the following technical solutions:

[0008] In a first aspect, the present application provides a front-end large file upload method, comprising:

[0009] The front end receives a large file uploaded by a user, dynamically segments the large file according to the network data at the time of uploading and the set dynamic segmentation model, determines the corresponding segments, performs content detection on the segments, determines the content features contained in each segment, prioritizes the segments according to the content features, and determines the corresponding priority order;

[0010] Acquire the network bandwidth and load of the server in real time, determine the corresponding number of concurrent upload fragments according to the network bandwidth, the load and the preset concurrency rule, and perform concurrent upload operations on the fragments according to the priority order and the number of concurrent upload fragments according to the preset multiplexing technology;

[0011] Determine whether the concurrent upload of each fragment is successful. If failed, retransmit the fragments that failed to be uploaded according to the breakpoint resume technology. If successful, trigger the back-end merging operation, splice the fragments into a complete file, and verify the complete file according to the preset hash algorithm to obtain the complete large file after uploading.

[0012] Furthermore, before dynamically slicing the large file according to the network data when the large file is uploaded and setting a dynamic slicing model to determine the corresponding slicing steps, the method includes:

[0013] Determine the corresponding initial dual network model according to the preset main network and the preset target network;

[0014] The initial dual-network model is trained according to a set experience buffer to determine a corresponding dynamic sharding model, wherein the experience buffer is used to store historical experience.

[0015] Furthermore, before performing model training on the initial dual-network model according to the set experience buffer and determining the corresponding dynamic sharding model, the method includes:

[0016] Collect historical network data, extract network bandwidth change trends from the historical network data using a sliding window technique, construct a corresponding state space based on the network bandwidth change trends, a preset network bandwidth, and a preset slice size, define an action space, adjust the slice change amplitude of the action space using a preset slice dynamic adjustment technique, and determine a corresponding continuous action space;

[0017] The preset main network performs action selection for the current state in the state space according to a greedy algorithm, and selects an action corresponding to the current state in the continuous action space, wherein the action is the action with the smallest Q value;

[0018] A corresponding reward and a new state are determined according to the execution result of the action, and an experience buffer is constructed according to the current state, the action, the reward, and the new state.

[0019] Furthermore, the performing model training on the initial dual network model according to the set experience buffer to determine the corresponding dynamic sharding model includes:

[0020] Performing action selection on the new state in the experience buffer according to the main network in the initial dual-network model to determine the corresponding optimal action;

[0021] The optimal action is evaluated according to the target network in the initial dual-network model to determine the optimal target Q value, and the main network parameters are updated according to the optimal target Q value to obtain the corresponding dynamic slicing model.

[0022] Furthermore, dynamically slicing the large file according to the network data when the large file is uploaded and setting a dynamic slicing model to determine the corresponding slicing includes:

[0023] Dynamically fragmenting the large file according to the network data when the large file is uploaded and the set dynamic fragmentation model, and determining the corresponding fragment sizes;

[0024] The large file is sliced ​​according to the preset file.slice method and the slice size to determine the corresponding slices.

[0025] Furthermore, the prioritizing the fragments according to the content features to determine the corresponding priority order includes:

[0026] If it is a video feature, key frame identification is performed on the video feature segment according to the preset frame difference method to determine the corresponding priority order; if it is a text feature, keyword extraction is performed on the text feature segment according to the preset natural language processing technology to determine the corresponding priority order; if it is an image feature, subject identification is performed on the image feature segment according to the preset saliency detection algorithm to determine the corresponding priority order.

[0027] Furthermore, the prioritizing the fragments according to the content features and determining the corresponding priority order further includes:

[0028] If it is a video feature, the scene change key frame of the video feature segment is identified according to the optical flow method, and the segment with the key frame is determined;

[0029] The slices with key frames are divided into high-priority slices, and the corresponding priority order is determined.

[0030] In a second aspect, the present application provides a front-end large file uploading device, comprising:

[0031] The large file segmentation module is used to receive large files uploaded by users at the front end, dynamically segment the large files according to the network data at the time of uploading the large files and the set dynamic segmentation model, determine the corresponding segments, perform content detection on the segments, determine the content characteristics of each segment, prioritize the segments according to the content characteristics, and determine the corresponding priority order;

[0032] A concurrent upload module is configured to obtain the network bandwidth and load of the server in real time, determine the corresponding number of concurrent upload fragments according to the network bandwidth, the load, and a preset concurrency rule, and concurrently upload the fragments according to the priority order and the number of concurrent upload fragments according to a preset multiplexing technology;

[0033] The upload verification module is used to determine whether the concurrent upload of each fragment is successful. If it fails, the fragments that failed to be uploaded are retransmitted according to the breakpoint resume technology. If it succeeds, the back-end merge operation is triggered to splice the fragments into a complete file. The complete file is verified according to the preset hash algorithm to obtain the complete large file after upload.

[0034] In a third aspect, the present application provides an electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps of the front-end large file uploading method when executing the program.

[0035] In a fourth aspect, the present application provides a computer-readable storage medium on which a computer program is stored, and when the computer program is executed by a processor, the steps of the front-end large file uploading method are implemented.

[0036] In a fifth aspect, the present application provides a computer program product, including a computer program / instruction, which, when executed by a processor, implements the steps of the front-end large file uploading method.

[0037] It can be seen from the above technical solution that the present application provides a front-end large file uploading method and device, which receives large files uploaded by users through the front-end, dynamically segments the large files according to the set dynamic segmentation model, determines the corresponding segments, determines the priority of each segment, determines the corresponding priority order, determines the corresponding number of concurrently uploaded segments according to the network bandwidth, load and preset concurrency rules, and performs concurrent upload operations on each segment according to the priority order and the number of concurrently uploaded segments according to the preset multiplexing technology, judges whether the concurrent upload of each segment is successful, and if failed, retransmits the failed upload segment according to the breakpoint resume technology, and if successful, triggers the back-end merge operation to obtain the complete large file after upload, thereby enabling file segmentation and concurrent upload based on dynamically adjusted segment size and concurrency, thereby improving the efficiency and flexibility of large file uploading. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0039] Figure 1 This is one of the flow charts of the front-end large file uploading method in the embodiment of this application;

[0040] Figure 2 This is the second flow chart of the front-end large file uploading method in the embodiment of this application;

[0041] Figure 3 This is the third flow chart of the front-end large file uploading method in the embodiment of this application;

[0042] Figure 4 This is the fourth flow chart of the front-end large file uploading method in the embodiment of the present application;

[0043] Figure 5 This is the fifth flow chart of the front-end large file uploading method in the embodiment of the present application;

[0044] Figure 6 This is the sixth flow chart of the front-end large file uploading method in the embodiment of the present application;

[0045] Figure 7 This is the seventh flow chart of the front-end large file uploading method in the embodiment of the present application;

[0046] Figure 8 This is a structural diagram of the front-end large file uploading device in an embodiment of the present application;

[0047] Figure 9 Schematic diagram of the structure of the electronic device in the embodiment of the present application.

[0048] Reference numerals:

[0049] Electronic device 9600, central processing unit 9100, memory 9140, communication module 9110, input unit 9120, audio processor 9130, display 9160, power supply 9170, buffer memory 9141, application / function storage unit 9142, data storage unit 9143, driver program storage unit 9144, antenna 9111, speaker 9131, microphone 9132. DETAILED DESCRIPTION

[0050] To make the purpose, technical solutions, and advantages of the embodiments of this application more clear, the technical solutions in the embodiments of this application will be clearly and completely described below in conjunction with the drawings in the embodiments of this application. Obviously, the described embodiments are part of the embodiments of this application, not all of the embodiments. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of this application.

[0051] The acquisition, storage, use, and processing of data in this application's technical solution comply with relevant national laws and regulations.

[0052] Considering that in the existing large file uploading technology, a fixed slice size is usually used to split the file, and these slices are uploaded to the server one by one, the upload speed and network resource utilization are often limited. The present application provides a front-end large file uploading method and device, which receives a large file uploaded by a user through the front end, dynamically slices the large file according to a set dynamic slicing model, determines the corresponding slices, determines the priority of each slice, determines the corresponding priority order, determines the corresponding number of concurrently uploaded slices according to the network bandwidth, load and preset concurrency rules, and concurrently uploads each slice according to the priority order and the number of concurrently uploaded slices according to the preset multiplexing technology, judges whether the concurrent upload of each slice is successful, and if it fails, retransmits the slice that failed to be uploaded according to the breakpoint resume technology. If it succeeds, triggers the back-end merge operation to obtain the complete large file after upload, thereby being able to perform file slicing and concurrent upload based on dynamically adjusted slice size and concurrency, thereby improving the efficiency and flexibility of large file uploading.

[0053] In order to perform file segmentation and concurrent upload based on dynamic adjustment of slice size and concurrency, and improve the efficiency and flexibility of large file upload, this application provides an embodiment of a front-end large file upload method, see Figure 1 The front-end large file upload method specifically includes the following contents:

[0054] Step S101: The front end receives a large file uploaded by a user, dynamically segments the large file according to the network data at the time of uploading the large file and a set dynamic segmentation model, determines the corresponding segments, performs content detection on the segments, determines the content features contained in each segment, prioritizes the segments according to the content features, and determines the corresponding priority order;

[0055] Optionally, in this embodiment, this step is a process of dynamically fragmenting a large file uploaded by a user and determining a priority order of each fragment.

[0056] Specifically, in this embodiment, a dynamic sharding model is set to dynamically segment large files uploaded by users.

[0057] Optionally, in this embodiment, a dynamic sharding model agent is constructed.

[0058] To dynamically adjust the slice size based on the network environment and enhance the user experience, this step models the slice resizing problem as a Markov decision process (MDP). This involves constructing an agent with a state space and an action space. This agent is then trained using the model to automatically adjust the slice size in different network environments, maximizing upload efficiency.

[0059] Specifically, the state space describes the current state of the system, including:

[0060] Current Network Bandwidth: The network bandwidth measured in real time (unit: MB / s).

[0061] Current Slice Size: The current slice size in use (unit: MB).

[0062] Upload Progress: The percentage of the file uploaded (0% to 100%).

[0063] Network delay: The current network delay time (unit: ms).

[0064] Packet loss rate: The current network packet loss rate (unit: %).

[0065] Optimally, the state space innovatively incorporates historical bandwidth change trends, using sliding window technology to calculate bandwidth change trends over a period of time (e.g., bandwidth changes over the past 5 seconds). This sliding window mechanism calculates historical bandwidth change trends, ensuring that the state space reflects the latest network environment. Using historical bandwidth change trends as a dimension of the state space enables the model to capture dynamic changes in network bandwidth, enabling more accurate slice adjustment decisions.

[0066] Specifically, the action space describes the slice adjustment strategies that the system can adopt, including:

[0067] Increase Slice Size: Increase the slice size by a certain percentage (e.g. 20%).

[0068] Reduce Slice Size: Reduce the slice size by a certain percentage (such as 20%).

[0069] Keep Slice Size: Do not change the current slice size.

[0070] Preferably, a continuous action space is introduced in addition to the fixed action space, allowing the model to output slice adjustment ratios of any size, rather than a few fixed options. Furthermore, a dynamic adjustment ratio is introduced in conjunction with the continuous action space, allowing the model to dynamically adjust the slice size change based on the current network environment. For example, when the network bandwidth is high, the slice size can be increased by a larger ratio; when the network bandwidth is low, the slice size can be decreased by a smaller ratio. By introducing dynamic adjustment ratios, the model can dynamically adjust the slice size change based on the current network environment, enhancing the model's flexibility.

[0071] Specifically, the reward function is used to evaluate the effect of each action. Designing a reasonable reward function can guide the model to learn the optimal strategy.

[0072] In this embodiment, the reward function introduces a multi-objective reward function, which enables the model to simultaneously optimize multiple objectives such as upload speed, success rate, and energy consumption. This includes:

[0073] Upload speed reward: The faster the upload speed, the higher the reward.

[0074] Success rate reward: The higher the slice upload success rate, the higher the reward.

[0075] Energy consumption penalty: If slice adjustment leads to increased device energy consumption, a penalty will be imposed.

[0076] Stability Bonus: A bonus is given if the upload speed remains stable after segment resizing.

[0077] Optionally, the reward function uses adaptive reward weighting, dynamically adjusting the reward function's weights based on the current network environment and upload progress. For example, in the early stages of an upload, you can prioritize upload speed; in the later stages, you can prioritize success rate. By introducing a multi-objective reward function, the model can simultaneously optimize multiple objectives, including upload speed, success rate, and energy consumption, improving the overall efficiency of the upload process.

[0078] After defining the state space, action space, and reward function, we can obtain an intelligent agent for model training.

[0079] Optionally, in this embodiment, a dynamic sharding model is defined.

[0080] When defining the DQN model, DQN directly uses the maximum Q value output by the target network as the update benchmark when calculating the target Q value. This design, where the same network is responsible for both "action selection" and "Q value evaluation," can easily lead to an excessive preference for high-reward actions, which in turn leads to a systematic overestimation of Q values. Therefore, in this example, two independent Q networks are introduced to address the problem of Q value estimation bias.

[0081] Specifically, a main network is introduced to select actions, while a target network is used to evaluate the value of actions. By separating action selection and action evaluation, Dual DQN can reduce the bias in Q-value estimation, thereby improving model stability and performance.

[0082] First, initialize the main network and the target network. The two have the same structure but independent parameters. The parameters of the target network are synchronized from the main network at regular intervals (for example, every 100 steps).

[0083] Then, the experience buffer is built based on the main network. At each time step, the main network is used to select an action for the state in the state space. Specifically, the current state is input to the main network, the Q value of each action is output, and the action with the largest Q value is selected.

[0084] For example, assume the current status is: network bandwidth = 2MB / s, slice size = 5MB, upload progress = 30%, network delay = 50ms, packet loss rate = 0.1%, network bandwidth change trend = -0.5%

[0085] The system inputs the current state into the main network, which outputs the Q value of each action: increase the slice size (Q value = 0.8), decrease the slice size (Q value = 0.9), and maintain the slice size (Q value = 0.7).

[0086] The system chooses the action with the largest Q value: reduce the slice size

[0087] The system performs a slice size reduction action, adjusting the slice size from 5MB to 4MB.

[0088] The system observes the following new status: Network bandwidth = 2.5MB / s, Segment size = 4MB, Upload progress = 35%, Network latency = 40ms, Packet loss rate = 0.05%, Network bandwidth change trend = 0.1%

[0089] The system stores the experience (current state, action, reward, new state) into the initial experience replay buffer and repeats the above steps to obtain the constructed experience buffer.

[0090] After the above steps, the initial dual network model is obtained based on the defined dual network and the constructed experience buffer, and then the model training is carried out.

[0091] Optionally, in this embodiment, dynamic sharding model training is performed.

[0092] Specifically, during the training process, the model collects a batch of experiences from the constructed experience buffer. Preferably, a priority experience replay mechanism is introduced to enable the model to learn important experiences (such as experiences with higher rewards) first, thereby accelerating the model's learning process.

[0093] For each collected experience, the primary network is used to select the optimal action in the new state, but the target network is used to evaluate the Q value of that action.

[0094] Calculate the target Q value: target Q value = reward + discount factor × Q value evaluated by the target network.

[0095] The target Q value is used to update the parameters of the main network and minimize the loss of the Q value function to obtain a dynamic sharding model.

[0096] At regular intervals (e.g., every 100 steps), the parameters of the main network are copied to the target network to ensure that the parameters of the target network are synchronized with the main network.

[0097] By introducing Dual DQN, the system can reduce the deviation of Q-value estimation, thereby improving model stability and performance. In scenarios where large files are uploaded, Dual DQN can more accurately and dynamically adjust the slice size, adapting to complex network environments, significantly improving upload efficiency and stability.

[0098] Optionally, in this embodiment, a dynamic sharding model is applied.

[0099] By monitoring network bandwidth, large files uploaded by users are segmented based on a trained dynamic segmentation model and the current network conditions, dynamically calculating the optimal segment size. For good network conditions, the segment size can be increased to reduce the number of requests; for poor network conditions, the segment size can be reduced. Once the segment size is determined, the large file is segmented accordingly, and each segment is uniquely identified.

[0100] Optionally, in this embodiment, content detection and priority order division are performed on each obtained fragment.

[0101] Specifically, depending on the type of the large file, the large file may include a video file, an image file, and a text file.

[0102] For large files with video features, a priority control technology is to identify key frames (I frames) in the video file, classify the video fragments with I frames as high-priority fragments, and the remaining fragments as ordinary fragments. When the network environment changes, high-priority fragments are uploaded first to ensure user experience.

[0103] Another priority control technology is to prioritize slices based on the file structure. This technology identifies keyframe locations by parsing the file format's metadata. Open-source multimedia libraries (such as FFmpeg, GStreamer, and OpenCV) are used to parse video files, obtain keyframe timestamps or frame numbers, and then classify video slices containing keyframes as high-priority slices, while the remaining slices are classified as normal slices. When network conditions change, high-priority slices are uploaded first to ensure a better user experience.

[0104] Another priority control technology is content-based slicing priority setting. This technology analyzes video content and identifies keyframes that change scenes. This technology uses the frame difference method, a computer vision technology, or alternatively, the optical flow method, which detects scene changes and identifies keyframes. Slices containing keyframes are then classified as high-priority slices, while the remaining slices are classified as standard slices. When network conditions change, high-priority slices are uploaded first to ensure a better user experience.

[0105] Another priority control technology is to identify key slices based on the hash value of the slice, use MD5 to calculate the hash for each slice, identify slices with large content changes, obtain key frames, and divide the video slices with key frames into high-priority slices, and the remaining slices into ordinary slices. When the network environment changes, high-priority slices are uploaded first to ensure user experience.

[0106] For large files with image features, we use computer vision technology to extract image features, obtain edge and texture characteristics, identify the main objects or significant areas in the image, assess the importance of the content, and assign high priority to the fragments containing important content. When the network environment changes, high-priority fragments are uploaded first to ensure a better user experience.

[0107] For large files with text features, natural language processing (NLP) technology extracts features from the text to obtain keywords, titles or chart features in the document, and dynamically adjusts the priority of slices based on changes and importance of the content to ensure that important content is uploaded first.

[0108] The above are the rules for sharding priority when a large file has only one type.

[0109] For example, if a user uploads a mixed large file containing videos, documents, and images:

[0110] Video section:

[0111] Use frame difference method to detect scene changes and identify key frames.

[0112] Mark slices containing keyframes as high priority.

[0113] Documentation section:

[0114] Use NLP technology to extract keywords and titles.

[0115] Mark pages containing keywords or titles as high priority.

[0116] Image part:

[0117] Use object detection technology to identify the main object.

[0118] Mark the area containing the subject object as high priority.

[0119] During the upload process, high-priority slices, pages, and regions are uploaded first to ensure that users can quickly preview important content.

[0120] It is understandable that after the above steps, we have achieved the dynamic determination of the fragment size according to the network environment to fragment large files, and prioritized the fragments to ensure that the fragments with high priority are uploaded first during each batch of fragment uploads, ensuring the efficiency and flexibility of upload.

[0121] Step S102: obtaining the network bandwidth and load of the server in real time, determining the corresponding number of concurrent upload segments according to the network bandwidth, the load, and a preset concurrency rule, and concurrently uploading the segments according to the priority order and the number of concurrent upload segments according to a preset multiplexing technology;

[0122] Optionally, in this embodiment, this step is a process of dynamically adjusting the concurrency data by detecting the network status, and performing concurrent uploading according to the concurrency and priority order.

[0123] Specifically, the Network Information API in the front-end JavaScript is used to measure the current network upload bandwidth in real time. The server returns the current load status to the front-end through the API interface, and the front-end adjusts the number of concurrent uploads based on the load.

[0124] The concurrency rule is: if the network bandwidth is high and the server load is low, increase the number of concurrent uploads. If the network bandwidth is low or the server load is high, reduce the number of concurrent uploads.

[0125] The server load factor is a value between 0 and 1, indicating the load of the server (0 means no load, 1 means full load).

[0126] The front-end periodically (e.g., every 10 seconds) obtains the current network bandwidth and server load, and calculates the current number of concurrent uploads based on the preset concurrency rules. If the calculated number of concurrent uploads is inconsistent with the currently used number of concurrent uploads, the number of shards for concurrent uploads is dynamically adjusted.

[0127] Based on the concurrency, HTTP / 2 multiplexing technology is used to perform concurrent uploads based on the shard priority and concurrency. HTTP / 2 allows multiple request and response streams to be transmitted simultaneously over a single TCP connection. Each stream can transmit data independently without blocking other streams.

[0128] The complete multi-part upload process is as follows:

[0129] Accept large files uploaded by users and calculate the shard size based on network status and dynamic sharding model;

[0130] Split large files into multiple fragments according to the fragment size (for example, each fragment is 1MB in size).

[0131] Generate a unique identifier (such as a hash value) for each shard and set the upload priority based on the priority control technology.

[0132] Based on the dynamically adjusted number of concurrent uploads, the shards are uploaded to the server concurrently according to the upload priority using HTTP / 2 multiplexing technology.

[0133] After the server receives the shards, it stores them in a temporary location and records the order and identifier of the shards.

[0134] Step S103: Determine whether the concurrent upload of each fragment is successful. If it fails, retransmit the fragment that failed to be uploaded according to the breakpoint resume technology. If it succeeds, trigger the back-end merging operation, splice the fragments into a complete file, and verify the complete file according to the preset hash algorithm to obtain the complete large file after uploading.

[0135] Optionally, in this embodiment, this step is a process of verifying the uploaded segments.

[0136] Specifically, the breakpoint resume technology means that during the upload process, if a slice fails to upload, the system can automatically retry and support continuing the upload from the breakpoint instead of re-uploading the entire file.

[0137] Generate a unique identifier (such as a hash value) for each segment and record the failed segment when the upload fails. When the upload is resumed, only upload the unsuccessful segments instead of re-uploading all segments.

[0138] This example demonstrates how this embodiment dynamically adjusts the size of large file fragments based on the reinforcement learning model to obtain large file fragments and divide the fragment priorities, and then uploads the fragments through concurrent uploading to ensure the efficiency and flexibility of large file uploading.

[0139] From the above description, it can be seen that the front-end large file upload method provided by the embodiment of the present application can receive large files uploaded by users through the front-end, dynamically segment the large files according to the set dynamic segmentation model, determine the corresponding segments, determine the priority of each segment, determine the corresponding priority order, determine the corresponding number of concurrently uploaded segments according to the network bandwidth, load and preset concurrency rules, and perform concurrent upload operations on each segment according to the priority order and the number of concurrently uploaded segments according to the preset multiplexing technology, judge whether the concurrent upload of each segment is successful, and if it fails, retransmit the failed upload segment according to the breakpoint resume technology. If successful, trigger the back-end merge operation to obtain the complete large file after upload, thereby enabling file segmentation and concurrent upload based on dynamically adjusted segment size and concurrency, thereby improving the efficiency and flexibility of large file upload.

[0140] In one embodiment of the front-end large file upload method of this application, see Figure 2 , and can also include the following:

[0141] Step S201: determining a corresponding initial dual network model according to a preset main network and a preset target network;

[0142] Step S202: performing model training on the initial dual-network model according to a set experience buffer to determine a corresponding dynamic sharding model, wherein the experience buffer is used to store historical experience.

[0143] Optionally, in this embodiment, this step is the process of building and training the network model.

[0144] Specifically, when defining the DQN model, DQN directly uses the maximum Q value output by the target network as the update benchmark when calculating the target Q value. This design, where the same network is responsible for both "action selection" and "Q value evaluation," can easily lead to an excessive preference for high-reward actions, which in turn leads to a systematic overestimation of Q values. Therefore, in this embodiment, two independent Q networks are introduced to address the problem of Q value estimation bias.

[0145] Specifically, a main network is introduced to select actions, while a target network is used to evaluate the value of actions. By separating action selection and action evaluation, Dual DQN can reduce the bias in Q-value estimation, thereby improving model stability and performance.

[0146] Specifically, a buffer is set, and an experience buffer is constructed based on the main network. At each time step, the main network is used to select an action for the state in the state space to construct an experience buffer.

[0147] Specifically, during the training process, the model collects a batch of experiences from the constructed experience buffer. Preferably, a priority experience replay mechanism is introduced to enable the model to learn important experiences (such as experiences with higher rewards) first, thereby accelerating the model's learning process.

[0148] For each collected experience, the primary network is used to select the optimal action in the new state, but the target network is used to evaluate the Q value of that action.

[0149] Calculate the target Q value: target Q value = reward + discount factor × Q value evaluated by the target network.

[0150] The target Q value is used to update the parameters of the main network and minimize the loss of the Q value function to obtain a dynamic sharding model.

[0151] Through step S202, this embodiment introduces dual DQN, and the system can reduce the deviation of Q value estimation, thereby improving the stability and performance of the model.

[0152] In one embodiment of the front-end large file upload method of this application, see Figure 3 , and can also include the following:

[0153] Step S301: Collect historical network data, extract network bandwidth change trends from the historical network data using a sliding window technique, construct a corresponding state space based on the network bandwidth change trends, a preset network bandwidth, and a preset slice size, define an action space, adjust the slice change amplitude of the action space using a preset slice dynamic adjustment technique, and determine a corresponding continuous action space;

[0154] Step S302: The preset main network performs action selection for the current state in the state space according to a greedy algorithm, and selects an action corresponding to the current state in the continuous action space, wherein the action is the action with the smallest Q value;

[0155] Step S303: determining a corresponding reward and a new state according to the execution result of the action, and constructing an experience buffer according to the current state, the action, the reward, and the new state.

[0156] Optionally, in this embodiment, this step is a process of constructing an experience buffer zone.

[0157] Specifically, the state space describes the current state of the system, including:

[0158] Current Network Bandwidth: The network bandwidth measured in real time (unit: MB / s).

[0159] Current Slice Size: The current slice size in use (unit: MB).

[0160] Upload Progress: The percentage of the file uploaded (0% to 100%).

[0161] Network delay: The current network delay time (unit: ms).

[0162] Packet loss rate: The current network packet loss rate (unit: %).

[0163] Optimally, the state space innovatively incorporates historical bandwidth change trends, using sliding window technology to calculate bandwidth change trends over a period of time (e.g., bandwidth changes over the past 5 seconds). This sliding window mechanism calculates historical bandwidth change trends, ensuring that the state space reflects the latest network environment. Using historical bandwidth change trends as a dimension of the state space enables the model to capture dynamic changes in network bandwidth, enabling more accurate slice adjustment decisions.

[0164] Specifically, the action space describes the slice adjustment strategies that the system can adopt, including:

[0165] Increase Slice Size: Increase the slice size by a certain percentage (e.g. 20%).

[0166] Reduce Slice Size: Reduce the slice size by a certain percentage (such as 20%).

[0167] Keep Slice Size: Do not change the current slice size.

[0168] Preferably, a continuous action space is introduced in addition to the fixed action space, allowing the model to output slice adjustment ratios of any size, rather than a few fixed options. Furthermore, a dynamic adjustment ratio is introduced in conjunction with the continuous action space, allowing the model to dynamically adjust the slice size change based on the current network environment. For example, when the network bandwidth is high, the slice size can be increased by a larger ratio; when the network bandwidth is low, the slice size can be decreased by a smaller ratio. By introducing dynamic adjustment ratios, the model can dynamically adjust the slice size change based on the current network environment, enhancing the model's flexibility.

[0169] Specifically, the reward function is used to evaluate the effect of each action. Designing a reasonable reward function can guide the model to learn the optimal strategy.

[0170] In this embodiment, the reward function introduces a multi-objective reward function, which enables the model to simultaneously optimize multiple objectives such as upload speed, success rate, and energy consumption. This includes:

[0171] Upload speed reward: The faster the upload speed, the higher the reward.

[0172] Success rate reward: The higher the slice upload success rate, the higher the reward.

[0173] Energy consumption penalty: If slice adjustment leads to increased device energy consumption, a penalty will be imposed.

[0174] Stability Bonus: A bonus is given if the upload speed remains stable after segment resizing.

[0175] Optionally, the reward function uses adaptive reward weighting, dynamically adjusting the reward function's weights based on the current network environment and upload progress. For example, in the early stages of an upload, you can prioritize upload speed; in the later stages, you can prioritize success rate. By introducing a multi-objective reward function, the model can simultaneously optimize multiple objectives, including upload speed, success rate, and energy consumption, improving the overall efficiency of the upload process.

[0176] After defining the state space, action space, and reward function, we can obtain an intelligent agent for model training.

[0177] The main network and the target network are introduced. An experience buffer is built based on the main network. At each time step, the main network is used to select an action based on the state in the state space. Specifically, the current state is input to the main network, the Q value of each action is output, and the action with the largest Q value is selected.

[0178] For example, assume the current status is: network bandwidth = 2MB / s, slice size = 5MB, upload progress = 30%, network delay = 50ms, packet loss rate = 0.1%, network bandwidth change trend = -0.5%

[0179] The system inputs the current state into the main network, which outputs the Q value of each action: increase the slice size (Q value = 0.8), decrease the slice size (Q value = 0.9), and maintain the slice size (Q value = 0.7).

[0180] The system chooses the action with the largest Q value: reduce the slice size

[0181] The system performs a slice size reduction action, adjusting the slice size from 5MB to 4MB.

[0182] The system observes the following new status: Network bandwidth = 2.5MB / s, Segment size = 4MB, Upload progress = 35%, Network latency = 40ms, Packet loss rate = 0.05%, Network bandwidth change trend = 0.1%

[0183] The system stores the experience (current state, action, reward, new state) into the initial experience replay buffer and repeats the above steps to obtain the constructed experience buffer.

[0184] Through step S303, this embodiment successfully constructs an experience buffer, laying a foundation for subsequent model training.

[0185] In one embodiment of the front-end large file upload method of this application, see Figure 4 , and can also include the following:

[0186] Step S401: performing action selection for the new state in the experience buffer according to the main network in the initial dual-network model to determine the corresponding optimal action;

[0187] Step S402: Evaluate the optimal action according to the target network in the initial dual-network model, determine the optimal target Q value, update the main network parameters according to the optimal target Q value, and obtain a corresponding dynamic slicing model.

[0188] Optionally, in this embodiment, this step is the process of model training.

[0189] Specifically, during the training process, the model collects a batch of experiences from the constructed experience buffer. Preferably, a priority experience replay mechanism is introduced to enable the model to learn important experiences (such as experiences with higher rewards) first, thereby accelerating the model's learning process.

[0190] For each collected experience, the primary network is used to select the optimal action in the new state, but the target network is used to evaluate the Q value of that action.

[0191] Calculate the target Q value: target Q value = reward + discount factor × Q value evaluated by the target network.

[0192] The target Q value is used to update the parameters of the main network and minimize the loss of the Q value function to obtain a dynamic sharding model.

[0193] At regular intervals (e.g., every 100 steps), the parameters of the main network are copied to the target network to ensure that the parameters of the target network are synchronized with the main network.

[0194] Through step S402, this embodiment successfully trains a dynamic sharding model, laying the foundation for subsequent model applications.

[0195] In one embodiment of the front-end large file upload method of this application, see Figure 5 , and can also include the following:

[0196] Step S501: dynamically fragmenting the large file according to the network data when the large file is uploaded and the set dynamic fragmentation model, and determining the corresponding fragment sizes;

[0197] Step S502: Slicing the large file according to the preset file.slice method and the slice size to determine corresponding slices.

[0198] Optionally, in this step, the dynamic sharding model obtains the optimal shard size based on network data, and subsequently creates file slices. The file.slice method is used to slice the file according to the "optimal shard size" to generate a list of sliced ​​Blob objects.

[0199] Through step S502, this embodiment successfully determines each fragment, laying a foundation for subsequent determination of the priority order of each fragment and concurrent uploading.

[0200] In one embodiment of the front-end large file upload method of this application, see Figure 6 , and can also include the following:

[0201] Step S601: If it is a video feature, perform key frame recognition on the video feature segment according to a preset frame difference method to determine the corresponding priority order;

[0202] Step S602: If it is a text feature, perform keyword extraction on the text feature segment according to a preset natural language processing technology to determine the corresponding priority order;

[0203] Step S603: If it is an image feature, the image feature segment is identified based on the preset saliency detection algorithm to determine the corresponding priority order.

[0204] Optionally, in this embodiment, for large files with video features, a priority control technology is to identify key frames (I frames) in the video file, divide the video fragments with I frames into high-priority fragments, and the remaining fragments into ordinary fragments. When the network environment changes, high-priority fragments are uploaded first to ensure user experience.

[0205] For large files with image features, we use computer vision technology to extract image features, obtain edge and texture characteristics, identify the main objects or significant areas in the image, assess the importance of the content, and assign high priority to the fragments containing important content. When the network environment changes, high-priority fragments are uploaded first to ensure a better user experience.

[0206] For large files with text features, natural language processing (NLP) technology extracts features from the text to obtain keywords, titles or chart features in the document, and dynamically adjusts the priority of slices based on changes and importance of the content to ensure that important content is uploaded first.

[0207] Through step S603, this embodiment successfully sets slice priorities based on the file structure, laying the foundation for subsequent concurrent uploading in priority order.

[0208] In one embodiment of the front-end large file upload method of this application, see Figure 7 , and can also include the following:

[0209] Step S701: If it is a video feature, identify the scene change key frame of the video feature segment according to the optical flow method, and determine the segment with the key frame;

[0210] Step S702: Divide the slices with key frames into high-priority slices and determine the corresponding priority order.

[0211] Optionally, in this embodiment, for large files with video features, a priority control technology is implemented: content-based slice priority setting. This technology analyzes the video content and identifies key frames where scenes change. Using computer vision technology, optical flow methods are used to detect scene changes and obtain key frames. Video slices with key frames are then classified as high-priority slices, while the remaining slices are classified as normal slices. When the network environment changes, high-priority slices are uploaded first to ensure a better user experience.

[0212] Through step S702, this embodiment successfully sets slice priorities based on content, laying the foundation for subsequent concurrent uploading in priority order.

[0213] In order to perform file segmentation and concurrent upload based on dynamic adjustment of slice size and concurrency, and improve the efficiency and flexibility of large file upload, the present application provides an embodiment of a front-end large file upload device for implementing all or part of the content of the front-end large file upload method, see Figure 8 The front-end large file upload device specifically includes the following contents:

[0214] The large file segmentation module 10 is used to receive large files uploaded by users at the front end, dynamically segment the large files according to the network data at the time of uploading the large files and the set dynamic segmentation model, determine the corresponding segments, perform content detection on the segments, determine the content characteristics of each segment, prioritize the segments according to the content characteristics, and determine the corresponding priority order;

[0215] The concurrent upload module 20 is configured to obtain the network bandwidth and load of the server in real time, determine the corresponding number of concurrent upload segments according to the network bandwidth, the load, and a preset concurrency rule, and concurrently upload the segments according to the priority order and the number of concurrent upload segments according to a preset multiplexing technology;

[0216] The upload verification module 30 is used to determine whether the concurrent upload of each fragment is successful. If it fails, the fragments that failed to be uploaded are retransmitted according to the breakpoint resumption technology. If it succeeds, the back-end merging operation is triggered to splice the fragments into a complete file, and the complete file is verified according to the preset hash algorithm to obtain the complete large file after upload.

[0217] From the above description, it can be seen that the front-end large file upload device provided by the embodiment of the present application can receive large files uploaded by users through the front-end, dynamically segment the large files according to the set dynamic segmentation model, determine the corresponding segments, determine the priority of each segment, determine the corresponding priority order, determine the corresponding number of concurrently uploaded segments according to the network bandwidth, load and preset concurrency rules, and perform concurrent upload operations on each segment according to the priority order and the number of concurrently uploaded segments according to the preset multiplexing technology, judge whether the concurrent upload of each segment is successful, and if it fails, retransmit the failed upload segment according to the breakpoint resumption technology. If successful, trigger the back-end merge operation to obtain the complete large file after upload, thereby enabling file segmentation and concurrent upload based on dynamically adjusted segment size and concurrency, thereby improving the efficiency and flexibility of large file upload.

[0218] From a hardware perspective, in order to perform file segmentation and concurrent upload based on dynamic adjustment of the slice size and concurrency, thereby improving the efficiency and flexibility of large file upload, the present application provides an embodiment of an electronic device for implementing all or part of the content of the front-end large file upload method, and the electronic device specifically includes the following content:

[0219] A processor, memory, a communications interface, and a bus; wherein the processor, memory, and communications interface communicate with each other via the bus; the communications interface is used to implement information transmission between the front-end large file upload method and related devices such as core business systems, user terminals, and related databases; the logic controller can be a desktop computer, a tablet computer, a mobile terminal, etc., but this embodiment is not limited to these. In this embodiment, the logic controller can be implemented with reference to the front-end large file upload method embodiment and the front-end large file upload method embodiment in the embodiment, and their contents are incorporated herein, and repeated parts are not repeated.

[0220] It is understandable that the user terminal may include a smart phone, a tablet electronic device, a network set-top box, a portable computer, a desktop computer, a personal digital assistant (PDA), a vehicle-mounted device, a smart wearable device, etc. Among them, the smart wearable device may include smart glasses, a smart watch, a smart bracelet, etc.

[0221] In practical applications, portions of the front-end large file upload method can be executed on the electronic device side as described above, or all operations can be completed on the client device. The specific selection can be based on the processing capabilities of the client device and the limitations of the user's usage scenario. This application does not impose any restrictions on this. If all operations are completed on the client device, the client device may also include a processor.

[0222] The aforementioned client device may include a communication module (i.e., a communication unit) capable of establishing a communication connection with a remote server to facilitate data transmission with the server. The server may include a server at the task scheduling center or, in other implementation scenarios, a server on an intermediate platform, such as a server on a third-party server platform that is communicatively linked to the task scheduling center server. The server may comprise a single computer device, a server cluster consisting of multiple servers, or a distributed server configuration.

[0223] Figure 9 Schematic block diagram of the system structure of the electronic device 9600 according to an embodiment of the present application. Figure 9 As shown, the electronic device 9600 may include a central processing unit 9100 and a memory 9140; the memory 9140 is coupled to the central processing unit 9100. It is worth noting that the Figure 9 is exemplary; other types of structures may also be used to supplement or replace this structure to implement telecommunication functions or other functions.

[0224] In one embodiment, the front-end large file upload method function can be integrated into the central processing unit 9100. The central processing unit 9100 can be configured to perform the following control:

[0225] Step S101: The front end receives a large file uploaded by a user, dynamically segments the large file according to the network data at the time of uploading the large file and a set dynamic segmentation model, determines the corresponding segments, performs content detection on the segments, determines the content features contained in each segment, prioritizes the segments according to the content features, and determines the corresponding priority order;

[0226] Step S102: obtaining the network bandwidth and load of the server in real time, determining the corresponding number of concurrent upload segments according to the network bandwidth, the load, and a preset concurrency rule, and concurrently uploading the segments according to the priority order and the number of concurrent upload segments according to a preset multiplexing technology;

[0227] Step S103: Determine whether the concurrent upload of each fragment is successful. If it fails, retransmit the fragment that failed to be uploaded according to the breakpoint resume technology. If it succeeds, trigger the back-end merging operation, splice the fragments into a complete file, and verify the complete file according to the preset hash algorithm to obtain the complete large file after uploading.

[0228] From the above description, it can be seen that the electronic device provided by the embodiment of the present application receives large files uploaded by users through the front end, dynamically segments the large files according to the set dynamic segmentation model, determines the corresponding segments, determines the priority of each segment, determines the corresponding priority order, determines the corresponding number of concurrently uploaded segments according to the network bandwidth, load and preset concurrency rules, and concurrently uploads each segment according to the priority order and the number of concurrently uploaded segments according to the preset multiplexing technology, determines whether the concurrent upload of each segment is successful, and if it fails, retransmits the failed upload segment according to the breakpoint resume technology. If it succeeds, triggers the back-end merge operation to obtain the complete large file after upload, thereby enabling file segmentation and concurrent upload based on dynamically adjusted segment size and concurrency, thereby improving the efficiency and flexibility of large file upload.

[0229] In another embodiment, the front-end large file upload method can be configured separately from the central processor 9100. For example, the front-end large file upload method can be configured as a chip connected to the central processor 9100, and the function of the front-end large file upload method can be realized through the control of the central processor.

[0230] like Figure 9 As shown, the electronic device 9600 may further include: a communication module 9110, an input unit 9120, an audio processor 9130, a display 9160, and a power supply 9170. It is worth noting that the electronic device 9600 does not necessarily have to include Figure 9 In addition, the electronic device 9600 may also include all components shown in Figure 9 For components not shown, reference may be made to the prior art.

[0231] like Figure 9 As shown, the central processing unit 9100 is sometimes also referred to as a controller or operation control, and may include a microprocessor or other processor device and / or logic device. The central processing unit 9100 receives input and controls the operation of various components of the electronic device 9600.

[0232] Memory 9140 can be, for example, one or more of a cache, flash memory, hard drive, removable media, volatile memory, non-volatile memory, or other suitable devices. It can store the aforementioned failure-related information and also store programs that execute the relevant information. The CPU 9100 can execute the programs stored in memory 9140 to implement information storage or processing.

[0233] The input unit 9120 provides input to the central processing unit 9100. The input unit 9120 may be, for example, a keypad or touch input device. The power supply 9170 is used to provide power to the electronic device 9600. The display 9160 is used to display objects such as images and text. The display may be, for example, an LCD display, but is not limited thereto.

[0234] The memory 9140 may be a solid-state memory, such as a read-only memory (ROM), random access memory (RAM), or SIM card. Alternatively, it may be a memory that retains information even when power is off, can be selectively erased, and is capable of storing additional data. Examples of such memory are sometimes referred to as EPROMs. The memory 9140 may also be some other type of device. The memory 9140 includes a buffer memory 9141 (sometimes referred to as a buffer). The memory 9140 may include an application / function storage unit 9142 for storing application programs and function programs, or processes used by the central processing unit 9100 to execute operations of the electronic device 9600.

[0235] The memory 9140 may also include a data storage unit 9143 for storing data, such as contacts, digital data, images, sounds, and / or any other data used by the electronic device. The driver storage unit 9144 of the memory 9140 may include various driver programs for communication functions of the electronic device and / or for executing other functions of the electronic device (such as messaging applications, address book applications, etc.).

[0236] The communication module 9110 is a transmitter / receiver that transmits and receives signals via the antenna 9111. The communication module 9110 is coupled to the central processing unit 9100 to provide input signals and receive output signals, which may be the same as the case of a conventional mobile communication terminal.

[0237] Based on different communication technologies, multiple communication modules 9110 can be provided in the same electronic device, such as a cellular network module, a Bluetooth module, and / or a wireless local area network module. The communication module 9110 is also coupled to a speaker 9131 and a microphone 9132 via an audio processor 9130 to provide audio output via the speaker 9131 and receive audio input from the microphone 9132, thereby implementing common telecommunication functions. The audio processor 9130 may include any suitable buffer, decoder, amplifier, etc. Furthermore, the audio processor 9130 is also coupled to the central processing unit 9100, enabling local recording via the microphone 9132 and playback of stored audio via the speaker 9131.

[0238] The embodiments of the present application also provide a computer-readable storage medium capable of implementing all steps of the front-end large file upload method in the above-mentioned embodiment, where the execution subject is a server or a client. The computer-readable storage medium stores a computer program. When the computer program is executed by a processor, all steps of the front-end large file upload method in the above-mentioned embodiment are implemented. For example, when the processor executes the computer program, the following steps are implemented:

[0239] Step S101: The front end receives a large file uploaded by a user, dynamically segments the large file according to the network data at the time of uploading the large file and a set dynamic segmentation model, determines the corresponding segments, performs content detection on the segments, determines the content features contained in each segment, prioritizes the segments according to the content features, and determines the corresponding priority order;

[0240] Step S102: obtaining the network bandwidth and load of the server in real time, determining the corresponding number of concurrent upload segments according to the network bandwidth, the load, and a preset concurrency rule, and concurrently uploading the segments according to the priority order and the number of concurrent upload segments according to a preset multiplexing technology;

[0241] Step S103: Determine whether the concurrent upload of each fragment is successful. If it fails, retransmit the fragment that failed to be uploaded according to the breakpoint resume technology. If it succeeds, trigger the back-end merging operation, splice the fragments into a complete file, and verify the complete file according to the preset hash algorithm to obtain the complete large file after uploading.

[0242] From the above description, it can be seen that the computer-readable storage medium provided in the embodiment of the present application receives large files uploaded by users through the front end, dynamically segments the large files according to the set dynamic segmentation model, determines the corresponding segments, determines the priority of each segment, determines the corresponding priority order, determines the corresponding number of concurrently uploaded segments according to the network bandwidth, load and preset concurrency rules, and concurrently uploads each segment according to the priority order and the number of concurrently uploaded segments according to the preset multiplexing technology, determines whether the concurrent upload of each segment is successful, and if it fails, retransmits the failed upload segment according to the breakpoint resume technology. If successful, triggers the back-end merge operation to obtain the complete large file after upload, thereby enabling file segmentation and concurrent upload based on dynamically adjusted segment size and concurrency, thereby improving the efficiency and flexibility of large file upload.

[0243] The embodiments of the present application also provide a computer program product capable of implementing all steps of the front-end large file upload method in the above-mentioned embodiment, where the execution subject is a server or a client. When the computer program / instruction is executed by a processor, the steps of the front-end large file upload method are implemented. For example, the computer program / instruction implements the following steps:

[0244] Step S101: The front end receives a large file uploaded by a user, dynamically segments the large file according to the network data at the time of uploading the large file and a set dynamic segmentation model, determines the corresponding segments, performs content detection on the segments, determines the content features contained in each segment, prioritizes the segments according to the content features, and determines the corresponding priority order;

[0245] Step S102: obtaining the network bandwidth and load of the server in real time, determining the corresponding number of concurrent upload segments according to the network bandwidth, the load, and a preset concurrency rule, and concurrently uploading the segments according to the priority order and the number of concurrent upload segments according to a preset multiplexing technology;

[0246] Step S103: Determine whether the concurrent upload of each fragment is successful. If it fails, retransmit the fragment that failed to be uploaded according to the breakpoint resume technology. If it succeeds, trigger the back-end merging operation, splice the fragments into a complete file, and verify the complete file according to the preset hash algorithm to obtain the complete large file after uploading.

[0247] From the above description, it can be seen that the computer program product provided by the embodiment of the present application receives large files uploaded by users through the front end, dynamically segments the large files according to the set dynamic segmentation model, determines the corresponding segments, determines the priority of each segment, determines the corresponding priority order, determines the corresponding number of concurrently uploaded segments according to the network bandwidth, load and preset concurrency rules, and concurrently uploads each segment according to the priority order and the number of concurrently uploaded segments according to the preset multiplexing technology, determines whether the concurrent upload of each segment is successful, and if it fails, retransmits the failed upload segment according to the breakpoint resume technology. If it succeeds, triggers the back-end merge operation to obtain the complete large file after upload, thereby enabling file segmentation and concurrent upload based on dynamically adjusted segment size and concurrency, thereby improving the efficiency and flexibility of large file upload.

[0248] Those skilled in the art will appreciate that embodiments of the present invention may be provided as methods, apparatuses, or computer program products. Thus, the present invention may take the form of an entirely hardware embodiment, an entirely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to magnetic disk storage, CD-ROMs, optical storage, etc.) containing computer-usable program code.

[0249] The present invention is described with reference to flowcharts and / or block diagrams of methods, devices (apparatus), and computer program products according to embodiments of the present invention. It should be understood that each process and / or block in the flowcharts and / or block diagrams, as well as combinations of processes and / or blocks in the flowcharts and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowcharts and / or block diagrams. Figure 1 a process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0250] These computer program instructions may also be stored in a computer readable memory that can direct a computer or other programmable data processing device to work in a specific manner, so that the instructions stored in the computer readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 a process or multiple processes and / or boxes Figure 1 The function specified in one or more boxes.

[0251] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operational steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing the instructions executed on the computer or other programmable device for implementing the process. Figure 1 a process or multiple processes and / or boxes Figure 1 A step that specifies a function in one or more boxes.

[0252] Specific embodiments are used in the present invention to illustrate the principles and implementation methods of the present invention. The description of the above embodiments is only used to help understand the method of the present invention and its core ideas. At the same time, for those skilled in the art, according to the ideas of the present invention, there may be changes in the specific implementation methods and application scopes. In summary, the contents of this specification should not be understood as limiting the present invention.

Claims

1. A front-end large file uploading method, characterized in that: The method comprises: The front end receives a large file uploaded by the user, and constructs an initial dual-network model based on a preset main network and a preset target network, wherein the main network is used to select an action, and the target network is used to evaluate the value of the action. The initial dual-network model is trained based on a set experience buffer to determine a corresponding dynamic sharding model, wherein the experience buffer is used to store historical experience. The large file is dynamically fragmented based on the network data when the large file is uploaded and the set dynamic sharding model to determine the corresponding individual shards. Content detection is performed on each shard to determine the content features contained in each shard. The shards are prioritized based on the content features to determine the corresponding priority order. Acquire the network bandwidth and load of the server in real time, determine the corresponding number of concurrent upload fragments according to the network bandwidth, the load and the preset concurrency rule, and perform concurrent upload operations on the fragments according to the priority order and the number of concurrent upload fragments according to the preset multiplexing technology; Determine whether the concurrent upload of each fragment is successful. If failed, retransmit the fragments that failed to be uploaded according to the breakpoint resume technology. If successful, trigger the back-end merging operation, splice the fragments into a complete file, and verify the complete file according to the preset hash algorithm to obtain the complete large file after uploading.

2. The front-end large file uploading method according to claim 1, characterized in that: Before performing model training on the initial dual-network model according to the set experience buffer to determine the corresponding dynamic sharding model, the method includes: Collect historical network data, extract network bandwidth change trends from the historical network data using a sliding window technique, construct a corresponding state space based on the network bandwidth change trends, a preset network bandwidth, and a preset slice size, define an action space, adjust the slice change amplitude of the action space using a preset slice dynamic adjustment technique, and determine a corresponding continuous action space; The preset main network performs a Q-value calculation operation on each action in the continuous action space according to the current state in the state space using a greedy algorithm, and selects the action with the largest Q-value to obtain the action corresponding to the current state, wherein the Q-value calculation operation includes: target Q-value = reward + discount factor × Q-value evaluated by the target network, and the reward is used to evaluate the effect of each action; A corresponding reward and a new state are determined according to the execution result of the action, and an experience buffer is constructed according to the current state, the action, the reward, and the new state.

3. The front-end large file uploading method according to claim 1, characterized in that: The performing model training on the initial dual network model according to the set experience buffer to determine the corresponding dynamic sharding model includes: Performing action selection on the new state in the experience buffer according to the main network in the initial dual-network model to determine the corresponding optimal action; The optimal action is evaluated according to the target network in the initial dual-network model to determine the optimal target Q value, and the main network parameters are updated according to the optimal target Q value to obtain the corresponding dynamic slicing model.

4. The front-end large file uploading method according to claim 1, characterized in that: The dynamically slicing the large file according to the network data when the large file is uploaded and the set dynamic slicing model to determine the corresponding slicing includes: Dynamically fragmenting the large file according to the network data when the large file is uploaded and the set dynamic fragmentation model, and determining the corresponding fragment sizes; The large file is sliced ​​according to the preset file.slice method and the slice size to determine the corresponding slices.

5. The front-end large file uploading method according to claim 1, characterized in that: Prioritizing the fragments according to the content features to determine the corresponding priority order includes: If it is a video feature, key frame recognition is performed on the video feature segment according to the preset frame difference method to determine the corresponding priority order; If it is a text feature, perform keyword extraction on the text feature segment according to the preset natural language processing technology to determine the corresponding priority order; If it is an image feature, the image feature segment is subject identified according to the preset saliency detection algorithm to determine the corresponding priority order.

6. The front-end large file uploading method according to claim 1, characterized in that: Prioritizing the fragments according to the content features to determine the corresponding priority order further includes: If it is a video feature, the scene change key frame of the video feature segment is identified according to the optical flow method, and the segment with the key frame is determined; The slices with key frames are divided into high-priority slices, and the corresponding priority order is determined.

7. A front-end large file uploading device, characterized in that: The device comprises: A large file sharding module is used to receive large files uploaded by users at the front end, build an initial dual-network model based on a preset main network and a preset target network, wherein the main network is used to select actions and the target network is used to evaluate the value of actions, perform model training on the initial dual-network model based on a set experience buffer, and determine a corresponding dynamic sharding model, wherein the experience buffer is used to store historical experience, dynamically segment the large file based on the network data when the large file is uploaded and the set dynamic sharding model, determine the corresponding individual shards, perform content detection on the individual shards, determine the content features contained in each shard, prioritize the individual shards based on the content features, and determine the corresponding priority order; A concurrent upload module is configured to obtain the network bandwidth and load of the server in real time, determine the corresponding number of concurrent upload fragments according to the network bandwidth, the load, and a preset concurrency rule, and concurrently upload the fragments according to the priority order and the number of concurrent upload fragments according to a preset multiplexing technology; The upload verification module is used to determine whether the concurrent upload of each fragment is successful. If it fails, the fragments that failed to be uploaded are retransmitted according to the breakpoint resume technology. If it succeeds, the back-end merge operation is triggered to splice the fragments into a complete file. The complete file is verified according to the preset hash algorithm to obtain the complete large file after upload.

8. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the program, the steps of the front-end large file uploading method described in any one of claims 1 to 6 are implemented.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the front-end large file uploading method described in any one of claims 1 to 6 are implemented.

Images