Buffer capacity processing method and device, electronic equipment, medium and product

Abstract

The application provides a buffer capacity processing method and device, electronic equipment, medium and product. It relates to the field of distributed storage. The method comprises the following steps: determining the request type of a network request, the request type comprising an internal request and an external request; according to the request type, adopting a preset corresponding buffer allocation strategy to allocate an initial buffer for the network request; in the data receiving process, detecting the usage state of the initial buffer; if it is detected that the capacity of the initial buffer is insufficient, dynamically adjusting the capacity of the initial buffer. Based on the buffer capacity processing method provided by the application, the buffer size can be dynamically adjusted, thereby significantly improving the performance and reliability of the distributed storage system.

Description

Technical Field

[0001] This application relates to the field of distributed storage, and more particularly to a method, apparatus, electronic device, medium, and product for processing buffer capacity. Background Technology

[0002] High-efficiency network data access systems are widely used in high-concurrency scenarios of distributed storage systems, especially in fields with extremely high requirements for data throughput and reliability, such as cloud computing, big data processing, real-time log analysis, and artificial intelligence model training. In these scenarios, distributed storage systems need to handle massive numbers of user requests simultaneously. The system needs to ensure the stability, write efficiency, and consistency of data transmission under high concurrency pressure, which can be achieved by temporarily storing data in a memory buffer and then asynchronously writing it to disk.

[0003] Currently, related technologies typically use fixed-size buffers. When the buffer is too small and the amount of data exceeds the buffer capacity, it may trigger an overflow or discard data; while an excessively large buffer will consume a lot of memory resources and increase system overhead. Summary of the Invention

[0004] This application provides a method, apparatus, electronic device, medium, and product for processing buffer capacity, which can dynamically adjust the buffer size, thereby significantly improving the performance and reliability of distributed storage systems.

[0005] In a first aspect, this application provides a method for processing buffer capacity, including:

[0006] Determine the type of network request, which includes internal and external requests;

[0007] Based on the request type, a preset buffer allocation strategy is used to allocate an initial buffer for the network request;

[0008] During data reception, the usage status of the initial buffer is detected. If the capacity of the initial buffer is insufficient, the capacity of the initial buffer is dynamically adjusted.

[0009] In one possible implementation, the capacity of the initial buffer is dynamically adjusted, specifically including at least one of the following:

[0010] Check if the amount of data already stored in the initial buffer exceeds the target value; if it does, call the memory management function to expand the capacity of the initial buffer.

[0011] A machine learning model is trained based on historical request data to predict the buffer capacity requirements of network requests; the initial buffer capacity is adjusted based on the prediction results.

[0012] In one possible implementation, after the step of determining the request type of the network request, the method further includes:

[0013] Get the status response code of the network request;

[0014] If the status response code is within the preset range, the network request is allowed to proceed to the next processing step; otherwise, the network request is discarded.

[0015] In one possible implementation, after the step of dynamically adjusting the capacity of the initial buffer, the method further includes:

[0016] The data in the initial buffer is split into multiple data fragments;

[0017] Record the storage location of each data segment and detect whether the storage location overlaps with the recorded target area.

[0018] In one possible implementation, the step of splitting the data in the initial buffer into multiple data segments specifically includes:

[0019] Select the corresponding sharding strategy based on the data type identifier;

[0020] The data is split into multiple data fragments according to the sharding strategy.

[0021] In one possible implementation, after the step of recording the storage location of each data segment, the method further includes:

[0022] Performs an atomic write operation on the data fragment and releases the memory of the initial buffer after the write is complete.

[0023] Secondly, this application provides a buffer capacity processing apparatus, comprising:

[0024] The determination module is used to determine the request type of a network request, which includes internal requests and external requests;

[0025] The processing module is used to allocate an initial buffer for a network request based on the request type and a preset corresponding buffer allocation strategy.

[0026] The processing module is also used to detect the usage status of the initial buffer during data reception. If the capacity of the initial buffer is insufficient, the capacity of the initial buffer is dynamically adjusted.

[0027] In one possible implementation, the processing module is specifically used to detect whether the amount of data already stored in the initial buffer exceeds the target value; if it exceeds the target value, the memory management function is called to expand the capacity of the initial buffer.

[0028] The processing module is also used to train a machine learning model based on historical request data, predict the buffer capacity requirements of network requests, and adjust the initial buffer capacity according to the prediction results.

[0029] In one possible implementation, the processing module is further configured to obtain the status response code of the network request;

[0030] The processing module is also used to allow the network request to proceed to the subsequent processing flow if the status response code is within a preset range, otherwise discard the network request.

[0031] In one possible implementation, the processing module is further configured to split the data in the initial buffer into multiple data fragments;

[0032] The processing module is also used to record the storage location of each data segment and detect whether the storage location overlaps with the recorded target area.

[0033] In one possible implementation, the processing module is further configured to select the corresponding sharding strategy based on the data type identifier;

[0034] The processing module is also used to split the data into multiple data fragments according to the sharding strategy.

[0035] In one possible implementation, the processing module is further configured to perform atomic write operations on the data fragment and release the memory of the initial buffer after the write is complete.

[0036] Thirdly, this application provides an electronic device, including: a processor, and a memory communicatively connected to the processor;

[0037] The memory stores the instructions that the computer executes;

[0038] The processor executes computer execution instructions stored in memory to implement the buffer capacity handling method as described in the first aspect and / or any possible implementation of the first aspect.

[0039] Fourthly, this application provides a computer-readable storage medium storing computer-executable instructions, which, when executed by a processor, are used to implement a buffer capacity processing method as described in the first aspect and / or any possible implementation of the first aspect.

[0040] Fifthly, this application provides a computer program product, including a computer program that, when executed by a processor, implements a buffer capacity processing method as described in the first aspect and / or any possible implementation of the first aspect.

[0041] The buffer capacity processing method, apparatus, electronic device, medium, and product provided in this application achieve precise allocation and expansion of buffers in high-concurrency scenarios, reducing memory fragmentation problems and improving resource utilization and system stability. Furthermore, the dynamic adjustment strategy reduces the system's dependence on preset buffer sizes and enhances its adaptability to sudden traffic surges, thereby significantly improving the performance and reliability of distributed storage systems. Attached Figure Description

[0042] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application.

[0043] Figure 1 This is a schematic diagram of a scenario provided for an embodiment of this application;

[0044] Figure 2 Flowchart of the buffer capacity processing method provided in this application Figure 1 ;

[0045] Figure 3 Flowchart of the buffer capacity processing method provided in this application Figure 2 ;

[0046] Figure 4 Flowchart of the buffer capacity processing method provided in this application Figure 3 ;

[0047] Figure 5 Flowchart of the buffer capacity processing method provided in this application Figure 4 ;

[0048] Figure 6 Flowchart of the buffer capacity processing method provided in this application Figure 5 ;

[0049] Figure 7 This is a schematic diagram of the buffer capacity processing device provided in the embodiments of this application;

[0050] Figure 8 A schematic diagram of the structure of the electronic device provided in this application.

[0051] The accompanying drawings illustrate specific embodiments of this application, which will be described in more detail below. These drawings and descriptions are not intended to limit the scope of the concept in any way, but rather to illustrate the concept of this application to those skilled in the art through reference to particular embodiments. Detailed Implementation

[0052] Exemplary embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numbers in different drawings denote the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this application. Rather, they are merely examples of apparatuses and methods consistent with some aspects of this application as detailed in the appended claims.

[0053] It should be noted that in the embodiments of this application, certain software, components, models and other existing solutions in the industry may be mentioned. These should be regarded as exemplary and are only intended to illustrate the feasibility of implementing the technical solution of this application. However, it does not mean that the applicant has used or necessarily used the solution.

[0054] In high-concurrency scenarios, distributed storage systems are particularly crucial, especially in fields like cloud computing, big data processing, real-time log analysis, and AI model training, where data throughput and reliability are paramount. In these scenarios, distributed storage systems need to handle massive numbers of user requests simultaneously. Examples include file uploads / downloads in cloud storage gateways, log data aggregation in real-time log acquisition and analysis platforms, and large-scale model parameter synchronization in AI training frameworks. Taking cloud storage gateways as an example, users may frequently upload or download large files (such as videos and virtual machine images) via the Hypertext Transfer Protocol (HTTP), and the system needs to ensure data transmission stability, write efficiency, and consistency under high concurrency pressure. Furthermore, in real-time log processing scenarios, the system needs to continuously receive log data streams from multiple terminal devices, temporarily store them in a memory buffer, and then asynchronously write them to disk, while avoiding data corruption caused by network fluctuations or server anomalies.

[0055] Currently, related technologies typically use fixed-size buffers. When the buffer is too small and the amount of data exceeds the buffer capacity, it may trigger an overflow or discard data; while an excessively large buffer will consume a lot of memory resources and increase system overhead.

[0056] Figure 1 This is a schematic diagram of a scenario provided for an embodiment of this application, such as... Figure 1As shown, the buffer capacity processing method, apparatus, electronic device, medium, and product provided in this application construct a network data access system adapted to high-concurrency scenarios by combining dynamic resource allocation with a hierarchical optimization strategy. The system adopts "on-demand dynamic adjustment + fine-grained task splitting" as its core design principle. It allocates an initial buffer for network requests and monitors the usage status of the initial buffer during data reception. If the initial buffer's capacity is insufficient, it dynamically adjusts the initial buffer's capacity. This achieves precise allocation and expansion of the buffer in high-concurrency scenarios, reducing memory fragmentation and improving resource utilization and system stability. Furthermore, the dynamic adjustment strategy reduces the system's dependence on the preset buffer size, enhancing its adaptability to sudden traffic surges, thereby significantly improving the performance and reliability of the distributed storage system.

[0057] The technical solution of this application and how the technical solution of this application solves the above-mentioned technical problems are described in detail below with specific embodiments. These specific embodiments can be combined with each other, and the same or similar concepts or processes may not be described again in some embodiments. The embodiments of this application will now be described with reference to the accompanying drawings.

[0058] Figure 2 Flowchart of the buffer capacity processing method provided in this application Figure 1 ,like Figure 2 As shown, the operation flow of the buffer capacity processing method is as follows:

[0059] S201. Determine the request type of the network request. Request types include internal requests and external requests.

[0060] In the context of specific scenarios, request type refers to the classification of the source or purpose of a network request, such as an internal request (communication between system components) or an external request (client communication). For example, an internal request might include system log synchronization operations, while an external request might include client file upload operations.

[0061] S202. Based on the request type, the corresponding preset buffer allocation strategy is used to allocate an initial buffer for the network request.

[0062] In the context of scenario examples, buffer allocation strategies refer to memory allocation rules selected based on request type. For instance, a fixed-size buffer might be allocated for internal requests, while a dynamically adjusted buffer might be allocated for external requests. For example, internal requests might use a preset 128MB buffer, while external requests might dynamically adjust to 256MB or larger based on the data volume. The initial buffer refers to the memory space initially allocated during the data reception phase, used for temporary storage of network request data. For example, when a client uploads a file, the system initially allocates a 128MB buffer.

[0063] S203. During the data reception process, the usage status of the initial buffer is detected. If the capacity of the initial buffer is insufficient, the capacity of the initial buffer is dynamically adjusted.

[0064] In the context of a scenario, insufficient capacity means that the amount of data already stored in the initial buffer exceeds its maximum capacity limit, making it unable to receive new data. For example, when the file size uploaded by the client exceeds 128MB, the initial buffer's capacity is insufficient. Dynamic adjustment refers to reallocating memory space based on the current data volume to expand the initial buffer's capacity. For example, the `realloc` function can be used to expand the initial buffer from 128MB to 256MB.

[0065] This example first distinguishes the source or purpose of the request (internal or external). For example, in a cloud storage gateway, the system determines the request type by checking the source Internet Protocol (IP) address or protocol header information. Second, it selects the corresponding buffer allocation strategy based on the request type: internal requests typically use a fixed-size buffer (e.g., 128MB), while external requests dynamically adjust the buffer size based on the data volume. For example, when a client uploads a large file, the system dynamically allocates an initial buffer. Finally, during data reception, the system monitors the usage status of the initial buffer. If the data volume exceeds the initial buffer capacity (e.g., the file size uploaded by the client exceeds 128MB), the `realloc` function expands the buffer capacity to 256MB or larger, ensuring uninterrupted data reception. The entire process uses request type to drive the switching of buffer allocation strategies and combines real-time data volume detection to achieve dynamic expansion, thereby adapting to memory requirements in different scenarios.

[0066] Based on the method provided in this example, differentiated buffer strategies are assigned to different types of requests by distinguishing between internal and external requests. For example, internal requests typically involve small and fixed amounts of data, so a fixed buffer strategy can reduce memory waste; external requests (such as large file uploads) require dynamic adjustment of the buffer capacity to adapt to fluctuations in data volume. Secondly, by monitoring the usage status of the initial buffer in real time, the system automatically expands the buffer when its capacity is insufficient, preventing data loss or processing interruptions. For example, when the file size uploaded by the client exceeds the initial buffer capacity, the system expands the buffer using the `realloc` function to ensure complete data reception. Through these techniques, precise allocation and expansion of the buffer are achieved in high-concurrency scenarios, reducing memory fragmentation and improving resource utilization and system stability. Furthermore, the dynamic adjustment strategy reduces the system's dependence on preset buffer sizes, enhancing its adaptability to sudden traffic surges, thereby significantly improving the performance and reliability of the distributed storage system.

[0067] Optional, Figure 3Flowchart of the buffer capacity processing method provided in this application Figure 2 ,like Figure 3 As shown, dynamically adjusting the initial buffer capacity specifically includes at least one of the following:

[0068] S301. Check if the amount of data already stored in the initial buffer exceeds the target value; if it does, call the memory management function to expand the capacity of the initial buffer.

[0069] In the example scenario, the preset threshold refers to the upper limit of the initial buffer capacity. Dynamic adjustment is triggered when the data volume exceeds this value. For instance, if the initial buffer is 128MB and the preset threshold is 120MB, expansion is triggered when the data volume exceeds 120MB. Memory management functions refer to system functions used to dynamically adjust memory space, such as `realloc`. `realloc` can be used to expand the initial buffer from 128MB to 256MB. In the step of dynamically adjusting the initial buffer capacity, the system first checks whether the amount of stored data exceeds the preset threshold. For example, when a client uploads a file, the system monitors the buffer usage in real time. If the data volume reaches 120MB (93.75% of the preset threshold of 128MB), the memory management function (such as `realloc`) is called to expand the buffer capacity to 256MB. This process, through the coordinated work of the detection mechanism and the memory management function, achieves automatic expansion of the buffer capacity, avoiding data overflow or processing interruption. Therefore, based on the method in this example, by combining preset threshold detection with memory management functions, the buffer capacity is accurately and dynamically adjusted. When dealing with sudden traffic or long-term load changes, the system's adaptability to dynamic resource allocation is significantly improved, and the stability of the distributed storage system is enhanced.

[0070] Optionally, the target value of the preset threshold can be dynamically adjusted based on the request type. Dynamically adjusting the preset threshold means resetting the upper limit of the buffer capacity according to the request type (internal / external). For example, the preset threshold for internal requests is 80MB, and the preset threshold for external requests is 120MB. Before detecting the amount of data in the buffer, the system dynamically adjusts the preset threshold value based on the request type (internal / external). For example, for internal requests (such as log synchronization), which have a smaller data volume, the preset threshold is set to 80MB; for external requests (such as file uploads), which have a larger data volume, the preset threshold is set to 120MB. This mechanism, through differentiated threshold settings, makes the buffer capacity adjustment more closely match actual needs, avoids unnecessary expansion operations triggered by internal requests, and thus improves resource utilization.

[0071] S302. Train a machine learning model based on historical request data to predict the buffer capacity requirements of network requests; adjust the initial buffer capacity according to the prediction results.

[0072] In the context of this example, a machine learning model refers to a predictive model trained on historical data, such as a Long Short-Term Memory (LSTM) network or a random forest. For instance, the system trains an LSTM model using historical request data to predict the buffer requirements of current requests. Based on the method provided in this example, by analyzing historical request data using a machine learning model (such as LSTM) to predict the buffer requirements of current requests, this mechanism improves the accuracy of buffer allocation through data-driven prediction. Furthermore, the predictive power of the machine learning model enables adaptive adjustment of buffer capacity. For example, in scenarios with sudden traffic surges, the model can anticipate capacity demands in advance, reducing the overhead of frequent buffer expansions and thus improving the system's adaptability to dynamic loads and resource utilization.

[0073] Optional, Figure 4 Flowchart of the buffer capacity processing method provided in this application Figure 3 ,like Figure 4 As shown, after the step of determining the request type of the network request, the following steps are also included:

[0074] S401. Obtain the status response code of the network request.

[0075] In conjunction with the scenario example, the status response code can be an HTTP response code. An HTTP response code refers to the status code in the HTTP protocol that the server responds to the client's request, such as 200 (success) and 500 (server error). For example, after the client uploads a file, the server returns 200 OK to indicate that the request was successful.

[0076] S402. If the status response code is within the preset range, the network request is allowed to proceed to the subsequent processing flow; otherwise, the network request is discarded.

[0077] With a scenario example, after identifying the request type, the system obtains the response code via the HTTP protocol and determines whether it falls within the 200-299 range. For instance, if the server returns a 200 OK response after the client uploads a file, the request enters the buffer allocation process; if it returns a 500 Server Error response, the request is discarded. This mechanism uses the HTTP protocol standard to implement data filtering, proactively shielding invalid data, ensuring that only valid requests are processed, and preventing error logs from being written to storage, thereby improving the accuracy and reliability of data processing.

[0078] Optional, Figure 5 Flowchart of the buffer capacity processing method provided in this application Figure 4 ,like Figure 5 As shown, after the step of dynamically adjusting the initial buffer capacity, the following steps are also included:

[0079] S501, Split the data in the initial buffer into multiple data segments.

[0080] Based on the scenario example, after dynamically adjusting the buffer capacity, the system splits the data into multiple data fragments.

[0081] S502. Record the storage location of each data segment and detect whether the storage location overlaps with the recorded target area.

[0082] In this scenario, the target region refers to the data area in a file that has been modified but not yet written to disk. After dynamically adjusting the buffer capacity, the system splits the data into multiple fragments and records the storage location of each fragment. For example, a large file is split into 10 128KB fragments, and the storage location of each fragment (such as its file offset) is recorded. The system further checks whether these storage locations overlap with the recorded target region (such as bytes 0-100), and only performs actual write operations on non-overlapping regions. This mechanism reduces disk I / O operations through fragmentation and target region detection. Simultaneously, by avoiding duplicate writing of modified data, it reduces disk load and improves write efficiency and storage performance.

[0083] Optional, Figure 6 Flowchart of the buffer capacity processing method provided in this application Figure 5 ,like Figure 6 As shown, step S501 specifically includes:

[0084] S601. Select the corresponding sharding strategy based on the data type identifier.

[0085] In the provided scenario, the type identifier can be a Multipurpose Internet Mail Extensions (MIME) type, such as text / plain (text) or image / jpeg (image). The system identifies the data content (such as text or image) through the MIME type and selects the corresponding sharding strategy. For example, text data is split line by line, and image data is split into blocks. This mechanism drives the sharding strategy through data characteristics, improving write efficiency and storage performance.

[0086] S602. Split the data into multiple data fragments according to the fragmentation strategy.

[0087] With scenario examples, a data feature-driven sharding strategy was used to optimize shard size and write order. For instance, splitting text data line by line improves log parsing efficiency, and splitting images into blocks reduces decompression overhead, thereby significantly improving storage efficiency and data processing speed.

[0088] Optionally, after the step of recording the storage location of each data segment, the following may also be included:

[0089] Performs an atomic write operation on the data fragment and releases the memory of the initial buffer after the write is complete.

[0090] In the context of this example, an atomic write operation refers to a single, uninterruptible write operation, such as the `pwrite` system call. For instance, `pwrite` writes a data fragment to a specified location in a file, preventing file pointer offsets. After recording the data fragment's storage location, the system uses atomic write operations like `pwrite` to write the data to disk and then calls `free` to release the initial buffer memory after the write is complete. This mechanism, through the coordination of atomic operations and memory reclamation, ensures data consistency and efficient resource utilization. Therefore, based on the method provided in this example, atomic writes and memory reclamation mechanisms achieve a unified approach to data consistency and resource reclamation, reducing memory leaks and significantly improving system stability and resource utilization.

[0091] Based on the buffer capacity processing method provided in this embodiment, precise allocation and expansion of the buffer are achieved in high-concurrency scenarios, reducing memory fragmentation and improving resource utilization and system stability. Furthermore, the dynamic adjustment strategy reduces the system's dependence on the preset buffer size, enhances its adaptability to sudden traffic surges, and thus significantly improves the performance and reliability of the distributed storage system.

[0092] Figure 7 This is a schematic diagram of the buffer capacity processing device provided in the embodiments of this application, as shown below. Figure 7 As shown, it includes:

[0093] The determination module 71 is used to determine the request type of the network request, which includes internal requests and external requests;

[0094] Processing module 72 is used to allocate an initial buffer for a network request according to the request type and a preset corresponding buffer allocation strategy;

[0095] The processing module 72 is also used to detect the usage status of the initial buffer during the data reception process, and dynamically adjust the capacity of the initial buffer if the capacity of the initial buffer is insufficient.

[0096] Optionally, the processing module 72 is specifically used to detect whether the amount of data already stored in the initial buffer exceeds the target value; if it exceeds the target value, the memory management function is called to expand the capacity of the initial buffer.

[0097] The processing module 72 is also used to train a machine learning model based on historical request data, predict the buffer capacity requirements of network requests, and adjust the capacity of the initial buffer according to the prediction results.

[0098] Optionally, the processing module 72 is also used to obtain the status response code of the network request;

[0099] The processing module 72 is also used to allow the network request to enter the subsequent processing flow if the status response code is within a preset range, otherwise discard the network request.

[0100] Optionally, the processing module 72 is also used to split the data in the initial buffer into multiple data fragments;

[0101] The processing module 72 is also used to record the storage location of each data segment and detect whether the storage location overlaps with the recorded target area.

[0102] Optionally, the processing module 72 is further configured to select the corresponding sharding strategy based on the data type identifier;

[0103] The processing module 72 is specifically used to split the data into multiple data fragments according to the fragmentation strategy.

[0104] Optionally, the processing module 72 is also configured to perform atomic write operations on the data fragment and release the memory of the initial buffer after the write is complete.

[0105] The buffer capacity processing device provided in this embodiment can execute the buffer capacity processing method provided in the above method embodiment. Its implementation principle and technical effect are similar, and will not be described in detail here.

[0106] Figure 8 A schematic diagram of the structure of the electronic device provided in this application. Figure 8 As shown, the electronic device 50 provided in this embodiment includes at least one processor 501 and a memory 502. Optionally, the electronic device 50 further includes a communication component 503. The processor 501, memory 502, and communication component 503 are connected via a bus.

[0107] In a specific implementation, at least one processor 501 executes computer execution instructions stored in memory 502, causing at least one processor 501 to perform the above-described method.

[0108] The specific implementation process of processor 501 can be found in the above method embodiments, and its implementation principle and technical effect are similar. It will not be repeated here.

[0109] In the above embodiments, it should be understood that the processor can be a Central Processing Unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), etc. The general-purpose processor can be a microprocessor or any conventional processor. The steps of the method disclosed in this invention can be directly implemented by a hardware processor, or implemented by a combination of hardware and software modules within the processor.

[0110] The memory may include random access memory (RAM) and may also include non-volatile memory (NVM), such as at least one disk storage device.

[0111] The bus can be an Industry Standard Architecture (ISA) bus, a Peripheral Component Interconnect (PCI) bus, or an Extended Industry Standard Architecture (EISA) bus, etc. Buses can be categorized as address buses, data buses, control buses, etc. For ease of illustration, the buses shown in the accompanying drawings are not limited to a single bus or a single type of bus.

[0112] This application also provides a computer program product, including a computer program that, when executed by a processor, implements the above-described method.

[0113] This application also provides a computer-readable storage medium storing computer-executable instructions, which, when executed by a processor, implement the above-described method.

[0114] The aforementioned readable storage medium can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic storage, flash memory, magnetic disk, or optical disk. The readable storage medium can be any available medium accessible to a general-purpose or special-purpose computer.

[0115] An exemplary readable storage medium is coupled to a processor, enabling the processor to read information from and write information to the readable storage medium. Of course, the readable storage medium can also be a component of the processor. The processor and the readable storage medium can reside in an Application Specific Integrated Circuit (ASIC). Alternatively, the processor and the readable storage medium can exist as discrete components in the device.

[0116] The division of units is merely a logical functional division; in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be indirect coupling or communication connection through some interfaces, devices, or units, and may be electrical, mechanical, or other forms.

[0117] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0118] In addition, the functional units in the various embodiments of the present invention can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit.

[0119] If a function is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this invention, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of the various embodiments of this invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0120] Those skilled in the art will understand that all or part of the steps of the above-described method embodiments can be implemented by hardware related to program instructions. The aforementioned program can be stored in a computer-readable storage medium. When executed, the program performs the steps of the above-described method embodiments; and the aforementioned storage medium includes various media capable of storing program code, such as ROM, RAM, magnetic disks, or optical disks.

[0121] Finally, it should be noted that other embodiments of the invention will readily occur to those skilled in the art upon consideration of the specification and practice of the invention disclosed herein. This invention is intended to cover any variations, uses, or adaptations of the invention that follow the general principles of the invention and include common knowledge or customary techniques in the art not disclosed herein, and is not limited to the precise structures described above and shown in the accompanying drawings, and various modifications and changes can be made without departing from its scope. The scope of the invention is limited only by the appended claims.

Claims

1. A method for processing buffer capacity, characterized in that, include: Determine the request type of the network request, which includes internal requests and external requests; Based on the request type, a preset corresponding buffer allocation strategy is used to allocate an initial buffer for the network request; During data reception, the usage status of the initial buffer is detected. If the capacity of the initial buffer is insufficient, the capacity of the initial buffer is dynamically adjusted.

2. The method according to claim 1, characterized in that, The dynamic adjustment of the initial buffer capacity specifically includes at least one of the following: Check whether the amount of data already stored in the initial buffer exceeds the target value; if it exceeds the target value, call the memory management function to expand the capacity of the initial buffer; A machine learning model is trained based on historical request data to predict the buffer capacity requirement of the network request; the capacity of the initial buffer is adjusted according to the prediction results.

3. The method according to claim 1, characterized in that, Following the step of determining the request type of the network request, the method further includes: Obtain the status response code of the network request; If the status response code is within a preset range, the network request is allowed to proceed to the subsequent processing flow; otherwise, the network request is discarded.

4. The method according to claim 1, characterized in that, Following the step of dynamically adjusting the capacity of the initial buffer, the method further includes: The data in the initial buffer is split into multiple data fragments; Record the storage location of each data segment and detect whether the storage location overlaps with the recorded target area.

5. The method according to claim 4, characterized in that, The step of splitting the data in the initial buffer into multiple data fragments specifically includes: Select the corresponding sharding strategy based on the data type identifier; The data is split into multiple data fragments according to the aforementioned fragmentation strategy.

6. The method according to claim 4, characterized in that, Following the step of recording the storage location of each data segment, the method further includes: Perform an atomic write operation on the data segment, and release the memory of the initial buffer after the write is complete.

7. A buffer capacity processing device, characterized in that, include: The determination module is used to determine the request type of the network request, which includes internal requests and external requests; The processing module is used to allocate an initial buffer for the network request according to the request type and by adopting a preset corresponding buffer allocation strategy; The processing module is also used to detect the usage status of the initial buffer during data reception, and dynamically adjust the capacity of the initial buffer if the capacity of the initial buffer is insufficient.

8. An electronic device, characterized in that, include: A processor, and a memory communicatively connected to the processor; The memory stores computer-executed instructions; The processor executes computer execution instructions stored in the memory to implement the buffer capacity processing method as described in any one of claims 1-6.

9. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer-executable instructions, which, when executed by a processor, are used to implement the buffer capacity processing method as described in any one of claims 1-6.

10. A computer program product, characterized in that, Includes a computer program that, when executed by a processor, implements the buffer capacity processing method as described in any one of claims 1-6.

Images