Cache-based file uploading cost optimization method, device and equipment
Through the cache-based file upload method, the array return and memory buffer batch upload mechanisms are used to solve the high file upload cost caused by frequent revisions of e-commerce pages, and efficient and low-cost file upload and download operations are achieved, improving the system Performance and user experience.
Patent Information
- Application Number
- CN202410920976.4
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-07-10
- Publication Date
- 2024-11-08
- Estimated Expiration
- 2044-07-10
AI Technical Summary
Frequent revision of e-commerce pages leads to high cost of uploading files. The existing technology requires calling interfaces for each upload, resulting in frequent storage and operation costs.
The cache-based file upload method is adopted, and multiple files are packaged and compressed through array return method, uploaded to cloud storage at one time, and the batch upload mechanism of the memory buffer is used to reduce the number of uploads and memory usage, and optimize resource utilization.
It significantly reduces file upload costs, improves upload efficiency and system performance, reduces time and resource usage, improves user experience, and supports flexible cache management and data storage security.
Smart Images

Figure CN118916334A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of computer technology, and in particular to a method, apparatus, and device for optimizing file upload costs based on caching. Background Technology
[0002] Currently, e-commerce pages frequently require redesign, leading to changes in the DOM structure and rendering the original parsing rules inapplicable, resulting in parsing failures and requiring re-parsing. However, the original page is not stored, and relying solely on the original page data for redesigns easily leads to failures. To solve this problem, it's necessary to obtain the redesigned original page, analyze its structure, formulate new parsing rules, and re-parse the original page. This process requires uploading the original page to a cloud platform for storage for a period of time. With tens of millions of page files uploaded daily, if each file requires a separate upload API call, this amounts to tens of millions of upload API calls per day, or approximately 1 billion per month, resulting in high upload costs. Existing technologies for uploading files to cloud platforms, besides storage costs, also incur charges for each upload. If the number of files is very large (billions), this leads to high upload costs and increased operational costs. Therefore, it is necessary to propose a file upload optimization scheme to significantly reduce upload costs. Summary of the Invention
[0003] The purpose of this invention is to provide a method, apparatus, and device for optimizing file upload costs based on caching. By packaging and compressing multiple files using an array return method, the files are uploaded to cloud storage in one go, which greatly reduces the cost of accessing the cloud, while improving performance and reducing time consumption and resource usage.
[0004] This invention provides a method for optimizing file upload costs based on caching, comprising:
[0005] In response to the request to re-parse the e-commerce page, the system's file message event listening function is invoked;
[0006] When a file message is detected, the page file and context data are obtained, wherein the context data is information related to the attributes of the page file;
[0007] The obtained page file and information related to the page file attributes are packaged for the first time to obtain initial file package data. The initial file package data includes the initial file package name written to the multiplexed array output stream and the effective length information of the underlying stream.
[0008] Based on the topic and date, the output stream of the target file package and the underlying output stream of the target file package are obtained from the mapping. Multiple initial file packages are aggregated as compression items by adding page files to the output stream of the target file package and a second packaging operation is performed, so that data can be written to the underlying stream / data carrier.
[0009] The target file package is cached using memory buffers with different storage capacities predefined based on the theme and date. The getByteArray function returns the effective length of the multiplexed array output stream and the underlying stream to determine whether the upload rule of accommodating the number of page files in the memory buffer is met and triggers the batch upload mechanism.
[0010] The target file package is uploaded to cloud storage, the mapping relationship between the page file and the target file package is saved, and the memory buffer is cleared after the upload to ensure cache reuse in the next time.
[0011] Preferably, determining whether the upload rule of balancing the number of page files with the memory buffer is met and triggering the batch upload mechanism further includes:
[0012] Determine whether the number of page files stored in the memory buffer has reached the threshold for storing page files or whether the remaining cache space of the memory buffer has reached the remaining cache threshold;
[0013] When the number of page files stored in the memory buffer reaches a threshold, the underlying stream is triggered to upload to cloud storage;
[0014] Determine whether the remaining cache space of the memory buffer is less than the remaining cache threshold. If the remaining cache space of the memory buffer is less than the remaining cache threshold, trigger the upload operation of the underlying stream to cloud storage.
[0015] The upload operation of the underlying stream refers to the batch transmission of the target files to cloud storage, the remaining cache threshold is 1M, and the initial file package of a standard unit is less than 1M.
[0016] Preferably, the step of determining whether the upload rule of accommodating the number of page files is met and triggering the batch upload mechanism by returning the effective length of the multiplexed array output stream and the underlying stream through the getByteArray function includes:
[0017] Create a ShareArrayOutputStream class, call the getByteArray function to return a reused array, and obtain the capacity and effective length of the underlying stream;
[0018] The fullness of the target file package is determined by whether the value of the underlying stream's capacity minus its effective length in the memory buffer is less than 1M.
[0019] When the value of the underlying stream capacity minus the effective length of the underlying stream is less than 1M, the upload mechanism is triggered for the current target file packet to prevent memory overflow.
[0020] When the initial file package continues to be added to the memory buffer, it is determined whether the remaining cache space is sufficient to store an initial file package. If the remaining cache space is insufficient to store an initial file package, the upload mechanism is triggered for the current target file package. Herein, a target file package includes multiple initial file packages.
[0021] Preferably, obtaining the output stream of the target file package and the underlying output stream of the target file package from the mapping based on the topic and date further includes:
[0022] Retrieve the maximum cache space capacity / maximum effective length historically occupied in the memory buffer based on the topic and date;
[0023] If the currently occupied cache space exceeds the historical maximum cache space capacity, then the currently occupied cache space capacity is updated to the maximum capacity corresponding to the memory buffer, and written into the mapping for subsequent statistics; and / or,
[0024] If the current effective length of the underlying stream exceeds the historical maximum effective length, then the current effective length of the underlying stream is updated to the maximum effective length and written into the mapping for subsequent statistics.
[0025] Preferably, when performing a download operation based on a request to re-parse the e-commerce page, the specific steps of the download operation include:
[0026] Based on the date and target file name, query the database for the corresponding target file package name to determine whether the downloaded target file package exists in the server cache directory;
[0027] If the target file package does not exist, download the target file package from the cloud storage and save it to the cache directory. Decompress the target file package to obtain the downloaded initial file package and then continue to the next step.
[0028] If a downloaded initial file package exists, retrieve the initial file package based on the target file name, and determine whether the initial file package has been decompressed.
[0029] If the initial file package is not decompressed, the initial file package is decompressed. After decompression, the file API is called through the server cache directory to obtain the original page file and context file.
[0030] Once the initial file package has been decompressed, the original page file and context file are obtained by directly calling the file API through the server cache directory.
[0031] Preferably, the step of querying the corresponding target file package name from the database based on the date and target file name includes:
[0032] Select the corresponding database table based on the date, create a target request class, call the TableNameHandler interface, and override the dynamicTableName method. The overridden dynamicTableName method includes setting the date, performing a bitwise AND operation between the date and the 0xf mask to obtain the sequence index, and reading data from the database table.
[0033] The target file package name is obtained by querying the index table of the serial number in the database table based on the target file name.
[0034] The present invention also provides a cache-based file upload cost optimization device, comprising:
[0035] The request module is used to respond to requests for re-parsing of e-commerce pages and to invoke the system's listening function for file message events;
[0036] The detection module is used to obtain page file and context data when a file message is detected, wherein the context data is information related to the attributes of the page file;
[0037] The first packaging module is used to perform a first packaging operation on the obtained page file and the information related to the attributes of the page file to obtain an initial file package, and write the name of the initial file package into the multiplexed array output stream, and record the effective length of the underlying stream;
[0038] The secondary packaging module is used to summarize multiple initial file packages as Entry compression items and then perform a second packaging operation to reduce upload operations and obtain the target file package;
[0039] The judgment module is used to reuse memory buffers with different storage capacities based on the topic and date. It uses the getByteArray function to return the effective length of the reused array output stream and the underlying stream to determine whether the upload rule of balancing the number of page files with the memory buffer is met and to trigger the upload mechanism.
[0040] The upload module is used to upload the target file package to cloud storage and save the mapping relationship between the page file and the target file package.
[0041] Preferably, the system also includes a download module for performing a download operation based on a request to re-parse the e-commerce page. The specific steps of the download operation include:
[0042] Based on the date and target file name, query the database for the corresponding target file package name to determine whether the downloaded target file package exists in the server cache directory;
[0043] If the target file package does not exist, download the target file package from the cloud storage and save it to the cache directory. Decompress the target file package to obtain the downloaded initial file package and then continue to the next step.
[0044] If a downloaded initial file package exists, retrieve the initial file package based on the target file name, and determine whether the initial file package has been decompressed.
[0045] If the initial file package is not decompressed, the initial file package is decompressed. After decompression, the file API is called through the server cache directory to obtain the original page file and context file.
[0046] Once the initial file package has been decompressed, the original page file and context file are obtained by directly calling the file API through the server cache directory.
[0047] This invention provides an electronic device, comprising:
[0048] The memory is used to store the processing program;
[0049] The processor, when executing the processing program, implements the cache-based file upload cost optimization method as described in the embodiments of the present invention.
[0050] The present invention provides a readable storage medium, characterized in that the readable storage medium stores a processing program, which, when executed by a processor, implements the cache-based file upload cost optimization method as described in the embodiments of the present invention.
[0051] Compared with the prior art, the present invention has the following beneficial effects:
[0052] 1. Improve upload efficiency: By combining multiple initial file packages and performing a second packaging operation, the number of upload operations is reduced, thereby improving upload efficiency.
[0053] 2. Reduce memory usage: Use a memory buffer based on theme and date to cache the target file package, and use the getByteArray function to determine whether the upload rules are met. This can effectively manage memory usage and prevent memory overflow.
[0054] 3. Optimize resource utilization: By predefining memory buffers with different storage capacities, the system can dynamically adjust the caching strategy according to actual needs, thereby improving resource utilization efficiency.
[0055] 4. Reduce cloud storage costs: By using a batch upload mechanism, the number of uploads to cloud storage is reduced, thereby lowering the cost of using cloud storage.
[0056] 5. Enhanced system performance: By optimizing the upload process, the system's file upload processing time has been reduced, thus improving the overall system performance.
[0057] 6. Improve user experience: By caching and optimizing the upload process, users can complete file uploads faster, thus improving the user experience.
[0058] 7. Support for download operations: The above methods also include performing download operations based on requests re-parsed from e-commerce pages. By caching downloaded files, download time can be reduced and download efficiency improved.
[0059] 8. Flexible cache management: Through topic- and date-based cache management strategies, the system can dynamically adjust the cache strategy according to different needs, improving the flexibility of cache management.
[0060] 9. Effective data storage: Saving the mapping relationship between page files and target file packages to a database such as MySQL is also a type of cloud storage. Saving the target file packages to cloud storage such as OSS object storage service is for the purpose of storing files. This can effectively manage data storage and improve data accessibility and reliability.
[0061] 10. Security and Scalability: The above methods include data encryption and secure transmission steps, ensuring the security of data transmission. At the same time, by dynamically adjusting caching strategies and upload rules, the system has good scalability. Attached Figure Description
[0062] Figure 1 This is an example diagram illustrating the file upload cost method in the background technology of this invention;
[0063] Figure 2 This is a schematic diagram illustrating the principle of the file upload cost optimization method based on caching in this embodiment of the invention.
[0064] Figure 3 This is a flowchart illustrating the file upload cost optimization method based on caching in an embodiment of the present invention.
[0065] Figure 4 This is a scenario example diagram illustrating the file upload cost optimization method based on caching in an embodiment of the present invention;
[0066] Figure 5 This is a flowchart illustrating the download operation in an embodiment of the present invention;
[0067] Figure 6 This is an example diagram of a common buffering method in an embodiment of the present invention;
[0068] Figure 7 This is an example diagram illustrating the application of a memory buffer buffering method in an embodiment of the present invention;
[0069] Figure 8 This is an example diagram illustrating the structure of each memory buffer in an embodiment of the present invention;
[0070] Figure 9 This is an example diagram illustrating array copying using toByteArray() in an embodiment of the present invention;
[0071] Figure 10 This is an example diagram showing how getByteArray() returns a raw byte array in an embodiment of the present invention;
[0072] Figure 11 This is an example diagram of a memory buffer clearing operation in an embodiment of the present invention;
[0073] Figure 12 This is an example diagram illustrating the mapping relationship between files and target file packages in an embodiment of the present invention;
[0074] Figure 13 This is an example diagram illustrating a batch insertion operation on a specified table in an embodiment of the present invention. Detailed Implementation
[0075] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.
[0076] Example 1
[0077] The term "comprising" and its variations as used herein are open-ended inclusion, meaning "including but not limited to". The term "based on" means "at least partially based on". The term "one embodiment" means "at least one embodiment"; the term "another embodiment" means "at least one additional embodiment"; the term "some embodiments" means "at least some embodiments". Definitions of other terms will be given in the description below.
[0078] It should be noted that the concepts of "first" and "second" mentioned in this application are only used to distinguish different devices, modules or units, and are not used to limit the order of functions performed by these devices, modules or units or their interdependencies.
[0079] It should be noted that the terms "a" and "a plurality of" used in this application disclosure are illustrative rather than restrictive, and those skilled in the art should understand that, unless otherwise expressly indicated in the context, they should be understood as "one or more".
[0080] In this embodiment, 1 billion files need to be uploaded each month, which is too large a number. Calling the upload interface for each file would be very costly. To address this cost issue, such as... Figure 1-5 As shown, this invention provides a cache-based method for optimizing file upload costs. It employs a method of uploading multiple files in a package, and persists the mapping between filenames and package names to a database. When a file needs to be downloaded, it first queries the local cache. If the file is not found there, it queries the database based on the date and filename to obtain the package name. After downloading from cloud storage, it is decompressed to retrieve the specified file. The specific steps of the method provided in this embodiment are as follows:
[0081] S1: In response to the request to re-parse the e-commerce page, the system's file message event listening function is invoked; in this embodiment, the first step is to listen to the message queue to see if there is a page file: 1.1 Listen to the specified topic in Kafka, which is generally an object used in the message queue to distinguish different business storage, that is, the obtained page will be placed in Kafka for caching; 1.2 If there is a file message, obtain the page file and context such as site, business, and date information.
[0082] S2: When a file message is detected, obtain the page file and context data. The context data is information related to the attributes of the page file; for example, extracting page data to obtain files from various dimensions, such as different business dimensions, different site dimensions, and obtaining publicly available pages from e-commerce platforms, such as Amazon. The parsing speedup in this embodiment involves obtaining valid numbers on the page, such as titles, prices, or other numerical information. If these cannot be parsed, some analysis is performed.
[0083] S3: Perform a first packaging operation on the obtained page file and information related to the page file attributes to obtain initial file package data. The initial file package data includes the initial file package name written to the multiplexed array output stream and the effective length information of the underlying stream. This can be understood as follows: In this embodiment, step three involves packaging the file and context into a small tarball: 3.1 Obtain the multiplexed array output stream from the thread context; if it exists, reset it; otherwise, create a new one. 3.2 Package the page file and context (context.json) as two Entries into a small tarball, with the package name being the file ID, and write it to the multiplexed output stream, recording the effective length. In this embodiment, the initial file package is the small tarball mentioned below, and the target file package is the large tarball mentioned below. tar is a compression and packaging tool on Unix and Unix-like systems that can merge multiple files into one file; the packaged file also has the .tar extension.
[0084] S4: Based on the topic and date, obtain the output stream and underlying output stream of the target file package from the mapping. Then, by adding page files to the output stream of the target file package, aggregate the multiple initial file packages as compression items and perform a second packaging operation, thereby writing the data to the underlying stream / data carrier. This can be understood as follows: In this embodiment, step four involves placing a small tarball as an Entry into a large tarball. The specific steps are as follows: 4.1 Obtain the output stream and underlying output stream of the large tarball from the mapping based on the topic and date; 4.2 Place the small tarball as an Entry into the large tarball; 4.3 Obtain the file list from the mapping based on the topic and date, and add the new file to the file list.
[0085] S5: Based on the predefined memory buffers with different storage capacities for the topic and date, the target file package is cached. The getByteArray function returns the effective length of the multiplexed array output stream and the underlying stream to determine whether the upload rule of accommodating the number of page files in the memory buffer is met and triggers the batch upload mechanism. Specifically, the determination process is as follows: Determine whether the number of page files stored in the memory buffer reaches the threshold for storing page files or whether the remaining cache space of the memory buffer reaches the remaining cache threshold. When the number of page files stored in the memory buffer reaches the threshold, the underlying stream upload operation is triggered to the cloud storage. Determine whether the remaining cache space of the memory buffer is less than the remaining cache threshold. When the remaining cache space of the memory buffer is less than the remaining cache threshold, the underlying stream upload operation is triggered to the cloud storage. The underlying stream upload operation refers to batch transmitting the target files to the cloud storage. The remaining cache threshold is 1M, and the initial file package of a standard unit is less than 1M. Cloud file storage, such as OSS (Object Storage Service), stores the mapping relationship between filenames and target file packages in its database. This can be understood as follows: Step 5 in this embodiment is to determine whether the number of files meets the standard, or whether the remaining buffer is less than 1M: 5.1 Whether the number of files reaches 220. If it does, proceed to step 6; otherwise, proceed to steps 1 and 2; 5.1.1 220 is a reference value. If it is too large, the large tar package will be too large, the upload time will be longer, the memory usage will also be larger, and the batch insertion into the database will also take longer; if it is too small, it will lead to too many upload API calls, increasing costs; 5.2 Whether the remaining buffer is less than 1M (a few hundred KB for a single file). If it is not, proceed to step 6; otherwise, proceed to steps 1 and 2; 5.2.1 1M is a reference value. Generally, files are a few hundred KB, leaving a certain margin.
[0086] S6: Upload the target file package to cloud storage, save the mapping relationship between the page file and the target file package, and clear the memory buffer after uploading to ensure cache reuse in the next instance. This can be understood as follows: In this embodiment, step six is uploading to cloud storage and saving the mapping relationship between the file and the large tar package: 6.1 Upload the large tar package to the specified directory in cloud storage ( / / <biz> / <date>6.2 Map the file list to the large tar package name and save it to the database; 6.2.1 Each record contains a file name and the corresponding large tar package name; 6.2.2 Sixteen tables are created in the database, starting with file_package and ending with a numeric index from 0 to 15; 6.2.3 DaysTableNameHandler implements the TableNameHandler interface of MybatisPlus, and overrides the dynamicTableName method: get the current date, perform a bitwise AND operation with 0xf to get the index, form the table: file_package_<index>, and then write data to this table.
[0087] This embodiment also includes the following steps after uploading in step six: maximum value statistics and clearing the buffer: Based on the topic and date, obtain the historical maximum cache space capacity / maximum effective length in the memory buffer; if the current cache space capacity exceeds the historical maximum cache space capacity, update the current cache space capacity to the corresponding maximum capacity of the memory buffer and write it into the mapping for subsequent statistics; if the current effective length of the underlying stream exceeds the historical maximum effective length, update the current effective length to the maximum effective length and write it into the mapping for subsequent statistics; for example, obtain the previous maximum size based on topic+date, obtain the size of this upload, and if the current size exceeds the previous maximum size, use the current size as the standard and write it into the mapping for subsequent statistics; in this embodiment, after uploading, perform a clearing operation on the memory buffer, i.e., clear the file list and reset the byte output stream for the next file processing and upload.
[0088] S7: When performing a download operation based on a request re-parsed from an e-commerce page, such as Figure 5 As shown, the specific steps of the download operation include:
[0089] S71: Based on the date and target file name, query the database for the corresponding target file package name to determine if the downloaded target file package exists in the server cache directory. In this embodiment, the specific steps for querying the database for the corresponding target file package name (i.e., querying the large tar package name) based on the date and target file name include: selecting the corresponding database table based on the date, creating a target request class and calling the TableNameHandler interface, and overriding the dynamicTableName method. The overriding of the dynamicTableName method includes setting the date, performing a bitwise AND operation between the date and a 0xf mask to obtain the sequence index, and reading data from the database table (i.e., reading data from the table: file_package_<sequence index>). The database table name is prefixed with file_package + sequence index, for example, 9. Data is read from the table file_package9. The target file package name is obtained by querying the database table's sequence index based on the target file name, and the large tar package name is then queried from the table: file_package_<sequence index>. In this embodiment, when determining whether a large tar package exists on the server, the server cache directory is used to determine whether a large tar package exists using the file API. If it does not exist, the large tar package is downloaded from cloud storage, saved to the cache directory, and then decompressed. If it exists, the next step is to determine whether the small tar package has been decompressed.
[0090] S72: If the target file package does not exist, download the target file package from the cloud storage and save it to the cache directory. Decompress the target file package to obtain the downloaded initial file package and continue to the next step.
[0091] S73: If a downloaded initial file package exists, obtain the initial file package based on the target file name, and determine whether the initial file package has been decompressed. In this embodiment, when determining whether the small tar package has been decompressed, the small tar package file name is: task ID plus .tar suffix, server cache directory. Use the file API to determine whether the small tar package has been decompressed. If it has not been decompressed, decompress it. If it exists, proceed to the next step of obtaining the page and context file.
[0092] S74: If the initial file package is not decompressed, the initial file package is decompressed. After decompression, the file API is called through the server cache directory to obtain the original page file and context file.
[0093] S75: When the initial file package has been decompressed, the original page file and context file are obtained directly by calling the file API through the server cache directory. The specific steps for obtaining the page and context file in this embodiment include: accessing the server cache directory, using the file API, obtaining the page and context file; the page is a compressed .gz file, and to view the original page, it needs to be decompressed; the context file (context.json) contains the task ID and the date / topic.
[0094] The core of this embodiment lies in the caching of the target file package in step S5, where memory buffers with different storage capacities are predefined based on the topic and date. This predefined memory buffer, used based on the topic and date, is reused, reducing the overhead of memory creation and destruction, and improving performance. Currently, ordinary caching methods involve allocating and expanding memory, which incurs significant performance overhead. (See [link to previous document]). Figure 6 As shown; however, this application uses a new buffering method based on memory buffers: the buffer size is predefined according to the number of files and the size of a single file, see [link to relevant documentation]. Figure 7-8 As shown, when using getByteArray(), it returns a byte array instead of a copied object, greatly improving performance. In the JDK, ByteArrayOutputStream acts as a memory buffer. When file storage reaches a certain threshold, it needs to call toByteArray() to retrieve the file data, which uses array copying and returns a new byte array, significantly impacting performance and memory usage.
[0095] public synchronized byte[]toByteArray(){
[0096] return Arrays.copyOf(buf,count);}
[0097] See Figure 9 As shown, ByteArrayOutputStream, toByteArray() will allocate a new memory block of the same size and perform array copying.
[0098] After improvement:
[0099] A custom ShareArrayOutputStream class is used to retrieve file data via getByteArray(), eliminating array copying and resulting in high performance.
[0100] public byte[]getByteArray(){
[0101] return buf;
[0102] However, this brings a problem that there may still be some unprocessed data in the buf. To solve this problem, the effective length is returned by the size() method.
[0103] public synchronized int size(){
[0104] return count;}
[0105] Specifically, the method of determining whether the upload rule for balancing the memory buffer and the number of page files is satisfied and triggering the batch upload mechanism by returning the effective lengths of the reusable array output stream and the underlying stream through the getByteArray function includes: establishing a ShareArrayOutputStream class, calling the getByteArray function to obtain the reusable array, and getting the capacity of the underlying stream and the effective length of the underlying stream; judging whether the target file package is full by whether the value of the capacity of the underlying stream minus the effective length of the underlying stream in the memory buffer is less than 1M; when the value of the capacity of the underlying stream minus the effective length of the underlying stream is less than 1M, triggering the upload mechanism for the current target file package to prevent memory overflow; when continuing to add the initial file package in the memory buffer, judging whether the remaining cache space can store an initial file package, and when the remaining cache space cannot store an initial file package, triggering the upload mechanism for the current target file package; wherein, one target file package includes multiple initial file packages.
[0106] Those skilled in the art can understand the usage scenarios of the getByteArray function in this embodiment:
[0107] 1. Pack the page and context files, and getByteArray() returns a byte array; 2. Judge whether the large tar package is full: underlyingStream.getByteArray().length - underlyingStream.size() <1048576>, where length is the capacity and size() is the effective length; 3. Upload the large tar package, return a byte array through underlyingStream.getByteArray(), and then upload the underlyingStream.size() part.
[0108] ShareArrayOutputSteream, getByteArray() returns the original byte array without copying, and配合size to obtain the file data, see Figure 10 As shown, to reduce the overhead and memory overflow issues caused by expansion, when memory usage approaches capacity, even if the number of files has not yet been reached, the files are immediately packaged, uploaded, written to the database, and the memory is cleared. See [link / reference]. Figure 11 As shown. There are 20 million file uploads daily, requiring the database to store 20 million records. The database is partitioned by date. Since only about 15 days' worth of records need to be retained, 16 tables starting with "file_package" were created, from file_package_0 to file_package_15. See [link / reference]. Figure 12 As shown. DaysTableNameHandler implements TableNameHandler, overrides the dynamicTableName method to get the current date, then performs a bitwise operation with &0xf to obtain the index of the file_package series table. Then, MybatisPlus performs batch insert operations on the specified table. See [link to relevant documentation]. Figure 13 As shown.
[0109] Example 2
[0110] Based on the same inventive concept, the present invention provides a cache-based file upload cost optimization device, comprising:
[0111] The request module is used to respond to requests for re-parsing of e-commerce pages and to invoke the system's listening function for file message events;
[0112] The detection module is used to obtain page file and context data when a file message is detected, wherein the context data is information related to the attributes of the page file;
[0113] The first packaging module is used to perform a first packaging operation on the obtained page file and the information related to the attributes of the page file to obtain an initial file package, and write the name of the initial file package into the multiplexed array output stream, and record the effective length of the underlying stream;
[0114] The secondary packaging module is used to summarize multiple initial file packages as Entry compression items and then perform a second packaging operation to reduce upload operations and obtain the target file package;
[0115] The judgment module is used to reuse memory buffers with different storage capacities based on the topic and date. It uses the getByteArray function to return the effective length of the reused array output stream and the underlying stream to determine whether the upload rule of balancing the number of page files with the memory buffer is met and to trigger the upload mechanism.
[0116] The upload module is used to upload the target file package to cloud storage and save the mapping relationship between the page file and the target file package.
[0117] The download module is used to perform download operations based on requests for re-parsing the e-commerce page. The specific steps of the download operation include:
[0118] Based on the date and target file name, query the database for the corresponding target file package name to determine whether the downloaded target file package exists in the server cache directory;
[0119] If the target file package does not exist, download the target file package from the cloud storage and save it to the cache directory. Decompress the target file package to obtain the downloaded initial file package and then continue to the next step.
[0120] If a downloaded initial file package exists, retrieve the initial file package based on the target file name, and determine whether the initial file package has been decompressed.
[0121] If the initial file package is not decompressed, the initial file package is decompressed. After decompression, the file API is called through the server cache directory to obtain the original page file and context file.
[0122] Once the initial file package has been decompressed, the original page file and context file are obtained by directly calling the file API through the server cache directory.
[0123] The principles and implementation methods of the above modules are as described in Embodiment 1, and will not be repeated here.
[0124] Example 3
[0125] In some embodiments of this application, an electronic device is also provided. This electronic device includes a memory and a processor, wherein the memory stores a processing program, and the processor executes the processing program according to instructions. When the processor executes the processing program, the cache-based file upload cost optimization method described in the foregoing embodiments is implemented.
[0126] In some embodiments of this application, a readable storage medium is also provided, which can be a non-volatile readable storage medium or a volatile readable storage medium. The readable storage medium stores instructions that, when executed on a computer, cause an electronic device containing such a readable storage medium to perform the aforementioned cache-based file upload cost optimization method.
[0127] It is understood that, for the aforementioned cache-based file upload cost optimization methods, if they are all implemented as software functional modules and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this invention, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of the various embodiments of this invention. The aforementioned storage medium includes: USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, optical disks, and other media capable of storing program code.
[0128] Computer-readable storage media may include data signals propagated in baseband or as part of a carrier wave, carrying readable program code. Such propagated data signals may take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. A readable storage medium may also be any readable medium other than a readable storage medium that can transmit, propagate, or transfer a program for use by or in connection with an instruction execution system, apparatus, or device. The program code contained on the readable storage medium may be transmitted using any suitable medium, including but not limited to wireless, wired, optical fiber, RF, etc., or any suitable combination thereof.
[0129] The program code for executing the technical solutions disclosed in this application can be written in any combination of one or more programming languages. These programming languages include object-oriented programming languages—such as Java and C++—and conventional procedural programming languages—such as C or similar languages. The program code can be executed entirely on the user's computing device, partially on the user's computing device, as a standalone software package, partially on the user's computing device and partially on a remote computing device, or entirely on a remote computing device or server. In cases involving remote computing devices, the remote computing device can be connected to the user's computing device via any type of network, including a local area network (LAN) or a wide area network (WAN), or it can be connected to an external computing device (e.g., via the Internet using an Internet service provider).
[0130] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the various embodiments of the present invention.< / date> < / biz>
Claims
1. A method for optimizing file upload costs based on cache, characterized in that: include: In response to the request for re-parsing the e-commerce page, the system's monitoring function for file message events is called; When a file message is detected, a page file and context data are obtained, wherein the context data is information related to the attributes of the page file; Performing a first packaging operation on the acquired page file and information related to the page file attributes to obtain initial file package data, wherein the initial file package data includes an initial file package name written into the multiplex array output stream and effective length information of the underlying stream; Based on the subject and date, the output stream of the target file package and the underlying output stream of the target file package are obtained from the mapping, and a plurality of the initial file packages are aggregated as compression items by adding a page file to the output stream of the target file package to perform a second packaging operation, so that the data can be written to the underlying stream / data carrier; Based on the subject and date, a memory buffer with different storage capacity spaces is predefined to cache the target file package, and the getByteArray function is used to return the effective length of the multiplexed array output stream and the underlying stream to determine whether the upload rule of the memory buffer taking into account the number of page files is met and trigger the batch upload mechanism; The target file package is uploaded to the cloud storage, the mapping relationship between the page file and the target file package is saved, and after uploading, the memory buffer is cleared to ensure the next cache reuse.
2. The method for optimizing file upload cost based on cache as claimed in claim 1, characterized in that: Determining whether the upload rule of the memory buffer taking into account the number of page files is satisfied and triggering the batch upload mechanism further includes: Determine whether the number of page files stored in the memory buffer reaches a threshold value of the number of page files that can be stored or whether the remaining cache space of the memory buffer reaches a remaining cache threshold value; When the number of page files stored in the memory buffer reaches a threshold, triggering an upload operation of the underlying stream to the cloud storage; Determine whether the remaining cache space of the memory buffer is less than the remaining cache threshold, and when the remaining cache space of the memory buffer is less than the remaining cache threshold, trigger an upload operation of the underlying stream to the cloud storage; The upload operation of the underlying stream refers to transferring the target files in batches to the cloud storage, the remaining cache threshold is 1M, and the initial file package of a standard unit is less than 1M.
3. The method for optimizing file upload cost based on cache as claimed in claim 1, characterized in that: The method of returning the effective length of the multiplexed array output stream and the underlying stream through the getByteArray function to determine whether the upload rule of the memory buffer taking into account the number of page files is met and triggering the batch upload mechanism includes: Create a SharedArrayOutputStream class, call the getByteArray function to return the multiplexed array, and get the capacity and effective length of the underlying stream; Whether the target file package is full is determined by determining whether the value of the capacity of the underlying stream minus the effective length of the underlying stream in the memory buffer is less than 1M; When the value of the capacity of the underlying stream minus the effective length of the underlying stream is less than 1M, the upload mechanism is triggered for the current target file package to prevent memory overflow; When the memory buffer continues to increase the initial file package, determine whether the remaining cache space can store an initial file package. When the remaining cache space cannot store an initial file package, trigger the upload mechanism for the current target file package; wherein, one target file package includes multiple initial file packages.
4. The method for optimizing file upload cost based on cache as claimed in claim 1, characterized in that: The output stream of the target file package is obtained from the mapping based on the subject and date, and the underlying output stream of the target file package further includes: Obtain the maximum cache space capacity / maximum effective length of the history occupied in the memory buffer according to the subject and date; If the currently occupied cache space capacity exceeds the maximum cache space capacity occupied in history, the currently occupied cache space capacity is updated to the maximum capacity corresponding to the memory buffer, and written into the mapping for subsequent statistics; and / or, If the effective length of the current underlying stream exceeds the historical maximum effective length, the effective length of the current underlying stream is updated to the maximum effective length and written into the mapping for subsequent statistics.
5. The method for optimizing file upload cost based on cache as claimed in claim 1, characterized in that: The download operation is also performed according to the request for re-parsing the e-commerce page, and the specific steps of the download operation include: According to the date and target file name, the corresponding target file package name is queried from the database to determine whether the downloaded target file package exists in the server cache directory; If the downloaded target file package does not exist, the target file package is downloaded from the cloud storage and saved to the cache directory, the target file package is decompressed, and the downloaded initial file package is obtained before continuing to the next step; If there is an initial file package that has been downloaded, obtain the initial file package according to the target file name, and determine whether the initial file package has been decompressed; When the initial file package is not decompressed, the initial file package is decompressed, and after decompression, a file API is called through a server cache directory to obtain the original page file and context file; When the decompression operation has been performed on the initial file package, the file API is directly called through the server cache directory to obtain the original page file and context file.
6. The method for optimizing file upload cost based on cache as claimed in claim 5, characterized in that: The querying of the corresponding target file package name from the database according to the date and the target file name includes: Select the corresponding database table according to the date, create a target request class to call the TableNameHandler interface, and rewrite the dynamicTableName method. The rewriting of the dynamicTableName method includes setting the date, performing an AND operation on the date and the 0xf mask to obtain a serial number index, and reading data from the database table; The target file package name is obtained by querying the serial number index table in the database table according to the target file name.
7. A cache-based file upload cost optimization device, characterized in that: include: The request module is used to respond to the request for re-parsing the e-commerce page and call the system's monitoring function for file message events; A detection module, for acquiring a page file and context data when a file message is detected, wherein the context data is information related to the attributes of the page file; A primary packaging module, used for performing a first packaging operation on the acquired page file and information related to the page file attributes to obtain an initial file package, and writing the initial file package name into the multiplexing array output stream, and recording the effective length of the underlying stream; A secondary packaging module, used to aggregate the multiple initial file packages as Entry compression items and then perform a secondary packaging operation to reduce upload operations, thereby obtaining a target file package; The judgment module is used to predefine memory buffers with different storage capacity spaces based on the subject and date for reuse, and return the effective length of the reuse array output stream and the underlying stream through the getByteArray function to judge whether the upload rule of the memory buffer taking into account the number of page files is met and trigger the upload mechanism; The upload module is used to upload the target file package to the cloud storage and save the mapping relationship between the page file and the target file package.
8. The cache-based file upload cost optimization device according to claim 7, characterized in that: It also includes a download module, which is used to perform a download operation according to the request for re-parsing the e-commerce page. The specific steps of the download operation include: According to the date and target file name, the corresponding target file package name is queried from the database to determine whether the downloaded target file package exists in the server cache directory; If the downloaded target file package does not exist, the target file package is downloaded from the cloud storage and saved to the cache directory, the target file package is decompressed, and the downloaded initial file package is obtained before continuing to the next step; If there is an initial file package that has been downloaded, obtain the initial file package according to the target file name, and determine whether the initial file package has been decompressed; When the initial file package is not decompressed, the initial file package is decompressed, and after decompression, a file API is called through a server cache directory to obtain the original page file and context file; When the decompression operation has been performed on the initial file package, the file API is directly called through the server cache directory to obtain the original page file and context file.
9. An electronic device, characterized in that: include: A memory, the memory being used to store a processing program; A processor, wherein when executing the processing program, the processor implements the cache-based file upload cost optimization method as described in any one of claims 1 to 7.
10. A readable storage medium, characterized in that: The readable storage medium stores a processing program, and when the processing program is executed by the processor, the cache-based file upload cost optimization method according to any one of claims 1 to 7 is implemented.
Citation Information
Patent Citations
Self-adaptive data synchronization system and method and storage medium
CN115934845A
Cloud object storage and versioning system
US20210389883A1
Cited By
Multi-level cache and file batch downloading method and system and storage medium
CN120201089A
Downloading progress monitoring method bypassing pipeline buffer and application thereof
CN122261946A
