File uploading method and file processing system

Abstract

The application provides a file uploading method and a file processing system, and relates to the technical field of data transmission. The method comprises the following steps: a front-end device acquires a resource file to be uploaded input by a user, calls a target distribution package based on a preset interface, the target distribution package comprises a portable compilation file, the portable compilation file is an executable file obtained by compiling and packaging a target logic code, the portable compilation file is used to generate a hash value, the target logic code is written based on a target programming language, the front-end device runs the target distribution package to obtain the hash value of the resource file, and the front-end device uploads the resource file to a file server according to the hash value of the resource file. The application introduces the portable compilation file written based on the target programming language, replaces the traditional hash calculation mode with the native compilation code execution with better performance, significantly improves the hash value generation efficiency of the large-volume resource file, and further improves the file uploading efficiency.

Description

Technical Field

[0001] The present application relates to the technical field of data transmission, and in particular to a file uploading method and a file processing system. Background Art

[0002] A Package File (PAK file) is a format used to package resources in Unreal Engine (UE). Based on the powerful rendering capabilities provided by the UE engine, it supports hot updates of resources through PAK files, enabling content updates without restarting the application.

[0003] In digital twin scenarios, efficient management of PAK files is crucial for dynamic scene loading and hot updates. Currently, front-ends typically use the JavaScript-based spark-md5 library to calculate file hashes for file uploads. However, as PAK files continue to grow in size, the time required to calculate hash values ​​increases significantly, severely impacting file upload efficiency. Summary of the Invention

[0004] The purpose of this application is to provide a file uploading method and a file processing system to address the deficiencies in the above-mentioned prior art, so as to solve the problem in the prior art that as the size of PAK files continues to increase, the time consumption of hash value calculation increases significantly, which seriously affects the efficiency of file uploading.

[0005] To achieve the above objectives, the technical solutions adopted in the embodiments of the present application are as follows:

[0006] In a first aspect, an embodiment of the present application provides a file uploading method, which is applied to a file processing system, wherein the file processing system includes a front-end device and a server, and the method includes:

[0007] The front-end device obtains a resource file to be uploaded input by a user, wherein the resource file includes model information of a model to be added;

[0008] The front-end device calls a target distribution package based on a preset interface, wherein the target distribution package includes a portable compiled file, which is an executable file obtained by compiling and packaging a target logic code, and the portable compiled file is used to generate a hash value, and the target logic code is written in a target programming language;

[0009] The front-end device runs the target distribution package to obtain a hash value of the resource file;

[0010] The front-end device uploads the resource file to the file server according to the hash value of the resource file.

[0011] As a possible implementation, the target distribution package is generated through the following process:

[0012] Acquire the target logic code written based on the target programming language;

[0013] Compiling and packaging the target logic code using a preset construction tool to obtain the portable compiled file;

[0014] The target distribution package is obtained by replacing an original hash generation file in the original distribution package with the portable compiled file, wherein the original hash generation file is generated based on an original programming language, and the hash value operation efficiency of the target programming language is higher than the hash value operation efficiency of the original programming language.

[0015] As a possible implementation, the method further includes:

[0016] A typed array file is created and loaded into the target distribution package, wherein the typed array file is used to convert the data format of the resource file.

[0017] As a possible implementation, running the target distribution package to obtain the hash value of the resource file includes:

[0018] Running the typed array file to obtain a typed array instance, and converting the data format of the resource file based on the typed array instance to obtain a processed resource file;

[0019] The portable compiled file is run, and a hash calculation is performed on the processed resource file to obtain a hash value of the processed resource file.

[0020] As a possible implementation manner, running the portable compiled file and performing hash calculation on the processed resource file to obtain the hash value of the processed resource file includes:

[0021] Performing data slicing processing on the processed resource file according to a preset data block size to obtain multiple data blocks;

[0022] Hash calculation is performed on each of the data blocks in sequence based on the fragmentation order to obtain a hash value of the processed resource file.

[0023] As a possible implementation, the method further includes:

[0024] If the size of the data block does not meet the preset data block size, the data block is padded according to a preset padding strategy.

[0025] As a possible implementation manner, uploading the resource file to the file server according to the hash value of the resource file includes:

[0026] Determining whether there is a historical resource file having the same hash value as the resource file in the file server;

[0027] If not, the resource file is fragmented according to a preset fragment size to obtain a plurality of resource fragments, and each resource fragment is uploaded to the file server in sequence according to the fragmentation order.

[0028] As a possible implementation, the method further includes:

[0029] The server determines the current operating system and reads the target resolution address corresponding to the current operating system from a preset database;

[0030] The server moves the resource file uploaded to the file server to the target resolution address corresponding to the current operating system;

[0031] The server reads the resource file from the target resolution address, and uses the resolution service corresponding to the current operating system to parse the resource file to obtain the model information of the model to be added.

[0032] As a possible implementation, before reading the target resolution address corresponding to the current operating system from the preset database, the method further includes:

[0033] Compile and generate the corresponding parsing service installation package for each operating system based on the preset programming language, and deploy the corresponding parsing service installation package for each operating system in each operating system;

[0034] The parsed address corresponding to each operating system is set, and the parsed address corresponding to each operating system is stored in the preset database.

[0035] As a possible implementation, after parsing the resource file using the parsing service corresponding to the current operating system to obtain the model information of the model to be added, the method further includes:

[0036] The model to be added is added to the stream rendering scene according to the model information, wherein the model information includes: model type, model identifier and model name.

[0037] In a second aspect, an embodiment of the present application provides a file processing system, the file processing system comprising a front-end device and a server;

[0038] The front-end device is used to execute the steps performed by the front-end device in the file uploading method according to any one of the first aspects above;

[0039] The server is used to execute the steps performed by the server in the file upload method described in any one of the first aspects above.

[0040] In a third aspect, an embodiment of the present application provides a front-end device comprising: a processor, a storage medium and a bus, wherein the storage medium stores machine-readable instructions executable by the processor. When the electronic device is running, the processor communicates with the storage medium through the bus, and the processor executes the machine-readable instructions to execute the steps performed by the front-end device in the file upload method as described in any one of the first aspects above.

[0041] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the steps of the file uploading method as described in any one of the above-mentioned first aspects are executed.

[0042] According to the file upload method and file processing system of the embodiment of the present application, the front-end device obtains the resource file to be uploaded input by the user, and calls the target distribution package based on the preset interface. The target distribution package includes a portable compiled file. The portable compiled file is an executable file obtained by compiling and packaging the target logic code, and the portable compiled file is used to generate a hash value. The target logic code is written based on the target programming language. By running the target distribution package, the hash value of the resource file is obtained, and then the resource file is uploaded to the file server according to the hash value of the resource file. According to the embodiment of the present application, by introducing a portable compiled file generated by compiling based on the target programming language, the traditional hash calculation method is replaced with a native compiled code execution with better performance. By efficiently running the target distribution package in the front-end device, the hash value of the resource file is quickly calculated, which effectively solves the problem in the prior art that the hash value calculation time increases significantly due to the continuous increase in the size of the PAK file, thereby affecting the file upload efficiency. Compared with the traditional JavaScript implementation method, the logic code written based on the target programming language rust has higher execution efficiency and lower memory usage, which greatly improves the hash calculation speed of large files. At the same time, this application supports fragmented processing and instant transmission judgment mechanism, which significantly reduces unnecessary upload operations while ensuring accuracy and completeness, improves the overall file upload response speed and user experience, and has good cross-platform compatibility and scalability. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the following is a brief introduction to the drawings required for use in the embodiments. It should be understood that the following drawings only show certain embodiments of the present application and therefore should not be regarded as limiting the scope. For ordinary technicians in this field, other relevant drawings can be obtained based on these drawings without creative work.

[0044] Figure 1 A schematic diagram of the architecture of a file processing system provided in an embodiment of the present application is shown;

[0045] Figure 2 A schematic diagram showing a process of a file uploading method provided in an embodiment of the present application is shown;

[0046] Figure 3 A schematic diagram showing a flow chart of a method for generating a target distribution package provided by an embodiment of the present application is shown;

[0047] Figure 4 A schematic diagram showing a flow chart of another method for generating a target distribution package provided by an embodiment of the present application is shown;

[0048] Figure 5 A schematic diagram of a process flow of a file parsing method provided in an embodiment of the present application is shown;

[0049] Figure 6 A schematic diagram showing a flow chart of a file processing method provided in an embodiment of the present application is shown;

[0050] Figure 7 A schematic structural diagram of a front-end device provided in an embodiment of the present application is shown. DETAILED DESCRIPTION

[0051] In order to make the purpose, technical solutions and advantages of the embodiments of the present application clearer, the technical solutions in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. It should be understood that the drawings in the present application only serve the purpose of illustration and description and are not used to limit the scope of protection of the present application. In addition, it should be understood that the schematic drawings are not drawn to scale. The flowcharts used in this application illustrate the operations implemented according to some embodiments of the present application. It should be understood that the operations of the flowcharts can be implemented out of sequence, and steps without logical context can be reversed or implemented simultaneously. In addition, those skilled in the art, under the guidance of the contents of this application, can add one or more other operations to the flowchart, or remove one or more operations from the flowchart.

[0052] In addition, the described embodiments are only a part of the embodiments of the present application, rather than all of the embodiments. The components of the embodiments of the present application generally described and shown in the drawings here can be arranged and designed in various configurations. Therefore, the following detailed description of the embodiments of the present application provided in the drawings is not intended to limit the scope of the claimed application, but merely represents selected embodiments of the present application. Based on the embodiments of the present application, all other embodiments obtained by those skilled in the art without making creative work are within the scope of protection of the present application.

[0053] It should be noted that the term "comprising" will be used in the embodiments of the present application to indicate the existence of the features declared thereafter, but does not exclude the addition of other features.

[0054] When managing PAK files, the PAK files need to be uploaded to the file server. In some business scenarios, the size of the PAK files may be very large, which makes the file upload process very time-consuming, especially the two operations of calculating the hash value of the PAK file and uploading the file content. For example, a PAK file of size 47G takes about 30 minutes to upload, and about 12 to 13 minutes of it is spent on calculating the hash value of the PAK file. To address this problem, this application proposes a file upload method that optimizes the operation of calculating the file hash value during the file upload process. Specifically, by migrating the core logic of the hash calculation from the traditional JavaScript library to the WebAssembly module written based on rust, the performance advantages of the compiled language are fully utilized, and the hash calculation efficiency of large-volume PAK files is significantly improved, thereby improving the overall performance of file upload and user experience.

[0055] Figure 1 FIG2 shows a schematic diagram of the architecture of a file processing system provided by an embodiment of the present application. Figure 1 As shown, the file processing system includes a front-end device and a server. The front-end device provides a user interface for uploading resource files to the file server. The server, as the back-end, retrieves and parses the uploaded resource files from the file server and stores the parsed model information in a database. This allows the model to be dynamically added to the stream rendering scene based on the model information.

[0056] Optionally, the user can input a resource file to be uploaded based on the user interface provided by the front-end device. The resource file is also known as a PAK file, which includes model information of the model to be added. The front-end device obtains the resource file to be uploaded input by the user and calls a pre-generated target distribution package based on a preset interface. The target distribution package includes a portable compiled file for generating a hash value. The front-end device obtains the hash value of the uploaded resource file by running the target distribution package and then uploads the resource file to the file server based on the hash value of the resource file.

[0057] Optionally, the resource file (PAK file) is a format used to package game resources in the UE engine and supports hot update functions. The UE engine itself supports the packaging and parsing of PAK files. However, since the UE engine is currently generally based on the Windows system and only supports manual input of command line parsing, it cannot achieve platform automation, and the back-end service is generally based on the Linux system, it is impossible to directly use the back-end service to call the UE engine for parsing. In this regard, the present application configures different parsing addresses for different operating systems, that is, the parsing paths of the parsing services corresponding to different operating systems are different. On this basis, after uploading the resource file to the file server, the server first determines the type of the current operating system, such as whether the current operating system is a Windows system or a Linux system, and moves the resource file uploaded to the file server to the parsing address corresponding to the current operating system, so as to read the resource file from the parsing address according to the corresponding parsing path and parse to obtain the model information.

[0058] Furthermore, after the server parses and obtains the model information, it can store the model information in a database, then open the UE stream rendering scene editor, load the model information from the database, and complete the dynamic addition of the model to be added in the stream rendering scene based on the model information.

[0059] Based on this, the file processing system provided in the embodiment of the present application solves the time-consuming problem of file hash value calculation during file upload and the low efficiency of manually entering command line parsing model information. By reducing the hash value calculation time and providing automated parsing methods for different operating systems, the file upload and parsing efficiency is significantly improved.

[0060] The following combines the above Figure 1 The content described in the file processing system shown is a detailed description of the file upload method provided in the embodiment of the present application.

[0061] Figure 2 The flowchart of a file uploading method provided by the embodiment of the present application is shown. The execution subject of the method is the front-end device in the file processing system. Figure 2 As shown, the method specifically includes the following steps:

[0062] S201: Obtain the resource file to be uploaded input by the user.

[0063] Optionally, the front-end device provides a user operation interface, based on which the user can select the resource file to be uploaded. The resource file is a stream rendering hot update package (that is, a PAK file with a .pak extension). The resource file includes model information of the model to be added, allowing the resource file to be updated without restarting the entire stream rendering application.

[0064] S202: Call the target distribution package based on a preset interface.

[0065] Optionally, the target distribution package includes a portable compiled file, which is an executable file obtained by compiling and packaging the target logic code, and the portable compiled file is used to generate a hash value, and the target logic code is written based on the target programming language.

[0066] Optionally, the JavaScript library spark-md5 is currently commonly used to calculate file hash values. In this application, in order to solve the problem of time-consuming hash value calculation, a portable compilation target (WebAssembly, wasm) is used for performance optimization. Wasm can be understood as a compiler target platform that can be run directly by a browser. It has high performance characteristics, is close to the speed of native code, and can call each other with JavaScript. On this basis, this application only replaces the original hash generation file corresponding to the spark-md5 library used to calculate hash values ​​in the original JavaScript distribution package with a portable compilation file (.wasm file). Since the .wasm file is a compact binary format, it is small in size, loads quickly, and saves bandwidth, which can effectively improve the efficiency of hash value calculation.

[0067] Optionally, refer to Figure 3 As shown, in this application, the target programming language rust is used to refer to the class library spark-md5 to write the relevant business code for hash value calculation, obtain the target logic code, and then use the construction tool wasm-pack to edit and package the target logic code to obtain a portable compiled file, which is an executable file. The portable compiled file is also a .wasm file, and the tool wasm-bindgen is used to implement type-safe interaction between rust and JavaScript, so that the .wasm file is used to replace the original hash generation file corresponding to the original JavaScript class library spark-md5 to obtain the target distribution package (that is, npm package), that is, an npm package containing the .wasm file is generated, so that the front-end device can run high-performance code.

[0068] Optionally, the application only replaces the original hash generation file corresponding to the class library spark-md5 in the original JavaScript distribution package for calculating the hash value with a portable compiled file (.wasm file), and the file uploading process is still implemented by using JavaScript, so the front-end device only calls the target distribution package through a preset interface rust api to calculate the hash value of the uploaded resource file by JavaScript, and the instantiation of the class library spark-md5 is replaced by loading the npm package containing the.wasm file in the target distribution package.

[0069] S203, running the target distribution package to obtain the hash value of the resource file.

[0070] Optionally, the portable compiled file (.wasm file) in the target distribution package is used to generate the hash value, the portable compiled file is an executable file generated by compiling and packaging the target logic code, and the target logic code is written based on the target programming language rust and the reference class library spark-md5, so the hash value calculation logic is similar to the hash value calculation logic of the class library spark-md5. On this basis, the front-end device can perform hash calculation on the resource file according to the hash value calculation logic defined in the portable compiled file (.wasm file) in the target distribution package by calling and running the target distribution package, so as to obtain the hash value of the resource file.

[0071] Optionally, the process of calculating the hash value by using the class library spark-md5 mainly includes object instantiation, file slicing, and loading each slice obtained by slicing into the instance until the hash value of the file is calculated after all the slices are loaded. Since the system memory is limited, when the file is relatively large, the system memory that can be allocated may not be enough to be loaded at one time, so the file can be divided into multiple parts and loaded one by one by slicing. Similarly, the portable compiled file (.wasm file) in the target distribution package also uses the above-mentioned hash value calculation logic of the class library spark-md5 to perform hash calculation on the resource file. Since several methods need to be called when calculating the hash value by using the class library spark-md5, such as the appendBinary method for loading the slice content and the end method for ending, the target distribution package also exposes similar appendBinary method and end method.

[0072] S204, uploading the resource file to the file server according to the hash value of the resource file.

[0073] Optionally, after obtaining the hash value of the resource file, the front-end device can request the back-end interface to verify whether there is a file with the same hash value as the resource file in the file server. If so, it means that the same resource file has been uploaded before, and the file can be uploaded instantly. It should be noted that "instant upload" here means that when a user uploads a file, it is detected that the file has been uploaded by other users. Therefore, there is no need to transfer the entire file again. Instead, the existing file resources are directly reused to achieve the effect of instant upload.

[0074] Optionally, if no file with the same hash value as the resource file exists on the file server, the resource file can be uploaded in pieces due to its large size. Specifically, the resource file can be split into pieces according to a certain size, for example, a 20GB resource file can be split into 300MB pieces to obtain multiple pieces, and each piece can be uploaded to the file server in sequence. After all pieces have been uploaded, the multiple pieces can be merged to obtain the original resource file.

[0075] Based on this, according to the file upload method of the embodiment of the present application, by introducing a portable compiled file generated by compiling based on the target programming language, the traditional hash calculation method is replaced with a native compiled code execution with better performance, and by efficiently running the target distribution package in the front-end device, the hash value of the resource file is quickly calculated, which effectively solves the problem in the prior art that the hash value calculation time increases significantly due to the increasing size of the PAK file, thereby affecting the efficiency of file upload. Compared with the traditional JavaScript implementation method, the logic code written in the target programming language rust has higher execution efficiency and lower memory usage, which greatly improves the hash calculation speed of large files. At the same time, the present application supports sharding processing and second-transfer judgment mechanism, which significantly reduces unnecessary upload operations while ensuring accuracy and completeness, improves the overall file upload response speed and user experience, and has good cross-platform compatibility and scalability.

[0076] Figure 4 FIG2 shows a flow chart of another method for generating a target distribution package provided by an embodiment of the present application. Figure 4 As shown, the target distribution package is generated through the following process:

[0077] S401: Obtain target logic code written in a target programming language.

[0078] Optionally, the target logic code is a new hash calculation logic code written in a high-performance target programming language, such as rust. The target logic code can implement a hash calculation function equivalent to that of the class library spark-md5, but with higher performance.

[0079] Alternatively, the original front-end JavaScript project uses the spark-md5 library to calculate file hash values, but JavaScript's execution efficiency is slow when processing large files, affecting the user experience. Therefore, this application uses the target programming language Rust to implement the same hash calculation logic, thereby improving file upload performance.

[0080] S402: Use a preset construction tool to compile and package the target logic code to obtain a portable compiled file.

[0081] Optionally, since the front-end device cannot directly run the target logic code written in the target programming language rust, it is necessary to compile the target logic code into a runnable .wasm format executable file and encapsulate it into a module that can be called on the front-end device to obtain a portable compiled file.

[0082] Optionally, the present application compiles and packages the target logic code through a specific tool chain of preset build tools. The preset build tools include wasm-pack and wasm-bindgen. The tool wasm-pack is used to compile and package the target logic code written in the target programming language rust into a target distribution package, and bind the portable compiled file (.wasm file) with JavaScript. The tool wasm-bindgen is used to establish a type-safe interactive interface between the target programming language rust and JavaScript.

[0083] It should be noted that the portable compiled file (.wasm file) in this application is in binary format, with small size, fast loading and high performance. It can be called by JavaScript in the front-end device to implement an API interface similar to native JavaScript.

[0084] S403: Replace the original hash generated file in the original distribution package with the portable compiled file to obtain a target distribution package.

[0085] Optionally, replace the original hash generation file in the original distribution package with a portable compiled file. This means replacing the original JavaScript implementation of the spark-md5 library with a .wasm file, and ultimately releasing a new npm package. The original distribution package is an npm package written in JavaScript, using the spark-md5 library to calculate file hashes. The newly generated target distribution package essentially replaces the original spark-md5 library with a wasm version of the hash calculation module. The overall functionality remains unchanged, but performance is significantly improved.

[0086] It should be noted that the file upload process in this application is still implemented in JavaScript, but the key hash calculation part has been replaced. Specifically, the original class library spark-md5 in the front-end JavaScript project is replaced with a newly generated portable compiled file (.wasm file). In this way, the front-end device automatically loads the .wasm file and calls its hash calculation logic to perform hash calculations, significantly shortening the calculation time of the file hash value and improving file upload efficiency. In terms of the distribution package structure, the target distribution includes the addition of a new .wasm file and the removal of the class library spark-md5 dependency. In other words, the final target distribution package includes the high-performance hash calculation library of the wasm module.

[0087] Optionally, the original hash generation file is generated based on the original programming language, and the hash value calculation efficiency of the target programming language is higher than that of the original programming language. Specifically, the original programming language JavaScript is an interpreted language and is relatively slow, while the portable compiled file compiled based on the target programming language Rust is closer to native speed. Therefore, as a target programming language, Rust has a much higher hash calculation efficiency than the original programming language JavaScript, making it particularly suitable for application scenarios where large files are uploaded.

[0088] Based on this, this application uses the target programming language rust to refer to the class library spark-md5 to write relevant business codes, obtain the target logic code, and then use the tools wasm-pack and wasm-bindgen to compile and package the target logic code written based on rust to obtain a portable compiled file (.wasm file), thereby generating an npm package containing the .wasm file (that is, the target distribution package), and integrating the generated npm package into the front-end JavaScript project, so that when the front-end device calculates the hash value of the resource file, it can directly call the target distribution package based on the preset interface through JavaScript, thereby realizing efficient calculation of the hash value of the resource file.

[0089] As a possible implementation, the method further includes: creating a typed array file and loading the typed array file into the target distribution package, wherein the typed array file is used to convert the data format of the resource file.

[0090] Alternatively, because the appendBinary method called by the spark-md5 library when calculating hash values ​​accepts a string type parameter, while the target distribution package generated by compiling based on Rust accepts a &[u8] type parameter, the data types received by the two are incompatible. Therefore, this application converts the data type received by the target distribution package, specifically using Rust's built-in format conversion method data.as_bytes().

[0091] Optionally, typed arrays are a mechanism for handling binary data in JavaScript, allowing data to be stored and manipulated using fixed-size numeric types (such as Uint8Array, which represents an array of unsigned 8-bit integers). Because WebAssembly expects raw binary data in linear memory, this application uses newUint8Array() to create a typed array file that can be directly mapped to WebAssembly's linear memory, avoiding additional data copying or format conversion steps. The typed array file includes typed array objects used by JavaScript to exchange data with WebAssembly. These objects are responsible for converting data from resource files passed in by the frontend into a format suitable for WebAssembly processing.

[0092] It should be noted that this application implements data format conversion in JavaScript by creating a typed array file. Then, when calculating the hash value, it directly calls the portable compiled file used to generate the hash value based on the preset Rust API interface. In this way, the data format conversion is completed directly in JavaScript, which reduces the need to convert the data format again when running the portable compiled file, and improves overall performance.

[0093] Furthermore, the above steps of running the target distribution package to obtain a hash value of the resource file include: running the typed array file to obtain a typed array instance, and converting the data format of the resource file based on the typed array instance to obtain a processed resource file; running the portable compilation file to perform a hash calculation on the processed resource file to obtain a hash value of the processed resource file.

[0094] Optionally, a typed array instance is created by running the typed array file. This typed array instance is a structure specifically designed for manipulating binary data and is more efficient than a regular JavaScript array. Based on this, the data format of the resource file is converted based on the typed array instance. The string data format received by the spark-md5 library is converted to the &[u8] data format received by the target distribution package compiled based on Rust. This further encapsulates the original resource file data into an array of the &[u8] type, allowing it to be directly passed to the called target distribution package for use.

[0095] Optionally, data sharding is performed on the processed resource file according to a preset data block size to obtain multiple data blocks, and hash calculation is performed on each data block in sequence based on the sharding order to obtain a hash value of the processed resource file.

[0096] Exemplarily, the preset data block size is, for example, 300M. By performing sharding processing tests according to the sizes of 100M, 300M, and 500M, it is finally determined that the hash calculation efficiency is the fastest when the sharding is performed with a size of 300M. On this basis, the processed resource file is sharded according to the data block size of 300M to obtain multiple data blocks. These multiple data blocks are essentially binary shards and have been processed by data format conversion. By calling the appendBinary method, the shard contents corresponding to each data block are loaded into the internal buffer in sequence based on the sharding order for the final hash calculation. Specifically, in this application, a sharding processing test is performed based on a file package of 2.66G. As shown in Table 1 below, for this file package of 2.66G, it is sharded according to the sizes of 100M, 300M, and 500M respectively, and hash calculations are performed based on the class library spark-md5 in the original JavaScript and the call to the target distribution package. Among them, the time consumption corresponding to 100M sharding is slightly longer, and the time consumption corresponding to 300M and 500M sharding is almost the same. However, after testing, in some old versions of Chrome browser, setting a larger sharding will cause a jam, so the hash calculation efficiency is finally optimized when the 300M size is selected for sharding.

[0097] Table 1 Comparison results based on 2.66G file packet fragmentation test

[0098]

[0099] Optionally, the method further includes: if the size of the data block does not meet the preset data block size, padding the data block according to a preset padding strategy. Exemplarily, the preset padding strategy is to add a 1 to the end of the data in the data block, and then pad with 0 until the length requirement is met, and append the total number of bits of the original data to the end of the padded data. After the shard contents corresponding to all databases are added, the end method is called to fill the data blocks that do not meet the preset data block size, for example, the data blocks that do not meet 64KB, and finally output a 128-bit (ie, 16-byte) hash value by appending the binary representation of the total length.

[0100] For example, as shown in the test results in Table 2 below, this application achieves initial optimization based on the shard addition mechanism of calling the appendBinary method, and then achieves data type compatibility by performing data format conversion in JavaScript, eliminating the step of compatible format conversion when compiling portable compiled files based on rust to achieve further optimization. Obviously, the generation time of the hash value of the resource file has been significantly improved.

[0101] Table 2 Hash value generation test results

[0102]

[0103] For example, in combination with the above Table 2, for a large file of 47 GB, it is obvious that the hash value calculation time is saved by about half, and for a smaller file of 2.6 GB, the hash value calculation time improvement effect is even more obvious.

[0104] Based on this, the front-end device creates a typed array in JavaScript, and then fills the data of the resource file into the typed array, completing the data format conversion directly in JavaScript, reducing the need for data format conversion again in wasm. Moreover, since the typed array can be directly mapped to the linear memory of wasm, unnecessary memory allocation and copy operations can also be reduced.

[0105] As a possible implementation method, the above steps upload the resource file to the file server based on the hash value of the resource file, including: determining whether there is a historical resource file with the same hash value as the resource file in the file server; if not, sharding the resource file according to the preset shard size to obtain multiple resource shards, and uploading each resource shard to the file server in sequence according to the sharding order.

[0106] Exemplarily, before uploading a resource file, the front-end device will first calculate the hash value of the uploaded resource file. The hash value can be used as a unique identifier to identify the resource file and to compare with the file stored in the file server. Specifically, the front-end device will send the calculated hash value of the resource file to the file server and ask the file server whether there is already a file with the same hash value. If the file server returns a confirmation that such a file exists, it means that the resource file that the user is trying to upload already has a copy on the file server. There is no need to upload it again, and the existing file can be used directly. On the contrary, if the file server feedback indicates that no file with the same hash value is found, the front-end device needs to split the original resource file according to the preset fragment size (such as 300M per fragment) to form multiple smaller file fragments, obtain multiple resource fragments, and upload each resource fragment to the file server in the fragment order.

[0107] Based on this, a match is performed based on the hash value of the resource file. If the file server finds a file with a matching hash value, it can directly inform the front-end device that the resource file to be uploaded already exists on the file server and does not need to be uploaded again. In this case, the user can immediately obtain the access link or other related information of the resource file, greatly improving efficiency and reducing unnecessary network traffic, thus avoiding the frequent upload of similar file content.

[0108] Figure 5 The flowchart of a file parsing method provided by an embodiment of the present application is shown, and the execution subject of the method is a server in a file processing system. Figure 5 As shown, the method specifically includes the following steps:

[0109] S501: Determine the current operating system, and read the target resolution address corresponding to the current operating system from a preset database.

[0110] Optionally, the target resolution address is also the storage path of the resource file. In this application, after uploading the resource file to the file server, the server first determines whether the currently running operating system is a Windows operating system or a Linux operating system. Then, based on the operation type of the current operating system, the server obtains the target resolution address corresponding to the current operating system from a preset database to ensure that subsequent resolution operations can correctly locate the resolution path that the current operating system should use.

[0111] S502: Move the resource file uploaded to the file server to the target resolution address corresponding to the current operating system.

[0112] Optionally, the front-end device uploads the resource file to the file server. Since the resolution service corresponding to the operating system is responsible for reading the resource file under a specific path, in order for the back-end server to obtain and resolve the resource file, it must first determine the target resolution address corresponding to the currently running operating system and move the resource file from the file server to the target resolution address corresponding to the current operating system. This ensures that the resolution service corresponding to the current operating system can access and process the resource file normally, avoiding the problem of resolution failure due to inconsistent paths.

[0113] S503: Read the resource file from the target resolution address, and use the resolution service corresponding to the current operating system to parse the resource file to obtain model information of the model to be added.

[0114] Optionally, the server reads the resource file from the target resolution address corresponding to the current operating system, and relies on the resolution service of the current operating system to parse the resource file to extract the model information of the model to be added. Further, the obtained model information of the model to be added is stored in the database.

[0115] Exemplarily, model information includes: model type, model identifier, and model name. The model type identifies the model's purpose or classification; the model identifier is a unique identifier used for resource location; and the model name can be used for interface display. The format of model information is model type_model identifier_model name, with parameters separated by underscores. For example, 015001_002_ShouFeiTing1, where 015001 represents the model type, 002 represents the model identifier, and ShouFeiTing1 represents the model name.

[0116] Based on this, this application uses rust to build a cross-platform Pak file parsing service, dynamically locates the parsing path according to the current operating system at runtime, and calls the parsing program of the corresponding system to achieve efficient extraction of model information.

[0117] As a possible implementation method, before reading the target resolution address corresponding to the current operating system from the preset database, the method also includes: compiling and generating the resolution service installation package corresponding to each operating system based on the preset programming language, and deploying the resolution service installation package corresponding to each operating system in each operating system, setting the resolution address corresponding to each operating system, and storing the resolution address corresponding to each operating system in the preset database.

[0118] Optionally, the application separately compiles corresponding executable installation packages for different operating systems and deploys the installation packages to the corresponding operating system environments. In the deployment process, the analysis path for storing resource files under each operating system is set, and the path information of the analysis path is written into the database for subsequent query and calling.

[0119] Illustratively, since programs written in the preset programming language rust can be cross-compiled very conveniently, the application uses the preset programming language rust to compile and generate analysis service installation packages corresponding to each operating system, thereby providing an independent and directly executable analysis service program for each operating system and ensuring that the analysis service can stably run on the respective operating system. The analysis service installation package can include a start script, a configuration template, a dependent library, and the like.

[0120] Illustratively, the analysis address refers to a specified folder path used by the analysis service when processing resource files. The path must exist and have read and write permissions, and is usually defined in the configuration file of the analysis service or can be passed in through a command line parameter. Further, after setting the analysis addresses corresponding to each operating system, the analysis addresses corresponding to each operating system are stored in a preset database, so that the server can quickly find the analysis path according to the type of the current operating system.

[0121] Based on this, the application will construct and deploy adaptive analysis services for different operating systems before analyzing resource files, configure their respective analysis paths, and store these path information in the database, thereby realizing an automated analysis process in a multi-system environment.

[0122] As a possible implementation manner, after the resource file is analyzed by the analysis service corresponding to the current operating system to obtain model information of a to-be-added model, the method further includes: adding the to-be-added model to a flow rendering scene according to the model information.

[0123] Optionally, after the server stores the model information in the database, the user can open a UE flow scene editor that supports dynamic loading and drag-and-drop placement of model resources, and the UE flow scene editor is connected with the backend server and can real-time acquire and load a model list.

[0124] For example, after opening a UE flow scene editor, the user can see the model list in the resource panel on the left side of the operation interface provided by the UE flow scene editor. When clicking on the model category in the model list, the corresponding resource files under this type of model can be resolved by path and registered, and the corresponding model resources can be loaded. Then, select a model icon with the mouse and drag it to the 3D scene view on the right side of the operation interface. When the mouse is released, the underlying API or engine interface is called to load the model and display it in the scene. Furthermore, the user can continue to drag other models to combine and build complex scenes.

[0125] Based on this, after completing the resource file parsing, the server extracts the model information, and based on the model information, realizes the automatic loading and drag-and-drop addition of the model in the UE flow scene editor to build a visual 3D scene.

[0126] Figure 6 The following is a flow chart showing a method for processing a file according to an embodiment of the present application. Figure 6 As shown, the present application provides a complete process from selecting and uploading resource files to dynamically adding models in the UE flow rendering scene. Specifically, the user selects the resource file to be uploaded, and the front-end device calculates the hash value of the file by loading the target distribution package, and then determines whether there is a file with the same hash value in the file server. If it exists, it is uploaded in seconds. If not, the resources are sorted and uploaded in pieces, and after the pieces are uploaded, the pieces are merged to obtain the original resource file. Furthermore, after the upload is completed, the deployment is started. The back-end server deploys the corresponding parsing service according to the type of the current operating system (Windows or Linux), and configures the corresponding parsing address. Then, the parsing service corresponding to the current operating system is started, the resource file is moved to the corresponding parsing address and the parsing task is triggered, so as to parse and obtain the model information, and store the model information in the database, so as to dynamically add the model in the UE flow rendering scene according to the model information in the database.

[0127] The specific implementation methods of each step in the above implementation process have been described in the above content and will not be repeated here.

[0128] The embodiment of the present application also provides a front-end device 700, such as Figure 7As shown, a schematic diagram of the structure of a front-end device 700 provided in an embodiment of the present application includes: a processor 701, a memory 702, and optionally, a bus 703. The memory 702 stores machine-readable instructions executable by the processor 701. When the front-end device 700 is running, the processor 701 communicates with the memory 702 via the bus 703. When the machine-readable instructions are executed by the processor 701, the steps performed by the front-end device in any of the above file uploading methods are performed.

[0129] An embodiment of the present application further provides a computer-readable storage medium having a computer program stored thereon. When the computer program is executed by a processor, the steps performed by the front-end device in any of the above file uploading methods are executed.

[0130] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working process of the system and device described above can refer to the corresponding process in the method embodiment, and will not be repeated in this application. In the several embodiments provided in this application, it should be understood that the disclosed system, device and method can be implemented in other ways. The device embodiments described above are merely schematic. For example, the division of the modules is only a logical function division. There may be other division methods in actual implementation. For example, multiple modules or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some communication interfaces, indirect coupling or communication connection of devices or modules, which can be electrical, mechanical or other forms.

[0131] In addition, the functional units in the various embodiments of the present application can be integrated into a single processing unit, each unit can exist physically separately, or two or more units can be integrated into a single unit. If the functions are implemented in the form of software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the portion that contributes to the prior art, or a portion of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions for causing a computer device (which can be a personal computer, server, or network device, etc.) to execute all or part of the steps of the method described in the various embodiments of the present invention. The aforementioned storage medium includes various media that can store program code, such as a USB flash drive, a mobile hard drive, a read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disk.

[0132] The above is only a specific implementation method of the present application, but the protection scope of the present application is not limited thereto. Any technician familiar with this technical field can easily think of changes or replacements within the technical scope disclosed in this application, which should be covered by the protection scope of the present application.

Claims

1. A file uploading method, characterized in that: Applied to a file processing system, the file processing system includes a front-end device and a server, and the method includes: The front-end device obtains a resource file to be uploaded input by a user, wherein the resource file includes model information of a model to be added; The front-end device calls a target distribution package based on a preset interface, wherein the target distribution package includes a portable compiled file, which is an executable file obtained by compiling and packaging a target logic code, and the portable compiled file is used to generate a hash value, and the target logic code is written in a target programming language; The front-end device runs the target distribution package to obtain a hash value of the resource file; The front-end device uploads the resource file to the file server according to the hash value of the resource file; The method further includes: the server determining a current operating system and reading a target resolution address corresponding to the current operating system from a preset database; The server moves the resource file uploaded to the file server to the target resolution address corresponding to the current operating system; The server reads the resource file from the target resolution address, and uses the resolution service corresponding to the current operating system to parse the resource file to obtain the model information of the model to be added.

2. The method according to claim 1, characterized in that The target distribution package is generated through the following process: Acquire the target logic code written based on the target programming language; Compiling and packaging the target logic code using a preset construction tool to obtain the portable compiled file; The target distribution package is obtained by replacing an original hash generation file in the original distribution package with the portable compiled file, wherein the original hash generation file is generated based on an original programming language, and the hash value operation efficiency of the target programming language is higher than the hash value operation efficiency of the original programming language.

3. The method according to claim 1, characterized in that The method further comprises: A typed array file is created and loaded into the target distribution package, wherein the typed array file is used to convert the data format of the resource file.

4. The method according to claim 3, characterized in that The step of running the target distribution package to obtain a hash value of the resource file includes: Running the typed array file to obtain a typed array instance, and converting the data format of the resource file based on the typed array instance to obtain a processed resource file; The portable compiled file is run, and a hash calculation is performed on the processed resource file to obtain a hash value of the processed resource file.

5. The method according to claim 4, characterized in that The step of running the portable compiled file and performing hash calculation on the processed resource file to obtain a hash value of the processed resource file includes: Performing data slicing processing on the processed resource file according to a preset data block size to obtain multiple data blocks; Hash calculation is performed on each of the data blocks in sequence based on the fragmentation order to obtain a hash value of the processed resource file.

6. The method according to claim 5, characterized in that The method further comprises: If the size of the data block does not meet the preset data block size, the data block is padded according to a preset padding strategy.

7. The method according to claim 1, characterized in that The uploading of the resource file to the file server according to the hash value of the resource file includes: Determining whether there is a historical resource file having the same hash value as the resource file in the file server; If not, the resource file is fragmented according to a preset fragment size to obtain a plurality of resource fragments, and each resource fragment is uploaded to the file server in sequence according to the fragmentation order.

8. The method according to claim 1, characterized in that Before reading the target resolution address corresponding to the current operating system from the preset database, the method further includes: Compile and generate the corresponding parsing service installation package for each operating system based on the preset programming language, and deploy the corresponding parsing service installation package for each operating system in each operating system; The parsed address corresponding to each operating system is set, and the parsed address corresponding to each operating system is stored in the preset database.

9. The method according to claim 1, characterized in that After parsing the resource file using the parsing service corresponding to the current operating system to obtain the model information of the model to be added, the method further includes: The model to be added is added to the stream rendering scene according to the model information, wherein the model information includes: model type, model identifier and model name.

10. A file processing system, characterized in that: The file processing system includes a front-end device and a server; The front-end device is used to perform the steps of the file uploading method performed by the front-end device in any one of claims 1 to 9; The server is used to execute the steps of the file uploading method executed by the server in any one of claims 1-9.

Images