Data processing method, device, equipment and storage medium

Abstract

The present disclosure provides a data processing method, apparatus, device, and storage medium, relating to the field of cloud computing technology and applicable to cloud platforms. The method comprises: obtaining target data for creating a mirror package; sharding the target data based on the metadata of the target data to obtain multiple shard data; uploading each shard data to a content distribution network based on identification information of each shard data; packaging the metadata and each shard data into a metadata mirror package, and uploading the metadata mirror package to the content distribution network. By using metadata mirroring technology, the method can run applications concurrently and quickly on multiple cloud phones on multiple ARM devices, allowing users to use them without waiting, thereby improving the user experience.

Description

Technical Field

[0001] The present disclosure relates to the field of cloud computing technology, and in particular to a data processing method, apparatus, device, and storage medium. Background Art

[0002] In the cloud phone application field, a typical ARM (Advanced RISC Machines) server contains 92 computing nodes, with each 23 ARM devices forming an ARM device group sharing a 1Gbit uplink port. The typical practice is to install games or applications directly using local storage on each ARM device. However, due to local motherboard capacity limitations, expansion is impossible, and data loss can occur if the local disk is damaged. Using general-purpose network storage solutions can lead to the duplication and storage of large amounts of data, as well as the inability to quickly run applications on multiple cloud phones, which in turn increases costs. Summary of the Invention

[0003] The present disclosure provides a data processing method, apparatus, device, and storage medium.

[0004] According to a first aspect of the present disclosure, a data processing method is provided, comprising: obtaining target data for producing a mirror package; slicing the target data based on metadata of the target data to obtain a plurality of slicing data; uploading each slicing data to a content distribution network based on identification information of each slicing data; packaging the metadata and each slicing data into a metadata mirror package, and uploading the metadata mirror package to the content distribution network.

[0005] According to a second aspect of the present disclosure, a data processing device is provided, comprising: an acquisition module configured to acquire target data for producing a mirror package; a sharding module configured to shard the target data based on metadata of the target data to obtain a plurality of shard data; an uploading module configured to upload each shard data to a content distribution network based on identification information of each shard data; and a packaging module configured to package the metadata and each shard data into a metadata mirror package, and upload the metadata mirror package to the content distribution network.

[0006] According to a third aspect of the present disclosure, an electronic device is provided, comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to execute the method described in any implementation manner in the first aspect.

[0007] According to a fourth aspect of the present disclosure, a non-transitory computer-readable storage medium storing computer instructions is provided, where the computer instructions are used to cause a computer to execute the method described in any implementation manner of the first aspect.

[0008] According to a fifth aspect of the present disclosure, a computer program product is provided, comprising a computer program, which implements the method described in any implementation manner of the first aspect when executed by a processor.

[0009] It should be understood that the contents described in this section are not intended to identify the key or important features of the embodiments of the present disclosure, nor are they intended to limit the scope of the present disclosure. Other features of the present disclosure will become readily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0010] The accompanying drawings are provided to facilitate a better understanding of the present invention and do not constitute a limitation of the present disclosure.

[0011] Figure 1 is an exemplary system architecture diagram in which the present disclosure may be applied;

[0012] Figure 2-1 is a flow chart of an embodiment of a data processing method according to the present disclosure;

[0013] Figure 2-2 This is a diagram of the network connections within the chassis;

[0014] Figure 2-3 It is a schematic diagram of the network connection between chassis;

[0015] Figure 3 is a flow chart of another embodiment of a data processing method according to the present disclosure;

[0016] Figure 4 is a flow chart of another embodiment of a data processing method according to the present disclosure;

[0017] Figure 5 is an application flow chart of the data processing method according to the present disclosure;

[0018] Figure 6 is a flow chart of another embodiment of a data processing method according to the present disclosure;

[0019] Figure 7 is another application flow chart of the data processing method according to the present disclosure;

[0020] Figure 8 This is a schematic diagram of the application of the data processing method disclosed herein in a data migration scenario in a computer room;

[0021] Figure 9is a structural diagram of an embodiment of a data processing device according to the present disclosure;

[0022] Figure 10 It is a block diagram of an electronic device used to implement the data processing method according to an embodiment of the present disclosure. DETAILED DESCRIPTION

[0023] The following description of exemplary embodiments of the present disclosure is made in conjunction with the accompanying drawings, including various details of the embodiments of the present disclosure to facilitate understanding. These details should be considered as merely exemplary. Therefore, those skilled in the art will recognize that various changes and modifications may be made to the embodiments described herein without departing from the scope and spirit of the present disclosure. Similarly, for the sake of clarity and conciseness, descriptions of well-known functions and structures are omitted in the following description.

[0024] It should be noted that, in the absence of conflict, the embodiments and features of the embodiments in the present disclosure may be combined with each other. The present disclosure will be described in detail below with reference to the accompanying drawings and in combination with the embodiments.

[0025] Figure 1 An exemplary system architecture 100 is shown to which embodiments of the data processing method or data processing apparatus of the present disclosure can be applied.

[0026] like Figure 1 As shown, system architecture 100 may include terminal devices 101, 102, 103, a network 104, and a server 105. Network 104 is a medium for providing communication links between terminal devices 101, 102, 103 and server 105. Network 104 may include various connection types, such as wired or wireless communication links or fiber optic cables.

[0027] Users can use terminal devices 101, 102, 103 to interact with server 105 via network 104 to receive or send information, etc. Various client applications can be installed on terminal devices 101, 102, 103.

[0028] Terminal devices 101, 102, and 103 can be either hardware or software. When terminal devices 101, 102, and 103 are hardware, they can be various electronic devices, including but not limited to smartphones, tablet computers, laptop computers, and desktop computers. When terminal devices 101, 102, and 103 are software, they can be installed in the aforementioned electronic devices. They can be implemented as multiple software programs or software modules, or as a single software program or software module. This is not specifically limited here.

[0029] The server 105 can provide various services. For example, the server 105 can analyze and process target data obtained from the terminal devices 101, 102, and 103, and generate a processing result (eg, a metadata image package).

[0030] It should be noted that the server 105 can be hardware or software. When the server 105 is hardware, it can be implemented as a distributed server cluster consisting of multiple servers, or it can be implemented as a single server. When the server 105 is software, it can be implemented as multiple software or software modules (for example, to provide distributed services), or it can be implemented as a single software or software module. No specific limitations are given here.

[0031] It should be noted that the data processing method provided in the embodiments of the present disclosure is generally executed by the server 105 , and accordingly, the data processing device is generally provided in the server 105 .

[0032] It should be understood that Figure 1 The number of terminal devices, networks and servers in the embodiment is merely illustrative. Any number of terminal devices, networks and servers may be provided as required.

[0033] Continue to refer Figure 2-1 , which shows a process 200 of an embodiment of a data processing method according to the present disclosure. The data processing method includes the following steps:

[0034] Step 201: Obtain target data for creating an image package.

[0035] In this embodiment, the execution subject of the data processing method (eg Figure 1 The server 105 shown in the figure will obtain the target data for making the mirror package. A mirror package refers to a complete backup file of a software or system, which contains all the programs, data and configurations of the software or system and can be used to quickly deploy or restore the software or system. The mirror package can quickly deploy the same environment on multiple servers, improving deployment efficiency; for operation and maintenance personnel, using the mirror package can quickly restore the system, reduce the losses caused by system crashes, and also reduce the workload of system operation and maintenance. In general, the mirror package is a convenient, fast and reliable software deployment and backup method that can greatly improve the efficiency and quality of software development and operation and maintenance work.

[0036] Based on this, in this embodiment, the metadata of the application data that needs to be run by multiple cloud phones on multiple ARM devices is made into a metadata mirror package. By using metadata mirroring technology, multiple cloud phones on multiple ARM devices can run applications concurrently and quickly, and users can use them without waiting.

[0037] It should be noted that, in this embodiment, 23 ARM devices form an ARM device group, and 4 ARM device groups form an ARM server chassis. The 23 ARM devices in each ARM group are connected inside the chassis through a LAN (Local Area Network). Multiple ARM server chassis are connected to CDN (Content Delivery Network) storage through the LAN network. Figure 2-2 , Figure 2-2 shows the network connections within the chassis, such as Figure 2-2 As shown, a server chassis contains four ARM device groups (i.e., device group 1, device group 2, device group 3, and device group 4). Each device group contains 23 ARM devices (i.e., ARM node 1, ARM node 2, ..., ARM node 23). Each ARM device group has 24 ports, operating at 1 Gbps (switching bandwidth). Within each ARM device group, there are two metadata management service devices, which manage all the disk devices of the 23 ARM nodes. These disks form a global storage space for running applications on the ARM devices. Each disk device corresponds to a program that communicates with the metadata management service and CDN.

[0038] Further, Figure 2-3 The network connections between the chassis are shown. Figure 2-3 As shown, the CDN storage device is connected to a cabinet switch, which is then connected to each chassis (Chassis 1, Chassis 2, ..., Chassis 6). The cabinet switch has a 10G uplink and a 1G downlink, meaning that the cabinet switch and the CDN storage device transmit data at 10Gbyte / s, and the cabinet switch and each chassis transmit data at 1Gbyte / s.

[0039] Step 202: Slice the target data based on the metadata of the target data to obtain a plurality of sliced ​​data.

[0040] In this embodiment, the execution entity segments the target data based on its metadata to obtain multiple shards. After obtaining the target data, the execution entity separates the target data into metadata and file data. Metadata is system data used to describe the characteristics of a file, such as access rights, file owner, and the distribution of file data blocks. File data is the actual data to be stored, i.e., the content data in the file.

[0041] The file data is then sliced ​​based on the metadata, that is, divided into several slices. Specifically, because the metadata stores the file size information of the file data, the file data can be sliced ​​according to a preset size based on the metadata, thereby obtaining multiple slices. For example, the file data can be sliced ​​into 4MB (megabyte) slices. For another example, if the file data is very large, the size of each slice can be adjusted accordingly, that is, the file data can be sliced ​​into 5MB (or larger) slices.

[0042] Step 203: Upload each fragment data to a content distribution network based on the identification information of each fragment data.

[0043] In this embodiment, the execution entity uploads each shard data to the content distribution network based on the identification information of each shard data. After obtaining multiple shard data, the execution entity uses a hash algorithm to calculate the hash value corresponding to each shard data, that is, hashing each shard to obtain the address value (i.e., hash value) of each shard data. Since some attribute information of the shard data stored in its metadata is used when calculating the hash value of the shard data, the hash value of each shard data is unique. The hash value is used as the identification information of the shard data, and the shard data is uniquely identified by the identification information.

[0044] Because the identification information for each data segment is unique, the execution entity uploads each data segment to the content distribution network based on the identification information. Specifically, the execution entity first determines whether the content distribution network already stores the data segment corresponding to the identification information. If so, there is no need to upload the data segment repeatedly. If not, the data segment is uploaded to the content distribution network, thereby avoiding duplicate uploads based on the identification information.

[0045] Step 204: Package the metadata and each fragment data into a metadata mirror package, and upload the metadata mirror package to a content distribution network.

[0046] In this embodiment, the execution entity packages the metadata and each shard data into a metadata image package and uploads the metadata image package to the content distribution network. Specifically, the execution entity sequentially writes the metadata information of each file and all shard information into the file and packages the metadata image package. The metadata image package is then uploaded to the CDN metadata area.

[0047] The data processing method provided by the embodiment of the present disclosure first obtains the target data for making a mirror package; then, the target data is sliced ​​based on the metadata of the target data to obtain a plurality of sliced ​​data; thereafter, each sliced ​​data is uploaded to the content distribution network based on the identification information of each sliced ​​data; finally, the metadata and each sliced ​​data are packaged into a metadata mirror package, and the metadata mirror package is uploaded to the content distribution network. The data processing method in this embodiment uses metadata mirroring technology to achieve the concurrent and rapid running of applications by multiple cloud phones on multiple ARM devices, and users can use them without waiting. In addition, the use of data slicing avoids repeated data transmission and repeated storage of data, which improves the efficiency of data transmission on the one hand, and improves the utilization efficiency of ARM node disk devices on the other hand, avoiding the problem of uneven use of disk space of each ARM device.

[0048] Continue to refer Figure 3 , Figure 3 A process 300 of another embodiment of a data processing method according to the present disclosure is shown. The data processing method includes the following steps:

[0049] Step 301: Obtain target data for creating an image package.

[0050] In this embodiment, the execution subject of the data processing method (eg Figure 1 The server 105 shown in FIG. 1 ) obtains target data for making a mirror package.

[0051] Step 301 is basically the same as step 201 in the aforementioned embodiment. For the specific implementation method, please refer to the aforementioned description of step 201, which will not be repeated here.

[0052] Step 302: Determine metadata and file data from the target data.

[0053] In this embodiment, the execution entity determines metadata and file data from the target data. That is, after obtaining the target data, the execution entity separates the target data into metadata and file data. Metadata is system data used to describe the characteristics of a file, such as access permissions, file owner, and the distribution of file data blocks. File data is the actual data to be stored, namely the content data in the file.

[0054] Step 303: Segment the file data into pieces according to the preset size based on the attribute information in the metadata to obtain a plurality of piece data.

[0055] In this embodiment, the execution entity will slice the file data into pieces according to a preset size based on the attribute information in the metadata, thereby obtaining multiple pieces of data. Since the metadata stores the file size information of the file data, the file data can be sliced ​​into pieces according to the preset size based on the metadata, thereby obtaining multiple pieces of data. For example, the file data can be sliced ​​into pieces of 4MB (megabytes); for another example, if the file data is very large, the size of each piece of data can be adjusted accordingly, that is, the file data can be sliced ​​into pieces of 5MB (or larger).

[0056] Step 304: Calculate the hash value of each shard data using a hash algorithm, and record the hash value as identification information of the shard data.

[0057] In this embodiment, the execution entity uses a hash algorithm to calculate the hash value of each shard data and records the calculated hash value as the identification information of the shard data. In other words, each shard is hashed to obtain the address value (i.e., hash value) of each shard data. Since some attribute information of the shard data stored in its metadata is used when calculating the hash value of the shard data, the hash value of each shard data is unique. The hash value is used as the identification information of the shard data, and the shard data is uniquely identified by the identification information.

[0058] Step 305: For each fragment data, determine whether the content distribution network has stored the fragment data based on the identification information of the fragment data.

[0059] In this embodiment, for each data segment, the execution entity determines whether the content distribution network already stores the data segment based on the identification information of the data segment. In other words, it determines whether the content distribution network already stores the data segment corresponding to the current identification information. If so, there is no need to upload it again.

[0060] Step 306 : In response to determining that the content distribution network does not store the segmented data, store the segmented data in the content distribution network.

[0061] In this embodiment, the execution subject will store the fragmented data in the content distribution network when it is determined that the content distribution network does not store the fragmented data, thereby avoiding repeated upload of the fragmented data, that is, achieving deduplication of the fragmented data based on the identification information.

[0062] Step 307: Package the metadata and each fragment data into a metadata mirror package, and upload the metadata mirror package to a content distribution network.

[0063] Step 307 is basically the same as step 204 in the aforementioned embodiment. For the specific implementation method, reference can be made to the aforementioned description of step 204 and will not be repeated here.

[0064] from Figure 3 It can be seen that, compared with the embodiment corresponding to Figure 2, the data processing method in this embodiment highlights the steps of sharding the target data and uploading it according to the identification information of the sharded data, thereby avoiding repeated data transmission and repeated storage of data by using data sharding. On the one hand, it improves the efficiency of data transmission, and on the other hand, it also improves the utilization efficiency of the ARM node disk device, avoiding the problem of uneven use of disk space of each ARM device.

[0065] Continue to refer Figure 4 , Figure 4 A process 400 of another embodiment of a data processing method according to the present disclosure is shown. The data processing method includes the following steps:

[0066] Step 401 : in response to receiving a mirror data read request, obtain a corresponding metadata mirror package, which is referred to as reading the mirror package.

[0067] In this embodiment, the execution subject of the data processing method (eg Figure 1 When receiving a mirror data read request, the server 105) will obtain the corresponding metadata mirror package from the CDN and record it as a read mirror package.

[0068] Step 402: According to the file handle of the data to be read, the fragment data corresponding to the data to be read is determined from the read mirror package, and recorded as the target read fragment.

[0069] In this embodiment, the execution subject determines the shard data corresponding to the data to be read from the read mirror package according to the file handle of the data to be read, and records it as the target read shard.

[0070] In file I / O (Input / Output), to read data from a file, an application first calls an operating system function, passing the file name and selecting a path to the file. This function returns a sequence number, or file handle, which uniquely identifies the open file.

[0071] Since the metadata mirror package stores the identification information of all shard data, the above-mentioned execution entity will determine the shard data corresponding to the data to be read from the read mirror package according to the file handle of the data to be read, and record it as the target read shard.

[0072] Step 403: Read the target read slice from the storage area corresponding to the location of the target read slice based on the identification information of the target read slice.

[0073] In this embodiment, the execution entity reads the target read shard from the storage area where the target read shard is located based on the identification information of the target read shard. Since the target read shard may be located on a local device (i.e., a local storage area), a shared device (i.e., a shared storage area), or a CDN, the execution entity reads the target read shard from the storage area corresponding to the location of the target read shard based on the identification information of the target read shard.

[0074] In some optional implementations of this embodiment, step 403 includes: in response to determining that the target read shard is stored in the local storage area, determining the target read node storing the target read shard; and reading the target read shard from the target read node based on the identification information of the target read shard.

[0075] In this implementation, the execution entity initiates a request to retrieve a shard from the metadata server. The metadata server queries all shards to determine whether a corresponding shard exists locally on the ARM device group. If so, it returns the device ID of the ARM node (i.e., the target read node) and sends a request to retrieve shard data to the node where the shard resides. Based on the identification information of the target read shard, the target read shard is retrieved from the target read node. Finally, the retrieved data is returned to the ARM device cloud phone application, thereby enabling the target read shard to be read locally.

[0076] In some optional implementations of this embodiment, step 403 includes: in response to determining that the target read shard is not stored in the local storage area, sending a read request to the corresponding node in the shared storage area; downloading the target read shard from the content distribution network and storing the target read shard in the shared storage area; and reading the target read shard from the corresponding node in the shared storage area based on the identification information of the target read shard.

[0077] In this implementation, the above-mentioned execution entity will initiate the step of obtaining the shards to the metadata server. The metadata server queries all shards to determine whether there are corresponding shards locally in the ARM device group. If not, it will send a read request to the ARM node assigned to obtain the shards and to the corresponding node in the shared storage area. After the ARM node receives the request to obtain the shared shard data sent by the metadata management server, it will obtain the shard data from the CDN data area, store it in the shard data area, and return the acquisition result to the metadata management server. The metadata management server returns the acquired data to the ARM node that requested the shard, thereby reading the target read shard from the ARM node in the shared storage area based on the identification information of the target read shard. Finally, the cloud phone on the ARM device closes the open file handle to complete the reading of this data. Therefore, when the local node does not store the data to be read, the data to be read is read from the shared storage area.

[0078] from Figure 4 It can be seen that Figure 3 Compared with the corresponding embodiment, the data processing method in this embodiment implements the reading operation of the mirror data and improves the data reading efficiency.

[0079] Further reference Figure 5 , Figure 5 An application flow chart of the data processing method disclosed herein is shown, which specifically includes:

[0080] Step 501: The cloud phone application initiates a data read request.

[0081] Step 502: Determine whether the file is opened successfully. If not, proceed to step 503; otherwise, proceed to step 504.

[0082] Step 503, exit.

[0083] Step 504: Obtain all fragment information corresponding to the read file.

[0084] Step 505: Determine whether the shard data is stored in the local shard cache. If so, proceed to step 506; otherwise, proceed to step 507.

[0085] Step 506: Read the shard data from the local shard cache.

[0086] Step 507: Obtain data from the CDN and store it in the shared shard cache.

[0087] Step 508: Return the acquired data to the cloud phone application.

[0088] Step 509: close the reading data file.

[0089] Continue to refer Figure 6 , Figure 6 A process 600 of another embodiment of a data processing method according to the present disclosure is shown. The data processing method includes the following steps:

[0090] Step 601: in response to receiving a mirror data write request, obtain a corresponding metadata mirror package, which is recorded as a write mirror package.

[0091] In this embodiment, the execution subject of the data processing method (eg Figure 1 When receiving a mirror data write request, the server 105) will obtain the corresponding metadata mirror package from the CDN and record it as a write mirror package.

[0092] Step 602: According to the file handle of the data to be written, the fragment data corresponding to the data to be written is determined from the write mirror package and recorded as the target write fragment.

[0093] In this embodiment, the execution subject determines the shard data corresponding to the data to be written from the write mirror package according to the file handle of the data to be written, and records it as the target write shard.

[0094] Since the metadata mirror package stores the identification information of all shard data, the above-mentioned execution entity will determine the shard data corresponding to the data to be written from the write mirror package according to the file handle of the data to be written, and record it as the target write shard.

[0095] Step 603: Determine whether the storage area where the node corresponding to the target write slice is located has stored the identification information of the target write slice, and write the data to be written based on the determination result.

[0096] In this embodiment, the execution entity writes the target write shard from the storage area corresponding to the location of the target write shard based on the identification information of the target write shard. Since the target write shard may be located in a local device (i.e., a local storage area), a shared device (i.e., a shared storage area), or a CDN, the execution entity first determines whether the storage area where the node corresponding to the target write shard is located already stores the identification information of the target write shard, and then writes the data to be written based on the determination result.

[0097] In some optional implementations of this embodiment, step 603 includes: in response to determining that the local storage area where the node corresponding to the target write shard is located does not store the identification information of the target write shard, writing the target write shard to a new node in the local storage area; storing the identification information of the new node in the write mirror package.

[0098] In this implementation, the above-mentioned execution subject will send a request to write the shard to the metadata management server, obtain the data access ARM node device number, and generally return the requested ARM node device. If the requested ARM node space is insufficient, the device number of the adjacent ARM node will be returned. After successfully obtaining the shard, a request to open up cache space is sent to the shard write ARM node. After receiving the request, the ARM device will open up cache space in the shard cache area to store the data to be written. When the above-mentioned execution subject determines that the ARM node is a local ARM node, it will directly write the target write shard into a new node in the local storage area; and store the identification information of the new node in the write mirror package. This will achieve writing the target write shard at the local node.

[0099] In some optional implementations of this embodiment, step 603 includes: in response to determining that the local storage area where the node corresponding to the target write shard is located has stored the identification information of the target write shard, writing the data to be written into the corresponding node.

[0100] In this implementation, the above-mentioned execution subject will send a request to write the shard to the metadata management server, obtain the data access ARM node device number, and generally return the requested ARM node device. If the requested ARM node space is insufficient, the device number of the adjacent ARM node will be returned. After successfully obtaining the shard, a request to open up cache space is sent to the ARM node that writes the shard. After receiving the request, the ARM device opens up cache space in the shard cache area to store the data to be written. When the above-mentioned execution subject determines that the ARM node is not a local ARM node, it sends the data to the write ARM node, and the write ARM node writes the data to the corresponding device. Finally, the shard information is written into the file metadata to complete the update of the file metadata information. The cloud phone on the ARM device closes the open file handle to complete the writing of this data. Therefore, when the local node does not store the data to be written, the data to be written is written to the corresponding node.

[0101] from Figure 6 It can be seen that Figure 4 Compared with the corresponding embodiment, the data processing method in this embodiment implements the writing operation of the mirror data and improves the data writing efficiency.

[0102] Further reference Figure 7 , Figure 7 Another application flow chart of the data processing method of the present disclosure is shown, which specifically includes:

[0103] Step 701: The cloud phone application initiates a data write request.

[0104] Step 702: Determine whether the file is opened successfully. If not, proceed to step 703; otherwise, proceed to step 704.

[0105] Step 703, exit.

[0106] Step 704: Create cache space in the local shard cache area.

[0107] Step 705: Obtain corresponding fragment information according to the file request location.

[0108] Step 706: Determine whether the shard data is stored in the local shard cache. If not, proceed to step 707; otherwise, proceed to step 708.

[0109] Step 707: Obtain data from the CDN and store it in the local shard cache.

[0110] Step 708: Complete writing of the fragmented data according to the file request location.

[0111] Step 709: Write the fragment information into the file metadata.

[0112] Step 710: Close the write data file.

[0113] Step 711: Upload the fragmented data asynchronously to the CDN.

[0114] Furthermore, the data processing method disclosed herein can be applied to the field of ARM cloud phone cloud gaming. In this application scenario, the method specifically includes:

[0115] 1. Create a game image, that is, install the game that needs to be mirrored on one of the cloud phones, mirror the game data metadata, and upload the game data and game metadata image to the CDN.

[0116] Specifically, it includes: 1) Installing a template game. A game that needs to be run on multiple ARM devices and multiple cloud phones is installed on a cloud phone on one of the ARM devices to create a game template image.

[0117] 2) Get metadata. Read metadata information of all game-related files, including file size, time, and other attribute information.

[0118] 3) Data Sharding: Obtain the file metadata information based on the previous step, split each file into 4MB shards, and use a hash algorithm to calculate the shard unique identifier based on the shard content.

[0119] 4) Upload fragment data. According to the unique identifier of each fragment, the CDN is checked to see whether the fragment exists. If it exists, it is skipped. If not, the fragment is uploaded to the CDN fragment data area.

[0120] 5) Create a game metadata image. Write the metadata information and data shard information of each file into files in turn and package them into a metadata image package.

[0121] 6) Upload the metadata image. Upload the metadata image package to the CDN metadata area.

[0122] 2. Obtain the game image. Multiple ARM devices obtain the game metadata image from the CDN and use it to quickly install the game on the cloud phone.

[0123] 3. Game data reading. After multiple ARM devices and multiple cloud phones start the game, read the game data required for game startup. This specifically includes:

[0124] 1) Open the file. When the ARM device cloud phone starts the game, it opens the file and obtains the file handle for reading the game data. If the opening is successful, it proceeds to the following sub-steps; otherwise, it exits.

[0125] 2) Get file fragment information. Based on the open file handle, get all the fragment information of the file, and get the fragment information of the game data to be read based on the file request location.

[0126] 3) Obtain shard location. Initiate a shard acquisition request to the metadata server. The metadata server queries all shards to determine whether there is a corresponding shard in the local ARM device group. If so, the device number of the ARM node is returned and the process proceeds to step 4) of obtaining shard data. If no ARM node is assigned to obtain the shard, the ARM node for the shard request is prioritized and a request for shared shard data is sent to that ARM node, proceeding to step 5) of obtaining shared shard data.

[0127] 4) Obtain shard data. Convert the file request location to a shard request location, send a shard data request to the ARM node where the shard resides, obtain the required data, and then proceed to step 6) of returning data.

[0128] 5) Obtaining shared shard data. After receiving the request from the metadata management server for shared shard data, the ARM node retrieves the shard data from the CDN data area, stores it in the shard data area, and returns the retrieval result to the metadata management server. The metadata management server then returns the retrieval result to the ARM node that requested the shard, and then proceeds to step 4) of obtaining shard data.

[0129] 6) Data return: The data obtained from the shard is returned to the ARM device cloud mobile game application.

[0130] 7) Close the file. The cloud phone on the ARM device closes the open file handle, completing the reading of the game data.

[0131] 4. Game data writing. After multiple ARM devices and multiple cloud phones run the game, the data generated by the game is written. This specifically includes:

[0132] 1) Open the file. When running the game on the cloud phone on the ARM device, open the file and obtain the file handle for writing game data. If the opening is successful, proceed to the following sub-steps; otherwise, exit.

[0133] 2) Create cache space. If the file is successfully opened, cache space is created in the local shard cache area to store the data to be written.

[0134] 3) Obtain shard information. Based on the file write location, obtain the shard information to be written. If the shard data is newly written or the shard data exists in the local shard cache, proceed to step 5) of writing the shard data. Otherwise, proceed to step 4) of processing the shard data.

[0135] 4) Shard data processing: First, check whether the shared shard cache contains the shard data to be written. If so, transfer the shared shard data to the local shard cache. If not, obtain the shard data from the CDN data area and store it in the local shard cache.

[0136] 5) Shard data writing: Convert the file request location to the shard request location and write the game data to the corresponding location of the corresponding shard in the local shard cache.

[0137] 6) Update metadata: Write the shard information into the file metadata to complete the update of the file metadata information.

[0138] 7) Close the file. The cloud phone on the ARM device closes the open file handle, completing the writing of the game data.

[0139] 8) Shard data is uploaded asynchronously. A hash algorithm is used to calculate the unique identifier of the shard based on the shard content. The shard unique identifier is used to check whether the shard data exists from the CDN. If not, the shard is uploaded to the CDN data area.

[0140] Therefore, through technologies such as metadata mirroring and device space sharing, multiple cloud mobile games can be quickly and concurrently run on multiple ARM devices, improving the efficiency of concurrent game running and effectively reducing the bandwidth and storage space occupied by game running.

[0141] Furthermore, the data processing method disclosed herein can be applied to the field of data migration in a computer room. In this application scenario, the method specifically includes:

[0142] 1. Create a data migration image. Create an image of the data to be migrated based on the cloud phone, and upload the relevant file data and metadata image package to the CDN. This includes:

[0143] 1) Obtain metadata. Read metadata information of all files related to the migration data, including file size, time, and other attribute information.

[0144] 2) Data Sharding: Obtain the file metadata information based on the previous step, split each file into 4MB shards, and use a hash algorithm to calculate the shard unique identifier based on the shard content.

[0145] 3) Upload fragment data. Based on the unique identifier of each fragment, the CDN is checked to see if the fragment exists. If it exists, it is skipped. If not, the fragment is uploaded to the CDN fragment data area.

[0146] 4) Create a metadata image. Write the metadata information and data shard information of each file into files in turn and package them into a metadata image package.

[0147] 5) Upload the metadata image. Upload the metadata image package to the CDN metadata area.

[0148] 2. Obtain the data migration image. The target data center obtains the metadata image from the CDN and presents the corresponding directory and file information through the metadata image.

[0149] 3. Migration data processing. Read the corresponding data through the presented file information and synchronize it to the target device. This specifically includes:

[0150] 1) Open the file. This is used to synchronize data with the target computer room and obtain the file handle for reading data. If the opening is successful, proceed to the following sub-steps; otherwise, exit.

[0151] 2) Get file fragment information. According to the open file handle, get all the fragment information of the file.

[0152] 3) Obtain the shard location. A request to obtain the shard is sent to the metadata server. The metadata server queries all shards to determine whether there is a corresponding shard in the local ARM device group. If so, the device number of the ARM node is returned, and the process proceeds to step 4) to obtain the shard data. If no ARM node is assigned to obtain the shard, the ARM node for the shard request is assigned first, and a request for shared shard data is sent to that ARM node, proceeding to step 5) to obtain shared shard data.

[0153] 4) Obtain shard data. Convert the file request location to a shard request location, send a shard data request to the ARM node where the shard resides, obtain the required data, and then proceed to step 6) of returning data.

[0154] 5) Obtaining shared shard data. After receiving the request from the metadata management server for shared shard data, the ARM node retrieves the shard data from the CDN data area, stores it in the shard data area, and returns the retrieval result to the metadata management server. The metadata management server then returns the retrieval result to the ARM node that requested the shard, and then proceeds to step 4) of obtaining shard data.

[0155] 6) Data synchronization: Synchronize the data obtained from the shard to the target device.

[0156] 7) Close the file. Close the open file handle and complete the reading of the data.

[0157] For reference Figure 8 , Figure 8 The application of the data processing method disclosed in the present invention in the data migration scenario of the computer room is shown. Figure 8As can be seen in the figure, the data processing method disclosed herein migrates data from multiple source ARM devices to multiple target ARM devices via CDN storage devices. By leveraging metadata mirroring, data sharding, and device space sharing, data in the computer room can be quickly migrated, effectively improving migration efficiency and reducing bandwidth usage and migration costs.

[0158] Further references Figure 9 As an implementation of the methods shown in the above figures, the present disclosure provides an embodiment of a data processing device. The device embodiment corresponds to the method embodiment shown in Figure 2, and the device can be specifically applied to various electronic devices.

[0159] like Figure 9 As shown, the data processing device 900 of this embodiment includes: an acquisition module 901, a fragmentation module 902, an upload module 903, and a packaging module 904. The acquisition module 901 is configured to acquire target data for creating a mirror package; the fragmentation module 902 is configured to fragment the target data based on the metadata of the target data to obtain multiple fragments; the upload module 903 is configured to upload each fragment to a content distribution network based on the identification information of each fragment; and the packaging module 904 is configured to package the metadata and each fragment into a metadata mirror package, and upload the metadata mirror package to the content distribution network.

[0160] In this embodiment, in the data processing device 900, the specific processing of the acquisition module 901, the fragmentation module 902, the upload module 903 and the packaging module 904 and the technical effects brought about by them can be referred to the relevant descriptions of steps 201-204 in the corresponding embodiment of Figure 2 respectively, and will not be repeated here.

[0161] In some optional implementations of this embodiment, the sharding module is further configured to: determine metadata and file data from the target data; and slice the file data into preset sizes based on attribute information in the metadata to obtain multiple shard data.

[0162] In some optional implementations of this embodiment, the upload module is further configured to: calculate a hash value for each shard data using a hash algorithm, and record the hash value as identification information of the shard data; for each shard data, determine whether the content distribution network has already stored the shard data based on the identification information of the shard data; and in response to determining that the content distribution network does not store the shard data, store the shard data in the content distribution network.

[0163] In some optional implementations of this embodiment, the above-mentioned data processing device 900 also includes: a read request receiving module, configured to obtain a corresponding metadata mirror package in response to receiving a mirror data read request, recorded as a read mirror package; a read shard determination module, configured to determine the shard data corresponding to the data to be read from the read mirror package according to the file handle of the data to be read, recorded as a target read shard; a reading module, configured to read the target read shard from the storage area corresponding to the location of the target read shard based on the identification information of the target read shard.

[0164] In some optional implementations of this embodiment, the reading module is further configured to: in response to determining that the target read shard is stored in the local storage area, determine the target read node storing the target read shard; and read the target read shard from the target read node based on the identification information of the target read shard.

[0165] In some optional implementations of this embodiment, the reading module is further configured to: in response to determining that the target read segment is not stored in the local storage area, send a read request to the corresponding node in the shared storage area; download the target read segment from the content distribution network and store the target read segment in the shared storage area; and read the target read segment from the corresponding node in the shared storage area based on the identification information of the target read segment.

[0166] In some optional implementations of this embodiment, the above-mentioned data processing device 900 also includes: a write request receiving module, configured to obtain a corresponding metadata mirror package in response to receiving a mirror data write request, recorded as a write mirror package; a write shard determination module, configured to determine the shard data corresponding to the data to be written from the write mirror package according to the file handle of the data to be written, recorded as a target write shard; a write module, configured to determine whether the storage area where the node corresponding to the target write shard is located has stored the identification information of the target write shard, and write the data to be written based on the judgment result.

[0167] In some optional implementations of this embodiment, the write module is further configured to: in response to determining that the local storage area where the node corresponding to the target write shard is located does not store the identification information of the target write shard, write the target write shard to a new node in the local storage area; and store the identification information of the new node in the write mirror package.

[0168] In some optional implementations of this embodiment, the write module is further configured to: in response to determining that the local storage area where the node corresponding to the target write shard is located has stored the identification information of the target write shard, write the data to be written into the corresponding node.

[0169] According to an embodiment of the present disclosure, the present disclosure also provides an electronic device, a readable storage medium, and a computer program product.

[0170] Figure 10 A schematic block diagram of an example electronic device 1000 that can be used to implement embodiments of the present disclosure is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital assistants, cellular phones, smartphones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present disclosure described and / or claimed herein.

[0171] like Figure 10 As shown, the device 1000 includes a computing unit 1001, which can perform various appropriate actions and processes according to a computer program stored in a read-only memory (ROM) 1002 or a computer program loaded from a storage unit 1008 into a random access memory (RAM) 1003. Various programs and data required for the operation of the device 1000 can also be stored in the RAM 1003. The computing unit 1001, the ROM 1002, and the RAM 1003 are connected to each other via a bus 1004. An input / output (I / O) interface 1005 is also connected to the bus 1004.

[0172] Various components in device 1000 are connected to I / O interface 1005, including an input unit 1006, such as a keyboard, mouse, etc.; an output unit 1007, such as various types of displays, speakers, etc.; a storage unit 1008, such as a magnetic disk, optical disk, etc.; and a communication unit 1009, such as a network card, modem, wireless communication transceiver, etc. The communication unit 1009 allows device 1000 to exchange information / data with other devices via a computer network such as the Internet and / or various telecommunication networks.

[0173] The computing unit 1001 can be a variety of general and / or special processing components with processing and computing capabilities. Some examples of the computing unit 1001 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various dedicated artificial intelligence (AI) computing chips, various computing units that run machine learning model algorithms, digital signal processors (DSPs), and any appropriate processors, controllers, microcontrollers, etc. The computing unit 1001 performs the various methods and processes described above, such as the data processing method. For example, in some embodiments, the data processing method can be implemented as a computer software program that is tangibly contained in a machine-readable medium, such as the storage unit 1008. In some embodiments, part or all of the computer program can be loaded and / or installed on the device 1000 via the ROM 1002 and / or the communication unit 1009. When the computer program is loaded into the RAM 1003 and executed by the computing unit 1001, one or more steps of the data processing method described above can be performed. Alternatively, in other embodiments, the computing unit 1001 can be configured to perform the data processing method in any other appropriate manner (e.g., by means of firmware).

[0174] Various embodiments of the systems and techniques described herein can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), system-on-chip systems (SOCs), programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments can include being implemented in one or more computer programs that are executable and / or interpreted on a programmable system that includes at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0175] The program code for implementing the method of the present disclosure can be written in any combination of one or more programming languages. These program codes can be provided to a processor or controller of a general-purpose computer, a special-purpose computer, or other programmable data processing device so that when the program code is executed by the processor or controller, the functions / operations specified in the flow chart and / or block diagram are implemented. The program code can be executed entirely on the machine, partially on the machine, as a stand-alone software package, partially on the machine and partially on a remote machine, or entirely on a remote machine or server.

[0176] In the context of the present disclosure, a machine-readable medium can be a tangible medium that can contain or store a program for use by or in conjunction with an instruction execution system, device or equipment. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium can include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or equipment, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium can include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0177] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user can provide input to the computer. Other types of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including acoustic input, voice input, or tactile input).

[0178] The systems and techniques described herein can be implemented in a computing system that includes back-end components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes front-end components (e.g., a user computer having a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such back-end components, middleware components, or front-end components. The components of the system can be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include a local area network (LAN), a wide area network (WAN), and the Internet.

[0179] A computer system may include a client and a server. The client and server are generally remote from each other and typically interact through a communication network. The client-server relationship arises through computer programs running on the respective computers and having a client-server relationship with each other. The server may be a cloud server, a server in a distributed system, or a server integrated with a blockchain.

[0180] It should be understood that the various forms of the processes shown above can be used to reorder, add, or delete steps. For example, the steps described in this disclosure can be performed in parallel, sequentially, or in a different order, as long as the desired results of the technical solutions disclosed in this disclosure can be achieved. This is not limited herein.

[0181] The above specific embodiments do not constitute a limitation on the scope of protection of this disclosure. Those skilled in the art will appreciate that various modifications, combinations, sub-combinations, and substitutions may be made based on design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this disclosure shall be included within the scope of protection of this disclosure.

Claims

1. A data processing method, comprising: Get the target data for making the image package; Slicing the target data based on the metadata of the target data to obtain a plurality of slicing data; Uploading each fragment data to a content distribution network based on identification information of each fragment data; Packing the metadata and each of the fragmented data into a metadata mirror package, and uploading the metadata mirror package to the content distribution network; The method further comprises: In response to receiving the mirror data read request, obtaining the corresponding metadata mirror package, which is recorded as reading the mirror package; According to the file handle of the data to be read, determine the fragment data corresponding to the data to be read from the read mirror package, and record it as the target read fragment; Reading the target read slice from a storage area corresponding to the location of the target read slice based on the identification information of the target read slice, wherein the storage area includes a local storage area and a shared storage area; The step of reading the target read slice from the storage area corresponding to the location of the target read slice based on the identification information of the target read slice includes: In response to determining that the target read shard is not stored in the local storage area, sending a read request to a corresponding node in the shared storage area; Downloading the target read segment from the content distribution network, and storing the target read segment in the shared storage area; The target read segment is read from a corresponding node in the shared storage area based on the identification information of the target read segment.

2. The method according to claim 1, wherein The target data is fragmented based on the metadata of the target data to obtain a plurality of fragmented data, including: determining metadata and file data from the target data; According to the attribute information in the metadata, the file data is fragmented according to a preset size to obtain the multiple fragmented data.

3. The method according to claim 2, wherein: The uploading of each fragment data to a content distribution network based on identification information of each fragment data includes: Calculate the hash value of each shard data using a hash algorithm, and record the hash value as the identification information of the shard data; For each piece of data, determining whether the content distribution network has stored the piece of data based on the identification information of the piece of data; In response to determining that the content distribution network does not store the segment data, the segment data is stored in the content distribution network.

4. The method according to claim 1, wherein The step of reading the target read slice from a storage area corresponding to a location of the target read slice based on the identification information of the target read slice includes: In response to determining that the target read shard is stored in the local storage area, determining a target read node storing the target read shard; The target read shard is read from the target read node based on the identification information of the target read shard.

5. The method according to claim 1, further comprising: In response to receiving the mirror data write request, obtaining a corresponding metadata mirror package, which is recorded as a write mirror package; According to the file handle of the data to be written, determine the shard data corresponding to the data to be written from the write mirror package, and record it as the target write shard; Determine whether the storage area where the node corresponding to the target write slice is located has stored the identification information of the target write slice, and write the data to be written based on the determination result.

6. The method according to claim 5, wherein: The step of writing the data to be written based on the judgment result includes: In response to determining that the local storage area where the node corresponding to the target write slice is located does not store the identification information of the target write slice, writing the target write slice to a new node in the local storage area; The identification information of the new node is stored in the write mirror package.

7. The method according to claim 5, wherein: The step of writing the data to be written based on the judgment result includes: In response to determining that the local storage area where the node corresponding to the target write slice is located has stored the identification information of the target write slice, the data to be written is written into the corresponding node.

8. A data processing device comprising: An acquisition module is configured to acquire target data for making an image package; A sharding module is configured to shard the target data based on the metadata of the target data to obtain a plurality of sharded data; an uploading module configured to upload each fragment data to a content distribution network based on identification information of each fragment data; a packaging module configured to package the metadata and each of the fragmented data into a metadata mirror package, and upload the metadata mirror package to the content distribution network; Wherein, the device further includes: a read request receiving module, configured to obtain a corresponding metadata mirror package in response to receiving a mirror data read request, which is recorded as a read mirror package; A read fragment determination module is configured to determine fragment data corresponding to the data to be read from the read mirror package according to the file handle of the data to be read, and record it as a target read fragment; a reading module configured to read the target read segment from a storage area corresponding to a location of the target read segment based on identification information of the target read segment, wherein the storage area includes a local storage area and a shared storage area; Wherein, the reading module is further configured to: In response to determining that the target read shard is not stored in the local storage area, sending a read request to a corresponding node in the shared storage area; Downloading the target read segment from the content distribution network, and storing the target read segment in the shared storage area; The target read segment is read from a corresponding node in the shared storage area based on the identification information of the target read segment.

9. The device according to claim 8, wherein The sharding module is further configured to: determining metadata and file data from the target data; According to the attribute information in the metadata, the file data is fragmented according to a preset size to obtain the multiple fragmented data.

10. The device according to claim 9, wherein The upload module is further configured to: Calculate the hash value of each shard data using a hash algorithm, and record the hash value as the identification information of the shard data; For each piece of data, determining whether the content distribution network has stored the piece of data based on the identification information of the piece of data; In response to determining that the content distribution network does not store the segment data, the segment data is stored in the content distribution network.

11. The device according to claim 8, wherein The reading module is further configured to: In response to determining that the target read shard is stored in the local storage area, determining a target read node storing the target read shard; The target read shard is read from the target read node based on the identification information of the target read shard.

12. The apparatus according to claim 8, further comprising: a write request receiving module configured to, in response to receiving a mirror data write request, obtain a corresponding metadata mirror package, recorded as a write mirror package; A write fragment determination module is configured to determine fragment data corresponding to the data to be written from the write mirror package according to the file handle of the data to be written, and record it as a target write fragment; The writing module is configured to determine whether the storage area where the node corresponding to the target write slice is located has stored the identification information of the target write slice, and write the data to be written based on the determination result.

13. The device according to claim 12, wherein The writing module is further configured to: In response to determining that the local storage area where the node corresponding to the target write slice is located does not store the identification information of the target write slice, writing the target write slice to a new node in the local storage area; The identification information of the new node is stored in the write mirror package.

14. The device according to claim 12, wherein The writing module is further configured to: In response to determining that the local storage area where the node corresponding to the target write slice is located has stored the identification information of the target write slice, the data to be written is written into the corresponding node.

15. An electronic device comprising: at least one processor; as well as a memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the method according to any one of claims 1 to 7.

16. A non-transitory computer-readable storage medium storing computer instructions, the computer instructions being configured to cause the computer to execute the method according to any one of claims 1 to 7.

17. A computer program product comprising a computer program, which, when executed by a processor, implements the method according to any one of claims 1 to 7.

Images