Data processing method and device, storage medium and electronic equipment

Abstract

The application discloses a kind of data processing method and device related to the container of computer, storage medium and electronic equipment. Among them, the method comprises: in response to the container creation request obtained, create the container of target graphics processor;First level data of target image file is obtained from the data pre-deployed locally in terminal, and the second level data of target image file except first level data is obtained from server, wherein the target image file is one of the plurality of image files corresponding to the service provided by the server, the file size of each of the plurality of image files is greater than the first threshold value, and the pre-deployed data is the level data included in at least two of the plurality of image files;First level data and second level data are processed in the container of target graphics processor to obtain the processing result.The application solves the technical problem of low data processing efficiency.

Description

Technical Field

[0001] This invention relates to the field of computer computing, and more specifically, to a data processing method and apparatus, a storage medium, and an electronic device. Background Technology

[0002] In existing technologies, it is typically necessary to create a graphics processing unit (GPU) container locally on the terminal device. The GPU container then provides computing power to the image file on the server. In practice, the GPU container is created locally on the terminal device, the image file is pulled from the server to the terminal device, and then the GPU container processes the image file to obtain the processing result.

[0003] However, since image files are usually quite large, the above methods result in long image file retrieval times and high failure rates when network conditions are poor, further leading to low data processing efficiency.

[0004] There is currently no effective solution to the above problems. Summary of the Invention

[0005] This invention provides a data processing method and apparatus, a storage medium and an electronic device to at least solve the technical problem of low data processing efficiency.

[0006] According to one aspect of the present invention, a data processing method is provided, comprising: responding to a received container creation request, creating a container for a target graphics processor, wherein the container for the target graphics processor is used to process a target image file; obtaining first-level data of the target image file from pre-deployed data on a terminal, and obtaining second-level data of the target image file other than the first-level data from a server, wherein the target image file is one of a plurality of image files corresponding to a service provided by the server, each of the plurality of image files has a file size greater than a first threshold, and the pre-deployed data is hierarchical data included in at least two of the plurality of image files; and processing the first-level data and the second-level data in the container of the target graphics processor to obtain a processing result.

[0007] According to another aspect of the present invention, a data processing method is provided, comprising: obtaining a data acquisition request sent by a terminal, wherein the data acquisition request is used to request the acquisition of second-level data of a target image file, wherein the target image file is one of a plurality of image files corresponding to a service provided by a server, the file size of each of the plurality of image files is greater than a first threshold, the second-level data is data in the target image file other than the first-level data, the first-level data is data pre-deployed on the terminal, and the data pre-deployed on the terminal is the level data included in at least two of the plurality of image files; and sending the second-level data to the terminal, so that the terminal processes the first-level data acquired locally on the terminal and the second-level data acquired from the server through a created GPU container.

[0008] According to another aspect of the present invention, a data processing apparatus is also provided, comprising: a creation unit, configured to create a container for a target graphics processor in response to an acquired container creation request, wherein the container for the target graphics processor is used to process a target image file; a first acquisition unit, configured to acquire first-level data of the target image file from pre-deployed data on a terminal, and acquire second-level data of the target image file other than the first-level data from a server, wherein the target image file is one of a plurality of image files corresponding to a service provided by the server, each of the plurality of image files has a file size greater than a first threshold, and the pre-deployed data is the hierarchical data included in at least two of the plurality of image files; and a processing unit, configured to process the first-level data and the second-level data in the container of the target graphics processor to obtain a processing result.

[0009] As an optional example, the first acquisition unit includes: a parsing module, configured to parse the file identifier of the target image file to be processed by the container of the target graphics processor from the container creation request; a first search module, configured to search for the first-level data of the target image file corresponding to the file identifier in the pre-deployed data on the terminal; a pull module, configured to pull the found first-level data if the first-level data is found; a first sending module, configured to send a data acquisition request to the server, wherein the data acquisition request is used to request the acquisition of the second-level data of the target image file other than the first-level data; and an acquisition module, configured to acquire the second-level data from the server.

[0010] As an optional example, the first search module includes: an acquisition submodule for acquiring the data hierarchy relationship in the target image file; and a processing submodule for determining each layer in the data hierarchy relationship as the current layer, and performing the following operations on the current layer: searching for the data corresponding to the current layer in the data pre-deployed locally on the terminal; and if the data corresponding to the current layer is found, determining the data corresponding to the current layer as the data in the first level of data.

[0011] As an optional example, the above processing submodule is also used to: after searching for the data corresponding to the current layer in the data pre-deployed locally on the terminal, if the data corresponding to the current layer is not found, determine the data corresponding to the current layer as the data in the second-level data.

[0012] As an optional example, the above apparatus further includes: a second acquisition unit, configured to acquire, before creating a container for the target graphics processor in response to an acquired container creation request, hierarchical data included in at least two of the plurality of image files from the server, provided that the real-time data transmission volume between the server and the local terminal is less than a second threshold; and a deployment unit, configured to deploy the hierarchical data included in at least two of the plurality of image files as the pre-deployed data on the local terminal.

[0013] As an optional example, the second acquisition unit includes: a second sending module, configured to send a data deployment request to the server; acquire hierarchical data included in both of the at least two image files sent by the server in response to the data deployment request; or a receiving module, configured to receive the hierarchical data included in both of the at least two image files sent by the server.

[0014] As an optional example, the above apparatus further includes: a second acquisition unit, configured to process the first-level data and the second-level data in the container of the target graphics processor, and after obtaining the processing result, acquire the changed first-level data from the server if the first-level data in the server changes; and a storage unit, configured to store the changed first-level data locally on the terminal.

[0015] As an optional example, the aforementioned storage unit includes: a monitoring module for monitoring the remaining storage space of the local storage space of the terminal used to store the first-level data; a second search module for searching for the latest first-level data in the storage space when the remaining storage space is less than a third threshold; and a deletion module for deleting all first-level data other than the latest first-level data from the storage space; or deleting the first-level data with the longest storage time among all first-level data other than the latest first-level data from the storage space.

[0016] According to another aspect of the present invention, a data processing apparatus is also provided, comprising: an acquisition unit, configured to acquire a data acquisition request sent by a terminal, wherein the data acquisition request is used to request acquisition of second-level data of a target image file, wherein the target image file is one of a plurality of image files corresponding to a service provided by a server, the file size of each of the plurality of image files is greater than a first threshold, the second-level data is data in the target image file excluding the first-level data, the first-level data is data pre-deployed on the terminal, and the data pre-deployed on the terminal is the level data included in at least two of the plurality of image files; and a sending unit, configured to send the second-level data to the terminal, so that the terminal processes the first-level data acquired locally on the terminal and the second-level data acquired from the server through a created GPU container.

[0017] As an optional example, the sending unit includes: an acquisition module, configured to acquire hierarchical data included in at least two of the plurality of image files from the server when the real-time data transmission volume between the server and the terminal is less than a second threshold; and a deployment module, configured to deploy the hierarchical data included in at least two of the plurality of image files as the data pre-deployed to the terminal on the terminal.

[0018] As an optional example, the acquisition module includes: a processing submodule, configured to receive a data deployment request sent by the terminal; respond to the data deployment request by sending the hierarchical data included in both of the at least two image files to the terminal; or send the hierarchical data included in both of the at least two image files to the terminal.

[0019] According to another aspect of the present invention, a computer-readable storage medium is also provided, wherein a computer program is stored in the computer program, and the computer program is configured to execute the above-described data processing method at runtime.

[0020] According to another aspect of the present invention, an electronic device is also provided, including a memory and a processor, wherein the memory stores a computer program and the processor is configured to execute the data processing method described above through the computer program.

[0021] In this embodiment of the invention, a container for a target graphics processor is created in response to a received container creation request. This container is used to process a target image file. The method involves obtaining first-level data of the target image file from pre-deployed data on the terminal and obtaining second-level data (excluding the first-level data) from the server. The target image file is one of multiple image files corresponding to a service provided by the server. Each of the multiple image files has a file size greater than a first threshold. The pre-deployed data consists of hierarchical data included in at least two of the multiple image files. The first-level and second-level data are then processed within the target graphics processor container to obtain a processing result. Because the first-level data of the target image file can be pre-deployed on the terminal, after creating the GPU container, when processing the target image file, the first-level data can be obtained from the terminal, and the second-level data from the server. This reduces the amount of data pulled from the server, improves the efficiency of pulling image files, further enhances data processing efficiency, and solves the technical problem of low data processing efficiency. Attached Figure Description

[0022] The accompanying drawings, which are included to provide a further understanding of the invention and form part of this application, illustrate exemplary embodiments of the invention and, together with their description, serve to explain the invention and do not constitute an undue limitation thereof. In the drawings:

[0023] Figure 1 This is a schematic diagram of an application environment for an optional data processing method according to an embodiment of the present invention;

[0024] Figure 2 This is a schematic diagram of an optional data processing method according to an embodiment of the present invention;

[0025] Figure 3 This is a schematic diagram of pre-deployment hierarchical data in an optional data processing method according to an embodiment of the present invention;

[0026] Figure 4 This is a schematic diagram illustrating the processing of first-level data and second-level data according to an optional data processing method based on an embodiment of the present invention.

[0027] Figure 5This is a schematic diagram of another optional data processing method according to an embodiment of the present invention;

[0028] Figure 6 This is a schematic diagram of the structure of an optional data processing apparatus according to an embodiment of the present invention;

[0029] Figure 7 This is a schematic diagram of the structure of another optional data processing device according to an embodiment of the present invention;

[0030] Figure 8 This is a schematic diagram of the structure of an optional electronic device according to an embodiment of the present invention.

[0031] Figure 9 This is a schematic diagram of the structure of another optional electronic device according to an embodiment of the present invention. Detailed Implementation

[0032] To enable those skilled in the art to better understand the present invention, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of the present invention.

[0033] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this invention are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of the invention described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover a non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0034] According to one aspect of the present invention, a data processing method is provided. Optionally, as an alternative implementation, the above-described data processing method may be applied to, but is not limited to, [examples of other methods]. Figure 1 In the environment shown.

[0035] like Figure 1As shown, the terminal device 102 includes a memory 104 for storing various data generated during the operation of the terminal device 102, a processor 106 for processing and calculating the aforementioned data, and a display 108 for displaying the processing results. The terminal device 102 can interact with the server 112 via a network 110. The server 112 includes a database 114 for storing various data and a processing engine 116 for processing the aforementioned data. Through steps S102 to S106, the terminal device 102 can obtain first-level data locally and second-level data from the server 112, and use a GPU container created on the terminal device 102 to process the first-level data and the second-level data to obtain the processing results.

[0036] Optionally, in this embodiment, the terminal device can be a terminal device configured with a GPU, and may include, but is not limited to, at least one of the following: mobile phone (such as Android phone, iOS phone, etc.), laptop computer, tablet computer, PDA, MID (Mobile Internet Device), PAD, desktop computer, smart TV, etc. The target client can be a video client, instant messaging client, browser client, educational client, etc. The network can include, but is not limited to, wired network and wireless network, wherein the wired network includes: local area network, metropolitan area network and wide area network, and the wireless network includes: Bluetooth, WIFI and other networks that enable wireless communication. The server can be a single server, a server cluster composed of multiple servers, or a cloud server. The above is only an example, and no limitation is made in this embodiment.

[0037] The server can be a standalone physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server providing basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, CDN, and big data and artificial intelligence platforms. The terminal can be a smartphone, tablet, laptop, desktop computer, smart speaker, smartwatch, etc., but is not limited to these. The terminal and server can be connected directly or indirectly through wired or wireless communication methods, which is not limited herein.

[0038] In this embodiment, the terminal device or server can be a node in the blockchain. The blockchain in this embodiment is a novel application model of computer technologies such as distributed data storage, peer-to-peer transmission, consensus mechanisms, and encryption algorithms. Essentially, a blockchain is a decentralized database, a chain of data blocks linked using cryptographic methods. Each data block contains information about a batch of network transactions, used to verify the validity of the information (anti-counterfeiting) and generate the next block. A blockchain can include a blockchain underlying platform, a platform product service layer, and an application service layer.

[0039] The underlying blockchain platform can include processing modules such as user management, basic services, smart contracts, and operational monitoring. The user management module is responsible for managing the identity information of all blockchain participants, including maintaining public and private key generation (account management), key management, and maintaining the correspondence between user real identities and blockchain addresses (access management). Furthermore, under authorization, it monitors and audits transactions of certain real identities and provides risk control rule configuration (risk control audit). The basic services module is deployed on all blockchain node devices to verify the validity of business requests. After consensus is reached on valid requests, they are recorded in storage. For a new business request, the basic services first perform interface adaptation parsing and authentication (interface adaptation), and then encrypt the business information through a consensus algorithm (consensus management). After encryption, the data is transmitted completely and consistently to the shared ledger (network communication) and recorded and stored. The smart contract module is responsible for contract registration, issuance, triggering, and execution. Developers can define contract logic using a programming language and publish it to the blockchain (contract registration). According to the contract terms, the key or other events are invoked to trigger execution and complete the contract logic. It also provides functions for contract upgrades and cancellations. The operation monitoring module is mainly responsible for deployment, configuration modification, contract settings, cloud adaptation, and real-time status visualization output during product release, such as alarms, monitoring network conditions, and monitoring the health status of node devices.

[0040] The platform's product service layer provides the basic capabilities and implementation frameworks for typical applications. Developers can leverage these basic capabilities, along with the specific characteristics of their business needs, to implement blockchain-based business logic. The application service layer provides blockchain-based application services for use by business stakeholders.

[0041] Alternatively, as an alternative implementation method, such as Figure 2 As shown, the above data processing method includes:

[0042] S202, In response to the received container creation request, create a container for the target graphics processor, wherein the container for the target graphics processor is used to process the target image file;

[0043] S204, obtain the first-level data of the target image file from the pre-deployed data on the terminal, and obtain the second-level data of the target image file other than the first-level data from the server. The target image file is one of multiple image files corresponding to the service provided by the server. The file size of each image file in the multiple image files is greater than a first threshold. The pre-deployed data is the hierarchical data included in at least two image files in the multiple image files.

[0044] S206, process the first-level data and the second-level data in the container of the target graphics processor to obtain the processing result.

[0045] Optionally, embodiments of this application can be applied, but are not limited to, to the process of providing services using a cloud server. For example, when a user uses a service provided by a cloud server, a GPU container needs to be created on a device used to create GPU containers. The GPU container processes the image file used to provide services on the cloud server and then obtains the processing result. The service content and type provided by the cloud server are not limited. For example, taking the provided service as an Artificial Intelligence (AI) service, the image file of the AI ​​service needs to be pulled to the terminal where the GPU container is created so that the GPU container can process the image file of the AI ​​service and obtain the processing result. In this embodiment, after the GPU container is created on the terminal, the first-level data of the image file can be obtained from the locally pre-deployed data, and the second-level data other than the first-level data can be obtained from the server. The GPU container can then process the first-level data and the second-level data. It is not necessary to pull all the level data of the image file from the server to the local terminal when providing services to the user. Instead, the second-level data that is not pre-deployed locally on the terminal is pulled from the server to the local terminal, which greatly reduces the amount of image file data pulled during the service provision process, improves the pulling efficiency, and further improves the data processing efficiency.

[0046] Of course, the type of service in this embodiment is not limited. The service can be any service that requires GPU data processing. It can be a function possessed by the client, such as services for taking photos or editing images.

[0047] Taking image editing as an example, for the image editing function's source file image file, the first level data of the image file can be deployed to the local terminal first. After creating a GPU container on the terminal, the first level data is obtained from the local terminal. Then, the second level data of the image file of the image editing function's source file is obtained from the server. Then, the first level data and the second level data are processed to obtain the editing result.

[0048] Optionally, in this embodiment, the GPU container can be a virtual container created on the terminal device for processing image files. After being created, the GPU container processes the image files, and after processing the image files, the GPU container can be destroyed. However, if the created GPU container is a reusable GPU container, it can wait to process another image file after processing the first one.

[0049] Optionally, the image file in this embodiment can be a mirror of the source file used to provide the service. The image file has exactly the same content as the source file, but the storage location can be different. By acquiring or processing the image file, the source file will not be affected.

[0050] Through this embodiment and the above method, after creating the GPU container, when obtaining data from the target image file, the first-level data can be obtained from the local terminal and the second-level data can be obtained from the server. This reduces the amount of data obtained from the server, improves the efficiency of pulling the image file, and further improves the efficiency of data processing.

[0051] As an optional example, obtaining the first-level data of the target image file from the pre-deployed data on the terminal, and obtaining the second-level data of the target image file in addition to the first-level data from the server, includes:

[0052] Parse the file identifier of the target image file that the container of the target graphics processor is to process from the container creation request;

[0053] Search for the first-level data of the target image file corresponding to the file identifier in the data pre-deployed locally on the terminal;

[0054] If the first level of data is found, retrieve the found first level of data;

[0055] Send a data retrieval request to the server, wherein the data retrieval request is used to request the second-level data of the target image file in addition to the first-level data;

[0056] Retrieve second-level data from the server.

[0057] Optionally, the file identifier in this embodiment can be an identifier used to uniquely identify the image file. It can use any one or more of characters, numbers, and letters for identification. For example, the identifier of the target image file can be 0001.

[0058] Optionally, in this embodiment, each layer of data in the target image file and each of the multiple image files can correspond to a layer identifier. The file identifier can include a layer identifier. For example, one of the file identifiers of the target image file is 0001-12. Here, 0001 represents the image file, and -12 represents the layer of the image file. Of course, the above is merely an example and is not intended to limit this application.

[0059] Once the file identifier of the target image file is obtained, the hierarchical data included in the target image file can be determined. The file identifier is used to check the pre-deployment data on the terminal's local machine to see if it includes the hierarchical data from the target image file, i.e., the first-level data. If the first-level data is included, it is not necessary to retrieve the first-level data from the server. Then, the second-level data is retrieved from the server, thus avoiding the need to retrieve a large image file from the server and improving the efficiency of pulling the image file.

[0060] As an optional example, the first-level data for finding the target image file corresponding to the file identifier in the locally pre-deployed data on the terminal includes:

[0061] Obtain the data hierarchy in the target image file;

[0062] Define each level in the data hierarchy as the current level, and perform the following operations on the current level:

[0063] Search for the data corresponding to the current layer in the pre-deployed data on the terminal;

[0064] If the data corresponding to the current layer is found, the data corresponding to the current layer is determined as the data in the first level of data.

[0065] Optionally, in this embodiment, the data of the target image file can be organized into layers. For each layer, the data is checked against the pre-deployed data on the terminal to see if it is included. If the layer is found locally on the terminal, it does not need to be retrieved from the server. Layers not found locally on the terminal are designated as second-level data, and these second-level data are retrieved from the server.

[0066] As an optional example, before creating the container for the target graphics processor in response to the received container creation request, the following may also be included:

[0067] If the real-time data transmission volume between the server and the local terminal is less than the second threshold, obtain the hierarchical data included in at least two of the multiple image files from the server.

[0068] Deploy the hierarchical data, which is included in at least two of the multiple image files, as pre-deployed data on the local terminal.

[0069] Optionally, in this embodiment, data included in at least two of the multiple image files on the server can be pre-deployed to the local terminal. For example, there are three image files: image file 1, image file 2, and image file 3. Image file 1 includes hierarchical data 1 and hierarchical data 3; image file 2 includes hierarchical data 1 and hierarchical data 4; and image file 3 includes hierarchical data 1, hierarchical data 3, and hierarchical data 5. Then, hierarchical data 1 and hierarchical data 3 will be pre-deployed to the local terminal.

[0070] In this embodiment, data can be pre-deployed during periods of low network traffic. Real-time data transmission volume between the local terminal and the server can be monitored. If the data transmission volume is below the threshold, the current traffic is considered low, and the hierarchical data included in at least two image files can be sent to the terminal for pre-deployment.

[0071] As an optional example, if the real-time data transfer volume between the server and the local terminal is less than the second threshold, the hierarchical data included in at least two of the multiple image files obtained from the server includes:

[0072] Send a data deployment request to the server; obtain the hierarchical data included in at least two image files sent by the server in response to the data deployment request; or

[0073] The receiving server sends hierarchical data that is included in at least two image files.

[0074] In other words, in this embodiment, the terminal can actively request the hierarchical data included in at least two image files from the server and then deploy it locally, or it can passively receive the hierarchical data included in at least two image files sent by the server and then deploy it locally. This embodiment is not limited to this.

[0075] As an optional example, after processing the first-level and second-level data within the target graphics processor's container to obtain the processing result, the following is also included:

[0076] If the first-level data in the server changes, retrieve the changed first-level data from the server.

[0077] The modified first-level data is saved locally on the terminal.

[0078] Optionally, in this embodiment, the image file on the server side may change. When the source file of the image file changes, the image file will also change. After the image file changes, the data included in at least two image files will also change. Therefore, it is necessary to obtain the new data included in at least two image files and then save the new data included in at least two image files locally on the terminal to ensure the accuracy of the first-level data obtained from the terminal.

[0079] As an optional example, storing the changed first-level data locally on the terminal includes:

[0080] The remaining storage space on the monitoring terminal's local storage space used to store the first-level data;

[0081] If the remaining storage space is less than the third threshold, search for the latest first-level data in the storage space;

[0082] Delete all first-level data except the latest first-level data from the storage space; or delete the first-level data with the longest storage time among all first-level data except the latest first-level data from the storage space.

[0083] Optionally, if the remaining storage space used to store the first-level data is less than the third threshold, it indicates that the remaining storage space is insufficient. In this case, older versions of the data in the storage space can be deleted, thereby increasing the remaining storage space.

[0084] Figure 3 This is a schematic diagram of a pre-deployment hierarchical data in this embodiment.

[0085] The system architecture is divided into hosts (terminals) and registry image servers. The logical software layer has various layers of containers and images, and there is pre-deployment in the process functions.

[0086] Host: In this solution, the host is specifically identified as the physical device of the GPU. This physical device will be connected to 8 GPU training cards to serve the offline training scenario of AI.

[0087] Registry Image Server: An image source service for offline AI training, storing AI training images and supporting bridgehead deployment based on proximity.

[0088] Container: A dynamic runtime of an image. User applications run inside the container. It has an independent runnable system state but shares the host kernel.

[0089] Image: A key module in this solution, the image layer consists of multiple layers, from bottom to top: operating system layer, system software layer, third-party dependencies (if any), and user application software layer. In the AI ​​training scenario of the computing power platform, the operating system layer uses a unified Linux operating system and a stable kernel version. The system software layer has multiple versions due to different NVIDIA drivers or CUDA library versions, but the number is enumerable. In the AI ​​training image of the unified computing power platform, over 60% of the image size is occupied by the operating system and system software layers, and the file organization of each layer is basically the same. Pre-deployment operations can be performed before container creation, batching and performing pull operations on the operating system and system software layers in container environments across multiple hosts during off-peak traffic periods. If there are version updates for the operating system and system software layers, then a pull operation is performed; older versions of the layers are not deleted. If the available image storage space on the host's local disk is insufficient, then a clean deletion operation is performed.

[0090] Pre-deployment: The pre-pull process for operating system and system software layers, that is, actively pulling from the registry image server cluster in the container environment of hosts, and the pull operation supports incremental pulling.

[0091] like Figure 3 As shown, both image file 1 and image file 2 on the server include hierarchical data 1 and hierarchical data 2. Therefore, hierarchical data 1 and hierarchical data 2 are pre-deployed on the terminal. When the terminal creates a GPU container to process image file 1, it can obtain hierarchical data 3 and hierarchical data 4 from the server and hierarchical data 1 to 2 from the terminal's local storage. When the terminal creates a GPU container to process image file 2, it can obtain hierarchical data 5 from the server and hierarchical data 1 to 2 from the terminal's local storage.

[0092] Figure 4 This is a schematic diagram illustrating one method of processing first-level and second-level data in this embodiment.

[0093] When a container is created on a specific host device, during the image pull phase, since the image is layered, the system queries whether each layer of the image exists. If a layer exists, the query continues for the next layer; otherwise, the image is pulled from the image server. Once all layers of the image exist, the runtime state of the image is built from the image, i.e., the container is started. Therefore, to reduce container creation latency, the host device pre-deploys the operating system layer and system software layer, and these two layers account for more than 60% of the image size. If the pre-deployed layers are selected, only the application software layer needs to be pulled to prepare all the layers of the image.

[0094] like Figure 4 Steps S402 to S412 involve obtaining pre-deployed hierarchical data after container creation. It is determined whether the hierarchical data for each layer of the target image file to be processed by the container exists locally on the terminal. If it exists, the created container processes the hierarchical data for each layer of the target image file. If the target image file contains hierarchical data that is not included locally on the terminal, the hierarchical data that does not exist locally on the terminal is requested from the server and deployed locally on the terminal so that the GPU container can process the target image file.

[0095] This embodiment decouples the image pulling time and advances the image pulling process. Especially when the image size is too large, it promotes the reduction of container creation time and the improvement of success rate. The reduced container creation critical path is also helpful for finding and locating problems after creation failure. The pre-deployment of the image operating system and system software layers can be selected during the low period of network traffic to avoid superimposing the peak traffic of normal business operations and ensure the stability of business operation.

[0096] According to another aspect of the present invention, a data processing method is also provided. For example... Figure 5 As shown, the method includes:

[0097] S502, Obtain a data acquisition request sent by the terminal, wherein the data acquisition request is used to request the acquisition of the second-level data of the target image file, wherein the target image file is one of multiple image files corresponding to the service provided by the server, the file size of each of the multiple image files is greater than a first threshold, the second-level data is the data in the target image file other than the first-level data, the first-level data is the data pre-deployed to the terminal, and the data pre-deployed to the terminal is the level data included in at least two of the multiple image files;

[0098] S504 sends the second-level data to the terminal so that the terminal can process the first-level data obtained locally from the terminal and the second-level data obtained from the server through the created GPU container.

[0099] Optionally, embodiments of this application can be applied, but are not limited to, to the process of providing services using a cloud server. For example, when a user uses a service provided by a cloud server, a GPU container needs to be created on a device used to create GPU containers. The GPU container processes the image file used to provide services on the cloud server and then obtains the processing result. The service content and type provided by the cloud server are not limited. For example, taking the provided service as an Artificial Intelligence (AI) service, the image file of the AI ​​service needs to be pulled to the terminal where the GPU container is created so that the GPU container can process the image file of the AI ​​service and obtain the processing result. In this embodiment, after the GPU container is created on the terminal, the first-level data of the image file can be obtained from the locally pre-deployed data, and the second-level data other than the first-level data can be obtained from the server. The GPU container can then process the first-level data and the second-level data. It is not necessary to pull all the level data of the image file from the server to the local terminal when providing services to the user. Instead, the second-level data that is not pre-deployed locally on the terminal is pulled from the server to the local terminal, which greatly reduces the amount of image file data pulled during the service provision process, improves the pulling efficiency, and further improves the data processing efficiency.

[0100] Of course, the type of service in this embodiment is not limited. The service can be any service that requires GPU data processing. It can be a function possessed by the client, such as services for taking photos or editing images.

[0101] For other examples of this embodiment, please refer to the examples above, which will not be repeated here.

[0102] It should be noted that, for the sake of simplicity, the foregoing method embodiments are all described as a series of actions. However, those skilled in the art should understand that the present invention is not limited to the described order of actions, because according to the present invention, some steps can be performed in other orders or simultaneously. Furthermore, those skilled in the art should also understand that the embodiments described in the specification are preferred embodiments, and the actions and modules involved are not necessarily essential to the present invention.

[0103] According to another aspect of the present invention, a data processing apparatus for implementing the above-described data processing method is also provided. For example... Figure 6 As shown, the device includes:

[0104] Creation unit 602 is used to respond to the obtained container creation request and create a container for the target graphics processor, wherein the container for the target graphics processor is used to process the target image file;

[0105] The first acquisition unit 604 is used to acquire the first-level data of the target image file from the pre-deployed data on the terminal, and acquire the second-level data of the target image file other than the first-level data from the server. The target image file is one of multiple image files corresponding to the service provided by the server. The file size of each image file in the multiple image files is greater than a first threshold. The pre-deployed data is the hierarchical data included in at least two image files in the multiple image files.

[0106] The processing unit 606 is used to process the first-level data and the second-level data in the container of the target graphics processor to obtain the processing result.

[0107] Optionally, embodiments of this application can be applied, but are not limited to, to the process of providing services using a cloud server. For example, when a user uses a service provided by a cloud server, a GPU container needs to be created on a device used to create GPU containers. The GPU container processes the image file used to provide services on the cloud server and then obtains the processing result. The service content and type provided by the cloud server are not limited. For example, taking the provided service as an Artificial Intelligence (AI) service, the image file of the AI ​​service needs to be pulled to the terminal where the GPU container is created so that the GPU container can process the image file of the AI ​​service and obtain the processing result. In this embodiment, after the GPU container is created on the terminal, the first-level data of the image file can be obtained from the locally pre-deployed data, and the second-level data other than the first-level data can be obtained from the server. The GPU container can then process the first-level data and the second-level data. It is not necessary to pull all the level data of the image file from the server to the local terminal when providing services to the user. Instead, the second-level data that is not pre-deployed locally on the terminal is pulled from the server to the local terminal, which greatly reduces the amount of image file data pulled during the service provision process, improves the pulling efficiency, and further improves the data processing efficiency.

[0108] Of course, the type of service in this embodiment is not limited. The service can be any service that requires GPU data processing. It can be a function possessed by the client, such as services for taking photos or editing images.

[0109] For other examples of this embodiment, please refer to the examples above, which will not be repeated here.

[0110] According to another aspect of the present invention, a data processing apparatus for implementing the above-described data processing method is also provided. For example... Figure 7 As shown, the device includes:

[0111] The acquisition unit 702 is used to acquire a data acquisition request sent by the terminal. The data acquisition request is used to request the acquisition of the second-level data of the target image file. The target image file is one of multiple image files corresponding to the service provided by the server. The file size of each image file in the multiple image files is greater than a first threshold. The second-level data is the data in the target image file other than the first-level data. The first-level data is the data pre-deployed to the terminal. The data pre-deployed to the terminal is the hierarchical data included in at least two of the multiple image files.

[0112] The sending unit 704 is used to send the second-level data to the terminal so that the terminal can process the first-level data obtained locally from the terminal and the second-level data obtained from the server through the created GPU container.

[0113] Optionally, embodiments of this application can be applied, but are not limited to, to the process of providing services using a cloud server. For example, when a user uses a service provided by a cloud server, a GPU container needs to be created on a device used to create GPU containers. The GPU container processes the image file used to provide services on the cloud server and then obtains the processing result. The service content and type provided by the cloud server are not limited. For example, taking the provided service as an Artificial Intelligence (AI) service, the image file of the AI ​​service needs to be pulled to the terminal where the GPU container is created so that the GPU container can process the image file of the AI ​​service and obtain the processing result. In this embodiment, after the GPU container is created on the terminal, the first-level data of the image file can be obtained from the locally pre-deployed data, and the second-level data other than the first-level data can be obtained from the server. The GPU container can then process the first-level data and the second-level data. It is not necessary to pull all the level data of the image file from the server to the local terminal when providing services to the user. Instead, the second-level data that is not pre-deployed locally on the terminal is pulled from the server to the local terminal, which greatly reduces the amount of image file data pulled during the service provision process, improves the pulling efficiency, and further improves the data processing efficiency.

[0114] Of course, the type of service in this embodiment is not limited. The service can be any service that requires GPU data processing. It can be a function possessed by the client, such as services for taking photos or editing images.

[0115] For other examples of this embodiment, please refer to the examples above, which will not be repeated here.

[0116] According to another aspect of the present invention, an electronic device for implementing the above-described data processing method is also provided. This electronic device may be... Figure 8 The terminal device or server shown. This embodiment uses this electronic device as an example for illustration. Figure 8 As shown, the electronic device includes a memory 802 and a processor 804. The memory 802 stores a computer program, and the processor 804 is configured to execute the steps in any of the above method embodiments via the computer program.

[0117] Optionally, in this embodiment, the aforementioned electronic device may be located in at least one of a plurality of network devices in a computer network.

[0118] Optionally, in this embodiment, the processor can be configured to perform the following steps via a computer program:

[0119] In response to the received container creation request, a container for the target graphics processor is created, wherein the container for the target graphics processor is used to process the target image file;

[0120] The first-level data of the target image file is obtained from the pre-deployed data on the terminal, and the second-level data of the target image file other than the first-level data is obtained from the server. The target image file is one of multiple image files corresponding to the service provided by the server. The file size of each image file in the multiple image files is greater than a first threshold. The pre-deployed data is the hierarchical data included in at least two image files in the multiple image files.

[0121] The first-level and second-level data are processed within the container of the target graphics processor to obtain the processing result.

[0122] Alternatively, as those skilled in the art will understand, Figure 8 The structure shown is for illustrative purposes only. Electronic devices can also be smartphones (such as Android phones, iOS phones, etc.), tablets, PDAs, mobile internet devices (MIDs), PADs, and other terminal devices. Figure 8 This does not limit the structure of the aforementioned electronic devices or electronic equipment. For example, electronic devices or electronic equipment may also include components that are more... Figure 8 The more or fewer components shown (such as network interfaces, etc.), or having the same Figure 8 The different configurations shown.

[0123] The memory 802 can be used to store software programs and modules, such as program instructions / modules corresponding to the data processing method and apparatus in this embodiment of the invention. The processor 804 executes various functional applications and data processing by running the software programs and modules stored in the memory 802, thereby implementing the aforementioned data processing method. The memory 802 may include high-speed random access memory, and may also include non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some instances, the memory 802 may further include memory remotely located relative to the processor 804, and these remote memories can be connected to the terminal via a network. Examples of such networks include, but are not limited to, the Internet, corporate intranets, local area networks, mobile communication networks, and combinations thereof. Specifically, the memory 802 may be used for, but is not limited to, pre-deployed data. As an example, such as... Figure 8 As shown, the memory 802 may include, but is not limited to, the creation unit 602, the first acquisition unit 604, and the processing unit 606 in the data processing device. Furthermore, it may include, but is not limited to, other module units in the data processing device, which will not be elaborated upon in this example.

[0124] Optionally, the transmission device 806 described above is used to receive or send data via a network. Specific examples of the network described above may include wired networks and wireless networks. In one example, the transmission device 806 includes a Network Interface Controller (NIC), which can be connected to other network devices and a router via a network cable to communicate with the Internet or a local area network. In another example, the transmission device 806 is a Radio Frequency (RF) module, used for wireless communication with the Internet.

[0125] In addition, the above-mentioned electronic device also includes: a display 808 for displaying processing results; and a connection bus 810 for connecting various module components in the above-mentioned electronic device.

[0126] In other embodiments, the aforementioned terminal device or server can be a node in a distributed system, wherein the distributed system can be a blockchain system, which is a distributed system formed by connecting multiple nodes through network communication. The nodes can form a peer-to-peer (P2P) network, and any form of computing device, such as a server, terminal, or other electronic device, can become a node in the blockchain system by joining this peer-to-peer network.

[0127] According to another aspect of the present invention, an electronic device for implementing the above-described data processing method is also provided. This electronic device may be...Figure 9 The terminal device or server shown. This embodiment uses the electronic device as a server as an example for illustration. Figure 9 As shown, the electronic device includes a memory 902 and a processor 904. The memory 902 stores a computer program, and the processor 904 is configured to execute the steps of any of the above method embodiments through the computer program.

[0128] Optionally, in this embodiment, the aforementioned electronic device may be located in at least one of a plurality of network devices in a computer network.

[0129] Optionally, in this embodiment, the processor can be configured to perform the following steps via a computer program:

[0130] The terminal sends a data acquisition request, wherein the data acquisition request is used to request the acquisition of the second-level data of the target image file, wherein the target image file is one of multiple image files corresponding to the service provided by the server, the file size of each of the multiple image files is greater than a first threshold, the second-level data is the data in the target image file other than the first-level data, the first-level data is the data pre-deployed to the terminal, and the data pre-deployed to the terminal is the level data included in at least two of the multiple image files;

[0131] The second-level data is sent to the terminal so that the terminal can process the first-level data obtained locally from the terminal and the second-level data obtained from the server through the created GPU container.

[0132] Alternatively, as those skilled in the art will understand, Figure 9 The structure shown is for illustrative purposes only. Electronic devices can also be smartphones (such as Android phones, iOS phones, etc.), tablets, PDAs, mobile internet devices (MIDs), PADs, and other terminal devices. Figure 9 This does not limit the structure of the aforementioned electronic devices or electronic equipment. For example, electronic devices or electronic equipment may also include components that are more... Figure 9 The more or fewer components shown (such as network interfaces, etc.), or having the same Figure 9 The different configurations shown.

[0133] The memory 902 can be used to store software programs and modules, such as program instructions / modules corresponding to the data processing method and apparatus in this embodiment of the invention. The processor 904 executes various functional applications and data processing by running the software programs and modules stored in the memory 902, thereby implementing the aforementioned data processing method. The memory 902 may include high-speed random access memory, and may also include non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some instances, the memory 902 may further include memory remotely located relative to the processor 904, and these remote memories can be connected to the terminal via a network. Examples of such networks include, but are not limited to, the Internet, corporate intranets, local area networks, mobile communication networks, and combinations thereof. Specifically, the memory 902 may be used for, but is not limited to, pre-deployed data. As an example, such as... Figure 9 As shown, the memory 902 may include, but is not limited to, the acquisition unit 702 and the transmission unit 704 from the data processing device described above. Furthermore, it may include, but is not limited to, other module units from the data processing device described above, which will not be elaborated upon in this example.

[0134] Optionally, the transmission device 906 described above is used to receive or send data via a network. Specific examples of the network described above may include wired networks and wireless networks. In one example, the transmission device 906 includes a Network Interface Controller (NIC), which can be connected to other network devices and routers via a network cable to communicate with the Internet or a local area network. In another example, the transmission device 906 is a Radio Frequency (RF) module, used for wireless communication with the Internet.

[0135] In addition, the aforementioned electronic device also includes a connection bus 908 for connecting various module components in the aforementioned electronic device.

[0136] In other embodiments, the aforementioned terminal device or server can be a node in a distributed system, wherein the distributed system can be a blockchain system, which is a distributed system formed by connecting multiple nodes through network communication. The nodes can form a peer-to-peer (P2P) network, and any form of computing device, such as a server, terminal, or other electronic device, can become a node in the blockchain system by joining this peer-to-peer network.

[0137] According to another aspect of the present invention, a computer-readable storage medium is also provided, wherein a computer program is stored therein, wherein the computer program is configured to perform the steps in any of the above method embodiments when it is run.

[0138] Optionally, in this embodiment, the computer-readable storage medium described above may be configured to store a computer program for performing the following steps:

[0139] In response to the received container creation request, a container for the target graphics processor is created, wherein the container for the target graphics processor is used to process the target image file;

[0140] The first-level data of the target image file is obtained from the pre-deployed data on the terminal, and the second-level data of the target image file other than the first-level data is obtained from the server. The target image file is one of multiple image files corresponding to the service provided by the server. The file size of each image file in the multiple image files is greater than a first threshold. The pre-deployed data is the hierarchical data included in at least two image files in the multiple image files.

[0141] The first-level and second-level data are processed within the container of the target graphics processor to obtain the processing result.

[0142] Alternatively, in this embodiment, the computer-readable storage medium described above may be configured to store a computer program for performing the following steps:

[0143] The terminal sends a data acquisition request, wherein the data acquisition request is used to request the acquisition of the second-level data of the target image file, wherein the target image file is one of multiple image files corresponding to the service provided by the server, the file size of each of the multiple image files is greater than a first threshold, the second-level data is the data in the target image file other than the first-level data, the first-level data is the data pre-deployed to the terminal, and the data pre-deployed to the terminal is the level data included in at least two of the multiple image files;

[0144] The second-level data is sent to the terminal so that the terminal can process the first-level data obtained locally from the terminal and the second-level data obtained from the server through the created GPU container.

[0145] Optionally, in this embodiment, those skilled in the art will understand that all or part of the steps in the various methods of the above embodiments can be implemented by a program instructing the hardware related to the terminal device. The program can be stored in a computer-readable storage medium, which may include: flash drive, read-only memory (ROM), random access memory (RAM), disk or optical disk, etc.

[0146] The sequence numbers of the above embodiments of the present invention are for descriptive purposes only and do not represent the superiority or inferiority of the embodiments.

[0147] If the integrated units in the above embodiments are implemented as software functional units and sold or used as independent products, they can be stored in the aforementioned computer-readable storage medium. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause one or more computer devices (which may be personal computers, servers, or network devices, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention.

[0148] In the above embodiments of the present invention, the descriptions of each embodiment have different focuses. For parts not described in detail in a certain embodiment, please refer to the relevant descriptions of other embodiments.

[0149] In the several embodiments provided in this application, it should be understood that the disclosed client can be implemented in other ways. The device embodiments described above are merely illustrative; for example, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces, indirect coupling or communication connection between units or modules, and may be electrical or other forms.

[0150] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0151] Furthermore, the functional units in the various embodiments of the present invention can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0152] The above description is only a preferred embodiment of the present invention. It should be noted that for those skilled in the art, several improvements and modifications can be made without departing from the principle of the present invention, and these improvements and modifications should also be considered within the scope of protection of the present invention.

Claims

1. A data processing method, characterized in that, include: In response to the received container creation request, a container for the target graphics processor is created, wherein the container for the target graphics processor is used to process the target image file; The first-level data of the target image file is obtained from the pre-deployed data on the terminal, and the second-level data of the target image file other than the first-level data is obtained from the server. The target image file is one of multiple image files corresponding to the service provided by the server. The file size of each of the multiple image files is greater than a first threshold. The pre-deployed data is the hierarchical data included in at least two of the multiple image files. The first-level data and the second-level data are processed in the container of the target graphics processor to obtain the processing result.

2. The method according to claim 1, characterized in that, Obtaining the first-level data of the target image file from the data pre-deployed locally on the terminal, and obtaining the second-level data of the target image file other than the first-level data from the server, includes: The file identifier of the target image file to be processed by the container of the target graphics processor is parsed from the container creation request; Search for the first-level data of the target image file corresponding to the file identifier in the data pre-deployed locally on the terminal; If the first level of data is found, retrieve the found first level of data; Send a data acquisition request to the server, wherein the data acquisition request is used to request the acquisition of the second level data of the target image file in addition to the first level data; Obtain the second-level data from the server.

3. The method according to claim 2, characterized in that, The step of searching for the first-level data of the target image file corresponding to the file identifier in the data pre-deployed locally on the terminal includes: Obtain the data hierarchy in the target image file; Each level in the data hierarchy is designated as the current level, and the following operations are performed on the current level: Search for the data corresponding to the current layer in the data pre-deployed locally on the terminal; If the data corresponding to the current layer is found, the data corresponding to the current layer is determined to be the data in the first level of data.

4. The method according to claim 3, characterized in that, After searching for the data corresponding to the current layer in the pre-deployed data on the terminal, the process further includes: If no data corresponding to the current layer is found, the data corresponding to the current layer is determined to be the data in the second level data.

5. The method according to claim 1, characterized in that, Before creating the container for the target graphics processor in response to the received container creation request, the process also includes: If the real-time data transmission volume between the server and the local terminal is less than the second threshold, hierarchical data included in at least two of the plurality of image files is obtained from the server. The hierarchical data included in at least two of the multiple image files is deployed locally on the terminal as the pre-deployed data.

6. The method according to claim 5, characterized in that, When the real-time data transmission volume between the server and the local terminal is less than a second threshold, obtaining the hierarchical data included in at least two of the plurality of image files from the server includes: Send a data deployment request to the server; obtain the hierarchical data included in both of the at least two image files sent by the server in response to the data deployment request; or Receive the hierarchical data included in both of the at least two image files sent by the server.

7. The method according to any one of claims 1 to 6, characterized in that, After processing the first-level data and the second-level data in the container of the target graphics processor to obtain the processing result, the process further includes: If the first-level data in the server changes, the changed first-level data is retrieved from the server. The modified first-level data is saved locally on the terminal.

8. The method according to claim 7, characterized in that, The step of saving the modified first-level data locally on the terminal includes: Monitor the remaining storage space of the local storage space of the terminal used to store the first level of data; If the remaining storage space is less than a third threshold, search for the latest first-level data in the storage space; Delete all first-level data except the latest first-level data from the storage space; or Delete the first-level data with the longest storage time from all the first-level data except the latest first-level data from the storage space.

9. A data processing method, characterized in that, include: The terminal sends a data acquisition request, wherein the data acquisition request is used to request the acquisition of the second-level data of the target image file, wherein the target image file is one of multiple image files corresponding to the service provided by the server, the file size of each of the multiple image files is greater than a first threshold, the second-level data is the data in the target image file other than the first-level data, the first-level data is the data pre-deployed on the terminal, and the data pre-deployed on the terminal is the level data included in at least two of the multiple image files; The second-level data is sent to the terminal so that the terminal can process the first-level data obtained locally from the terminal and the second-level data obtained from the server through the created GPU container.

10. The method according to claim 9, characterized in that, Sending the second-level data to the terminal includes: If the real-time data transmission volume between the server and the local terminal is less than the second threshold, hierarchical data included in at least two of the plurality of image files is obtained from the server. The hierarchical data included in at least two of the multiple image files are deployed locally on the terminal as the data to be pre-deployed on the terminal.

11. The method according to claim 10, characterized in that, When the real-time data transmission volume between the server and the local terminal is less than a second threshold, obtaining the hierarchical data included in at least two of the plurality of image files from the server includes: Receive a data deployment request sent by the terminal; respond to the data deployment request by sending the hierarchical data included in both of the at least two image files to the terminal; or Send the hierarchical data included in both of the at least two image files to the terminal.

12. A data processing apparatus, characterized in that, include: A creation unit is used to respond to a received container creation request and create a container for the target graphics processor, wherein the container for the target graphics processor is used to process the target image file; The first acquisition unit is used to acquire the first-level data of the target image file from the data pre-deployed locally on the terminal, and to acquire the second-level data of the target image file other than the first-level data from the server. The target image file is one of multiple image files corresponding to the service provided by the server. The file size of each of the multiple image files is greater than a first threshold. The pre-deployed data is the hierarchical data included in at least two of the multiple image files. The processing unit is used to process the first-level data and the second-level data in the container of the target graphics processor to obtain the processing result.

13. A data processing apparatus, characterized in that, include: An acquisition unit is used to acquire a data acquisition request sent by a terminal, wherein the data acquisition request is used to request the acquisition of the second-level data of a target image file, wherein the target image file is one of multiple image files corresponding to the service provided by the server, the file size of each of the multiple image files is greater than a first threshold, the second-level data is the data in the target image file other than the first-level data, the first-level data is the data pre-deployed on the terminal, and the data pre-deployed on the terminal is the level data included in at least two of the multiple image files; The sending unit is configured to send the second-level data to the terminal, so that the terminal can process the first-level data obtained locally from the terminal and the second-level data obtained from the server through the created GPU container.

14. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed, it performs the method described in any one of claims 1 to 8 or 9 to 13.

15. An electronic device comprising a memory and a processor, characterized in that, The memory stores a computer program, and the processor is configured to execute the method described in any one of claims 1 to 8 or 9 to 13 via the computer program.

Images