A data set management method, system, device, electronic device and storage medium

Abstract

The present disclosure relates to a data set management method and system, device, electronic device and storage medium, the method comprising: receiving a data set processing request for a business container deployed on a physical machine, the data set processing request comprising an identifier of a data set used by the business container, and a storage directory where the business container is mounted locally on the physical machine; in response to the data set processing request, processing operations are performed on the data set corresponding to the identifier in the storage directory, the processing operations comprising: downloading the data set corresponding to the identifier to the storage directory, or deleting the data set corresponding to the identifier from the storage directory. The embodiments of the present disclosure can achieve fast access to data sets.

Description

Technical Field

[0001] The present disclosure relates to the field of computer technology, and in particular to a data set management method and system, device, electronic device and storage medium. Background Art

[0002] In fields such as artificial intelligence and data science, large amounts of data are usually processed. For example, in machine learning of artificial intelligence, large amounts of data are usually used for machine learning; in research related to big data, large amounts of data are also analyzed. In fields such as artificial intelligence and data science, a collection of large amounts of data is usually called a dataset.

[0003] Due to the large amount of data in the dataset, in a container-based distributed server (for example, Kubernetes, a distributed container management system developed by Google), the dataset is usually stored in shared storage, and multiple containers process data by remotely accessing the shared storage. However, if a container needs to modify or delete the data in the shared storage, it will affect the use of other containers. Summary of the invention

[0004] The present disclosure proposes a technical solution for data set management.

[0005] According to one aspect of the present disclosure, there is provided a data set management method, comprising:

[0006] Receive a data set processing request for a business container deployed on a physical machine, wherein the data set processing request includes an identifier of a data set used by the business container and a storage directory where the business container is mounted locally on the physical machine;

[0007] In response to the data set processing request, a processing operation is performed on the data set corresponding to the identifier in the storage directory, and the processing operation includes: downloading the data set corresponding to the identifier to the storage directory, or deleting the data set corresponding to the identifier from the storage directory.

[0008] In a possible implementation manner, after receiving the data set processing request for the business container, the method further includes:

[0009] determining a processing progress of the processing operation;

[0010] The processing progress is returned to the front end through the message middleware and / or the WebSocket protocol.

[0011] In a possible implementation, the data set management method is applied to a distributed server, and the distributed server includes a request receiving module and a data set processing module;

[0012] The step of performing a processing operation on the data set corresponding to the identifier in response to the data set processing request in the storage directory includes:

[0013] The request receiving module receives a data set processing request, generates a data set processing message supported by the message middleware according to the data set processing request, and pushes the data set processing message to the message queue corresponding to the physical machine in the message middleware;

[0014] The data set processing module located in the physical machine obtains the data set processing message in the message queue, and performs a processing operation on the data set corresponding to the identifier under the storage directory according to the identifier in the data set processing message and the storage directory.

[0015] In a possible implementation, the distributed server further includes a processing progress push module and a Websocket server, and the processing progress is returned to the front end through the message middleware and the WebSocket protocol, including:

[0016] The processing progress push module generates a processing progress message according to the processing progress, and pushes the processing progress message to the message middleware;

[0017] The Websocket server obtains the processing progress message from the message middleware, and sends the processing progress in the processing progress message to the front-end Websocket client.

[0018] In a possible implementation manner, after downloading the data set corresponding to the identifier to the storage directory, the method further includes:

[0019] receiving a service processing request for the service container;

[0020] In response to the business processing request, a business processing operation is performed using the data set in the storage directory.

[0021] In a possible implementation, the method further includes:

[0022] In response to closing the service container, the data set in the storage directory is deleted.

[0023] In a possible implementation, downloading the data set corresponding to the identifier to the storage directory includes:

[0024] The data set corresponding to the identifier is downloaded from the storage directory of the database or other container to the storage directory of the business container mounted on the local physical machine.

[0025] According to one aspect of the present disclosure, there is provided a data set management device, comprising:

[0026] A request receiving unit, configured to receive a data set processing request for a business container deployed on a physical machine, wherein the data set processing request includes an identifier of a data set used by the business container and a storage directory where the business container is mounted locally on the physical machine;

[0027] A data set processing unit is used to respond to the data set processing request and perform a processing operation on the data set corresponding to the identifier in the storage directory, wherein the processing operation includes: downloading the data set corresponding to the identifier to the storage directory, or deleting the data set corresponding to the identifier from the storage directory.

[0028] In a possible implementation manner, the device further includes:

[0029] a progress determination unit, configured to determine a processing progress of the processing operation;

[0030] The progress returning unit is used to return the processing progress to the front end through the message middleware and / or the web socket WebSocket protocol.

[0031] In a possible implementation, the device is applied to a distributed server, and the distributed server includes a request receiving module and a data set processing module;

[0032] The data set processing unit includes the request receiving module and the data set processing module, wherein the request receiving module receives a data set processing request, generates a data set processing message supported by the message middleware according to the data set processing request, and pushes the data set processing message to the message queue corresponding to the physical machine in the message middleware; the data set processing module located in the physical machine obtains the data set processing message in the message queue, and according to the identifier in the data set processing message and the storage directory, performs processing operations on the data set corresponding to the identifier under the storage directory.

[0033] In a possible implementation, the distributed server also includes a processing progress push module and a Websocket server, and the progress return unit includes the processing progress push module and the Websocket server, wherein the processing progress push module generates a processing progress message according to the processing progress, and pushes the processing progress message to the message middleware; the Websocket server obtains the processing progress message from the message middleware, and sends the processing progress in the processing progress message to the front-end Websocket client.

[0034] In a possible implementation manner, the device further includes:

[0035] A service processing request receiving unit, configured to receive a service processing request for the service container;

[0036] The business processing unit is used to respond to the business processing request and perform business processing operations using the data set in the storage directory.

[0037] In a possible implementation manner, the device further includes:

[0038] The deleting unit is configured to delete the data set in the storage directory in response to closing the service container.

[0039] In a possible implementation, the data set processing unit is used to download the data set corresponding to the identifier from a storage directory of a database or other container to a storage directory mounted locally on the physical machine by the business container.

[0040] According to one aspect of the present disclosure, there is provided a data set management system, comprising:

[0041] A request receiving module receives a data set processing request for a business container deployed on a physical machine, wherein the data set processing request includes an identifier of a data set used by the business container and a storage directory where the business container is mounted locally on the physical machine;

[0042] The data set processing module, in response to the data set processing request, performs a processing operation on the data set corresponding to the identifier in the storage directory, wherein the processing operation includes: downloading the data set corresponding to the identifier to the storage directory, or deleting the data set corresponding to the identifier from the storage directory.

[0043] According to one aspect of the present disclosure, an electronic device is provided, comprising: a processor; and a memory for storing processor-executable instructions; wherein the processor is configured to call the instructions stored in the memory to execute the above method.

[0044] According to one aspect of the present disclosure, a computer-readable storage medium is provided, on which computer program instructions are stored, and the computer program instructions implement the above method when executed by a processor.

[0045] In an embodiment of the present disclosure, after a business container is created on a physical machine, when a data set processing request for the business container is received, based on the identifier of the data set used by the business container contained in the data set processing request, a processing operation is performed on the data set corresponding to the identifier in a storage directory locally mounted on the physical machine by the business container, and the processing operation includes: downloading the data set corresponding to the identifier to the storage directory, or deleting the data set corresponding to the identifier from the storage directory. Thus, by synchronizing the data set to the physical machine where the user container is located and mounting it to the storage directory locally on the physical machine, the user can access his own data set just like accessing local data, and will not be affected by network delays. In addition, since the data set is stored in the storage directory mounted on the container, it has a better isolation effect, and the operations on the data set between containers do not affect each other, and the user can freely modify, delete, and other operations on the data set.

[0046] It should be understood that the above general description and the following detailed description are exemplary and explanatory only and do not limit the present disclosure. Other features and aspects of the present disclosure will become clear from the following detailed description of exemplary embodiments with reference to the accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0047] The drawings herein are incorporated into the specification and constitute a part of the specification. These drawings illustrate embodiments consistent with the present disclosure and are used to illustrate the technical solutions of the present disclosure together with the specification.

[0048] Figure 1 A flowchart showing a data set management method according to an embodiment of the present disclosure;

[0049] Figure 2 An architecture diagram of a distributed server provided according to an embodiment of the present disclosure is shown;

[0050] Figure 3 A schematic diagram showing the structure of a data set management system according to an embodiment of the present disclosure is shown;

[0051] Figure 4 A schematic diagram showing the structure of another data set management system according to an embodiment of the present disclosure is shown;

[0052] Figure 5 A schematic diagram showing the structure of another data set management system according to an embodiment of the present disclosure is shown;

[0053] Figure 6 A schematic diagram showing the structure of another data set management system according to an embodiment of the present disclosure is shown;

[0054] Figure 7 A block diagram of a data set management device according to an embodiment of the present disclosure is shown;

[0055] Figure 8 A block diagram of an electronic device according to an embodiment of the present disclosure is shown;

[0056] Fig. 9 A block diagram of an electronic device according to an embodiment of the present disclosure is shown. DETAILED DESCRIPTION

[0057] Various exemplary embodiments, features and aspects of the present disclosure will be described in detail below with reference to the accompanying drawings. The same reference numerals in the accompanying drawings represent elements with the same or similar functions. Although various aspects of the embodiments are shown in the accompanying drawings, the drawings are not necessarily drawn to scale unless otherwise specified.

[0058] The word “exemplary” is used exclusively herein to mean “serving as an example, example, or illustration.” Any embodiment described herein as “exemplary” is not necessarily to be construed as preferred or advantageous over other embodiments.

[0059] The term "and / or" herein is only a description of the association relationship of the associated objects, indicating that there may be three relationships. For example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone. In addition, the term "at least one" herein represents any combination of at least two of any one or more of a plurality of. For example, including at least one of A, B, and C can represent including any one or more elements selected from the set consisting of A, B, and C.

[0060] In addition, in order to better illustrate the present disclosure, numerous specific details are given in the following specific embodiments. It should be understood by those skilled in the art that the present disclosure can also be implemented without certain specific details. In some examples, methods, means, components and circuits well known to those skilled in the art are not described in detail in order to highlight the subject matter of the present disclosure.

[0061] With the development of computer technology, programs have entered the era of virtualized deployment. Programs can be deployed virtually through containers. In container-based distributed servers (for example, the distributed container management system Kubernetes developed by Google), by deploying the same program in multiple containers, distributed processing of tasks by the program can be achieved.

[0062] In the field of artificial intelligence, an artificial intelligence platform can be deployed on a Kubernetes cluster for users to conduct artificial intelligence-related learning, research, and applications on the platform, such as machine learning, data science, and data processing. Different users conduct the above activities through multiple containers deployed in the machine. The above activities often process data sets, for example, using data sets for machine learning.

[0063] In large-scale container clusters, it is crucial to manage data sets. If data sets are stored in shared storage, network latency may cause performance bottlenecks when users use data sets, greatly reducing computing efficiency. In addition, data set isolation is difficult to achieve in shared storage. If a container needs to modify or delete data in shared storage, it will affect the use of other containers.

[0064] In the disclosed embodiment, by synchronizing the data set to the physical machine where the user container is located and mounting it to the storage directory of the container on the local physical machine, the user can access his data set just like accessing local data, without being affected by network delays. In addition, since the data set is stored in the storage directory mounted by the container, it has a good isolation effect, and the operations of the data set between containers do not affect each other, and the user can freely modify, delete, and other operations on the data set.

[0065] In a possible implementation, the data set management method can be executed by an electronic device such as a terminal device or a server. The terminal device can be a user equipment (UE), a mobile device, a user terminal, a terminal, a cellular phone, a cordless phone, a personal digital assistant (PDA), a handheld device, a computing device, a vehicle-mounted device, a wearable device, etc. The method can be implemented by a processor calling a computer-readable instruction stored in a memory. Alternatively, the method can be executed by a server, for example, the method can be executed by a distributed server. The following takes the execution subject as an example of a distributed server to exemplify the possible implementation of the present disclosure.

[0066] In the embodiments of the present disclosure, a business container can be created on a physical machine. After receiving a business processing request sent by a user front end, a corresponding business container can be created. The business container is used to process the business corresponding to the request, and the process for processing the business is loaded in the container. The business can be, for example, a machine learning task in the field of artificial intelligence, a data processing business in data science, and the like.

[0067] A business container is created on a physical machine. Multiple business containers can be created on the same physical machine. After the business container is created, the address of the physical machine where the business container is located can be recorded in the business container, and the local storage directory of the physical machine can be mounted to the business container. The storage directory can be used by the business container to store, read, modify, and delete data sets. The storage directories mounted by different business containers are not affected by each other.

[0068] The above service processing request may be sent by a user front end, which may be a browser or a client, that is, the user may send the service processing request by operating on a web page of the browser.

[0069] After the business container is created, a message indicating successful creation of the business container may be sent to the user front end. In addition, a storage directory where the business container is mounted on a local physical machine may be sent to the user front end.

[0070] After creating a business container, you can manage the data set for the business container. Figure 1 A flowchart of a data set management method according to an embodiment of the present disclosure is shown. Figure 1 As shown, the data set management method includes:

[0071] In step S11, a data set processing request for a business container deployed on a physical machine is received, wherein the data set processing request includes an identifier of a data set used by the business container and a storage directory where the business container is mounted locally on the physical machine.

[0072] The user front end will display the data sets available to the user for selection. After the user selects the data set used by the business container, the user front end will encapsulate the identifier of the data set selected by the user into the data set processing request. In addition, the request can also encapsulate the storage directory where the business container is mounted on the local physical machine.

[0073] The user front end sends the encapsulated data set processing request to the back end server, wherein the back end server is the execution subject of this implementation method.

[0074] The data set processing request may be a user request to download the data set to a local storage directory of the physical machine to facilitate subsequent use of the data set; or, the request may be a user request to delete the data set from the local storage directory of the physical machine.

[0075] In the embodiments of the present disclosure, a data set may be a data set consisting of data having a common theme and / or similar attributes, for example, it may be a data set used for machine learning in artificial intelligence, or it may also be a data set used for data science.

[0076] In step S12, in response to the data set processing request, a processing operation is performed on the data set corresponding to the identifier in the storage directory, and the processing operation includes: downloading the data set corresponding to the identifier to the storage directory, or deleting the data set corresponding to the identifier from the storage directory.

[0077] After receiving the data set processing request, if the request requests to download the data set to the local storage directory of the physical machine, the data set corresponding to the identifier is downloaded to the storage directory. Specifically, the data set corresponding to the identifier can be downloaded from the shared object storage according to the identifier.

[0078] In the case where the request is to delete the data set from the local storage directory of the physical machine, the data set corresponding to the identifier is deleted from the storage directory.

[0079] See also Figure 2 , Figure 2 The following is an architecture diagram of a distributed server provided by an embodiment of the present disclosure, such as Figure 2 As shown, the distributed server includes multiple physical machines. User 1 to user N send user requests through the front end, and the user requests are distributed to the container of a physical machine through load balancing. Multiple business containers can be deployed in one physical machine, and each business container corresponds to a data set. The data sets are isolated from each other and do not interfere with each other.

[0080] In an embodiment of the present disclosure, after a business container is created on a physical machine, when a data set processing request for the business container is received, based on the identifier of the data set used by the business container contained in the data set processing request, a processing operation is performed on the data set corresponding to the identifier in a storage directory locally mounted on the physical machine by the business container, and the processing operation includes: downloading the data set corresponding to the identifier to the storage directory, or deleting the data set corresponding to the identifier from the storage directory. Thus, by synchronizing the data set to the physical machine where the user container is located and mounting it to the storage directory locally on the physical machine, the user can access his own data set just like accessing local data, and will not be affected by network delays. In addition, since the data set is stored in the storage directory mounted on the container, it has a better isolation effect, and the operations on the data set between containers do not affect each other, and the user can freely modify, delete, and other operations on the data set.

[0081] In a possible implementation, downloading the data set corresponding to the identifier to the storage directory includes: downloading the data set corresponding to the identifier from the storage directory of a database or other container to the storage directory mounted by the business container on the local physical machine.

[0082] The database here can be a storage space for centrally storing data sets. Then, when downloading a data set, it can be downloaded from the database. In addition, if the data set corresponding to the identifier is also stored in the storage directory of other containers, it can also be downloaded from the storage directory of other containers to reduce the occupancy of the database bandwidth and improve the download efficiency of the data set.

[0083] In a possible implementation, the data sets under the storage directories of other containers may be edited data sets. Then, downloading the edited data sets from the storage directories of other containers can achieve sharing of the edited data sets, thereby improving the efficiency of business processing of the data sets.

[0084] In a possible implementation, after downloading the data set corresponding to the identifier to the storage directory, the method further includes: receiving a business processing request for the business container; and performing a business processing operation using the data set in the storage directory in response to the business processing request.

[0085] As mentioned above, the business container can be used to process the business requested by the business processing request. The business here can be, for example, the machine learning business in artificial intelligence machine learning, or the data analysis business in big data related research, etc.

[0086] In the process of processing business, the data sets in the storage directory will be used to perform business processing operations, such as using the data sets to train neural networks, or using the data sets to perform user behavior analysis, etc.

[0087] In the embodiment of the present disclosure, business processing operations are performed by using the data sets in the storage directory. Since the data sets are stored in the storage directory mounted by the container, a good isolation effect is achieved, and operations on the data sets between containers do not affect each other. Users can freely modify, delete, and perform other operations on the data sets.

[0088] In a possible implementation, after receiving the data set processing request for the business container, the method further includes: determining the processing progress of the processing operation; and returning the processing progress to the front end via a message middleware and / or a web socket WebSocket protocol.

[0089] The processing progress here is, for example, the progress of downloading a dataset, for example, dataset 01 has been downloaded completely, or dataset 02 has been downloaded 65%; the processing progress can also be the progress of deleting a dataset, for example, dataset 03 has been deleted, or dataset 03 failed to be deleted, please try again.

[0090] In the disclosed embodiment, the processing progress can be returned to the front end through the message middleware and the WebSocket protocol. The message middleware can asynchronously transmit the processing progress through the message queue to improve the transmission efficiency of the processing progress between different components of the distributed server. In addition, the processing progress is pushed from the server to the front end through the WebSocket protocol, which reduces the pressure on the server compared to the user front end requesting the processing progress from the server in a polling manner.

[0091] In one possible implementation, the dataset management method is applied to a distributed server, which includes a request receiving module and a dataset processing module; in response to the dataset processing request, a processing operation is performed on the dataset corresponding to the identifier under the storage directory, including: the request receiving module receives the dataset processing request, generates a dataset processing message supported by a message middleware according to the dataset processing request, and pushes the dataset processing message to a message queue corresponding to the physical machine in the message middleware; the dataset processing module located in the physical machine obtains the dataset processing message in the message queue, and performs a processing operation on the dataset corresponding to the identifier under the storage directory according to the identifier in the dataset processing message and the storage directory.

[0092] In this implementation, the message middleware can work in a publish-subscribe mode, in which a message sender (producer) publishes a message and one or more message receivers (consumers) subscribe to the message. There is no direct coupling between the message sender (producer) and the message receiver (consumer). The message sender sends the message to the distributed message queue in the message middleware and ends the processing of the message. The message receiver obtains the message from the distributed message queue and then performs subsequent processing.

[0093] In this implementation, after the request receiving module receives the data set processing request, it will generate a message supported by the message middleware based on the data set processing request. For the sake of ease of expression, the generated message is called a data set processing message. The data set processing message will still include the identifier of the data set used by the business container and the storage directory where the business container is mounted locally on the physical machine.

[0094] The request receiving module acts as a producer in the "publish-subscribe mode" and pushes the data set processing message to the message middleware. Specifically, the data set processing message is placed in the message queue corresponding to the physical machine where the business container is located.

[0095] The data set processing module in the physical machine where the business container is located will act as a consumer in the "publish-subscribe mode", obtain the data set processing message from the message queue corresponding to the physical machine, and parse the data set processing message to obtain the data set identifier contained in the data set processing message and the storage directory where the business container is mounted on the local physical machine. Then, the data set corresponding to the identifier can be processed under the storage directory.

[0096] In the disclosed embodiment, the identifier and storage directory of the dataset contained in the dataset processing request can be asynchronously transmitted to the dataset processing module in the physical machine where the business container is located through the message queue in the message middleware, with high transmission efficiency.

[0097] In a possible implementation, the distributed server also includes a processing progress push module and a Websocket server, which returns the processing progress to the front end through the message middleware and / or the web socket WebSocket protocol, including: the processing progress push module generates a processing progress message according to the processing progress, and pushes the processing progress message to the message middleware; the Websocket server obtains the processing progress message from the message middleware, and sends the processing progress in the processing progress message to the Websocket client of the front end.

[0098] In this implementation, the message middleware can still work in the publish-subscribe mode. The processing progress push module will act as a producer in the "publish-subscribe mode" to obtain the processing progress of data processing, and then encapsulate the processing progress into a message supported by the message middleware, that is, a processing progress message. The processing progress push module will push the processing progress message to the message queue of the message middleware, and specifically put the processing progress message into the message queue corresponding to the business container.

[0099] The Websocket server connected to the user front end will act as a consumer in the "publish-subscribe mode" to obtain the processing progress message from the message queue corresponding to the business container, parse the processing progress message, obtain the processing progress, and then send the processing progress to the Websocket client of the user front end. The Websocket server can encapsulate the processing progress into a data transmission format supported by the Websocket protocol, and then send it to the Websocket client. After receiving the processing progress, the Websocket client can display the processing progress to the user through the user front end, for example, the processing progress can be displayed through a web page in a browser.

[0100] In the embodiments of the present disclosure, the processing progress can be asynchronously transmitted to the Websocket server through the message queue in the message middleware, with high transmission efficiency. It is pushed to the front end through the Websocket server, which reduces the pressure on the server compared to the user front end requesting the processing progress from the server in a polling manner.

[0101] In a possible implementation, the method further includes: in response to closing the business container, deleting the data set in the storage directory. When the business container is destroyed, the data set in the local storage directory of the physical machine where the business container is mounted can be destroyed at the same time to save storage space in the physical machine.

[0102] In addition, the present disclosure also provides a data set management system 30, such as Figure 3 As shown, including:

[0103] The request receiving module 301 receives a data set processing request for a business container deployed on a physical machine, wherein the data set processing request includes an identifier of a data set used by the business container and a storage directory where the business container is mounted locally on the physical machine;

[0104] The data set processing module 303, in response to the data set processing request, performs a processing operation on the data set corresponding to the identifier in the storage directory, and the processing operation includes: downloading the data set corresponding to the identifier to the storage directory, or deleting the data set corresponding to the identifier from the storage directory.

[0105] In an embodiment of the present disclosure, after a business container is created on a physical machine, when a data set processing request for the business container is received, based on the identifier of the data set used by the business container contained in the data set processing request, a processing operation is performed on the data set corresponding to the identifier in a storage directory locally mounted on the physical machine by the business container, and the processing operation includes: downloading the data set corresponding to the identifier to the storage directory, or deleting the data set corresponding to the identifier from the storage directory. Thus, by synchronizing the data set to the physical machine where the user container is located and mounting it to the storage directory locally on the physical machine, the user can access his own data set just like accessing local data, and will not be affected by network delays. In addition, since the data set is stored in the storage directory mounted on the container, it has a better isolation effect, and the operations on the data set between containers do not affect each other, and the user can freely modify, delete, and other operations on the data set.

[0106] In a possible implementation, the data set management system 30 further includes a message middleware 302, see Figure 4 In this implementation, the request receiving module 301 receives a data set processing request for a business container, generates a data set processing message supported by the message middleware 302 according to the data set processing request, and pushes the data set processing message to the message middleware 302. The data set processing request includes an identifier of the data set used by the business container and a storage directory where the business container is mounted locally on the physical machine.

[0107] The message middleware 302 puts the data set processing message into a message queue;

[0108] The data set processing module 303 obtains the data set processing message in the message queue 302, and performs a processing operation on the data set corresponding to the identifier in the storage directory.

[0109] In the embodiment of the present disclosure, the message middleware 302 may be Rabbit MQ (Message queue, MQ), or may also be Active QM, Kafka, Rocket MQ, Zero MQ, etc., which is not limited in the embodiment of the present disclosure.

[0110] The message middleware 302 may include an exchange, which can distribute messages sent by producers. In the process of distributing messages, the distribution mode varies depending on the message type. For example, it may include the following four modes: direct mode, fanout mode, topic mode, and headers mode.

[0111] When the message middleware 302 puts the data set processing message into the message queue, the switch can adopt the topic mode, in which the data set processing message can be put into the message queue corresponding to the physical machine where the business container is located according to the physical machine identifier contained in the routing key of the data set processing message. The data set processing module 303 on the physical machine will obtain the data set processing message in the message queue, and process the data set corresponding to the data set identifier in the storage directory.

[0112] In the disclosed embodiment, by synchronizing the data set to the physical machine where the user container is located and mounting it to the storage directory of the container locally on the physical machine, users can access their data sets just like accessing local data, without being affected by network delays. In addition, since the data set is stored in the storage directory mounted on the container, it has a good isolation effect, and the operations on the data set between containers do not affect each other, and users can freely modify, delete, and perform other operations on the data set. In addition, through the message queue in the message middleware, the identifier and storage directory of the data set contained in the data set processing request are asynchronously transmitted to the data set processing module in the physical machine where the business container is located, and the transmission efficiency is relatively high.

[0113] In a possible implementation, the system 30 further includes a processing progress push module 304 and a Websocket server 305. Figure 5 As shown, where:

[0114] The processing progress push module 304 generates a processing progress message according to the processing progress of the processing operation, and pushes the processing progress message to the message middleware 302;

[0115] The message middleware 302 puts the processing progress message into a message queue;

[0116] The Websocket server 305 obtains the processing progress message from the message queue of the message middleware 302, and sends the processing progress in the processing progress message to the front-end Websocket client.

[0117] When the message middleware 302 puts the processing progress message into the message queue, the switch can adopt the fanout mode, in which the message queue is bound to the switch, and the processing progress message sent to the switch will be forwarded to all message queues bound to the switch. Then the Websocket server can obtain the processing progress message from the message queue of the message middleware 302.

[0118] In the embodiments of the present disclosure, the processing progress can be asynchronously transmitted to the Websocket server through the message queue in the message middleware, with high transmission efficiency. It is pushed to the front end through the Websocket server, which reduces the pressure on the server compared to the user front end requesting the processing progress from the server in a polling manner.

[0119] See also Figure 6 , which is a structural diagram of a data set management system provided by the present application, wherein the system includes: a request receiving module, a message middleware, a data set processing module, a processing progress push module, and a Websocket server, wherein the message middleware includes a switch and a message queue.

[0120] After the user sends a dataset processing request through the front end, the dataset processing request will be distributed to the request receiving module after load balancing. The request receiving module generates a dataset processing message supported by the message middleware based on the dataset processing request, and pushes the dataset processing message to the message middleware. The dataset processing request contains the identifier of the dataset used by the business container and the storage directory where the business container is mounted locally on the physical machine.

[0121] The switch of the message middleware can adopt the topic mode, and put the data set processing message into the message queue corresponding to the physical machine where the business container is located according to the physical machine identifier contained in the routing key of the data set processing message. The data set processing module on the physical machine will obtain the data set processing message in the message queue, and process the data set corresponding to the data set identifier in the storage directory.

[0122] The processing progress push module generates a processing progress message based on the processing progress of the processing operation and pushes the processing progress message to the message middleware;

[0123] The switch of the message middleware adopts fanout mode, binds the message queue to the switch, and the processing progress message sent to the switch will be forwarded to all message queues bound to the switch. The Websocket server obtains the processing progress message from the message queue of the message middleware and sends the processing progress in the processing progress message to the front-end Websocket client.

[0124] In the disclosed embodiment, by synchronizing the data set to the physical machine where the user container is located and mounting it to the storage directory of the container locally on the physical machine, users can access their data sets just like accessing local data, without being affected by network delays. In addition, since the data set is stored in the storage directory mounted on the container, it has a good isolation effect, and the operations on the data set between containers do not affect each other, and users can freely modify, delete, and perform other operations on the data set. In addition, through the message queue in the message middleware, the identifier and storage directory of the data set contained in the data set processing request are asynchronously transmitted to the data set processing module in the physical machine where the business container is located, and the transmission efficiency is relatively high.

[0125] In the embodiments of the present disclosure, the processing progress can be asynchronously transmitted to the Websocket server through the message queue in the message middleware, with high transmission efficiency. It is pushed to the front end through the Websocket server, which reduces the pressure on the server compared to the user front end requesting the processing progress from the server in a polling manner.

[0126] For the implementation of the specific functions of each module, please refer to the relevant discussion in this disclosure, which will not be repeated here.

[0127] It can be understood that the above-mentioned various method embodiments mentioned in the present disclosure can be combined with each other to form a combined embodiment without violating the principle logic. Due to space limitations, the present disclosure will not repeat them. It can be understood by those skilled in the art that in the above-mentioned method of the specific implementation method, the specific execution order of each step should be determined according to its function and possible internal logic.

[0128] In addition, the present disclosure also provides a data set management device, an electronic device, a computer-readable storage medium, and a program, all of which can be used to implement any data set management method provided by the present disclosure. The corresponding technical solutions and descriptions are referred to in the corresponding records of the method part and will not be repeated here.

[0129] Figure 7 A block diagram of a data set management device according to an embodiment of the present disclosure is shown as follows: Figure 7 As shown, the device 70 includes:

[0130] The request receiving unit 71 is used to receive a data set processing request for a business container deployed on a physical machine, wherein the data set processing request includes an identifier of a data set used by the business container and a storage directory where the business container is mounted locally on the physical machine;

[0131] The data set processing unit 72 is used to respond to the data set processing request and perform a processing operation on the data set corresponding to the identifier in the storage directory, wherein the processing operation includes: downloading the data set corresponding to the identifier to the storage directory, or deleting the data set corresponding to the identifier from the storage directory.

[0132] In a possible implementation manner, the device further includes:

[0133] a progress determination unit, configured to determine a processing progress of the processing operation;

[0134] The progress returning unit is used to return the processing progress to the front end through the message middleware and / or the WebSocket protocol.

[0135] In a possible implementation, the device is applied to a distributed server, and the distributed server includes a request receiving module and a data set processing module;

[0136] The data set processing unit 72 includes the request receiving module and the data set processing module, wherein the request receiving module receives a data set processing request, generates a data set processing message supported by the message middleware according to the data set processing request, and pushes the data set processing message to the message queue corresponding to the physical machine in the message middleware; the data set processing module located in the physical machine obtains the data set processing message in the message queue, and according to the identifier in the data set processing message and the storage directory, performs processing operations on the data set corresponding to the identifier under the storage directory.

[0137] In a possible implementation, the distributed server also includes a processing progress push module and a Websocket server, and the progress return unit includes the processing progress push module and the Websocket server, wherein the processing progress push module generates a processing progress message according to the processing progress, and pushes the processing progress message to the message middleware; the Websocket server obtains the processing progress message from the message middleware, and sends the processing progress in the processing progress message to the front-end Websocket client.

[0138] In a possible implementation manner, the device further includes:

[0139] A service processing request receiving unit, configured to receive a service processing request for the service container;

[0140] The business processing unit is used to respond to the business processing request and perform business processing operations using the data set in the storage directory.

[0141] In a possible implementation manner, the device further includes:

[0142] The deleting unit is configured to delete the data set in the storage directory in response to closing the service container.

[0143] In a possible implementation, the data set processing unit 72 is used to download the data set corresponding to the identifier from the storage directory of the database or other container to the storage directory of the business container mounted locally on the physical machine.

[0144] In some embodiments, the functions or modules included in the device provided by the embodiments of the present disclosure can be used to execute the method described in the above method embodiments. The specific implementation can refer to the description of the above method embodiments, and for the sake of brevity, it will not be repeated here.

[0145] The embodiment of the present disclosure also provides a computer-readable storage medium, on which computer program instructions are stored, and when the computer program instructions are executed by a processor, the above method is implemented. The computer-readable storage medium may be a non-volatile computer-readable storage medium.

[0146] An embodiment of the present disclosure further proposes an electronic device, comprising: a processor; and a memory for storing instructions executable by the processor; wherein the processor is configured to call the instructions stored in the memory to execute the above method.

[0147] The embodiments of the present disclosure also provide a computer program product, including computer-readable codes. When the computer-readable codes are executed on a device, a processor in the device executes instructions for implementing the data set management method provided in any of the above embodiments.

[0148] The embodiments of the present disclosure also provide another computer program product for storing computer-readable instructions, which, when executed, enable a computer to perform the operations of the data set management method provided in any of the above embodiments.

[0149] The electronic device may be provided as a terminal, a server, or a device in other forms.

[0150] Figure 8A block diagram of an electronic device 800 according to an embodiment of the present disclosure is shown. For example, the electronic device 800 may be a mobile phone, a computer, a digital broadcast terminal, a messaging device, a game console, a tablet device, a medical device, a fitness device, a personal digital assistant, or the like.

[0151] Reference Figure 8 , the electronic device 800 may include one or more of the following components: a processing component 802 , a memory 804 , a power component 806 , a multimedia component 808 , an audio component 810 , an input / output (I / O) interface 812 , a sensor component 814 , and a communication component 816 .

[0152] The processing component 802 generally controls the overall operation of the electronic device 800, such as operations associated with display, phone calls, data communications, camera operations, and recording operations. The processing component 802 may include one or more processors 820 to execute instructions to complete all or part of the steps of the above-mentioned method. In addition, the processing component 802 may include one or more modules to facilitate the interaction between the processing component 802 and other components. For example, the processing component 802 may include a multimedia module to facilitate the interaction between the multimedia component 808 and the processing component 802.

[0153] The memory 804 is configured to store various types of data to support operations on the electronic device 800. Examples of such data include instructions for any application or method operating on the electronic device 800, contact data, phone book data, messages, pictures, videos, etc. The memory 804 can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, magnetic disk or optical disk.

[0154] The power supply component 806 provides power to the various components of the electronic device 800. The power supply component 806 may include a power management system, one or more power supplies, and other components associated with generating, managing, and distributing power to the electronic device 800.

[0155] The multimedia component 808 includes a screen that provides an output interface between the electronic device 800 and the user. In some embodiments, the screen may include a liquid crystal display (LCD) and a touch panel (TP). If the screen includes a touch panel, the screen may be implemented as a touch screen to receive input signals from the user. The touch panel includes one or more touch sensors to sense touch, slide, and gestures on the touch panel. The touch sensor may not only sense the boundaries of the touch or slide action, but also detect the duration and pressure associated with the touch or slide operation. In some embodiments, the multimedia component 808 includes a front camera and / or a rear camera. When the electronic device 800 is in an operating mode, such as a shooting mode or a video mode, the front camera and / or the rear camera may receive external multimedia data. Each front camera and rear camera may be a fixed optical lens system or have a focal length and optical zoom capability.

[0156] The audio component 810 is configured to output and / or input audio signals. For example, the audio component 810 includes a microphone (MIC), and when the electronic device 800 is in an operating mode, such as a call mode, a recording mode, and a voice recognition mode, the microphone is configured to receive an external audio signal. The received audio signal can be further stored in the memory 804 or sent via the communication component 816. In some embodiments, the audio component 810 also includes a speaker for outputting audio signals.

[0157] I / O interface 812 provides an interface between processing component 802 and peripheral interface modules, such as keyboards, click wheels, buttons, etc. These buttons may include but are not limited to: home button, volume button, start button, and lock button.

[0158] The sensor assembly 814 includes one or more sensors for providing various aspects of status assessment for the electronic device 800. For example, the sensor assembly 814 can detect the open / closed state of the electronic device 800, the relative positioning of the components, such as the display and keypad of the electronic device 800, and the sensor assembly 814 can also detect the position change of the electronic device 800 or a component of the electronic device 800, the presence or absence of contact between the user and the electronic device 800, the orientation or acceleration / deceleration of the electronic device 800, and the temperature change of the electronic device 800. The sensor assembly 814 may include a proximity sensor configured to detect the presence of a nearby object without any physical contact. The sensor assembly 814 may also include an optical sensor, such as a complementary metal oxide semiconductor (CMOS) or a charge coupled device (CCD) image sensor, for use in imaging applications. In some embodiments, the sensor assembly 814 may also include an acceleration sensor, a gyroscope sensor, a magnetic sensor, a pressure sensor, or a temperature sensor.

[0159] The communication component 816 is configured to facilitate wired or wireless communication between the electronic device 800 and other devices. The electronic device 800 can access a wireless network based on a communication standard, such as a wireless network (WiFi), a second generation mobile communication technology (2G) or a third generation mobile communication technology (3G), or a combination thereof. In an exemplary embodiment, the communication component 816 receives a broadcast signal or broadcast-related information from an external broadcast management system via a broadcast channel. In an exemplary embodiment, the communication component 816 also includes a near field communication (NFC) module to facilitate short-range communication. For example, the NFC module can be implemented based on radio frequency identification (RFID) technology, infrared data association (IrDA) technology, ultra-wideband (UWB) technology, Bluetooth (BT) technology and other technologies.

[0160] In an exemplary embodiment, the electronic device 800 may be implemented by one or more application-specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field programmable gate arrays (FPGAs), controllers, microcontrollers, microprocessors, or other electronic components to perform the above methods.

[0161] In an exemplary embodiment, a non-volatile computer-readable storage medium is also provided, such as a memory 804 including computer program instructions, which can be executed by a processor 820 of an electronic device 800 to perform the above method.

[0162] Fig. 9 1 is a block diagram of an electronic device 1900 according to an embodiment of the present disclosure. For example, the electronic device 1900 may be provided as a server. Fig. 9 , the electronic device 1900 includes a processing component 1922, which further includes one or more processors, and a memory resource represented by a memory 1932 for storing instructions executable by the processing component 1922, such as an application. The application stored in the memory 1932 may include one or more modules, each corresponding to a set of instructions. In addition, the processing component 1922 is configured to execute instructions to perform the above method.

[0163] The electronic device 1900 may also include a power supply component 1926 configured to perform power management of the electronic device 1900, a wired or wireless network interface 1950 configured to connect the electronic device 1900 to a network, and an input / output (I / O) interface 1958. The electronic device 1900 may operate based on an operating system stored in the memory 1932, such as a Microsoft Server operating system (Windows Server 2003). TM), a graphical user interface operating system launched by Apple (Mac OSX TM ), a multi-user, multi-process computer operating system (Unix TM ), a free and open source Unix-like operating system (Linux TM ), an open source Unix-like operating system (FreeBSD TM ) or similar.

[0164] In an exemplary embodiment, a non-volatile computer-readable storage medium is also provided, such as a memory 1932 including computer program instructions, which can be executed by the processing component 1922 of the electronic device 1900 to perform the above method.

[0165] The present disclosure may be a system, a method and / or a computer program product. The computer program product may include a computer-readable storage medium carrying computer-readable program instructions for causing a processor to implement various aspects of the present disclosure.

[0166] Computer readable storage medium can be a tangible device that can hold and store instructions used by an instruction execution device. Computer readable storage medium can be, for example, (but not limited to) an electrical storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or any suitable combination thereof. More specific examples (non-exhaustive list) of computer readable storage medium include: a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), a static random access memory (SRAM), a portable compact disk read-only memory (CD-ROM), a digital versatile disk (DVD), a memory stick, a floppy disk, a mechanical encoding device, for example, a punch card or a convex structure in a groove on which instructions are stored, and any suitable combination thereof. The computer readable storage medium used here is not interpreted as a transient signal itself, such as a radio wave or other freely propagating electromagnetic wave, an electromagnetic wave propagated by a waveguide or other transmission medium (for example, a light pulse by an optical fiber cable), or an electrical signal transmitted by a wire.

[0167] The computer-readable program instructions described herein can be downloaded from a computer-readable storage medium to each computing / processing device, or downloaded to an external computer or external storage device via a network, such as the Internet, a local area network, a wide area network, and / or a wireless network. The network can include copper transmission cables, optical fiber transmissions, wireless transmissions, routers, firewalls, switches, gateway computers, and / or edge servers. The network adapter card or network interface in each computing / processing device receives the computer-readable program instructions from the network and forwards the computer-readable program instructions for storage in the computer-readable storage medium in each computing / processing device.

[0168] The computer program instructions for performing the operation of the present disclosure may be assembly instructions, instruction set architecture (ISA) instructions, machine instructions, machine-related instructions, microcode, firmware instructions, state setting data, or source code or object code written in any combination of one or more programming languages, including object-oriented programming languages, such as Smalltalk, C++, etc., and conventional procedural programming languages, such as "C" language or similar programming languages. Computer-readable program instructions may be executed completely on a user's computer, partially on a user's computer, as an independent software package, partially on a user's computer, partially on a remote computer, or completely on a remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer via any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., using an Internet service provider to connect via the Internet). In some embodiments, an electronic circuit, such as a programmable logic circuit, a field programmable gate array (FPGA), or a programmable logic array (PLA), may be customized by utilizing the state information of the computer-readable program instructions, and the electronic circuit may execute the computer-readable program instructions, thereby realizing various aspects of the present disclosure.

[0169] Various aspects of the present disclosure are described herein with reference to the flowcharts and / or block diagrams of the methods, devices (systems) and computer program products according to the embodiments of the present disclosure. It should be understood that each box in the flowchart and / or block diagram and the combination of each box in the flowchart and / or block diagram can be implemented by computer-readable program instructions.

[0170] These computer-readable program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device, thereby producing a machine, so that when these instructions are executed by the processor of the computer or other programmable data processing device, a device that implements the functions / actions specified in one or more boxes in the flowchart and / or block diagram is generated. These computer-readable program instructions can also be stored in a computer-readable storage medium, and these instructions cause the computer, programmable data processing device, and / or other equipment to work in a specific manner, so that the computer-readable medium storing the instructions includes a manufactured product, which includes instructions for implementing various aspects of the functions / actions specified in one or more boxes in the flowchart and / or block diagram.

[0171] Computer-readable program instructions may also be loaded onto a computer, other programmable data processing apparatus, or other device so that a series of operating steps are performed on the computer, other programmable data processing apparatus, or other device to produce a computer-implemented process, thereby causing the instructions executed on the computer, other programmable data processing apparatus, or other device to implement the functions / actions specified in one or more boxes in the flowchart and / or block diagram.

[0172] The flow chart and block diagram in the accompanying drawings show the possible architecture, function and operation of the system, method and computer program product according to multiple embodiments of the present disclosure. In this regard, each square box in the flow chart or block diagram can represent a part of a module, program segment or instruction, and a part of the module, program segment or instruction includes one or more executable instructions for realizing the specified logical function. In some alternative implementations, the function marked in the square box can also occur in a sequence different from that marked in the accompanying drawings. For example, two continuous square boxes can actually be executed substantially in parallel, and they can sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each square box in the block diagram and / or flow chart, and the combination of the square boxes in the block diagram and / or flow chart can be implemented with a dedicated hardware-based system that performs the specified function or action, or can be implemented with a combination of special hardware and computer instructions.

[0173] The computer program product may be implemented in hardware, software or a combination thereof. In one optional embodiment, the computer program product is embodied as a computer storage medium, and in another optional embodiment, the computer program product is embodied as a software product, such as a software development kit (SDK) and the like.

[0174] The embodiments of the present disclosure have been described above, and the above description is exemplary, not exhaustive, and is not limited to the disclosed embodiments. Many modifications and changes will be apparent to those of ordinary skill in the art without departing from the scope and spirit of the described embodiments. The selection of terms used herein is intended to best explain the principles of the embodiments, practical applications, or improvements to the technology in the market, or to enable other persons of ordinary skill in the art to understand the embodiments disclosed herein.

Claims

1. A data set management method, characterized in that: include: Receive a data set processing request for a business container deployed on a physical machine, wherein the data set processing request includes an identifier of a data set used by the business container and a storage directory where the business container is mounted locally on the physical machine; In response to the data set processing request, a processing operation is performed on the data set corresponding to the identifier in the storage directory, wherein the processing operation includes: downloading the data set corresponding to the identifier to the storage directory, or deleting the data set corresponding to the identifier from the storage directory; After receiving the data set processing request for the business container, the method further includes: determining a processing progress of the processing operation; Returning the processing progress to the front end via the message middleware and / or the WebSocket protocol; The data set management method is applied to a distributed server, and the distributed server includes a request receiving module and a data set processing module; The step of performing a processing operation on the data set corresponding to the identifier in response to the data set processing request in the storage directory includes: The request receiving module receives a data set processing request, generates a data set processing message supported by the message middleware according to the data set processing request, and pushes the data set processing message to the message queue corresponding to the physical machine in the message middleware; The data set processing module located in the physical machine obtains the data set processing message in the message queue, and performs a processing operation on the data set corresponding to the identifier under the storage directory according to the identifier in the data set processing message and the storage directory.

2. The method according to claim 1, characterized in that: The distributed server also includes a processing progress push module and a Websocket server, and the processing progress is returned to the front end through the message middleware and the WebSocket protocol, including: The processing progress push module generates a processing progress message according to the processing progress, and pushes the processing progress message to the message middleware; The Websocket server obtains the processing progress message from the message middleware, and sends the processing progress in the processing progress message to the front-end Websocket client.

3. The method according to any one of claims 1-2, characterized in that: After downloading the data set corresponding to the identifier to the storage directory, the method further includes: receiving a service processing request for the service container; In response to the business processing request, a business processing operation is performed using the data set in the storage directory.

4. The method according to any one of claims 1-2, characterized in that: The method further comprises: In response to closing the service container, the data set in the storage directory is deleted.

5. The method according to any one of claims 1-2, characterized in that: Downloading the data set corresponding to the identifier to the storage directory includes: The data set corresponding to the identifier is downloaded from the storage directory of the database or other container to the storage directory of the business container mounted on the local physical machine.

6. A data set management system, characterized in that: include: A request receiving module receives a data set processing request for a business container deployed on a physical machine, wherein the data set processing request includes an identifier of a data set used by the business container and a storage directory where the business container is mounted locally on the physical machine; a data set processing module, in response to the data set processing request, performing a processing operation on the data set corresponding to the identifier in the storage directory, the processing operation including: downloading the data set corresponding to the identifier to the storage directory, or deleting the data set corresponding to the identifier from the storage directory; The system further comprises: A progress determination module, which determines the processing progress of the processing operation; The progress return module returns the processing progress to the front end through the message middleware and / or the WebSocket protocol; A request receiving module and a data set processing module; wherein the request receiving module receives a data set processing request, generates a data set processing message supported by the message middleware according to the data set processing request, and pushes the data set processing message to the message queue corresponding to the physical machine in the message middleware; the data set processing module located in the physical machine obtains the data set processing message in the message queue, and according to the identifier in the data set processing message and the storage directory, performs processing operations on the data set corresponding to the identifier under the storage directory.

7. A data set management device, characterized in that: include: A request receiving unit, configured to receive a data set processing request for a business container deployed on a physical machine, wherein the data set processing request includes an identifier of a data set used by the business container and a storage directory where the business container is mounted locally on the physical machine; a data set processing unit, configured to, in response to the data set processing request, perform a processing operation on the data set corresponding to the identifier in the storage directory, wherein the processing operation includes: downloading the data set corresponding to the identifier to the storage directory, or deleting the data set corresponding to the identifier from the storage directory; The device also includes: a progress determination unit, configured to determine a processing progress of the processing operation; A progress returning unit, used to return the processing progress to the front end via a message middleware and / or a web socket protocol. The device is applied to a distributed server, and the distributed server includes a request receiving module and a data set processing module; The data set processing unit includes the request receiving module and the data set processing module, wherein the request receiving module receives a data set processing request, generates a data set processing message supported by the message middleware according to the data set processing request, and pushes the data set processing message to the message queue corresponding to the physical machine in the message middleware; the data set processing module located in the physical machine obtains the data set processing message in the message queue, and according to the identifier in the data set processing message and the storage directory, performs processing operations on the data set corresponding to the identifier under the storage directory.

8. An electronic device, characterized in that: include: processor; a memory for storing processor-executable instructions; The processor is configured to call the instructions stored in the memory to execute the method described in any one of claims 1 to 5.

9. A computer-readable storage medium having computer program instructions stored thereon, characterized in that: When the computer program instructions are executed by a processor, the method according to any one of claims 1 to 5 is implemented.

Images