Containerization-based cloud-edge collaborative ocean data quality control model deployment method and system

Abstract

The invention provides a containerization-based cloud-edge collaborative ocean data quality control model deployment method and system, and the method comprises the steps: carrying out the packaging and deployment of a model through employing a container technology, and achieving the quick migration and efficient operation of the model on heterogeneous edge equipment; on the basis of reuse of a Docker layer, a content definition partitioning algorithm is proposed to carry out segmentation processing, so that the mirror image pulling amount is reduced to the maximum extent, and rapid deployment of model containerization is realized.

Description

Technical Field

[0001] The present invention belongs to the field of computer technology, and in particular to a container-based cloud-edge collaborative ocean data quality control model deployment method and system. Background Art

[0002] The statements in this section merely provide background information related to the present invention and do not necessarily constitute prior art.

[0003] With the rapid development and popularization of Internet of Things technology, the collection and processing of ocean data has become an important part of marine scientific research. A large number of ocean sensors are deployed around the world to monitor various aspects of the marine environment, such as temperature, salinity, tides, and water quality. The amount of data collected by these sensors is huge and highly heterogeneous and uncertain. In order to ensure the accuracy and reliability of these data in scientific research, data quality control has become an essential step.

[0004] With the development of big data and cloud computing technologies, more and more researchers have begun to use cloud platforms to train ocean data quality control models and process ocean data through machine learning methods. Cloud platforms have powerful computing power and flexible resource scheduling capabilities, and can efficiently process massive amounts of data. However, in the process of collecting data and uploading it to the cloud platform, there will be network delays and bandwidth limitations, making it difficult to respond to rapid changes in the ocean environment in real time.

[0005] In order to overcome the limitations of cloud computing, edge computing has emerged. Edge computing can reduce data transmission delays and improve real-time processing capabilities by performing calculations and processing near the source of data. However, the deployment of edge computing devices varies greatly, with significant differences in hardware and software configurations, showing a high degree of heterogeneity. This makes the models trained in the cloud face great limitations when deploying and updating them to edge devices, and even fail to operate normally after deployment. Given the complexity and variability of the marine environment, the cloud needs to continuously train and optimize the marine data quality control model based on the latest data, and regularly push updates to edge devices. In addition, during the model update process, when a new version of the model image is pulled through Docker, multiple or all layers of the new version of the model image will differ from the previous version. Although these differences may be small, Docker's layer reuse mechanism may still not be able to treat these layers as the same, resulting in the edge node being unable to use the stored image layers. Therefore, each update requires re-downloading all image layers with all differences, which not only prolongs the time required for the update, but also significantly increases bandwidth consumption.

[0006] In summary, how to solve the heterogeneity of edge devices, as well as the lag in model updates and the limitations of the Docker layer reuse mechanism are technical problems that need to be solved at present. Summary of the invention

[0007] In order to overcome the shortcomings of the above-mentioned prior art, the present invention provides a container-based cloud-edge collaborative ocean data quality control model deployment method and system, which uses container technology to encapsulate and deploy the model, and realizes the rapid migration and efficient operation of the model on heterogeneous edge devices; based on the reuse of the Docker layer, it proposes to use the content-defined block algorithm for segmentation processing, in order to minimize the amount of image pulling and realize the rapid deployment of model containerization.

[0008] To achieve the above objectives, a first aspect of the present invention provides a container-based cloud-edge collaborative ocean data quality control model deployment method, the method comprising:

[0009] Use Docker to package the ocean data quality control model and model dependencies into a Docker image;

[0010] Determine the image layer to be updated from the cloud node image repository based on the layer reuse mechanism, use the content-defined block algorithm to segment the data of the image layer to be updated and the data of the image layer stored in the local image repository of the edge node, and calculate the hash value of each data segment;

[0011] Compare the data segment hash of the image layer to be updated with the data segment hash value of the storage image layer to determine the data segment used for image layer reconstruction;

[0012] The data segments used for image layer reconstruction are transmitted, and the ocean data quality control model is updated and deployed.

[0013] A second aspect of the present invention provides a containerized cloud-edge collaborative ocean data quality control model deployment system, including:

[0014] The packaging module is used to package the ocean data quality control model and model dependencies into a Docker image using Docker;

[0015] A calculation module is used to determine the image layer to be updated pulled from the image repository according to the layer reuse mechanism, segment the data of the image layer to be updated and the data of the image layer stored in the local image repository using the content-defined block algorithm, and calculate the hash value of each data segment;

[0016] A comparison module, used to compare the hash value of the data segment of the image layer to be updated with the hash value of the data segment of the storage image layer, and determine the data segment used for image layer reconstruction;

[0017] The update deployment module is used to transmit the data segments used for image layer reconstruction and update the deployment of the ocean data quality control model.

[0018] The third aspect of the present invention provides a computer device, including: a processor, a memory and a bus, the memory storing machine-readable instructions executable by the processor, when the computer device is running, the processor and the memory communicate through the bus, and when the machine-readable instructions are executed by the processor, a containerized cloud-edge collaborative ocean data quality control model deployment method is executed.

[0019] A fourth aspect of the present invention provides a computer-readable storage medium having a computer program stored thereon, and when the computer program is run by a processor, a method for deploying a containerized cloud-edge collaborative ocean data quality control model is executed.

[0020] One or more of the above technical solutions have the following beneficial effects:

[0021] In the present invention, Docker is used to encapsulate the marine data quality control model and model dependencies. The Docker container deployment model technology ensures the consistency of the application environment and the standardization of dependencies, allowing different devices to run the same application under the same configuration, thereby simplifying software deployment and management in diverse hardware and operating system environments. In addition, containerization also improves the scalability and portability of the system, allowing the work of marine data processing and analysis to be more flexibly migrated to different computing environments, including the cloud and edge nodes. This greatly enhances the real-time data processing capabilities of marine scientific research, optimizes resource utilization, and reduces maintenance costs.

[0022] In the present invention, after utilizing the layer reuse mechanism, a content-defined blocking algorithm is used to supplement it, thereby reducing the amount of data transmission and update delay in the process of updating the model through the container. By more finely controlling the reuse of data segments, even if only a small part of the data has changed and layer reuse cannot be performed, the existing data can be effectively utilized, thereby improving update efficiency and saving bandwidth resources.

[0023] Advantages of additional aspects of the present invention will be given in part in the following description, and in part will become obvious from the following description, or will be learned through practice of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] The accompanying drawings in the specification, which constitute a part of the present invention, are used to provide a further understanding of the present invention. The exemplary embodiments of the present invention and their descriptions are used to explain the present invention and do not constitute improper limitations on the present invention.

[0025] Figure 1 This is a diagram of the existing Docker image before and after the update;

[0026] Figure 2 This is a schematic diagram of image layer segmentation in Embodiment 1 of the present invention;

[0027] Figure 3 This is a block diagram of the containerized cloud-edge collaborative ocean data quality control model deployment method in Example 1 of the present invention. DETAILED DESCRIPTION

[0028] It should be noted that the following detailed descriptions are exemplary and are intended to provide further explanation of the present invention. Unless otherwise specified, all technical and scientific terms used herein have the same meanings as those commonly understood by those skilled in the art to which the present invention belongs.

[0029] It should be noted that the terms used herein are for describing specific embodiments only and are not intended to be limiting of exemplary embodiments according to the present invention.

[0030] In the absence of conflict, the embodiments of the present invention and the features of the embodiments may be combined with each other.

[0031] Embodiment 1

[0032] This embodiment discloses a container-based cloud-edge collaborative ocean data quality control model deployment method, including:

[0033] Use Docker to package the ocean data quality control model and model dependencies into a Docker image;

[0034] Determine the image layer to be updated from the cloud node image repository based on the layer reuse mechanism, use the content-defined block algorithm to segment the data of the image layer to be updated and the data of the image layer stored in the local image repository of the edge node, and calculate the hash value of each data segment;

[0035] Compare the data segment hash of the image layer to be updated with the data segment hash value of the storage image layer to determine the data segment used for image layer reconstruction;

[0036] The data segments used for image layer reconstruction are transmitted, and the ocean data quality control model is updated and deployed.

[0037] The following is a detailed description of the improved Docker method for pulling a new version image in this embodiment:

[0038] This embodiment chooses to use Docker to deploy the model in containerized form to solve the problem of heterogeneity of edge devices. Through containerization technology, the model and its dependencies can be encapsulated in an independent container, thereby ensuring that the model can run consistently in different hardware and operating system environments. However, in the process of using Docker to update and deploy the model, due to the limitations of its own layer reuse mechanism, new images will be pulled frequently, and these image files are generally large, which puts great pressure on network transmission, thereby affecting deployment efficiency. Therefore, this embodiment chooses to study the deployment mechanism of Docker, in order to minimize the amount of image pulls and achieve rapid deployment of model containerization.

[0039] Docker images are composed of multiple layers, each of which represents a part of the file system. Each time a new version of the image is updated, the image layers will change. When using Docker to update and deploy a model, after Docker receives the update instruction, it will connect to the image library and request the required image version. According to the layer reuse mechanism, when pulling a new version of the image, Docker will analyze each layer that makes up the image, and some layers that exist in the local cache will be directly reused. For layers that do not exist in the local cache, the Docker client will send a request to the remote repository and only download those necessary layers, thereby reducing unnecessary data transmission. Once all the required layers are pulled locally, Docker will stack these layers in a specific order, rebuild the complete image, and start the container to run the new model.

[0040] However, if Figure 1 As shown in the figure, when the update of the Docker image involves changes to most or even all layers, the advantages of the layer reuse mechanism cannot be fully utilized. Because even if each image layer has only a small change, the layer reuse mechanism cannot take effect, which results in the need to re-download a large amount of data from the image library, resulting in a huge amount of data transmission, especially in the face of large-scale cloud-edge update deployment, which will put a huge pressure on data transmission and reduce the efficiency of model update deployment. At the same time, according to research, most of the files in the image are identical and redundant, and only 3% of the files are critical and unique. Therefore, based on this feature, the layer reuse mechanism can be studied and further optimized to improve deployment efficiency.

[0041] Considering that the time required to download the image for updating and deploying a new model is T, T is related to the image transmission volume W and the network transmission speed V, that is, Since the network bandwidth is limited and there is data uploaded from the edge to the cloud, when W is too large, it will cause network congestion, resulting in a decrease in the network transmission speed V, which will further increase the value of time T. Therefore, under the current network transmission constraints, only by reducing W as much as possible can the deployment efficiency of the model be further improved.

[0042] In order to reduce the data transmission volume W as much as possible, this embodiment further proposes a Docker update mechanism of segment reuse based on layer reuse.

[0043] When the model is updated, first, the image layer that needs to be pulled from the image repository is determined according to the layer reuse mechanism. min =W0-W1(W1>=0), where W0 is the data volume of the new version image layer, and W1 is the data volume of the locally stored image layer. When all layers of the new and old versions of the image are completely different, W1=0. Figure 2 As shown, in the next step, since the change of the image layer can be regarded as inserting, deleting and modifying data on the image layer of the old version, the CDC algorithm can be used to analyze the data content of the old version of the image layer, determine the segment boundaries and segment them to form the old version data segment set C, and calculate the hash value of each data segment in C and store it in the hash set H C Then, the data content of the new version image layer is analyzed in the same way, the segment boundaries are determined and segmented to form a new version data segment set C′, the hash value of each data segment in C′ is calculated and stored in the hash set H′ C middle.

[0044] Among them, the CDC algorithm is used to slide a 4KB window on the data stream, moving one step (512 bytes) each time, and the hash value of the data is calculated in each sliding window. When the hash value of the data in the window matches a preset specific value, the position of the window is determined as the segment boundary.

[0045] In this way, the set of reusable data segments is E = C∩C′, where the data segment d is:

[0046]

[0047] Among them, h d The hash value of the data segment.

[0048] Based on this, the minimum amount of data that needs to be transmitted for model update deployment is:

[0049]

[0050] Among them, W E is the amount of data in the reusable data segment set E.

[0051] This method can greatly reduce the amount of data transmission during the container update process, reduce network burden, and improve deployment efficiency.

[0052] This embodiment combines Figure 3A container-based cloud-edge collaborative ocean data quality control model deployment method proposed in this embodiment is described in detail.

[0053] 1. Cloud Node

[0054] A computing platform with powerful computing capabilities is selected as the cloud node, which is responsible for executing the training of ocean data quality control models, automatically generating Docker images, issuing model update commands, and preparing for model update deployment.

[0055] 1.1 Training the ocean data quality control model: Regularly collect raw data, post-quality control data, quality control anomaly data and other data from each edge terminal, and train and optimize the ocean data quality control model based on this data to continuously improve the accuracy of the model.

[0056] 1.2 Automatically generate Docker images: Set up a listener to monitor whether new models are generated in the cloud. When a new model is generated by cloud training, call Docker to execute the image building operation, package the model and its required environment and dependencies (including a specific version of Python, Pytorch framework, etc.) into an image, and upload it to the Docker Hub image repository.

[0057] 1.3 Publishing model update commands: Use the Webhook in the Docker Hub image repository to monitor the generation of new images. When a new image is detected, publish the model update command to the edge node and call the model update deployment preparation function.

[0058] 1.4 Model update deployment preparation: After being called, the running version image layer set L = {l0,l1,l2,...,l n} and the new version image layer (i.e., the image of the model generated by new training that needs to be updated to the edge node) set L′={l′0,l′1,l′2,...,l′ n}, first calculate the hash value of each image layer in the two image layer sets and store them in the hash table H L and H L′ In the hash table H L′ The hash value and hash table H in L The hash values ​​in the hash table H are compared one by one. When the hash values ​​are found to be the same, the hash value and the hash table H L′ The layer serial number of the corresponding image layer in is inserted into the build table (used to reconstruct the image accordingly), and the hash value and hash table H L The layer serial number of the corresponding image layer is inserted into the hash index (used for fast reuse of the corresponding layer during reconstruction).

[0059] Next, the different image layers corresponding to the remaining hash values ​​in the image layer set L and the image layer set L′ are segmented, and the hash value of each segment is calculated. The hash value of the segmented data segment of the remaining image layers in the image layer set L and its relative address are inserted into the hash index, and the hash value of the segmented data segment of the remaining layers in the image layer set L′, as well as the layer serial number of the corresponding image layer and the order of data segment generation are inserted into the build table.

[0060] Finally, the hash value in the build table is compared with the hash value in the hash index. If the hash value does not exist in the hash index, the corresponding data segment is inserted into the data segment library, which stores the data segments that need to be transmitted to build a new version of the container image layer. In addition, the hash value also needs to be inserted into the hash index, and the address of the data segment in the data segment library is recorded. After completing the above operations, the hash index, build table, and data segment library are packaged and sent to the edge node, and a signal that the model update deployment is ready is sent.

[0061] 2. Edge Nodes

[0062] Edge devices with certain computing capabilities close to data acquisition devices are selected as edge nodes. Their main functions include ocean data quality control, model update, image layer reconstruction, etc.

[0063] 2.1 Ocean data quality control: Use the running ocean data quality control model to perform real-time quality control on the ocean observation data uploaded by the data acquisition equipment to provide users with accurate ocean observation data.

[0064] 2.2 Model update: After receiving the model update deployment instruction from the cloud, call the image layer reconstruction function, wait for the feedback of the image layer reconstruction completion, and start the container to complete the model update.

[0065] 2.3 Image layer reconstruction: After receiving the instruction to update the model, pull the image layer of the local running version model and wait for the cloud model update deployment to be ready.

[0066] After receiving the hash index, build table, and data segment library from the cloud, the hash index is queried according to the hash value in the build table, and the required image layer and data segment are obtained from the local image layer and data segment library; then the image layer is reconstructed according to the layer sequence number and data segment generation order of the image layer corresponding to the hash value in the build table, and feedback is given to the model update deployment function after the reconstruction is completed.

[0067] Embodiment 2

[0068] The purpose of this embodiment is to provide a container-based cloud-edge collaborative ocean data quality control model deployment system, including:

[0069] The packaging module is used to package the ocean data quality control model and model dependencies into a Docker image using Docker;

[0070] A calculation module is used to determine the image layer to be updated pulled from the image repository according to the layer reuse mechanism, segment the data of the image layer to be updated and the data of the image layer stored in the local image repository using the content-defined block algorithm, and calculate the hash value of each data segment;

[0071] A comparison module, used to compare the hash value of the data segment of the image layer to be updated with the hash value of the data segment of the storage image layer, and determine the data segment used for image layer reconstruction;

[0072] The update deployment module is used to transmit the data segments used for image layer reconstruction and update the deployment of the ocean data quality control model.

[0073] Embodiment 3

[0074] The purpose of this embodiment is to provide a computing device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps of the above method when executing the program.

[0075] Embodiment 4

[0076] The purpose of this embodiment is to provide a computer-readable storage medium.

[0077] A computer-readable storage medium stores a computer program, which executes the steps of the above method when executed by a processor.

[0078] The steps involved in the apparatuses of the above embodiments 2, 3 and 4 correspond to the method embodiment 1, and the specific implementation methods can refer to the relevant description part of embodiment 1. The term "computer-readable storage medium" should be understood as a single medium or multiple media including one or more instruction sets; it should also be understood to include any medium that can store, encode or carry an instruction set for execution by a processor and enable the processor to execute any method in the present invention.

[0079] Those skilled in the art should understand that the modules or steps of the present invention described above can be implemented by a general-purpose computer device, or alternatively, they can be implemented by a program code executable by a computing device, so that they can be stored in a storage device and executed by the computing device, or they can be made into individual integrated circuit modules, or multiple modules or steps therein can be made into a single integrated circuit module for implementation. The present invention is not limited to any specific combination of hardware and software.

[0080] Although the above describes the specific implementation mode of the present invention in conjunction with the accompanying drawings, it is not intended to limit the scope of protection of the present invention. Those skilled in the art should understand that various modifications or variations that can be made by those skilled in the art on the basis of the technical solution of the present invention without creative work are still within the scope of protection of the present invention.

Claims

1. A container-based cloud-edge collaborative ocean data quality control model deployment method, characterized in that: The method comprises: Use Docker to package the ocean data quality control model and model dependencies into a Docker image; Determine the image layer to be updated from the cloud node image repository based on the layer reuse mechanism, use the content-defined block algorithm to segment the data of the image layer to be updated and the data of the image layer stored in the local image repository of the edge node, and calculate the hash value of each data segment; Compare the data segment hash of the image layer to be updated with the data segment hash value of the storage image layer to determine the data segment used for image layer reconstruction; The ocean data quality control model is updated and deployed based on the data segments used for image layer reconstruction.

2. The containerized cloud-edge collaborative ocean data quality control model deployment method according to claim 1 is characterized in that: Compare the data segment hash of the image layer to be updated with the data segment hash value of the storage image layer to determine the data segment used for image layer reconstruction, specifically: The cloud node pulls the edge node running version image layer set and the new version image layer set from the image repository; Calculate the hash value of each image layer in the two image layer sets respectively and store them in the corresponding hash tables respectively; Compare the hash values ​​in the two hash tables. If the hash values ​​are the same, insert the hash value and the layer serial number of the corresponding new version image layer into the build table, and insert the hash value and the layer serial number of the corresponding edge node running version image layer into the hash index. The hash value in the build table is compared with the hash value in the index table. If the hash value does not exist in the hash index, the corresponding data segment is inserted into a data segment library; wherein the data segment library stores the transmitted data segments to construct the reconstructed image layer.

3. The containerized cloud-edge collaborative ocean data quality control model deployment method according to claim 2 is characterized in that: Before comparing the hash values ​​in the build table with the hash values ​​in the index table, it also includes: The different image layers corresponding to the remaining hash values ​​in the two hash tables are segmented and the hash value of each image layer is calculated; Insert the hash value of the segmented data segment in the edge node running version image layer and the corresponding address into the hash index; Insert the hash value of the data segment after segmentation of the new version of the image layer, as well as the corresponding image layer serial number and the order in which the segmented data segments are generated into the build table.

4. The containerized cloud-edge collaborative ocean data quality control model deployment method according to claim 2 or 3, characterized in that: After the edge node receives the component table, data segment library and hash index sent by the cloud node, it queries the hash index according to the hash value in the build table, obtains the required image layer and data segment from the local image layer and data segment library, and reconstructs the image layer according to the image layer serial number and data segment generation order corresponding to the hash value in the build table.

5. The container-based cloud-edge collaborative ocean data quality control model deployment method according to claim 1, characterized in that: The content-defined block algorithm is used to segment the data of the image layer to be updated and the data of the image layer stored in the local image repository. Specifically, a sliding window is used to slide on the data segment, and the hash value of the data in each sliding window is calculated. When the calculated hash value of the data in the window matches a preset specific value, the position of the sliding window is determined as the segment boundary.

6. The container-based cloud-edge collaborative ocean data quality control model deployment method according to claim 1 is characterized in that: The cloud node obtains the training data of each edge node, trains and optimizes the ocean data quality control model, and uses the listener to monitor whether the cloud node has a new ocean data quality control model generated. When the cloud node generates a new ocean data quality control model through training, it calls Docker to execute the image building operation, packages the new ocean data quality control model and its required environment and dependencies into a Docker image, and uploads it to the Docker Hub image repository.

7. A containerized cloud-edge collaborative ocean data quality control model deployment system, characterized by: include: The packaging module is used to package the ocean data quality control model and model dependencies into a Docker image using Docker; A calculation module is used to determine the image layer to be updated from the cloud node image repository according to the layer reuse mechanism, and to segment the data of the image layer to be updated and the data of the image layer stored in the local image repository of the edge node by using the content-defined block algorithm, and to calculate the hash value of each data segment; A comparison module, used to compare the hash value of the data segment of the image layer to be updated with the hash value of the data segment of the storage image layer, and determine the data segment used for image layer reconstruction; The update deployment module is used to transmit the data segments used for image layer reconstruction and update the deployment of the ocean data quality control model.

8. The containerized cloud-edge collaborative ocean data quality control model deployment system according to claim 7, wherein in the comparison module, the data segment hash of the image layer to be updated is compared with the data segment hash value of the storage image layer to determine the data segment for image layer reconstruction, specifically: The cloud node pulls the edge node running version image layer set and the new version image layer set from the image repository; Calculate the hash value of each image layer in the two image layer sets respectively and store them in the corresponding hash tables respectively; Compare the hash values ​​in the two hash tables. If the hash values ​​are the same, insert the hash value and the layer serial number of the corresponding new version image layer into the build table, and insert the hash value and the layer serial number of the corresponding edge node running version image layer into the hash index. The hash value in the build table is compared with the hash value in the index table. If the hash value does not exist in the hash index, the corresponding data segment is inserted into a data segment library; wherein the data segment library stores the transmitted data segments to construct the reconstructed image layer.

9. A computer device, characterized in that: include: A processor, a memory and a bus, wherein the memory stores machine-readable instructions executable by the processor. When the computer device is running, the processor and the memory communicate through the bus. When the machine-readable instructions are executed by the processor, the containerized cloud-edge collaborative ocean data quality control model deployment method according to any one of claims 1 to 6 is executed.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by the processor, the containerized cloud-edge collaborative ocean data quality control model deployment method according to any one of claims 1 to 6 is executed.

Images