Data processing method and device based on encrypted data, equipment and medium
By encrypting and decrypting the original data, model training is carried out in a preset training container, and model files are encrypted, the problem of model training and data processing is solved while ensuring data security, and a high-security data processing process is realized.
Patent Information
- Application Number
- CN202411888453.2
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-12-20
- Publication Date
- 2025-03-28
AI Technical Summary
On the premise of ensuring data security, it is difficult for the existing technology to effectively solve this problem using data for model training and using the trained model to process the data.
By encrypting the original data, generating a copy of the encrypted data, decrypting and model training in a preset training container, a data processing model is obtained, and then the model file is encrypted and deployed to the target environment for decryption and data processing.
Improve the security of using data for model training and data processing, ensure the security of data during transmission and storage, and avoid data leakage and abuse.
Smart Images
Figure CN119720251A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of data processing technology, and in particular to a data processing method, device, equipment and medium based on encrypted data. Background Art
[0002] With the advent of the big data era, sensitive data is increasingly being used in various fields. Among these sensitive data, some information is extremely important. For enterprises, if the business secrets and customer information in the sensitive data are illegally leaked, the enterprise will lose its competitive advantage and cause great economic losses.
[0003] At the same time, with the development of computer technology, neural networks, as a powerful data analysis tool, are widely used in various research projects, such as in universities and external collaborative projects. However, since neural networks require a large amount of data for training, these training data may contain sensitive data.
[0004] From the above, it can be seen that how to use data for model training while ensuring data security and use the trained model to process the data is an urgent problem to be solved. Summary of the invention
[0005] In view of this, the purpose of the present invention is to provide a data processing method, device, equipment and medium based on encrypted data, which can improve the security of using data for model training and using the trained model to process data. The specific scheme is as follows:
[0006] In a first aspect, the present application provides a data processing method based on encrypted data, comprising:
[0007] Encrypting the original data to generate a data encryption copy corresponding to the obtained original encrypted data, and creating a model training task; the model training task includes the data encryption copy and the model training algorithm;
[0008] The model training task is transmitted to a preset training container, and a network communication port of the preset training container is closed, and then the preset training container is controlled to decrypt the encrypted copy of the data to obtain a copy of the data;
[0009] Inputting the data copy into a preset model training framework in the preset training container, so as to train the preset model training framework using the model training algorithm to obtain a data processing model, and destroying the data copy and the preset training container;
[0010] The model file corresponding to the data processing model is encrypted, and the encrypted model file is deployed to the target environment, so that the encrypted model file can be decrypted in the target environment, and the target data can be processed using the data processing model obtained after decryption.
[0011] Optionally, encrypting the original data to generate a data encrypted copy corresponding to the obtained original encrypted data includes:
[0012] Performing permission authentication based on the user authentication information issued by the current user interface to obtain a permission authentication result, and judging whether the user has data access rights based on the permission authentication result;
[0013] If the user does not have data access rights, return to the current user interface to perform permission authentication on the user authentication information;
[0014] If the user has data access rights, the original data selected by the user is encrypted using a preset encryption algorithm to obtain original encrypted data, and the original encrypted data is transmitted to a preset object storage system, and the original encrypted data is decrypted in the preset object storage system, and then a corresponding data encryption copy is generated based on the obtained original data.
[0015] Optionally, after encrypting the original data to generate a data encrypted copy corresponding to the obtained original encrypted data, the method further includes:
[0016] The original encrypted data and the encrypted copy of the data are stored in a preset object storage system, and respond to various data processing instructions; the data processing instructions include data query instructions, data addition instructions, data modification instructions and data deletion instructions;
[0017] The response to the data query instruction is to display the query results of the data in a list form; the query results of the data include the data name, data type, data label, data storage location, data creator and data creation time;
[0018] In response to the data adding instruction, the data name, data label, data description and model file encryption method input by the user are received;
[0019] Responding to the data modification instruction is receiving the user's modification content of the data name, data label or data description, and saving the modified content after receiving the save instruction;
[0020] The data deletion instruction is responded to by deleting the data selected by the user.
[0021] Optionally, after creating the model training task, the step further includes:
[0022] Respond to each task function instruction corresponding to the model training task; the task function instruction includes a model training task query instruction, a model training task addition instruction, a model training task deletion instruction, and a task history execution instruction;
[0023] Wherein, responding to the model training task query instruction is to display the query results of the model training task in the form of a list; the query results of the model training task include the task name, task description, algorithm image name, data name, task status, task creator and task creation time;
[0024] Responding to the new model training task instruction includes receiving a task name, task description, algorithm image, training data, and a path for loading the training data into a preset training container corresponding to the new model training task;
[0025] Responding to the model training task deletion instruction is to delete the model training task selected by the user;
[0026] Responding to the task history execution instruction is to view the result record of the last execution of the model training task selected by the user.
[0027] Optionally, before transferring the model training task to a preset training container, the method further includes:
[0028] Configure the operating environment of the preset training container, select the target model training framework from various model training frameworks for integration based on user needs, and package the obtained integration results into a deployable container image;
[0029] The deployable container image is scheduled to create the preset training container based on the task requirements corresponding to the model training task, and corresponding computer resources are allocated to each model training task so that the preset model training framework uses the corresponding computer resources to execute the model training task to obtain a data processing model.
[0030] Optionally, before selecting a target model training framework from various model training frameworks for integration based on user needs, the process further includes:
[0031] Respond to each algorithm function instruction corresponding to the model training algorithm; the algorithm function instruction includes an algorithm query instruction, an algorithm addition instruction, and an algorithm deletion instruction;
[0032] Wherein, the response to the algorithm query instruction is to display the query results based on the preset algorithm rules in the form of a list; the query results based on the preset algorithm rules include the algorithm name, algorithm description, algorithm type, algorithm creator and algorithm creation time;
[0033] In response to the algorithm adding instruction, the algorithm image name, algorithm image description, algorithm image type and image file input by the user are received, and the newly added algorithm content is saved when the algorithm saving instruction is obtained;
[0034] The algorithm deletion instruction is responded to by deleting the algorithm image file selected by the user.
[0035] Optionally, before deploying the encrypted model file to the target environment, the process further includes:
[0036] Respond to each task deployment function instruction corresponding to the encrypted model file; the task deployment function instruction includes a model deployment task query instruction, a model deployment task addition instruction, a model deployment task deletion instruction, and a model deployment task stop instruction;
[0037] Among them, respond to the model deployment task query instruction and display the query result of the model deployment task in the form of a list; the query result of the model deployment task is to display the deployment task name, deployment task description, model type, image name, model name, deployment task port, deployment task status, creator of the deployment task and creation time of the deployment task in the form of a task list;
[0038] In response to the new instruction of the model deployment task, the deployment task name, deployment task description, path for loading the model into a preset training container, model type, and container image for inference are received from the user, and the newly added model deployment task is saved after obtaining the deployment task save instruction;
[0039] In response to the model deployment task deletion instruction, the deployed model selected by the user is deleted;
[0040] Responding to the model deployment task stop instruction is to stop the deployment task being executed that is selected by the user.
[0041] In a second aspect, the present application provides a data processing device based on encrypted data, comprising:
[0042] A data encryption module, used to encrypt the original data so as to generate a data encryption copy corresponding to the obtained original encrypted data, and create a model training task; the model training task includes the data encryption copy and the model training algorithm;
[0043] A task transmission module, used to transmit the model training task to a preset training container, close the network communication port of the preset training container, and then control the preset training container to decrypt the encrypted copy of the data to obtain a data copy;
[0044] A copy transmission module, used to input the data copy into the preset model training framework in the preset training container, so as to train the preset model training framework using the model training algorithm to obtain a data processing model, and destroy the data copy and the preset training container;
[0045] The file encryption module is used to encrypt the model file corresponding to the data processing model and deploy the encrypted model file to the target environment so as to decrypt the encrypted model file in the target environment and process the target data using the data processing model obtained after decryption.
[0046] In a third aspect, the present application provides an electronic device, including:
[0047] Memory, used to store computer programs;
[0048] The processor is used to execute the computer program to implement the aforementioned data processing method based on encrypted data.
[0049] In a fourth aspect, the present application provides a computer-readable storage medium for storing a computer program, wherein the computer program, when executed by a processor, implements the aforementioned data processing method based on encrypted data.
[0050] As can be seen from the above, before performing data processing based on encrypted data, the present application needs to encrypt the original data in order to generate an encrypted data copy corresponding to the obtained original encrypted data, and create a model training task; the model training task includes the encrypted data copy and the model training algorithm; the model training task is transmitted to a preset training container, and the network communication port of the preset training container is closed, and then the preset training container is controlled to decrypt the encrypted data copy to obtain a data copy; the data copy is input into the preset model training framework in the preset training container, so that the preset model training framework executes the model training task to obtain a data processing model, and destroys the data copy and the preset training container; the model file corresponding to the data processing model is encrypted, and the encrypted model file is deployed to the target environment, so that the encrypted model file is decrypted in the target environment, and the target data is processed using the data processing model obtained after decryption.
[0051] It can be seen that the present application generates a data encryption copy based on the original encrypted data, transfers the model training task to the preset training container, closes the network communication port of the preset training container, and then inputs the data copy into the preset model training framework in the preset training container, so that the preset model training framework executes the model training task to obtain the data processing model, and destroys the data copy and the preset training container; encrypts the model file corresponding to the data processing model, and deploys the encrypted model file to the target environment, so that the encrypted model file can be decrypted in the target environment, and the data processing model obtained after decryption can be used to process the target data. In this way, the security of using data for model training and using the model obtained after training to process data can be improved, and the user experience can be improved. BRIEF DESCRIPTION OF THE DRAWINGS
[0052] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on the provided drawings without paying creative work.
[0053] Figure 1 A flow chart of a data processing method based on encrypted data disclosed in this application;
[0054] Figure 2 A flowchart of performing corresponding operations based on user authentication information disclosed in this application;
[0055] Figure 3 A schematic diagram of the principle of a specific data processing method based on encrypted data disclosed in this application;
[0056] Figure 4 A schematic diagram of the technical architecture of a specific data processing method based on encrypted data disclosed in this application;
[0057] Figure 5 This is a schematic diagram of the structure of a data processing device based on encrypted data disclosed in this application;
[0058] Figure 6 This is a structural diagram of an electronic device disclosed in this application. DETAILED DESCRIPTION
[0059] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.
[0060] With the advent of the big data era, sensitive data is being used more and more widely in various fields. Among these sensitive data, some information is extremely important. If the above information is leaked or abused, it will cause unpredictable losses to individuals, organizations or countries. To this end, the present application provides a data processing method based on encrypted data, which can improve the security of using data for model training and using the trained model to process data.
[0061] See also Figure 1 As shown, an embodiment of the present invention discloses a data processing method based on encrypted data, comprising:
[0062] Step S11, encrypt the original data to generate a data encryption copy corresponding to the obtained original encrypted data, and create a model training task; the model training task includes the data encryption copy and the model training algorithm.
[0063] In this embodiment, the existing data is mostly public data, and the concept that data is an important asset of the owner has not yet been deeply rooted in people's minds. Most of the existing data access systems do not fully consider whether data can be safely accessed and used. In addition, the access and use of some non-public data is only protected by setting the user's access rights. For some sensitive data and important data, it is far from enough to protect the data by only using the method of setting the user's access rights.
[0064] In this embodiment, before encrypting the original data, the user authentication information needs to be authenticated to determine whether the user has the data access authority based on the authentication result. Whether the user has the data access authority determines the subsequent operations. Figure 2 As shown. The original data can only be obtained if the user has data access rights. Specifically, the encrypting of the original data to generate a data encrypted copy corresponding to the obtained original encrypted data may include: performing permission authentication based on the user authentication information issued by the current user interface to obtain a permission authentication result, and judging whether the user has data access rights based on the permission authentication result; if the user does not have data access rights, returning to the current user interface to perform permission authentication on the user authentication information.
[0065] After having data access rights, the user needs to obtain the original data, encrypt the original data using a preset encryption algorithm, and generate a corresponding data encryption copy based on the encrypted original data. That is, if the user has data access rights, the original data selected by the user is encrypted using a preset encryption algorithm to obtain original encrypted data, and the original encrypted data is transmitted to a preset object storage system, and the original encrypted data is decrypted in the preset object storage system, and then a corresponding data encryption copy is generated based on the obtained original data.
[0066] In this embodiment, both the original encrypted data and the encrypted copy of the data can be securely stored in MINIO (i.e., a distributed object storage system) of the local workstation, or can be securely transmitted over the network. Since the data has been encrypted, even if data leakage occurs during storage or transmission, the attacker cannot directly obtain the content of the encrypted data.
[0067] It is worth mentioning that the embodiment of the present application can perform corresponding operations on the data by clicking various data function buttons. Specifically, after encrypting the original data to generate a data encryption copy corresponding to the obtained original encrypted data, it can also include: storing the original encrypted data and the data encryption copy in a preset object storage system, and responding to various data processing instructions; the data processing instructions include data query instructions, data addition instructions, data modification instructions and data deletion instructions; wherein, responding to the data query instruction is to display the query results of the data in a list form; the query results of the data include data name, data type, data label, data storage location, data creator and data creation time; responding to the data addition instruction is to receive the data name, data label, data description and model file encryption method entered by the user; responding to the data modification instruction is to receive the user's modification content of the data name, data label or data description, and save the modification content after receiving the save instruction; responding to the data deletion instruction is to delete the data selected by the user.
[0068] In the first specific implementation, the user can respond to the data query instruction by clicking the data query button, and the results of the data query are displayed in a list format, and the displayed table header includes data name, data type, data label, storage location, data creator and data creation time.
[0069] In the second specific implementation, the user can intuitively see all current training tasks in the MINIO of the local workstation, and can perform fuzzy matching of data by inputting data labels.
[0070] In a third specific implementation, the user can respond to the training data adding instruction by clicking the Add button, that is, the user can search for data by entering the data name, data label, and selecting the data encryption method and encryption file.
[0071] In a fourth specific implementation, the user may respond to the save instruction by clicking a save button, that is, after the user uploads new data, the user may save the data by clicking a save button.
[0072] In the fifth specific implementation, the user can respond to the data modification instruction by clicking a modification button, that is, the user can modify the data label and data description, and click a data save button after the data modification is completed to save the modified data.
[0073] In a sixth specific implementation, the user can respond to the data deletion instruction by clicking a delete button, that is, the user can delete the currently selected data by clicking the delete button.
[0074] In addition, the embodiment of the present application can also perform corresponding operations on the task by clicking various task function buttons. Specifically, after creating the model training task, it can also include: responding to each task function instruction corresponding to the model training task; the task function instruction includes a model training task query instruction, a model training task new instruction, a model training task deletion instruction, and a task history execution instruction; wherein, responding to the model training task query instruction is to display the query results of the model training task in a list form; the query results of the model training task include the task name, task description, algorithm image name, data name, task status, task creator, and task creation time; responding to the model training task new instruction includes receiving the task name, task description, algorithm image, training data corresponding to the new model training task, and the path of loading the training data into the preset training container; responding to the model training task deletion instruction is to delete the model training task selected by the user; responding to the task history execution instruction is to view the result record of the last execution of the model training task selected by the user.
[0075] In the first specific implementation, the user can respond to the training task query instruction by clicking the training task query button, wherein the training task query results are displayed in a list format, and the list header includes the task name, task description, algorithm image name, data name, status, creator, and creation time. In addition, the user can intuitively see all current training tasks and perform fuzzy matching on the tasks by entering the task name.
[0076] In the second specific implementation, the user can respond to the training task addition instruction by clicking the training task addition button, that is, the user adds a training task by entering the task name, task description, selecting the algorithm image, selecting the data, entering the path of the container into which the data is loaded, and saves the training task after clicking the Save button.
[0077] In a third specific implementation, the user can respond to the training task deletion instruction by clicking the training task deletion button, that is, the user can delete the currently selected training task by clicking the training task deletion button.
[0078] In a fourth specific implementation, the user can respond to the data dashboard task history execution instruction by clicking the task history execution button, that is, the user can view the execution record of the last execution operation of the currently selected training task.
[0079] Step S12: transmitting the model training task to a preset training container, closing the network communication port of the preset training container, and then controlling the preset training container to decrypt the encrypted copy of the data to obtain a data copy.
[0080] In this embodiment, before transferring the model training task to the preset training container, the operating environment corresponding to the preset training container needs to be pre-configured in this application, and the selected target model training framework is integrated to be packaged as a deployable container image, and the deployable container image is used to create the image. It is worth mentioning that when selecting a model training framework for integration based on user needs, the embodiment of the present application can use a container as a running carrier. Therefore, the embodiment of the present application can support a variety of popular deep learning framework options, such as PyTorch (i.e., a machine learning library), TensorFlow (i.e., a symbolic mathematical system based on data flow programming), MXNet (i.e., a deep learning framework), etc. Users can choose the most suitable framework according to their needs to write code and frameworks for training and reasoning, and package them into deployable container images.
[0081] Specifically, before transmitting the model training task to the preset training container, it may also include: configuring the operating environment of the preset training container and selecting a target model training framework from various model training frameworks for integration based on user needs, and packaging the obtained integration results into a deployable container image; scheduling the deployable container image to create the preset training container based on the task requirements corresponding to the model training task, and allocating corresponding computer resources to each model training task, so that the preset model training framework uses the corresponding computer resources to execute the model training task to obtain a data processing model.
[0082] In a specific implementation, when configuring the operating environment of the preset training container, the embodiment of the present application uses Docker (i.e., an open source project for building, publishing, and running applications) as a container operating platform to achieve container operation and management. Each model training task is isolated and managed through a Docker container to ensure the consistency of the operating environment and the isolation of computer resources.
[0083] In this embodiment, before selecting the target model training framework, the embodiment of the present application needs to select a model training algorithm based on user needs, so as to train the target model training framework using the model training algorithm. Specifically, before selecting the target model training framework from each model training framework for integration based on user needs, it can also include: responding to each algorithm function instruction corresponding to the model training algorithm; the algorithm function instruction includes an algorithm query instruction, an algorithm addition instruction, and an algorithm deletion instruction; wherein, responding to the algorithm query instruction is to display the query results based on the preset algorithm rules in a list form; the query results based on the preset algorithm rules include the algorithm name, algorithm description, algorithm type, algorithm creator, and algorithm creation time; responding to the algorithm addition instruction is to receive the algorithm image name, algorithm image description, algorithm image type, and image file input by the user, and save the newly added algorithm content when obtaining the algorithm save instruction; responding to the algorithm deletion instruction is to delete the algorithm image file selected by the user.
[0084] Step S13: input the data copy into the preset model training framework in the preset training container, so as to train the preset model training framework using the model training algorithm to obtain a data processing model, and destroy the data copy and the preset training container.
[0085] In this embodiment, an encrypted data copy corresponding to the original encrypted data is obtained through an internal API (Application Programming Interface) interface, and after decrypting the encrypted data copy, the encrypted data copy is loaded into a preset training container for training or reasoning.
[0086] It is worth mentioning that the internal API interface in the embodiment of the present application includes but is not limited to: training status API interface, data upload API interface and reasoning API standard interface. Among them, the training status API interface is for the application in the preset training container to feedback the training results of the model by calling the training status API interface. The data upload API interface is used to support users to upload training data files, and encrypt the training data files and store them in MINIO. The reasoning API standard interface is used to provide an HTTP interface for the reasoning service and support data input and output.
[0087] In a specific implementation, the embodiment of the present application needs to transfer the trained data processing model to MINIO object storage for storage as a persistent storage solution. Among them, MINIO is a high-performance, distributed object storage system suitable for storing large-capacity unstructured data, such as pictures, videos, log files, backup data, container images, and virtual machine images. It is worth mentioning that MINIO provides high availability, data redundancy and secure access control functions to ensure the security and availability of data and data processing models used for model training.
[0088] Step S14: encrypt the model file corresponding to the data processing model, and deploy the encrypted model file to the target environment, so that the encrypted model file can be decrypted in the target environment, and the target data can be processed using the data processing model obtained after decryption.
[0089] It is worth mentioning that data decryption operations can be performed in both training scenarios and download scenarios. Among them, in the first specific implementation, in the training scenario, when the user needs to access encrypted data, a data access request is first initiated, and the data access request is verified to ensure that the requester has the authority to access the data. Subsequently, the copy of the encrypted data is decrypted using the corresponding key and decryption algorithm, and the encrypted copy of the data is restored to the original data. Then, a model training task is created and deployed in the preset model training container. At the same time, the network communication function of the preset model training container is turned off to further improve the security of the data.
[0090] In the second specific implementation, in the download scenario, when a user needs to access encrypted data, a data access request is first initiated, and the data access request is verified to ensure that the requester has the right to access the data. Subsequently, the encrypted data copy is decrypted using the corresponding key and decryption algorithm to restore the encrypted data copy to the original data and generate a temporary file. Finally, after the user downloads the decrypted data, the decrypted temporary file needs to be cleared and the log of the decryption and download operations needs to be recorded.
[0091] In addition, this embodiment provides a model deployment service. Therefore, users can quickly and flexibly put various models into various practical applications through the model deployment service. At the same time, the model deployment service serves as a key bridge from training to application of machine learning models, ensuring that the data processing model can play the greatest value in the actual environment and provide users with accurate and reliable prediction or decision support. Specifically, before deploying the encrypted model file to the target environment, it may also include: responding to each task deployment function instruction corresponding to the encrypted model file; the task deployment function instruction includes a model deployment task query instruction, a model deployment task addition instruction, a model deployment task deletion instruction, and a model deployment task stop instruction; wherein, responding to the model deployment task query instruction and displaying the query result of the model deployment task in a list form; the query result of the model deployment task is to display the deployment task name, deployment task description, model type, image name, model name, deployment task port, deployment task status, the creator of the deployment task, and the creation time of the deployment task in the form of a task list; responding to the model deployment task addition instruction is to receive the deployment task name, deployment task description, model loading path to the preset training container, model type, and inference container image input by the user, and save the newly added model deployment task after obtaining the deployment task save instruction; responding to the model deployment task deletion instruction is to delete the deployed model selected by the user; responding to the model deployment task stop instruction is to stop the deployment task selected by the user and being executed.
[0092] As can be seen from the above, before performing data processing based on encrypted data, the embodiment of the present application needs to generate a data encryption copy based on the original encrypted data, transfer the model training task to the preset training container, and close the network communication port of the preset training container, and then input the data copy into the preset model training framework in the preset training container, so that the preset model training framework executes the model training task to obtain the data processing model, and destroys the data copy and the preset training container; encrypt the model file corresponding to the data processing model, and deploy the encrypted model file to the target environment, so that the encrypted model file can be decrypted in the target environment, and the data processing model obtained after decryption can be used to process the target data. In this way, the security of using data for model training and using the model obtained after training to process data can be improved.
[0093] Accordingly, see Figure 3 As shown, the present application also provides a data processing method based on encrypted data, comprising:
[0094] In the external collaborative unit, the operations performed in the embodiments of the present application are user authentication operation, algorithm upload operation, business operation and result query.
[0095] The process of user authentication operation is as follows: the client sends the user authentication request to the authentication and authorization function module, so that the authentication and authorization function module can identify whether the user authentication information in the user authentication request has data access rights, and return the authority identification result to the client.
[0096] When the returned permission identification result indicates that the user has data access rights, the original data can be obtained and the algorithm upload operation needs to be performed. The process of algorithm upload operation is as follows: the user uploads the algorithm image to the algorithm rule function module so that the algorithm rule function module can perform virus detection on the algorithm image uploaded by the user, and upload the algorithm image in the training task service module under the premise that the algorithm image is safe. Subsequently, the operating environment corresponding to the preset training container is configured in the training task service module, and the selected target model training framework is integrated to be packaged into a deployable container image, and the deployable container image is used to create the image, and the image creation result is fed back to the external collaboration unit through the result feedback function module.
[0097] After the deployable container is successfully created, business operations are required. The process of business operations is as follows: first, the business operation request is sent to the business operation function module, and the target model training framework is selected from several preset model training frameworks based on the user requirements contained in the business operation request, and the target model training framework is uploaded to the model management function module for storage, so that the model management function module can upload the target model training framework to the model deployment service function module. Subsequently, the original data is uploaded to the model prediction service function module through the REST (Representational State Transfer) standard interface, so that the model prediction service function module can infer the original data, and upload the inferred data to the Docker container orchestration function module in the model deployment service function module, so that the Docker container orchestration function module can be based on the deployable container, the target model training framework and the inferred data to obtain the result of data processing.
[0098] After the Docker container orchestration function module transmits the result of data processing to the external collaboration unit, the external collaboration unit can receive the result of data processing in order to perform result query operations.
[0099] In addition, during the above steps, the auditor can monitor the status of the device and the device connection at any time. If an abnormal situation is monitored, an alarm message is generated and corresponding remedial measures are taken based on the alarm message.
[0100] It is worth mentioning that the technical architecture diagram corresponding to the above steps is as follows Figure 4 As shown:
[0101] The technical architecture includes services and applications, scheduling layer, data layer and operating environment.
[0102] Among them, data query, log module, internal API, training tasks, performance indicators, user interface, user interface and reasoning service are all services and applications in this embodiment.
[0103] The scheduling layer includes container images, which include deep learning frameworks PyTorch and TensorFlow, and are also used for code training and reasoning. When training and reasoning the code, it can manage training algorithms, track and verify information, manage private and public models, and perform service deployment operations.
[0104] The data layer includes MINIO and database, which can store the original training data and model files. It is worth mentioning that the operating environment can be CUDA (Compute Unified Device Architecture, a general parallel computing architecture and programming model), Ubuntu (a Linux distribution operating system mainly based on desktop applications) and Docker.
[0105] As can be seen from the above, before the data processing based on encrypted data is performed in the embodiment of the present application, it is necessary for the external unit to provide data for analysis and processing by the system after authentication and authorization. Subsequently, the system realizes the collection, processing, analysis and application of data through the interaction between multiple business components and subsystems. Finally, the system provides users with data security analysis support by providing prediction, alarm and other functions. In this way, the security of using data for model training and using the trained model to process data can be improved.
[0106] Accordingly, see Figure 5 As shown, the present application also provides a data processing device based on encrypted data, comprising:
[0107] A data encryption module 11 is used to encrypt the original data so as to generate a data encryption copy corresponding to the obtained original encrypted data, and to create a model training task; the model training task includes the data encryption copy and the model training algorithm;
[0108] A task transmission module 12 is used to transmit the model training task to a preset training container, close the network communication port of the preset training container, and then control the preset training container to decrypt the encrypted copy of the data to obtain a data copy;
[0109] A copy transmission module 13 is used to input the data copy into the preset model training framework in the preset training container, so as to train the preset model training framework using the model training algorithm to obtain a data processing model, and destroy the data copy and the preset training container;
[0110] The file encryption module 14 is used to encrypt the model file corresponding to the data processing model and deploy the encrypted model file to the target environment so as to decrypt the encrypted model file in the target environment and process the target data using the data processing model obtained after decryption.
[0111] As can be seen from the above, before performing data processing based on encrypted data, the embodiment of the present application needs to encrypt the original data in order to generate a data encryption copy corresponding to the obtained original encrypted data, and create a model training task; the model training task includes the data encryption copy and the model training algorithm; the model training task is transmitted to a preset training container, and the network communication port of the preset training container is closed, and then the preset training container is controlled to decrypt the data encryption copy to obtain a data copy; the data copy is input into the preset model training framework in the preset training container, so that the preset model training framework executes the model training task to obtain a data processing model, and destroys the data copy and the preset training container; the model file corresponding to the data processing model is encrypted, and the encrypted model file is deployed to the target environment, so that the encrypted model file is decrypted in the target environment, and the data processing model obtained after decryption is used to process the target data. In this way, the security of using data for model training and using the model obtained after training to process data can be improved.
[0112] In some specific implementations, the data encryption module 11 may specifically include:
[0113] An information authentication unit, used to perform permission authentication based on the user authentication information issued by the current user interface to obtain a permission authentication result, and to determine whether the user has data access rights based on the permission authentication result;
[0114] An operation return unit, configured to return to the current user interface to perform authority authentication on the user authentication information if the user does not have data access authority;
[0115] The data encryption subunit is used to encrypt the original data selected by the user using a preset encryption algorithm to obtain original encrypted data if the user has data access rights, and transmit the original encrypted data to a preset object storage system, decrypt the original encrypted data in the preset object storage system, and then generate a corresponding data encryption copy based on the obtained original data.
[0116] In some specific implementations, the data processing device based on encrypted data may further include:
[0117] An information storage unit, used to store the original encrypted data and the encrypted copy of the data in a preset object storage system, and respond to various data processing instructions; the data processing instructions include data query instructions, data addition instructions, data modification instructions and data deletion instructions;
[0118] A first data processing instruction response unit, configured to respond to the data query instruction by displaying the data query result in a list form; the data query result includes the data name, data type, data label, data storage location, data creator and data creation time;
[0119] A second data processing instruction response unit, configured to respond to the data adding instruction by receiving a data name, a data label, a data description and a model file encryption method input by a user;
[0120] A third data processing instruction response unit, configured to respond to the data modification instruction by receiving the user's modification content of the data name, data label or data description, and save the modified content after receiving the save instruction;
[0121] The fourth data processing instruction response unit is used to respond to the data deletion instruction to delete the data selected by the user.
[0122] In some specific implementations, the data processing device based on encrypted data may further include:
[0123] A first task function instruction response unit, used to respond to each task function instruction corresponding to the model training task; the task function instruction includes a model training task query instruction, a model training task addition instruction, a model training task deletion instruction, and a task history execution instruction;
[0124] The second task function instruction response unit is used to respond to the model training task query instruction to display the query results of the model training task in the form of a list; the query results of the model training task include the task name, task description, algorithm image name, data name, task status, task creator and task creation time;
[0125] A third task function instruction response unit, used to respond to the model training task new addition instruction, including receiving the task name, task description, algorithm image, training data corresponding to the new model training task, and the path for loading the training data into a preset training container;
[0126] A fourth task function instruction response unit, configured to respond to the model training task deletion instruction to delete the model training task selected by the user;
[0127] The fifth task function instruction response unit is used to respond to the task history execution instruction to view the result record of the last execution of the model training task selected by the user.
[0128] In some specific implementations, the data processing device based on encrypted data may further include:
[0129] An operating environment configuration unit, used to configure the operating environment of the preset training container, and select a target model training framework from various model training frameworks for integration based on user needs, and package the obtained integration results into a deployable container image;
[0130] A container creation unit is used to schedule the deployable container image to create the preset training container based on the task requirements corresponding to the model training tasks, and allocate corresponding computer resources to each of the model training tasks so that the preset model training framework uses the corresponding computer resources to execute the model training tasks to obtain the data processing model.
[0131] In some specific implementations, the data processing device based on encrypted data may further include:
[0132] A first algorithm function instruction response unit, used to respond to each algorithm function instruction corresponding to the model training algorithm; the algorithm function instruction includes an algorithm query instruction, an algorithm addition instruction, and an algorithm deletion instruction;
[0133] A second algorithm function instruction response unit, configured to respond to the algorithm query instruction by displaying query results based on preset algorithm rules in a list format; the query results based on preset algorithm rules include algorithm name, algorithm description, algorithm type, algorithm creator, and algorithm creation time;
[0134] A third algorithm function instruction response unit, configured to respond to the algorithm new addition instruction by receiving the algorithm image name, algorithm image description, algorithm image type and image file input by the user, and save the newly added algorithm content when obtaining the algorithm save instruction;
[0135] The fourth algorithm function instruction response unit is used to respond to the algorithm deletion instruction to delete the algorithm image file selected by the user.
[0136] In some specific implementations, the data processing device based on encrypted data may further include:
[0137] A first task deployment function instruction response unit, used to respond to each task deployment function instruction corresponding to the encrypted model file; the task deployment function instruction includes a model deployment task query instruction, a model deployment task addition instruction, a model deployment task deletion instruction, and a model deployment task stop instruction;
[0138] The second task deployment function instruction response unit is used to respond to the model deployment task query instruction and display the query result of the model deployment task in the form of a list; the query result of the model deployment task is to display the deployment task name, deployment task description, model type, image name, model name, deployment task port, deployment task status, creator of the deployment task and creation time of the deployment task in the form of a task list;
[0139] The third task deployment function instruction response unit is used to respond to the newly added instruction of the model deployment task to receive the deployment task name, deployment task description, model loading path to the preset training container, model type and inference container image input by the user, and save the newly added model deployment task after obtaining the deployment task saving instruction;
[0140] A fourth task deployment function instruction response unit, configured to respond to the model deployment task deletion instruction to delete the deployed model selected by the user;
[0141] The fifth task deployment function instruction response unit is used to respond to the model deployment task stop instruction to stop the deployment task selected by the user and being executed.
[0142] Furthermore, the present application also discloses an electronic device. Figure 6 This is a structural diagram of an electronic device 20 according to an exemplary embodiment, and the content in the figure cannot be regarded as any limitation on the scope of use of this application. The electronic device 20 may specifically include: at least one processor 21, at least one memory 22, a power supply 23, a communication interface 24, an input / output interface 25, and a communication bus 26. The memory 22 is used to store a computer program, which is loaded and executed by the processor 21 to implement the relevant steps in the data processing method based on encrypted data disclosed in any of the aforementioned embodiments. In addition, the electronic device 20 in this embodiment may specifically be an electronic computer.
[0143] In this embodiment, the power supply 23 is used to provide working voltage for each hardware device on the electronic device 20; the communication interface 24 can create a data transmission channel between the electronic device 20 and the external device, and the communication protocol it follows is any communication protocol that can be applied to the technical solution of the present application, and is not specifically limited here; the input and output interface 25 is used to obtain external input data or output data to the outside world, and its specific interface type can be selected according to specific application needs and is not specifically limited here.
[0144] In addition, the memory 22, as a carrier for storing resources, can be a read-only memory, a random access memory, a disk or an optical disk, etc. The resources stored thereon can include an operating system 221, a computer program 222, etc., and the storage method can be temporary storage or permanent storage.
[0145] The operating system 221 is used to manage and control the hardware devices and computer programs 222 on the electronic device 20, and can be Windows Server, Netware, Unix, Linux, etc. In addition to including a computer program that can be used to complete the data processing method based on encrypted data executed by the electronic device 20 disclosed in any of the aforementioned embodiments, the computer program 222 can further include a computer program that can be used to complete other specific tasks.
[0146] Furthermore, the present application also discloses a computer-readable storage medium for storing a computer program; wherein, when the computer program is executed by a processor, the aforementioned disclosed data processing method based on encrypted data is implemented. The specific steps of the method can refer to the corresponding contents disclosed in the aforementioned embodiments, and will not be repeated here.
[0147] In this specification, each embodiment is described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the embodiments can be referred to each other. For the device disclosed in the embodiment, since it corresponds to the method disclosed in the embodiment, the description is relatively simple, and the relevant parts can be referred to the method part.
[0148] Professionals may further appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of the two. In order to clearly illustrate the interchangeability of hardware and software, the composition and steps of each example have been generally described in the above description according to function. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professionals and technicians may use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.
[0149] The steps of the method or algorithm described in conjunction with the embodiments disclosed herein may be implemented directly using hardware, a software module executed by a processor, or a combination of the two. The software module may be placed in a random access memory (RAM), a memory, a read-only memory (ROM), an electrically programmable ROM, an electrically erasable programmable ROM, a register, a hard disk, a removable disk, a CD-ROM, or any other form of storage medium known in the art.
[0150] Finally, it should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the presence of other identical elements in the process, method, article or device including the elements.
[0151] The technical solution provided by the present application is introduced in detail above. Specific examples are used in this article to illustrate the principles and implementation methods of the present application. The description of the above embodiments is only used to help understand the method of the present application and its core idea. At the same time, for general technicians in this field, according to the idea of the present application, there will be changes in the specific implementation methods and application scope. In summary, the content of this specification should not be understood as a limitation on the present application.
Claims
1. A data processing method based on encrypted data, characterized in that: include: Encrypting the original data to generate an encrypted copy of the data corresponding to the obtained original encrypted data, and creating a model training task; The model training task includes the encrypted copy of the data and the model training algorithm; The model training task is transmitted to a preset training container, and a network communication port of the preset training container is closed, and then the preset training container is controlled to decrypt the encrypted copy of the data to obtain a copy of the data; Inputting the data copy into a preset model training framework in the preset training container, so as to train the preset model training framework using the model training algorithm to obtain a data processing model, and destroying the data copy and the preset training container; The model file corresponding to the data processing model is encrypted, and the encrypted model file is deployed to the target environment, so that the encrypted model file can be decrypted in the target environment, and the target data can be processed using the data processing model obtained after decryption.
2. The data processing method based on encrypted data according to claim 1, characterized in that: The encrypting of the original data to generate a data encrypted copy corresponding to the obtained original encrypted data includes: Performing permission authentication based on the user authentication information issued by the current user interface to obtain a permission authentication result, and judging whether the user has data access rights based on the permission authentication result; If the user does not have data access rights, return to the current user interface to perform permission authentication on the user authentication information; If the user has data access rights, the original data selected by the user is encrypted using a preset encryption algorithm to obtain original encrypted data, and the original encrypted data is transmitted to a preset object storage system, and the original encrypted data is decrypted in the preset object storage system, and then a corresponding data encryption copy is generated based on the obtained original data.
3. The data processing method based on encrypted data according to claim 1, characterized in that: After the original data is encrypted to generate a data encrypted copy corresponding to the obtained original encrypted data, the method further includes: The original encrypted data and the encrypted copy of the data are stored in a preset object storage system, and respond to various data processing instructions; the data processing instructions include data query instructions, data addition instructions, data modification instructions and data deletion instructions; The response to the data query instruction is to display the query results of the data in a list form; the query results of the data include the data name, data type, data label, data storage location, data creator and data creation time; In response to the data adding instruction, the data name, data label, data description and model file encryption method input by the user are received; Responding to the data modification instruction is receiving the user's modification content of the data name, data label or data description, and saving the modified content after receiving the save instruction; The data deletion instruction is responded to by deleting the data selected by the user.
4. The data processing method based on encrypted data according to claim 1, characterized in that: After creating the model training task, the following steps are also included: Respond to each task function instruction corresponding to the model training task; the task function instruction includes a model training task query instruction, a model training task addition instruction, a model training task deletion instruction, and a task history execution instruction; Wherein, responding to the model training task query instruction is to display the query results of the model training task in the form of a list; the query results of the model training task include the task name, task description, algorithm image name, data name, task status, task creator and task creation time; Responding to the new model training task instruction includes receiving a task name, task description, algorithm image, training data, and a path for loading the training data into a preset training container corresponding to the new model training task; Responding to the model training task deletion instruction is to delete the model training task selected by the user; Responding to the task history execution instruction is to view the result record of the last execution of the model training task selected by the user.
5. The data processing method based on encrypted data according to claim 1, characterized in that: Before transferring the model training task to the preset training container, the method further includes: Configure the operating environment of the preset training container, select the target model training framework from various model training frameworks for integration based on user needs, and package the obtained integration results into a deployable container image; The deployable container image is scheduled to create the preset training container based on the task requirements corresponding to the model training task, and corresponding computer resources are allocated to each model training task so that the preset model training framework uses the corresponding computer resources to execute the model training task to obtain a data processing model.
6. The data processing method based on encrypted data according to claim 5, characterized in that: Before selecting a target model training framework from each model training framework for integration based on user needs, the method further includes: Respond to each algorithm function instruction corresponding to the model training algorithm; the algorithm function instruction includes an algorithm query instruction, an algorithm addition instruction, and an algorithm deletion instruction; Wherein, the response to the algorithm query instruction is to display the query results based on the preset algorithm rules in the form of a list; the query results based on the preset algorithm rules include the algorithm name, algorithm description, algorithm type, algorithm creator and algorithm creation time; In response to the algorithm adding instruction, the algorithm image name, algorithm image description, algorithm image type and image file input by the user are received, and the newly added algorithm content is saved when the algorithm saving instruction is obtained; The algorithm deletion instruction is responded to by deleting the algorithm image file selected by the user.
7. The data processing method based on encrypted data according to any one of claims 1 to 6, characterized in that: Before deploying the encrypted model file to the target environment, the method further includes: Respond to each task deployment function instruction corresponding to the encrypted model file; the task deployment function instruction includes a model deployment task query instruction, a model deployment task addition instruction, a model deployment task deletion instruction, and a model deployment task stop instruction; Among them, respond to the model deployment task query instruction and display the query result of the model deployment task in the form of a list; the query result of the model deployment task is to display the deployment task name, deployment task description, model type, image name, model name, deployment task port, deployment task status, creator of the deployment task and creation time of the deployment task in the form of a task list; In response to the new instruction of the model deployment task, the deployment task name, deployment task description, path for loading the model into a preset training container, model type, and container image for inference are received from the user, and the newly added model deployment task is saved after obtaining the deployment task save instruction; In response to the model deployment task deletion instruction, the deployed model selected by the user is deleted; Responding to the model deployment task stop instruction is to stop the deployment task being executed that is selected by the user.
8. A data processing device based on encrypted data, characterized in that: include: A data encryption module, used to encrypt the original data so as to generate a data encrypted copy corresponding to the obtained original encrypted data and create a model training task; The model training task includes the encrypted copy of the data and the model training algorithm; A task transmission module, used to transmit the model training task to a preset training container, close the network communication port of the preset training container, and then control the preset training container to decrypt the encrypted copy of the data to obtain a data copy; A copy transmission module, used to input the data copy into the preset model training framework in the preset training container, so as to train the preset model training framework using the model training algorithm to obtain a data processing model, and destroy the data copy and the preset training container; The file encryption module is used to encrypt the model file corresponding to the data processing model and deploy the encrypted model file to the target environment so as to decrypt the encrypted model file in the target environment and process the target data using the data processing model obtained after decryption.
9. An electronic device, characterized in that: include: Memory, used to store computer programs; A processor, configured to execute the computer program to implement the steps of the data processing method based on encrypted data as described in any one of claims 1 to 7.
10. A computer-readable storage medium, characterized in that: Used to store a computer program, wherein when the computer program is executed by a processor, the steps of the data processing method based on encrypted data as described in any one of claims 1 to 7 are implemented.