Deep fake and detection model application system and method based on plug-in deployment

Abstract

The present invention particularly relates to a deep fake and detection model application system based on plug-in deployment, including an application layer: enabling users to interact with the deployment platform through the client; a Web service layer: used to encapsulate the common functional interface of the deployment platform; an application service layer: used to perform service deployment, task scheduling, GPU scheduling, result preservation, and program monitoring operations on each model reasoning service; a model interface layer: a collection of interfaces provided by all plug-in, encapsulated model reasoning services; a model deployment layer: used to deploy encapsulated model reasoning services and perform reasoning tasks; a data storage layer: used to store various temporary or permanent data. Through this solution, on the one hand, the encapsulation problem of different models using different AI frameworks is solved (encapsulated into independent services or dockers, and exposing specific interfaces); on the other hand, the problem of horizontal expansion of new models can be achieved without downtime when the deep fake model is upgraded or a new algorithm is generated.

Description

Technical Field

[0001] The present invention relates to the field of deep fake and detection technology, and in particular to a deep fake and detection model application system and method based on plug-in deployment. Background Art

[0002] Deep fake technology is a technology that forges audio and video media based on artificial intelligence, especially DeepFake-like forgery technology based on deep learning (such as face-changing videos, fake sounds, face synthesis, video generation, etc.); Deep fake detection technology is a technology that detects audio and video media on the Internet to determine whether they are DeepFake-forged audio and video media. Deep fake and detection technology are based on artificial intelligence technologies such as deep learning. At present, when various research teams conduct research on deep fake related technologies, they use different AI frameworks (tensorflow, caffe, pytorch, mxnet, etc.), different versions, and different technology stacks (C++, python, MATLAB, etc.), which makes the deep fake detection model they have studied encounter great difficulties in engineering model deduction and prediction. In addition, since deep fake and detection technology are constantly developing, the forgery and detection models will continue to provide new versions or new algorithms. How to quickly and seamlessly publish new algorithms to the algorithm engineering platform is also a key issue that needs to be solved in the deployment method.

[0003] At present, the mainstream deployment methods are the following three solutions, each of which has its own advantages and disadvantages.

[0004] 1. Python service interface: deploy the model file on the Python server and provide an http service. The backend can call the model for input and output through this service. Advantages: the working boundary between the algorithm and the backend is clear, and there is no need for much communication; data preprocessing is required before using the model, and no additional code migration is required. Disadvantages: the server needs to install the Python environment, the maintenance cost increases, there are communication problems between the interfaces of the servers, and it may take longer to obtain the model output.

[0005] 2. Load the model directly in Java: Currently, the more mature deployment solution in the industry is to use the tensorflow java package and then load the trained model file. The model file needs to be saved in pb format in advance, then add dependencies in the java environment, and finally load the model. Advantages: No additional interface is required, easy to call; no additional python environment needs to be installed. Disadvantages: The data preprocessing code needs to be migrated to java and added to the backend project code. In addition, Google does not attach much importance to this method, there is no detailed documentation, and the code is rarely updated and maintained. The model prediction speed is about the same as calling the python interface.

[0006] 3. Docker+tf-serving deployment model: This is the deployment method recommended by Google, and the deployment document is relatively detailed. It is also a widely used method. This method directly deploys the model in the docker container, and then provides two interfaces, namely Grpc interface and http interface. The former is said to have more advantages in reading batch data, such as image data; after the service is started, as long as it can be connected, any language can be called. Advantages: easy deployment, not limited by the server; multiple models can be deployed at the same time, which is convenient for model management and version control; the model reasoning speed is fast, and it has been tested to be 2 times faster than the previous two. Disadvantages: It is also necessary to write data preprocessing code, and the data input format needs to meet the requirements of the document.

[0007] Among the current common deployment solutions, first of all, all solutions need to be completed by engineers on the deployment platform for pre-processing of input data. Each new model requires the deployment platform to develop code and launch the entire program, which is not conducive to the continuous increase of models; secondly, the above deployment solutions have requirements for deep learning AI frameworks. For example, the TFserving solution is only applicable to models of the Tensorflow framework, but is helpless for Pytorch models; thirdly, deep fake and detection modes have their own unique characteristics, and there is no universal solution that can perfectly adapt to the online deployment of deep fake models; finally, some highly stable deep learning deployment architectures are relatively heavy in size, have high learning costs, and cannot give full play to the advantages of professionals. Summary of the invention

[0008] The primary purpose of the present invention is to provide a deep fake and detection model application system based on plug-in deployment, which solves the packaging problem of different AI frameworks and can realize the horizontal expansion of new models without stopping the system.

[0009] To achieve the above objectives, the technical solution adopted by the present invention is: a deep fake and detection model application system based on plug-in deployment, including an application layer: allowing users to interact with the deployment platform through the client; a Web service layer: used to encapsulate the common functional interface of the deployment platform; an application service layer: used to perform service deployment, task scheduling, GPU scheduling, result preservation and program monitoring operations on the application server; a model interface layer: an interface provided by a plug-in, encapsulated model inference service; a model deployment layer: used to deploy encapsulated model inference services and execute inference tasks; a data storage layer: used to store various temporary or permanent data.

[0010] Another object of the present invention is to provide a deep fake and detection model application method based on plug-in deployment, which solves the packaging problem of different AI frameworks and can realize the horizontal expansion of new models without stopping the system.

[0011] To achieve the above objectives, the technical solution adopted by the present invention is: a deep fake and detection model application method based on plug-in deployment, comprising the following steps: S100, the deployment platform completes the maintenance of GPU server parameters, database, and file server parameters; S200, each deep fake or detection model encapsulates the model reasoning process into an independently run model service package according to the interface standard; S300, the model service package is input into the service deployment module of the deployment platform, and according to the running video memory size requirements, CPU core number, and memory requirements of the model service package, the service deployment module selects a qualified GPU server to upload the model service package to the designated GPU server; S400, before the model service package is uploaded to the GPU server, it is necessary to configure the running port, service type, service program name, number of input parameters and type information; S500, after the model service program completes the deployment and settlement of the platform, the user selects the model service through the user portal to create a new deduction task.

[0012] Compared with the prior art, the present invention has the following technical effects: by formulating the input and output interface standards for the deep fake model deduction process, the model generator is required to complete the encapsulation of the model deduction process through its own technology stack and AI framework to form an independent deduction service program, so that each deduction service program is an independent plug-in service, exposing an interface that supports http to the outside world, and outputting a json string containing deduction result information; through this solution, on the one hand, the encapsulation problem of different models using different AI frameworks is solved (encapsulated into independent services or docker, and exposing specific interfaces); on the other hand, the problem of horizontal expansion of new models can be achieved without shutting down when the deep fake model is upgraded or a new algorithm is generated. BRIEF DESCRIPTION OF THE DRAWINGS

[0013] Figure 1It is the overall architecture diagram of the deployment platform;

[0014] Figure 2 It is the main use flow chart of the present invention. DETAILED DESCRIPTION

[0015] Combine the following Figure 1 to Figure 2 , the present invention is further described in detail.

[0016] See also Figure 1 A deep fake and detection model application system based on plug-in deployment, including an application layer: enabling users to interact with the deployment platform through the client; a Web service layer: used to encapsulate the common functional interface of the deployment platform; an application service layer: used to perform service deployment, task scheduling, GPU scheduling, result preservation, and program monitoring operations on the application server; a model interface layer: an interface provided by a plug-in, encapsulated model inference service; a model deployment layer: used to deploy encapsulated model inference services and perform inference tasks; a data storage layer: used to store various temporary or permanent data.

[0017] The deep fake and detection model has certain particularities, namely: the input and output of the model are relatively fixed. The input is generally one or more of the pictures, audio, and video files, and the output is generally a forged file or a text result, picture result, or video result of the detection. This lays the foundation for encapsulating the deduction task process for each model. The deployment scheme adopted by the present invention relies on the above-mentioned particularities. By formulating the input and output interface standards for the deep fake model deduction process, the model generator is required to complete the encapsulation of the model deduction process through its own technology stack and AI framework to form an independent deduction service program. Of course, if conditions permit, the engineering team can also be responsible for encapsulating each model deduction process. For the deployment platform, it does not care about how to encapsulate it, but only cares whether the encapsulated deduction service program is consistent with the standard input and output we set. In this way, each deduction service program is an independent plug-in service, which exposes an interface that supports http to the outside and outputs a json string containing deduction result information. In addition, in the backend, the Java general framework is used to perform service deployment, task scheduling, GPU scheduling, result preservation, program monitoring and other operations on each deduction service program, and with the help of a certain front-end UI interface, an online deduction platform based on deep fake and detection technology can be created. Through this solution, on the one hand, the packaging problem of different models using different AI frameworks is solved (encapsulated into independent services or docker, and exposing specific interfaces); on the other hand, the problem of horizontal expansion of new models can be achieved without stopping when the deep fake model is upgraded or a new algorithm is generated.

[0018] In the present invention, the application service layer realizes important logical functions and provides corresponding API interfaces for the Web service layer to call. Specifically, the application service layer includes: a service deployment module, which is used to upload the packaged model reasoning service to the GPU server for deployment and operation; a GPU scheduling module, which is used to realize the resource utilization monitoring and allocation of multiple GPU graphics cards of a GPU server, and ensure that the model service has sufficient video memory for the normal operation of the model reasoning when it is running; a task scheduling module, which is responsible for reasonably arranging the running queue of the forgery or detection task of the deployment platform, and arranging the reasonable and orderly execution of the reasoning task on multiple GPU servers under the premise of ensuring that each model service runs at most one instance at the same time; a status monitoring module, which is used to monitor the running status of each forgery or detection task, and is also responsible for monitoring whether the model reasoning service stops abnormally, and ensuring that the model reasoning service can be quickly started when it stops; a result saving module, which is used to store the execution process and result information of each forgery or detection task in a database, and provide an interface for the Web service layer to call and return the execution status of the forgery or detection task to the front-end page; a log monitoring module, which is responsible for maintaining the abnormal or running logs of each model reasoning service and the entire deployment platform, and assisting in locating and troubleshooting problems.

[0019] The model interface layer is an interface provided by the plug-in packaged model reasoning service, which is mainly composed of two interfaces: the first interface provides the function of mobilizing the model reasoning service - DoJobs, which can pass in one or more media file paths in pictures, videos or audios and the serial number of the graphics card on which the model will run according to the requirements of the reasoning model; the second interface is the task execution status interface for obtaining the model reasoning service - GetStatus, through which the execution status of the reasoning task (which step is executed, success or failure) can be obtained, and the reasoning result is returned through a json string. The fields in the JSon string are agreed upon in the encapsulation standard, such as the output video is defined as the output_video_file field, the output audio is defined as the output_audeo_file field, the output detection report is located as the output_detection_report field, etc. The result saving module of the application service layer can save the execution results of the task to the relational database according to the standard agreement, so that the front end can query the task execution status and results.

[0020] Furthermore, the model deployment layer is a cluster composed of multiple GPU servers, each of which has at least one GPU graphics card. The packaged model inference service will be deployed in these servers, and the actual inference tasks will also be executed in these servers. When the number of model inference services exceeds the capacity of the server cluster, it can be expanded horizontally, adding more GPU servers to the cluster to form a larger distributed model deployment layer.

[0021] Furthermore, the data storage layer is the medium for the deployment platform of the present invention to store various temporary or permanent data. It consists of three major parts: relational database, Redis cache, and file server. The database stores the business data of the deployment platform, the Redis cache stores temporary data generated during the service operation to improve the interface performance of the system, and the file server is used to store multimedia files and report file data input and output by running model reasoning tasks.

[0022] Further, the Web service layer includes a login module for user login and permission setting, a forged task module for creating a new forged task, a detection task module for creating a new detection task, a forged report module for viewing the forged result, and a detection report module for viewing the detection result. The Web service encapsulates some common functional interfaces of the deployment platform. The specific functions need to be implemented by the Web server layer and the application service layer through the API interface interaction. In addition to the functions mentioned above, some other functions can be encapsulated as needed. The application layer allows users to interact with the deployment platform through the client. By calling the relevant API interface provided by the application service layer, user login and creation, execution of forged detection tasks and viewing execution results can be achieved. Since the overall deployment scheme adopts the technology of front-end and back-end separation, the main calculations are on the back-end and GPU server, and there is no rigid requirement for the front-end entrance. Therefore, the application layer entrance of the present invention can be one or more of PC web pages, WeChat applets, APPs or windows desktop applications.

[0023] See also Figure 2The present invention also discloses a deep fake and detection model application method based on plug-in deployment, comprising the following steps: S100, the deployment platform completes the maintenance of GPU server parameters, database, and file server parameters; S200, each deep fake or detection model encapsulates the model reasoning process into an independently run model service package according to the interface standard; S300, the model service package is input into the service deployment module of the deployment platform, and according to the running video memory size requirements, CPU core number, and memory requirements of the model service package, the service deployment module selects a qualified GPU server to upload the model service package to the specified GPU server; S400, before the model service package is uploaded to the GPU server, it is necessary to configure the running port, service type, service program name, number of input parameters, and type information; S500, after the model service program completes the deployment and settlement of the platform, the user selects the model service through the user portal to create a new deduction task.

[0024] Furthermore, the application service layer will execute the task request through the task scheduling module and the GPU scheduling module to call the Dojobs interface to perform model deduction when the GPU resources of the GPU service where the specified model service is located are idle. At the same time, the status monitoring module will periodically call the GetStatus interface of the model to obtain the execution status and results of the task; the task scheduling module ensures that at any time, each service model has at most one deduced instance running; the GPU scheduling module ensures that the GPU resources consumed by the service modules running on the same GPU server will not exceed the resource consumption of the normal operation of the model; when the status monitoring module obtains the end of the model service deduction or the deduction fails, it will save the deduction results to the database and return the deduction results to the application layer.

[0025] In the implementation process of the present invention, the key to achieving plug-in operation is to encapsulate the model deduction process into an independently run http service that provides a specified functional interface. On the one hand, it is necessary to provide a practical standard interface and output field standard suitable for deep fake and detection model reasoning; on the other hand, it is necessary to have a technical means to package and encapsulate both the reasoning framework and the operating environment.

[0026] In terms of standard interfaces, the present invention has been verified through implementation and only needs to provide two interfaces: execution reasoning and acquisition status. The execution reasoning interface needs to specify the input parameters required for reasoning the model and the GPU graphics card serial number information, and standardize the output result fields. This standard will be continuously updated and supplemented as the deployment platform connects to more and more deep fake related models, especially different models often have some additional fields in the inferred results. The present invention currently integrates 20 algorithm models, and has basically completed the standardization of interface standards and output fields.

[0027] In terms of packaging technology, a simple method is to use the Anconda virtual environment with the python+wsgi web framework (flask or django)+pyinstaller packaging solution to encapsulate the reasoning process of the deep learning model into an http service that conforms to the standard interface. This method is simple and easy to use, but the general deployment package is relatively large and the running speed is slow, and the stability is slightly worse, but it is still more applicable as a deep fake detection scenario that does not require high real-time performance. Another method is to encapsulate the reasoning process of the model through c++ or java language. For example, the libtorch library that can be used in c++ can be used to encapsulate the reasoning process of the torch framework model and encapsulate it into an http service; the JAVA language can use the Deeplearing4j library to implement the deduction process call and encapsulation of the deep learning model. In short, in terms of packaging technology, the current mainstream technology stacks support the call of deep learning models more maturely, which is also the premise that the present invention can deploy deep fake models implemented by different AI frameworks of different technology stacks in a plug-in manner.

[0028] The present invention also discloses a computer-readable storage medium on which a computer program is stored. When the computer program is executed by a processor, the aforementioned deep fake and detection model application method based on plug-in deployment is implemented. An electronic device is also disclosed, including a memory, a processor, and a computer program stored on the memory. When the processor executes the computer program, the aforementioned deep fake and detection model application method based on plug-in deployment is implemented.

Claims

1. A deep fake and detection model application system based on plug-in deployment, characterized by: Includes the application layer: enables users to interact with the deployment platform through the client; Web service layer: used to encapsulate the common functional interfaces of the deployment platform; Application service layer: used for service deployment, task scheduling, GPU scheduling, result storage, and program monitoring operations on application servers; Model interface layer: It has two interfaces provided by the plug-in and encapsulated model reasoning service. The first interface provides the function of calling the model reasoning service, and the second interface obtains the execution status interface of the reasoning task of the model reasoning service. Model deployment layer: used to deploy packaged model inference services and perform inference tasks; Data storage layer: used to store various temporary or permanent data.

2. The deep fake and detection model application system based on plug-in deployment as claimed in claim 1, characterized in that: The application service layer includes The service deployment module is used to upload the packaged model inference service to the GPU server for deployment and operation; GPU scheduling module, used to monitor and allocate resource utilization of multiple GPU graphics cards on a GPU server; The task scheduling module is responsible for arranging the running queue of the forgery or detection tasks of the deployment platform; The result saving module is used to store the execution process and result information of each forgery or detection task in the data storage layer, and provide an interface for the Web service layer to call and return the execution status of the forgery or detection task to the front-end page; The status monitoring module is used to monitor the running status of each forgery or detection task and is responsible for monitoring whether the model inference service stops abnormally; The log monitoring module is responsible for maintaining the exception or operation logs of each model inference service and the entire deployment platform.

3. The deep fake and detection model application system based on plug-in deployment as claimed in claim 1, characterized in that: The model deployment layer is a cluster composed of multiple GPU servers, each of which has at least one GPU graphics card.

4. The deep fake and detection model application system based on plug-in deployment as claimed in claim 1, characterized in that: The data storage layer includes a database, a Redis cache and a file server. The database stores the business data of the deployment platform. The Redis cache stores temporary data generated during service operation to improve the interface performance of the system. The file server is used to store multimedia files and report file data input and output of running model reasoning tasks.

5. The deep fake and detection model application system based on plug-in deployment as claimed in claim 1, characterized in that: The Web service layer includes a login module for users to log in and set permissions, a forgery task module for creating new forgery tasks, a detection task module for creating new detection tasks, a forgery report module for viewing forgery results, and a detection report module for viewing detection results; the application layer is one or more of a PC web page, WeChat applet, APP or Windows desktop application.

6. A deep fake and detection model application method based on plug-in deployment, characterized in that: The steps include: S100, the deployment platform completes the maintenance of GPU server parameters, database, and file server parameters; S200, each deep fake or detection model encapsulates the model reasoning process into an independently run model service package according to the interface standard; S300, input the model service program package into the service deployment module of the deployment platform, and according to the running video memory size requirement, number of CPU cores, and memory requirement of the model service program package, the service deployment module selects a qualified GPU server to upload the model service program package to the designated GPU server; S400, before uploading the model service package to the GPU server, it is necessary to configure the running port, service type, service program name, number and type information of input parameters; S500: After the model service program completes the deployment and entry into the platform, the user selects the model service through the user portal to create a new deduction task.

7. The deep fake and detection model application method based on plug-in deployment as claimed in claim 6, characterized in that: The application service layer will execute the task request through the task scheduling module and the GPU scheduling module to call the Dojobs interface to perform model deduction when the GPU resources of the GPU service where the specified model service is located are idle. At the same time, the status monitoring module will periodically call the GetStatus interface of the model to obtain the execution status and results of the task; The task scheduling module ensures that at any time, at most one deduction instance of each service model is running; the GPU scheduling module ensures that the GPU resources consumed by the service modules running on the same GPU server do not exceed the resource consumption of the normal operation of the model; When the status monitoring module obtains the model service deduction completion or deduction failure, it saves the deduction result to the database and returns the deduction result to the application layer.

8. A computer-readable storage medium, characterized in that: A computer program is stored thereon, and when the computer program is executed by a processor, the deep fake and detection model application method based on plug-in deployment as described in claim 6 or 7 is implemented.

9. An electronic device, characterized in that: It includes a memory, a processor and a computer program stored in the memory. When the processor executes the computer program, it implements the deep fake and detection model application method based on plug-in deployment as described in claim 6 or 7.

Images