Model deployment method and device, information processing method and device, equipment, storage medium and product

By splicing the feature extraction model and the fusion model to be deployed into a media information processing model and deploying it on the target server, the problem of inefficient data processing caused by unreasonable model deployment in the existing technology is solved, and the East Data and West Calculation Project is realized efficient data processing.

CN118890422APending Publication Date: 2024-11-01CHINA MOBILE GROUP DESIGN INST +1
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202410918039.5
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-07-10
Publication Date
2024-11-01

AI Technical Summary

Technical Problem

In the prior art, unreasonable model deployment leads to inefficient data processing, especially in video services, where clients and servers need multiple data interactions, resulting in high requirements for communication networks and difficult to apply to the East Data and West Computing Project.

Method used

By obtaining the feature extraction model to be deployed and the fusion model to be deployed, splicing it into a media information processing model, and deploying it on the target server, reducing the number of data interactions between the client and the server, using the GPU server to perform all model-related calculations, reducing costs and capabilities Consumption.

Benefits of technology

It improves data processing efficiency, reduces the delay caused by multiple communications, optimizes the computing power utilization rate, improves the overall inference speed and efficiency, and is suitable for Eastern Data and Western computing projects.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118890422A_ABST
    Figure CN118890422A_ABST
Patent Text Reader

Abstract

The invention discloses a model deployment method and device, an information processing method and device, equipment, a storage medium and a product, and the method comprises the steps: splicing a to-be-deployed feature extraction model and a to-be-deployed fusion model which are suitable for an actual operation environment, so as to obtain a media information processing model, the to-be-deployed feature extraction model is used for extracting the features of the received original media information, and the to-be-deployed fusion model is used for generating target media information according to the features of the original media information and deploying the media information processing model on a target server, so that reasonable deployment of the model is realized, and the user experience is improved. The data interaction frequency of the client and the server is reduced, the time delay caused by multiple times of communication is further reduced, the data processing efficiency is improved, and the method is suitable for east-west calculation engineering.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the field of media data processing, and in particular to a model deployment method, information processing method, device, equipment, storage medium and product. Background Art

[0002] Video service is one of the most active and promising services. The economically developed eastern regions generate tens of millions of videos every day. Each short video needs to be tested by a large number of video algorithms before reaching users. It is necessary to use feature extraction models (such as text models, voice models, and video models) to extract features, and input the features into the fusion model to obtain the final inference results. The existing technology is mainly a single model deployment solution, that is, in multimodality, the feature extraction model is deployed on a server based on a graphics processing unit (GPU), the client's central processing unit (CPU) is responsible for audio and video decoding frame extraction and preprocessing, the GPU server uses the feature extraction model to extract multiple features, and then the client CPU performs feature post-processing and fusion, and finally inputs it into the fusion model to obtain the final inference results. In this process, the client and the server need to interact with a large amount of data multiple times, which has high requirements for the communication network, easily leads to low data processing efficiency, and is difficult to apply in the East-West computing project. Summary of the invention

[0003] Based on this, the present invention provides a model deployment method, information processing method, device, equipment, storage medium and product to solve the defect of low data processing efficiency caused by unreasonable model deployment in the prior art.

[0004] To achieve the above object, an embodiment of the present invention provides a model deployment method, including:

[0005] Acquire a feature extraction model to be deployed and a fusion model to be deployed, wherein the feature extraction model to be deployed is used to extract features of the received original media information, and the fusion model to be deployed is used to generate target media information according to the features of the original media information;

[0006] The feature extraction model to be deployed and the fusion model to be deployed are spliced ​​to obtain a media information processing model to be deployed on a target server.

[0007] As an improvement of the above solution, the feature extraction model to be deployed and the fusion model to be deployed are spliced ​​to obtain a media information processing model to be deployed on a target server, including:

[0008] Constructing a pre-processing model of the feature extraction model to be deployed according to the pre-processing logic and input requirements of the feature extraction model to be deployed;

[0009] According to the output requirements of the business and the post-processing logic of the feature extraction model to be deployed, determine the output node of the feature extraction model to be deployed, and build a corresponding post-processing model for the output node;

[0010] Sequentially splicing the pre-processing model of the feature extraction model to be deployed, the feature extraction model to be deployed, and the post-processing model of the feature extraction model to be deployed to obtain a target information feature extraction model;

[0011] Constructing a pre-processing model of the fusion model to be deployed according to the dimension information of the output node of the feature extraction model to be deployed and the pre-processing logic of the fusion model to be deployed, and splicing the pre-processing model of the fusion model to be deployed and the fusion model to be deployed to obtain a target fusion model;

[0012] A media information processing model is obtained according to the target information feature extraction model and the target fusion model so as to be deployed on a target server.

[0013] As an improvement of the above solution, the method further includes:

[0014] According to the post-processing logic of the fusion model to be deployed, a post-processing model of the fusion model to be deployed is constructed; wherein the post-processing model of the fusion model to be deployed is used to process the output information to be used of the fusion module to be used; the target fusion model is obtained by sequentially splicing the pre-processing model of the fusion model to be deployed, the fusion model to be deployed and the post-processing model of the fusion model to be deployed.

[0015] As an improvement of the above-mentioned scheme, the pre-processing model of the feature extraction model to be deployed is used to process the input information of the feature extraction model to be deployed so as to be suitable for the feature extraction model to be deployed and the target server; the pre-processing model of the fusion model to be deployed is used to process the input information to be input into the fusion model to be deployed so as to be suitable for the fusion model to be deployed.

[0016] As an improvement of the above solution, the step of determining the output node of the feature extraction model to be deployed according to the output requirements of the service and the post-processing logic of the feature extraction model to be deployed includes:

[0017] According to the output requirements of the preset business, the input requirements of the fusion model to be deployed and the post-processing logic of the feature extraction model to be deployed, the output node of the feature extraction model to be deployed is determined; wherein the post-processing model of the feature extraction model to be deployed is used to process the output information of the feature extraction model to be deployed so as to be suitable for the fusion model to be deployed and meet the business requirements.

[0018] As an improvement of the above scheme, it also includes:

[0019] When the feature extraction model to be deployed includes a speech feature extraction model to be deployed, constructing an audio frame extraction model according to input node parameter information of the speech feature extraction model to be deployed, so as to be spliced ​​with the target information feature extraction model;

[0020] When the feature extraction model to be deployed includes a video feature extraction model to be deployed, a video frame extraction model is constructed according to input node parameter information of the video feature extraction model to be deployed to be spliced ​​with the target information feature extraction model.

[0021] As an improvement of the above-mentioned scheme, the feature extraction model to be deployed includes at least one of a text feature extraction model to be deployed, a speech feature extraction model to be deployed, and a video feature extraction model to be deployed, wherein the text feature extraction model to be deployed is used to extract text features, the speech feature extraction model to be deployed is used to extract speech features, and the video feature extraction model to be deployed is used to extract video features.

[0022] As an improvement of the above solution, the feature extraction model to be deployed and the fusion model to be deployed are obtained by the following method:

[0023] Obtaining a trained original information feature extraction model, a trained original fusion model, and actual operating environment information;

[0024] The original information feature extraction model and the original fusion model are respectively converted according to the actual operating environment information to obtain a feature extraction model to be deployed and a fusion model to be deployed suitable for the actual operating environment.

[0025] As an improvement of the above solution, all the pre-processing models and all the post-processing models are of the type of computational graphs.

[0026] To achieve the above object, an embodiment of the present invention provides an information processing method, including:

[0027] The original media information is input into a media information processing model deployed on a target server to obtain target media information; wherein the media information processing model includes a spliced ​​feature extraction model to be deployed and a fusion model to be deployed, the feature extraction model to be deployed is used to extract features of the received original media information, and the fusion model to be deployed is used to generate target media information based on the features of the original media information.

[0028] As an improvement of the above solution, the media information processing model is obtained through the model deployment method described in any of the above embodiments.

[0029] To achieve the above purpose, an embodiment of the present invention further provides a model deployment device, including:

[0030] A model acquisition module, used to acquire a feature extraction model to be deployed and a fusion model to be deployed, wherein the feature extraction model to be deployed is used to extract features of the received original media information, and the fusion model to be deployed is used to generate target media information according to the features of the original media information;

[0031] The model deployment module is used to splice the feature extraction model to be deployed and the fusion model to be deployed to obtain a media information processing model for deployment on a target server.

[0032] To achieve the above object, an embodiment of the present invention further provides an information processing device, including:

[0033] An information processing module is used to input the original media information into a media information processing model deployed on a target server to obtain target media information; wherein the media information processing model includes a spliced ​​feature extraction model to be deployed and a fusion model to be deployed, the feature extraction model to be deployed is used to extract the features of the received original media information, and the fusion model to be deployed is used to generate target media information according to the features of the media information, and the feature extraction model to be deployed and the fusion model to be deployed are suitable for an actual operating environment.

[0034] To achieve the above objectives, an embodiment of the present invention further provides an electronic device, comprising a processor, a memory, and a computer program stored in the memory and configured to be executed by the processor, wherein when the processor executes the computer program, the model deployment method or information processing method as described in any of the above embodiments is implemented.

[0035] To achieve the above objectives, an embodiment of the present invention further provides a computer-readable storage medium, wherein the computer-readable storage medium includes a stored computer program, wherein when the computer program is running, the device where the computer-readable storage medium is located is controlled to execute the model deployment method or information processing method as described in any of the above embodiments.

[0036] To achieve the above objectives, an embodiment of the present invention further provides a computer program product, including a computer program / instructions, which, when executed by a processor, implements the model deployment method or information processing method as described in any of the above embodiments.

[0037] Compared with the prior art, the model deployment method, information processing method, device, equipment, storage medium and product disclosed in the embodiments of the present invention obtain a media information processing model by splicing a feature extraction model to be deployed and a fusion model to be deployed that are suitable for the actual operating environment, wherein the feature extraction model to be deployed is used to extract the features of the received original media information, and the fusion model to be deployed is used to generate target media information according to the features of the original media information, and the media information processing model is deployed on the target server, thereby realizing the reasonable deployment of the model, reducing the number of data interactions between the client and the server, thereby reducing the delay caused by multiple communications, and improving data processing efficiency, and is suitable for East-West Computing Project. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] In order to more clearly illustrate the technical solution of the present invention, the drawings required for use in the implementation mode will be briefly introduced below. Obviously, the drawings described below are only some implementation modes of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0039] Figure 1 It is a flowchart of a model deployment method provided by an embodiment of the present invention;

[0040] Figure 2 It is a schematic diagram of a model building process provided by an embodiment of the present invention;

[0041] Figure 3 is a schematic diagram of the architecture of a media information processing model provided by an embodiment of the present invention;

[0042] Figure 4 It is a structural schematic diagram of a model deployment device provided by an embodiment of the present invention;

[0043] Figure 5 It is a structural schematic diagram of an electronic device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0044] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0045] Video business is one of the most active and promising businesses at present. The economically developed eastern regions generate tens of millions of videos every day. Each short video needs to go through a large number of video algorithm tests before reaching users. It is necessary to use text models, voice models and video models to extract multiple features, and integrate these features into the fusion model to obtain the final inference results, which requires huge computing power. With the implementation of the East-West Computing Project, the eastern developed regions will deploy video algorithm models on the western GPU computing servers, thereby giving full play to the advantages of abundant green power resources and low electricity prices in the western region, which can reduce the cost of video business and gain a larger market.

[0046] The existing technology is mainly a single model deployment scheme, that is, in multimodality, multiple models are deployed on a server based on a graphics processing unit (GPU). Then, the central processing unit (CPU) of the client is responsible for audio and video decoding frame extraction and preprocessing, and uses the above model to extract multiple features, and then the client CPU performs feature post-processing and fusion, and finally inputs it into the fusion model to obtain the final reasoning result. Since a large amount of floating-point data will be generated in the model output stage, in the prior art, the transmission of these floating-point data will not only occupy a large amount of network bandwidth, resulting in a significant reduction in transmission speed, but also make the CPU and GPU in a waiting state for a long time, thereby causing low computing power utilization, slowing down the reasoning speed, and finally affecting the processing volume per second (TPS) of the entire reasoning service. In addition, when fusing multiple features, due to the inconsistent calculation speeds of each model, there will be a situation where the models wait for each other, which not only leads to a waste of computing power resources, but also affects the overall computing efficiency.

[0047] Based on this, an embodiment of the present invention provides a model deployment method, see Figure 1 , Figure 1 : is a flow chart of a model deployment method provided by an embodiment of the present invention, the model deployment method comprises steps S11 to S12:

[0048] S11, obtaining a feature extraction model to be deployed and a fusion model to be deployed, wherein the feature extraction model to be deployed is used to extract features of received original media information, and the fusion model to be deployed is used to generate target media information according to the features of the original media information;

[0049] S12: splicing the feature extraction model to be deployed and the fusion model to be deployed to obtain a media information processing model to be deployed on a target server.

[0050] Specifically, the target server is a GPU server, and the feature extraction model to be deployed for extracting media information features and the fusion model to be deployed for fusing multiple media information features are spliced ​​to generate a media information processing model, which is deployed on the GPU server in the western region. The client CPU server in the eastern region only needs to send the original media information (such as the original video file) to the western GPU server in a binary stream, and the western GPU server completes all model-related calculations. The eastern CPU is responsible for business logic processing with a small amount of calculation, thereby reducing costs and energy consumption. It can be seen that the embodiment of the present invention splices multiple models into a new model and runs the entire multimodal model on a single GPU. The GPU server does not need to interact with the CPU server for multiple times with a large amount of data, and also eliminates the situation in the prior art where the models wait for each other due to the inconsistent calculation speeds of the models, and also reduces the number of GPU cards required, thereby effectively avoiding the situation in the prior art where the video memory and utilization of each GPU may have a large amount of surplus due to the deployment of multiple models on multiple GPUs, reducing computing power fragmentation, optimizing computing power utilization, and improving the overall reasoning speed and efficiency.

[0051] In one embodiment, the step S12 of combining the feature extraction model to be deployed and the fusion model to be deployed to obtain a media information processing model for deployment on a target server includes:

[0052] Constructing a pre-processing model of the feature extraction model to be deployed according to the pre-processing logic and input requirements of the feature extraction model to be deployed;

[0053] According to the output requirements of the business and the post-processing logic of the feature extraction model to be deployed, determine the output node of the feature extraction model to be deployed, and build a corresponding post-processing model for the output node;

[0054] Sequentially splicing the pre-processing model of the feature extraction model to be deployed, the feature extraction model to be deployed, and the post-processing model of the feature extraction model to be deployed to obtain a target information feature extraction model;

[0055] Constructing a pre-processing model of the fusion model to be deployed according to the dimension information of the output node of the feature extraction model to be deployed and the pre-processing logic of the fusion model to be deployed, and splicing the pre-processing model of the fusion model to be deployed and the fusion model to be deployed to obtain a target fusion model;

[0056] A media information processing model is obtained according to the target information feature extraction model and the target fusion model so as to be deployed on a target server.

[0057] Exemplarily, assuming that the feature extraction models to be deployed include a text feature extraction model B1 to be deployed, a speech feature extraction model B2 to be deployed, and a video feature extraction model B3 to be deployed, and the fusion model to be deployed is a fusion model B4 to be deployed suitable for actual operation wake-up, each feature extraction model to be deployed will be deployed on the target server.

[0058] 1. Pre-processing model of the feature extraction model to be deployed. The type of the target server is a GPU server. When the original media information is sent from the CPU of the client to the GPU server for processing, the conversion of the server type needs to be considered, and the original media information is converted from a format suitable for CPU calculation to a format suitable for GPU calculation. Therefore, the pre-processing logic may include converting the original media information into a format suitable for the target server type. In addition, each feature extraction model to be deployed also has certain requirements for the format of the information input to the model, such as data size, dimension or data type. Therefore, based on the above reasons, it is necessary to set a pre-processing model for the feature extraction model to be deployed according to the pre-processing logic and input requirements of the feature extraction model 1 to be deployed. For example, assuming that the type of the pre-processing model is a calculation graph, it is necessary to develop a pre-processing calculation graph Bpre1 based on GPU operators for the text feature extraction model B1 to be deployed according to the pre-processing logic and input requirements of the text feature extraction model B1 to be deployed. Similarly, a pre-processing calculation graph Bpre2 based on GPU operators and a pre-processing calculation graph Bpre3 based on GPU operators are set for the speech feature extraction model B2 to be deployed and the video feature extraction model B3 to be deployed.

[0059] It is worth noting that the pre-processing logic and input requirements of the feature extraction model to be deployed are not limited to uploading specific settings and can be set according to actual conditions.

[0060] 2. About the post-processing model of the feature extraction model to be deployed. The post-processing logic is set according to the actual situation, and is used to process the output data of the feature extraction model to be deployed before output. The information features required for the output requirements of different services are different. It is necessary to consider the output requirements of the business to process the output information of the extraction model to be deployed to extract the features required by the business. Therefore, the output requirements of the entire business and each modal model (each feature extraction model to be deployed) are combined to determine the positions and names of multiple output nodes of each modality, where different output nodes have different output features, data dimensions, etc.), and build corresponding post-processing models for the output nodes. For example, a post-processing calculation graph Bpost1 of the text feature extraction model B1 to be deployed based on the GPU operator is constructed, a post-processing calculation graph Bpost2 of the speech feature extraction model B2 to be deployed based on the GPU operator is constructed, and a post-processing calculation graph Bpost3 of the video feature extraction model B3 to be deployed based on the GPU operator is constructed.

[0061] 3. Pre-processing model of the fusion model to be deployed. According to the dimensional information of the output nodes of Bpost1, Bpost2 and Bpost3, combined with the pre-processing logic (such as the input requirements of the model) and post-processing logic of the fusion model to be deployed, the pre-processing calculation graph Bpre4 and post-processing calculation graph Bpost4 of the fusion model to be deployed based on GPU operators are constructed. It can be understood that the types of each pre-processing model and post-processing model are not limited to the calculation graph, and can be set according to the actual situation.

[0062] In this implementation, pre-processing, modeling, and post-processing are all executed on the GPU, making full use of the GPU parallel processing capabilities to accelerate the calculation process. There is no need to execute on the client's CPU, avoiding high client CPU load and thus avoiding affecting the upper-level business logic of the entire reasoning service. In addition, the large amount of floating-point data generated at each stage does not need to be transmitted between the client CPU and the GPU server, and does not need to occupy a large amount of network bandwidth. The CPU and GPU do not need to be in a waiting state for a long time, thereby improving the reasoning speed and the processing capacity per second (TPS) of the entire reasoning service.

[0063] In one embodiment, the method further comprises:

[0064] According to the post-processing logic of the fusion model to be deployed, a post-processing model of the fusion model to be deployed is constructed; wherein the post-processing model of the fusion model to be deployed is used to process the output information to be used of the fusion module to be used; the target fusion model is obtained by sequentially splicing the pre-processing model of the fusion model to be deployed, the fusion model to be deployed and the post-processing model of the fusion model to be deployed.

[0065] Exemplarily, the output information of the fusion model to be deployed can be post-processed according to actual conditions to make it meet the set output requirements. Therefore, by constructing a post-processing calculation graph Bpost4 of the model to be fused based on GPU operators according to the set post-processing logic of the fusion model to be deployed.

[0066] In one embodiment, the pre-processing model of the feature extraction model to be deployed is used to process the input information of the feature extraction model to be deployed so as to make it suitable for the feature extraction model to be deployed and the target server; the pre-processing model of the fusion model to be deployed is used to process the input information to be input into the fusion model to be deployed so as to make it suitable for the fusion model to be deployed.

[0067] In one implementation, determining the output node of the feature extraction model to be deployed according to the output requirements of the service and the post-processing logic of the feature extraction model to be deployed includes:

[0068] According to the output requirements of the preset business, the input requirements of the fusion model to be deployed and the post-processing logic of the feature extraction model to be deployed, the output node of the feature extraction model to be deployed is determined; wherein the post-processing model of the feature extraction model to be deployed is used to process the output information of the feature extraction model to be deployed so as to be suitable for the fusion model to be deployed and meet the business requirements.

[0069] Specifically, the feature extraction model to be deployed has multiple output nodes. Different input nodes correspond to different data formats such as data dimensions. The data corresponding to different output nodes do not necessarily meet the input requirements of the fusion model to be deployed. Therefore, it is necessary to determine the output nodes that meet the input requirements of the fusion model to be deployed.

[0070] In one embodiment, it further includes:

[0071] When the feature extraction model to be deployed includes a speech feature extraction model to be deployed, constructing an audio frame extraction model according to input node parameter information of the speech feature extraction model to be deployed, so as to be spliced ​​with the target information feature extraction model;

[0072] When the feature extraction model to be deployed includes a video feature extraction model to be deployed, a video frame extraction model is constructed according to input node parameter information of the video feature extraction model to be deployed to be spliced ​​with the target information feature extraction model.

[0073] Specifically, the audio and video preprocessing module includes a video frame extraction module and an audio frame extraction module. According to the input node parameter information of the speech feature extraction model B2 to be deployed, an audio frame extraction calculation graph Apre1 based on the GPU operator is developed. Similarly, a video frame extraction calculation graph Apre2 is developed according to the video model B3. The audio frame extraction calculation graph Apre1 receives the original audio file as binary input, and after processing, it is input into the pre-processing calculation graph Bpre1 of the speech feature model B2 to be deployed for further processing. The video frame extraction calculation graph Apre2 receives the original video file as binary input; wherein, a calculation graph is a graphical model used to describe mathematical calculations, and the specific types of the video frame extraction model and the audio frame extraction model in this embodiment are not limited to calculation graphs.

[0074] In this embodiment, the preprocessing of voice and video and subsequent pre-processing, the model itself, and post-processing are concentrated in the target server GPU, which significantly reduces the amount of intermediate data transmitted between networks, improves the inference speed, and optimizes the overall performance.

[0075] In one embodiment, the feature extraction model to be deployed includes at least one of a text feature extraction model to be deployed, a speech feature extraction model to be deployed, and a video feature extraction model to be deployed, wherein the text feature extraction model to be deployed is used to extract text features, the speech feature extraction model to be deployed is used to extract speech features, and the video feature extraction model to be deployed is used to extract video features.

[0076] In one implementation, the feature extraction model to be deployed and the fusion model to be deployed are obtained in the following manner:

[0077] Obtaining a trained original information feature extraction model, a trained original fusion model, and actual operating environment information;

[0078] The original information feature extraction model and the original fusion model are respectively converted according to the actual operating environment information to obtain a feature extraction model to be deployed and a fusion model to be deployed suitable for the actual operating environment.

[0079] Specifically, the original text feature extraction model M1, the original speech feature extraction model M2, the original video feature extraction model M3 and the original fusion model M4 are trained respectively using the training samples. The specific training method can be found in the prior art and will not be described here. The trained model is converted into a deployment model suitable for the actual operating environment, such as converting the original text feature extraction model M1 into the text feature extraction model B1 to be deployed, the original speech feature extraction model M2 into the speech feature extraction model B2 to be deployed, the original video feature extraction model M3 into the video feature extraction model B3 to be deployed, and the original fusion model M4 into the fusion model B4 to be deployed.

[0080] See also Figure 2 The model building process shown and Figure 3 The model architecture diagram shown in the figure, in order to make the specific construction process of the media information processing model clearer, its construction process is introduced again as follows: convert each modal training model (original information feature extraction model) and the original fusion model into a deployment model (feature extraction model to be deployed, fusion model to be deployed), develop audio preprocessing and video preprocessing calculation graphs based on GPU operators, develop pre-processing calculation graphs for each modal model based on GPU operators, develop post-processing calculation graphs for each modal model based on GPU operators, develop pre-processing and post-processing calculation graphs for fusion models based on GPU operators, and according to the preset model reasoning processes, fuse the pre-processing, model and post-processing of each model into a new model (such as target text feature extraction model C1, target speech feature extraction model C2, target video feature extraction model C3, target fusion model C4), and according to the preset business reasoning process, fuse the above new models into a single large model (media information processing model) as the final deployment model D, and then deploy the final deployment model D on the GPU server in the computing network. In actual applications, the client sends a video binary stream to the final deployment model D, which processes the data according to the model reasoning process and business reasoning process, and returns the reasoning result to the client.

[0081] Compared with the prior art, the model deployment method disclosed in the embodiment of the present invention obtains a media information processing model by splicing a feature extraction model to be deployed and a fusion model to be deployed that are suitable for the actual operating environment, wherein the feature extraction model to be deployed is used to extract the features of the received original media information, and the fusion model to be deployed is used to generate target media information according to the features of the original media information, and the media information processing model is deployed on the target server, thereby realizing the reasonable deployment of the model, reducing the number of data interactions between the client and the server, thereby reducing the time delay caused by multiple communications, and improving data processing efficiency, and is suitable for East-West Computing projects.

[0082] The embodiment of the present invention further provides an information processing method, including:

[0083] The original media information is input into a media information processing model deployed on a target server to obtain target media information; wherein the media information processing model includes a spliced ​​feature extraction model to be deployed and a fusion model to be deployed, the feature extraction model to be deployed is used to extract features of the received original media information, and the fusion model to be deployed is used to generate target media information based on the features of the original media information.

[0084] Furthermore, the media information processing model is obtained through the model deployment method described in any of the above embodiments.

[0085] It is worth noting that the working principle of the information processing method provided in the above embodiments can be found in the content of the model deployment method provided in any of the above embodiments, which will not be elaborated here.

[0086] Compared with the prior art, the information processing method disclosed in the embodiment of the present invention obtains target media information by inputting original media information into a media information processing model deployed on a target server; wherein the media information processing model includes a spliced ​​feature extraction model to be deployed and a fusion model to be deployed, the feature extraction model to be deployed is used to extract the features of the received original media information, and the fusion model to be deployed is used to generate target media information according to the features of the original media information. It can be seen that this implementation utilizes the media information processing model spliced ​​from multiple models deployed on the target server to perform information processing, thereby reducing the number of data interactions between the client and the server, thereby reducing the delay caused by multiple communications, improving data processing efficiency, and is suitable for East-West Computing projects.

[0087] See also Figure 4 , an embodiment of the present invention further provides a model deployment device, comprising:

[0088] A model acquisition module 21, used to acquire a feature extraction model to be deployed and a fusion model to be deployed, wherein the feature extraction model to be deployed is used to extract features of the received original media information, and the fusion model to be deployed is used to generate target media information according to the features of the original media information;

[0089] The model deployment module 22 is used to splice the feature extraction model to be deployed and the fusion model to be deployed to obtain a media information processing model for deployment on a target server.

[0090] In one implementation, the model deployment module 22 is specifically configured to:

[0091] The step of splicing the feature extraction model to be deployed and the fusion model to be deployed to obtain a media information processing model for deployment on a target server includes:

[0092] Constructing a pre-processing model of the feature extraction model to be deployed according to the pre-processing logic and input requirements of the feature extraction model to be deployed;

[0093] According to the output requirements of the business and the post-processing logic of the feature extraction model to be deployed, determine the output node of the feature extraction model to be deployed, and build a corresponding post-processing model for the output node;

[0094] Sequentially splicing the pre-processing model of the feature extraction model to be deployed, the feature extraction model to be deployed, and the post-processing model of the feature extraction model to be deployed to obtain a target information feature extraction model;

[0095] Constructing a pre-processing model of the fusion model to be deployed according to the dimension information of the output node of the feature extraction model to be deployed and the pre-processing logic of the fusion model to be deployed, and splicing the pre-processing model of the fusion model to be deployed and the fusion model to be deployed to obtain a target fusion model;

[0096] A media information processing model is obtained according to the target information feature extraction model and the target fusion model so as to be deployed on a target server.

[0097] In one embodiment, the model deployment module 22 is further used to:

[0098] According to the post-processing logic of the fusion model to be deployed, a post-processing model of the fusion model to be deployed is constructed; wherein the post-processing model of the fusion model to be deployed is used to process the output information to be used of the fusion module to be used; the target fusion model is obtained by sequentially splicing the pre-processing model of the fusion model to be deployed, the fusion model to be deployed and the post-processing model of the fusion model to be deployed.

[0099] In one embodiment, the pre-processing model of the feature extraction model to be deployed is used to process the input information of the feature extraction model to be deployed so as to make it suitable for the feature extraction model to be deployed and the target server; the pre-processing model of the fusion model to be deployed is used to process the input information to be input into the fusion model to be deployed so as to make it suitable for the fusion model to be deployed.

[0100] In one implementation, determining the output node of the feature extraction model to be deployed according to the output requirements of the service and the post-processing logic of the feature extraction model to be deployed includes:

[0101] According to the output requirements of the preset business, the input requirements of the fusion model to be deployed and the post-processing logic of the feature extraction model to be deployed, the output node of the feature extraction model to be deployed is determined; wherein the post-processing model of the feature extraction model to be deployed is used to process the output information of the feature extraction model to be deployed so as to be suitable for the fusion model to be deployed and meet the business requirements.

[0102] In one embodiment, the model deployment module 22 is further used to:

[0103] When the feature extraction model to be deployed includes a speech feature extraction model to be deployed, constructing an audio frame extraction model according to input node parameter information of the speech feature extraction model to be deployed, so as to be spliced ​​with the target information feature extraction model;

[0104] When the feature extraction model to be deployed includes a video feature extraction model to be deployed, a video frame extraction model is constructed according to input node parameter information of the video feature extraction model to be deployed to be spliced ​​with the target information feature extraction model.

[0105] In one embodiment, the feature extraction model to be deployed includes at least one of a text feature extraction model to be deployed, a speech feature extraction model to be deployed, and a video feature extraction model to be deployed, wherein the text feature extraction model to be deployed is used to extract text features, the speech feature extraction model to be deployed is used to extract speech features, and the video feature extraction model to be deployed is used to extract video features.

[0106] In one implementation, the model acquisition module is specifically used to:

[0107] Obtaining a trained original information feature extraction model, a trained original fusion model, and actual operating environment information;

[0108] The original information feature extraction model and the original fusion model are respectively converted according to the actual operating environment information to obtain a feature extraction model to be deployed and a fusion model to be deployed suitable for the actual operating environment.

[0109] In one embodiment, all of the pre-processing models and all of the post-processing models are of the type of computational graphs.

[0110] It is worth noting that the working principle of the model deployment device provided in the above embodiments can refer to the workflow of the model deployment method provided in any of the above embodiments, which will not be elaborated here.

[0111] Compared with the prior art, the model deployment device provided in the embodiment of the present invention obtains a media information processing model by splicing a feature extraction model to be deployed and a fusion model to be deployed that are suitable for the actual operating environment, wherein the feature extraction model to be deployed is used to extract the features of the received original media information, and the fusion model to be deployed is used to generate target media information according to the features of the original media information, and the media information processing model is deployed on the target server, thereby realizing the reasonable deployment of the model, reducing the number of data interactions between the client and the server, thereby reducing the time delay caused by multiple communications, and improving data processing efficiency, and is suitable for East-West Computing Project.

[0112] An embodiment of the present invention further provides an information processing device, including:

[0113] An information processing module is used to input the original media information into a media information processing model deployed on a target server to obtain target media information; wherein the media information processing model includes a spliced ​​feature extraction model to be deployed and a fusion model to be deployed, the feature extraction model to be deployed is used to extract the features of the received original media information, and the fusion model to be deployed is used to generate target media information according to the features of the media information, and the feature extraction model to be deployed and the fusion model to be deployed are suitable for an actual operating environment.

[0114] It is worth noting that the working principle of the information processing device provided in the above embodiments can refer to the working process of the information processing method provided in any of the above embodiments, which will not be elaborated here.

[0115] Compared with the prior art, the information processing device disclosed in the embodiment of the present invention obtains target media information by inputting original media information into a media information processing model deployed on a target server; wherein the media information processing model includes a spliced ​​feature extraction model to be deployed and a fusion model to be deployed, the feature extraction model to be deployed is used to extract the features of the received original media information, and the fusion model to be deployed is used to generate target media information according to the features of the original media information. It can be seen that this embodiment uses the media information processing model spliced ​​by multiple models deployed on the target server to perform information processing, thereby reducing the number of data interactions between the client and the server, thereby reducing the delay caused by multiple communications, improving data processing efficiency, and is suitable for East-West Computing projects.

[0116] See also Figure 5 The embodiment of the present invention further provides an electronic device, including a processor 31, a memory 32, and a computer program stored in the memory 32 and configured to be executed by the processor 31. When the processor 31 executes the computer program, the steps in the above-mentioned model deployment method embodiment or the above-mentioned information processing method are implemented, for example Figure 1 S11 to S12 in the above description; or, the processor 31 implements the functions of the modules in the above-mentioned device embodiments when executing the computer program.

[0117] Exemplarily, the computer program may be divided into one or more modules, which are stored in the memory 32 and executed by the processor 31 to complete the present invention. The one or more modules may be a series of computer program instruction segments capable of completing specific functions, which are used to describe the execution process of the computer program in the electronic device. For example, the computer program may be divided into multiple modules, and the specific functions of each module are as follows:

[0118] A model acquisition module 21, used to acquire a feature extraction model to be deployed and a fusion model to be deployed, wherein the feature extraction model to be deployed is used to extract features of the received original media information, and the fusion model to be deployed is used to generate target media information according to the features of the original media information;

[0119] The model deployment module 22 is used to splice the feature extraction model to be deployed and the fusion model to be deployed to obtain a media information processing model for deployment on a target server.

[0120] The specific working process of each module can refer to the working process of the model deployment device described in the above embodiment, which will not be repeated here.

[0121] The electronic device may be a computing device such as a desktop computer, a notebook, a PDA, a cloud server, etc. The electronic device may include, but is not limited to, a processor 31 and a memory 32. Those skilled in the art will appreciate that the electronic device may also include an input / output device, a network access device, a bus, etc.

[0122] The processor 31 may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or any conventional processor, etc. The processor 31 is the control center of the electronic device, and uses various interfaces and lines to connect various parts of the entire electronic device.

[0123] The memory 32 can be used to store the computer program and / or module, and the processor 31 realizes various functions of the electronic device by running or executing the computer program and / or module stored in the memory 32, and calling the data stored in the memory 32. The memory 32 can mainly include a program storage area and a data storage area, wherein the program storage area can store an operating system, at least one application required for a function (such as an image playback function, etc.); the data storage area can store data created according to the use of the mobile phone, etc. In addition, the memory 32 can include a high-speed random access memory, and can also include a non-volatile memory, such as a hard disk, a memory, a plug-in hard disk, a smart memory card (Smart Media Card, SMC), a secure digital (Secure Digital, SD) card, a flash card (Flash Card), at least one disk storage device, a flash memory device, or other volatile solid-state storage devices.

[0124] Wherein, if the module integrated in the electronic device is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on such an understanding, the present invention implements all or part of the processes in the above-mentioned embodiment method, and can also be completed by instructing the relevant hardware through a computer program. The computer program can be stored in a computer-readable storage medium, and the computer program can implement the steps of the above-mentioned various method embodiments when executed by the processor 31. Wherein, the computer program includes computer program code, and the computer program code can be in source code form, object code form, executable file or some intermediate form. The computer-readable medium may include: any entity or device capable of carrying the computer program code, recording medium, U disk, mobile hard disk, disk, optical disk, computer memory, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), electrical carrier signal, telecommunication signal and software distribution medium, etc.

[0125] An embodiment of the present invention also provides a computer program product, including a computer program / instruction, which, when executed by a processor, implements the model deployment method or information processing method as described in any of the above embodiments.

[0126] The above is a preferred embodiment of the present invention. It should be pointed out that a person skilled in the art can make several improvements and modifications without departing from the principle of the present invention. These improvements and modifications are also considered to be within the scope of protection of the present invention.

Claims

1. A model deployment method, characterized in that: include: Acquire a feature extraction model to be deployed and a fusion model to be deployed, wherein the feature extraction model to be deployed is used to extract features of the received original media information, and the fusion model to be deployed is used to generate target media information according to the features of the original media information; The feature extraction model to be deployed and the fusion model to be deployed are spliced ​​to obtain a media information processing model to be deployed on a target server.

2. The model deployment method according to claim 1, characterized in that: The step of splicing the feature extraction model to be deployed and the fusion model to be deployed to obtain a media information processing model for deployment on a target server includes: Constructing a pre-processing model of the feature extraction model to be deployed according to the pre-processing logic and input requirements of the feature extraction model to be deployed; According to the output requirements of the business and the post-processing logic of the feature extraction model to be deployed, determine the output node of the feature extraction model to be deployed, and build a corresponding post-processing model for the output node; Sequentially splicing the pre-processing model of the feature extraction model to be deployed, the feature extraction model to be deployed, and the post-processing model of the feature extraction model to be deployed to obtain a target information feature extraction model; Constructing a pre-processing model of the fusion model to be deployed according to the dimension information of the output node of the feature extraction model to be deployed and the pre-processing logic of the fusion model to be deployed, and splicing the pre-processing model of the fusion model to be deployed and the fusion model to be deployed to obtain a target fusion model; A media information processing model is obtained according to the target information feature extraction model and the target fusion model so as to be deployed on a target server.

3. The model deployment method according to claim 2, characterized in that: The method further comprises: According to the post-processing logic of the fusion model to be deployed, a post-processing model of the fusion model to be deployed is constructed; wherein the post-processing model of the fusion model to be deployed is used to process the output information to be used of the fusion module to be used; the target fusion model is obtained by sequentially splicing the pre-processing model of the fusion model to be deployed, the fusion model to be deployed and the post-processing model of the fusion model to be deployed.

4. The model deployment method according to claim 2, characterized in that: The pre-processing model of the feature extraction model to be deployed is used to process the input information of the feature extraction model to be deployed so as to make it suitable for the feature extraction model to be deployed and the target server; the pre-processing model of the fusion model to be deployed is used to process the input information of the fusion model to be deployed so as to make it suitable for the fusion model to be deployed.

5. The model deployment method according to claim 2, characterized in that: The step of determining the output node of the feature extraction model to be deployed according to the output requirements of the business and the post-processing logic of the feature extraction model to be deployed includes: According to the output requirements of the preset business, the input requirements of the fusion model to be deployed and the post-processing logic of the feature extraction model to be deployed, the output node of the feature extraction model to be deployed is determined; wherein the post-processing model of the feature extraction model to be deployed is used to process the output information of the feature extraction model to be deployed so as to be suitable for the fusion model to be deployed and meet the business requirements.

6. The model deployment method according to claim 1, characterized in that: Also includes: When the feature extraction model to be deployed includes a speech feature extraction model to be deployed, constructing an audio frame extraction model according to input node parameter information of the speech feature extraction model to be deployed, so as to be spliced ​​with the target information feature extraction model; When the feature extraction model to be deployed includes a video feature extraction model to be deployed, a video frame extraction model is constructed according to input node parameter information of the video feature extraction model to be deployed to be spliced ​​with the target information feature extraction model.

7. The model deployment method according to claim 1, characterized in that: The feature extraction model to be deployed includes at least one of a text feature extraction model to be deployed, a speech feature extraction model to be deployed, and a video feature extraction model to be deployed, wherein the text feature extraction model to be deployed is used to extract text features, the speech feature extraction model to be deployed is used to extract speech features, and the video feature extraction model to be deployed is used to extract video features.

8. The model deployment method according to claim 1, characterized in that: The feature extraction model to be deployed and the fusion model to be deployed are obtained in the following manner: Obtaining a trained original information feature extraction model, a trained original fusion model, and actual operating environment information; The original information feature extraction model and the original fusion model are respectively converted according to the actual operating environment information to obtain a feature extraction model to be deployed and a fusion model to be deployed suitable for the actual operating environment.

9. The model deployment method according to claim 2, characterized in that: The types of all the pre-processing models and all the post-processing models are computational graphs.

10. An information processing method, characterized in that: include: The original media information is input into a media information processing model deployed on a target server to obtain target media information; wherein the media information processing model includes a spliced ​​feature extraction model to be deployed and a fusion model to be deployed, the feature extraction model to be deployed is used to extract features of the received original media information, and the fusion model to be deployed is used to generate target media information based on the features of the original media information.

11. The information processing method according to claim 10, characterized in that: The media information processing model is obtained by the model deployment method as described in any one of claims 1 to 9.

12. A model deployment device, characterized in that: include: A model acquisition module, used to acquire a feature extraction model to be deployed and a fusion model to be deployed, wherein the feature extraction model to be deployed is used to extract features of the received original media information, and the fusion model to be deployed is used to generate target media information according to the features of the original media information; The model deployment module is used to splice the feature extraction model to be deployed and the fusion model to be deployed to obtain a media information processing model for deployment on a target server.

13. An information processing device, characterized in that: include: An information processing module is used to input the original media information into a media information processing model deployed on a target server to obtain target media information; wherein the media information processing model includes a spliced ​​feature extraction model to be deployed and a fusion model to be deployed, the feature extraction model to be deployed is used to extract the features of the received original media information, and the fusion model to be deployed is used to generate target media information according to the features of the media information, and the feature extraction model to be deployed and the fusion model to be deployed are suitable for an actual operating environment.

14. An electronic device, characterized in that: The method comprises a processor, a memory, and a computer program stored in the memory and configured to be executed by the processor, wherein when the processor executes the computer program, the model deployment method as described in any one of claims 1 to 9 or the information processing method as described in claim 10 or 11 is implemented.

15. A computer-readable storage medium, characterized in that: The computer-readable storage medium includes a stored computer program, wherein when the computer program is running, the device where the computer-readable storage medium is located is controlled to execute the model deployment method according to any one of claims 1 to 9 or the information processing method according to claim 10 or 11.

16. A computer program product comprising a computer program / instructions, characterized in that When the computer program / instructions are executed by a processor, the model deployment method according to any one of claims 1 to 9 or the information processing method according to claim 10 or 11 is implemented.