Data generation method and device, electronic equipment and computer readable medium

Abstract

The invention discloses a data generation method and device, electronic equipment and a computer readable medium, and the method comprises the steps that after a client side receives a data generation request, the client side processes the data generation request through a target plug-in to obtain generated data, so that data generation processing is carried out on the client side based on the plug-in, and the generated data is sent to the client side; therefore, all related processes of the data generation request are generated at the same end, the defects existing when data generation processing is achieved by means of a client and server collaboration mode can be effectively overcome, and the data generation effect can be improved.

Description

Technical Field

[0001] This application relates to the technical field of data processing, and in particular, to a data generation method, apparatus, electronic device, and computer-readable medium. Background Art

[0002] In some application scenarios, for data such as images, videos, texts, etc., the data can be generated by means of the cooperation between the client and the server. For the sake of easy understanding, the following takes the image generation process in the text-to-image scenario as an example for illustration.

[0003] As an example, the image generation process in the text-to-image scenario can specifically be: after the client detects a text-to-image request triggered by the user, the client forwards the text-to-image request to the server, so that the server can process the request by using a model with text-to-image function that has been deployed in the server to obtain a generated image, and the server feeds back the generated image to the client for display. Summary of the Invention

[0004] This application provides a data generation method, apparatus, electronic device, and computer-readable medium, which is beneficial to improving the data generation effect.

[0005] To achieve the above object, the technical solutions provided by this application are as follows:

[0006] This application provides a data generation method, which is applied to a client, and the method includes:

[0007] Receiving a data generation request;

[0008] Processing the data generation request by using a target plug-in to obtain generated data; the target plug-in is constructed according to a data generation model corresponding to the data generation request, and the data generation model is used to process the data generation request.

[0009] In a possible implementation manner, the data generation request is used to request content generation processing based on reference information; the reference information includes at least one of text and image; the data generation model is a content generation model; the generated data includes at least one generated image.

[0010] In a possible implementation manner, the client is deployed on a terminal device;

[0011] The processing the data generation request by using a target plug-in to obtain generated data includes:

[0012] Sending the data generation request to a target plug-in deployed on the terminal device through a plug-in service deployed on the client;

[0013] Receive the generated data generated by the target plug-in in response to the data generation request.

[0014] In a possible implementation manner, the target plug-in includes a target model determined according to the data generation model; the target plug-in is used to process the data generation request by using the target model to obtain the generated data.

[0015] In a possible implementation manner, the target plug-in includes at least two candidate models, and the data processing processes implemented by different candidate models are different. The at least two candidate models include the target model; the target plug-in is further used to determine the target model corresponding to the data generation request from the at least two candidate models.

[0016] In a possible implementation manner, before using the target plug-in to process the data generation request, the method further includes:

[0017] In response to a client startup request, start the client and load the target plug-in.

[0018] In a possible implementation manner, after loading the target plug-in, the method further includes:

[0019] Perform an initialization process on the loaded target plug-in;

[0020] The using the target plug-in to process the data generation request includes:

[0021] Use the initialized target plug-in to process the data generation request.

[0022] In a possible implementation manner, the loading the target plug-in includes: asynchronously preloading the target plug-in.

[0023] In a possible implementation manner, the target plug-in is constructed according to at least one data generation model, and the data generation processes implemented by different data generation models are different. The at least one data generation model includes the data generation model corresponding to the data generation request.

[0024] In a possible implementation manner, the target plug-in includes a base model and at least one fine-tuning model; the base model and the at least one fine-tuning model are determined according to the at least one data generation model;

[0025] Before using the target plug-in to process the data generation request, the method further includes:

[0026] In response to a client startup request, start the client and load the base model in the target plug-in;

[0027] The process of obtaining the generated data includes:

[0028] Determine the fine-tuning model corresponding to the data generation request from the at least one fine-tuning model;

[0029] Use the fine-tuning model to perform parameter correction processing on the loaded base model to obtain the target model;

[0030] Use the target model to process the data generation request.

[0031] In a possible implementation manner, after receiving the data generation request, the method further includes:

[0032] Determine whether the target plugin is in an available state;

[0033] The using the target plugin to process the data generation request includes:

[0034] If it is determined that the target plugin is in an available state, use the target plugin to process the data generation request.

[0035] In a possible implementation manner, after determining whether the target plugin is in an available state, the method further includes:

[0036] If it is determined that the target plugin is in an unavailable state, send the data generation request to the server, and the server is used to process the data generation request by using the data generation model deployed on the server to obtain the generated data;

[0037] Receive the generated data fed back by the server.

[0038] In a possible implementation manner, the client is deployed on a terminal device;

[0039] After determining whether the target plugin is in an available state, the method further includes:

[0040] If it is determined that the target plugin is in an unavailable state, determine whether the plugin description resource of the target plugin is stored in the terminal device;

[0041] If it is determined that the plugin description resource of the target plugin is not stored in the terminal device, send a resource requirement request to the server;

[0042] Receive and store the plugin description resource fed back by the server for the resource requirement request; the plugin description resource is determined by the server according to the data generation model deployed on the server.

[0043] In a possible implementation, the target plug-in includes at least two data processing modules arranged in sequence, and the at least two data processing modules are obtained by splitting the data generation model;

[0044] The process of obtaining the generated data includes:

[0045] Load the i-th data processing module; i is a positive integer, and the initial value of i is 1;

[0046] Use the i-th data processing module to process data;

[0047] After obtaining the output result of the i-th data processing module, release the memory occupied by the i-th data processing module;

[0048] Update the i, and continue to execute the step of loading the i-th data processing module until, when a preset stop condition is reached, determine the generated data according to the output result of the i-th data processing module.

[0049] In a possible implementation, the process of splitting the data generation model includes:

[0050] Split the data generation model into at least two sub-models, and the data processing functions implemented by different sub-models are different;

[0051] For any one of the sub-models, if the data representing the memory requirement of the sub-model does not exceed a preset memory threshold, then determine the sub-model as the data processing module; if the data representing the memory requirement of the sub-model exceeds the preset memory threshold, then split the sub-model into at least two model segments, and determine each of the model segments as the data processing module, and the data representing the memory requirement of each of the model segments does not exceed the preset memory threshold.

[0052] In a possible implementation, the process of obtaining the at least two model segments includes:

[0053] Obtain at least one candidate splitting description information, and the model splitting methods described by different candidate splitting description information are different;

[0054] For any one of the candidate splitting description information, perform a splitting process on the sub-model according to the candidate splitting description information to obtain a model splitting result corresponding to the candidate splitting description information, and determine resource usage representation data corresponding to the candidate splitting description information according to the model splitting result corresponding to the candidate splitting description information, where the resource usage representation data includes computing resource reuse representation data and memory usage representation data;

[0055] Select a target splitting description information from the at least one candidate splitting description information according to the resource usage characterization data corresponding to each candidate splitting description information, where the resource usage balance presented by the resource usage characterization data corresponding to the target splitting description information is higher than that presented by the resource usage characterization data corresponding to any other candidate splitting description information among the at least one candidate splitting description information except the target splitting description information;

[0056] Determine the at least two model segments according to the model splitting result corresponding to the target splitting description information.

[0057] This application provides a data generation device, including:

[0058] A receiving unit, configured to receive a data generation request;

[0059] A processing unit, configured to process the data generation request by using a target plug-in to obtain generated data; the target plug-in is constructed according to a data generation model corresponding to the data generation request, and the data generation model is used to process the data generation request.

[0060] This application provides an electronic device, where the device includes: a processor and a memory;

[0061] The memory is configured to store instructions or computer programs;

[0062] The processor is configured to execute the instructions or computer programs in the memory, so that the electronic device executes the data generation method provided by this application.

[0063] This application provides a computer-readable medium, characterized in that instructions or computer programs are stored in the computer-readable medium, and when the instructions or computer programs run on a device, the device is enabled to execute the data generation method provided by this application.

[0064] This application provides a computer program product, which includes a computer program carried on a non-transitory computer-readable medium, and the computer program includes program codes for executing the data generation method provided by this application.

[0065] Compared with the related art, this application has at least the following advantages:

[0066] In the technical solution provided by this application, for a client, after the client receives a data generation request, the client uses a target plug-in to process the data generation request to obtain generated data. In this way, data generation processing can be realized based on the plug-in on the client, which can effectively overcome the defects existing when realizing data generation processing by means of the cooperation between the client and the server, and thus is beneficial to improving the data generation effect. Among them, since the target plug-in is constructed according to the data generation model corresponding to the data generation request, the data processing function of the target plug-in includes the data processing function of the data generation model; also, since the data generation model can be used to process the data generation request, the target plug-in can also be used to process the data generation request. Therefore, after the client receives the data generation request, the client can directly use the target plug-in to process the data generation request, so that all relevant processes of the data generation request occur on the same side. In this way, a single-end closed loop of data generation processing can be realized, which can effectively improve the effects presented by the data generation process in terms of real-time performance, security, reliability, etc. Brief Description of the Drawings

[0067] In order to more clearly illustrate the technical solutions in the embodiments of this application or related technologies, the following will briefly introduce the drawings required for use in the description of the embodiments or related technologies. Obviously, the drawings in the following description are only some embodiments recorded in this application. For those of ordinary skill in the art, without creative efforts, other drawings can also be obtained based on these drawings.

[0068] Figure 1 It is a flowchart of a data generation method provided by an embodiment of this application;

[0069] Figure 2 It is a schematic diagram of a content generation process provided by an embodiment of this application;

[0070] Figure 3 It is a schematic diagram of a content generation process provided by an embodiment of this application;

[0071] Figure 4 It is a schematic diagram of a model splitting method provided by an embodiment of this application;

[0072] Figure 5 It is a schematic diagram of another model splitting method provided by an embodiment of this application;

[0073] Figure 6 It is a schematic diagram of an implementation method of a variety of data generation processes provided by an embodiment of this application;

[0074] Figure 7Structural schematic diagram of a data generation device provided by an embodiment of the present application;

[0075] Figure 8 Structural schematic diagram of an electronic device provided by an embodiment of the present application. Detailed implementation manners

[0076] After research, it is found that for some application scenarios, such as content generation scenarios similar to text-to-image, image-to-image, text-to-video, and image-to-video, the content generation process in these scenarios can be realized by means of cooperation between the client and the server; and the specific implementation process can be: after the client detects a content generation request triggered by the user, such as a text-to-image request, the client forwards the content generation request to the server, so that the server can process the request by using a model with content generation function deployed on the server to obtain the generated content, and the server feeds back the generated content to the client, so that the client can display the generated content.

[0077] After research, it is also found that for the content generation process realized by the cooperation between the client and the server shown in the above paragraph, it has the defects shown in ①-④ below.

[0078] ① Defects caused by network latency, specifically: because the client needs to forward the above content generation request to the server through the network, so when the network has latency, it will affect the response time of the content generation request, thus affecting the user experience.

[0079] ② Defects presented in terms of data security, specifically: because the client needs to forward the above content generation request to the server for processing, so some problems that may affect data security may occur during the forwarding process and processing process of the content generation request, thus affecting the user experience.

[0080] ③ Defects in terms of stability and reliability, specifically: because the client and the server are connected through the network, so the stability and reliability of the network may affect the response process of the above content generation request. For example, if the network fails or the network bandwidth is limited, it may lead to an increase in the response time of the content generation request, or even the response processing for the content generation request cannot be completed.

[0081] ④ Defects in terms of content generation cost, specifically: because using the computing resources and storage resources in the server may incur additional costs, so the cost required for the content generation process realized by the cooperation between the client and the server is relatively high.

[0082] Based on the above research, in order to better improve the data generation experience, such as content generation experience, etc., the present application provides a data generation method, which includes: for a client, after the client receives a data generation request, the client uses a target plugin to process the data generation request to obtain generated data. In this way, data generation processing can be realized based on the plugin on the client, thereby effectively overcoming the defects existing when data generation processing is realized by means of the cooperation between the client and the server, and further facilitating the improvement of the data generation effect. Among them, the target plugin is constructed according to the data generation model corresponding to the data generation request, so that the data processing function of the target plugin includes the data processing function of the data generation model; also because the data generation model can be used to process the data generation request, so that the target plugin can also be used to process the data generation request, so that after the client receives the data generation request, the client can directly use the target plugin to process the data generation request, and further all relevant processes of the data generation request occur on the same end. In this way, a single-end closed loop of data generation processing can be realized, thereby effectively improving the effects presented by the data generation process in terms of real-time performance, security, reliability, etc.

[0083] In order to enable those skilled in the art to better understand the solution of the present application, the following will clearly and completely describe the technical solutions in the embodiments of the present application with reference to the accompanying drawings in the embodiments of the present application. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without making creative efforts belong to the scope of protection of the present application.

[0084] To better understand the technical solution provided by the present application, the data generation method provided by the present application will be described below with reference to some drawings first. As Figure 1 shown, the data generation method provided by the embodiment of the present application includes S1-S2 below. Among them, the Figure 1 is a flowchart of a data generation method provided by an embodiment of the present application.

[0085] S1: The client receives a data generation request.

[0086] Among them, the client is used to provide some services for users, such as data generation services similar to text-to-image, etc. For example, the client can be implemented using the Figure 2 or Figure 3 shown client.

[0087] It should be noted that the present application does not limit the implementation manner of the data generation service in the above paragraph. For example, in some application scenarios, the data generation service may refer to a content generation service; and the present application does not limit the content generation service. For example, the content generation service may include a text-to-image service, an image-to-image service, a text-to-video service, an image-to-video service, etc.

[0088] In addition, the present application does not limit the implementation manner of the above-mentioned client. For example, the client may be implemented using an existing or future user terminal capable of interacting with the user, such as an application (app) or a web page, etc.

[0089] Furthermore, for the above-mentioned client, the client may be deployed on a terminal device so that the user of the terminal device can use the client to meet some of the user's needs, such as data generation requirements for text-to-image, image-to-image, text-to-video, image-to-video, etc. It should be noted that the present application does not limit the implementation manner of the terminal device. For example, the terminal device may be a smart phone, a computer, a personal digital assistant (PDA), a tablet computer, etc.

[0090] A data generation request refers to a request triggered by a user on a client for requesting a certain data generation process; and the present application does not limit the data generation request. For example, in some application scenarios, the data generation request may be implemented using Figure 2 or Figure 3 the content generation request shown. Among them, the content generation request is used to request a content generation process, such as text-to-image, image-to-image, text-to-video, image-to-video, etc. It can be seen that in a possible implementation manner, the data generation request may be used to request a certain content generation process, such as text-to-image, image-to-image, text-to-video, image-to-video, etc.

[0091] Based on the above content, it can be known that in some application scenarios, such as content generation scenarios, the above-mentioned data generation request may be used to request content generation processing based on reference information, so that the data generation request carries the reference information. Among them, the reference information refers to the information required for content generation processing; and the present application does not limit the reference information. For example, in some application scenarios, the reference information may include at least one of text and image. For the sake of understanding, some application scenarios are described below.

[0092] Scenario 1. In some scenarios, such as text-to-image and text-to-video scenarios, the above data generation request can be used to request text-to-image processing or text-to-video processing; moreover, when the data generation request carries at least a prompt, the data generation request can be used to request text-to-image processing or text-to-video processing based on the prompt. Based on this, it can be known that in a possible implementation manner, the reference information carried by the data generation request can at least include the prompt, so that the data generation request can be used to request content generation processing based on the prompt, such as text-to-image or text-to-video processing, etc.

[0093] Scenario 2. In some scenarios, such as image-to-image and image-to-video scenarios, the above data generation request can be used to request image-to-image processing or image-to-video processing; moreover, when the data generation request carries at least a source image, the data generation request can be used to request image-to-image processing or image-to-video processing based on the source image. Based on this, it can be known that in a possible implementation manner, the reference information carried by the data generation request can at least include the source image, so that the data generation request can be used to request content generation processing based on the source image, such as image-to-image or image-to-video processing, etc.

[0094] Scenario 3. In some scenarios, such as image generation scenarios based on multiple control information or video generation scenarios based on multiple control information, etc., the above data generation request can carry multiple control information, so that the data generation request can be used to request image generation processing or video generation processing based on these control information. Among them, the multiple control information is used to describe the constraints required in image generation processing or video generation processing; and the constraints described by different control information are different. In addition, the present application does not limit the implementation manner of the multiple control information. For example, the multiple control information can include at least one text and at least one image. Based on this, it can be known that in a possible implementation manner, the reference information carried by the data generation request can at least include the at least one text and the at least one image, so that the data generation request can be used to request content generation processing based on the at least one text and the at least one image.

[0095] Based on the relevant content of S1 above, for the terminal device used by the user, if a client is deployed on the terminal device, then after the user triggers a data generation request on the client, the client can perform some response processing on the data generation request, such as the response processing shown in S2 below, etc., to meet the user's data generation requirements.

[0096] S2: The client uses the target plugin to process the data to generate a request and obtains the generated data. The target plugin is constructed based on the data generation model corresponding to the data generation request, and the data generation model is used to process the data generation request.

[0097] Among them, the target plugin refers to the plugin required when the client processes the above data generation request, so that the client can complete the relevant processing process for the data generation request with the help of the target plugin, so that all the relevant processing processes for the data generation request occur on the same side. In this way, the data generation processing can be realized in a single-end closed loop, thus effectively overcoming the defects existing when the data generation processing is realized by means of multi-end collaboration.

[0098] In addition, for the above target plugin, the target plugin is constructed based on the data generation model corresponding to the above data generation request, so that the data processing function of the target plugin includes the data processing function of the data generation model, so that when the data generation model can be used to process the data generation request, the target plugin can also be used to process the data generation request. Among them, the data generation model refers to a pre-constructed model with the function of processing the data generation request; moreover, the implementation manner of the above data generation model in this application is not limited. For example, in some application scenarios, the data generation model can be implemented using an Artificial Intelligence Generated Content (AIGC) model, so that the data generation model can be used to process any content generation request, such as a text-to-image request, an image-to-image request, a text-to-video request, or an image-to-video request, etc.

[0099] In addition, for the data generation model in the above paragraph, the data generation model is deployed on a device independent of the client, such as Figure 2 the server shown, etc., so that the device can provide the relevant resources of the target plugin to the client according to the data generation model. It can be seen that in a possible implementation manner, the target plugin is constructed based on the data generation model deployed on the server that can process the data generation request, so that the target plugin can be used instead of the data generation model. Among them, data communication can be carried out between the server and the client.

[0100] Furthermore, the implementation manner of the above target plugin in this application is not limited. For the convenience of understanding, some examples are given below for illustration.

[0101] Example 1. In some application scenarios, such as when the model volume is relatively small, when the target plug-in is constructed based on a data generation model deployed on the server that can process the data generation request, the target plug-in may include a target model determined based on the data generation model, so that the target plug-in can be used to directly process the data generation request using the target model. Among them, the target model refers to the model required when using the target plug-in to process the data generation request; and the target model is determined based on the data generation model corresponding to the data generation request, so that the data generation function of the target model is consistent with the data generation function of the data generation model, so that the target model can be used instead of the data generation model.

[0102] It should be noted that the present application does not limit the determination process of the target model in the above paragraph. For example, the determination process of the target model may specifically be: directly determining the data generation model corresponding to the above data generation request as the target model. Another example is that the determination process of the target model may be: first extracting the core logic from the data generation model corresponding to the data generation request; then implementing the core logic in a preset expression manner to obtain the target model. Among them, the preset expression manner refers to a manner that is preset and can be interpreted on the terminal device; and the present application does not limit the preset expression manner. For example, it may be pure C++.

[0103] Example 2. In some application scenarios, in order to better save resources, one plug-in can be used to replace multiple data processing models on the server, such as AIGC models, speech recognition models, etc. Based on this, the present application also provides a possible implementation manner of the above target plug-in. In this implementation manner, the target plug-in is constructed based on at least two data processing models, and the at least two data processing models include the data generation model corresponding to the above data generation request, so that the target plug-in includes candidate models determined based on each data processing model, so that the functions of the target plug-in include the functions of these data processing models, and further enabling the target plug-in to replace these data processing models for use. Based on this, it can be known that in a possible implementation manner, the target plug-in may include at least two candidate models, and the at least two candidate models include the above target model. Among them, because different candidate models are constructed based on different data processing models deployed on the server, so that the data processing processes implemented by different candidate models are different, so that the target plug-in including these candidate models can replace these data processing models deployed on the server for use.

[0104] Based on the above content, in a possible implementation, when the above-mentioned target plug-in includes at least two candidate models, and the at least two candidate models include the target model determined according to the data generation model corresponding to the data generation request in the above text, the target plug-in can be used to determine the target model corresponding to the data generation request in the above text from the at least two candidate models, and use the target model to process the data generation request. It should be noted that this application does not limit the implementation manner of the step of "determining the target model corresponding to the data generation request in the above text from the at least two candidate models". For example, when the data generation request carries model description information, this step can specifically be: matching the model description information carried by the data generation request with the model description information of each candidate model, and determining the candidate model that best matches the model description information carried by the data generation request as the target model corresponding to the data generation request. Among them, the model description information is used to describe the characteristics of a model; and this application does not limit the implementation manner of the model description information. For example, it can be implemented using a model configuration file.

[0105] Example 3, in some application scenarios, such as scenarios where the model size is relatively large, etc., in order to better overcome the memory overflow defect caused by the relatively large model size, this application also provides a possible implementation manner of the above-mentioned target plug-in. In this implementation manner, the target plug-in includes at least two data processing modules arranged in sequence. The at least two data processing modules are obtained by splitting the data generation model corresponding to the data generation request in the above text, so that the target plug-in can complete the processing process for the data generation request by sequentially using these data processing modules, such as Figure 5 Or Figure 6 The module usage method shown to complete the processing process for the data generation request. Among them, the two data processing modules refer to the modules required to be used when the target plug-in processes the data generation request.

[0106] In addition, this application does not limit the splitting process of the data generation model corresponding to the data generation request in the above text. For example, it can specifically be: directly splitting the data generation model into multiple equal parts, and using each equal part obtained by splitting as a data processing module.

[0107] For another example, in order to better avoid the occurrence of memory overflow, this application also provides a possible implementation manner of the splitting process of the data generation model corresponding to the data generation request in the above text. In this implementation manner, the splitting process may include the following steps 11 - step 13.

[0108] Step 11: Split the data generation model corresponding to the data generation request in the above text into at least two sub-models, and the data processing functions implemented by different sub-models are different.

[0109] Among them, a sub-model refers to a model existing in the data generation model corresponding to the above data generation request that can implement a certain data processing function. For example, the sub-model can be Figure 4 or Figure 5 the text feature extraction model, image feature encoding model, other information extraction model, picture feature decoding model, as well as the denoising model and control information adaptation model shown, etc.

[0110] In addition, the present application does not limit the implementation manner of step 11 above. For example, step 11 can specifically be: according to the data processing function, perform model splitting processing on the data generation model corresponding to the above data generation request to obtain at least two sub-models, so that each sub-model represents a data processing function.

[0111] For another example, step 11 above can specifically be: according to the pre-set model splitting rule, split at least two sub-models from the data generation model corresponding to the above data generation request, so that each sub-model can be used for one or more data processing functions. Among them, the model splitting rule refers to the rule determined in advance for the actual application scenario and used to split a model into multiple sub-models; and the present application does not limit this model splitting rule.

[0112] For yet another example, in order to better improve the flexibility of sub-model splitting, step 11 above can specifically be: based on the correlation degree between each pair of adjacent network layers in the data generation model corresponding to the above data generation request, split at least two sub-models from the data generation model, so that the correlation degree between any pair of network layers in each sub-model is higher than the preset correlation degree threshold, so that each sub-model can as completely as possible implement one or more data processing functions, and thus can automatically perform model splitting processing on the premise of ensuring the functional integrity of each sub-model. Among them, for any pair of adjacent network layers in the data generation model, the correlation degree between this pair of adjacent network layers is used to characterize the correlation between this pair of adjacent network layers, so that the correlation degree between this pair of adjacent network layers can indicate whether this pair of adjacent network layers belong to different steps under the same data processing process; and the present application does not limit the implementation manner of this correlation degree. For example, in some application scenarios, this correlation degree can be obtained through manual annotation. For another example, in some application scenarios, this correlation degree can be automatically determined during the training process of the data generation model, so that this correlation degree can be updated accordingly with the training and updating process of the data generation model.

[0113] Step 12: If the memory requirement characterization data of the j-th sub-model in the data generation model corresponding to the above data generation request does not exceed the preset memory threshold, then determine the j-th sub-model as the data processing module, where j is a positive integer, j ≤ J, and J is a positive integer representing the number of sub-models in at least two sub-models in the data generation model corresponding to the above data generation request.

[0114] Among them, the j-th sub-model is used to represent any one of the sub-models in the data generation model corresponding to the above data generation request; and the j-th sub-model is used to implement one or more data processing functions. In addition, the present application does not limit the implementation manner of the j-th sub-model. For example, when the data generation model includes Figure 4 or Figure 5 the text feature extraction model, image feature encoding model, other information extraction model, picture feature decoding model, denoising model, and control information adaptation model shown, the j-th sub-model can be the text feature extraction model, the image feature encoding model, the other information extraction model, or the picture feature decoding model; or, the j-th sub-model can be the denoising model and the control information adaptation model.

[0115] The memory requirement characterization data of the above j-th sub-model is used to represent the memory required to be consumed when using the j-th sub-model; and the present application does not limit the determination method of the memory requirement characterization data of the j-th sub-model. For example, it can be implemented by using any existing or future method that can measure the memory requirement of a model.

[0116] The preset memory threshold is used to represent the upper limit of the memory consumption of a model; and the present application does not limit the acquisition method of the preset memory threshold. For example, it can be manually set by the user. Again, in some application scenarios, in order to better avoid the occurrence of memory overflow, the preset memory threshold can be determined according to the memory parameters of the terminal device to ensure that the preset memory threshold can more accurately represent the memory upper limit required when using a model on the terminal device. Among them, the memory parameter is used to describe the memory resource situation in the terminal device, such as the maximum available memory, etc.; and the present application does not limit the memory parameter. In addition, the present application does not limit the implementation manner of determining the preset memory threshold according to the memory parameter.

[0117] Based on the relevant content of Step 12 above, for the j-th sub-model in the data generation model corresponding to the above data generation request, if the memory requirement characterization data of the j-th sub-model does not exceed the preset memory threshold, it can be determined that there will be no memory overflow when using the j-th sub-model. Therefore, in order to better improve efficiency, there is no need to further split the j-th sub-model, and only need to directly determine the j-th sub-model as a data processing module.

[0118] Step 13: If the memory requirement characterization data of the j-th sub-model in the above data generation model corresponding to the data generation request exceeds the preset memory threshold, then split the j-th sub-model into at least two model segments, and determine each model segment as a data processing module, and the memory requirement characterization data of each model segment does not exceed the preset memory threshold.

[0119] In this application, for the j-th sub-model in the above data generation model, if the memory requirement characterization data of the j-th sub-model exceeds the preset memory threshold, it can be determined that there is a relatively high possibility of memory overflow when using the j-th sub-model. Therefore, in order to better avoid the occurrence of memory overflow, the j-th sub-model is split into at least two model segments, such as Figure 4 Or Figure 5 As shown in Segment 1 - Segment 4, etc., so that the memory requirement characterization data of each model segment does not exceed the preset memory threshold, so that there will be no memory overflow when using each model segment, and each model segment is determined as a data processing module.

[0120] In addition, for the above-mentioned j-th sub-model, the at least two model segments obtained by splitting the j-th sub-model have the following characteristics: Characteristic 1, for any one model segment, the model segment includes one or more network layers in the j-th sub-model, so that the model segment can represent a certain part of the j-th sub-model, so that the model segment can implement the functions required by the corresponding part in the j-th sub-model. Characteristic 2, different model segments are used to represent different parts in the j-th sub-model, so that the union of the at least two model segments can implement the data processing function of the j-th sub-model. Characteristic 3, there is no intersection between any two model segments.

[0121] In addition, this application does not limit the implementation manner of the step of "splitting the j-th sub-model into at least two model segments" in the above Step 13. For example, it can evenly divide the j-th sub-model into multiple model segments. Another example is that it can be implemented according to a preset sub-model splitting rule. The sub-model splitting rule refers to a rule that is preset according to the application scenario and is used as a basis for splitting a sub-model with relatively large memory consumption.

[0122] Furthermore, in order to better improve the flexibility of sub-model splitting, this application also provides a possible implementation manner of the splitting process of the above-mentioned j-th sub-model. In this implementation manner, the splitting process may specifically include the following Steps 131 - 134.

[0123] Step 131: Obtain at least one candidate splitting description information, and different candidate splitting description information describes different model splitting methods.

[0124] The k-th candidate splitting description information refers to the information required for splitting a sub-model according to the k-th splitting method, so that the k-th candidate splitting description information can represent the characteristics presented by the k-th splitting method. k is a positive integer, and k ≤ the number of candidate splitting description information in at least one candidate splitting description information above.

[0125] In addition, the present application does not limit the implementation manner of the k-th candidate splitting description information above. For example, the k-th candidate splitting description information can describe each splitting position required for splitting a sub-model according to the k-th splitting method. Another example is that the k-th candidate splitting description information can be used to describe the splitting rule of the k-th splitting method, such as the splitting position determination rule, etc.

[0126] Furthermore, the present application does not limit the acquisition method of the k-th candidate splitting description information above. For example, it can be pre-set by relevant personnel.

[0127] Step 132: For any candidate splitting description information, split the j-th sub-model above according to the candidate splitting description information to obtain the model splitting result corresponding to the candidate splitting description information, and determine the resource usage characterization data corresponding to the candidate splitting description information based on the model splitting result corresponding to the candidate splitting description information. The resource usage characterization data includes computing resource reuse characterization data and memory usage characterization data.

[0128] The model splitting result corresponding to the k-th candidate splitting description information refers to the result obtained by splitting the j-th sub-model above according to the k-th candidate splitting description information, so that the "model splitting result corresponding to the k-th candidate splitting description information" can represent the multiple segments obtained by splitting the j-th sub-model in the k-th splitting method. k is a positive integer, and k ≤ the number of candidate splitting description information in at least one candidate splitting description information above.

[0129] The resource usage characterization data corresponding to the k-th candidate splitting description information refers to the resource usage situation presented when the data processing function described by the j-th sub-model above is implemented using the model splitting result corresponding to the k-th candidate splitting description information, such as the computing resource reuse situation and the memory occupancy situation.

[0130] In addition, the present application does not limit the implementation manner of the resource usage characterization data corresponding to the k-th candidate splitting description information above. For example, it may at least include the computing resource reuse characterization data corresponding to the k-th candidate splitting description information and the memory usage characterization data corresponding to the k-th candidate splitting description information. The computing resource reuse characterization data corresponding to the k-th candidate splitting description information refers to the computing reuse situation presented when implementing the data processing function described by the j-th sub-model above using the model splitting result corresponding to the k-th candidate splitting description information. The memory usage characterization data corresponding to the k-th candidate splitting description information refers to the memory occupancy situation presented when implementing the data processing function described by the j-th sub-model using the model splitting result corresponding to the k-th candidate splitting description information.

[0131] Furthermore, the present application does not limit the acquisition manner of the resource usage characterization data corresponding to the k-th candidate splitting description information above. For example, it can be implemented using any existing or future method that can analyze the computing reuse situation and memory occupancy situation of some models. As an example, the acquisition process of the resource usage characterization data corresponding to the k-th candidate splitting description information can be: using the model splitting result corresponding to the k-th candidate splitting description information to process a sample request, and recording the computing reuse information and memory occupancy information involved in the process of processing the sample request, so that after the processing of the sample request is completed, based on these computing reuse information and memory occupancy information, determine the resource usage characterization data corresponding to the k-th candidate splitting description information. Wherein, the sample request refers to the request required when analyzing the computing reuse situation and memory occupancy situation presented by the j-th sub-model in a certain candidate splitting description information; and this sample request is similar to the data generation request above.

[0132] Step 133: Select a target splitting description information from at least one candidate splitting description information above according to the resource usage characterization data corresponding to each candidate splitting description information. The resource usage balance degree presented by the resource usage characterization data corresponding to the target splitting description information is higher than the resource usage balance degree presented by the resource usage characterization data corresponding to any other candidate splitting description information except the target splitting description information among the at least one candidate splitting description information.

[0133] In this application, for the j-th sub-model mentioned above, after determining the resource usage characterization data corresponding to each candidate split description information for the j-th sub-model, the target split description information can be selected from the at least one candidate split description information mentioned above based on these resource usage characterization data, so that the target split description information is used to represent the candidate split description information that matches the j-th sub-model, thereby enabling the target split description information to represent the split method finally selected for the j-th sub-model. Among them, the target split description information refers to the candidate split description information selected from the at least one candidate split description information that matches the j-th sub-model, so that the target split description information can represent the information required when performing split processing on the j-th sub-model, such as each split position, etc.

[0134] In addition, for the j-th sub-model mentioned above, the conditions required when selecting the target split description information that matches the j-th sub-model from the at least one candidate split description information mentioned above include: the resource usage balance degree presented by the resource usage characterization data corresponding to the target split description information is higher than the resource usage balance degree presented by the resource usage characterization data corresponding to any other candidate split description information except the target split description information among the at least one candidate split description information. Among them, the resource usage balance degree presented by the resource usage characterization data corresponding to the k-th candidate split description information is used to characterize the balance degree presented between computing reuse and memory occupancy when implementing the data processing function described by the j-th sub-model above using the model split result corresponding to the k-th candidate split description information, so that the resource usage balance degree can represent the balance presented by the model split result corresponding to the k-th candidate split description information under multi-dimensional performance; moreover, this application does not limit the acquisition process of the resource usage balance degree. For example, it can adopt any existing or future method for calculating the balance degree under multi-dimensional performance, such as a method based on a pre-set balance degree calculation rule or a method based on a pre-set balance degree calculation formula, etc., for implementation. k is a positive integer, and k ≤ the number of candidate split description information in the at least one candidate split description information.

[0135] Based on the relevant content of the above-mentioned step 133, for the j-th sub-model above, after determining the resource usage characterization data corresponding to each candidate splitting description information for the j-th sub-model, the target splitting description information can be selected from the above-mentioned at least one candidate splitting description information according to these resource usage characterization data, so that the result obtained by splitting the j-th sub-model according to the target splitting description information can present the greatest balance between computational reuse and memory occupancy. In this way, it is possible to reduce the number of computational reuses as much as possible on the premise of ensuring no memory overflow, which is beneficial to improving the response speed as much as possible on the premise of not occurring memory overflow.

[0136] Step 134: Determine at least two model segments corresponding to the j-th sub-model above according to the model splitting result corresponding to the above-mentioned target splitting description information.

[0137] In this application, for the j-th sub-model above, after selecting the target splitting description information for the j-th sub-model from the above-mentioned at least one candidate splitting description information, at least two model segments corresponding to the j-th sub-model can be determined according to the model splitting result corresponding to the target splitting description information, so that the at least two model segments can represent the result obtained by splitting the j-th sub-model according to the splitting method described in the target splitting description information.

[0138] Based on the relevant content of the above-mentioned steps 131 to 134, in some application scenarios, for the j-th sub-model in the data generation model corresponding to the above-mentioned data generation request, if the memory requirement characterization data of the j-th sub-model exceeds the preset memory threshold, it can be determined that the possibility of memory overflow is relatively large when using the j-th sub-model. Therefore, in order to better avoid the occurrence of memory overflow, the target splitting description information matching the j-th sub-model can be first selected from some candidate splitting description information set in advance; then, at least two model segments corresponding to the j-th sub-model can be determined according to the model splitting result of the j-th sub-model under the target splitting description information. This is beneficial to reducing the number of computational reuses as much as possible on the premise of ensuring no memory overflow, and thus is beneficial to improving the response speed as much as possible on the premise of not occurring memory overflow.

[0139] Based on the relevant content of steps 11 to 13 above, in some application scenarios, for any sub-model in the data generation model corresponding to the above data generation request, if the memory requirement characterization data of the sub-model does not exceed the preset memory threshold, it can be determined that there will be no memory overflow when using this sub-model. Therefore, this sub-model can be directly determined as the data processing module. However, if the memory requirement characterization data of the sub-model exceeds the preset memory threshold, it can be determined that there is a relatively high possibility of memory overflow when using this sub-model. Therefore, in order to better avoid the occurrence of memory overflow, the sub-model can be split first to obtain at least two model fragments, so that the memory requirement characterization data of each model fragment does not exceed the preset memory threshold. Then, each model fragment is determined as the data processing module. In this way, it can be effectively ensured that the memory requirement characterization data of all data processing modules split from this data generation model do not exceed the preset memory threshold, so as to effectively ensure that there will be no memory overflow when using these data processing modules to replace the data generation model to implement the corresponding data generation process, and thus it is beneficial to avoid defects caused by memory overflow. In addition, the present application realizes automatically splitting the data generation model into at least two data processing modules by executing steps 11 - 13. This is beneficial to improving the flexibility of model splitting, and thus can effectively avoid defects caused by manual model splitting.

[0140] In addition, for the data generation model corresponding to the above data generation request, the at least two data processing modules split from this data generation model have the following characteristics: the at least two data processing modules are arranged in sequence, so that the data generation process described by this data generation model can be realized by using the at least two data processing modules in sequence according to this arrangement order in the subsequent process. In this way, the function described by this data generation model can be better realized with the help of the at least two data processing modules. It should be noted that the present application does not limit the determination method of the arrangement serial numbers of each data processing module. For example, the arrangement serial numbers of each data processing module can be determined during the determination process of each data processing module.

[0141] Based on the relevant content of at least two data processing modules above, it can be known that in some application scenarios, for the target plug-in constructed according to the data generation model corresponding to the data generation request above, in order to better avoid memory overflow when the data generation processing is implemented with the help of the target plug-in, the target plug-in may include at least two data processing modules arranged in sequence, and the at least two data processing modules are obtained by splitting the data generation model, so that the target plug-in can implement the function described by the data generation model with the help of the at least two data processing modules, such as processing the data generation request to obtain the generated data, etc., so that it can be ensured that there will be no memory overflow when the data generation request is processed with the help of the target plug-in, thereby effectively ensuring the stability of the data generation processing implemented by a single end. Among them, the generated data refers to the data obtained by processing the data generation request; and the present application does not limit the generated data. For example, in some application scenarios, when the data generation request is used to request content generation processing based on reference information, the generated data may include at least one generated image, which refers to the result obtained by performing content generation processing based on the reference information. Example 1, for a text-generated image scene or a picture-generated image scene, the generated data is a generated image. Example 2: For a scene of text-generated video or image-generated video, the generated data is a generated image sequence, such as a generated video.

[0142] In addition, for the target plug-in shown in the previous paragraph, since the target plug-in includes at least two data processing modules arranged in sequence, in order to avoid memory overflow as much as possible, the target plug-in can implement the corresponding data generation process by sequentially loading and releasing each data processing module, so that sufficient memory is available during the use of each data processing module, thereby effectively ensuring that memory overflow does not occur when the target plug-in is used to implement data generation processing. To facilitate understanding of the above content, the following is an example.

[0143] As an example, when the above target plug-in includes at least two data processing modules arranged in sequence, and the at least two data processing modules are obtained by splitting the data generation model corresponding to the above data generation request, in order to avoid memory overflow, the above client can use the target plug-in to process the data generation request to obtain generated data, and the determination process of the generated data can include the following steps 21-24.

[0144] Step 21: Load the i-th data processing module; i is a positive integer, and the initial value of i is 1.

[0145] Here, i refers to the serial number of the data processing module to be used in the current round; and the initial value of i is 1.

[0146] The i-th data processing module refers to the data processing module that exists among the at least two data processing modules above and is in the i-th arrangement position.

[0147] Based on the relevant content of step 21 above, for the current round, the i-th data processing module above is loaded into the memory so that the corresponding data processing process can be implemented using the i-th data processing module that has been loaded into the memory subsequently.

[0148] Step 22: Perform data processing using the i-th data processing module.

[0149] In this application, for the current round, after determining that the i-th data processing module above has been loaded into the memory, it can be determined that the i-th data processing module is in an available state, so the i-th data processing module can be directly used to perform data processing on the input data of the i-th data processing module. Among them, the input data of the i-th data processing module refers to the data input to the i-th data processing module; and this application does not limit the implementation manner of the input data of the i-th data processing module. For example, if i = 1, the input data of the i-th data processing module is determined according to the data generation request above, so that the input data of the i-th data processing module can at least include the reference information carried by the data generation request; if i > 1, the input data of the i-th data processing module is determined according to the output data of the (i - 1)-th data processing module, so that the input data of the i-th data processing module at least includes all or part of the output data of the (i - 1)-th data processing module.

[0150] In addition, in some application scenarios, in order to better improve the data processing effect, some data processing modules need to be initialized before use. Based on this, this application also provides a possible implementation manner of step 22 above. In this implementation manner, step 22 can specifically be: if the i-th data processing module above meets the preset initialization condition, then after determining that the i-th data processing module has been loaded into the memory, perform initialization processing on the i-th data processing module so that the initialized i-th data processing module has a better data processing function; then perform data processing using the initialized i-th data processing module, which is beneficial to improving the data generation effect. Among them, the initialization condition refers to the condition that the model to be initialized reaches; and this application does not limit the implementation manner of the initialization condition. For example, in some application scenarios, the initialization condition can specifically be: the initialization parameters of the i-th data processing module are recorded in the preset storage space of the terminal device.

[0151] Based on the above content, for the current round, after determining that the i-th data processing module in the above text has been loaded into the memory, it is necessary to determine whether the initialization parameters corresponding to the i-th data processing module are recorded in the preset storage space of the terminal device. If the initialization parameters corresponding to the i-th data processing module are recorded, it can be determined that the i-th data processing module meets the initialization conditions, and thus it can be determined that initialization processing still needs to be performed on the i-th data processing module. Furthermore, it can be determined that the i-th data processing module is in an unavailable state. Therefore, the initialization processing is first performed on the i-th data processing module to obtain the initialized i-th data processing module, so that the initialized i-th data processing module is in an available state; then the initialized i-th data processing module is used for data processing, which is beneficial to improving the data processing effect.

[0152] Step 23: After obtaining the output result of the i-th data processing module, release the memory occupied by the i-th data processing module.

[0153] Among them, the output result of the i-th data processing module refers to the result obtained by the i-th data processing module for data processing of the input data of the i-th data processing module.

[0154] Based on the relevant content of Step 23 above, for the current round, after obtaining the output result of the i-th data processing module in the above text, it can be determined that the corresponding data processing process has been completed using the i-th data processing module, and thus it can be determined that the memory occupied by the i-th data processing module can be recycled. Therefore, in order to better avoid memory overflow, the memory occupied by the i-th data processing module can be directly released to reduce the memory occupancy, so as to meet the memory requirements of subsequent modules as much as possible, and thus effectively avoid the occurrence of memory overflow.

[0155] Step 24: Update i, and continue to execute Step 21 above and its subsequent steps until, when the preset stop condition is reached, generate data based on the output result of the i-th data processing module.

[0156] Among them, the present application does not limit the implementation manner of "updating i" above. For example, it can be implemented using the following formula (1).

[0157] i’ = i + 1 (1)

[0158] In the formula, i’ represents the updated value; i represents the value before updating.

[0159] The preset stop condition refers to the condition required to stop the execution of the multi-round iteration process; moreover, the present application does not limit this preset stop condition. For example, the preset stop condition can be: traversing all modules in the at least two data processing modules. Another example is that when the step of "updating i" above is implemented using formula (1) above, the preset stop condition can be: i is equal to the number of modules in the at least two data processing modules above.

[0160] In addition, the present application does not limit the timing for judging the preset stop condition. For example, in some application scenarios, in order to improve the response speed as much as possible, the timing for judging the preset stop condition can be earlier than the execution timing of the step of "updating i" above. Based on this, it can be known that in a possible implementation manner, step 24 above can specifically be: after releasing the memory occupied by the i-th data processing module, determine whether the preset stop condition is met. If the preset stop condition is not met, update i and continue to execute step 21 above and its subsequent steps; however, if the preset stop condition is met, the generated data can be determined based on the output result of the i-th data processing module.

[0161] Furthermore, the present application does not limit the implementation manner of the step of "determining the generated data based on the output result of the i-th data processing module" above. For example, in some application scenarios, it can specifically be: determining the generated data based on the output result of the i-th data processing module, so that the generated data at least includes the output result of the i-th data processing module. Another example is that it can specifically be: directly determining the output result of the i-th data processing module as the generated data. Still another example is that it can specifically be: determining the generated data based on the output results of at least one data processing module, so that the generated data includes the output results of the at least one data processing module, and the at least one data processing module includes the i-th data processing module.

[0162] Based on the relevant content of steps 21 to 24 above, for the target plug-in above, if the target plug-in includes at least two data processing modules arranged in sequence, and the at least two data processing modules are obtained by splitting the data generation model corresponding to the above data generation request, then when the client uses the target plug-in to process the data generation request to obtain the generated data, it can be in a certain manner, such as Figure 4 Or Figure 5 The shown manner is to sequentially use these data processing modules to obtain the generated data, so as to effectively ensure that there is sufficient memory during the use of each data processing module, thereby effectively avoiding defects caused by memory overflow, such as defects like being unable to respond to the data generation request.

[0163] Based on the relevant content of the above-mentioned target plug-in, for this target plug-in, since the target plug-in is constructed according to the data generation model corresponding to the request generated based on the above data, so that the data processing function of the target plug-in includes the data processing function of the data generation model, thus enabling the target plug-in to replace the data generation model to implement corresponding functions; also because the data generation model can be used to process the data generation request, so that the target plug-in can also be used to process the data generation request, so that after the client receives the data generation request, the client can directly use the target plug-in to process the data generation request, and further enabling all relevant processes of the data generation request to occur on the same end, so as to realize a single-end closed-loop for data generation processing, thereby effectively improving the effects presented by the data generation process in terms of real-time performance, security, reliability, etc.

[0164] In addition, this application does not limit the usage mode of the above-mentioned client for the target plug-in, that is, the implementation mode of S2 above. For the convenience of understanding, some situations will be described below.

[0165] Situation 1, in some application scenarios, in order to better improve the response speed of the above-mentioned data generation request, this application also provides a possible implementation mode of S2 above. In this implementation mode, when the target plug-in has been deployed in the client, S2 can specifically be: after the client receives the data generation request, the client directly uses the target plug-in deployed in the client to process the data generation request to obtain generated data.

[0166] It can be seen that in a possible implementation mode, the above-mentioned target plug-in can be directly deployed in the client, so that the client can directly use the target plug-in to process the data generation request, so that the data communication process is hardly involved in the processing process of the data generation request, so as to effectively reduce the time consumed due to data communication, thereby being beneficial to improving the response speed for the data generation request, and further being beneficial to improving the data generation experience.

[0167] Situation 2, in some application scenarios, in order to minimize the impact of plug-in deployment on the performance of the client, this application also provides a possible implementation mode of S2 above. In this implementation mode, when the client is deployed on the terminal device and the above-mentioned target plug-in is also deployed on the terminal device, and the client can communicate with the target plug-in through the plug-in service deployed on the client, S2 can specifically include the following steps 31-step 32.

[0168] Step 31: The client sends a data generation request to the target plug-in deployed on the terminal device through the plug-in service deployed on the client.

[0169] Among them, the plugin service is used to assist the above-mentioned client to call the target plugin; and the plugin service is deployed within the client.

[0170] In addition, the present application does not limit the implementation manner of the above-mentioned plugin service. For example, it can be implemented in any existing or future manner that can assist the client to call the plugin. As an example, in a possible implementation manner, as Figure 3 shown, the plugin service may include a plugin service interface and a plugin host. Among them, the plugin service interface is used to obtain a data generation request triggered by the user and send the data generation request to the plugin host. The plugin host is used to send the data generation request to the target plugin so that the target plugin can process the data generation request. It should be noted that the present application does not limit the implementation manners of the plugin service interface and the plugin host.

[0171] Furthermore, in order to better save the plugin deployment cost, the present application also provides a possible implementation manner of the above-mentioned plugin service. In this implementation manner, one plugin service corresponds to multiple plugins, so that the client can call different plugins through the same plugin service.

[0172] Based on the above content, the present application also provides a possible implementation manner of the determination process of the above-mentioned target plugin. In this implementation manner, when the plugin service deployed on the client corresponds to multiple candidate plugins, the determination process of the target plugin can specifically be: determining the target plugin corresponding to the above-mentioned data generation request from the multiple candidate plugins. Among them, the candidate plugin refers to a plugin that can be called by means of the plugin service; and the corresponding relationship between the multiple candidate plugins and the plugin service can be preset in advance. In addition, the present application does not limit the implementation manners of the multiple candidate plugins. For example, the multiple candidate plugins may include a content generation plugin, a speech recognition plugin, a 3D model construction plugin, etc. The content generation plugin is used to process any content generation request. The speech recognition plugin is used to process any speech recognition request. The 3D model construction plugin is used to process any 3D model construction request.

[0173] Step 32: The client receives the generated data fed back by the target plugin deployed on the terminal device for the data generation request.

[0174] Based on the relevant content of the above steps 31 to 32, in a possible implementation, the above target plugin can be completely independent of the client, and both the target plugin and the client are deployed on the same terminal device, so that the client can call the target plugin by means of the plugin service deployed in the client; and the specific process of this call can be: after the user triggers a data generation request on the client, the client first sends a data generation request to the target plugin deployed on the terminal device through the plugin service deployed on the client, so that the target plugin processes the data generation request to obtain generated data and feeds the generated data back to the client for display. Among them, since both the target plugin and the client are deployed on the same terminal device, all relevant processes of the data generation request occur on the terminal device, so that single-end processing of the data generation request can be achieved; also, since the target plugin is completely independent of the client, the relevant processes of the target plugin are also completely independent of the relevant processes of the client, so that the relevant processes of the target plugin hardly affect the relevant processes of the client, so that single-end processing of the data generation request can be achieved without affecting the performance of the client, which is beneficial to improving the data generation experience without affecting the user experience of the client.

[0175] Based on the relevant content of the above S1 to S2, for the data generation method provided in the embodiments of the present application, after the client receives a data generation request, the client uses the target plugin to process the data generation request to obtain generated data, so that data generation processing can be achieved based on the plugin on the client, which can effectively overcome the defects existing when data generation processing is achieved by means of the cooperation between the client and the server, and is thus beneficial to improving the data generation effect. Among them, since the target plugin is constructed according to the data generation model corresponding to the data generation request, the data processing function of the target plugin includes the data processing function of the data generation model; also, since the data generation model can be used to process the data generation request, the target plugin can also be used to process the data generation request, so that after the client receives the data generation request, the client can directly use the target plugin to process the data generation request, so that all relevant processes of the data generation request occur on the same end, so that single-end closed-loop of data generation processing can be achieved, and thus the effects presented by the data generation process in terms of real-time performance, security, reliability, etc. can be effectively improved.

[0176] In fact, in order to better improve the response speed, the present application also provides a possible implementation manner of the above data generation method. In this implementation manner, the data generation method not only includes the above S1 - S2, but may also include the following step 41. Among them, the execution time of step 41 is earlier than the execution time of S2.

[0177] Step 41: In response to a client startup request, start the client and load the target plugin.

[0178] Among them, the client startup request is used to request to start the client; moreover, the present application does not limit the implementation manner of the client startup request. For example, it can be implemented using any existing or future client startup request.

[0179] Based on the relevant content of the above step 41, in some application scenarios, for the terminal device used by a user, after the terminal device detects that the user has triggered a client startup request, start the client and load the target plugin. In this way, it can be ensured that the target plugin is loaded as early as possible, so that when the user triggers a data generation request on the client later, the client can directly use the already loaded target plugin to process the data generation request and obtain the generated data. In this way, it can effectively improve the response speed for the data generation request, thereby being beneficial to improving the user experience.

[0180] In fact, in some application scenarios, in order to minimize the impact of plugin loading on client startup as much as possible, the present application also provides a possible implementation manner of the above data generation method. In this implementation manner, the data generation method not only includes the above S1 - S2, but may also include the following step 42. Among them, the execution time of step 42 is earlier than the execution time of S2.

[0181] Step 42: In response to a client startup request, start the client and asynchronously pre - load the target plugin.

[0182] It should be noted that the present application does not limit the implementation manner of the asynchronous pre - loading in the above step 42. For example, it can be implemented using any existing or future asynchronous pre - loading method.

[0183] Based on the relevant content of the above step 42, in some application scenarios, for the terminal device used by a user, after the terminal device detects that the user has triggered a client startup request, start the client and asynchronously pre - load the target plugin, so that the loading process of the target plugin does not affect the startup process of the client, thereby being able to load the target plugin as early as possible without affecting the startup speed of the client. This is beneficial to improving the user experience.

[0184] In fact, in some application scenarios, for the above-mentioned target plug-in, such as the target plug-in constructed based on a certain diffusion model, etc., the target plug-in may need to be initialized to improve performance. Based on this, the present application also provides a possible implementation manner of the above-mentioned data generation method. In this implementation manner, the data generation method may at least include step 43 below. Among them, the execution time of step 43 is later than the execution time of the above-mentioned step 41 or the execution time of the above-mentioned step 42.

[0185] Step 43: Initialize the loaded target plug-in so that the subsequent client can use the initialized target plug-in to process the above-mentioned data generation request.

[0186] It should be noted that the present application does not limit the implementation manner of the initialization process in the above-mentioned step 43. For example, it can use the initialization parameters pre-stored for the target plug-in in the terminal device to initialize the loaded target plug-in, so that the initialized target plug-in has better performance.

[0187] Based on the relevant content of the above-mentioned step 43, in some application scenarios, for a terminal device used by a user, after the terminal device detects that the user triggers a client startup request, start the client and load (or asynchronously pre-load) the target plug-in; then initialize the loaded target plug-in to obtain an initialized target plug-in, so that after the client receives the data generation request triggered by the user, the client can directly use the initialized target plug-in to process the data generation request to obtain generated data. Among them, because the initialized target plug-in has better performance, the generated data obtained by using the initialized target plug-in is more accurate, which is beneficial to improving the data generation effect.

[0188] In fact, in order to better improve the diversity of data generation, the present application also provides a possible implementation manner of the above-mentioned target plug-in. In this implementation manner, the target plug-in is constructed based on at least one data generation model. The data generation processes implemented by different data generation models are different. The at least one data generation model includes the data generation model corresponding to the above-mentioned data generation request. Among them, the at least one data generation model refers to the model required to be based on when constructing the target plug-in. Moreover, the present application does not limit the at least one data generation model. For example, in some application scenarios, such as multi-style image generation scenarios or multi-style video generation scenarios, the at least one data generation model may include at least one style content generation model. Moreover, different style content generation models are used to implement content generation processing under different styles, such as image generation processing or video generation processing, etc. Among them, the nth style content generation model is used to implement content generation processing under the nth style, where n is a positive integer, n ≤ N, and N is a positive integer, and N represents the number of models in the at least one style content generation model. In addition, the present application does not limit the implementation manner of the at least one style content generation model. For example, the at least one style content generation model may include Figure 6 the image generation model of style 1, the image generation model of style 2,..., the image generation model of style N shown.

[0189] In addition, in some application scenarios, in order to be able to implement as diverse data generation processes as possible on terminal devices with limited resources, such as image generation processes in N styles, etc. The present application also provides a possible implementation manner of the above-mentioned target plug-in. In this implementation manner, when the target plug-in is constructed based on at least one data generation model, the data generation processes implemented by different data generation models are different, and when the at least one data generation model includes the data generation model corresponding to the above-mentioned data generation request, the target plug-in may include a base model determined based on the at least one data generation model and at least one fine-tuning model, so that the target plug-in can implement multiple data generation processes by means of the base model and the at least one fine-tuning model. Among them, the base model refers to the model determined based on the at least one data generation model and can be used as a base model, so that the base model can present some characteristics common to the multiple data generation processes described by the at least one data generation model, so that the base model can represent the base model required to be used when implementing any data generation processing described by the at least one data generation model by using the target plug-in, such as Figure 6The basic model shown, etc. The nth fine-tuning model refers to the model used to provide parameter fine-tuning information determined according to the nth data generation model, such as the LoRA model, etc., so that the nth fine-tuning model can exhibit some characteristics unique to the nth data generation model, so that the nth data generation model can be obtained by using the nth fine-tuning model to correct the parameters of the basic model subsequently, so that the nth fine-tuning model and the basic model can be used to replace the nth data generation model subsequently; moreover, the implementation manner of the nth fine-tuning model is not limited in this application. For example, the nth fine-tuning model can be implemented using Figure 6 the fine-tuning model of style n shown, where n is a positive integer, n ≤ N, and N is a positive integer, and N represents the number of models in the at least one fine-tuning model. Among them, since the storage resources occupied by the basic model and the at least one fine-tuning model are much smaller than the storage resources occupied by the at least one data generation model, the terminal device used to store the basic model and the at least one fine-tuning model can implement as diverse image generation processing as possible under limited storage resources, which is conducive to improving the diversity of data generation and thus conducive to better meeting the user's data generation needs.

[0190] In addition, for the target plug-in shown in the above paragraph, when using the target plug-in to implement a certain data generation process, in order to better avoid the occurrence of memory overflow, the basic model in the target plug-in and the fine-tuning model corresponding to the data generation process can be loaded at different time points, so that the memory pressure caused by loading the basic model and the fine-tuning model at the same time can be effectively avoided, and thus the defect caused by the large amount of memory required when directly loading the data generation model used to implement the data generation process, such as Figure 6 the image generation model of style 1 shown, etc., can be effectively overcome, which is conducive to better avoiding the occurrence of memory overflow and thus conducive to improving the stability of data generation processing. For better understanding, a possible implementation manner of the above data generation method is used as an example for description below.

[0191] As an example, in a possible implementation manner, when the above target plug-in includes a basic model and at least one fine-tuning model, the basic model and the at least one fine-tuning model are determined according to at least one data generation model, the data generation processes implemented by different data generation models are different, and the at least one data generation model includes the data generation model corresponding to the above data generation request, the data generation method provided in this application may include the following steps 51-step 54.

[0192] Step 51: In response to the client startup request, start the client and load (or asynchronously pre-load) the basic model in the target plug-in.

[0193] Step 52: After the client receives a data generation request, determine the fine-tuning model corresponding to the data generation request from at least one fine-tuning model of the target plugin.

[0194] It should be noted that the present application does not limit the implementation manner of step 52 above. For example, when the above data generation request carries model description information, step 52 may specifically be: match the model description information carried in the data generation request with the model description information of each fine-tuning model in the target plugin, and determine the fine-tuning model that best matches the model description information carried in the data generation request as the fine-tuning model corresponding to the data generation request.

[0195] It should also be noted that the present application does not limit the execution entity of the step of "determining the fine-tuning model corresponding to the data generation request from at least one fine-tuning model of the target plugin" in step 52 above. For example, the execution entity of this step may be the client or the target plugin.

[0196] Step 53: Use the fine-tuning model corresponding to the above data generation request to perform parameter correction processing on the loaded base model to obtain the data generation model corresponding to the data generation request.

[0197] It should be noted that the present application does not limit the implementation manner of step 53 above. For example, it may be implemented using any existing or future method that can perform parameter correction processing on a base model according to a fine-tuning model, such as the LoRA model, etc.

[0198] It should be noted that the present application does not limit the execution entity of step 53 above. For example, the execution entity of step 53 may be the client or the target plugin.

[0199] Step 54: The client uses the data generation model corresponding to the above data generation request to process the data generation request.

[0200] Based on the relevant content of steps 51 to 54 above, it can be seen that in some application scenarios, the target plugin can implement various data generation processes by means of a base model and some fine-tuning models. In this way, it is possible to achieve as diverse data generation processes as possible on resource-limited terminal devices, thereby effectively overcoming the defects caused by limited resources of terminal devices, and further being beneficial to improving data generation diversity, and thus being beneficial to improving the user experience.

[0201] Actually, in some application scenarios, in order to better improve the user experience, the present application also provides a possible implementation manner of the above data generation method. In this implementation manner, the data generation method at least includes the following steps 61 - 63.

[0202] Step 61: The client receives a data generation request.

[0203] It should be noted that for the relevant content of Step 61, please refer to S1 above.

[0204] Step 62: The client determines whether the target plugin is in an available state.

[0205] In this application, for the client, after the client receives a data generation request triggered by the user, the client can determine whether the target plugin corresponding to the data generation request is in an available state. If the target plugin is in an available state, it can be determined that the target plugin can immediately process the data generation request, so the client can directly use the target plugin to process the data generation request; however, if the target plugin is in an unavailable state, it can be determined that the target plugin cannot immediately (or even cannot) process the data generation request, so the client can use other methods, such as the method shown in Step 64 below, to complete the processing process of the data generation request.

[0206] It should be noted that this application does not limit the implementation manner of the above step of "determining whether the target plugin is in an available state". For example, specifically, if the client determines that the target plugin is in the deployment stage, such as the loading stage or the initialization stage, etc., the client can determine that the target plugin is in an unavailable state; however, if the client determines that the target plugin has completed the entire deployment stage, the client can determine that the target plugin is in an available state. Among them, the deployment stage refers to the stage of deploying the target plugin to the client or the terminal device; and this application does not limit the deployment stage. For example, the deployment stage can include the loading stage and the initialization stage.

[0207] Step 63: If the client determines that the target plugin is in an available state, the client uses the target plugin to process the above data generation request to obtain generated data.

[0208] Based on the relevant content of the above Steps 61 to 63, in some application scenarios, for the above client, after the client receives a data generation request triggered by the user, the client can determine whether the target plugin corresponding to the data generation request is in an available state, so that when it is determined that the target plugin is in an available state, the client can directly use the target plugin to process the data generation request, which is beneficial to improving the response speed.

[0209] In fact, in some application scenarios, in order to better improve the response speed, the present application also provides a possible implementation manner of the above data generation method. In this implementation manner, the data generation method may at least include the above steps 61 - step 63, and the following steps 64 - step 65. Among them, the execution time of step 64 is later than that of step 62.

[0210] Step 64: If the client determines that the target plug-in is in an unavailable state, the client sends the above data generation request to the server, and the server is used to process the data generation request by using the data generation model deployed on the server to obtain generated data.

[0211] Among them, the data generation model deployed on the server refers to the data generation model corresponding to the above data generation request, so that the data generation model deployed on the server can be used to construct the above target plug-in; and for the relevant content of this data generation model, please refer to the above.

[0212] Based on the relevant content of the above step 64, for the above client, after the client receives the data generation request triggered by the user, if the client determines that the target plug-in corresponding to the data generation request is in an unavailable state, it can be determined that the client cannot complete the processing process of the data generation request with the help of the target plug-in. Therefore, in order to improve the response speed as much as possible, the client can directly send the data generation request to the server, so that the server can use the data generation model that has been deployed on the server to process the data generation request, obtain the generated data, and feedback the generated data to the client for display, so that the client can display the generated data to the user in the shortest possible time. This is beneficial to improving the response speed and thus beneficial to improving the user experience.

[0213] It should be noted that the present application does not limit the association relationship between the model involved in the above target plug-in and the data generation model deployed on the server. For example, the model involved in the target plug-in is determined based on the data generation model deployed on the server, so that the model involved in the target plug-in and the data generation model deployed on the server have the same function, so that the target plug-in can realize the function described by the data generation model deployed on the server with the help of the model involved in the target plug-in, and further the target plug-in can replace the data generation model deployed on the server to realize the corresponding function.

[0214] It should also be noted that the present application does not limit the implementation manner of the above server. For example, it can be implemented by using any existing or future server, such as an independent server, a cluster server, or a cloud server, etc.

[0215] Step 65: The client receives the generated data fed back by the server.

[0216] Based on the relevant content of the above steps 61 to 65, in some application scenarios, for the above client, after the client receives the data generation request triggered by the user, the client can determine whether the target plug-in corresponding to the data generation request is in an available state. If the target plug-in is in an available state, it can be determined that the target plug-in can immediately process the data generation request. Therefore, the client can directly use the target plug-in to process the data generation request. However, if the target plug-in is in an unavailable state, it can be determined that the target plug-in cannot immediately (or even cannot) process the data generation request. Therefore, the client can directly send the data generation request to the server, so that the client can complete the processing process of the data generation request with the corresponding model deployed on the server, obtain and display the generated data fed back by the server. This is beneficial to improving the response speed and thus beneficial to improving the user experience.

[0217] In fact, in some application scenarios, there may be various reasons for the target plug-in to be in an unavailable state. For example, the target plug-in is in the deployment stage or the relevant resources of the target plug-in do not exist in the terminal device. Therefore, in order to better improve the data generation effect, the present application also provides a possible implementation manner of the above data generation method. In this implementation manner, when the above client is deployed on the terminal device, the data generation method may at least include the following steps 71-step 73.

[0218] Step 71: If the client determines that the target plug-in is in an unavailable state, the client determines whether the plug-in description resources of the target plug-in are stored in the terminal device.

[0219] Among them, the plug-in description resources of the target plug-in refer to the resources required when deploying the target plug-in on the client or the terminal device. Moreover, the present application does not limit the implementation manner of the plug-in description resources of the target plug-in. For example, the plug-in description resources of the target plug-in may include loading stage resources and initialization stage resources. The loading stage resources refer to the resources required for loading the target plug-in. The initialization stage resources refer to the resources required for initializing and processing the target plug-in.

[0220] Step 72: If the client determines that the plug-in description resources of the target plug-in are not stored in the terminal device, the client sends a resource requirement request to the server.

[0221] Among them, the resource requirement request is used to request the server to provide the plug-in description resources of the target plug-in. Moreover, the present application does not limit the implementation manner of the resource requirement request. For example, the resource requirement request may adopt Figure 2Implement the plug-in download request shown above.

[0222] Based on the relevant content of step 72 above, for the client deployed on the terminal device, after the client receives the data generation request triggered by the user, if the client determines that the target plug-in corresponding to the data generation request is in an unavailable state and determines that there is no plug-in description resource of the target plug-in stored in the terminal device, it can be determined that the client receives for the first time a data generation requirement that is the same as or similar to the data generation requirement described in the data generation request. Thus, it can be determined that the client does not yet have the ability to independently process the data generation requirement. Therefore, the client can directly send a resource requirement request to the server. After the server receives the resource requirement request, the server can feedback the plug-in description resource of the target plug-in determined according to the data generation model in the server to the client, so that the client can complete the deployment process of the target plug-in by using the plug-in description resource, so that the client can use the deployed target plug-in to process subsequent triggered data generation requirements that are the same as or similar to the data generation requirement described in the data generation request. It should be noted that this application does not limit the implementation manner of the deployment process of the target plug-in. For example, it may include the loading process and the initialization process of the target plug-in.

[0223] Step 73: The client receives and stores the plug-in description resource feedback by the server for the above resource requirement request; the plug-in description resource is determined by the server according to the data generation model deployed on the server.

[0224] Based on the relevant content of steps 71 to 73 above, in some application scenarios, for the client deployed on the above terminal device, when the client receives a data generation request, if the client has never processed a data generation requirement that is the same as or similar to the data generation requirement described in the data generation request, the client not only needs to rely on the server to complete the processing process of the data generation request, but also needs to obtain the relevant resources of the target plug-in corresponding to the data generation request from the server, so that after the target plug-in is deployed based on these resources on the client or the terminal device, the client can use the deployed target plug-in to process subsequent received data generation requirements that are the same as or similar to the data generation requirement described in the data generation request, etc. In this way, on-demand plug-in deployment can be realized, thus effectively avoiding resource waste caused by deploying a large number of plug-ins that users do not need on the terminal device, and further enabling more efficient use of the limited resources in the terminal device.

[0225] Based on the data generation method described above, the present application provides a technical solution for implementing data generation processing, such as content generation processing, on a single end based on a plug-in, and this technical solution has the advantages shown in (i) to (viii) below.

[0226] (i) For the target plug-in described above, the target plug-in implements data processing by using some lightweight algorithms, so as to overcome to a certain extent the defects caused by the relatively large data generation model required for constructing the target plug-in, such as the inability to deploy the target plug-in on a terminal device or the inability to run the model involved in the target plug-in on a terminal device.

[0227] (ii) For the target plug-in described above, the model involved in the target plug-in is determined by the server by extracting the core logic from the data generation model deployed on the server, so as to effectively overcome the defects caused by the relatively large data generation model required for constructing the target plug-in.

[0228] (iii) For the target plug-in described above, the model involved in the target plug-in is obtained by the server by converting the data generation model deployed on the server from one expression form to another, so that the expression form of the model involved in the target plug-in is more compatible with the operating environment of the terminal device, thus effectively solving the defects caused by the different operating environments of the server and the terminal device. Among them, the former expression form refers to the expression form required when the server runs the data generation model deployed on the server, such as Python and other ways. The latter expression form refers to the expression form required when the terminal device runs the model involved in the target plug-in, such as pure C++ and other ways.

[0229] (iv) For the target plug-in described above, heterogeneous computing methods can be used to implement the computationally intensive parts involved in the use of the target plug-in, which is conducive to giving full play to the computing resources on the terminal device, such as the computing resources of chips such as graphics processing unit (GPU), neural processing unit (NPU), and digital signal processing (DSP).

[0230] (5) In this application, by plugging in models such as AIGC models, the model does not occupy the installation package space of the client, so that users can download and dynamically load the model on demand on the client. In this way, the problem of the model volume can be effectively overcome. For example, since the model deployed on the server is relatively large, when the model is deployed along with the installation package of the client, the installation package volume will be relatively large, thus affecting the user's installation package download experience and the client's update experience.

[0231] (6) In this application, by splitting the large model into a series of small models and loading the currently needed small model according to the configuration and immediately destroying it after the small model runs to free up enough memory for the subsequent operation of the small model, the occurrence of memory overflow is avoided. This is beneficial to improving memory stability.

[0232] (7) This application not only adopts the technical solution of realizing data generation processing by the client with the help of plugins, but also adopts the collaborative method between the client and the server to realize data generation processing when the first data processing or the plugin is not completely deployed. In this way, the response speed can be effectively improved, which is beneficial to improving the user experience.

[0233] (8) This application adopts a basic model and multiple fine-tuned models, such as multiple LoRA small models, to realize various data generation processes. In this way, the problem that it is difficult to expand the richness of the data processing process due to limited resources of the terminal device can be effectively solved, which is beneficial to improving the diversity of data generation processing and further beneficial to improving the user experience.

[0234] Based on the data generation method provided by the embodiments of this application, the embodiments of this application also provide a data generation device. The following will be combined with Figure 7 for explanation and illustration. Among them, Figure 7 is a schematic structural diagram of a data generation device provided by the embodiments of this application. It should be noted that for the technical details of the data generation device provided by the embodiments of this application, please refer to the relevant content of the above data generation method.

[0235] As Figure 7 shown, the data generation device 700 provided by the embodiments of this application includes:

[0236] A receiving unit 701, configured to receive a data generation request;

[0237] A processing unit 702, configured to process the data generation request by using a target plugin to obtain generated data; the target plugin is constructed according to the data generation model corresponding to the data generation request, and the data generation model is used to process the data generation request.

[0238] In a possible implementation manner, the data generation request is used to request content generation processing based on reference information; the reference information includes at least one of text and image; the data generation model is a content generation model; the generated data includes at least one generated image.

[0239] In a possible implementation manner, the client is deployed on a terminal device;

[0240] The processing unit 702 is specifically configured to: send the data generation request to a target plugin deployed on the terminal device through a plugin service deployed on the client; receive the generated data fed back by the target plugin for the data generation request.

[0241] In a possible implementation manner, the target plugin includes a target model determined according to the data generation model; the target plugin is used to process the data generation request by using the target model to obtain the generated data.

[0242] In a possible implementation manner, the target plugin includes at least two candidate models, and the data processing processes implemented by different candidate models are different. The at least two candidate models include the target model; the target plugin is further used to determine the target model corresponding to the data generation request from the at least two candidate models.

[0243] In a possible implementation manner, the data generation device 700 further includes:

[0244] A first response unit, configured to start the client and load the target plugin in response to a client start request.

[0245] In a possible implementation manner, the data generation device 700 further includes:

[0246] An initialization unit, configured to perform initialization processing on the loaded target plugin;

[0247] The processing unit 702 is specifically configured to: process the data generation request by using the initialized target plugin.

[0248] In a possible implementation manner, the first response unit is specifically configured to: start the client and asynchronously preload the target plugin in response to a client start request.

[0249] In a possible implementation manner, the target plugin is constructed according to at least one data generation model, and the data generation processes implemented by different data generation models are different. The at least one data generation model includes the data generation model corresponding to the data generation request.

[0250] In a possible implementation, the target plug-in includes a base model and at least one fine-tuning model; the base model and the at least one fine-tuning model are determined according to the at least one data generation model;

[0251] The data generation device 700 further includes:

[0252] A second response unit, configured to respond to a client startup request, start the client, and load the base model in the target plug-in;

[0253] The processing unit 702 is specifically configured to: determine the fine-tuning model corresponding to the data generation request from the at least one fine-tuning model; use the fine-tuning model to perform parameter correction processing on the loaded base model to obtain the target model; and use the target model to process the data generation request.

[0254] In a possible implementation, the data generation device 700 further includes:

[0255] A first judgment unit, configured to judge whether the target plug-in is in an available state;

[0256] The processing unit 702 is specifically configured to: if it is determined that the target plug-in is in an available state, use the target plug-in to process the data generation request.

[0257] In a possible implementation, the data generation device 700 further includes:

[0258] A first sending unit, configured to, if it is determined that the target plug-in is in an unavailable state, send the data generation request to a server, and the server is configured to use a data generation model deployed on the server to process the data generation request to obtain the generated data;

[0259] A first receiving unit, configured to receive the generated data fed back by the server.

[0260] In a possible implementation, the client is deployed on a terminal device;

[0261] The data generation device 700 further includes:

[0262] A second judgment unit, configured to, if it is determined that the target plug-in is in an unavailable state, judge whether a plug-in description resource of the target plug-in is stored in the terminal device;

[0263] A second sending unit, configured to, if it is determined that the terminal device does not store the plug-in description resource of the target plug-in, send a resource requirement request to the server;

[0264] A second receiving unit, configured to receive and store the plug-in description resources fed back by the server for the resource requirement request; the plug-in description resources are determined by the server according to a data generation model deployed on the server.

[0265] In a possible implementation manner, the target plug-in includes at least two data processing modules arranged in sequence, and the at least two data processing modules are obtained by splitting the data generation model.

[0266] The processing unit 702 is specifically configured to:

[0267] Load the i-th data processing module; i is a positive integer, and the initial value of i is 1.

[0268] Perform data processing by using the i-th data processing module.

[0269] After obtaining the output result of the i-th data processing module, release the memory occupied by the i-th data processing module.

[0270] Update the i, and continue to execute the step of loading the i-th data processing module until, when a preset stop condition is reached, determine the generated data according to the output result of the i-th data processing module.

[0271] In a possible implementation manner, the splitting process of the data generation model includes: splitting the data generation model into at least two sub-models, and different sub-models implement different data processing functions; for any one of the sub-models, if the data representing the memory requirement of the sub-model does not exceed a preset memory threshold, then determine the sub-model as the data processing module; if the data representing the memory requirement of the sub-model exceeds the preset memory threshold, then split the sub-model into at least two model segments, and determine each of the model segments as the data processing module, and the data representing the memory requirement of each of the model segments does not exceed the preset memory threshold.

[0272] In a possible implementation manner, the process of obtaining the at least two model segments includes: obtaining at least one candidate splitting description information, where the model splitting manners described by different candidate splitting description information are different; for any one of the candidate splitting description information, splitting the sub-model according to the candidate splitting description information to obtain a model splitting result corresponding to the candidate splitting description information, and determining resource usage characterization data corresponding to the candidate splitting description information based on the model splitting result corresponding to the candidate splitting description information, where the resource usage characterization data includes computing resource reuse characterization data and memory usage characterization data; selecting target splitting description information from the at least one candidate splitting description information based on the resource usage characterization data corresponding to each candidate splitting description information, where the resource usage balance degree presented by the resource usage characterization data corresponding to the target splitting description information is higher than the resource usage balance degree presented by the resource usage characterization data corresponding to any other candidate splitting description information except the target splitting description information in the at least one candidate splitting description information; and determining the at least two model segments based on the model splitting result corresponding to the target splitting description information.

[0273] In a possible implementation manner, the client includes the data generation device 700.

[0274] Based on the relevant content of the above data generation device 700, for the data generation device 700 provided in the embodiments of the present application, after the data generation device 700 receives a data generation request, the data generation device 700 uses a target plug-in to process the data generation request to obtain generated data. In this way, data generation processing based on the plug-in can be realized on the data generation device 700, thereby effectively overcoming the defects existing when data generation processing is realized by means of multi-terminal collaboration, and further being beneficial to improving the data generation effect. Among them, since the target plug-in is constructed according to the data generation model corresponding to the data generation request, the data processing function of the target plug-in includes the data processing function of the data generation model; also, since the data generation model can be used to process the data generation request, the target plug-in can also be used to process the data generation request. Therefore, after the data generation device 700 receives the data generation request, the data generation device 700 can directly use the target plug-in to process the data generation request, and then all relevant processes of the data generation request occur on the same end. In this way, single-end closed-loop of data generation processing can be realized, thereby effectively improving the effects presented by the data generation process in terms of real-time performance, security, reliability, etc.

[0275] In addition, an embodiment of the present application further provides an electronic device, which includes a processor and a memory: the memory is used to store instructions or computer programs; the processor is used to execute the instructions or computer programs in the memory so that the electronic device executes any implementation manner of the data generation method provided in the embodiment of the present application.

[0276] See Figure 8 , which shows a schematic structural diagram of an electronic device 800 suitable for implementing the embodiments of the present disclosure. The terminal devices in the embodiments of the present disclosure may include, but are not limited to, mobile terminals such as mobile phones, laptop computers, digital broadcast receivers, PDAs (Personal Digital Assistants), PADs (Tablet Computers), PMPs (Portable Multimedia Players), in-vehicle terminals (such as in-vehicle navigation terminals), etc., and fixed terminals such as digital TVs, desktop computers, etc. Figure 8 The electronic device shown is only an example and should not impose any limitation on the functions and usage scope of the embodiments of the present disclosure.

[0277] As Figure 8 shown, the electronic device 800 may include a processing device (such as a central processing unit, a graphics processing unit, etc.) 801, which may execute various appropriate actions and processes according to the programs stored in the read-only memory (ROM) 802 or the programs loaded from the storage device 808 into the random access memory (RAM) 803. In the RAM 803, various programs and data required for the operation of the electronic device 800 are also stored. The processing device 801, the ROM 802, and the RAM 803 are connected to each other through a bus 804. The input / output (I / O) interface 805 is also connected to the bus 804.

[0278] Generally, the following devices may be connected to the I / O interface 805: an input device 806 including, for example, a touch screen, a touchpad, a keyboard, a mouse, a camera, a microphone, an accelerometer, a gyroscope, etc.; an output device 807 including, for example, a liquid crystal display (LCD), a speaker, a vibrator, etc.; a storage device 808 including, for example, a magnetic tape, a hard disk, etc.; and a communication device 809. The communication device 809 may allow the electronic device 800 to communicate with other devices wirelessly or wiredly to exchange data. Although Figure 8 shows the electronic device 800 having various devices, it should be understood that it is not required to implement or have all the shown devices. Instead, more or fewer devices may be implemented or had.

[0279] In particular, according to an embodiment of the present disclosure, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, an embodiment of the present disclosure includes a computer program product that includes a computer program carried on a non-transitory computer-readable medium, and the computer program includes program code for performing the methods shown in the flowcharts. In such an embodiment, the computer program can be downloaded and installed from the network via the communication device 809, or installed from the storage device 808, or installed from the ROM 802. When the computer program is executed by the processing device 801, the above-mentioned functions defined in the method of the embodiment of the present disclosure are performed.

[0280] The electronic device provided by the embodiment of the present disclosure and the method provided by the above embodiment belong to the same inventive concept. Technical details not described in detail in this embodiment can be referred to the above embodiment, and this embodiment has the same beneficial effects as the above embodiment.

[0281] An embodiment of the present application also provides a computer-readable medium, in which instructions or a computer program are stored. When the instructions or the computer program run on a device, the device is caused to execute any implementation manner of the data generation method provided by the embodiment of the present application.

[0282] It should be noted that the computer-readable medium described above can be a computer-readable signal medium, a computer-readable storage medium, or any combination of the two. A computer-readable storage medium can be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination of the above. More specific examples of the computer-readable storage medium can include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the present disclosure, the computer-readable storage medium can be any tangible medium that contains or stores a program, and this program can be used by or in combination with an instruction execution system, apparatus, or device. In the present disclosure, a computer-readable signal medium can include a data signal propagated in a baseband or as part of a carrier wave, which carries computer-readable program code. Such a propagated data signal can take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. A computer-readable signal medium can also be any computer-readable medium other than a computer-readable storage medium, and this computer-readable signal medium can send, propagate, or transmit a program for use by or in combination with an instruction execution system, apparatus, or device. The program code contained on the computer-readable medium can be transmitted by any appropriate medium, including but not limited to: wires, optical cables, RF (radio frequency), etc., or any suitable combination of the above.

[0283] In some embodiments, the client and the server can communicate using any currently known or future-developed network protocol such as HTTP (Hyper Text Transfer Protocol), and can be interconnected with digital data communication in any form or medium (e.g., a communication network). Examples of communication networks include local area networks ("LANs"), wide area networks ("WANs"), the Internet (e.g., the Internet), and end-to-end networks (e.g., ad hoc end-to-end networks), as well as any currently known or future-developed networks.

[0284] The above computer-readable medium can be included in the above electronic device; or it can exist separately without being assembled into the electronic device.

[0285] The above computer-readable medium carries one or more programs, and when the above one or more programs are executed by the electronic device, the electronic device can execute the above method.

[0286] Computer program code for performing the operations of this disclosure can be written in one or more programming languages or combinations thereof. The programming languages include, but are not limited to, object-oriented programming languages such as Java, Smalltalk, C++, and also include conventional procedural programming languages such as the "C" language or similar programming languages. The program code can be executed entirely on the user's computer, partially on the user's computer, executed as a stand-alone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the case of a remote computer, the remote computer can be connected to the user's computer through any kind of network, including a local area network (LAN) or a wide area network (WAN), or can be connected to an external computer (e.g., through the Internet using an Internet service provider).

[0287] The flowcharts and block diagrams in the accompanying drawings illustrate the possible architectures, functions, and operations of systems, methods, and computer program products according to various embodiments of this disclosure. In this regard, each block in the flowchart or block diagram can represent a module, a program segment, or a part of code that contains one or more executable instructions for implementing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the blocks can occur in a different order than marked in the accompanying drawings. For example, two consecutive blocks shown can actually be executed substantially in parallel, and they can sometimes be executed in the reverse order, depending on the functions involved. It should also be noted that each block in the block diagram and / or flowchart, and combinations of blocks in the block diagram and / or flowchart, can be implemented by a dedicated hardware-based system for performing the specified functions or operations, or can be implemented by a combination of dedicated hardware and computer instructions.

[0288] The units involved in the embodiments described in this disclosure can be implemented in software or in hardware. Among them, the name of the unit / module does not constitute a limitation to the unit itself in some cases.

[0289] The functions described above in this document can be performed at least in part by one or more hardware logic components. For example, by way of non-limitation, exemplary types of hardware logic components that can be used include: field programmable gate arrays (FPGA), application specific integrated circuits (ASIC), application specific standard products (ASSP), system on a chip (SOC), complex programmable logic devices (CPLD), and so on.

[0290] In the context of the present disclosure, a machine-readable medium may be a tangible medium that can contain or store a program for use by or in connection with an instruction execution system, apparatus, or device. The machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. The machine-readable medium may include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. More specific examples of the machine-readable storage medium would include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fibers, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0291] It should be noted that the various embodiments in this specification are described in a progressive manner, with each embodiment focusing on the differences from other embodiments. The same or similar parts among the various embodiments can be referred to each other. For the systems or apparatuses disclosed in the embodiments, since they correspond to the methods disclosed in the embodiments, the descriptions are relatively simple, and the relevant parts can be referred to the descriptions in the method section.

[0292] It should be understood that in this application, "at least one (item)" means one or more, and "a plurality" means two or more. "And / or" is used to describe the association relationship of associated objects, indicating that three relationships can exist. For example, "A and / or B" can mean: only A exists, only B exists, and both A and B exist at the same time. Here, A and B can be singular or plural. The character " / " generally indicates an "or" relationship between the associated objects before and after. "At least one (one) of the following" or its similar expressions refer to any combination of these items, including any combination of single item (one) or plural items (ones). For example, at least one (one) of a, b, or c can mean: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, and c can be single or multiple.

[0293] It should also be noted that in this text, relational terms such as first and second are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any actual relationship or order between these entities or operations. Moreover, the terms "comprising", "including" or any other variant thereof are intended to cover non-exclusive inclusion, such that a process, method, article or device comprising a series of elements not only includes those elements but also includes other elements not expressly listed, or elements inherent to such process, method, article or device. Without further limitation, an element defined by the phrase "comprising a..." does not exclude the presence of additional identical elements in the process, method, article or device comprising said element.

[0294] The steps of the methods or algorithms described in connection with the embodiments disclosed herein may be implemented directly in hardware, in a software module executed by a processor, or in a combination thereof. The software module may be placed in a random access memory (RAM), internal memory, read-only memory (ROM), electrically programmable ROM, electrically erasable programmable ROM, registers, hard disk, removable disk, CD-ROM, or any other form of storage medium known in the art.

[0295] The foregoing description of the disclosed embodiments enables those skilled in the art to make or use the present application. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present application. Thus, the present application is not intended to be limited to the embodiments shown herein but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A data generation method, characterized in that, the method is applied to a client, and the method includes: receiving a data generation request; processing the data generation request by using a target plug-in to obtain generated data; the target plug-in is constructed according to a data generation model corresponding to the data generation request, and the data generation model is used to process the data generation request.

2. The method according to claim 1, characterized in that, the data generation request is used to request content generation processing based on reference information; the reference information includes at least one of text and image; the data generation model is a content generation model; the generated data includes at least one generated image.

3. The method according to claim 1, characterized in that, the client is deployed on a terminal device; the processing the data generation request by using a target plug-in to obtain generated data includes: sending the data generation request to a target plug-in deployed on the terminal device through a plug-in service deployed on the client; receiving the generated data fed back by the target plug-in for the data generation request.

4. The method according to claim 1, characterized in that, the target plug-in includes a target model determined according to the data generation model; the target plug-in is used to process the data generation request by using the target model to obtain the generated data.

5. The method according to claim 4, characterized in that, the target plug-in includes at least two candidate models, and the data processing processes implemented by different candidate models are different, and the at least two candidate models include the target model; the target plug-in is further used to determine the target model corresponding to the data generation request from the at least two candidate models.

6. The method according to claim 1, characterized in that, before processing the data generation request by using a target plug-in, the method further includes: responding to a client startup request, starting the client, and loading the target plug-in.

7. The method according to claim 6, characterized in that, after loading the target plug-in, the method further includes: performing initialization processing on the loaded target plug-in; the processing the data generation request by using a target plug-in includes: processing the data generation request by using the initialized target plug-in.

8. The method according to claim 6, characterized in that, the loading the target plug-in includes: asynchronously preloading the target plug-in.

9. The method according to claim 1, characterized in that, the target plug-in is constructed according to at least one data generation model, and the data generation processes implemented by different data generation models are different, and the at least one data generation model includes the data generation model corresponding to the data generation request.

10. The method according to claim 9, characterized in that, the target plug-in includes a base model and at least one fine-tuning model; the base model and the at least one fine-tuning model are determined according to the at least one data generation model; before processing the data generation request by using a target plug-in, the method further includes: In response to a client startup request, start the client and load the base model in the target plugin; The process of obtaining the generated data includes: Determine the fine-tuning model corresponding to the data generation request from the at least one fine-tuning model; Use the fine-tuning model to perform parameter correction processing on the loaded base model to obtain the target model; Use the target model to process the data generation request.

11. The method according to claim 1, wherein, after receiving the data generation request, the method further includes: Determine whether the target plugin is in an available state; The process of using the target plugin to process the data generation request includes: If it is determined that the target plugin is in an available state, use the target plugin to process the data generation request.

12. The method according to claim 11, wherein, after determining whether the target plugin is in an available state, the method further includes: If it is determined that the target plugin is in an unavailable state, send the data generation request to the server, and the server is used to process the data generation request by using the data generation model deployed on the server to obtain the generated data; Receive the generated data fed back by the server.

13. The method according to claim 11, wherein, The client is deployed on a terminal device; after determining whether the target plugin is in an available state, the method further includes: If it is determined that the target plugin is in an unavailable state, determine whether the plugin description resource of the target plugin is stored in the terminal device; If it is determined that the plugin description resource of the target plugin is not stored in the terminal device, send a resource requirement request to the server; Receive and store the plugin description resource fed back by the server for the resource requirement request; the plugin description resource is determined by the server according to the data generation model deployed on the server.

14. The method according to claim 1, wherein, The target plugin includes at least two data processing modules arranged in sequence, and the at least two data processing modules are obtained by splitting the data generation model; The process of obtaining the generated data includes: Load the i-th data processing module; i is a positive integer, and the initial value of i is 1; Use the i-th data processing module to perform data processing; After obtaining the output result of the i-th data processing module, release the memory occupied by the i-th data processing module; Update i and continue to execute the step of loading the i-th data processing module until a preset stop condition is reached, and determine the generated data according to the output result of the i-th data processing module.

15. The method according to claim 14, wherein, The process of splitting the data generation model includes: Split the data generation model into at least two sub-models, and the data processing functions implemented by different sub-models are different; For any of the sub-models, if the memory requirement characterization data of the sub-model does not exceed a preset memory threshold, then determine the sub-model as the data processing module; if the memory requirement characterization data of the sub-model exceeds the preset memory threshold, then split the sub-model into at least two model segments, and determine each of the model segments as the data processing module, and the memory requirement characterization data of each of the model segments does not exceed the preset memory threshold.

16. The method according to claim 15, wherein, the process of obtaining the at least two model segments includes: obtaining at least one candidate splitting description information, and different candidate splitting description information describes different model splitting methods; For any of the candidate splitting description information, split the sub-model according to the candidate splitting description information to obtain a model splitting result corresponding to the candidate splitting description information, and determine resource usage characterization data corresponding to the candidate splitting description information according to the model splitting result corresponding to the candidate splitting description information, where the resource usage characterization data includes computing resource reuse characterization data and memory usage characterization data; According to the resource usage characterization data corresponding to each of the candidate splitting description information, select target splitting description information from the at least one candidate splitting description information, and the resource usage balance degree presented by the resource usage characterization data corresponding to the target splitting description information is higher than that presented by the resource usage characterization data corresponding to any other candidate splitting description information except the target splitting description information among the at least one candidate splitting description information; Determine the at least two model segments according to the model splitting result corresponding to the target splitting description information.

17. A data generation device, wherein, it includes: a receiving unit, configured to receive a data generation request; a processing unit, configured to process the data generation request by using a target plug-in to obtain generated data; The target plug-in is constructed according to a data generation model corresponding to the data generation request, and the data generation model is used to process the data generation request.

18. An electronic device, wherein, the device includes: a processor and a memory; the memory is configured to store instructions or computer programs; the processor is configured to execute the instructions or computer programs in the memory so that the electronic device executes the method according to any one of claims 1-16.

19. A computer-readable medium, wherein, instructions or computer programs are stored in the computer-readable medium, and when the instructions or computer programs run on a device, the device executes the method according to any one of claims 1-16.

Images