A scheduling method, device, equipment and storage medium for a multimodal engine

Abstract

The embodiment of the present application provides a scheduling method, device, equipment and storage medium of a multimodal engine, any multimodal engine is deployed on a cluster server, and completes a Wensheng map task in a single-threaded mode, the method is: obtain a task request, the task request includes a Wensheng map prompt word; determine the complexity of the Wensheng map prompt word according to the preset Wensheng map description dimension; determine the target server matching the complexity based on the status information of any server in the cluster server; send the task request to the target server, and complete the task request through the multimodal engine of the target server. By matching the server corresponding to the complexity, the server is used as the multimodal engine of the target server to process the Wensheng map task request, so that the server processing the Wensheng map task request will neither fail to complete the Wensheng map task due to low configuration nor waste server resources due to high configuration.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence technology, and in particular to a scheduling method, device, equipment and storage medium for a multimodal engine. Background Art

[0002] The text map is a technology based on the diffusion model, which can generate corresponding images according to the input text map prompt words. The underlying layer of the diffusion model text map relies on the multimodal engine. When multiple users initiate text map task requests, when calling the multimodal engine, the traditional method uses polling to match the task requests that call the multimodal engine with the server where the multimodal engine is located in turn.

[0003] However, the traditional method of calling task requests will query all servers where multimodal engines are located, and determine whether the servers meet the requirements for executing the task requests in turn, which will generate a large number of invalid queries that waste server resources. Summary of the invention

[0004] Embodiments of the present application provide a scheduling method, apparatus, device, and storage medium for a multimodal engine, which are used to improve the efficiency of multimodal engine scheduling.

[0005] In a first aspect, an embodiment of the present application provides a scheduling method for a multimodal engine, wherein any multimodal engine is deployed on a cluster server and completes a text-generating graph task in a single-threaded mode, and the method includes:

[0006] Get the task request, which contains the prompt words of the text image;

[0007] According to the preset description dimension of the text image, determine the complexity of the text image prompt words;

[0008] Based on the status information of any server in the cluster server, determine the target server matching the complexity;

[0009] The task request is sent to the target server and completed through the multimodal engine of the target server.

[0010] In an embodiment of the present application, when processing a task request for a text diagram, the complexity of the text diagram prompt words in the task request is first calculated to understand the complexity of the text diagram prompt words; then, by matching a server corresponding to the complexity, the server is used as the target server's multimodal engine to process the text diagram task request, so that the server processing the text diagram task request will neither be unable to complete the text diagram task due to low configuration, nor will it waste server resources due to high configuration.

[0011] Optionally, according to a preset description dimension of the text image, the complexity of the text image prompt word is determined, including:

[0012] According to the preset description dimensions of the text image, the text image prompt words are divided into multiple dimensions;

[0013] Score the detail level of the text diagram prompt words in any dimension to get the dimensional complexity;

[0014] The complexity of each dimension is calculated comprehensively to obtain the complexity of the prompt words in the text diagram.

[0015] In the embodiment of the present application, by presetting the description dimensions of the text diagram prompt words, after obtaining the text diagram prompt words, they can be divided according to the preset description dimensions, which is convenient for understanding the complexity of the text diagram prompt words; by scoring the degree of detail of multiple preset text diagram description dimensions, the complexity of the text diagram in each dimension can be accurately known, and then the complexity of the text diagram prompt words is obtained by combining the complexity of each dimension, so that the complexity calculation of the text diagram prompt words is more accurate.

[0016] Optionally, before obtaining the task request, the following is also included:

[0017] According to the configuration information of each server in the cluster server, the cluster server is divided into multiple server subsets; different server subsets correspond to different complexities;

[0018] Based on the status information of any server in the cluster server, determine the target server that matches the complexity, including:

[0019] Determine a subset of servers matching the complexity from the clustered servers;

[0020] Get an idle server from the matching server subset as the target server.

[0021] In an embodiment of the present application, first, by dividing the server into multiple server subsets according to the configuration information of the server, and determining the corresponding complexity according to the server subset, it is convenient to quickly find the corresponding target server; then, by determining the idle server in the server subgroup as the target server, it is avoided that the task request is not processed in time or even cannot be processed due to the use of the occupied server.

[0022] Optionally, it also includes:

[0023] If no idle server is found in the matching server subset, the task execution progress of each server in the matching server subset is obtained;

[0024] When it is detected that the server with the fastest task execution progress is in an idle state, the task request is sent to the target server.

[0025] In the embodiment of the present application, if there is no idle server, the server with the fastest execution progress is used as the target server, and the task request is sent to the server when the server with the fastest execution progress is idle, thereby improving the processing efficiency of the task request.

[0026] Optionally, after obtaining the task request and before determining the complexity of the text image prompt word, the method further includes:

[0027] Determining that there is no historical preferred prompt word that matches the Wensheng map prompt word;

[0028] Also includes:

[0029] If there is a historical preferred prompt word that matches the literary image prompt word, the historical image corresponding to the historical preferred prompt word is used as the execution result of the task request.

[0030] In the embodiment of the present application, by matching the text image prompt word with the historical preferred prompt word, if they match, the historical image corresponding to the historical preferred prompt word is used as the execution result of the task request, and there is no need to execute the current text image task request, which not only reduces the processing time of the task request, but also can obtain a better result of this task request.

[0031] Optionally, after obtaining the task request and before determining the complexity of the text image prompt word, the method further includes:

[0032] Determine modification suggestions for text image prompt words through machine learning models;

[0033] Obtain updated text-image prompt words based on user modification suggestions until the machine learning model determines that the text-image prompt words meet the text-image description dimensions; the machine learning model is trained based on prompt words of historical samples, image scoring results, and resource usage of the multimodal engine.

[0034] In an embodiment of the present application, a machine learning model is used to provide modification suggestions for the text image prompt words, so that when the text image prompt words are not accurate or detailed enough, the machine learning model is used to update the text image prompt words, thereby enhancing the quality of the text image prompt words, thereby ensuring the output of more accurate pictures and improving the user experience.

[0035] In a second aspect, an embodiment of the present application provides a scheduling device for a multimodal engine, wherein any multimodal engine is deployed on a cluster server and completes a text-generating image task in a single-threaded mode, and the device includes:

[0036] An acquisition module is used to acquire a task request, wherein the task request includes a text image prompt word;

[0037] A determination module, used to determine the complexity of the text diagram prompt words according to the preset text diagram description dimension;

[0038] A determination module, used for determining a target server matching the complexity based on the status information of any server in the cluster server;

[0039] The sending module is used to send the task request to the target server and complete the task request through the multimodal engine of the target server.

[0040] Optionally, the determination module is specifically used for:

[0041] According to the preset description dimensions of the text image, the text image prompt words are divided into multiple dimensions;

[0042] Score the detail level of the text diagram prompt words in any dimension to get the dimensional complexity;

[0043] The complexity of each dimension is calculated comprehensively to obtain the complexity of the prompt words in the text diagram.

[0044] Optionally, the acquisition module is also used to:

[0045] According to the configuration information of each server in the cluster server, the cluster server is divided into multiple server subsets; different server subsets correspond to different complexities;

[0046] Optionally, the determination module is specifically used for:

[0047] Determine a subset of servers matching the complexity from the clustered servers;

[0048] Get an idle server from the matching server subset as the target server.

[0049] Optionally, the acquisition module is also used to:

[0050] If no idle server is found in the matching server subset, the task execution progress of each server in the matching server subset is obtained;

[0051] When it is detected that the server with the fastest task execution progress is in an idle state, the task request is sent to the target server.

[0052] Optionally, the determination module is further configured to:

[0053] Determining that there is no historical preferred prompt word that matches the Wensheng map prompt word;

[0054] If there is a historical preferred prompt word that matches the literary image prompt word, the historical image corresponding to the historical preferred prompt word is used as the execution result of the task request.

[0055] Optionally, the determination module is further configured to:

[0056] Determine modification suggestions for text image prompt words through machine learning models;

[0057] Obtain updated text-image prompt words based on user modification suggestions until the machine learning model determines that the text-image prompt words meet the text-image description dimensions; the machine learning model is trained based on prompt words of historical samples, image scoring results, and resource usage of the multimodal engine.

[0058] In a third aspect, an embodiment of the present application provides a computer device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the steps of the above method are implemented when the processor executes the program.

[0059] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium storing a computer program executable by a computer device, which enables the computer device to perform the steps of the above method when the program is run on the computer device.

[0060] In a fifth aspect, an embodiment of the present application provides a computer program product, which includes a computer program stored on a computer-readable storage medium, and the computer program includes program instructions. When the program instructions are executed by a computer device, the computer device executes the steps of the above method.

[0061] In an embodiment of the present application, when processing a task request for a text diagram, the complexity of the text diagram prompt words in the task request is first calculated to understand the complexity of the text diagram prompt words; then, by matching a server corresponding to the complexity, the server is used as the target server's multimodal engine to process the text diagram task request, so that the server processing the text diagram task request will neither be unable to complete the text diagram task due to low configuration, nor will it waste server resources due to high configuration. BRIEF DESCRIPTION OF THE DRAWINGS

[0062] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.

[0063] Figure 1 A system architecture diagram provided for an embodiment of the present application;

[0064] Figure 2 A flowchart of a scheduling method for a multimodal engine provided in an embodiment of the present application (I);

[0065] Figure 3A schematic diagram of a process for determining the complexity of a text image prompt word provided in an embodiment of the present application;

[0066] Figure 4 A flowchart of a scheduling method for a multimodal engine provided in an embodiment of the present application (II);

[0067] Figure 5 A schematic diagram of the structure of a scheduling device for a multimodal engine provided in an embodiment of the present application;

[0068] Figure 6 A schematic diagram of the structure of a computer device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0069] In order to make the purpose, technical solution and beneficial effects of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0070] To facilitate understanding of this solution, the application scenario of this solution is introduced below.

[0071] Wenshengtu is an application that generates corresponding pictures based on Wenshengtu prompt words. Wenshengtu relies on a multimodal engine at the bottom layer. The multimodal engine itself has a large overhead on the graphics processor (GPU), and the implementation and deployment methods based on the multimodal engine itself usually adopt single-machine deployment and single-threaded execution. When a Wenshengtu task request is in the process of execution, another Wenshengtu task request can only block and wait for the previous task request to complete. Therefore, the number of tasks processed per second (TPS) of Wenshengtu task requests can easily become a bottleneck. In addition, the deployment of the multimodal engine cannot adopt the traditional load balancing solution, because the load balancing solution supports multi-threaded mode, but the multimodal engine only supports single-threaded mode. When calling the multimodal engine interface, the interface processing time is relatively long and the time consumption of receiving different types of task requests is quite different. If the task request call is sent frequently, the waiting queue of the multimodal engine will accumulate and timeout problems will occur.

[0072] Usually, polling is used to check the server cluster where the multimodal engine is located, pair the server with the task request, and determine the server that is most suitable for processing the text and image task request. However, this requires the task request to actively query the execution status of all servers in the cluster continuously, which will generate a large number of invalid queries that waste resources. Therefore, this application proposes a scheduling method for a multimodal engine, and the specific operation steps are as follows:

[0073] First, obtain a text map task request, which includes text map prompt words. When the description of the text map prompt words does not meet the requirements, the machine learning model makes modification suggestions for the input text map prompt words, and obtains the resubmitted text map prompt words according to the modification suggestions. Then, the text map prompt words are divided into multiple dimensions, the complexity of each dimension is determined respectively, and the complexity of the text map prompt words is obtained by integrating the complexity of multiple dimensions. Then, according to the complexity of the text map prompt words and the status information of the servers in the cluster, the server matching the text map task request is determined as the target server, and finally, the text map task request is processed by the target server. Among them, before determining the complexity of each dimension, when training the machine learning model, the historical sample prompt words, the picture scoring results output according to the historical sample prompt words, and the resource occupancy of the multimodal engine are used to train the machine learning model, so that the machine learning model can accurately make modification suggestions according to the text map prompt words, thereby meeting the text map description dimension. When determining the target server, firstly, the clustered servers are divided into multiple server subsets according to the configuration information of the servers, each server subset corresponds to a different complexity of the Vincent diagram prompt words, and then the server subset matching the complexity of the Vincent diagram prompt is determined. If there is an idle server in the server subset, the idle server is used as the target server; if there is no idle server in the server subset, the task execution progress of each server in the server subset is obtained, and the server with the fastest task execution progress is monitored. When it is in an idle state, it is used as the target server.

[0074] refer to Figure 1 , is a system architecture diagram provided in an embodiment of the present application, which includes: a terminal device 101 and a cluster server 102. The terminal device 101 can be a tablet computer, a laptop computer, a desktop computer, etc., but is not limited thereto. The cluster server 102 can be an independent physical server, or a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, content delivery networks (CDN), and big data and artificial intelligence platforms. The terminal device 101 and the cluster server 102 can be connected by wire or wirelessly.

[0075] based on Figure 1 As shown in the system architecture diagram, the present application embodiment provides a scheduling method for a multimodal engine, where any multimodal engine is deployed on a cluster server and completes the text map task in a single-threaded mode. Figure 2 As shown, the method comprises the following steps:

[0076] Step 201: Obtain a task request, where the task request includes a text image prompt word.

[0077] Specifically, the task request is a task request for a text image initiated to a cluster server. The task request includes text image prompt words. The text image prompt words are text descriptions for the generated image. For example, if the task request is "please generate a horse", "please generate a girl wearing glasses and sitting in a library reading a book", etc., the text image prompt words are "a horse", "library", "sitting", "wearing glasses", and "girl".

[0078] Step 202: Determine the complexity of the text image prompt words according to the preset text image description dimension.

[0079] Specifically, the description dimensions of the text image are preset. For example, in a text image task request for generating an object, the preset text image description dimensions are: subject description, subject details, environment description, etc.; in a text image task for generating a font, the preset text image description dimensions are: font size, font style, font color, etc. The complexity of the text image prompt words is determined according to each description dimension, that is, the finer the description dimension, the higher the complexity of the text image prompt words.

[0080] In some embodiments, the complexity of the text diagram prompt words is determined according to the preset text diagram description dimension, such as Figure 3 As shown, the following steps are included:

[0081] Step 301: Divide the text image prompt words into multiple dimensions according to the preset text image description dimensions.

[0082] Specifically, the description dimension of the text map is pre-set, and the text map description dimension is based on the detailed description of the generated image in different dimensions. The more the text map description dimensions are, the more complex the text map prompt words are, and the more details of the generated image are. The text map prompt words are divided into multiple dimensions according to the preset description dimensions, and the specific content of each dimension is the detailed description of the generated image. For example, the text map description dimensions for the generated object can be the following dimensions:

[0083] Subject description: Identify the main object or person in the image, which is the basis for generating the image;

[0084] Subject details: further describe the subject's appearance, movements, expressions, etc. to increase the richness and accuracy of the image;

[0085] Environmental description: describes the background, scene and atmosphere in the image, providing necessary contextual information for the image;

[0086] Artistic style: Specify the style of the image, such as realism, cartoon, oil painting, etc., to meet different aesthetic needs;

[0087] Image quality and viewing angle: Describes the clarity, color saturation, and viewing angle of an image to control the visual effect of the image.

[0088] Any description dimension may contain multiple text-image prompt words, or may not contain any text-image prompt words.

[0089] For example, in the task request of generating a text image of an object, the preset text image description dimensions include: subject description, subject details, environment description, artistic style, image quality, and viewing angle. If the text image prompt is "Please generate a girl who is sitting at a table with a serious expression, wearing glasses, and reading a book in the library", the text image prompt is split in the description dimension, the subject description is "girl", the subject details are "serious expression", "wearing glasses", "reading a book with her head down", the environment description is "library", the artistic style is "realistic", the image quality is "high definition", and the viewing angle is "sitting at the table"; if the text image prompt is "Please generate a girl who is reading a book", the text image prompt only exists in the two dimensions of subject description "girl" and subject details "reading a book".

[0090] Step 302: Score the detail description of the text diagram prompt words in any dimension to obtain the dimensional complexity.

[0091] Specifically, the higher the degree of detail description in any dimension, the higher the score and the higher the dimensional complexity. For example, in the generated object's text image prompt, if it is "please generate a girl reading a book", it is scored according to the text image prompt with the description dimension, and the dimension without the text image prompt is 0 points; if it is "please generate a girl sitting at a table with a serious expression, wearing glasses, reading a book in the library", the text image prompt is scored according to the description dimension. Since the text image prompt exists in any dimension and the degree of detail description in multiple dimensions is higher than the text image prompt "please generate a girl reading a book", the dimensional complexity of the latter text image prompt is higher than the former.

[0092] Step 303: Comprehensively calculate the complexity of each dimension to obtain the complexity of the text image prompt word.

[0093] Specifically, the complexity of each dimension is comprehensively calculated to obtain the complexity of the text diagram prompt word. The comprehensive calculation can be to add up the scores of each dimension as the complexity of the text diagram prompt word; or it can be to weight the sum of the detail description degree of each dimension according to the weight of each dimension to obtain the complexity of the text diagram prompt word.

[0094] For example, for the text image prompt “Please generate a girl who is reading a book in a library, with a serious expression, wearing glasses, and sitting at a table”, the degree of detail description in each dimension is summed up to get the complexity of the text image prompt; alternatively, the weights of the subject description and subject details are set to 0.3 and 0.3, the weight of the environment description is set to 0.2, the weight of the artistic style is set to 0.1, and the weights of the image quality and viewing angle are set to 0.05. After obtaining the detail description degree scores of each dimension, a weighted sum is performed to get the complexity of the text image prompt.

[0095] In the embodiment of the present application, by presetting the description dimensions of the text diagram prompt words, after obtaining the text diagram prompt words, they can be divided according to the preset description dimensions, which is convenient for understanding the complexity of the text diagram prompt words; by scoring the degree of detail of multiple preset text diagram description dimensions, the complexity of the text diagram in each dimension can be accurately known, and then the complexity of the text diagram prompt words is obtained by combining the complexity of each dimension, so that the complexity calculation of the text diagram prompt words is more accurate.

[0096] Step 203: Determine a target server that matches the complexity based on the status information of any server in the cluster server.

[0097] Specifically, in a cluster server, different servers have different status information, and the status information includes the running status and configuration information of the server. The target server matching the complexity of the text image prompt word is determined according to the status information.

[0098] For example, based on the status information of the server, the server score is determined to be one of 0 to 10 points, and the complexity score of the text diagram prompt word is also 0 to 10 points. If the complexity score of a text diagram prompt word is 8 points, then a subset of servers whose status information is also 8 points is obtained from multiple servers as the target server; or, servers whose status information is 8 points or above are obtained from multiple servers as the target server.

[0099] Step 204: Send the task request to the target server, and complete the task request through the multimodal engine of the target server.

[0100] In an embodiment of the present application, when processing a task request for a text diagram, the complexity of the text diagram prompt words in the task request is first calculated to understand the complexity of the text diagram prompt words; then, by matching a server corresponding to the complexity, the server is used as the target server's multimodal engine to process the text diagram task request, so that the server processing the text diagram task request will neither be unable to complete the text diagram task due to low configuration, nor will it waste server resources due to high configuration.

[0101] In some embodiments, before obtaining a task request, the process includes:

[0102] According to the configuration information of each server in the cluster server, the cluster server is divided into multiple server subsets; different server subsets correspond to different complexities.

[0103] Specifically, there are multiple servers in the cluster server, and each server has its own configuration information, such as operating system, memory, hard disk and other information. According to different configuration information, the cluster server is divided into multiple server subsets. The servers in the server subsets have the same or similar configuration information, and different server subsets correspond to different text map prompt word complexities.

[0104] In some embodiments, based on the status information of any server in the cluster server, determining the target server matching the complexity includes: determining a server subset matching the complexity from the cluster server; obtaining an idle server from the matching server subset as the target server.

[0105] Specifically, the server subsets are scored according to the configuration information of the server subsets, and a server subset corresponding to the complexity of the text map prompt words is determined from multiple server subsets; and then a server in an idle state is determined from the server subsets as a target server.

[0106] For example, the servers are scored according to their configuration information and divided into three server subsets, namely 9 points, 8 points, and 5 points. If the complexity of the Wenshengtu prompt word is 8 points, the server subset with a score of 5 points is first eliminated, and then an idle server is selected from the servers with a score of 9 points and 8 points as the target server. Alternatively, an idle server in the server subset with a score of 8 points is directly determined as the target server. In addition, the corresponding server can be directly matched to the user according to the user's resource consumption.

[0107] Each server in different server subsets can execute a unique task at the same time. The task status is divided into initial state, running state, and final state. The final state is divided into completion and timeout. When the server task status is initial state or running state, the server status is unavailable (i.e. non-idle state); when the task execution is in the final state, the server status is idle. The task execution will also set whether to time out to avoid the task occupying server resources for too long. If the running state time exceeds the timeout, the server will interrupt the task and change the state to idle state.

[0108] When the task is in the running state, the progress of the task will be reported to the flink streaming computing platform once a second through the message middleware KAFKA. flink is usually used to generate real-time report scenarios. Flink is introduced as a distributed real-time sorting middleware to summarize and sort the progress of each server through the flink platform. flink will update the real-time progress data of each server's task, such as high-end9-1:99%, low-end2-8:52%, etc., indicating that the task execution progress of the server with serial number 1 of high-end configuration level 9 is 99%, and the execution progress of the server with serial number 8 of low-end configuration level 2 is 52%, so as to manage server resources in a refined manner.

[0109] At the same time, the real-time task execution progress of each server and the task queue after analysis and classification are obtained through the Flink platform. If there is an idle server that meets the complexity, the server will be directly matched as the target server.

[0110] In an embodiment of the present application, first, by dividing the server into multiple server subsets according to the configuration information of the server, and determining the corresponding complexity according to the server subset, it is convenient to quickly find the corresponding target server; then, by determining the idle server in the server subgroup as the target server, it is avoided that the task request is not processed in time or even cannot be processed due to the use of the occupied server.

[0111] In some embodiments, if no idle server is obtained in the matching server subset, the task execution progress of each server in the matching server subset is obtained; when it is monitored that the server with the fastest task execution progress is in an idle state, the task request is sent to the target server.

[0112] Specifically, after obtaining the real-time task execution progress of each server in the server subset through the flink platform, if no idle server is obtained and there is no available server to allocate temporarily, the top 10 servers in task execution progress will be obtained through the flink platform. The task execution progress is sorted according to the completion status of the previous task execution progress. For example, if the previous task execution progress is 98%, it means that the previous task is about to be completed. The shorter the waiting time, the server will be locked and the task request will be assigned to the server. When the task request is completed, the server will be unlocked and the server will re-enter the idle state. At this point, the service process of a single task request is completed, and new task requests will be repeated according to the above steps.

[0113] In the embodiment of the present application, if there is no idle server, the server with the fastest execution progress is used as the target server, and the task request is sent to the server when the server with the fastest execution progress is idle, thereby improving the processing efficiency of the task request.

[0114] In some embodiments, after obtaining the task request and before determining the complexity of the text image prompt word, it also includes: determining that there is no historical preferred prompt word matching the text image prompt word; if there is a historical preferred prompt word matching the text image prompt word, then taking the historical picture corresponding to the historical preferred prompt word as the execution result of the task request.

[0115] Specifically, after obtaining the task request, it is determined whether there is a historical preferred prompt word matching the text image prompt word according to the prompt word in the task request. The hbase of big data is used to store the historical preferred prompt words and the corresponding historical images.

[0116] If it does not exist, perform the Wenshengtu task according to the above method.

[0117] If it exists, the historical image corresponding to the historical preferred prompt word is obtained as the execution result of this task request. This can speed up the progress of task execution.

[0118] In the embodiment of the present application, by matching the text image prompt word with the historical preferred prompt word, if they match, the historical image corresponding to the historical preferred prompt word is used as the execution result of the task request, and there is no need to execute the current text image task request, which not only reduces the processing time of the task request, but also can obtain a better result of this task request.

[0119] In some embodiments, after obtaining the task request and before determining the complexity of the text diagram prompt word, the method further includes:

[0120] Determine modification suggestions for text image prompt words through a machine learning model; obtain updated text image prompt words from users based on modification suggestions until the machine learning model determines that the text image prompt words meet the text image description dimensions; the machine learning model is trained based on prompt words of historical samples, image scoring results, and resource usage of a multimodal engine.

[0121] Specifically, the data generated by the above user behaviors and their feedback results are provided to the machine learning model for learning. Because the data contains user feedback results, the machine learning model will use the reinforcement learning algorithm to learn the rules of text image request prompt words with higher image rating results and lower resource consumption. Users continuously provide the machine learning model with image rating results, and the machine learning model in turn generates images that better meet user needs.

[0122] Hbase can also store information including but not limited to: prompt words of historical samples, image scoring results, and resource usage of the multimodal engine, such as task execution time, CPU, memory, video memory, etc. This data will be provided to the machine learning model as training data.

[0123] The mathematical model of the machine learning model is shown in formula (1):

[0124] V{t+1}(s)=max{a}sum{s ′}P(s ′ |s,a)[R(s,a,s ′ )

[0125] +gamma Vt(s ′ )]...Formula (1)

[0126] Where V{t+1}(s) is the value function of the next execution step, Vt(s) is the value function of the current execution step, s is the current state, a is the action to be executed, s' is the next state, R(s,a,s') is the reward for performing action a in state s and transferring to state s', gamma is the discount factor, and P(s'|s,a) is the probability of transferring to state s' after taking action a in state s.

[0127] The value function of the model agent is continuously updated through multiple rounds of attempts and feedback (the model agent is an intelligent entity that can learn autonomously, make decisions and perform corresponding actions, which can be expressed in various forms, such as software programs, robots or other automated equipment; the value function is a mathematical function used to evaluate the expected long-term return of the agent after taking a specific execution action in a given state). In each round of iteration, for each state (that is, under a given strategy X, starting from state S and following strategy X, the expected cumulative return can be obtained), the expected reward of all possible actions is calculated. Then, the maximum reward is selected as the value of the current state, so that it gradually approaches the optimal value. In each round of attempts and feedback, the algorithm will update the value of all possible states. When the value function no longer changes after multiple rounds of iterations, or reaches the preset maximum number of iterations, the algorithm stops, so that the agent can select the most optimized prompt word to perform actions in any state.

[0128] When the machine learning model is trained and the task request is obtained, the document image prompt words are first analyzed, the document image prompt word keywords are disassembled, and then check whether there are similar or identical historical preferred prompt words. If there are, the historical images corresponding to the historical preferred prompt words are directly found and returned to the user without calling the server for execution; if the server is called for execution according to the above method.

[0129] Input the text image prompt words into the machine learning model, and the machine learning model will give modification suggestions for this text image prompt words. For example, if the text image prompt words are "Please help me generate a girl's portrait", the machine learning model will prompt "This text image prompt word is too general, and the detailed description can be strengthened, such as please strengthen the subject description of the girl's age, appearance, and clothing-related features, and the subject details. For example, further describe the girl's details such as hairstyle and color, skin color, etc., character clothing, expression, etc., environmental description such as where the background of the character is, what the surrounding environment is like, what the artistic style is, such as character portrait or anime style, Japanese style or Chinese style or some other style series, etc. Finally, the picture quality and perspective will also suggest soft natural light, backlight, and whether it highlights a fresh and quiet atmosphere." It is equivalent to making optimization suggestions for overly simple text image prompt words from various description dimensions. Users update the text image prompt words in various description dimensions according to the modification suggestions. When the text image prompt words are updated, the updated task request is submitted. For example, "Please help me generate a girl's portrait" is modified to be as shown below:

[0130] “Create a realistic portrait of a young girl, in the style of Japanese film, Hideaki Hamada, portrait, ultra-high definition, rich light and shadow, and super complex details.

[0131] Core features: A young woman in a Japanese school uniform, placed in the center or in one-third of the composition, with the sea in the background, and extremely high-precision photography.

[0132] Character color: long black hair, black pupils, pink lips, fair skin, high contrast;

[0133] Character clothing: Japanese school uniform;

[0134] Character expressions and movements: looking directly at the camera, smiling;

[0135] Accessories and background elements: simple necklace, sea background;

[0136] Light and shadow effects: soft natural light, backlight;

[0137] Photographic equipment and parameters: Fuji, 35mm fixed focus, 1.4 large aperture;

[0138] Atmosphere and mood: Fresh, tranquil. ”

[0139] The above is a text image prompt with richer dimensions, which is enough to generate a picture that better meets user needs.

[0140] After modifying the text image prompt words, the optimizer can be used to polish and enrich the text image prompt words based on the machine learning model to generate new text image prompt words. The optimizer uses the good text image prompt words obtained by the optimizer to optimize the image generation results, reduce the number of user trial and error calls, thereby reducing the waste of invalid server resources and enhancing the quality of text image prompt words.

[0141] In an embodiment of the present application, a machine learning model is used to provide modification suggestions for the text image prompt words, so that when the text image prompt words are not accurate or detailed enough, the machine learning model is used to update the text image prompt words, thereby enhancing the quality of the text image prompt words, thereby ensuring the output of more accurate pictures and improving the user experience.

[0142] In order to further explain the present application, the method for scheduling a multimodal engine is described in detail below with a specific implementation scenario, which specifically includes the following steps: Figure 4 As shown:

[0143] First, let’s introduce the machine learning model training phase:

[0144] Step 401: Input historical sample prompt words into the machine learning model, where the historical sample prompt words are historical input text and image prompt words.

[0145] Step 402: The machine learning model outputs historical pictures based on the historical sample prompt words, and the user scores the pictures to obtain picture rating results.

[0146] Step 403: When the machine learning model goes from input to output, it calculates the resource usage of the multimodal engine (such as the memory or hard disk size, task execution time).

[0147] Step 404: Feedback each input historical sample prompt word, the corresponding image scoring result, and resource usage status to the machine learning model.

[0148] Step 405: The machine learning model continuously adjusts parameters based on feedback until a picture that meets user needs is obtained.

[0149] The following describes the scheduling method of the multimodal engine and the execution process of the text graph task:

[0150] Step 406: Score the configuration information of each server;

[0151] Step 407: divide the servers with the same score into the same server subset;

[0152] Step 408: Obtain a task request for a text image, such as "generate a girl wearing glasses and reading in a library".

[0153] Step 409: Determine whether there is a relevant historical text image prompt word in the database. If so, execute step 410; if not, execute step 411.

[0154] Step 410: Use the picture corresponding to the historical cultural image prompt word in the database as the picture requested for this cultural image task.

[0155] Step 411: Divide the text image task request into five description dimensions: subject description, subject details, environment description, image quality, and viewing angle.

[0156] Step 412: Divide the description dimensions corresponding to "generate a girl wearing glasses and reading in the library", and obtain the description content of each dimension as follows: subject description "girl", subject details "wearing glasses", environment description "library", image quality and viewing angle "none".

[0157] Step 413: Input the dimensionally divided text image prompt words into the machine learning model, and “supplement the image quality and perspective” according to the suggestions of the machine learning model.

[0158] Step 414, update the prompt words of the Wensheng picture to: "Generate a girl wearing glasses reading a book in the library, with a serious expression and lowering her head to read the book."

[0159] Step 415: Calculate the complexity score of each description dimension, 3 points for subject description, 2 points for subject details, 2 points for environment description, and 2 points for image quality and viewing angle.

[0160] Step 416: Calculate the complexity of the Wenshengtu prompt word "generate a girl wearing glasses reading in the library" as 9 points.

[0161] Step 417: Select a server subset with configuration information of 9 points from the server subsets.

[0162] Step 418: Determine an idle server from the 9-point server subset.

[0163] Step 419: Send the task request of the text image prompt "generate a girl wearing glasses reading in the library" to the idle servers in the 9-point server subset.

[0164] Step 420: Use idle servers in the server subset of 9 to generate corresponding images from text-based image prompt words.

[0165] In an embodiment of the present application, when processing a task request for a text diagram, the complexity of the text diagram prompt words in the task request is first calculated to understand the complexity of the text diagram prompt words; then, by matching a server corresponding to the complexity, the server is used as the target server's multimodal engine to process the text diagram task request, so that the server processing the text diagram task request will neither be unable to complete the text diagram task due to low configuration, nor will it waste server resources due to high configuration.

[0166] Based on the same technical concept, the embodiment of the present application provides a scheduling device for a multimodal engine, wherein any multimodal engine is deployed on a cluster server and completes the text-generating image task in a single-threaded mode, such as Figure 5 As shown, the device 500 includes:

[0167] An acquisition module 501 is used to acquire a task request, wherein the task request includes a text image prompt word;

[0168] A determination module 502 is used to determine the complexity of the text diagram prompt words according to the preset text diagram description dimension;

[0169] The determination module 502 is used to determine a target server matching the complexity based on the status information of any server in the cluster server;

[0170] The sending module 503 is used to send the task request to the target server and complete the task request through the multimodal engine of the target server.

[0171] Optionally, the determination module 502 is specifically configured to:

[0172] According to the preset description dimensions of the text image, the text image prompt words are divided into multiple dimensions;

[0173] Score the detail level of the text diagram prompt words in any dimension to get the dimensional complexity;

[0174] The complexity of each dimension is calculated comprehensively to obtain the complexity of the prompt words in the text diagram.

[0175] Optionally, the acquisition module 501 is further used for:

[0176] According to the configuration information of each server in the cluster server, the cluster server is divided into multiple server subsets; different server subsets correspond to different complexities;

[0177] Optionally, the determination module 502 is specifically configured to:

[0178] Determine a subset of servers matching the complexity from the clustered servers;

[0179] Get an idle server from the matching server subset as the target server.

[0180] Optionally, the acquisition module 501 is further used for:

[0181] If no idle server is found in the matching server subset, the task execution progress of each server in the matching server subset is obtained;

[0182] When it is detected that the server with the fastest task execution progress is in an idle state, the task request is sent to the target server.

[0183] Optionally, the determining module 502 is further configured to:

[0184] Determining that there is no historical preferred prompt word that matches the Wensheng map prompt word;

[0185] If there is a historical preferred prompt word that matches the literary image prompt word, the historical image corresponding to the historical preferred prompt word is used as the execution result of the task request.

[0186] Optionally, the determining module 502 is further configured to:

[0187] Determine modification suggestions for text image prompt words through machine learning models;

[0188] Obtain updated text-image prompt words based on user modification suggestions until the machine learning model determines that the text-image prompt words meet the text-image description dimensions; the machine learning model is trained based on prompt words of historical samples, image scoring results, and resource usage of the multimodal engine.

[0189] Based on the same technical concept, the embodiment of the present application provides a computer device, which may be a terminal or a server, such as Figure 6 As shown, it includes at least one processor 601 and a memory 602 connected to the at least one processor. The specific connection medium between the processor 601 and the memory 602 is not limited in the embodiment of the present application. Figure 6 For example, the processor 601 and the memory 602 are connected via a bus. The bus can be divided into an address bus, a data bus, a control bus, and the like.

[0190] In the embodiment of the present application, the memory 602 stores instructions that can be executed by at least one processor 601. The at least one processor 601 can execute the steps included in the above-mentioned controlled transaction method by executing the instructions stored in the memory 602.

[0191] Among them, the processor 601 is the control center of the computer device, and can use various interfaces and lines to connect various parts of the computer device, and perform master-slave abnormal recovery by running or executing instructions stored in the memory 602 and calling data stored in the memory 602. Optionally, the processor 601 may include one or more processing units, and the processor 601 may integrate an application processor and a modem processor, wherein the application processor mainly processes the operating system, user interface, and application programs, and the modem processor mainly processes wireless communications. It is understandable that the above-mentioned modem processor may not be integrated into the processor 601. In some embodiments, the processor 601 and the memory 602 may be implemented on the same chip, and in some embodiments, they may also be implemented separately on independent chips.

[0192] Processor 601 can be a general-purpose processor, such as a central processing unit (CPU), a digital signal processor, an application-specific integrated circuit (ASIC), a field programmable gate array or other programmable logic device, a discrete gate or transistor logic device, a discrete hardware component, and can implement or execute the methods, steps and logic block diagrams disclosed in the embodiments of the present application. A general-purpose processor can be a microprocessor or any conventional processor, etc. The steps of the method disclosed in the embodiments of the present application can be directly embodied as a hardware processor for execution, or can be executed by a combination of hardware and software modules in the processor.

[0193] The memory 602 is a non-volatile computer-readable storage medium that can be used to store non-volatile software programs, non-volatile computer executable programs and modules. The memory 602 may include at least one type of storage medium, such as a flash memory, a hard disk, a multimedia card, a card-type memory, a random access memory (Random Access Memory, RAM), a static random access memory (Static Random Access Memory, SRAM), a programmable read-only memory (Programmable Read Only Memory, PROM), a read-only memory (Read Only Memory, ROM), an electrically erasable programmable read-only memory (Electrically Erasable Programmable Read-Only Memory, EEPROM), a magnetic memory, a disk, an optical disk, etc. The memory 602 is any other medium that can be used to carry or store a desired program code in the form of an instruction or data structure and can be accessed by a computer, but is not limited thereto. The memory 602 in the embodiment of the present application can also be a circuit or any other execution device that can realize a storage function, for storing program instructions and / or data.

[0194] Based on the same inventive concept, an embodiment of the present application provides a computer-readable storage medium, which stores a computer program that can be executed by a computer device. When the program runs on the computer device, the computer device executes the steps of the above-mentioned controlled transaction method.

[0195] Based on the same inventive concept, an embodiment of the present application provides a computer program product, characterized in that the computer program product includes a computer program stored on a computer-readable storage medium, and the computer program includes program instructions. When the program instructions are executed by a computer device, the computer device executes the steps of the above-mentioned controlled transaction method.

[0196] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment in combination with software and hardware. Moreover, the present application may adopt the form of a computer program product implemented in one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) that include computer-usable program code.

[0197] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0198] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0199] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0200] Obviously, those skilled in the art can make various changes and modifications to the present application without departing from the spirit and scope of the present application. Thus, if these modifications and variations of the present application fall within the scope of the claims of the present application and their equivalents, the present application is also intended to include these modifications and variations.

Claims

1. A scheduling method for a multimodal engine, characterized in that: Any multimodal engine is deployed on a cluster server and completes the text-to-image task in a single-threaded mode, the method includes: Obtaining a task request, wherein the task request includes a text image prompt word; Determining the complexity of the prompt words of the text image according to the preset description dimension of the text image; Determine a target server matching the complexity based on status information of any server in the cluster server; The task request is sent to the target server, and the task request is completed through the multimodal engine of the target server.

2. The method according to claim 1, characterized in that According to the preset description dimensions of the text image, determine the complexity of the text image prompt words, including: According to the preset description dimensions of the text image, the text image prompt words are divided into multiple dimensions; Scoring the detail description of the text diagram prompt words in any dimension to obtain the dimensional complexity; The complexity of each dimension is calculated comprehensively to obtain the complexity of the text image prompt word.

3. The method according to claim 1, characterized in that Before getting the task request, it also includes: According to the configuration information of each server in the cluster server, the cluster server is divided into a plurality of server subsets; different server subsets correspond to different complexities; Based on the status information of any server in the cluster server, determine the target server that matches the complexity, including: Determining a subset of servers matching the complexity from the clustered servers; A server in an idle state is obtained from the matched server subset as the target server.

4. The method according to claim 1, characterized in that Also includes: If no server in the idle state is obtained in the matched server subset, the task execution progress of each server in the matched server subset is obtained; When it is detected that the server with the fastest task execution progress is in an idle state, the task request is sent to the target server.

5. The method according to any one of claims 1 to 4, characterized in that: After obtaining the task request and before determining the complexity of the text image prompt word, the method further includes: Determining that there is no historical preferred prompt word matching the text map prompt word; The method further comprises: If there is a historical preferred prompt word that matches the cultural image prompt word, the historical image corresponding to the historical preferred prompt word is used as the execution result of the task request.

6. The method according to any one of claims 1 to 4, characterized in that: After obtaining the task request and before determining the complexity of the text image prompt word, the method further includes: Determining modification suggestions for the text image prompt words through a machine learning model; Obtain updated text image prompt words based on modification suggestions by the user until the machine learning model determines that the text image prompt words meet the text image description dimensions; the machine learning model is trained based on prompt words of historical samples, image scoring results, and resource occupancy status of a multimodal engine.

7. A scheduling device for a multimodal engine, characterized in that: Any multimodal engine is deployed on a cluster server and completes the text-generated image task in a single-threaded mode. The device includes: An acquisition module, used for acquiring a task request, wherein the task request includes a text image prompt word; A determination module, used to determine the complexity of the prompt words of the text diagram according to the preset text diagram description dimension; A determination module, used for determining a target server matching the complexity based on the status information of any server in the cluster server; The sending module is used to send the task request to the target server and complete the task request through the multimodal engine of the target server.

8. A computer device comprising a memory, a processor and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the program, the steps of the method described in any one of claims 1 to 6 are implemented.

9. A computer-readable storage medium, characterized in that: It stores a computer program executable by a computer device, and when the program is run on the computer device, the computer device executes the steps of any method described in claims 1 to 6.

10. A computer program product, characterized in that The computer program product comprises a computer program stored on a computer-readable storage medium, wherein the computer program comprises program instructions, and when the program instructions are executed by a computer device, the computer device executes the steps of the method according to any one of claims 1 to 6.

Images