Training method of belief generation model, related method and device

Abstract

The invention discloses a belief generation model training method, which is applied to mobile equipment and comprises the following steps: acquiring a plurality of experiences, each experience comprising an experience data set and an experience priority corresponding to the experience data set; determining a target experience from the plurality of experiences based on the experience priorities of the plurality of experiences; the target experience is input into the belief generation model for training, the belief generation model is obtained, and the belief generation model is used for calculation unloading and resource allocation of video diffusion tasks. The invention further discloses a related method and device. The data utilization rate in reinforcement learning is improved.

Description

Technical Field

[0001] The disclosed embodiments of the present application relate to the field of artificial intelligence, and more specifically, to a training method, related methods and devices for a belief generation model. Background Art

[0002] Reinforcement Learning (RL) is used to describe and solve the problem of how an agent learns strategies to maximize rewards or achieve specific goals during its interaction with the environment. When using the diffusion model to implement tasks such as video generation, the video diffusion task can be offloaded to the cloud / edge for calculation and the results can be returned. The diffusion model is trained in combination with reinforcement learning to obtain a reasonable calculation offloading and resource allocation strategy. However, the traditional reinforcement learning algorithm has a low data utilization rate, which affects the learning efficiency of the related models. Summary of the invention

[0003] According to the embodiments of the present application, the present application proposes a training method, related methods and devices for a belief generation model to solve the above-mentioned problems.

[0004] The first aspect of the present application discloses a training method for a belief generation model, which is applied to a mobile device, including: obtaining multiple experiences, each of which includes an experience data set and an experience priority corresponding to the experience data set; determining a target experience from the multiple experiences based on the experience priorities of the multiple experiences; inputting the target experience into the belief generation model for training to obtain the belief generation model, wherein the belief generation model is used for computational offloading and resource allocation of video diffusion tasks.

[0005] In some embodiments, the empirical data set includes: first empirical data, used to characterize server information of computational offloading and policy information of resource allocation; second empirical data, used to characterize environmental information of a server connected to the mobile device, wherein the server connected to the mobile device includes at least one of a cloud server and an edge server; and third empirical data, used to characterize a corresponding free energy return, wherein the free energy return represents a return for measuring the belief generation model.

[0006] In some embodiments, the second experience data in the target experience includes current second experience data and next second experience data; the belief generation model includes a belief encoder, a dynamic transfer model and a belief error model; the target experience is input into the belief generation model for training, including: using the belief encoder to encode the current second experience data in the target experience as the current prior belief, and encoding the next second experience data in the target experience as the next prior belief; using the dynamic transfer model to predict the current prior belief and the current first experience data in the target experience to obtain the next posterior belief; using the belief error model, based on a preset loss function, calculate the error between the next prior belief and the next posterior belief to obtain a belief error score corresponding to the current prior belief.

[0007] In some embodiments, the belief generation model also includes a free energy preference model; the inputting of the target experience into the belief generation model for training further includes: inputting the current prior belief and the next prior belief into the free energy preference model in sequence to obtain the free energy preference score corresponding to the current prior belief and the free energy preference score corresponding to the next prior belief, respectively.

[0008] In some embodiments, the free energy corresponding to the current prior belief is calculated using the belief error score corresponding to the current prior belief and the free energy preference score corresponding to the current prior belief; and the experience priority in the target experience is updated using the free energy corresponding to the current prior belief.

[0009] In some embodiments, the third empirical data includes the final scores corresponding to the mobile device and the server and the belief error of the belief generation model; obtaining the final scores corresponding to the mobile device and the server includes: based on the video diffusion task, taking the device information of the mobile device and the device information of the server as variables, constructing a data transmission function and a task calculation function between the mobile device and the corresponding server; weighting the data transmission function and the task calculation function to obtain a final score function, wherein the final score function is added with a preset function constraint and is used to determine the final score corresponding to the mobile device and the server.

[0010] The second aspect of the present application discloses a method for computational offloading and resource allocation, comprising: being applied to a mobile device, the mobile device being connected to at least one server, comprising: the mobile device obtaining environmental information of the mobile device and the at least one server from the at least one server; using a belief generation model, converting the environmental information of the mobile device and the at least one server to obtain corresponding beliefs; predicting the corresponding beliefs to generate a plurality of candidate action sequences, and obtaining free energies of the plurality of candidate action sequences, wherein each candidate action in the plurality of candidate action sequences represents a candidate strategy for computational offloading and resource allocation; and determining a target action from the plurality of candidate action sequences based on the free energies of the plurality of candidate action sequences, wherein the target action represents a target strategy for computational offloading and resource allocation, and is an action in the candidate action sequence with the smallest free energy among the plurality of candidate action sequences; wherein the belief generation model is obtained by the training method of the belief generation model described in the first aspect.

[0011] In some embodiments, the predicting of the corresponding beliefs to generate multiple candidate action sequences and obtain the free energy of the multiple candidate action sequences includes: performing a current prediction on the corresponding beliefs to generate a current candidate action sequence and obtain the free energy of the current candidate action sequence; performing a next prediction on the corresponding beliefs to generate a next candidate action sequence and obtain the free energy of the next candidate action sequence; wherein the first preset number of candidate actions in the current candidate action sequence are used for the next prediction, and the free energy of the first preset number of candidate actions decreases successively.

[0012] In some embodiments, the current prediction of the corresponding belief to generate a current candidate action sequence and obtain the free energy of the current candidate action sequence includes: using Gaussian distribution to currently sample the corresponding belief to obtain a first candidate action subsequence, and using a policy network to currently sample the corresponding belief to obtain a second candidate action subsequence; the first candidate action subsequence and the second candidate action subsequence form the current candidate action sequence; through the belief generation model, the corresponding free energy preference score and the belief error score are obtained to obtain the corresponding free energy as the free energy of the current candidate action sequence; wherein the first preset number of candidate actions in the current candidate action sequence are used to update the mean of the Gaussian distribution of the next prediction.

[0013] The third aspect of the present application discloses a method for video diffusion, which is applied to a mobile device, wherein the mobile device is connected to at least one server, and comprises: determining a video diffusion task; sending the video diffusion task to the at least one server to determine a computational offloading and resource allocation strategy for the video diffusion task; performing computational offloading and resource allocation operations based on the computational offloading and resource allocation strategy for the video diffusion task, so that the at least one server returns a video corresponding to the video diffusion task to the mobile device in accordance with the computational offloading and resource allocation strategy for the video diffusion task; wherein the computational offloading and resource allocation strategy for the video diffusion task is determined by the computational offloading and resource allocation method described in the second aspect.

[0014] The fourth aspect of the present application discloses an electronic device, comprising a memory and a processor coupled to each other, wherein the processor is used to execute program instructions stored in the memory to implement the belief generation model training method described in the first aspect, or to implement the computational offloading and resource allocation method described in the second aspect, or to implement the video diffusion method described in the third aspect.

[0015] The fifth aspect of the present application discloses a non-volatile computer-readable storage medium having program instructions stored thereon. When the program instructions are executed by a processor, the method for training the belief generation model described in the first aspect is implemented, or the method for computational offloading and resource allocation described in the second aspect is implemented, or the method for video diffusion described in the third aspect is implemented.

[0016] The beneficial effects of the present application include: obtaining multiple experiences, each of which includes an experience data set and an experience priority corresponding to the experience data set, determining a target experience from the multiple experiences based on the experience priorities of the multiple experiences, inputting the target experience into a belief generation model for training, and obtaining a belief generation model, wherein the belief generation model is used for computational offloading and resource allocation of video diffusion tasks, thereby improving data utilization in reinforcement learning. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] The present application will be further described below in conjunction with the accompanying drawings and implementation methods, in which:

[0018] Figure 1 It is a flowchart of a training method of a belief generation model in an embodiment of the present application;

[0019] Figure 2 It is a structural diagram of a belief generation model in an embodiment of the present application;

[0020] Figure 3 is a schematic diagram of belief generation model training in an embodiment of the present application;

[0021] Figure 4 is a schematic diagram of belief generation model training in one embodiment of the present application;

[0022] Figure 5 It is a flowchart of a method for computing offloading and resource allocation according to an embodiment of the present application;

[0023] Figure 6 is a schematic diagram of a process for determining a target action according to an embodiment of the present application;

[0024] Figure 7 It is a flowchart of a method for computing offloading and resource allocation according to an embodiment of the present application;

[0025] Figure 8 is a schematic diagram of a scenario of a video diffusion method according to an embodiment of the present application;

[0026] Fig. 9 is a schematic diagram of the structure of an electronic device according to an embodiment of the present application;

[0027] Fig.10 It is a schematic diagram of the structure of the non-volatile computer-readable storage medium of an embodiment of the present application. DETAILED DESCRIPTION

[0028] Reference to "embodiments" in this application means that a particular feature, structure, or characteristic described in conjunction with the embodiments may be included in at least one embodiment of the present application. The appearance of the phrase in various locations in the specification does not necessarily refer to the same embodiment, nor is it an independent or alternative embodiment that is mutually exclusive with other embodiments. It is explicitly and implicitly understood by those skilled in the art that the embodiments described herein may be combined with other embodiments.

[0029] The term "and / or" in this application is only a description of the association relationship of associated objects, indicating that there may be three relationships. For example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone. In addition, the character " / " in this article generally indicates that the objects associated before and after are in an "or" relationship. In addition, "many" in this article means two or more than two. In addition, the term "at least one" in this article means any combination of at least two of any one or more of a plurality of, for example, including at least one of A, B, and C, and can mean including any one or more elements selected from the set consisting of A, B, and C. In addition, the terms "first", "second", and "third" in this application are used for descriptive purposes only and cannot be understood as indicating or implying relative importance or implicitly indicating the number of indicated technical features.

[0030] The video diffusion task is a task that uses diffusion models to achieve video generation, video editing or other video understanding. With the increasing popularity and demand for text-to-video generation applications on mobile devices, resource-constrained mobile terminals face the difficulty of insufficient computing power. With the rise of IoT cloud / edge computing, users' video diffusion tasks can be offloaded to the cloud / edge for computing and returning the results. In the process, how to design reasonable computing offloading and resource allocation strategies is the key. The former mainly offloads a large number of tasks to different cloud / edge servers on demand, while the latter requires coordinated scheduling of resources for each cloud / edge server. At present, reinforcement learning (RL) can be used to optimize its own decision-making by using the continuous interaction between the agent and the environment. However, traditional RL algorithms have low data utilization, usually requiring a large amount of interaction data to learn effective strategies, and in actual environments, the task load may change over time, and traditional RL algorithms need to be retrained or adjusted to adapt to new environmental conditions. These problems affect the performance of the system on the one hand, requiring the system to continuously collect data and update in real time; on the other hand, they also affect the deployment of cloud / edge applications and user experience.

[0031] Therefore, the present application provides a belief generation model training method, a computation offloading and resource allocation method, a video diffusion method, and related equipment.

[0032] In order to enable those skilled in the art to better understand the technical solution of the present application, the technical solution of the present application is further described in detail below in conjunction with the accompanying drawings and specific implementation methods.

[0033] See also Figure 1 , Figure 1 1 is a flow chart of a training method for a belief generation model according to an embodiment of the present application. The execution subject of the method may be a mobile device with computing functions, such as a microcomputer, a server, and a mobile device such as a laptop computer and a tablet computer.

[0034] It should be noted that if there are substantially the same results, the method of the present application is not limited to Figure 1 The process sequence shown is limited.

[0035] In some possible implementations, the method may be implemented by a processor calling computer-readable instructions stored in a memory, such as Figure 1 As shown, the method may include the following steps:

[0036] S11: Acquire multiple experiences, where each experience includes an experience data set and an experience priority corresponding to the experience data set.

[0037] Get multiple experiences, for example, get data set D, which includes n experiences, data set Among them, (ξ i ,g i )(1≤i≤n) represents an experience, that is, the data of a series of continuous interactions between the agent and the environment. Each experience (ξ i ,g i ) includes the empirical data set ξ i and the empirical data set ξ i Corresponding experience priority g i , g i Expressed as the free-energy-based sampling priority for the current data.

[0038] S12: Based on the experience priority of multiple experiences, determine the target experience from multiple experiences.

[0039] Temporarily store the multiple experiences obtained in the experience pool. For example, store n experiences {(ξ1,g1),(ξ2,g2),…,(ξ n ,g n )} is cached to a preset playback buffer. Based on the experience priorities of multiple experiences, the target experience is determined from multiple experiences. For example, among n experiences {(ξ1,g1),(ξ2,g2),…,(ξ n ,g n )}, based on the corresponding experience priority g i Sampling is performed to select one experience from n experiences as the target experience. For example, g2>g1, the experience (ξ2, g2) corresponding to the experience priority g2 is preferentially selected as the target experience for model training.

[0040] S13: Input the target experience into the belief generation model for training to obtain the belief generation model, wherein the belief generation model is used for computational offloading and resource allocation of the video diffusion task.

[0041] The target experience is input into the belief generation model to train the belief generation model, where the belief generation model is used for computational offloading and resource allocation of video diffusion tasks. For example, a reasonable computational offloading and resource allocation strategy can be determined based on the output results of the belief generation model to realize the on-demand offloading of a large number of tasks to different server ends, while coordinating and scheduling the resources of each server end.

[0042] In this embodiment, multiple experiences are obtained, each of which includes an experience data set and an experience priority corresponding to the experience data set. Based on the experience priorities of the multiple experiences, a target experience is determined from the multiple experiences, and the target experience is input into a belief generation model for training to obtain a belief generation model, wherein the belief generation model is used for computational offloading and resource allocation of video diffusion tasks, thereby improving data utilization in reinforcement learning.

[0043] In some embodiments, the empirical data set includes: first empirical data, used to characterize server information of computational offloading and policy information of resource allocation; second empirical data, used to characterize environmental information of a server connected to a mobile device, wherein the server connected to the mobile device includes at least one of a cloud server and an edge server; and third empirical data, used to characterize a corresponding free energy return, wherein the free energy return represents a return for measuring a belief generation model.

[0044] Empirical Datasetξ i Including first experience data Second empirical data And the third experience data, the first experience data That is, a series of continuous computing offload server information and resource allocation strategy information, where a i (1≤i≤m) can represent the actions corresponding to the server information of the offloaded computation and the resource allocation strategy. Environmental information for characterizing a server connected to a mobile device, wherein the server connected to the mobile device includes at least one of a cloud server and an edge server, wherein o i (1≤i≤n) may include the joint state of each server running the cloud computing network model and / or the edge computing network model (ie, the cloud / edge computing network model). The third empirical data is the corresponding free energy return R, which represents the reward for measuring the belief generation model.

[0045] Servers connected to mobile devices include cloud servers and edge servers. A cloud server is a server based on cloud computing, usually deployed in a remote data center, and provides services over the network. An edge server is a server based on edge computing, deployed at the edge of the network, that is, located closer to the user or device, and used to process and store data close to the user to reduce data transmission delays and network congestion. For the sake of simplicity, the term "cloud / edge server" may appear in this article, which can be used to represent at least one of multiple cloud servers and edge servers.

[0046] See also Figure 2-Figure 3 , Figure 2 is a schematic diagram of the structure of the belief generation model of the embodiment of the present application, Figure 3 It is a schematic diagram of the belief generation model training of an embodiment of the present application.

[0047] In some embodiments, the belief generation model (Generative Model) includes a belief encoder, a dynamic transfer model, and a belief error model.

[0048] Input the target experience into the belief generation model, for example, the target experience is (ξ i ,gi ), (ξ i ,g i ) includes the first empirical data Second empirical data And the corresponding third experience data, wherein the second experience data includes the current second experience data and the next second experience data. For example, the target experience data set ξ i Include {o t ,A t,T ,R t,T ,o t,T}, where o t A is the current second empirical data, that is, the joint state of each server in the current cloud / edge computing network model; t,T Server information for current computing offload and resource allocation policy information; R t,T is the corresponding free energy return; o t,T is the next second experience data, that is, the joint state of each server of the cloud / edge computing network model after executing the current strategy, such as the next second experience data o t,T It may include multiple consecutive joint states, ie, the joint states of each server after the corresponding policies are executed consecutively.

[0049] Using the belief encoder, the current second experience data o in the target experience t Encoded as the current prior belief s t , and the next second experience data o in the target experience t,T Encoded as the next prior belief s', that is, the joint state o of the cloud / edge computing network model is converted into an internal belief that the agent can understand. Using the dynamic transfer model, the current prior belief s t The current first experience data A in the target experience t,T Make a prediction to get the next posterior belief s, that is, jointly predict the intrinsic belief and action generated by the belief encoder to infer the future belief. Using the belief error model, based on the preset loss function, calculate the error between the next prior belief s' and the next posterior belief s to get the belief error score (belieferror) corresponding to the current prior belief, that is, learn the belief error between the intrinsic belief generated by the belief encoder and the future belief generated by the dynamic transfer model. Among them, with repeated generative training, the overall belief error gradually decreases to ensure that the data is effectively used, so there is no need to interact with the environment many times.

[0050] like Figure 3 As shown, the belief encoder is used to encode the current second experience data o t Encode and get the current prior belief s t, using EMA (exponential moving average, exponential moving average), and using a prior encoder to the next second experience data o t,T {o t+1 ,o t+2 ,…o n} is encoded to obtain the next prior belief s'{s′ t+1 ,s′ t+2 ,…s′ n}. Using the dynamic transfer model (dynamics), the current prior belief s t The current first experience data A in the target experience t,T {a t ,a t+1 ,…a n} to make predictions and obtain the next posterior belief s{s t ,s t+1 ,s t+2 ,…s n}. Further, using the belief error model, based on the preset loss function (mse loss), the next prior belief s'{s′ t+1 ,s′ t+2 ,…s′ n} and the next posterior belief s{s t ,s t+1 ,s t+2 ,…s n} to obtain the belief error score {be t ,be t+1 ,be t+2 ,…be n}.

[0051] In this embodiment, the belief generation model encodes high-dimensional external state information into low-dimensional internal beliefs through a belief encoder so that the agent can reason, wherein the belief can support the agent in long-term planning optimization strategy and reduce the dependence on environmental interaction and data collection. At the same time, the dynamic transfer model is used to predict the state of future time steps and minimize the model error to ensure the time consistency of belief encoding and transfer, thereby obtaining an efficient belief generation model, thereby significantly improving the data utilization of the model.

[0052] Please continue reading Figure 2-Figure 3 In some embodiments, the belief generation model (Generative Model) includes a belief encoder, a dynamic transfer model and a belief error model, as well as a free energy preference model.

[0053] The free energy preference model is used to calculate the free energy preference score (prefe-rencestate) corresponding to the prior belief. For example, the current prior belief s t and the next prior belief {s t+1 ,s t+2 ,…s n} are sequentially input into the free energy preference model to obtain the current prior belief s t The corresponding free energy preference score pref t and the next prior belief {s t+1 ,s t+2 ,…s n}corresponding free energy preference score {pref t ,pref t+1 ,pref t+2 ,…pref n}.

[0054] Further, in some embodiments, the free energy corresponding to the current prior belief is calculated using the belief error score corresponding to the current prior belief and the free energy preference score corresponding to the current prior belief; and the experience priority in the target experience is updated using the free energy corresponding to the current prior belief.

[0055] Use the belief error score be corresponding to the current prior belief i The free energy preference score pref corresponding to the current prior belief i , calculate the free energy G corresponding to the current prior belief, and then use the free energy G corresponding to the current prior belief to update the experience priority g in the target experience i .

[0056] The free energy can be expressed as follows:

[0057]

[0058] In this embodiment, the corresponding belief error score and free energy preference score are combined, and the minimization of free energy is used as the optimization criterion to balance the decision-making process of the intelligent agent, so that the intelligent agent can actively explore and learn when facing dynamic environmental changes, rather than just pursuing short-term high returns, thereby effectively reducing the sensitivity of the intelligent agent to the instability of the dynamic environment and improving the stability and adaptability of the intelligent agent in complex environments.

[0059] In some embodiments, the third experience data includes the final scores corresponding to the mobile device and the server and the belief error of the belief generation model. Obtaining the final scores corresponding to the mobile device and the server includes: based on the video diffusion task, taking the device information of the mobile device and the device information of the server as variables, constructing a data transmission function and a task calculation function between the mobile device and the corresponding server; weighting the data transmission function and the task calculation function to obtain a final score function, wherein the final score function is added with a preset function constraint and is used to determine the final score corresponding to the mobile device and the server.

[0060] The device information of the mobile device may include the task offloading size, task offloading time, CPU frequency, etc. of the mobile device, and the device information of the server may include the remaining computing resources, available channels and relative positions of the cloud / edge server, etc. The device information of the mobile device and the device information of the cloud / edge server are used as variables to construct the data transmission function and task calculation function between the mobile device and the corresponding cloud / edge server. It can be understood that the task offloading size, task offloading time, CPU frequency of the mobile device and the remaining computing resources, available channels and relative positions of the cloud / edge server are used as computing task offloading strategy variables, and based on the video diffusion model of the cloud / edge server, the data transmission model and task calculation model between the mobile device and the cloud / edge server are constructed.

[0061] Among them, the data transmission function between the mobile device and the corresponding server can be expressed as:

[0062]

[0063] Among them, W i represents the i-th transmission channel assigned by the server to the user equipment, P i and G i They represent the transmission power in the ith subchannel and the information gain of the ith subchannel respectively. N represents the noise coefficient, σ 2 Represents background noise.

[0064] The data transmission function and the task calculation function are weighted to obtain a final score function, wherein the final score function is added with a preset function constraint and is used to determine the final scores corresponding to the mobile device and the server, which is expressed as follows:

[0065]

[0066] Among them, P(T) represents the final score, L represents the task transmission calculation time, and VQ is the video quality score returned by the task calculation model. Task offloading and resource allocation need to meet the above constraints, where constraint 1 represents the remaining bandwidth of server s at time t. Memory The resource cannot be less than 0; Constraint 2 means that the computing time of the task cannot be greater than the maximum limit time of the system; Constraint 3 means that at time t, the channel bandwidth resources in the s server cannot exceed the maximum channel bandwidth resources; Constraint 4 means that at time t, the number of users carried by each transmission sub-channel of the server cannot exceed the maximum limit.

[0067] For ease of understanding, the training method of the belief generation model in the embodiment of the present application is illustrated by way of example. Figure 4 As shown, Figure 4 Schematic diagram of belief generation model training in one embodiment of the present application, where multiple acquired experiences are temporarily stored in a playback buffer, each experience including an experience data set ξ i The empirical priority g corresponding to the empirical data set i , select an experience data from multiple experiences as the target experience by random sampling, for example, select g i =2.0 empirical data to train the belief generation model, encode the high-dimensional external state information into low-dimensional internal belief s based on the belief generation model, and then update the network parameter θ, calculate the corresponding free energy G, and update the empirical priority g according to the calculation result of the free energy i That is to say, this application samples data through free energy guidance, that is, the distribution of data sampling is adjusted through the concept of free energy, and data with larger belief errors are preferentially selected for training, so that the intelligent agent can preferentially learn external data that differs greatly from the current belief, while increasing the priority of new sampled data, thereby quickly reducing the error of the belief model and enhancing the adaptability and training efficiency of the model in a dynamic environment.

[0068] Furthermore, by introducing free energy into the objective function, the present application achieves the coordination of the estimation error and the estimation preference, thereby balancing the adaptive exploration and utilization of the environment. At the same time, by using the belief generation model to predict the future state, the ability to make long-term balanced decisions on the environment is achieved, ensuring the superiority of the system in terms of performance and adaptability.

[0069] See also Figure 5 , Figure 5 1 is a flow chart of a method for computing offloading and resource allocation according to an embodiment of the present application. The method can be applied to a mobile device with computing and other functions, and the mobile device is connected to at least one server. It should be noted that if there is substantially the same result, the method of the present application is not limited to Figure 5 The process sequence shown is limited.

[0070] In some possible implementations, the method may be implemented by a processor calling computer-readable instructions stored in a memory, such as Figure 5 As shown, the method may include the following steps:

[0071] S51: The mobile device obtains environment information of the mobile device and at least one server from at least one server.

[0072] The mobile device obtains environmental information of the mobile device and at least one server from at least one server, for example, the mobile device obtains environmental information of the mobile device and the cloud server from a cloud server, or the mobile device obtains environmental information of the mobile device and the edge server from an edge server.

[0073] The environmental information of the mobile device and at least one server may include the task offloading size, task offloading time, CPU frequency of the mobile device, and the remaining computing resources, available channels and relative positions of the server, etc. After preprocessing, a set of environmental data is obtained, including the joint status of each server of the cloud / edge computing network model. t .

[0074] S52: using the belief generation model, converting the environmental information of the mobile device and at least one server to obtain corresponding beliefs.

[0075] The belief generation model is used to convert the environmental information of the mobile device and at least one server to obtain the corresponding belief. For example, the belief generation model is used to convert the joint state of each server of the current cloud / edge computing network model into t Transformed into corresponding beliefs t .

[0076] The belief generation model is obtained through the above-mentioned belief generation model training method.

[0077] S53: Predict the corresponding beliefs to generate multiple candidate action sequences and obtain free energy of the multiple candidate action sequences, wherein each candidate action in the multiple candidate action sequences represents a candidate strategy for computation offloading and resource allocation.

[0078] The corresponding belief t Predictions are made to generate multiple candidate action sequences, each of which includes n candidate actions, and the free energy G of the multiple candidate action sequences is calculated, where each candidate action in the multiple candidate action sequences can be used to represent a candidate strategy for computational offloading and resource allocation, that is, the belief s output by the belief generation model t Predictions are made to obtain multiple candidate strategies, which can choose to offload tasks to edge servers with average performance but short distance or to cloud servers with high performance but long distance.

[0079] S54: Based on the free energy of multiple candidate action sequences, determine a target action from the multiple candidate action sequences, wherein the target action represents a target strategy for computational offloading and resource allocation, and is an action in the candidate action sequence with the smallest free energy among the multiple candidate action sequences.

[0080] Based on the free energy G of multiple candidate action sequences, a target action is determined from the multiple candidate action sequences, and the target action is the action in the candidate action sequence with the smallest free energy G among the multiple candidate action sequences, for example, the target action is the first action in the candidate action sequence with the smallest free energy G among the multiple candidate action sequences. The target action represents a target strategy for computation offloading and resource allocation, that is, based on the calculation result of the free energy, the target strategy is selected from the multiple candidate strategies.

[0081] In this embodiment, a mobile device obtains environmental information of the mobile device and at least one server from at least one server, converts the environmental information of the mobile device and at least one server using a belief generation model to obtain corresponding beliefs, predicts the corresponding beliefs to generate multiple candidate action sequences, and calculates the free energy of the multiple candidate action sequences. Furthermore, based on the free energy of the multiple candidate action sequences, a target action is determined from the multiple candidate action sequences. By adopting a free energy-guided action decision mechanism, data exploration and utilization are optimized, thereby improving the adaptability of the intelligent agent in a dynamic environment and the efficiency of computational offloading and resource allocation.

[0082] In some embodiments, corresponding beliefs are predicted to generate multiple candidate action sequences and obtain free energies of multiple candidate action sequences, including: performing current predictions on corresponding beliefs to generate current candidate action sequences and obtain free energies of current candidate action sequences; performing next predictions on corresponding beliefs to generate next candidate action sequences and obtain free energies of next candidate action sequences; wherein the first preset number of candidate actions in the current candidate action sequence are used for the next prediction, and the free energies of the first preset number of candidate actions decrease successively.

[0083] The corresponding belief is currently predicted to generate the current candidate action sequence and obtain the free energy of the current candidate action sequence, that is, the free energy of the candidate actions in the current candidate action sequence is calculated. At this time, the first preset number of candidate actions in the current candidate action sequence are used to participate in the next strategy prediction, and the free energy of the first preset number of candidate actions decreases in sequence, for example, the k actions with the lowest free energy are selected in the current action sequence. Further, using the information of the first preset number of candidate actions in the current candidate action sequence, the corresponding belief is predicted again to generate the next candidate action sequence and obtain the free energy of the next candidate action sequence, so as to realize the process of continuously iteratively optimizing the prediction strategy based on the free energy of the strategy and improve the adaptability of the intelligent agent in a dynamic environment.

[0084] In some embodiments, current predictions are made on corresponding beliefs to generate a current candidate action sequence and obtain the free energy of the current candidate action sequence, including: using Gaussian distribution to currently sample corresponding beliefs to obtain a first candidate action subsequence, and using a policy network to currently sample corresponding beliefs to obtain a second candidate action subsequence; the first candidate action subsequence and the second candidate action subsequence are combined into a current candidate action sequence; through a belief generation model, corresponding free energy preference scores and belief error scores are obtained to obtain corresponding free energy as the free energy of the current candidate action sequence; wherein, the first preset number of candidate actions in the current candidate action sequence are used to update the mean of the Gaussian distribution of the next prediction.

[0085] The corresponding belief is sampled by Gaussian distribution to obtain the first candidate action subsequence, that is, n candidate actions are predicted and generated. The corresponding belief is sampled by policy network to obtain the second candidate action subsequence, that is, n candidate actions are predicted and generated, and the first candidate action subsequence and the second candidate action subsequence are combined into the current candidate action sequence. Furthermore, through the above belief generation model, the free energy preference score pref and the belief error score be corresponding to the action strategy are obtained to obtain the corresponding free energy G as the free energy G of the current candidate action sequence. At this time, the first preset number of candidate actions in the current candidate action sequence are used to update the mean of the Gaussian distribution of the next prediction, for example, the first k candidate actions in the current candidate action sequence are used to update the mean of the Gaussian distribution of the next prediction, and the candidate actions in the current candidate action sequence are arranged from small to large according to free energy.

[0086] For ease of understanding, the method of computing offloading and resource allocation in the embodiment of the present application is illustrated by way of example. Figure 6 As shown, Figure 6 The mobile device obtains the environment information of the mobile device and at least one server from at least one server, and uses the belief generation model to convert the current joint state of each server into the state of the target action. t Transformed into corresponding beliefs t , and update the corresponding network parameters θ, predict the corresponding beliefs to generate multiple candidate action sequences, for example, by using the model prediction path integral method to perform n-step prediction, that is, combining Gaussian sampling and policy networks to generate candidate action sequences and generate corresponding beliefs And calculate the corresponding strategy free energy {G1,G2,…G K-1 ,G K…}, and select the mean of the first k actions with the lowest free energy G as the Gaussian mean of the next iteration. Through multiple iterations to optimize the action distribution, the first action of the optimal action sequence can be selected as the target action, that is, executed with the decision of the current time step.

[0087] See also Figure 7 , Figure 7 1 is a flow chart of a method for computing offloading and resource allocation according to an embodiment of the present application. The method can be applied to a mobile device with computing and other functions, and the mobile device is connected to at least one server. It should be noted that if there is substantially the same result, the method of the present application is not limited to Figure 7 The process sequence shown is limited.

[0088] In some possible implementations, the method may be implemented by a processor calling computer-readable instructions stored in a memory, such as Figure 7 As shown, the method may include the following steps:

[0089] S71: Determine a video diffusion task.

[0090] Determine the video tasks to be performed on the mobile device, such as video generation, video editing or other video understanding tasks, and generate corresponding video diffusion tasks.

[0091] S72: Send the video diffusion task to at least one server to determine a strategy for computation offloading and resource allocation of the video diffusion task.

[0092] For example, the mobile device sends the video diffusion task to at least one server, and then obtains the environmental information of the mobile device and at least one server, so as to determine the computation offloading and resource allocation strategy of the video diffusion task through the above-mentioned computation offloading and resource allocation method.

[0093] S73: Based on the strategy of computation offloading and resource allocation of the video diffusion task, perform computation offloading and resource allocation operations, so that at least one server returns the video corresponding to the video diffusion task to the mobile device according to the strategy of computation offloading and resource allocation of the video diffusion task.

[0094] The mobile device performs corresponding computing offloading and resource allocation operations based on the determined computing offloading and resource allocation strategy. For example, the mobile device sends the specific content of the video task, such as text, pictures, etc., to at least one server determined in the strategy, so that at least one server returns the corresponding pictures or videos of the video diffusion task to the mobile device in accordance with the computing offloading and resource allocation strategy of the video diffusion task.

[0095] For ease of understanding, the application scenario of the video diffusion method of the embodiment of the present application is illustrated by example. Figure 8 As shown, Figure 8 The present invention is a scene diagram of the video diffusion method of an embodiment of the present application. The mobile device is connected to at least one server. When the mobile device sends the video diffusion task to at least one server, the mobile device obtains the environmental information of the mobile device and at least one server, and then uses the above-mentioned method of computational offloading and resource allocation to determine the computational offloading and resource allocation strategy of the video diffusion task on the mobile device side, such as determining a target server from at least one server to perform computational offloading. After determining the target server, the mobile device sends information such as text and pictures corresponding to the video task to the target server. For example, a user request is sent: "(1) Tom Cruise's face reflects concentration, and his eyes are full of goals and motivation; (2) A child swings excitedly on a rusty swing and laughs loudly". The target server responds, and the video generation model (for example, VideoCrafter2) generates pictures (1) and (2) based on the user request, and returns pictures (1) and (2) to the mobile device.

[0096] Those skilled in the art will appreciate that, in the above method of specific implementation, the order in which the steps are written does not imply a strict execution order and does not constitute any limitation on the implementation process. The specific execution order of the steps should be determined by their functions and possible internal logic.

[0097] See also Fig. 9 , Fig. 9 : is a schematic diagram of the structure of an electronic device of an embodiment of the present application. The electronic device 90 includes a memory 91 and a processor 92 coupled to each other, and the processor 92 is used to execute the program instructions stored in the memory 91 to implement the steps of the above-mentioned belief generation model training method embodiment, or to implement the steps of the above-mentioned calculation offloading and resource allocation method embodiment, or to implement the steps of the above-mentioned video diffusion method embodiment. In a specific implementation scenario, the electronic device 90 may include but is not limited to: a microcomputer, a server, which is not limited here.

[0098] Specifically, the processor 92 is used to control itself and the memory 91 to implement the steps of the training method embodiment of the belief generation model, or to implement the steps of the method embodiment of the above-mentioned calculation offloading and resource allocation, or to implement the steps of the method embodiment of the above-mentioned video diffusion. The processor 92 can also be called a CPU (Central Processing Unit), and the processor 92 may be an integrated circuit chip with signal processing capabilities. The processor 92 can also be a general-purpose processor, a digital signal processor (Digital Signal Processor, DSP), an application-specific integrated circuit (Application Specific Integrated Circuit, ASIC), a field-programmable gate array (Field-Programmable Gate Array, FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components. The general-purpose processor can be a microprocessor or the processor can also be any conventional processor, etc. In addition, the processor 92 can be implemented by an integrated circuit chip.

[0099] See also Fig.10 , Fig.10 1 is a schematic diagram of the structure of a non-volatile computer-readable storage medium according to an embodiment of the present application. The non-volatile computer-readable storage medium 100 is used to store a computer program 1001. When the computer program 1001 is executed by a processor, for example, the above-mentioned Fig. 9 When the processor 92 in the embodiment is executed, it is used to implement the steps of the above-mentioned training method embodiment for the belief generation model, or to implement the steps of the above-mentioned method embodiment for computational offloading and resource allocation, or to implement the steps of the above-mentioned method embodiment for video diffusion.

[0100] The above description of various embodiments tends to emphasize the differences between the various embodiments. The same or similar aspects can be referenced to each other, and for the sake of brevity, they will not be repeated herein.

[0101] In the several embodiments provided in the present application, it should be understood that the disclosed methods and related devices can be implemented in other ways. For example, the above-described related device implementations are only schematic, for example, the division of modules or units is only a logical function division, and there may be other division methods in actual implementation, such as units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication disconnection shown or discussed can be through some interfaces, indirect coupling or communication disconnection of devices or units, which can be electrical, mechanical or other forms.

[0102] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0103] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application is essentially or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions to enable a computer device (which can be a personal computer, server, or network device, etc.) or a processor (processor) to perform all or part of the steps of each implementation method of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk and other media that can store program code.

[0104] It is easy for a person skilled in the art to know that many modifications and changes can be made to the device and method while maintaining the teaching content of the present application. Therefore, the above disclosure should be considered as limited only by the scope of the appended claims.

Claims

1. A training method for a belief generation model, characterized in that: Applicable to mobile devices, including: Acquire multiple experiences, wherein each experience includes an experience data set and an experience priority corresponding to the experience data set; Determining a target experience from among the multiple experiences based on the experience priorities of the multiple experiences; The target experience is input into the belief generation model for training to obtain the belief generation model, wherein the belief generation model is used for computational offloading and resource allocation of video diffusion tasks.

2. The method according to claim 1, characterized in that The empirical data set includes: First experience data, used to characterize server information of computing offload and policy information of resource allocation; second experience data, used to characterize environmental information of a server connected to the mobile device, wherein the server connected to the mobile device includes at least one of a cloud server and an edge server; and The third empirical data is used to characterize the corresponding free energy return, where the free energy return represents a return for measuring the belief generation model.

3. The method according to claim 1 or 2, characterized in that: The second experience data in the target experience includes current second experience data and next second experience data; The belief generation model includes a belief encoder, a dynamic transfer model and a belief error model; The step of inputting the target experience into the belief generation model for training includes: Using the belief encoder, encoding the current second experience data in the target experience as the current prior belief, and encoding the next second experience data in the target experience as the next prior belief; Using the dynamic transfer model, predicting the current prior belief with the current first experience data in the target experience to obtain the next posterior belief; The belief error model is used to calculate the error between the next priori belief and the next posteriori belief based on a preset loss function to obtain a belief error score corresponding to the current priori belief.

4. The method according to claim 3, characterized in that The belief generation model also includes a free energy preference model; The step of inputting the target experience into the belief generation model for training further includes: The current prior belief and the next prior belief are sequentially input into the free energy preference model to obtain a free energy preference score corresponding to the current prior belief and a free energy preference score corresponding to the next prior belief, respectively.

5. The method according to claim 4, characterized in that Also includes: Calculating the free energy corresponding to the current prior belief by using the belief error score corresponding to the current prior belief and the free energy preference score corresponding to the current prior belief; The free energy corresponding to the current prior belief is used to update the experience priority in the target experience.

6. The method according to claim 2, characterized in that The third experience data includes the final scores of the mobile device and the server and the belief error of the belief generation model; Obtaining the final scores corresponding to the mobile device and the server, including: Based on the video diffusion task, taking the device information of the mobile device and the device information of the server as variables, constructing a data transmission function and a task calculation function between the mobile device and the corresponding server; The data transmission function and the task calculation function are weighted to obtain a final score function, wherein the final score function is added with a preset function constraint and is used to determine the final scores corresponding to the mobile device and the server.

7. A method for computing offloading and resource allocation, characterized in that: Applied to a mobile device, the mobile device is connected to at least one server, comprising: The mobile device obtains, from the at least one server, environmental information of the mobile device and the at least one server; Using a belief generation model, converting environmental information of the mobile device and the at least one server to obtain corresponding beliefs; Predicting the corresponding beliefs to generate a plurality of candidate action sequences and obtaining free energies of the plurality of candidate action sequences, wherein each candidate action in the plurality of candidate action sequences represents a candidate strategy for computation offloading and resource allocation; and Based on the free energies of the multiple candidate action sequences, determining a target action from the multiple candidate action sequences, wherein the target action represents a target strategy for computation offloading and resource allocation and is an action in the candidate action sequence with the smallest free energy among the multiple candidate action sequences; Wherein, the belief generation model is obtained by the belief generation model training method described in any one of claims 1-6.

8. The method according to claim 7, characterized in that The predicting of the corresponding beliefs to generate a plurality of candidate action sequences and obtaining free energies of the plurality of candidate action sequences includes: Performing current prediction on the corresponding belief to generate a current candidate action sequence and obtaining the free energy of the current candidate action sequence; Making a next prediction for the corresponding belief to generate a next candidate action sequence and obtaining the free energy of the next candidate action sequence; Among them, the first preset number of candidate actions in the current candidate action sequence are used for the next prediction, and the free energies of the first preset number of candidate actions decrease in sequence.

9. The method according to claim 8, characterized in that The current prediction of the corresponding belief to generate a current candidate action sequence and obtain the free energy of the current candidate action sequence includes: Using Gaussian distribution, currently sampling the corresponding belief to obtain a first candidate action subsequence, and using a policy network, currently sampling the corresponding belief to obtain a second candidate action subsequence; Combining the first candidate action subsequence and the second candidate action subsequence into the current candidate action sequence; Obtaining corresponding free energy preference scores and belief error scores through the belief generation model to obtain corresponding free energy as the free energy of the current candidate action sequence; The first preset number of candidate actions in the current candidate action sequence are used to update the mean of the Gaussian distribution of the next prediction.

10. A method for video diffusion, characterized in that: Applied to a mobile device, the mobile device is connected to at least one server, comprising: Determine the video diffusion task; Sending the video diffusion task to the at least one server to determine a strategy for computation offloading and resource allocation of the video diffusion task; Based on the strategy of computing offloading and resource allocation of the video diffusion task, performing computing offloading and resource allocation operations, so that the at least one server returns the video corresponding to the video diffusion task to the mobile device according to the strategy of computing offloading and resource allocation of the video diffusion task; The strategy of computational offloading and resource allocation of the video diffusion task is determined by the method of computational offloading and resource allocation described in any one of claims 7 to 9.

11. An electronic device, characterized in that: It includes a memory and a processor coupled to each other, and the processor is used to execute program instructions stored in the memory to implement the training method of the belief generation model described in any one of claims 1 to 6, or to implement the method of computational offloading and resource allocation described in any one of claims 7 to 9, or to implement the method of video diffusion described in claim 10.

12. A non-volatile computer-readable storage medium having program instructions stored thereon, characterized in that: When the program instructions are executed by the processor, they implement the training method of the belief generation model described in any one of claims 1 to 6, or the method of computational offloading and resource allocation described in any one of claims 7 to 9, or the method of video diffusion described in claim 10.

Images