Model training method and device
By generating cross-task graphs and updating diffusion model parameters, the problem of inefficiency in existing diffusion models in multi-task prediction is solved, and more efficient multi-task prediction capabilities are achieved.
Patent Information
- Application Number
- CN202411901657.5
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-12-23
- Publication Date
- 2025-04-11
AI Technical Summary
Existing diffusion models are inefficient in prediction in multiple intensive prediction tasks, requiring training of multiple models to meet the needs of different tasks.
By inputting the training image into the image encoder to extract features, input multiple task modules to generate cross-task maps, obtain prediction results for different tasks based on the cross-task maps and features, and update the diffusion model parameters.
The prediction efficiency of the diffusion model in multi-task scenarios is improved, and the requirements of different intensive prediction tasks can be met, reducing the complexity of model training.
Smart Images
Figure CN119810589A_ABST
Abstract
Description
Technical Field
[0001] The present application belongs to the field of artificial intelligence technology, and specifically relates to a model training method and device thereof. Background Art
[0002] In the related art, dense prediction tasks such as semantic segmentation and depth estimation are usually implemented using dense prediction methods based on diffusion models. However, for multiple different dense prediction tasks, multiple corresponding diffusion models need to be trained to meet the prediction requirements of different dense prediction tasks.
[0003] It can be seen that the diffusion model in the related art has the problem of low prediction efficiency. Summary of the invention
[0004] The purpose of the embodiments of the present application is to provide a model training method and a device thereof, which can solve the problem of low prediction efficiency existing in the diffusion model in the related art.
[0005] In order to solve the above technical problems, this application is implemented as follows:
[0006] In a first aspect, an embodiment of the present application provides a model training method, comprising:
[0007] The training image is input into the image encoder to extract the first feature of the training image at the i-th layer. The i-th layer is a target coding network layer in the image encoder;
[0008] The first feature Input into T different task modules respectively to obtain the second feature T is an integer greater than 1, and t∈{1,2,…,T};
[0009] Generate a cross-task graph based on the T task labels corresponding to the T different task modules;
[0010] Based on the inter-task relationship graph corresponding to the cross-task graph and the second feature Get the prediction results of T different tasks;
[0011] Obtaining loss function values of the T different tasks based on the prediction results of the T different tasks;
[0012] The parameters of the diffusion model are updated based on the loss function values of the T different tasks to obtain an updated diffusion model, where the diffusion model is used to perform information prediction on the intensive prediction task.
[0013] In a second aspect, an embodiment of the present application provides a model training device, comprising:
[0014] A processing module is used to input the training image into the image encoder and extract the first feature of the training image at the i-th layer. The i-th layer is a target coding network layer in the image encoder;
[0015] The processing module is further configured to convert the first feature Input into T different task modules respectively to obtain the second feature T is an integer greater than 1, and t∈{1,2,…,T};
[0016] A generation module, used for generating a cross-task graph based on the T task labels corresponding to the T different task modules;
[0017] The processing module is further configured to: Get the prediction results of T different tasks;
[0018] An acquisition module, used for acquiring loss function values of T different tasks based on the prediction results of the T different tasks;
[0019] A training update module is used to update the parameters of the diffusion model based on the loss function values of the T different tasks to obtain an updated diffusion model, wherein the diffusion model is used to perform information prediction on the intensive prediction task.
[0020] In a third aspect, an embodiment of the present application provides an electronic device, which includes a processor and a memory, wherein the memory stores programs or instructions that can be executed on the processor, and when the program or instructions are executed by the processor, the steps of the method described in the first aspect are implemented.
[0021] In a fourth aspect, an embodiment of the present application provides a readable storage medium, on which a program or instruction is stored, and when the program or instruction is executed by a processor, the steps of the method described in the first aspect are implemented.
[0022] In a fifth aspect, an embodiment of the present application provides a chip, comprising a processor and a communication interface, wherein the communication interface is coupled to the processor, and the processor is used to run a program or instruction to implement the method described in the first aspect.
[0023] In a sixth aspect, an embodiment of the present application provides a computer program product, comprising computer instructions, which, when executed by a processor, implement the method described in the first aspect.
[0024] In the embodiment of the present application, during the training process of the diffusion model, a cross-task graph is generated, and based on the task relationship graph corresponding to the cross-task graph and the second feature By performing result prediction, we can get the prediction results of different tasks, and then based on the prediction results of T different tasks, we can get the loss function values of different tasks, so that we can update the parameters of the diffusion model based on the obtained loss function values, and realize the training of the diffusion model, so that the trained diffusion model can meet the prediction requirements of different prediction tasks, and effectively improve the prediction efficiency of the diffusion model. BRIEF DESCRIPTION OF THE DRAWINGS
[0025] Figure 1 is a flow chart of the model training method provided in an embodiment of the present application;
[0026] Figure 2a It is a link diagram of hard parameter sharing provided in an embodiment of the present application;
[0027] Figure 2b It is a link diagram of soft parameter sharing provided in an embodiment of the present application;
[0028] Figure 3 is a structural block diagram of a diffusion model provided by an embodiment of the present application;
[0029] Figure 4 is a structural diagram of a model training device provided in an embodiment of the present application;
[0030] Figure 5 It is one of the structural diagrams of the electronic device provided in the embodiment of the present application;
[0031] Figure 6 This is the second structural diagram of the electronic device provided in the embodiment of the present application. DETAILED DESCRIPTION
[0032] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.
[0033] The terms "first", "second", etc. in the specification and claims of the present application are used to distinguish similar objects, and are not used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable under appropriate circumstances, so that the embodiments of the present application can be implemented in an order other than those illustrated or described here, and the objects distinguished by "first", "second", etc. are generally of one type, and the number of objects is not limited. For example, the first object can be one or more. In addition, "and / or" in the specification and claims represents at least one of the connected objects, and the character " / " generally indicates that the objects associated with each other are in an "or" relationship.
[0034] The model training method provided in the embodiment of the present application is described in detail below through specific embodiments and their application scenarios in conjunction with the accompanying drawings.
[0035] See also Figure 1 , Figure 1 is a flow chart of a model training method provided in an embodiment of the present application. Figure 1 As shown, the following steps are included:
[0036] Step 101: input a training image into an image encoder, and extract the first feature of the training image at the i-th layer.
[0037] The i-th layer is the target coding network layer in the image encoder, and the first feature can be expressed as the first feature
[0038] For example, when the image encoder is an encoder including a 12-layer encoding network, the value of i can be set to 3, 6, 9, 12; that is, the first feature of the training image in the third layer of the encoding network can be extracted. The first feature of the network encoded at layer 6 The first feature of the network encoded at layer 9 And the first feature of the network encoded at layer 12
[0039] Moreover, the above-mentioned first feature can be understood as a conventional feature or a general feature of the training image, such as a semantic feature, a depth feature, etc. of the training image.
[0040] In some embodiments, the above-mentioned image encoder can be a shared encoder, for example, different encoder paths can be set for different tasks to meet the needs of multi-task prediction. This sharing method is called hard parameter sharing; and, in contrast to hard parameter sharing, there is soft parameter sharing that uses different encoders but shares information between modules.
[0041] Among them, the link diagram of the above hard parameter sharing can be as follows Figure 2a As shown, the link diagram of the above soft parameter sharing can be as follows Figure 2b shown.
[0042] Step 102: Input the first feature into T different task modules respectively to obtain the second feature.
[0043] Where T is an integer greater than 1, and t∈{1,2,…,T}, the above second feature can be expressed as the second feature
[0044] The above-mentioned task modules can be understood as functional modules included in the diffusion model. For example, when the diffusion model includes prediction capabilities such as semantic segmentation and depth estimation, the above-mentioned task modules can be understood as semantic segmentation modules, depth estimation modules, etc.
[0045] For example, the first feature The semantic segmentation module and the depth estimation module are input respectively to obtain the corresponding speech segmentation features and depth estimation features.
[0046] In some embodiments, the task module may include two stacked convolution modules.
[0047] Step 103: Generate a cross-task graph based on the T task labels corresponding to the T different task modules.
[0048] Different task modules may be set with different task labels, that is, corresponding task labels may be set for T different task modules, and a corresponding cross-task graph may be generated based on the T task labels to associate different tasks.
[0049] Step 104: Based on the task relationship graph corresponding to the cross-task graph and the second feature Get the prediction results of T different tasks.
[0050] In this step, a corresponding task relationship graph can be generated based on the cross-task graph to associate different tasks, and then based on the task relationship graph and the second feature Get the prediction results of T different tasks.
[0051] Step 105: Obtain loss function values of the T different tasks based on the prediction results of the T different tasks.
[0052] In this step, the loss function values of different tasks can be obtained based on the prediction results of T different tasks, and then the parameters of the diffusion model can be updated based on the obtained loss function values to achieve the training of the diffusion model.
[0053] Step 106: Update the parameters of the diffusion model based on the loss function values of the T different tasks to obtain an updated diffusion model.
[0054] The diffusion model is used to perform information prediction on intensive prediction tasks.
[0055] In this embodiment, during the training of the diffusion model, a cross-task graph is generated, and based on the task relationship graph corresponding to the cross-task graph and the second feature By performing result prediction, we can get the prediction results of different tasks, and then based on the prediction results of T different tasks, we can get the loss function values of different tasks, so that we can update the parameters of the diffusion model based on the obtained loss function values, and realize the training of the diffusion model, so that the trained diffusion model can meet the prediction requirements of different prediction tasks, and effectively improve the prediction efficiency of the diffusion model.
[0056] In some embodiments, the task relationship graph corresponding to the cross-task graph and the second feature Get prediction results for T different tasks, including:
[0057] Obtaining an inter-task relationship graph corresponding to the cross-task graph;
[0058] Based on the inter-task relationship graph and the second feature Multiply to get the task fusion feature
[0059] The corresponding task features of different layers of the target task are fused After fusion processing, the prediction result of the target task is obtained.
[0060] Wherein, obtaining the inter-task relationship graph corresponding to the cross-task graph includes:
[0061] Performing noise processing on the cross-task graph based on Gaussian noise to obtain a noisy cross-task graph;
[0062] Performing mapping processing on the noisy cross-task graph to obtain the mapped noisy cross-task graph;
[0063] Function processing is performed on the mapped noisy cross-task graph to obtain an inter-task relationship graph.
[0064] In this embodiment, based on the task relationship diagram and the second feature The product of , we get the task fusion feature To associate different tasks.
[0065] In some embodiments, the cross-task graph can be denoised by a function γ(s) that controls the signal-to-noise ratio, and this process can be formulated as:
[0066]
[0067] Among them, ∈ represents Gaussian noise; l s represents the noisy cross-task graph, that is, the cross-task graph after noise processing; l s The s in represents the random sampling time, and s∈{1,2,…,S}, and S is the total number of denoising steps of the diffusion model.
[0068] Moreover, the noise cross-task graph and the second feature They can be sent together to the cross-task diffusion decoder for joint denoising processing to obtain the prediction results of the corresponding tasks.
[0069] For the noisy cross-task graph, we can first use two consecutive multi-layer convolution modules to map it, and finally map its channel number to T 2 ; And the Sigmoid function can be used to process the mapped noisy cross-task graph to obtain the corresponding task relationship graph A i .
[0070] Among them, the relationship graph between tasks can be obtained by and the second feature of the corresponding layer Multiply them together to get the task fusion feature This process can be formulated as:
[0071] In some embodiments, the task fusion feature It can be sent to the subsequent layer interaction stage. Specifically, the task fusion features corresponding to different layers of the same task can be first The concatenation is performed along the channel dimension, and a convolution module is used to map the concatenated features to M with N channels. t ; Wherein, the number of i in step 101 is N, if the value of i is 3, 6, 9, or 12, then the value of N is 4.
[0072] According to the mapped features, the task fusion features of different layers can be fused on the layer. This process can be formulated as: Moreover, the features after layer fusion are passed through three multi-layer convolution modules to obtain the prediction results of different tasks.
[0073] In some embodiments, generating a cross-task graph based on the T task labels corresponding to the T different task modules includes:
[0074] Encoding the T task labels corresponding to the T different task modules into a continuous feature space through a cross-task label encoder;
[0075] The T task labels in the continuous feature space are concatenated to obtain a cross-task graph.
[0076] In this embodiment, by splicing the T task labels in the continuous feature space and obtaining a cross-task graph, a correlation relationship between different tasks can be established, so that the diffusion model that updates parameters based on the loss function value obtained from the task relationship graph of the cross-task graph can meet the prediction requirements of different prediction tasks and effectively improve the prediction efficiency of the diffusion model.
[0077] For example, you can set the T task labels corresponding to T different task modules to L t , where t∈{1,2,…,T}, so as to encode the task labels representing different tasks into the continuous feature space through the cross-task label encoder encoding.
[0078] Specifically, the cross-task label encoder can include two layers of convolution, where the first layer is a convolution unique to different tasks. Moreover, when inputting, discrete labels, such as labels for semantic segmentation and human parsing tasks, are represented by one-hot codes, while continuous labels are normalized and input into their corresponding first layer of convolution. After obtaining the first encoded feature L′ t Afterwards, these features are concatenated along the channel dimension and fed into the second convolutional module to obtain the cross-task graph l.
[0079] Among them, the cross-task graph l can be normalized to [-1, +1], and then it can be multiplied by a hyperparameter to control its value range.
[0080] In some embodiments, updating the parameters of the diffusion model based on the loss function values of the T different tasks to obtain an updated diffusion model includes:
[0081] Determining weighted loss function values of the T different tasks based on the loss function values of the T different tasks;
[0082] The parameters of the diffusion model are updated based on the weighted loss function value to obtain an updated diffusion model.
[0083] In this embodiment, the loss function values of T different tasks can be weightedly calculated to obtain weighted loss function values of T different tasks, and then the parameters of the diffusion model can be updated based on the weighted loss function values, thereby improving the accuracy of information prediction of the diffusion model for intensive prediction tasks.
[0084] like Figure 3 As shown, an embodiment of the present application provides a structural block diagram of a diffusion model, which includes an image encoder 31 and a diffusion decoder 32, and the diffusion decoder includes a cross-task label encoder, a cross-task graph and a cross-task diffusion decoder.
[0085] In some embodiments, the training image may be first input into the image encoder 31 to extract the first feature of the training image at the i-th layer. Then the first feature can be Input into T different task modules respectively to obtain the second feature
[0086] In some embodiments, L1, L2, ..., L T The T task labels are encoded into a unified continuous feature space, and the T task labels are concatenated in the continuous feature space to obtain a cross-task graph l; then, the cross-task graph l can be denoised by Gaussian noise ∈, and the noisy cross-task graph l is obtained. s .
[0087] In some embodiments, the cross-task graph may be s And the second feature The input is sent to the cross-task diffusion decoder for joint denoising to obtain the prediction results of different tasks.
[0088] During the training process of the diffusion model, the diffusion model can be denoised by a step-by-step denoising method, which specifically includes the following steps:
[0089] Step 201: The second feature can be obtained in the same manner as step 101. Then it is mixed with Gaussian noise, which is also the noisy cross-task graph l in the Sth step S Input the cross-task diffusion decoder to obtain the prediction result after denoising the Gaussian noise
[0090] Step 202: Use a cross-task label encoder to encode the labels corresponding to different tasks. Encode into a unified continuous feature space to obtain the encoded denoising features
[0091] Specifically, the previous step noise is first predicted based on the encoded features Specifically, the formula Calculated, where γ(S) is the signal-to-noise ratio in the Sth step.
[0092] Moreover, after predicting the noise at this step, we can use the denoising diffusion implicit model to Together with γ(s′) and the predicted noise, we can calculate the noisy cross-task graph l of the previous step. s′ , where s′ is the time step several steps ago to be predicted, and this process can be formalized as:
[0093]
[0094] In this way, by transforming the noisy cross-task graph l s′ And the second feature Input the cross-task diffusion decoder to obtain the prediction result after a new round of denoising
[0095] Step 203, repeat the process of step 202 until the time step 1 without noise is predicted. At this time, the diffusion model predicts is the final prediction result.
[0096] In an embodiment of the present application, the relationship between tasks is explicitly modeled through a diffusion model, which effectively improves the pre-efficiency of multi-task intensive prediction; moreover, by using a joint denoising process, the efficiency of multi-round denoising in a multi-task scenario is improved; in addition, the cumbersome encoding problem of different tasks can also be solved.
[0097] The model training method of the embodiment of the present application is to input the training image into the image encoder to extract the first feature of the training image at the i-th layer. The i-th layer is the target coding network layer in the image encoder; the first feature Input into T different task modules respectively to obtain the second feature T is an integer greater than 1, and t∈{1,2,…,T}; based on the T task labels corresponding to the T different task modules, a cross-task graph is generated; based on the task relationship graph corresponding to the cross-task graph and the second feature The prediction results of T different tasks are obtained; based on the prediction results of the T different tasks, the loss function values of the T different tasks are obtained; based on the loss function values of the T different tasks, the parameters of the diffusion model are updated to obtain an updated diffusion model, and the diffusion model is used to perform information prediction on the intensive prediction task. In this way, the prediction efficiency of the diffusion model can be improved.
[0098] The model training method provided in the embodiment of the present application can be executed by a model training device. In the embodiment of the present application, the model training device executing the model training method is taken as an example to illustrate the model training device provided in the embodiment of the present application.
[0099] See also Figure 4 , Figure 4 is a structural diagram of the model training device provided in the embodiment of the present application, such as Figure 4 As shown, the model training device 400 includes:
[0100] Processing module 401 is used to input the training image into the image encoder and extract the first feature of the training image at the i-th layer. The i-th layer is a target coding network layer in the image encoder;
[0101] The processing module 401 is further configured to convert the first feature Input into T different task modules respectively to obtain the second feature T is an integer greater than 1, and t∈{1,2,…,T};
[0102] A generating module 402, configured to generate a cross-task graph based on the T task labels corresponding to the T different task modules;
[0103] The processing module 401 is further configured to: Get the prediction results of T different tasks;
[0104] An acquisition module 403 is used to acquire loss function values of T different tasks based on the prediction results of the T different tasks;
[0105] The training update module 404 is used to update the parameters of the diffusion model based on the loss function values of the T different tasks to obtain an updated diffusion model, and the diffusion model is used to perform information prediction on the intensive prediction task.
[0106] Optionally, the processing module 401 is specifically configured to:
[0107] Obtaining an inter-task relationship graph corresponding to the cross-task graph;
[0108] Based on the inter-task relationship graph and the second feature Multiply to get the task fusion feature
[0109] The corresponding task features of different layers of the target task are fused After fusion processing, the prediction result of the target task is obtained.
[0110] Optionally, the processing module 401 is specifically configured to:
[0111] Performing noise processing on the cross-task graph based on Gaussian noise to obtain a noisy cross-task graph;
[0112] Performing mapping processing on the noisy cross-task graph to obtain the mapped noisy cross-task graph;
[0113] Function processing is performed on the mapped noisy cross-task graph to obtain an inter-task relationship graph.
[0114] Optionally, the generating module 402 is specifically configured to:
[0115] Encoding the T task labels corresponding to the T different task modules into a continuous feature space through a cross-task label encoder;
[0116] The T task labels in the continuous feature space are concatenated to obtain a cross-task graph.
[0117] Optionally, the training update module 404 is specifically used to:
[0118] Determining weighted loss function values of the T different tasks based on the loss function values of the T different tasks;
[0119] The parameters of the diffusion model are updated based on the weighted loss function value to obtain an updated diffusion model.
[0120] The model training device in the embodiment of the present application can be an electronic device, or a component in an electronic device, such as an integrated circuit or a chip. The electronic device can be a terminal, or other devices other than a terminal. Exemplary, the electronic device can be a mobile phone, a tablet computer, a laptop computer, a PDA, a vehicle-mounted electronic device, a mobile Internet device (Mobile Internet Device, MID), an augmented reality (Augmented Reality, AR) / virtual reality (Virtual Reality, VR) device, a robot, a wearable device, a super mobile personal computer (Ultra-Mobile Personal Computer, UMPC), a netbook or a personal digital assistant (Personal Digital Assistant, PDA), etc., and can also be a network attached storage (Network Attached Storage, NAS), a personal computer (Personal Computer, PC), a television (Television, TV), a teller machine or a self-service machine, etc., and the embodiment of the present application is not specifically limited.
[0121] The model training device in the embodiment of the present application may be a device having an operating system. The operating system may be an Android operating system, an iOS operating system, or other possible operating systems, which are not specifically limited in the embodiment of the present application.
[0122] The model training device provided in the embodiment of the present application can achieve Figure 1 To avoid repetition, the various processes implemented by the method embodiment are not described here.
[0123] See also Figure 5 , Figure 5 This is one of the structural diagrams of the electronic device provided in the embodiment of the present application. Figure 5 As shown, an embodiment of the present application also provides an electronic device 500, including a processor 501 and a memory 502, wherein the memory 502 stores programs or instructions that can be executed on the processor 501, and when the program or instructions are executed by the processor 501, the various steps of the above-mentioned model training method embodiment are implemented, and the same technical effect can be achieved. To avoid repetition, they are not described here.
[0124] It should be noted that the electronic devices in the embodiments of the present application include the mobile electronic devices and non-mobile electronic devices mentioned above.
[0125] See also Figure 6 , Figure 6 This is the second structural diagram of the electronic device provided in the embodiment of the present application. Figure 6 As shown, the electronic device 600 includes but is not limited to: a radio frequency unit 601, a network module 602, an audio output unit 603, an input unit 604, a sensor 605, a display unit 606, a user input unit 607, an interface unit 608, a memory 609, and a processor 610 and other components.
[0126] Those skilled in the art will appreciate that the electronic device 600 may also include a power source (such as a battery) for supplying power to each component, and the power source may be logically connected to the processor 610 through a power management system, thereby implementing functions such as managing charging, discharging, and power consumption management through the power management system. Figure 6 The electronic device structure shown in the figure does not constitute a limitation on the electronic device. The electronic device may include more or fewer components than shown in the figure, or combine certain components, or arrange the components differently, which will not be described in detail here.
[0127] The processor 610 is configured to:
[0128] The training image is input into the image encoder to extract the first feature of the training image at the i-th layer. The i-th layer is a target coding network layer in the image encoder;
[0129] The first feature Input into T different task modules respectively to obtain the second feature T is an integer greater than 1, and t∈{1,2,…,T};
[0130] Generate a cross-task graph based on the T task labels corresponding to the T different task modules;
[0131] Based on the inter-task relationship graph corresponding to the cross-task graph and the second feature Get the prediction results of T different tasks;
[0132] Obtaining loss function values of the T different tasks based on the prediction results of the T different tasks;
[0133] The parameters of the diffusion model are updated based on the loss function values of the T different tasks to obtain an updated diffusion model, where the diffusion model is used to perform information prediction on the intensive prediction task.
[0134] Optionally, the processor 610 is configured to:
[0135] Obtaining an inter-task relationship graph corresponding to the cross-task graph;
[0136] Based on the inter-task relationship graph and the second feature Multiply to get the task fusion feature
[0137] The corresponding task features of different layers of the target task are fused After fusion processing, the prediction result of the target task is obtained.
[0138] Optionally, the processor 610 is configured to:
[0139] Performing noise processing on the cross-task graph based on Gaussian noise to obtain a noisy cross-task graph;
[0140] Performing mapping processing on the noisy cross-task graph to obtain the mapped noisy cross-task graph;
[0141] Function processing is performed on the mapped noisy cross-task graph to obtain an inter-task relationship graph.
[0142] Optionally, the processor 610 is configured to:
[0143] Encoding the T task labels corresponding to the T different task modules into a continuous feature space through a cross-task label encoder;
[0144] The T task labels in the continuous feature space are concatenated to obtain a cross-task graph.
[0145] Optionally, the processor 610 is configured to:
[0146] Determining weighted loss function values of the T different tasks based on the loss function values of the T different tasks;
[0147] The parameters of the diffusion model are updated based on the weighted loss function value to obtain an updated diffusion model.
[0148] It should be understood that in the embodiment of the present application, the input unit 604 may include a graphics processor (GPU) 6041 and a microphone 6042, and the graphics processor 6041 processes the image data of a static picture or video obtained by an image capture device (such as a camera) in a video capture mode or an image capture mode. The display unit 606 may include a display panel 6061, and the display panel 6061 may be configured in the form of a liquid crystal display, an organic light emitting diode, etc. The user input unit 607 includes a touch panel 6071 and at least one of other input devices 6072. The touch panel 6071 is also called a touch screen. The touch panel 6071 may include two parts: a touch detection device and a touch controller. Other input devices 6072 may include, but are not limited to, a physical keyboard, function keys (such as a volume control button, a switch button, etc.), a trackball, a mouse, and a joystick, which will not be repeated here.
[0149] The memory 609 can be used to store software programs and various data. The memory 609 may mainly include a first storage area for storing programs or instructions and a second storage area for storing data, wherein the first storage area may store an operating system, an application program or instructions required for at least one function (such as a sound playback function, an image playback function, etc.), etc. In addition, the memory 609 may include a volatile memory or a non-volatile memory, or the memory 609 may include both volatile and non-volatile memories. Among them, the non-volatile memory may be a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), or a flash memory. The volatile memory may be a random access memory (RAM), a static random access memory (SRAM), a dynamic random access memory (DRAM), a synchronous dynamic random access memory (SDRAM), a double data rate synchronous dynamic random access memory (DDRSDRAM), an enhanced synchronous dynamic random access memory (ESDRAM), a synchronous link dynamic random access memory (SLDRAM) and a direct memory bus random access memory (DRRAM). The memory 609 in the embodiment of the present application includes but is not limited to these and any other suitable types of memory.
[0150] The processor 610 may include one or more processing units; optionally, the processor 610 integrates an application processor and a modem processor, wherein the application processor mainly processes operations related to an operating system, a user interface, and application programs, and the modem processor mainly processes wireless communication signals, such as a baseband processor. It is understandable that the modem processor may not be integrated into the processor 610.
[0151] An embodiment of the present application also provides a readable storage medium, on which a program or instruction is stored. When the program or instruction is executed by a processor, each process of the above-mentioned model training method embodiment is implemented, and the same technical effect can be achieved. To avoid repetition, it will not be repeated here.
[0152] The processor is the processor in the electronic device described in the above embodiment. The readable storage medium includes a computer readable storage medium, such as a computer read-only memory ROM, a random access memory RAM, a magnetic disk or an optical disk.
[0153] An embodiment of the present application further provides a chip, which includes a processor and a communication interface, wherein the communication interface is coupled to the processor, and the processor is used to run programs or instructions to implement the various processes of the above-mentioned model training method embodiment, and can achieve the same technical effect. To avoid repetition, it will not be repeated here.
[0154] It should be understood that the chip mentioned in the embodiments of the present application can also be called a system-level chip, a system chip, a chip system or a system-on-chip chip, etc.
[0155] An embodiment of the present application provides a computer program product, including computer instructions. When the computer instructions are executed by a processor, the various processes of the above-mentioned model training method embodiment are implemented, and the same technical effect can be achieved. To avoid repetition, they are not repeated here.
[0156] It should be noted that, in this article, the terms "comprise", "include" or any other variant thereof are intended to cover non-exclusive inclusion, so that the process, method, article or device including a series of elements includes not only those elements, but also includes other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise one..." do not exclude the presence of other identical elements in the process, method, article or device including the element. In addition, it should be pointed out that the scope of the methods and devices in the embodiments of the present application is not limited to performing functions in the order shown or discussed, and may also include performing functions in a substantially simultaneous manner or in reverse order according to the functions involved, for example, the described method may be performed in an order different from that described, and various steps may also be added, omitted, or combined. In addition, the features described with reference to certain examples may be combined in other examples.
[0157] Through the description of the above implementation methods, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of software plus a necessary general hardware platform, and of course by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present application, or the part that contributes to the prior art, can be embodied in the form of a computer product, which is stored in a storage medium (such as ROM / RAM, a disk, or an optical disk), and includes a number of instructions for a terminal (which can be a mobile phone, a computer, a server, or a network device, etc.) to execute the methods described in each embodiment of the present application.
[0158] The embodiments of the present application are described above in conjunction with the accompanying drawings, but the present application is not limited to the above-mentioned specific implementation methods. The above-mentioned specific implementation methods are merely illustrative and not restrictive. Under the guidance of the present application, ordinary technicians in this field can also make many forms without departing from the purpose of the present application and the scope of protection of the claims, all of which are within the protection of the present application.
Claims
1. A model training method, characterized in that: include: The training image is input into the image encoder to extract the first feature of the training image at the i-th layer. The i-th layer is a target coding network layer in the image encoder; The first feature Input into T different task modules respectively to obtain the second feature T is an integer greater than 1, and t∈{1,2,…,T}; Generate a cross-task graph based on the T task labels corresponding to the T different task modules; Based on the inter-task relationship graph corresponding to the cross-task graph and the second feature Get the prediction results of T different tasks; Obtaining loss function values of the T different tasks based on the prediction results of the T different tasks; The parameters of the diffusion model are updated based on the loss function values of the T different tasks to obtain an updated diffusion model, where the diffusion model is used to perform information prediction on the intensive prediction task.
2. The method according to claim 1, characterized in that The inter-task relationship graph corresponding to the cross-task graph and the second feature Get prediction results for T different tasks, including: Obtaining an inter-task relationship graph corresponding to the cross-task graph; Based on the inter-task relationship graph and the second feature Multiply to get the task fusion feature The corresponding task features of different layers of the target task are fused After fusion processing, the prediction result of the target task is obtained.
3. The method according to claim 2, characterized in that The obtaining of the inter-task relationship graph corresponding to the cross-task graph includes: Performing noise processing on the cross-task graph based on Gaussian noise to obtain a noisy cross-task graph; Performing mapping processing on the noisy cross-task graph to obtain the mapped noisy cross-task graph; Function processing is performed on the mapped noisy cross-task graph to obtain an inter-task relationship graph.
4. The method according to any one of claims 1 to 3, characterized in that The generating a cross-task graph based on the T task labels corresponding to the T different task modules includes: Encoding the T task labels corresponding to the T different task modules into a continuous feature space through a cross-task label encoder; The T task labels in the continuous feature space are concatenated to obtain a cross-task graph.
5. The method according to any one of claims 1 to 3, characterized in that The updating of the parameters of the diffusion model based on the loss function values of the T different tasks to obtain an updated diffusion model includes: Determining weighted loss function values of the T different tasks based on the loss function values of the T different tasks; The parameters of the diffusion model are updated based on the weighted loss function value to obtain an updated diffusion model.
6. A model training device, characterized in that: include: A processing module is used to input the training image into the image encoder and extract the first feature of the training image at the i-th layer. The i-th layer is a target coding network layer in the image encoder; The processing module is further configured to convert the first feature Input into T different task modules respectively to obtain the second feature T is an integer greater than 1, and t∈{1,2,…,T}; A generation module, used for generating a cross-task graph based on the T task labels corresponding to the T different task modules; The processing module is further configured to: Get the prediction results of T different tasks; An acquisition module, used for acquiring loss function values of T different tasks based on the prediction results of the T different tasks; A training update module is used to update the parameters of the diffusion model based on the loss function values of the T different tasks to obtain an updated diffusion model, wherein the diffusion model is used to perform information prediction on the intensive prediction task.
7. The device according to claim 6, characterized in that The processing module is specifically used for: Obtaining an inter-task relationship graph corresponding to the cross-task graph; Based on the inter-task relationship graph and the second feature Multiply to get the task fusion feature The corresponding task features of different layers of the target task are fused After fusion processing, the prediction result of the target task is obtained.
8. The device according to claim 7, characterized in that The processing module is specifically used for: Performing noise processing on the cross-task graph based on Gaussian noise to obtain a noisy cross-task graph; Performing mapping processing on the noisy cross-task graph to obtain the mapped noisy cross-task graph; Function processing is performed on the mapped noisy cross-task graph to obtain an inter-task relationship graph.
9. The device according to any one of claims 6 to 8, characterized in that The generating module is specifically used for: Encoding the T task labels corresponding to the T different task modules into a continuous feature space through a cross-task label encoder; The T task labels in the continuous feature space are concatenated to obtain a cross-task graph.
10. The device according to any one of claims 6 to 8, characterized in that The training update module is specifically used for: Determining weighted loss function values of the T different tasks based on the loss function values of the T different tasks; The parameters of the diffusion model are updated based on the weighted loss function value to obtain an updated diffusion model.