Model training method and device, courseware element layout method and device and storage medium

Abstract

The invention provides a courseware element layout-based model training method, a courseware element layout method, a courseware element layout device and a computer storage medium. The model training method comprises the steps that to-be-trained courseware layout data and real courseware layout data are acquired, and the courseware layout data comprise element positions of courseware elements; inputting the to-be-trained courseware layout data into a to-be-trained courseware layout model, and obtaining a predicted element position of each courseware element output by the courseware layout model; and training the courseware layout model based on the predicted element position and the real element position of each courseware element until a training condition is obtained. Through the model training method, the consideration of element layout when a worker makes the courseware is simplified, and the courseware making efficiency is greatly improved.

Description

Technical Field

[0001] The present application relates to the technical field of page layout, and in particular to a model training method based on courseware element layout, a courseware element layout method, a courseware element layout device and a computer storage medium. Background Art

[0002] In the process of making PPT, staff often encounter situations where they don’t know how to layout the elements, and the layout of elements plays an important role in the aesthetics of PPT.

[0003] The current solution is mainly to preset templates, then extract the corresponding templates according to the template requirements, and fill the elements into the templates. This solution requires preset templates, so a large number of different templates need to be made for different PPT pages, which is very time-consuming and labor-intensive. At the same time, the templates made may not completely match the content that needs to be filled in, resulting in more spaces or overlapping content in some places. Summary of the invention

[0004] In order to solve the above technical problems, the present application proposes a model training method based on courseware element layout, a courseware element layout method, a courseware element layout device and a computer storage medium.

[0005] In order to solve the above technical problems, the present application proposes a model training method based on courseware element layout, and the model training method includes:

[0006] Obtaining the layout data of the courseware to be trained and the layout data of the real courseware, wherein the courseware layout data includes the element position of each courseware element;

[0007] Inputting the to-be-trained courseware layout data into the to-be-trained courseware layout model, and obtaining the predicted element position of each courseware element output by the courseware layout model;

[0008] Based on the predicted element position and the actual element position of each courseware element, the courseware layout model is trained until the training conditions are obtained.

[0009] The element position is composed of element coordinates and element size, wherein the element coordinates are the upper left corner coordinates, lower right corner coordinates, or center coordinates of the courseware element.

[0010] Wherein, the step of obtaining the predicted element position of each courseware element output by the courseware layout model includes:

[0011] Obtaining the position noise predicted by the courseware layout model;

[0012] The position noise is superimposed on the position of the element to be trained to obtain and output the predicted element position.

[0013] Wherein, the model training method further includes:

[0014] Constructing a courseware layout model to be trained based on a diffusion model, wherein the diffusion model is DDPM, D3PM, or VQ-Diffusion;

[0015] The noise predictor in the courseware layout model is used to predict the position noise of the courseware layout data to be trained.

[0016] Wherein, after training the courseware layout model until the training conditions are obtained, the model training method further comprises:

[0017] Get the current courseware layout model that meets the training conditions for the current batch;

[0018] Obtain the target courseware layout model by obtaining historical batch training;

[0019] Input the courseware layout data to be tested into the current courseware layout model and the target courseware layout model respectively, and obtain the current loss value of the current courseware layout model and the target loss value of the target courseware layout model;

[0020] When the current loss value is less than the target loss value, the current courseware layout model is updated to the latest target courseware layout model.

[0021] The courseware layout data further includes element categories of each courseware element, wherein the element categories include pictures, audio, video, tables, and / or text.

[0022] In order to solve the above technical problems, the present application also proposes a courseware element layout method, which includes:

[0023] Obtaining a pre-trained courseware layout model, wherein the courseware layout model is trained by the above-mentioned model training method;

[0024] The courseware to be laid out is input into the courseware layout model, and the optimized courseware output by the courseware layout model is obtained.

[0025] Wherein, after obtaining the optimized courseware output by the courseware layout model, the courseware element layout method further includes:

[0026] Aligning courseware elements of the same element category in the optimized courseware in a preset direction;

[0027] And / or, the distance between adjacent courseware elements of the same element category in the optimized courseware is set to a preset value.

[0028] In order to solve the above-mentioned technical problems, the present application also proposes a courseware element layout device, which includes a memory and a processor coupled to the memory; wherein the memory is used to store program data, and the processor is used to execute the program data to implement the model training method and / or courseware element layout method as mentioned above.

[0029] In order to solve the above technical problems, the present application also proposes a computer storage medium, which is used to store program data. When the program data is executed by a computer, it is used to implement the above model training method and / or courseware element layout method.

[0030] Compared with the prior art, the beneficial effects of the present application are as follows: the courseware element layout device obtains the courseware layout data to be trained and the real courseware layout data, wherein the courseware layout data includes the element position of each courseware element; the courseware layout data to be trained is input into the courseware layout model to be trained, and the predicted element position of each courseware element output by the courseware layout model is obtained; based on the predicted element position and the real element position of each courseware element, the courseware layout model is trained until the training conditions are obtained. Through the above-mentioned model training method, the staff's consideration of element layout when making courseware is simplified, which greatly improves the efficiency of courseware production. BRIEF DESCRIPTION OF THE DRAWINGS

[0031] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0032] in:

[0033] Figure 1 It is a flowchart of an embodiment of a model training method based on courseware element layout provided by the present application;

[0034] Figure 2 It is a flowchart of an embodiment of a courseware element layout method provided by the present application;

[0035] Figure 3 It is a structural diagram of an embodiment of a courseware element layout device provided by the present application;

[0036] Figure 4 It is a structural diagram of an embodiment of a computer storage medium provided by the present application. DETAILED DESCRIPTION

[0037] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0038] The terms "first", "second", "third", "fourth", etc. (if any) in the specification and claims of the present application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present application described herein can, for example, be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units that are clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0039] Please refer to Figure 1 , Figure 1 It is a flow chart of an embodiment of a model training method based on courseware element layout provided in this application.

[0040] The courseware element layout method of the present application is applied to a courseware element layout device, wherein the courseware element layout device of the present application can be a server, a terminal device, or a system composed of a server and a terminal device. Accordingly, the various parts included in the courseware element layout device, such as various units, sub-units, modules, and sub-modules, can all be set in the server, can all be set in the terminal device, or can be set in the server and the terminal device respectively.

[0041] Furthermore, the above-mentioned server can be hardware or software. When the server is hardware, it can be implemented as a distributed server cluster consisting of multiple servers, or it can be implemented as a single server. When the server is software, it can be implemented as multiple software or software modules, such as software or software modules used to provide distributed servers, or it can be implemented as a single software or software module, which is not specifically limited here.

[0042] like Figure 1 As shown, the specific steps are as follows:

[0043] Step S11: Obtain the layout data of the courseware to be trained and the layout data of the real courseware, wherein the courseware layout data includes the element position of each courseware element.

[0044] In an embodiment of the present application, before model training, the present application first constructs a data set, specifically including a training data set and a verification data set.

[0045] Specifically, the staff first obtains a batch of original PPT courseware, and then obtains a batch of beautiful PPT courseware through manual adjustment. The courseware element layout device extracts the element category c and element position (x, y, w, h) in the original PPT courseware and the beautiful PPT courseware. Among them, (w, h) is the size of the courseware element, and (x, y) is a certain coordinate point of the courseware element, including but not limited to: upper left coordinate point, lower left coordinate point, upper right coordinate point, lower right coordinate point, center coordinate point, etc. Element category c includes but is not limited to: pictures, audio, video, tables, text, etc.

[0046] The original PPT courseware extracts the layout data of the courseware to be trained or the layout data of the courseware to be verified, and the beautiful PPT courseware extracts the real courseware layout data. The courseware element layout device forms data pairs with the layout data of the courseware to be trained and its corresponding real courseware layout data. All layout data pairs form a data set, and the data set is divided into a training data set and a verification data set according to a ratio of 4:1.

[0047] Then, the courseware element layout device builds the courseware layout model to be trained. This application adopts a model solution based on diffusion (diffusion model), which specifically includes but is not limited to: DDPM (probability diffusion model), D3PM (discrete denoising diffusion probability model), VQ-Diffusion (vector quantization diffusion model), etc.

[0048] Step S12: inputting the to-be-trained courseware layout data into the to-be-trained courseware layout model, and obtaining the predicted element position of each courseware element output by the courseware layout model.

[0049] In the embodiment of the present application, the courseware element layout device inputs the courseware layout data to be trained into the courseware layout model constructed in step S11, so as to predict the predicted element position of the optimized courseware layout through the courseware layout model.

[0050] Specifically, the training process of the courseware layout model of the present application can be understood as training a noise predictor using a layout generation model. The generation model can generate new samples based on the original samples. For example, in the field of image processing, the generation model can generate new images based on the original image by adding or eliminating noise. In the field of courseware layout of the present application, the noise predictor can continuously add noise to the element position (x, y, w, h) of each courseware element in the forward prediction process of the courseware layout model, that is, obtain position noise, which can be understood as the position offset distance.

[0051] Furthermore, the noise predictor removes the noise in the reverse process of prediction, that is, after the position noise is predicted by the noise predictor, the position noise is subtracted or added to the input original element position to obtain the predicted element position after denoising.

[0052] Step S13: Based on the predicted element position and the actual element position of each courseware element, the courseware layout model is trained until the training conditions are obtained.

[0053] In an embodiment of the present application, the courseware element layout device calculates the loss, i.e., the loss value, using the position deviation between the predicted element position and the actual element position of each courseware element obtained in step S12, and uses the loss value to evaluate the training effect of the courseware layout model. The smaller the loss, the better the training effect.

[0054] Furthermore, the courseware element layout device iterates the courseware layout model through the above process in the training data set, and tests the training effect of the current courseware layout model in the validation data set every N training epochs, that is, obtains the loss of the courseware layout model on the validation data set at each epoch. Epoch means that during training, all training data sets are trained once.

[0055] Specifically, the courseware layout model after each epoch verification needs to be compared with the courseware layout model trained in the previous epoch to obtain the best courseware layout model, that is, the courseware layout model with the smallest loss. If the loss of the courseware layout model after subsequent epoch verification is less than the current best courseware layout model, the current best courseware layout model can be replaced as the latest courseware layout model and saved.

[0056] In the embodiment of the present application, the courseware element layout device obtains the courseware layout data to be trained and the real courseware layout data, wherein the courseware layout data includes the element position of each courseware element; the courseware layout data to be trained is input into the courseware layout model to be trained, and the predicted element position of each courseware element output by the courseware layout model is obtained; based on the predicted element position and the real element position of each courseware element, the courseware layout model is trained until the training condition is obtained. Through the above-mentioned model training method, the staff's consideration of element layout when making courseware is simplified, which greatly improves the efficiency of courseware production.

[0057] Please continue reading Figure 2 , Figure 2 It is a flow chart of an embodiment of a method for layout of courseware elements provided in this application.

[0058] like Figure 2 As shown, the specific steps are as follows:

[0059] Step S21: Obtain a pre-trained courseware layout model.

[0060] Step S22: input the courseware to be laid out into the courseware layout model, and obtain the optimized courseware output by the courseware layout model.

[0061] In an embodiment of the present application, the courseware element layout device extracts the category and position of the PPT page elements through a pre-trained courseware layout model, and then predicts a new layout, that is, a new element position, based on the category and position of the elements.

[0062] Furthermore, since the present application Figure 1 In the training process, PPT data with beautiful layout is used, and the layout output by the pre-trained courseware layout model is also relatively beautiful, but some special cases are not excluded. For this purpose, this application further introduces a post-processing algorithm.

[0063] Specifically, the courseware element layout device uses a post-processing algorithm that is highly relevant to data such as PPT. For example, left alignment in PPT is more beautiful. The courseware element layout device divides the PPT page into N equal parts from left to right, for example, N=10, and then layouts the courseware elements in each space. The courseware element layout device takes the upper left corner coordinates of the courseware elements of the same category. If the same page part is processed, the courseware element layout device aligns these courseware elements of the same category in the x direction. The courseware element layout device adjusts the distances between adjacent text boxes / pictures up, down, left, and right to make them consistent.

[0064] This application creatively introduces the layout generation model into the field of courseware production, and uses the capabilities of the deep learning model to greatly simplify the consideration of element layout when manually producing courseware, greatly improving the efficiency of courseware production.

[0065] This application carefully analyzes the characteristics of PPT element layout, post-processes the model output results, and further optimizes the PPT element layout.

[0066] This application does not require manual production and selection of templates. Generally, after manual selection, users will check the rendering results and compare the rendering effects of different templates, which will not only make selection difficult, but also waste time. This application can liberate the productivity of workers and improve work efficiency.

[0067] Those skilled in the art will appreciate that, in the above method of specific implementation, the order in which the steps are written does not imply a strict execution order and does not constitute any limitation on the implementation process. The specific execution order of the steps should be determined by their functions and possible internal logic.

[0068] In order to implement the above-mentioned courseware element layout method, this application also proposes a courseware element layout device, please refer to Figure 3 , Figure 3 It is a structural diagram of an embodiment of a courseware element layout device provided in this application.

[0069] The courseware element layout device 400 of this embodiment includes a processor 41 , a memory 42 , an input and output device 43 , and a bus 44 .

[0070] The processor 41, memory 42, and input / output device 43 are respectively connected to a bus 44. The memory 42 stores program data. The processor 41 is used to execute the program data to implement the model training method and / or the courseware element layout method described in the above embodiments.

[0071] In the embodiment of the present application, the processor 41 may also be referred to as a CPU (Central Processing Unit). The processor 41 may be an integrated circuit chip having the ability to process signals. The processor 41 may also be a general-purpose processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field programmable gate array (FPGA), or other programmable logic devices, discrete gates or transistor logic devices, or discrete hardware components. A general-purpose processor may be a microprocessor or the processor 41 may also be any conventional processor, etc.

[0072] This application also provides a computer storage medium, please continue to refer to Figure 4 , Figure 4 It is a structural diagram of an embodiment of a computer storage medium provided in the present application. The computer storage medium 600 stores a computer program 61. When the computer program 61 is executed by the processor, it is used to implement the model training method and / or the courseware element layout method of the above-mentioned embodiment.

[0073] When the embodiments of the present application are implemented in the form of software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application is essentially or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including several instructions to enable a computer device (which can be a personal computer, a server, or a network device, etc.) or a processor to perform all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes: various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a disk or an optical disk.

[0074] The above description is only an implementation method of the present application, and does not limit the patent scope of the present application. Any equivalent structure or equivalent process transformation made using the contents of the present application specification and drawings, or directly or indirectly used in other related technical fields, are also included in the patent protection scope of the present application.

Claims

1. A model training method based on courseware element layout, characterized in that: The model training method comprises: Obtaining the layout data of the courseware to be trained and the layout data of the real courseware, wherein the courseware layout data includes the element position of each courseware element; Inputting the to-be-trained courseware layout data into the to-be-trained courseware layout model, and obtaining the predicted element position of each courseware element output by the courseware layout model; Based on the predicted element position and the actual element position of each courseware element, the courseware layout model is trained until the training conditions are obtained.

2. The model training method according to claim 1, characterized in that: The element position is composed of element coordinates and element size, wherein the element coordinates are the upper left corner coordinates, lower right corner coordinates, or center coordinates of the courseware element.

3. The model training method according to claim 1 or 2, characterized in that: The step of obtaining the predicted element position of each courseware element output by the courseware layout model includes: Obtaining the position noise predicted by the courseware layout model; The position noise is superimposed on the position of the element to be trained to obtain and output the predicted element position.

4. The model training method according to claim 3, characterized in that: The model training method also includes: Constructing a courseware layout model to be trained based on a diffusion model, wherein the diffusion model is DDPM, D3PM, or VQ-Diffusion; The noise predictor in the courseware layout model is used to predict the position noise of the courseware layout data to be trained.

5. The model training method according to claim 1, characterized in that: After training the courseware layout model until the training conditions are obtained, the model training method further includes: Get the current courseware layout model that meets the training conditions for the current batch; Obtain the target courseware layout model by obtaining historical batch training; Input the courseware layout data to be tested into the current courseware layout model and the target courseware layout model respectively, and obtain the current loss value of the current courseware layout model and the target loss value of the target courseware layout model; When the current loss value is less than the target loss value, the current courseware layout model is updated to the latest target courseware layout model.

6. The model training method according to claim 1, characterized in that: The courseware layout data also includes element categories of each courseware element, wherein the element categories include pictures, audio, video, tables, and / or text.

7. A courseware element layout method, characterized in that: The courseware element layout method comprises: Obtaining a pre-trained courseware layout model, wherein the courseware layout model is trained by the model training method according to any one of claims 1 to 6; The courseware to be laid out is input into the courseware layout model, and the optimized courseware output by the courseware layout model is obtained.

8. The courseware element layout method according to claim 7, characterized in that: After obtaining the optimized courseware output by the courseware layout model, the courseware element layout method further includes: Aligning courseware elements of the same element category in the optimized courseware in a preset direction; And / or, the distance between adjacent courseware elements of the same element category in the optimized courseware is set to a preset value.

9. A courseware element layout device, characterized in that: The courseware element layout device includes a memory and a processor coupled to the memory; Wherein, the memory is used to store program data, and the processor is used to execute the program data to implement the model training method as described in any one of claims 1 to 6, and / or the courseware element layout method as described in any one of claims 7 to 8.

10. A computer storage medium, characterized in that: The computer storage medium is used to store program data, and when the program data is executed by a computer, it is used to implement the model training method as described in any one of claims 1 to 6, and / or the courseware element layout method as described in any one of claims 7 to 8.

Images