Model training method and device, equipment, storage medium and computer program product

Abstract

The invention provides a model training method and device, equipment, a storage medium and a computer program product. The method comprises the following steps: determining a first preference difference of a target model for a training sample, and determining a second preference difference of a reference model for the training sample; determining a first loss value based on the first preference difference and a first preference scaling factor of the target model for the training sample, and determining a second loss value based on the first preference difference and the second preference difference; determining a convergence trend of the target model, and determining a loss scale factor based on the convergence trend; and based on the loss scale factor, fusing the first loss value and the second loss value to obtain a total loss value, and based on the total loss value, updating parameters of the target model. According to the invention, the correctness of the iteration direction of the target model can be ensured.

Description

Technical Field

[0001] The present application relates to artificial intelligence technology, and in particular, to a model training method, apparatus, device, storage medium, and computer program product. Background Art

[0002] Model training is one of the core technologies in the field of artificial intelligence. It refers to using a large amount of data to iteratively optimize through algorithms, enabling a machine learning model to gradually acquire the ability to simulate the human learning process, and thus being able to efficiently complete the automated processing of specific tasks. Among them, reinforcement learning is an important branch of machine learning, which trains the model by directly comparing the quality of multiple generated results as feedback information. However, in related technologies, the training process of the model is easily interfered by various factors, resulting in the model deviating from the correct optimization direction during the iteration process. Summary of the Invention

[0003] Embodiments of the present application provide a model training method, apparatus, device, storage medium, and computer program product, which can ensure the correctness of the iteration direction of the target model.

[0004] The technical solution of the embodiments of the present application is implemented as follows:

[0005] Embodiments of the present application provide a model training method, the method comprising:

[0006] Predicting a training sample through a target model to obtain a first positive preference probability and a first negative preference probability, and determining a first preference difference of the target model for the training sample based on a difference between the first positive preference probability and the first negative preference probability;

[0007] Predicting the training sample through a reference model for reinforcing the target model to obtain a second positive preference probability and a second negative preference probability, and determining a second preference difference of the reference model for the training sample based on a difference between the second positive preference probability and the second negative preference probability;

[0008] Determining a first loss value based on the first preference difference and a first preference scaling factor of the target model for the training sample, and determining a second loss value based on the first preference difference and the second preference difference;

[0009] Determining a convergence trend of the target model, and determining a loss ratio factor based on the convergence trend;

[0010] Fusing the first loss value and the second loss value based on the loss ratio factor to obtain a total loss value, and updating parameters of the target model based on the total loss value.

[0011] An embodiment of the present application provides a model training device, which includes:

[0012] A forward propagation module, configured to predict a training sample through a target model to obtain a first positive preference probability and a first negative preference probability, and determine a first preference difference of the target model for the training sample based on a difference between the first positive preference probability and the first negative preference probability;

[0013] The forward propagation module is further configured to predict the training sample through a reference model for reinforcement learning the target model to obtain a second positive preference probability and a second negative preference probability, and determine a second preference difference of the reference model for the training sample based on a difference between the second positive preference probability and the second negative preference probability;

[0014] A loss value determination module, configured to determine a first loss value based on the first preference difference and a first preference scaling factor of the target model for the training sample, and determine a second loss value based on the first preference difference and the second preference difference;

[0015] A loss value fusion module is further configured to determine a convergence trend of the target model and determine a loss ratio factor based on the convergence trend;

[0016] The loss value fusion module is further configured to fuse the first loss value and the second loss value based on the loss ratio factor to obtain a total loss value;

[0017] A backpropagation module, configured to update parameters of the target model based on the total loss value.

[0018] An embodiment of the present application provides an electronic device, which includes:

[0019] A memory, configured to store computer-executable instructions or a computer program;

[0020] A processor, configured to implement the model training method provided by the embodiment of the present application when executing the computer-executable instructions or the computer program stored in the memory.

[0021] An embodiment of the present application provides a computer-readable storage medium, storing a computer program or computer-executable instructions, which are configured to implement the model training method provided by the embodiment of the present application when being executed by a processor.

[0022] An embodiment of the present application provides a computer program product, including a computer program or computer-executable instructions, which are configured to implement the model training method provided by the embodiment of the present application when being executed by a processor.

[0023] The embodiments of the present application have the following beneficial effects:

[0024] Based on the first preference difference of the target model for the training samples and the second preference difference of the reference model for the training samples, the preference differences of different models for the samples are more comprehensively reflected, thereby providing richer information for the subsequent optimization of the target model. The first loss value is determined through the first preference difference of the target model for the training samples and the first preference scaling factor, and the second loss value is determined based on the first preference difference and the second preference difference. Furthermore, through the multi-dimensional loss value determination method, the performance of the target model on the current training samples is accurately measured. Further, based on the influence of the training samples on the convergence trend of the target model, the loss ratio factor corresponding to the training samples is determined, and based on the loss ratio factor, the first loss value and the second loss value are fused to obtain the total loss. The parameters of the target model are updated based on the total loss value, which not only ensures the correctness of the iterative direction of the target model but also realizes personalized optimization for the training samples. Description of the Drawings

[0025] Figure 1 is a schematic structural diagram of the model training system architecture provided by the embodiments of the present application;

[0026] Figure 2 is a schematic structural diagram of the electronic device for model training provided by the embodiments of the present application;

[0027] Figure 3A is a schematic first process diagram of the model training method provided by the embodiments of the present application;

[0028] Figure 3B is a schematic second process diagram of the model training method provided by the embodiments of the present application;

[0029] Figure 3C is a schematic third process diagram of the model training method provided by the embodiments of the present application;

[0030] Figure 3D is a schematic fourth process diagram of the model training method provided by the embodiments of the present application;

[0031] Figure 4 is a schematic first principle diagram of direct preference optimization learning provided by the embodiments of the present application;

[0032] Figure 5 is a schematic second principle diagram of direct preference optimization learning provided by the embodiments of the present application;

[0033] Figure 6 is a schematic first principle diagram of the model training method provided by the embodiments of the present application;

[0034] Figure 7 is a target model training flowchart of the model training method provided by the embodiments of the present application;

[0035] Figure 8 It is the second schematic diagram of the model training method provided by the embodiments of the present application;

[0036] Figure 9 It is the fifth process schematic diagram of the model training method provided by the embodiments of the present application;

[0037] Figure 10 It is the third schematic diagram of the model training method provided by the embodiments of the present application.

[0038] It should be noted that the above "first" and "second" are only used to distinguish different solutions, and do not represent the distinction of the advantages and disadvantages of the solutions or the priority in the implementation process. Specific Embodiments

[0039] In order to make the objectives, technical solutions and advantages of the present application clearer, the present application will be further described in detail below with reference to the accompanying drawings. The described embodiments should not be regarded as limitations of the present application. All other embodiments obtained by those of ordinary skill in the art without creative efforts shall fall within the scope of protection of the present application.

[0040] In the following description, reference is made to "some embodiments", which describe a subset of all possible embodiments. However, it can be understood that "some embodiments" can be the same subset or different subsets of all possible embodiments, and can be combined with each other without conflict.

[0041] In the following description, the terms "first / second / third" are only used to distinguish similar objects, and do not represent a specific order for the objects. It can be understood that "first / second / third" can be interchanged with a specific order or sequence when allowed, so that the embodiments of the present application described here can be implemented in an order other than that illustrated or described here.

[0042] In the embodiments of the present application, the term "module" or "unit" refers to a computer program with a predetermined function or a part of a computer program, which works together with other related parts to achieve a predetermined goal, and can be fully or partially implemented by using software, hardware (such as a processing circuit or a memory), or a combination thereof. Similarly, one processor (or multiple processors or memories) can be used to implement one or more modules or units. In addition, each module or unit can be a part of the overall module or unit that includes the function of the module or unit.

[0043] Unless otherwise defined, all technical and scientific terms used in the embodiments of the present application have the same meaning as commonly understood by those skilled in the technical field to which the present application belongs. The terms used in the embodiments of the present application are only for the purpose of describing the embodiments of the present application, and are not intended to limit the present application.

[0044] In the embodiments of this application, when collecting and processing relevant data in practical applications, the informed consent or separate consent of the personal information subject should be obtained in strict accordance with the requirements of relevant laws and regulations, and subsequent data use and processing should be carried out within the scope authorized by laws, regulations and the personal information subject.

[0045] Before further elaborating on the embodiments of this application, the nouns and terms involved in the embodiments of this application are described. The nouns and terms involved in the embodiments of this application are applicable to the following explanations.

[0046] 1) Model training: Model training refers to the process of enabling a machine learning model to perform specific tasks by learning from a dataset. Model training mainly involves two processes: forward propagation and backpropagation.

[0047] 2) Direct Preference Optimization (DPO): Direct Preference Optimization is a type of reinforcement learning method. Its core lies in optimizing model parameters by comparing the preference degrees of different output results. In this process, the reference model serves as a benchmark, usually a model version that has been fine-tuned with supervision, providing a stable behavior reference for the target model. The target model is the model that is continuously optimized during training, and its model parameters are optimized by comparing with the reference model.

[0048] 3) Training samples: Sample data refers to the data used to train, validate, and test a machine learning model. Each training sample includes input feature information (Features) and corresponding output label information (Labels). Among them, the feature information is the information describing the observed values or attributes of the training samples, used to train the model to identify patterns and regularities in the data, and the label information is the target value that the model is expected to identify or predict.

[0049] 4) Loss value: The loss value is a quantitative indicator that measures the difference between the model's prediction result and the true result. It is often used to evaluate the model's performance and guide parameter optimization. In model training, the loss value is calculated through a specific loss function. A smaller loss value indicates that the model's prediction is closer to the true value.

[0050] The embodiments of this application provide a model training method, device, electronic device, computer-readable storage medium, and computer program product, which can ensure the correctness of the iteration direction of the target model.

[0051] The model training method described in the embodiments of this application can be applied to various fields (such as the image field, the text field, the video field, the audio field, etc.). For example, in the text field, it can be applied to the application scenario of intelligent script understanding for training summary models, character models, etc. That is, the model training method in the embodiments of this application is not limited to a certain field.

[0052] See Figure 1 , Figure 1 FIG. is a schematic structural diagram of the model training system architecture provided by the embodiments of this application. In the model training system 10 provided by the embodiments of this application, to support a model training application, the terminal 400 is connected to the server 200 through the network 300. The network 300 can be a wide area network, a local area network, or a combination of the two.

[0053] The terminal 400 can be used to obtain a model training request for a target model carrying training samples.

[0054] In some embodiments, a model training plugin can be implanted in the client running on the terminal 400 to implement the model training method locally on the client. For example, the terminal 400 calls the model training plugin to implement the model training method, determine the first preference difference of the target model for the training sample, and the second preference difference of the reference model for the training sample used for reinforcement learning of the target model. Based on the first preference difference and the first preference scaling factor of the target model for the training sample, determine the first loss value, and based on the first preference difference and the second preference difference, determine the second loss value. Based on the convergence trend of the target model, determine the loss ratio factor. Based on the loss ratio factor, fuse the first loss value and the second loss value to obtain the total loss value. Based on the total loss value, update the parameters of the target model.

[0055] It should be noted that the target model with updated parameters can be stored locally on the terminal 400 and called when needed.

[0056] In some embodiments, after the terminal 400 obtains a model training request for a target model carrying training samples, it calls the model training interface of the server 200 (which can be provided in the form of a cloud service, i.e., a model training service). The server 200 implements the model training method through the model training plugin, determines the first preference difference of the target model for the training sample, and the second preference difference of the reference model for the training sample used for reinforcement learning of the target model. Based on the first preference difference and the first preference scaling factor of the target model for the training sample, determine the first loss value, and based on the first preference difference and the second preference difference, determine the second loss value. Based on the convergence trend of the target model, determine the loss ratio factor. Based on the loss ratio factor, fuse the first loss value and the second loss value to obtain the total loss value. Based on the total loss value, update the parameters of the target model.

[0057] It should be noted that the target model after parameter update can be stored locally in the server 200 and called when needed, or returned to the terminal 400.

[0058] The model training method of the embodiments of the present application can be applied to various application scenarios in various fields. For example, in the field of images, it can be applied to the application scenario of medical image diagnosis. By using the model training method of the embodiments of the present application to train an auxiliary diagnosis model, the lesions can be accurately detected and assist doctors in judging the condition. In the field of text, it can be applied to the application scenario of intelligent script understanding. By using the model training method of the embodiments of the present application to train a summary model for summary extraction, and by using the model training method of the embodiments of the present application to train a character setting model for character setting and character story line extraction. The following is a specific description.

[0059] In an implementation scenario, the server 200 or the terminal 400 can implement the extraction of the scene summary of the script, obtain training samples for training the summary model. The training samples include plot texts (i.e., feature information), accepted summary texts (i.e., positive labels), and unaccepted summary texts (i.e., negative labels). Determine the first preference difference of the summary model (i.e., the target model) for the training samples, and the second preference difference of the reference model for the training samples. Based on the first preference difference and the first preference scaling factor of the summary model for the training samples, determine the first loss value, and based on the first preference difference and the second preference difference, determine the second loss value. Based on the convergence trend of the summary model, determine the loss ratio factor. Based on the loss ratio factor, fuse the first loss value and the second loss value to obtain the total loss value. Based on the total loss value, update the parameters of the summary model. Then, obtain the script text, and perform scene splitting on the script text to obtain text information of multiple scenes. Based on the summary model with updated parameters, perform summary extraction processing on the text information of each scene in the multiple scenes in sequence to obtain summary information corresponding to each scene, and store the summary information of the multiple scenes in the plot library to assist relevant practitioners in quickly understanding the script plot.

[0060] In an implementation scenario, the server 200 or the terminal 400 can implement the extraction of the setting information and the character storylines of the characters appearing in each scene of the script, obtain training samples for training the character model. The training samples include the summary information (i.e., feature information) of the scenes related to the characters appearing, the accepted character settings and character storyline texts (i.e., positive labels), and the unaccepted character settings and character storyline texts (i.e., negative labels). Determine the first preference difference of the character model (i.e., the target model) for the training samples, and the second preference difference of the reference model for the training samples. Based on the first preference difference and the first preference scaling factor of the character model for the training samples, determine the first loss value, and based on the first preference difference and the second preference difference, determine the second loss value. Based on the convergence trend of the character model, determine the loss ratio factor. Based on the loss ratio factor, fuse the first loss value and the second loss value to obtain the total loss value. Based on the total loss value, update the parameters of the character model. Then, obtain the text information of multiple scenes included in the script text, perform character splitting on the text information of each scene to obtain the characters appearing in that scene, and extract the relevant plot of the characters appearing from the plot library. Input the relevant plot of the characters appearing into the character model with updated parameters to obtain the character setting information (i.e., the character profile) and the character storyline of the characters appearing, so as to assist relevant practitioners in quickly understanding the characters appearing.

[0061] In some embodiments, the server 200 can be an independent physical server, or a server cluster or distributed system composed of multiple physical servers. It can also be a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, content delivery network (CDN, Content Delivery Network), and big data and artificial intelligence platforms.

[0062] The terminal 400 can be a smart phone, a tablet computer, a notebook computer, a desktop computer, a smart speaker, a smart watch, a smart voice interaction device, a smart home appliance, a vehicle terminal, an aircraft, etc., but is not limited thereto. The terminal and the server can be directly or indirectly connected through wired or wireless communication methods, which are not limited in the embodiments of the present application.

[0063] See Figure 2 , Figure 2 is a schematic structural diagram of an electronic device for model training provided by an embodiment of the present application. Figure 2 The electronic device 500 shown can be Figure 1The terminal 400 or the server 200 in it, the electronic device 500 includes: at least one processor 510, a memory 550, and at least one network interface 520. Each component in the server 200 is coupled together through a bus system 540. It can be understood that the bus system 540 is used to realize the connection and communication between these components. In addition to the data bus, the bus system 540 also includes a power bus, a control bus, and a status signal bus. However, for the sake of clear illustration, in Figure 2 all kinds of buses are labeled as the bus system 540.

[0064] The processor 510 can be an integrated circuit chip with signal processing capabilities, such as a general-purpose processor, a digital signal processor (DSP, Digital Signal Processor), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. Among them, the general-purpose processor can be a microprocessor or any conventional processor, etc.

[0065] The user interface 530 includes one or more output devices 531 that enable the presentation of media content, including one or more speakers and / or one or more visual display screens. The user interface 530 also includes one or more input devices 532, including user interface components that assist the user in input, such as a keyboard, a mouse, a microphone, a touch screen display, a camera, other input buttons, and controls;

[0066] In some embodiments, when the embodiment independently completes the model training task by the terminal 400, the server 200 provided by the embodiments of the present application does not include the user interface 530.

[0067] The memory 550 can be removable, non-removable, or a combination thereof. Exemplary hardware devices include solid-state memory, hard disk drives, optical disc drives, etc. The memory 550 optionally includes one or more storage devices that are physically located away from the processor 510.

[0068] The memory 550 includes volatile memory or non-volatile memory, and can also include both volatile and non-volatile memory. The non-volatile memory can be a read-only memory (ROM, Read Only Memory), and the volatile memory can be a random access memory (RAM, Random Access Memory). The memory 550 described in the embodiments of the present application is intended to include any suitable type of memory.

[0069] In some embodiments, the memory 550 is capable of storing data to support various operations. Examples of these data include programs, modules, and data structures, or subsets or supersets thereof, which are illustrated below.

[0070] The operating system 551 includes system programs for handling various basic system services and performing hardware-related tasks, such as the framework layer, the core library layer, the driver layer, etc., for implementing various basic services and handling hardware-based tasks;

[0071] The network communication module 552 is used to reach other computing devices via one or more (wired or wireless) network interfaces 520. Exemplary network interfaces 520 include: Bluetooth, Wi-Fi (Wireless Fidelity), and USB (Universal Serial Bus), etc.;

[0072] The presentation module 553 is used to enable the presentation of information (such as a user interface for operating peripheral devices and displaying content and information) via one or more output devices 531 associated with the user interface 530 (such as a display screen, a speaker, etc.);

[0073] In some embodiments, when the terminal 400 independently completes the model training task, the server 200 provided by the embodiments of the present application may not include the presentation module 553.

[0074] The input processing module 554 is used to detect and translate one or more user inputs or interactions from one of one or more input devices 532; In some embodiments, when the terminal 400 independently completes the model training task, the server 200 provided by the embodiments of the present application may not include the presentation module 553.

[0075] In some embodiments, the device provided by the embodiments of the present application may be implemented in software. Figure 2 The model training device 555 stored in the memory 550 is shown, which may be software in the form of programs and plugins, etc., and includes the following software modules: the forward propagation module 5551, the loss value determination module 5552, the loss value fusion module 5553, and the backpropagation module 5554. These modules are logical, so they can be combined arbitrarily or further split according to the functions to be implemented. The functions of each module will be described below.

[0076] In some other embodiments, the device provided by the embodiments of the present application can be implemented in a hardware manner. As an example, the device provided by the embodiments of the present application can be a processor in the form of a hardware decoding processor, which is programmed to execute the model training method provided by the embodiments of the present application. For example, the processor in the form of a hardware decoding processor can adopt one or more application specific integrated circuits (ASICs), digital signal processors (DSPs), programmable logic devices (PLDs), complex programmable logic devices (CPLDs), field programmable gate arrays (FPGAs) or other electronic components.

[0077] It should be noted that in the examples of model training shown below, those skilled in the art can, based on the understanding of the following text, apply the target model obtained by training the model training method provided by the embodiments of the present application to abstract extraction, character setting, and character story line extraction in the application scenario of intelligent script understanding.

[0078] As mentioned above, the electronic device for implementing the model training method of the embodiments of the present application can be a terminal, a server, or a combination of the two. Therefore, the execution subject of each step will not be repeated below. Refer to Figure 3A , Figure 3A is the first process schematic diagram of the model training method provided by the embodiments of the present application, and will be described in combination with Figure 3A the steps shown.

[0079] In step 101, the training sample is predicted by the target model to obtain a first positive preference probability and a first negative preference probability, and based on the difference between the first positive preference probability and the first negative preference probability, the first preference difference of the target model for the training sample is determined.

[0080] Among them, the target model is the model participating in iterative training, the first positive preference probability is the likelihood probability that the prediction result of the target model is a positive label, the first negative preference probability is the likelihood probability that the prediction result of the target model is a negative label, the first preference difference is used to measure the preference tendency of the target model for the training sample, the training sample includes feature information, a positive label, and a negative label, the training sample is the j-th training sample among the J training samples included in the training sample set, J is a positive integer, j is a positive integer that increases sequentially, and j ≤ J.

[0081] For example, in the application scenario of intelligent script understanding, the target model can be a summary model for scene summary extraction. Each training sample in the training sample set can be represented as (text, y l , y w ), where text is the script text (i.e., feature information), y l is the accepted summary (i.e., positive label), and y w is the unaccepted summary (i.e., negative label). Each training sample in the training sample set can be constructed in the following way: Obtain the script text text, construct a prompt for guiding the large language model to extract summary information based on the script text. The prompt can be "The following is the script of a certain scene in a play. Please extract the summary of the plot of this scene, requiring a description of the specific things that happened in the scene, without speculation and without abstract description. The script text is: text". Input the prompt into the large language model to obtain the summary information output by the large language model, determine the summary information output by the large language model as the unaccepted summary y w , and manually correct the summary information output by the large language model, and determine the corrected summary information as the accepted summary y l .

[0082] As an example, in the current iteration round, the target model predicts the feature information of the j-th training sample in the training sample set to obtain the first positive preference probability and the first negative preference probability.

[0083] Based on the difference between the first positive preference probability and the first negative preference probability, determine the first preference difference of the target model for the training sample through formula (1.1). Formula (1.1) is as follows:

[0084] diff3 = β(logπ θ (y w |x) - logπ θ (y l |x)) (1.1)

[0085] where diff3 is the first preference difference, π θ is the target model, x is the feature information of the training sample, y w is the positive label, y l is the negative label, π θ (y w |x) is the first positive preference probability, π θ (y l |x) is the first negative preference probability, and β is a balance parameter (usually taken as 0.1).

[0086] When the current iteration round is not the first round, the target model is the model obtained after the previous round of iterative training. When the current iteration round is the first round, the target model is the model obtained after pre-training the initial neural network model. The initial neural network model can directly adopt the open-source Baichuan 7B model, including an embedding layer, 32 cascaded decoding layers, and a fully connected layer. Each decoding layer sequentially includes a normalization layer, an attention layer, another normalization layer, and a multi-layer perceptron. Of course, the model structure of the initial neural network model in the embodiments of the present application is not limited thereto.

[0087] In step 102, the reference model for strengthening the learning target model is used to predict the training samples, obtaining the second positive preference probability and the second negative preference probability, and based on the difference between the second positive preference probability and the second negative preference probability, determining the second preference difference of the reference model for the training samples.

[0088] Among them, the reference model is a model for assisting the target model in reinforcement learning. The model structure of the reference model is the same as that of the target model. The reference model is the model obtained after pre-training the initial neural network model, and the reference model does not participate in iterative training (parameter freezing).

[0089] As an example, the reference model is used to predict the feature information of the j-th training sample in the training sample set, obtaining the second positive preference probability and the second negative preference probability. The second positive preference probability is the likelihood probability that the prediction result of the reference model is a positive label, and the second negative preference probability is the likelihood probability that the prediction result of the reference model is a negative label. Based on the difference between the second positive preference probability and the second negative preference probability, the second preference difference of the reference model for the training samples is determined through formula (1.2). The second preference difference is used to measure the preference tendency of the reference model for the training samples. Formula (1.2) is as follows:

[0090] diff4 = β(logπ ref (y w (y ref (y l (y

[0091] Among them, diff4 is the second preference difference, π ref is the reference model, x is the feature information of the training sample, y w is the positive label, y l is the negative label, π ref (y w (y ref (y l (y

[0092] In some embodiments, referring to Figure 3B , Figure 3B FIG. is a second process schematic diagram of the model training method provided by the embodiments of the present application. Before the step 103 shown in Figure 3A , the following steps 201 to 202 can be executed to implement, and the following is a specific description.

[0093] In step 201, determine the current iteration round of the target model, and statistically analyze the preference differences in the reference space distribution of the current iteration round to obtain a first maximum value and a second average value.

[0094] As an example, based on the current iteration round of the target model, determine the reference space distribution of the current iteration round, and statistically analyze the preference differences in the reference space distribution of the current iteration round to obtain a first maximum value and a second average value, where the reference space distribution is a space distribution including the preference differences of J training samples.

[0095] In some embodiments, the training sample is the jth training sample among the J training samples included in the training sample set, J is a positive integer, j is a sequentially increasing positive integer, and j ≤ J. Figure 3B The step 201 shown in FIG. can be implemented by the following steps: When the current iteration round is the first round, obtain the reference space distribution composed of the preference differences of each training sample among the J training samples by the reference model, and determine the average value of the J preference differences in the reference space distribution as the second average value, and determine the maximum value among the J preference differences in the reference space distribution as the first maximum value; when the current iteration round is not the first round, obtain the reference space distribution composed of the preference differences of each training sample by the target model in the previous round, and determine the average value of the J preference differences in the reference space distribution as the second average value, and determine the maximum value among the J preference differences in the reference space distribution as the first maximum value.

[0096] As an example, when the current iteration round is the first round, the reference space distribution of the current iteration round is the space distribution composed of the preference differences of each training sample among the J training samples by the reference model, that is, the reference space is distributed with J preference differences of the reference model for the J training samples. Determine the average value of the J preference differences as the second average value, and determine the maximum value among the J preference differences as the first maximum value. Among them, the reference space distribution is obtained through the following processing: Based on the reference model, predict each training sample included in the training sample set in turn to obtain the positive preference probability and negative preference probability of each training sample, and based on the difference between the positive preference probability and negative preference probability of each training sample, determine the preference difference of the reference model for each training sample, and determine the space distribution composed of the J differences of the reference model for the J training samples as the reference space distribution.

[0097] When the current iteration round is not the first round, the reference space distribution of the current iteration round is the space distribution formed by the preference differences of the target model for each training sample in the previous round, that is, the reference space distributes J preference differences of the target model for J training samples in the previous round. The average value of the J preference differences is determined as the second average value, and the maximum value among the J preference differences is determined as the first maximum value. For example, when the current iteration round is the third round, the reference space distribution is the space distribution formed by 66 preference differences of the target model for 66 training samples in the second round. The average value of the 66 preference differences is determined as the second average value, and the maximum value among the 66 preference differences is determined as the first maximum value.

[0098] In the embodiments of the present application, in the first round, the reference space distribution is initialized by the preference differences of the reference model. In subsequent iteration rounds, the reference space distribution is updated by using the preference differences of the target model itself in the previous round. By dynamically updating the reference space distribution in different iteration rounds, the preference change situation of the target model during the training process can be better reflected.

[0099] In step 202, based on the first maximum value, the second average value, and the first preference difference, a first preference scaling factor is determined.

[0100] Among them, the first preference scaling factor is the expected preference difference of the target model for the training sample, and is used to guide the target model to learn in the direction indicated by the first preference scaling factor.

[0101] In some embodiments, Figure 3B The shown step 202 can be implemented by the following steps: determining a second ratio between the first maximum value and the second average value, and determining the product of the second ratio and the first preference difference as the second candidate scaling factor; based on the first maximum value and the first preference difference, cropping the second candidate scaling factor to obtain the first preference scaling factor.

[0102] As an example, the first maximum value can be represented as max_diff3, the second average value can be represented as avg_diff3, and the first preference difference can be represented as diff3. Determine the second ratio between the first maximum value and the second average value and determine the product of the second ratio and the first preference difference as the second candidate scaling factor, and the second candidate scaling factor is represented as

[0103] In the embodiments of the present application, by determining the second ratio between the first maximum value and the second average value and multiplying it by the first preference difference, a second candidate scaling factor is generated, which can more accurately reflect the relative importance of the training sample in the reference space distribution. Furthermore, the second candidate scaling factor corresponding to each training sample is dynamically adjusted, making the second candidate scaling factor corresponding to the training sample more adaptable and flexible.

[0104] Based on the value range [diff3, max_diff3] from the first preference difference to the first maximum value, the part of the second candidate scaling factor that exceeds the above value range is clipped to obtain the first preference scaling factor.

[0105] In some embodiments, "clipping the second candidate scaling factor based on the first maximum value and the first preference difference to obtain the first preference scaling factor" can be achieved through the following steps: when the second candidate scaling factor is greater than or equal to the first maximum value, clip the second candidate scaling factor to the first maximum value to obtain the first preference scaling factor; when the second candidate scaling factor is less than or equal to the first preference difference, determine the first preference difference as the first preference scaling factor; when the second candidate scaling factor is less than the first maximum value and greater than the first preference difference, clip the second candidate scaling factor to the first preference scaling factor to obtain the first preference scaling factor.

[0106] As an example, when the second candidate scaling factor is greater than or equal to the first maximum value, clip the part of the second candidate scaling factor that is greater than the first maximum value, that is, clip the second candidate scaling factor to the first maximum value to obtain the first preference scaling factor, and the first preference scaling factor can be represented as ratio_diff3, ratio_diff3 = max_diff3; when the second candidate scaling factor is less than or equal to the first preference difference, clip the part of the second candidate scaling factor that is less than the first preference difference, that is, clip the second candidate scaling factor to the first preference difference to obtain the first preference scaling factor, represented as ratio_diff3 = diff3; when the second candidate scaling factor is less than the first maximum value and greater than the first preference difference, that is, the second candidate scaling factor is within the value range [diff3, max_diff3] from the first preference difference to the first maximum value, determine the second candidate scaling factor as the first preference scaling factor, represented as

[0107] In the embodiment of the present application, based on the value range from the first preference difference to the first maximum value, the part of the second candidate scaling factor that exceeds the value range is cropped to obtain the first preference scaling factor, effectively controlling the value range of the first scaling factor and avoiding adverse effects on the training of the target model due to it being too large or too small. This cropping mechanism not only simplifies the calculation process, improves the efficiency of the target model training, but also enhances the adaptability and robustness of the target model to different training samples.

[0108] In step 103, based on the first preference difference and the first preference scaling factor of the target model for the training sample, a first loss value is determined, and based on the first preference difference and the second preference difference, a second loss value is determined.

[0109] As an example, a first mapping value can be determined based on the first difference between the first preference difference and the first preference scaling factor, and the first mapping value is determined as the first loss value, and a second loss value is determined based on the first preference difference and the second preference difference; alternatively, a second mapping value can be determined based on the second preference difference and the second preference scaling factor, and the sum of the first mapping value and the second mapping value is determined as the first loss value, and a second loss value is determined based on the first preference difference and the second preference difference, which is not limited herein.

[0110] In some embodiments, refer to Figure 3C , Figure 3C is the third process schematic diagram of the model training method provided by the embodiment of the present application, Figure 3A The "determining the first loss value based on the first preference difference and the first preference scaling factor of the target model for the training sample" in step 103 shown can be implemented through the following steps 1031 to 1032, which are specifically described below.

[0111] In step 1031, a first difference between the first preference difference and the first preference scaling factor is determined.

[0112] As an example, the first preference difference can be represented as dif3, the first preference difference is subjected to an exponential operation to obtain the logarithm logdiff3 of the first preference difference, the first preference scaling factor can be represented as ratio_diff3, the first preference scaling factor is subjected to an exponential operation to obtain the logarithm logratio_diff3 of the first preference scaling factor, and the difference between the logarithm of the first preference difference and the logarithm of the first preference scaling factor is determined as the first difference, and the first difference can be represented as

[0113] In step 1032, a first mapping is performed on the first difference to obtain a first mapping value, and based on the first mapping value, the first loss value is determined.

[0114] Among them, the first difference is positively correlated with the first loss value, and the first mapping value is used to quantify the relative entropy (KL divergence) between the first preference difference and the first preference scaling factor.

[0115] As an example, through formula (1.3), the first difference is subjected to the first mapping to obtain the first mapping value, and the first mapping value is determined as the first loss value. Formula (1.3) is as follows:

[0116]

[0117] Where loss is the first loss, D(diff3‖ratio_diff3) is the first mapping value, diff3 is the first preference difference, ratio_diff3 is the first preference scaling factor, and D represents the first mapping.

[0118] In the embodiments of the present application, by mapping the difference between the first preference difference and the first preference scaling factor to the first loss value, not only can the KL divergence between the first preference difference and the first preference scaling factor be quantified, but also the target model can be effectively guided to adjust its own parameters during subsequent parameter updates, so that the first preference difference gradually approaches the first preference scaling factor.

[0119] Referring to the above, the first mapping value can be directly determined as the first loss value, or the second mapping value can be determined through steps 10321 to 10323 shown below, and the sum of the first mapping value and the second mapping value is determined as the first loss value, which is not limited herein.

[0120] In some embodiments, referring to Figure 3D , Figure 3D is the fourth process schematic diagram of the model training method provided by the embodiments of the present application. "Determining the first loss value based on the first mapping value" can be implemented through the following steps 10321 to 10323, which will be specifically described below.

[0121] In step 10321, determine the second preference scaling factor of the reference model for the training sample, and determine the second difference between the second preference difference and the second preference scaling factor.

[0122] As an example, determine the second preference scaling factor of the reference model for the training sample. The second preference scaling factor can be represented as ratio_diff4. Perform an exponential operation on the second preference scaling factor to obtain the logarithm of the second preference scaling factor, log ratio_diff4. The second preference difference can be represented as diff4. Perform an exponential operation on the second preference difference to obtain the logarithm of the second preference difference, log diff4. Determine the difference between the logarithm of the second preference difference and the logarithm of the second preference scaling factor as the second difference, and the second difference can be represented as

[0123] In some embodiments, the training sample is the j-th training sample among the J training samples included in the training sample set, where J is a positive integer, j is a positive integer that increases sequentially, and j ≤ J. Figure 3D The "determining the second preference scaling factor of the reference model for the training sample" in step 10321 shown can be implemented by the following processing: obtaining the reference space distribution formed by the preference differences of the reference model for each of the J training samples, determining the average value of the J preference differences in the reference space distribution as the first average value, and determining the minimum value among the J preference differences in the reference space distribution as the first minimum value; determining the second preference scaling factor based on the first minimum value, the first average value, and the second preference difference.

[0124] As an example, referring to step 201 shown above, obtain the reference space distribution formed by the preference differences of the reference model for each of the J training samples. The reference space distributes the J preference differences of the reference model for the J training samples. Determine the average value of the J preference differences as the first average value, and determine the minimum value among the J preference differences as the first minimum value. Determine the second preference scaling factor based on the first minimum value, the first average value, and the second preference difference.

[0125] In the embodiments of the present application, using the preference differences of the reference model to initialize the reference space distribution provides a constant benchmark for subsequent iterations, which can effectively improve the stability and accuracy of the target model training, enabling the target model to better adapt to the preference differences of different training samples in the subsequent iteration process, thereby improving the overall performance of the target model.

[0126] In some embodiments, the "determining the second preference scaling factor based on the first minimum value, the first average value, and the second preference difference" can be implemented by the following processing: determining the first ratio between the first minimum value and the first average value, and determining the product of the first ratio and the second preference difference as the first candidate scaling factor; clipping the first candidate scaling factor based on the first minimum value and the second preference difference to obtain the second preference scaling factor.

[0127] Among them, the second preference scaling factor is the expected preference difference of the reference model for the training sample, and is used to provide a stable reference benchmark for the target model.

[0128] As an example, the second preference difference can be characterized as diff4, the first minimum value can be characterized as min_diff4, the first average value can be characterized as avg_diff4, and determine the first ratio between the first minimum value and the first average value And determine the product of the first ratio and the second preference difference as the first candidate scaling factor, where the first candidate scaling factor is characterized as

[0129] Based on the value range [min_diff4, diff4] from the first minimum value to the second preference difference, clip the part of the first candidate scaling factor that exceeds the above value range to obtain the second preference scaling factor. The specific processing method of "clipping the first candidate scaling factor to obtain the second preference scaling factor" can refer to step 202 shown above and will not be elaborated here.

[0130] In the embodiment of the present application, when the target model learns the training samples, the parameters of the reference model are not updated, that is, the second preference difference output by the reference model remains unchanged. The second preference scaling factor is the expected preference difference of the reference model for the training samples, which is used to provide a relatively stable reference benchmark for the target model, thereby helping the target model to correct deviations and improve generalization ability during the learning process. Among them, based on the value range from the first minimum value to the second preference difference, clip the part of the first candidate scaling factor that exceeds the above value range to obtain the second preference scaling factor, which can effectively control the value range of the second preference scaling factor.

[0131] In step 10322, perform a second mapping on the second difference to obtain a second mapping value.

[0132] Among them, the second mapping value is used to quantify the relative entropy (KL divergence) between the second preference difference and the second preference scaling factor.

[0133] As an example, perform a second mapping on the second difference through formula (1.4) to obtain the second mapping value. Formula (1.4) is as follows:

[0134]

[0135] Among them, D(diff4‖ratio_diff4) is the second mapping value, diff4 is the second preference difference, ratio_diff4 is the second preference scaling factor, and D represents the second mapping.

[0136] In step 10323, fuse the first mapping value and the second mapping value to obtain the first loss value.

[0137] As an example, through formula (1.5), determine the sum of the first mapping value and the second mapping value as the first loss value. Formula (1.5) is as follows:

[0138] loss = D(diff3‖ratio_diff3) + D(diff4‖ratio_diff4) (1.5)

[0139] Wherein, loss is the first loss value, D(diff3‖ratio_diff3) is the first mapping value, diff3 is the first preference difference, ratio_diff3 is the first preference scaling factor, D(diff4‖ratio_diff4) is the second mapping value, diff4 is the second preference difference, and ratio_diff4 is the second preference scaling factor.

[0140] In the embodiments of the present application, by fusing the first mapping value corresponding to the first preference difference and the second mapping value corresponding to the second preference difference to determine the first loss value, not only the first preference difference of the target model itself is considered, but also the second preference difference of the reference model is introduced as a stable reference benchmark, enabling the target model to more comprehensively adjust its own parameters during the learning process, further improving the training effect and performance of the target model.

[0141] In some embodiments, Figure 3A "Determining the second loss value based on the first preference difference and the second preference difference" in step 103 shown can be implemented by the following processing: determining a third difference between the first preference difference and the second preference difference; performing a third mapping on the third difference to obtain the second loss value.

[0142] As an example, the difference between the first preference difference and the second preference difference is determined as the third difference, and through formula (1.6), the third difference is subjected to a third mapping to obtain the second loss value, and the third difference is negatively correlated with the second loss value. Formula (1.6) is as follows:

[0143]

[0144] Wherein, is the second loss value, diff3 is the first preference difference, diff4 is the second preference difference, and σ is the temperature parameter.

[0145] In the embodiments of the present application, by introducing the third difference and performing a third mapping on it to obtain the second loss value, not only the ability of the target model to distinguish between positive and negative labels of training samples and the convergence stability of the target model are improved, but also the interpretability and adaptability of the target model are enhanced.

[0146] In step 104, determine the convergence trend of the target model, and based on the convergence trend, determine the loss ratio factor.

[0147] As an example, when the current iteration round of the target model is the first round, determine the convergence trend of the target model based on the magnitude relationship between the first preference difference of the target model for the training samples and the second preference difference of the reference model for the training samples, and determine the loss ratio factor based on the convergence trend; when the current iteration round of the target model is not the first round, determine the convergence trend of the target model based on the change trend of the first preference difference of the target model for the training samples relative to the third preference difference of the target model for the training samples in the previous round, and determine the loss ratio factor based on the convergence trend.

[0148] In some embodiments, Figure 3A "Determining the convergence trend of the target model" in step 104 shown can be achieved through the following processing: when the current iteration round of the target model is the first round and the second preference difference is less than the first preference difference, determine the positive convergence as the convergence trend of the target model; when the current iteration round is the first round and the second preference difference is greater than or equal to the first preference difference, determine the negative convergence as the convergence trend of the target model; when the current iteration round is not the first round, obtain the third preference difference of the target model for the training samples in the previous round, and when the third preference difference is less than the first preference difference, determine the positive convergence as the convergence trend of the target model; when the third preference difference is greater than or equal to the first preference difference, determine the negative convergence as the convergence trend of the target model.

[0149] Among them, positive convergence indicates that the likelihood probability of the prediction result of the target model being a positive label is significantly greater than the likelihood probability of the prediction result being a negative label, that is, the prediction result of the target model has a strong tendency and is more inclined to the positive label, and negative convergence indicates that the likelihood probability of the prediction result of the target model being a positive label is relatively close to the likelihood probability of the prediction result being a negative label, that is, the prediction result of the target model lacks a clear tendency.

[0150] As an example, in the case where the current iteration round of the target model is the first round, judge the magnitude relationship between the first preference difference and the second preference difference. When the second preference difference is less than the first preference difference, determine the positive convergence as the convergence trend of the target model. When the second preference difference is greater than or equal to the first preference difference, determine the negative convergence as the convergence trend of the target model.

[0151] In the case where the current iteration round of the target model is not the first round, obtain the third preference difference of the target model for the training samples in the previous round, judge the magnitude relationship between the first preference difference and the third preference difference. When the third preference difference is less than the first preference difference, determine the positive convergence as the convergence trend of the target model. When the third preference difference is greater than or equal to the first preference difference, determine the negative convergence as the convergence trend of the target model.

[0152] In some embodiments, Figure 3A "Determining a loss ratio factor based on a convergence trend" in step 104 shown can be implemented through the following processing: when the convergence trend of the target model is positive convergence, determining the first value corresponding to the positive convergence as the loss ratio factor; when the convergence trend of the target model is negative convergence, determining the second value corresponding to the negative convergence as the loss ratio factor.

[0153] Wherein, the first value is a positive number less than the second value.

[0154] As an example, when the convergence trend of the target model is positive convergence, it indicates that the training samples are positive convergence samples, and the positive convergence samples are samples with a significant difference between the positive label and the negative label, and determining the first value corresponding to the positive convergence as the loss ratio factor; when the convergence trend of the target model is negative convergence, it indicates that the training samples are negative convergence samples, and the negative convergence samples are samples with an insignificant difference between the positive label and the negative label, and determining the second value corresponding to the negative convergence as the loss ratio factor.

[0155] In the embodiments of the present application, when the current iteration round of the target model is the first round, by comparing the first preference difference of the target model and the second preference difference of the reference model, the initial convergence direction of the target model can be quickly determined, and in subsequent rounds, by comparing the first preference difference and the third preference difference of the target model for the training samples in the previous round, the real-time convergence situation of the target model is determined, and then based on the convergence trend of the target model, the identification of positive convergence samples and negative convergence samples in the training sample set is realized, so as to dynamically adjust the loss ratio factor based on the types (positive convergence samples and negative convergence samples) of the training samples of the target model.

[0156] In step 105, based on the loss ratio factor, the first loss value and the second loss value are fused to obtain a total loss value, and based on the total loss value, the parameters of the target model are updated.

[0157] Wherein, the loss ratio factor is the weight of the first loss in the total loss, and the total loss is the loss used for the iterative training of the target model.

[0158] As an example, based on the loss ratio factor, the first loss value and the second loss value are weighted and fused to obtain a total loss value, and by minimizing the total loss value, backpropagation is performed on the target model to update the parameters of the target model. Among them, backpropagation on the target model can use gradient descent or other optimization algorithms, which are not limited herein.

[0159] It can be understood that in each iteration round, the target model can be iteratively trained with each training sample included in the training sample set as the dimension, or the J training samples included in the training sample set can be evenly divided into multiple training batches, and the target model can be iteratively trained with each training batch as the dimension. The embodiments of the present application do not limit this.

[0160] In some embodiments, Figure 3A "Based on the loss ratio factor, fusing the first loss value and the second loss value to obtain the total loss value" in step 105 shown can be achieved through the following processing: determining the loss ratio factor as the first weight of the first loss value, and determining the second weight of the second loss value based on the loss ratio factor; based on the first weight and the second weight, performing weighted summation on the first loss value and the second loss value to obtain the total loss value.

[0161] As an example, determining the loss ratio factor as the first weight of the first loss value, the first weight can be represented as a, and determining the difference between 1 and the loss ratio factor as the second weight of the second loss value, the second weight can be represented as 1 - a. Through formula (1.7), based on the first weight and the second weight, performing weighted summation on the first loss value and the second loss value to obtain the total loss value. Formula (1.7) is as follows:

[0162]

[0163] where L is the total loss value, a is the loss ratio factor and the first weight, (1 - a) is the second weight, loss is the first loss, is the second loss.

[0164] In the embodiments of the present application, when the convergence trend of the target model is positive convergence, it indicates that the training sample is a positive convergence sample. The weight of the first loss in the total loss is less than the weight of the second loss in the total loss. The learning of the target model for the training sample will be dominated by the second loss value. Then, by minimizing the total loss, the target model is guided to learn in the direction of expanding the difference between the first preference difference and the second preference difference; when the convergence trend of the target model is negative convergence, it indicates that the training sample is a negative convergence sample. The weight of the first loss in the total loss is greater than the weight of the second loss in the total loss. The learning of the target model for the training sample will be dominated by the first loss value. By minimizing the total loss, the preference difference of the target model for the training sample is guided to approach the first preference scaling factor.

[0165] In the application scenario of autonomous driving, the model training method provided by the embodiments of this application can be used for training a target detection model. The target detection model needs to identify target objects such as vehicles, pedestrians, and traffic signs from road images and accurately frame their positions. The following details the specific training process of the target detection model.

[0166] It should be noted that the target detection sample set (i.e., the training sample set) for training the target detection model (i.e., the target model) includes J target detection samples (i.e., training samples). Through each of the J target detection samples, the target detection model is iteratively trained, where J is a positive integer greater than 1. The target detection sample includes: an image showing the target object (i.e., feature information), the position information of the target object in the image (i.e., the positive label), and the position information of the area of non-target objects or the background area in the image (i.e., the negative label).

[0167] First, the target detection model performs target detection on the image showing the target object, obtaining the first positive preference probability that the target detection result of the target detection model is the position information of the target object in the image, and the first negative preference probability that the target detection result of the target detection model is the position information of the area of non-target objects or the background area in the image. Based on the difference between the first positive preference probability and the first negative preference probability, the first preference difference of the target detection model for the target detection sample is determined.

[0168] Then, the reference model for strengthening the learning of the target detection model performs target detection on the image showing the target object, obtaining the second positive preference probability that the target detection result of the reference model is the position information of the target object in the image, and the second negative preference probability that the target detection result of the reference model is the position information of the area of non-target objects or the background area in the image. Based on the difference between the second positive preference probability and the second negative preference probability, the second preference difference of the reference model for the target detection sample is determined.

[0169] Finally, based on the first preference difference and the first preference scaling factor of the target detection model for the target detection sample, the first loss value is determined, and based on the first preference difference and the second preference difference, the second loss value is determined. And based on the convergence trend of the target detection model, the loss ratio factor is determined. Based on the loss ratio factor, the first loss value and the second loss value are fused to obtain the total loss value, and based on the total loss value, the parameters of the target detection model are updated.

[0170] In the application scenario of smart home control, the model training method provided by the embodiments of the present application can be used for training a voice command recognition model. The voice command recognition model needs to recognize operation instructions such as "turn on the light" and "close the curtain" from the user's voice instructions. The following will detail the specific training process of the voice command recognition model.

[0171] It should be noted that the voice command sample set (i.e., the training sample set) for training the voice command recognition model (i.e., the target model) includes J voice command samples (i.e., training samples). Through each voice command sample in the J voice command samples, the voice command recognition model is iteratively trained, where J is a positive integer greater than 1. The voice command sample includes: the user's voice signal (i.e., the feature information), the correct voice command text (i.e., the positive label), and the incorrect voice command text (i.e., the negative label).

[0172] First, the voice command recognition model is used to recognize the user's voice signal, obtaining the first positive preference probability that the recognition result of the voice command recognition model is the correct voice command text, and the first negative preference probability that the recognition result of the voice command recognition model is the incorrect voice command text. Based on the difference between the first positive preference probability and the first negative preference probability, the first preference difference of the voice command recognition model for the voice command sample is determined.

[0173] Then, the reference model for strengthening the learning of the voice command recognition model is used to recognize the user's voice signal, obtaining the second positive preference probability that the recognition result of the reference model is the correct voice command text, and the second negative preference probability that the recognition result of the reference model is the incorrect voice command text. Based on the difference between the second positive preference probability and the second negative preference probability, the second preference difference of the reference model for the voice command sample is determined.

[0174] Finally, based on the first preference difference and the first preference scaling factor of the voice command recognition model for the target detection sample, the first loss value is determined, and based on the first preference difference and the second preference difference, the second loss value is determined. And based on the convergence trend of the voice command recognition model, the loss ratio factor is determined. Based on the loss ratio factor, the first loss value and the second loss value are fused to obtain the total loss value, and based on the total loss value, the parameters of the voice command recognition model are updated.

[0175] Next, the exemplary application of the embodiments of the present application in the application scenario of intelligent script understanding will be described.

[0176] Intelligent script understanding is an important link to improve efficiency in the film and television industry from drama reviews to production. Extracting the summaries of script scenes based on large language models can help the platform quickly extract the content of each scene in the script, which is conducive to film and television reviewers quickly understanding the plot and scoring and evaluating the script. Furthermore, it can help the relevant production parties understand the plot in advance to judge whether a script is worth shooting. On this basis, as the script understanding ability of large language models improves, more tasks related to script understanding can be supported, such as tasks related to character development trajectories, character settings, and character relationships.

[0177] Generally, a script is hundreds of thousands of words long, consisting of dozens of episodes and up to tens of thousands of scenes. Manual reading is very time-consuming. In the method of automated script understanding based on artificial intelligence, it is necessary to collect script samples for training large language models. However, due to the ever-changing nature of language expression, it is difficult to obtain an absolutely correct summary text description of each script plot. In related technologies, the quality of multiple summaries predicted by the target model and the reference model is evaluated or compared, and this is used as feedback information for training the target model. For example, in the Direct Preference Optimization (DPO) reinforcement learning method, the model is trained based on training samples in the format of a feature information and two labels (positive label and negative label). Although it can improve the quality of the model's generated results, it is difficult to ensure the correctness of the model iteration direction, and the following problems are likely to occur:

[0178] 1) The optimization goal of the DPO reinforcement learning method is in the form of a - b. Among them, a is the probability distribution of the model's prediction results, and b is the probability distribution of the reference model's prediction results. The optimization result of DPO reinforcement learning may be that a becomes larger (the probability distribution becomes more dispersed) and b becomes smaller (the probability distribution becomes more concentrated), or both a and b may become smaller but the decrease in b is less than that in a, resulting in the model's training falling into a non-positive iteration direction.

[0179] See Figure 4 , Figure 4It is the first schematic diagram of direct preference optimization learning provided by an embodiment of the present application. Probability 401 represents the likelihood probability that the prediction result of the reference model (SFT model) is a bad answer (negative label), probability 402 represents the likelihood probability that the prediction result of the reference model is a good answer (positive label), probability 403 represents the likelihood probability that the prediction result of the model is a bad answer (negative label), probability 404 represents the likelihood probability that the prediction result of the model is a good answer (positive label). The purpose of the DPO reinforcement learning method is to make the difference diff3(a) between the likelihood probability that the prediction result of the model is a bad answer (negative label) and the likelihood probability that the prediction result of the model is a good answer (positive label) greater than the difference diff4(b) between the likelihood probability that the prediction result of the reference model is a bad answer (negative label) and the likelihood probability that the prediction result of the reference model is a good answer (positive label).

[0180] 2) When the output of the model includes many answers with insignificant differences, that is, the model will fall into a non-positive iteration direction, thereby compressing the loss space of the model, restricting the space for the model to generate good answers, and causing the reinforcement learning effect of the model to deteriorate.

[0181] See Figure 5 , Figure 5 It is the second schematic diagram of direct preference optimization learning provided by an embodiment of the present application. When there is an iteration direction in which both diff3 and diff4 are decreasing, that is, diff3'<diff3( Figure 4 where probability 404 shifts left to Figure 5 probability 501 in Figure 4 after one round of training adjustment, diff4'<diff4( Figure 5 where probability 402 shifts left to probability 502 in after one round of training adjustment), although the ratio between diff3' and diff4' is greater than the ratio between diff3 and diff4 at this time, the value range of the loss composed of diff3 and diff4 will be shrunk, and as the iteration progresses, it will cause a serious compression of the value range of diff3 (from range 503 to range 504), that is, it represents that the change in the likelihood probability of the model for good answers relative to the likelihood probability of the model for bad answers is limited, resulting in the compression of the solution space output by the model, that is, it cannot generate good answers with an obvious optimization effect compared to bad answers.

[0182] In view of the problems existing in the related art, an embodiment of the present application proposes a model training method. By performing data analysis on the feature space of training samples under a model, a comparison distribution of globally initialized prediction results (i.e., the reference space distribution mentioned above) is obtained. A constraint term that maintains positive convergence (i.e., the first loss value mentioned above) is set in contrastive learning, and this constraint term is adjusted by a sample-by-sample personalized scaling factor (i.e., the first preference scaling factor mentioned above), thereby controlling the learning direction of the target model. When any sample is a non-positively convergent sample (i.e., the negatively convergent sample mentioned above), the learning direction of the target model is adjusted through the constraint term, so as to prevent the target model from falling into a bad convergence direction, and the scaling factor is used to prevent the loss space of the target model from shrinking. Before learning, non-positively convergent samples are identified through the comparison distribution performance of all training samples, and optimization control is focused on non-positively convergent samples. Specifically, based on the relationship between each sample and the comparison distribution of the prediction result, a contrastive learning target is set. During model training, the comparison relationship predicted by the target model (i.e., the first preference difference mentioned above) is aligned with this target. Finally, the trained target model is applied to an intelligent script understanding system to support business use. Specifically:

[0183] 1) In view of the problem that the DPO reinforcement learning method in the related art cannot guarantee the correctness of the model convergence direction, contrastive learning based on non-positively convergent samples (positive alignment loss) is proposed, and each sample is learned by combining the reinforcement learning loss (DPO loss) and the positive alignment loss.

[0184] 2) Perform refined optimization control for each sample: Through the comparison distribution of each sample, the identification of non-positively convergent samples is realized, and different control methods are adopted for samples with different type tendencies (positively convergent samples and non-positively convergent samples).

[0185] 3) Form a script understanding system and downstream applications with rich generalization capabilities: In the script understanding system and the downstream character introduction system, the corresponding models are trained by the model training method provided in the embodiment of the present application, so as to accurately understand the plot information or summary based on the trained models, thereby improving their generalization capabilities for script information, and generating character trajectories and character settings.

[0186] Next, the implementation of the model training method provided in the embodiment of the present application in the script understanding system will be described.

[0187] The input of the script understanding system is a script, and the output can include information such as the script plot, character trajectories, and character settings. The script understanding system includes an abstract model and a character setting model. Among them, the abstract model and the character setting model can be two different models (trained separately with corresponding data using the model training method provided in the embodiments of the present application), or they can be one model (learned using the model training method provided in the embodiments of the present application and trained by mixing all the data).

[0188] See Figure 6 , Figure 6 is the first schematic diagram of the model training method provided in the embodiments of the present application.

[0189] Step 601, scene splitting, transfer to Step 602 and Step 603.

[0190] After uploading the script, split the script into scenes to obtain multiple scenes for each episode.

[0191] Step 602, abstract extraction.

[0192] Use the abstract model to extract the abstract for each scene, obtain the abstract for each scene, and save the abstract for each scene to the plot library. Among them, the plot library stores the plot, plot embedding information, plot abstract, character list, and scene time information (for example, the 3rd scene in the 1st episode). Among them, the plot embedding information can be applied to downstream scenarios such as similar plot retrieval or sorting, and the character list can be applied to any task that depends on character information downstream.

[0193] Step 603, character splitting, transfer to Step 604.

[0194] Split each scene into characters to obtain character information.

[0195] Step 604, obtain the query result of the relevant scenes of the target character, transfer to Step 605.

[0196] Based on the character information, obtain the query result of the relevant scenes of the target character from the plot library. For example, for the important character A, find all the plots related to the important character A from the plot library (search the character list of each scene, and the plot with this character is the relevant plot, and sort them according to the time sequence of the scenes).

[0197] Step 605, character setting extraction.

[0198] Use the character setting model to process all the plots related to the important character A, obtain the character setting and character story line of the important character A, and save them to the database. Furthermore, the plot understanding can be carried out based on the plot story line in the plot library and the character setting and character story line of the important character A.

[0199] The model training method provided by the embodiments of the present application can be applied not only to the script understanding system, but also to the story creation management system: for novelists and playwrights, after each scene of the story plot is written, it can be input into the story creation management system. The story creation management system extracts the abstract outline and compares it with the historical outlines in the inventory to check whether the story is full of ups and downs and reasonable, etc., thereby helping novelists and playwrights quickly sort out and plan the story line during the creation process and improve the creation efficiency; it can also be applied to the text value evaluation system: for film researchers, script critics, and script reviewers, the text value evaluation system can help them quickly understand and analyze the plot structure of the film, and score from various evaluation dimensions (for example, the proportion of twists and turns in the plot setting, the growth route of characters as the plot develops, and the proportion of suspense plots, etc.) according to the understanding results of the script or text for in-depth research and review.

[0200] The model training method provided by the embodiments of the present application can be applied to the training of large language models that use a contrast method to determine the reinforcement learning loss (DPO loss). For example, it can be used for large language models for knowledge answering (for certain texts (scripts, novels, news, blogs, etc.), large language models that provide knowledge answering).

[0201] Next, the training processes of the abstract model and the character model in the script understanding system will be described from a technical perspective.

[0202] The embodiments of the present application optimize the calculation of the loss value in contrastive learning and identify the type tendency of each sample (positive convergence sample or non-positive convergence sample), and can be applied to contrastive learning with an optimization target in the form of a - b.

[0203] See Figure 7 , Figure 7It is a flowchart for training a target model provided by an embodiment of the present application. The target model can be an abstract model or a persona model. First, the training set is predicted by a reference model 702 to obtain a prediction result 704 (a prediction matrix of size M*6400). Based on the prediction result 704, a loss space 705 in the form of a-b for the training set is determined. The reference model 702 can be an initial parameter version of the target model 701. Then, during the training process of the target model 701, each sample is predicted by the target model 701 to obtain a prediction result 703 (a prediction matrix of size M*6400) for each sample. Based on the prediction result 703 and the loss space 705, a scaling factor 706 is determined. Then, based on the scaling factor 706 and the prediction result 703, the loss space 705 is scaled and adjusted to obtain a reference loss space 707. Then, the reference loss space 707 is used to analyze the prediction result 703 to obtain a positive alignment loss 708 (i.e., the first loss above) and a contrast loss 709 (i.e., the second loss above) for training the target model 701. Since there is an obvious gap in the distribution of the prediction results of positive convergence samples and non-positive convergence samples in the loss space 705, based on the distribution performance of the prediction result 703, the sample type tendency (positive convergence sample or non-positive convergence sample) can be inferred. Then, the positive alignment loss 708 is constructed, and the weights of the contrast loss 709 and the positive alignment loss 708 are adjusted to obtain a total loss 710.

[0204] Among them, the positive alignment loss is a constraint loss used to prevent the prediction result of the target model for a sample from causing spatial compression. This constraint loss takes the prediction result of the target model as the learning objective and the loss space scaled according to certain rules as the supervision information, so that the prediction result of the target model converges to a non-compressed state. At the same time, based on the different sample type tendencies, the weights of the contrast loss (e.g., DPO loss) and the positive alignment loss are adjusted to obtain a total loss, thereby achieving the effect of reducing the proportion of reinforcement learning for non-positive convergence samples or samples with low confidence.

[0205] The model training method provided by the embodiment of the present application can be applied to the training of DPO reinforcement learning. It is equivalent to adding a positive alignment loss to the DPO loss. The obtained total loss can be expressed as: loss = (1 - a)*loss1 + a*loss2, where loss1 is the DPO loss and loss2 represents the positive alignment loss. For positive convergence samples, the total loss is dominated by loss1. For non-positive convergence samples, since directly using the DPO loss for learning easily causes the learning of the target model to fall into non-positive convergence, the positive alignment loss loss2 plays a dominant role.

[0206] Next, the model training method provided by the embodiments of the present application will be introduced from aspects such as data collection and preparation, model preparation, model training, and model application.

[0207] 1. Data collection and preparation.

[0208] Prepare the scene plot comparison data for reinforcement learning. The scene plot comparison data includes the script text and two summaries for the same script text. Mark the accepted summary as chosen (positive label) and the unaccepted summary as rejected (negative label). A piece of scene plot comparison data can be represented as (text, yl, yw), where text is the script text (i.e., the feature information above), yl is the unaccepted summary, and yw is the accepted summary.

[0209] Use a large language model (e.g., GPT4, Hunyuan, and Baichuan) to ask questions about the obtained script text. The question statement is "The following is the script text of a certain scene in a drama. Please extract the summary of this scene plot, requiring a description of the specific events that occurred in the scene without speculation or abstract description. The script text is text". Since the information output by the large language model may have detailed errors or character misalignments at this time, the information output by the large language model can be used as the unaccepted summary, and then the information output by the large language model is manually corrected to obtain the accepted summary.

[0210] The character setting data and the character development trajectory can be extracted in the same way. Among them, the prompt for extracting the character setting data using the large language model can be: "The following is the plot related to character A in drama A. Please summarize the character setting of this role. The plot is xxx."; The prompt for extracting the character development trajectory using the large language model can be: "The following is the plot related to character A in drama A. Please summarize the character growth trajectory of this role. The plot is xxx.", where "xxx" represents the plot summary of the scenes where character A appears.

[0211] Since the data preparation and model training processes of the character setting model are the same as those of the summary model, only the summary model will be introduced as an example below.

[0212] According to multiple plot comparison data collected (text, yl, yw), a training dataset D0 for training the summary model is constructed. The format of each piece of training data is as follows: {"prompt": "The following is the script of a scene in a drama. Please extract the summary of the plot of this scene, requiring a description of the specific things that happened in the scene, without speculation or abstract description. The script text is xxx.", "prompt_id": "0001", "chosen": [{"content": "The following is the script of a scene in a drama. Please extract the summary of the plot of this scene, requiring a description of the specific things that happened in the scene, without speculation or abstract description. The script text is xxx.", "role": "user"}, {"content": "A", "role": "assistant"}], "rejected": [{"content": "The following is the script of a scene in a drama. Please extract the summary of the plot of this scene, requiring a description of the specific things that happened in the scene, without speculation or abstract description. The script text is xxx.", "role": "user"}, {"content": "B", "role": "assistant"}]}, where {} encloses a piece of training data. A represents the accepted plot summary yw, B represents the unaccepted plot summary yl, the script text xxx is the text of a specific scene in a specific movie or TV drama script, Id is the serial number of this piece of training data, the text of role user is input to the input end of the model, and the loss related to the predicted text of role assistant in the above data is calculated for the predicted text output by the model.

[0213] 2. Model preparation.

[0214] The summary model can adopt the Baichuan-7B model, and its structure is mainly a structure stacked with multiple attention layers (transformer), as shown in Table 1:

[0215] Table 1: Summary model structure [[ID=Il]]

[0216] Layer Name Type Number of Layers Embedding_tokens Embedding Layer 1 Model layers Decoding Layer 32 lm_head Fully Connected Layer 1

[0217] Among them, the output of lm_head is the prediction of the embedded encoding (token ids). For example, when the dictionary size is 64,000 (i.e., ids length = 64000), the prediction result is a 1×64000 vector. The structure of the decoding layer can be seen in Table 2:

[0218] Table 2: Decoding layer structure of the summary model

[0219] Layer Name Type Number of Layers input_layernorm Root Mean Square Normalization Layer 1 self_attn Attention Layer 1 post_attention_layernorm Root Mean Square Normalization Layer 1 mlp Multi-Layer Perceptron 1

[0220] First, based on the open-source language model (Baichuan-7B model), the open-source language model is fine-tuned through a training dataset to obtain a reference model (SFT fine-tuned model). The summary model is initialized with the parameters of the SFT fine-tuned model, and the parameters of the summary model are updated through reinforcement learning, while the parameters of the reference model do not participate in the update. Among them, the structures of the Baichuan-7B model, the reference model, and the summary model are exactly the same, but the training samples or training methods are different, and the model parameters are also different.

[0221] 3. Model training.

[0222] Before globally training the summary model, the model training method provided in the embodiment of the present application needs to predict the loss space of the training dataset through the reference model to obtain the global reference loss space (i.e., the reference space distribution mentioned above). Then, during training, the global reference loss space is stretched and projected according to the distribution performance of each sample in the global reference loss space to obtain the target of the reference loss space of the sample (i.e., the first preference stretching factor mentioned above). Based on the target of the reference loss space of the sample, the learning target is reset, and then the current sample to be learned is aligned with the target of the reference loss space of the sample to achieve the forward loss learning after the target reset for the current sample to be learned.

[0223] See Figure 8 , Figure 8 is the second schematic diagram of the model training method provided in the embodiment of the present application. During the forward propagation process, the script text is used as the input text and input into the summary model 801 (i.e., Figure 7 the model in) and the reference model 802 respectively. The input is converted into words in the dictionary through the encoder, and then the text embedding representation (word embedding) of the script text is obtained. For example, for a dictionary with 6400 words, the representation of each word in the dictionary is a 1*4096-dimensional vector, resulting in a 6400*4096 text embedding representation. After the model outputs the prediction code (prediction id) of the first word, the first word passes through the encoder again to obtain its text embedding representation and then is input into the model to obtain the prediction id of the second word, and so on until all words are predicted, obtaining the prediction results 803 and 804 corresponding to the summary model 801 and the reference model 802 respectively.

[0224] See Figure 9 , Figure 9 is the x-process schematic diagram of the model training method provided in the embodiment of the present application. The training process of reinforcement learning will be described below in combination with steps 901 to 903.

[0225] Step 901, parameter initialization.

[0226] Obtain a reference model trained through fine-tuning, and initialize the summary model using the reference model. Among them, the parameters of the reference model are not involved in the update. The model structures of the reference model and the summary model can refer to the open-source Baichuan-7B model, and the pre-trained Baichuan-7B model can also be used to initialize the reference model.

[0227] Step 902, set the learning parameters.

[0228] Perform full-parameter fine-tuning on the summary model, that is, update all the parameters of the summary model.

[0229] Step 903, reinforcement learning.

[0230] Randomly extract multiple unlearned training samples from the training dataset and perform batch training until each training sample has been traversed once, which is considered to complete one round (epoch) of iteration. Repeat multiple rounds of iteration until the average loss of any round (the average of the losses of each batch in this round of iteration) no longer decreases.

[0231] During the reinforcement learning process of each batch, steps 9031 to 9036 can be executed, which will be specifically described below.

[0232] Step 9031, obtain the training data of the current batch.

[0233] Step 9032, model forward propagation. Respectively predict the current sample to be learned in the training data through the reference model and the summary model to obtain the prediction results.

[0234] Step 9033, calculate the scaling factor of the current sample according to the global reference loss space.

[0235] Step 9034, calculate the forward alignment loss and the reinforcement learning loss according to the scaling factor of the current sample, and determine the loss ratio factor. Based on the loss ratio factor, perform weighted summation on the forward alignment loss and the reinforcement learning loss to obtain the total loss.

[0236] Step 9035, model backward propagation. Calculate the gradients of each parameter of the network based on the total loss backpropagating through the network.

[0237] Step 9036, model parameter update. Update the parameters according to the gradients of each parameter in the network.

[0238] Next, the method for determining the total loss of reinforcement learning will be described.

[0239] The total loss can be characterized as: loss = (1 - α) * loss1 + α * loss2, where loss1 is the reinforcement learning loss (DPO loss), loss2 represents the forward alignment loss, and α is the loss ratio factor. For forward convergence samples, the learning direction of reinforcement learning is dominated by the DPO loss (loss1), and α can be less than 0.5. For non-forward convergence samples, the learning direction of reinforcement learning is dominated by the forward alignment loss (loss2), and α can be greater than 0.5.

[0240] 1) Reinforcement learning loss (DPO loss).

[0241] See Figure 8 , the supervision information 805 is the supervision information of the accepted summary (good summary summ1), and the supervision information 806 is the supervision information of the unaccepted summary (bad summary summ2). Based on the prediction result of the summary model 801 (the output result of the fully connected layer of the summary model 801), the supervision information 805, and the supervision information 806, determine the prediction probability 807 (Pw) of the summary model 801 outputting an accepted summary and the prediction probability 808 (Pl) of an unaccepted summary. At the same time, based on the prediction result of the reference model 802 (model0) (the output result of the fully connected layer of the reference model 802), the supervision information 805, and the supervision information 806, determine the prediction probability 810 (Pw_0) of the reference model 802 outputting an accepted summary and the prediction probability 809 (Pl_0) of an unaccepted summary. Among them, the reference model 802 has the same model structure as the summary model 801 but different model parameters. During the reinforcement learning process, the parameters of the reference model 802 are not updated.

[0242] Then, based on the prediction probability 807, the prediction probability 808, the prediction probability 809, and the prediction probability 810, determine the DPO loss (i.e., the reinforcement learning loss 811). The determination formula (2.1) of the DPO loss is as follows:

[0243]

[0244] Formula (2.1) can be rewritten as the following formula (2.2) to characterize the relative relationship of the difference between the prediction probabilities of good and bad answers (accepted summary and unaccepted summary) under the summary model and the reference model. The determination formula (2.2) is as follows:

[0245]

[0246] Among them, is the DPO loss, x is the input text (e.g., script text), y w is the accepted summary (chosen), y lIs the rejected abstract, π θ Is the abstract model, π ref Is the reference model, π θ (y w (y|x) is the probability that the abstract model π θ Produces a predicted result that the abstract y is accepted w , π θ (y l (y|x) is the probability that the abstract model π θ Produces a predicted result that the abstract y is not accepted l , π ref (y w (y|x) is the probability that the reference model π ref Produces a predicted result that the abstract y is accepted w , π ref (y l (y|x) is the probability that the reference model π ref Produces a predicted result that the abstract y is not accepted l , σ is the temperature parameter, and β is the balance parameter (usually taken as 0.1).

[0247] The abstract model is a model that has been iteratively optimized. The reference model is a model that does not participate in iterative optimization. Under fixed input text, the probability prediction π ref (y w (y|x) of the preferred data (accepted abstract) and the probability prediction π ref (y l (y|x) of the non-preferred data (unaccepted abstract) are fixed. On this basis, the training objective of the abstract model is to increase the generation probability of the abstract model for the preferred data π θ (y w (y|x), while reducing the generation probability of the non-preferred data π θ (y l (y|x). Therefore, the abstract model can be optimized through the DPO loss, and then the abstract model can learn in the direction of "generating preferred data with a greater probability than the reference model".

[0248] 2) Forward alignment loss.

[0249] The formula (2.2) for determining the DPO loss shown above can be equivalently transformed into -logσ(diff3 - diff4), where diff3 is the difference between the likelihood probability of the preference data and the likelihood probability of the non-preference data of the summary model (i.e., the first preference difference above), diff4 is the difference between the likelihood probability of the preference data and the likelihood probability of the non-preference data of the reference model (i.e., the second preference difference above), and the summary model makes the DPO loss smaller by learning, which is equivalent to making diff3 - diff4 larger, that is, promoting the value of diff3 relative to diff4 to increase continuously during the training process.

[0250] When the model training method provided by the embodiments of this application is applied to contrastive learning with an optimization target in the form of a - b, in order to avoid any sample causing a negative loss trend (for example, diff3 and diff4 both decrease), it is necessary to dominate the training of the summary model by the positive alignment loss for loss correction. When any sample causes a positive loss trend, it is necessary to dominate the training of the summary model by the DPO loss.

[0251] The positive alignment loss is used to guide diff3 - diff4 to increase. Refer to Figure 5 the situation where both diff3 and diff4 shown in Figure 4 decrease, that is, diff3'< diff3 ( Figure 5 the probability 404 in Figure 4 shifts to the left to Figure 5 the probability 501 in

[0252] Through the prediction of all samples in the training dataset by the reference model, the loss space composed of the prediction results of all samples can be obtained (the space with dotted lines as its sides generated by the four circular points in the figure, i.e., the initial loss space provided by the reference model for the first round of training). To prevent the loss space from being compressed, the average value avg_diff3 (i.e., the second average value mentioned above) and the maximum value max_diff3 (i.e., the first maximum value mentioned above) of all diffs in the loss space are determined respectively. Then, based on the diff3 of each sample in the training dataset predicted by the summary model, the scaling factor ratio_diff3 is determined, where ratio_diff3 = max_diff3 * diff3 / avg_diff3. Then, ratio_diff3 is clipped: when ratio_diff3 > max_diff3, ratio_diff3 is set to max_diff3; when ratio_diff3 < diff3, ratio_diff3 is set to diff3; in other cases, the value of ratio_diff3 remains unchanged. Through formula (2.3), the forward alignment loss corresponding to the current sample is determined. Formula (2.3) is as follows:

[0253]

[0254] where loss2 is the forward alignment loss, diff3 is the diff3 of each sample, and ratio_diff3 is the scaling factor.

[0255] In addition to determining the forward alignment loss through the method shown above, the forward alignment loss can also be determined through formula (2.4) using the combined constraint term of diff3 and diff4. The goal is to shrink diff4 and enlarge diff3. Refer to Figure 10 , Figure 10 which is the third schematic diagram of the model training method provided in the embodiments of this application. Using the combined constraint term of diff3 and diff4 to determine the forward alignment loss, diff4 is shrunk to diff4', that is, probability 1001 is shifted left to probability 1002 after one round of training adjustment, that is, diff3 is enlarged to diff3', and probability 1003 is shifted left to probability 1004 after one round of training adjustment. Formula (2.4) is as follows:

[0256] loss2 = D(diff3 ‖ ratio_diff3) + D(diff4 ‖ ratio_diff4) (2.4)

[0257] where loss2 is the forward alignment loss, and ratio_diff4 is the scaling factor (i.e., the second scaling factor mentioned above).

[0258] The determination method of the scaling factor ratio_diff3 shown above can be referred to to determine the scaling factor ratio_diff4: Through the prediction of all samples in the training dataset by the reference model, the loss space composed of the prediction results of all samples can be obtained. The average value avg_diff4 (i.e., the first average value above) and the minimum value min_diff4 (i.e., the first minimum value above) of all diffs in the loss space are determined respectively. Then, based on the prediction of diff4 for each sample in the training dataset by the reference model, ratio_diff4 = min_diff4 * diff4 / avg_diff4 is determined. The ratio_diff4 is clipped: when ratio_diff4 < min_diff4, ratio_diff4 is set to min_diff4; when ratio_diff4 > diff4, ratio_diff4 is set to diff4; in other cases, the value of ratio_diff4 remains unchanged.

[0259] 3) Identify the forward convergence samples and determine the loss ratio factor.

[0260] In the first round of training, the diff of each sample is obtained through the reference model (model0) and recorded as the initial diff of each sample. Before calculating the loss, the new_diff of the target model is obtained first. By comparing the initial diff with the new_diff, del_diff = new_diff - diff is calculated. When del_diff < 0, the current sample is a non-forward convergence sample, and the loss ratio factor a can be determined to be 0.9. When del_diff > 0, the current sample is a forward convergence sample, and the loss ratio factor a can be determined to be 0.1.

[0261] After the first round of training is completed, for each sample, the prediction result of the previous round is used as the initial diff. Each time before calculating the loss, the new_diff predicted by the target model in the current round is obtained first. By comparing the initial diff with the new_diff, del_diff = new_diff - diff is calculated. When del_diff < 0, the current sample is a non-forward convergence sample, and the loss ratio factor a can be determined to be 0.9. When del_diff > 0, the current sample is a forward convergence sample, and the loss ratio factor a can be determined to be 0.1.

[0262] 4. Model application.

[0263] The model training method provided by the embodiments of this application can be directly applied to the optimization of the above DPO reinforcement learning, and can also be applied to other similar tasks of a-b learning. At the same time, it can be applied not only to the reinforcement learning training of script summarization based on large language models, but also to other similar reinforcement learning tasks. For example, the reinforcement learning training of character setting Q&A models for scripts, the reinforcement learning training of general Q&A models for large language models, etc. Among them, for different tasks, only the training data corresponding to the task needs to be constructed. For example, when applied to the reinforcement learning training of character setting Q&A models for scripts, first, each script dialogue is processed by a large language model to collect the main character settings, and samples in the format of (script sentence, setting 1, setting 2...) are obtained. Then, based on the above samples, the model training method provided by the embodiments of this application is executed.

[0264] Based on the model training method provided by the embodiments of this application, a plot summary model and a character setting model can be trained to generate information such as the plot summary, character settings, and character storylines of the script, thereby assisting reviewers in quickly reading the script and understanding the plot content. For example, see Figure 6 Extract the summaries and character settings of each scene in sequence and output them to the user for preview in the format of "the character story development line of a certain number of scenes or episodes".

[0265] The beneficial effects produced by the model training method provided by the embodiments of this application include:

[0266] 1) Learn for each sample by combining the reinforcement learning loss and the positive alignment loss.

[0267] 2) Identify non-positively convergent samples by comparing the distribution, and perform different learning processes on different samples: train the model dominated by the DPO loss for positively convergent samples, and train the model dominated by the positive alignment loss for non-positively convergent samples.

[0268] 3) A script analysis system that supports the development of events (plot) and the development of characters (character settings, growth trajectories, etc.), and can be migrated to fields such as novel analysis and continuous news event text analysis.

[0269] Next, continue to describe the exemplary structure of the model training device 555 provided by the embodiments of this application as a software module. In some embodiments, as Figure 2 shown, the software modules in the model training device 555 stored in the memory 550 may include:

[0270] The forward propagation module 5551 is used to predict the training samples through the target model to obtain the first positive preference probability and the first negative preference probability, and determine the first preference difference of the target model for the training samples based on the difference between the first positive preference probability and the first negative preference probability.

[0271] The forward propagation module 5551 is further configured to use a reference model for strengthening the learning of the target model to predict the training sample, obtain a second positive preference probability and a second negative preference probability, and determine a second preference difference of the reference model for the training sample based on a difference between the second positive preference probability and the second negative preference probability.

[0272] The loss value determination module 5552 is configured to determine a first loss value based on the first preference difference and a first preference scaling factor of the target model for the training sample, and determine a second loss value based on the first preference difference and the second preference difference.

[0273] The loss value fusion module 5553 is configured to determine a convergence trend of the target model and determine a loss ratio factor based on the convergence trend.

[0274] The loss value fusion module 5553 is further configured to fuse the first loss value and the second loss value based on the loss ratio factor to obtain a total loss value.

[0275] The backpropagation module 5554 is configured to update parameters of the target model based on the total loss value.

[0276] In some embodiments, the loss value determination module 5552 is further configured to determine a first difference between the first preference difference and the first preference scaling factor; perform a first mapping on the first difference to obtain a first mapping value, and determine the first loss value based on the first mapping value, where the first difference and the first loss value are positively correlated.

[0277] In some embodiments, the loss value determination module 5552 is further configured to determine a second preference scaling factor of the reference model for the training sample, and determine a second difference between the second preference difference and the second preference scaling factor; perform a second mapping on the second difference to obtain a second mapping value; fuse the first mapping value and the second mapping value to obtain the first loss value.

[0278] In some embodiments, the training sample is the j-th training sample among the J training samples included in the training sample set, where J is a positive integer, j is a positive integer that increases sequentially, and j ≤ J; the loss value determination module 5552 is further configured to obtain a reference space distribution formed by the preference differences of each of the J training samples by the reference model, and determine the average value of the J preference differences in the reference space distribution as the first average value, and determine the minimum value among the J preference differences in the reference space distribution as the first minimum value; based on the first minimum value, the first average value, and the second preference difference, determine the second preference scaling factor.

[0279] In some embodiments, the loss value determination module 5552 is further configured to determine a first ratio between the first minimum value and the first average value, and determine the product of the first ratio and the second preference difference as a first candidate scaling factor; based on the first minimum value and the second preference difference, clip the first candidate scaling factor to obtain the second preference scaling factor.

[0280] In some embodiments, the loss value determination module 5552 is further configured to determine a third difference between the first preference difference and the second preference difference; perform a third mapping on the third difference to obtain the second loss value, where the third difference is negatively correlated with the second loss value.

[0281] In some embodiments, the loss value determination module 5552 is further configured to determine the current iteration round of the target model, and perform statistics on the preference differences in the reference space distribution of the current iteration round to obtain a first maximum value and a second average value; based on the first maximum value, the second average value, and the first preference difference, determine the first preference scaling factor.

[0282] In some embodiments, the training sample is the j-th training sample among the J training samples included in the training sample set, where J is a positive integer, j is a sequentially increasing positive integer, and j ≤ J; the loss value determination module 5552 is further configured to, when the current iteration round is the first round, obtain the reference space distribution formed by the preference differences of each of the J training samples by the reference model, and determine the average value of the J preference differences in the reference space distribution as the second average value, and determine the maximum value among the J preference differences in the reference space distribution as the first maximum value; when the current iteration round is not the first round, obtain the reference space distribution formed by the preference differences of each of the training samples by the target model in the previous round, and determine the average value of the J preference differences in the reference space distribution as the second average value, and determine the maximum value among the J preference differences in the reference space distribution as the first maximum value.

[0283] In some embodiments, the loss value determination module 5552 is further configured to determine a second ratio between the first maximum value and the second average value, and determine the product of the second ratio and the first preference difference as the second candidate scaling factor; based on the first maximum value and the first preference difference, clip the second candidate scaling factor to obtain the first preference scaling factor.

[0284] In some embodiments, the loss value determination module 5552 is further configured to, when the second candidate scaling factor is greater than or equal to the first maximum value, clip the second candidate scaling factor to the first maximum value to obtain the first preference scaling factor; when the second candidate scaling factor is less than or equal to the first preference difference, determine the first preference difference as the first preference scaling factor; when the second candidate scaling factor is less than the first maximum value and greater than the first preference difference, clip the second candidate scaling factor to the first preference scaling factor to obtain the first preference scaling factor.

[0285] In some embodiments, the loss value fusion module 5553 is further configured to, when the current iteration round of the target model is the first round and the second preference difference is less than the first preference difference, determine the positive convergence as the convergence trend of the target model; when the current iteration round is the first round and the second preference difference is greater than or equal to the first preference difference, determine the negative convergence as the convergence trend of the target model; when the current iteration round is not the first round, obtain the third preference difference of the target model for the training sample in the previous round, and when the third preference difference is less than the first preference difference, determine the positive convergence as the convergence trend of the target model; when the third preference difference is greater than or equal to the first preference difference, determine the negative convergence as the convergence trend of the target model.

[0286] In some embodiments, the loss value fusion module 5553 is further configured to, when the convergence trend of the target model is the positive convergence, determine the first value corresponding to the positive convergence as the loss ratio factor; when the convergence trend of the target model is the negative convergence, determine the second value corresponding to the negative convergence as the loss ratio factor, where the first value is a positive number less than the second value.

[0287] In some embodiments, the loss value fusion module 5553 is further configured to determine the loss ratio factor as the first weight of the first loss value, and determine the second weight of the second loss value based on the loss ratio factor; perform a weighted sum of the first loss value and the second loss value based on the first weight and the second weight to obtain the total loss value.

[0288] An embodiment of the present application provides a computer program product, which includes computer executable instructions or a computer program, and the computer executable instructions or the computer program are stored in a computer-readable storage medium. A processor of an electronic device reads the computer executable instructions or the computer program from the computer-readable storage medium, and the processor executes the computer executable instructions or the computer program, so that the electronic device executes the model training method described above in the embodiments of the present application.

[0289] An embodiment of the present application provides a computer-readable storage medium, in which computer executable instructions or a computer program are stored, and when the computer executable instructions or the computer program are executed by a processor, the processor will be caused to execute the model training method provided by the embodiments of the present application.

[0290] In some embodiments, the computer-readable storage medium may be a memory such as RAM, ROM, flash memory, magnetic surface memory, optical disc, or CD-ROM; or may be various devices including one or any combination of the above memories.

[0291] In some embodiments, the computer-executable instructions may be in the form of a program, software, a software module, a script, or code, written in any form of programming language (including compiled or interpreted languages, or declarative or procedural languages), and may be deployed in any form, including being deployed as a stand-alone program or being deployed as a module, a component, a subroutine, or other unit suitable for use in a computing environment.

[0292] As an example, the computer-executable instructions may or may not correspond to a file in a file system, may be stored as part of a file that holds other programs or data, for example, in one or more scripts in a HyperText Markup Language (HTML) document, stored in a single file dedicated to the program being discussed, or, stored in multiple cooperating files (such as files that store one or more modules, subroutines, or portions of code).

[0293] As an example, the computer-executable instructions may be deployed to execute on one electronic device, or on multiple electronic devices located at one location, or, on multiple electronic devices distributed across multiple locations and interconnected by a communication network.

[0294] As described above, the above are only embodiments of the present application and are not intended to limit the protection scope of the present application. Any modifications, equivalent replacements, and improvements made within the spirit and scope of the present application are all included in the protection scope of the present application.

Claims

1. A model training method, characterized in that, The method includes: Predicting a training sample through a target model to obtain a first positive preference probability and a first negative preference probability, and determining a first preference difference of the target model for the training sample based on a difference between the first positive preference probability and the first negative preference probability; Predicting the training sample through a reference model for strengthening the target model to obtain a second positive preference probability and a second negative preference probability, and determining a second preference difference of the reference model for the training sample based on a difference between the second positive preference probability and the second negative preference probability; Determining a first loss value based on the first preference difference and a first preference scaling factor of the target model for the training sample, and determining a second loss value based on the first preference difference and the second preference difference; Determining a convergence trend of the target model, and determining a loss ratio factor based on the convergence trend; Fusing the first loss value and the second loss value based on the loss ratio factor to obtain a total loss value, and updating parameters of the target model based on the total loss value.

2. The method according to claim 1, wherein The determining the first loss value based on the first preference difference and the first preference scaling factor of the target model for the training sample includes: Determining a first difference between the first preference difference and the first preference scaling factor; Performing a first mapping on the first difference to obtain a first mapping value, and determining the first loss value based on the first mapping value, where the first difference and the first loss value are positively correlated.

3. The method according to claim 2, characterized in that, The determining the first loss value based on the first mapping value includes: Determining a second preference scaling factor of the reference model for the training sample, and determining a second difference between the second preference difference and the second preference scaling factor; Performing a second mapping on the second difference to obtain a second mapping value; Fusing the first mapping value and the second mapping value to obtain the first loss value.

4. The method according to claim 3, wherein The training sample is the j-th training sample among J training samples included in a training sample set, J is a positive integer, j is a positive integer that increases in sequence, and j ≤ J; The determining the second preference scaling factor of the reference model for the training sample includes: Obtaining a reference space distribution formed by preference differences of the reference model for each of the J training samples, determining an average value of the J preference differences in the reference space distribution as a first average value, and determining a minimum value among the J preference differences in the reference space distribution as the first minimum value; Determining the second preference scaling factor based on the first minimum value, the first average value, and the second preference difference.

5. The method according to claim 4, wherein The determining the second preference scaling factor based on the first minimum value, the first average value, and the second preference difference includes: Determining a first ratio between the first minimum value and the first average value, and determining a first candidate scaling factor as a product of the first ratio and the second preference difference; Based on the first minimum value and the second preference difference, clip the first candidate scaling factor to obtain the second preference scaling factor.

6. The method according to any one of claims 1-5, characterized in that, The determining the second loss value based on the first preference difference and the second preference difference includes: Determine a third difference between the first preference difference and the second preference difference; Perform a third mapping on the third difference to obtain the second loss value, where the third difference is negatively correlated with the second loss value.

7. The method according to claim 1, wherein Before determining the first loss value based on the first preference difference and the first preference scaling factor of the target model for the training sample, the method further includes: Determine the current iteration round of the target model, and statistically analyze the preference differences in the reference space distribution of the current iteration round to obtain a first maximum value and a second average value; Based on the first maximum value, the second average value, and the first preference difference, determine the first preference scaling factor.

8. The method according to claim 7, wherein The training sample is the j-th training sample among the J training samples included in the training sample set, J is a positive integer, j is a positive integer that increases sequentially, and j ≤ J; The statistically analyzing the preference differences in the reference space distribution of the current iteration round to obtain a first maximum value and a second average value includes: When the current iteration round is the first round, obtain the reference space distribution composed of the preference differences of the reference model for each of the J training samples, and determine the average value of the J preference differences in the reference space distribution as the second average value, and determine the maximum value among the J preference differences in the reference space distribution as the first maximum value; When the current iteration round is not the first round, obtain the reference space distribution composed of the preference differences of the target model in the previous round for each of the training samples, and determine the average value of the J preference differences in the reference space distribution as the second average value, and determine the maximum value among the J preference differences in the reference space distribution as the first maximum value.

9. The method according to claim 7, characterized in that The determining the first preference scaling factor based on the first maximum value, the second average value, and the first preference difference includes: Determine a second ratio between the first maximum value and the second average value, and determine the product of the second ratio and the first preference difference as the second candidate scaling factor; Based on the first maximum value and the first preference difference, clip the second candidate scaling factor to obtain the first preference scaling factor.

10. The method according to claim 9, characterized in that, The clipping the second candidate scaling factor based on the first maximum value and the first preference difference to obtain the first preference scaling factor includes: When the second candidate scaling factor is greater than or equal to the first maximum value, clip the second candidate scaling factor to the first maximum value to obtain the first preference scaling factor; When the second candidate scaling factor is less than or equal to the first preference difference, determine the first preference difference as the first preference scaling factor; When the second candidate scaling factor is less than the first maximum value and greater than the first preference difference, clip the second candidate scaling factor to the first preference scaling factor to obtain the first preference scaling factor.

11. The method according to any one of claims 1-10, wherein: The determining the convergence trend of the target model includes: When the current iteration round of the target model is the first round and the second preference difference is less than the first preference difference, determine the positive convergence as the convergence trend of the target model; When the current iteration round is the first round and the second preference difference is greater than or equal to the first preference difference, determine the negative convergence as the convergence trend of the target model; When the current iteration round is not the first round, obtain the third preference difference of the target model for the training sample in the previous round. When the third preference difference is less than the first preference difference, determine the positive convergence as the convergence trend of the target model; when the third preference difference is greater than or equal to the first preference difference, determine the negative convergence as the convergence trend of the target model; The determining the loss ratio factor based on the convergence trend includes: When the convergence trend of the target model is the positive convergence, determine the first value corresponding to the positive convergence as the loss ratio factor; when the convergence trend of the target model is the negative convergence, determine the second value corresponding to the negative convergence as the loss ratio factor, where the first value is a positive number less than the second value.

12. The method according to claim 11, wherein The fusing the first loss value and the second loss value based on the loss ratio factor to obtain the total loss value includes: Determine the loss ratio factor as the first weight of the first loss value, and determine the second weight of the second loss value based on the loss ratio factor; Based on the first weight and the second weight, perform weighted summation on the first loss value and the second loss value to obtain the total loss value.

13. A model training device, characterized in that, The device includes: A forward propagation module, configured to predict a training sample through a target model to obtain a first positive preference probability and a first negative preference probability, and determine a first preference difference of the target model for the training sample based on a difference between the first positive preference probability and the first negative preference probability; The forward propagation module is further configured to predict the training sample through a reference model for strengthening the target model to obtain a second positive preference probability and a second negative preference probability, and determine a second preference difference of the reference model for the training sample based on a difference between the second positive preference probability and the second negative preference probability; A loss value determination module, configured to determine a first loss value based on the first preference difference and the first preference scaling factor of the target model for the training sample, and determine a second loss value based on the first preference difference and the second preference difference; A loss value fusion module, further configured to determine the convergence trend of the target model, and determine a loss ratio factor based on the convergence trend; The loss value fusion module is further configured to fuse the first loss value and the second loss value based on the loss ratio factor to obtain a total loss value; A backpropagation module, configured to update the parameters of the target model based on the total loss value.

14. An electronic device, characterized in that, The electronic device includes: A memory, configured to store computer-executable instructions or a computer program; A processor, configured to implement the model training method according to any one of claims 1 to 12 when executing the computer-executable instructions or the computer program stored in the memory.

15. A computer-readable storage medium storing computer-executable instructions or a computer program, characterized in that, The computer-executable instructions or the computer program, when executed by the processor, implement the model training method according to any one of claims 1 to 12.

16. A computer program product, comprising computer-executable instructions or a computer program, characterized in that, The computer-executable instructions or the computer program, when executed by the processor, implement the model training method according to any one of claims 1 to 12.

Images