Method and device for training large language model

Abstract

The present application discloses a method and apparatus for training a large language model, the method comprising: an acquisition step of acquiring a training data set, wherein the training data set comprises multimodal training data; a training step of using the training data set to train the large language model; during the training process, based on the correlation between the gradient of each modality and the gradient of other modalities, determining the collaborative adjustment gradient of each modality, for adjusting the model parameters for each modality in the large language model in real time, thereby obtaining a trained large language model, wherein the trained large language model is used to identify medical-related questionnaires. The present invention can improve the accuracy of questionnaire recognition.

Description

Technical Field

[0001] The present application relates to the field of artificial intelligence, and in particular to methods, devices, media, electronic devices, and computer program products for training large language models. Background Art

[0002] In the field of medical information technology, clinical questionnaires, as a core vehicle for collecting patient information, are facing a growing demand for digital processing. Existing clinical questionnaire recognition technology primarily relies on a combination of traditional optical character recognition (OCR) technology and rule-based engines. Handwritten text recognition typically employs a single-modal model based on a convolutional neural network (CNN). Checkmark recognition relies on an image binarization threshold segmentation algorithm. Symbols (such as arrows and asterisks) often rely on matching with preset templates.

[0003] Core challenges in automating clinical questionnaire processing include: difficult-to-decipher physicians' illegible handwriting, complex abbreviations for specialized terminology, ambiguous associations between checkmarks and text options, and diverse and information-dense table structures. Traditional methods break questionnaire content down into independent modalities and process them separately, resulting in a loss of semantic connections. Specifically, in traditional multimodal training methods, feature extractors for each modality are typically optimized independently, with only a shallow integration achieved at the fusion layer. This leads to semantic inconsistencies and feature mismatches between modalities. Consequently, accurate questionnaire recognition is impossible. Summary of the Invention

[0004] Embodiments of the present application provide a method for training a large language model, an apparatus, a medium, an electronic device, and a computer program product for training a large language model.

[0005] In a first aspect, an embodiment of the present application provides a method for training a large language model for an electronic device, the method comprising:

[0006] An acquisition step of acquiring a training data set, wherein the training data set includes multimodal training data;

[0007] A training step, using the training data set to train the large language model, wherein during the training process, based on the correlation between the gradient of each modality and the gradient of other modalities, a collaborative adjustment gradient of each modality is determined, which is used to adjust the model parameters for each modality in the large language model in real time, thereby obtaining a trained large language model,

[0008] The trained large language model is used to identify medical-related questionnaires.

[0009] In a second aspect, an embodiment of the present application provides a device for training a large language model, the device comprising:

[0010] An acquisition unit, which acquires a training data set, wherein the training data set includes multimodal training data;

[0011] A training unit trains the large language model using the training data set, wherein during the training process, based on the correlation between the gradient of each modality and the gradients of other modalities, a collaborative adjustment gradient of each modality is determined, and used to adjust the model parameters for each modality in the large language model in real time, thereby obtaining a trained large language model.

[0012] The trained large language model is used to identify medical-related questionnaires.

[0013] In a third aspect, an embodiment of the present application provides a computer-readable storage medium having instructions stored thereon, which, when executed on a computer, causes the computer to execute the method for training a large language model in the first aspect described above.

[0014] In a fourth aspect, an embodiment of the present application provides an electronic device comprising: one or more processors; one or more memories; the one or more memories storing one or more programs, which, when executed by the one or more processors, enables the electronic device to execute the method for training a large language model in the first aspect above.

[0015] In a fifth aspect, an embodiment of the present application provides a computer program product, comprising computer-executable instructions, which are executed by a processor to implement the method for training a large language model in the first aspect.

[0016] The basic idea of ​​the CCGCR mechanism used in this invention is to dynamically and collaboratively adjust the gradient flows of different modalities (such as vision, text, and structure) during the backpropagation process, so that the features extracted by the feature extractors of each modality can "perceive" and adapt to each other during the training process.

[0017] The present invention can enhance the association between medical text and labels. By collaborating on the gradients of visual and linguistic modalities, the large language model's understanding of key associations is enhanced. For example, in a diabetes risk assessment form, patients are required to simultaneously fill in their blood glucose values ​​and select the time point (fasting / postprandial). Traditional methods often separate number recognition from checkbox recognition, resulting in association errors. The CCGCR mechanism in the present invention uses cross-modal gradient collaboration to enable the trained large language model to learn the semantic association between numerical values ​​and time point labels, accurately understanding the complete medical semantics of "fasting blood glucose 6.2mmol / L."

[0018] This invention optimizes the recognition of medical terminology. By leveraging the gradient synergy between language and visual modalities, it enhances the recognition of medical abbreviations (e.g., "HTN" for hypertension and "CVA" for cerebrovascular accident). For example, in a cardiology questionnaire, even when presented with a physician's handwritten phrase "Pt c AF+RVR" (patient with atrial fibrillation and rapid ventricular rate response), the trained large language model can correctly identify these abbreviations within the context.

[0019] This invention can adapt to complex handwriting scenarios, promoting the integration of visual features and language context through CCGCR, significantly improving the recognition of illegible handwriting. In emergency medical records, even when faced with extremely illegible handwriting due to urgent medical emergencies, the trained large language model can accurately interpret medical context (such as common symptom combinations and medication regimens), reducing the character error rate from approximately 15% to approximately 3.3%. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] Figure 1 According to an embodiment of the present application, a flowchart of a method for training a large language model is shown;

[0021] Figure 2 According to an embodiment of the present application, a structural diagram of an apparatus for training a large language model is shown;

[0022] Figure 3 According to an embodiment of the present application, a block diagram of an electronic device is shown. DETAILED DESCRIPTION

[0023] The illustrative embodiments of the present application include, but are not limited to, methods, apparatuses, media, electronic devices, and computer program products for training large language models.

[0024] The embodiments of the present application will be described in further detail below with reference to the accompanying drawings.

[0025] Figure 1 The method for training a large language model in an embodiment of the present application is shown. Figure 3 Electronic devices shown.

[0026] Large language models include Tongyi Qianwen, GPT-4, Wenyan Yixin, DeepSeek-V2, etc. In this embodiment, Tongyi Qianwen is used as an example for detailed explanation.

[0027] In the acquisition step S11 , a training data set is acquired, where the training data set includes multimodal training data.

[0028] Specifically, we first collected a comprehensive clinical questionnaire dataset, which included, for example: 100,000 completed samples of 20 international standardized scales, 300,000 actual clinical questionnaires from 30 tertiary hospitals, 100,000 special samples (such as cursive handwriting and various marking methods), and a corpus of 850,000 medical terms.

[0029] Secondly, a three-level mechanism was used to annotate the data in the collected dataset. For example, basic recognition annotation: annotating handwritten text character by character, marking position and status; semantic association annotation: the relationship between tags and corresponding text options, and the correspondence between numbers and units; and structured output annotation: converting questionnaire content into a standardized data format.

[0030] Next, the data undergoes specialized preprocessing tailored to medical scenarios. For example, these preprocessing algorithms utilize adaptive image enhancement, questionnaire layout analysis, and medical scenario data enhancement. The adaptive image enhancement algorithm can address issues like low-quality scans and paper creases. The questionnaire layout analysis algorithm can identify table structures and field types. The medical scenario data enhancement algorithm can simulate sloppy handwriting in emergency settings, shaky handwriting in elderly patients, and multiple abbreviations of drug names.

[0031] It can be understood that by performing the above preprocessing, the final training data can be made clearer.

[0032] The multimodal training data acquired in step S11 includes visual training data, language training data, structural training data, etc. It is understood that multimodality includes vision, language, structure, etc., and visual training data includes, for example, images, language training data includes, for example, text, and structural training data includes, for example, tables.

[0033] In training step S12, the large language model is trained using the training data set. During the training process, the collaborative adjustment gradient of each modality is determined based on the correlation between the gradient of each modality and the gradient of other modalities, which is used to adjust the model parameters for each modality in the large language model in real time, thereby obtaining a trained large language model.

[0034] In this embodiment, the Cross-Collaborative Gradient Consistency Regularization (CCGCR) mechanism is used. The CCGCR mechanism dynamically calculates the correlation between the gradients of different modalities through a gradient coordination function, thereby determining the coordinated adjustment gradient of each modality. The optimization direction of each modality is adjusted based on the coordinated adjustment gradient, achieving deep coordination at the gradient level. Specifically, the coordinated adjustment gradient of each modality is determined based on the following formulas (1) and (2).

[0035] Formula (1):

[0036] ;

[0037] Formula (2):

[0038] ;

[0039] in, represents the model parameters of the ith mode, θj Represents the model parameters of the j-th modality, such as visual encoder parameters, language encoder parameters, or fusion layer parameters, etc. Represents the gradient of the i-th mode, that is, the task loss to the parameter The gradient of , that is, the original optimization direction of the i-th mode in the current training step, is understandable. During the training process, gradients are continuously generated and the gradients continue to decrease. Is the synergy strength adjustment coefficient, ranging from 0 to 1, used to control the strength of inter-modal synergy, and dynamically adjusted from small to large during training. represents the gradient of the j-th mode.

[0040] It is a gradient coordination function used to calculate the correlation between gradients of different modes, and the output range is 0~1. Represents the inner product of two gradients, which is used to measure the similarity of the gradient directions of the two modalities. represents the norm of the gradient of the i-th mode, represents the norm of the gradient of the j-th mode. is the adjustment coefficient used to control the steepness of the correlation curve. A larger β value makes the synergy coefficient closer to 0 or 1. is the sigmoid function, which is used to map the correlation to the (0,1) interval so that the synergy coefficient changes smoothly. is the collaborative adjustment gradient of the i-th mode, which is used to update the model parameters of each mode.

[0041] It is understood that the gradient cooperation function is used to determine the correlation between the gradient of each modality and the gradient of other modalities, that is, the gradient cooperation function is used to determine the correlation between the gradients of different modalities. The gradient cooperation function adopts an adaptive weighting strategy, for example, to strengthen the gradient direction when the gradient directions of the two modalities are consistent, and to weaken the gradient direction when the gradient directions of the two modalities conflict.

[0042] It's easy to understand that in deep learning, the gradient is the vector of partial derivatives of the loss function with respect to the model parameters. It indicates the direction in which the loss function increases fastest. During optimization, parameters are typically updated in the opposite direction of the gradient (i.e., gradient descent) to minimize the loss function. Therefore, the gradient direction can be intuitively understood as the direction in parameter space that increases the loss function.

[0043] In a multimodal model, the gradient directions of two different modalities (such as vision and text) may have the following relationship:

[0044] 1) Gradient direction is consistent

[0045] When the gradient directions of two modalities are the same (i.e., the gradient vector angle is close to 0° and the inner product is positive), it indicates that their influence on the loss function is consistent. For example, in clinical questionnaire recognition, the gradient directions of the visual modality (recognizing the image features of the number "3") and the language modality (parsing the text of "number of days") are consistent, indicating that they jointly drive the model toward correctly identifying "lasting 3 days."

[0046] 2) Gradient direction conflict

[0047] When the gradients of the two modalities point in opposite directions (i.e., the angle between the gradient vectors is close to 180°, resulting in a negative inner product), their impact on the loss function is conflicting. For example, if the visual modality incorrectly identifies "3" as "8," its gradient direction will push the model in the wrong direction for optimization; while the gradient of the language modality (correctly interpreting "days") points in the correct direction, the two gradients conflict.

[0048] In the CCGCR mechanism, the gradient direction refers to the direction of the gradient of the loss function with respect to the parameters of a modality (such as vision or text) during the current training step. Using an adaptive weighting strategy, CCGCR dynamically evaluates the consistency of gradient directions across modalities, strengthening collaborative optimization and mitigating conflicting interference, thereby achieving deep fusion and semantic alignment of multimodal feature extractors. This mechanism is particularly critical in clinical questionnaire recognition, effectively resolving the complex associations between handwritten text, markup symbols, and text semantics.

[0049] It is understandable that after calculating the collaborative adjustment gradient of the i-th mode Then, the model parameters of the i-th mode are Adjust and train multiple times.

[0050] It is understandable that during the training process, visual training data, such as questionnaire images, are input into the visual encoder to obtain visual features, and language training data, such as those containing relevant text prompts, are input into the language encoder to obtain language features. The fusion representation is calculated through the multimodal fusion layer to generate the prediction result. The prediction result is compared with the marked true result to calculate the task loss of each modality in the current training step. Mission Loss This involves multimodal task losses, such as visual loss, text loss, and structural loss. The total task loss, the sum of all task losses, is then calculated. For example, the sum of visual loss, text loss, and structural loss is calculated. When the total task loss stabilizes, the large language model training is complete, resulting in the trained large language model. The trained large language model is then used to identify medical questionnaires.

[0051] It is understood that in this embodiment, the correlation between the gradients of each two modalities is calculated separately, for example, the correlation between the gradients of vision-language, vision-structure, and language-structure, thereby adjusting the gradient direction of each modality and applying it to the model parameter update (adjustment). A lower synergy strength is used in the early stages of training to allow each modality to independently learn basic features, while the synergy strength is gradually increased in the later stages of training to strengthen the deep fusion between the modalities.

[0052] The basic idea of ​​the CCGCR mechanism used in this invention is to dynamically and collaboratively adjust the gradient flows of different modalities (such as vision, text, and structure) during the backpropagation process, so that the features extracted by the feature extractors of each modality can "perceive" and adapt to each other during the training process.

[0053] The present invention can enhance the association between medical text and labels. Through the gradient collaboration of visual modalities and language modalities, the large language model's understanding of key associations is enhanced. For example, in a diabetes risk assessment form, patients need to fill in their blood sugar values ​​and check the time point (fasting / postprandial) at the same time. Traditional methods often separate the recognition of numbers and the recognition of check boxes, resulting in association errors. The CCGCR mechanism in the present invention uses cross-modal gradient collaboration to enable the trained large language model to learn the semantic association between numerical values ​​and time point labels, accurately understanding the complete medical semantics of "fasting blood sugar 6.2mmol / L."

[0054] This invention optimizes the recognition of medical terminology. By leveraging the gradient synergy between language and visual modalities, it enhances the recognition of medical abbreviations (e.g., "HTN" for hypertension and "CVA" for cerebrovascular accident). For example, in a cardiology questionnaire, even when presented with a physician's handwritten phrase "Pt c AF+RVR" (patient with atrial fibrillation and rapid ventricular rate response), the trained large language model can correctly identify these abbreviations within the context.

[0055] This invention can adapt to complex handwriting scenarios, promoting the integration of visual features and language context through CCGCR, significantly improving the recognition of illegible handwriting. In emergency medical records, even when faced with extremely illegible handwriting due to urgent medical emergencies, the trained large language model can accurately interpret medical context (such as common symptom combinations and medication regimens), reducing the character error rate from approximately 15% to approximately 3.3%.

[0056] Table 1 shows the accuracy of various key indicators when using traditional OCR, a general large model, and a large model trained using the present invention (CCGCR mechanism) to recognize questionnaires. As can be seen from Table 1, the large language model trained using the present invention achieves significant improvements in all key indicators, increasing the accuracy of questionnaire recognition.

[0057] Table 1

[0058]

[0059] The present invention is based on a general large model (such as Tongyi Qianwen), and through the cross-collaborative gradient consistency regularization (CCGCR) mechanism, constructs a large language model for intelligent recognition of clinical questionnaires. It breaks through the limitations of traditional methods and realizes the deep fusion and understanding of multimodal information such as handwritten text and marked symbols in medical scenarios. The core innovation of the CCGCR mechanism is to achieve deep collaboration at the gradient level, so that the feature extractors of each modality can perceive and adapt to each other during the training process, thereby solving the key problems in clinical questionnaire recognition: recognition of sloppy handwriting, understanding of professional terms, and analysis of mark-text association relationships. Through the training method of the present invention, the trained large language model gradually transitions from medical field knowledge adaptation to deep modal fusion, and ultimately achieves high-precision clinical questionnaire recognition and structuring.

[0060] This invention has been put into practical application in different scenarios in many hospitals, significantly improving the efficiency and accuracy of clinical data processing, shortening questionnaire processing time by about 80%, reducing data entry manpower requirements by about 70%, and reducing the error rate by more than 75%.

[0061] The present invention provides a device for training a large language model, such as Figure 2As shown, the device 200 includes: an acquisition unit 201, which acquires a training data set, wherein the training data set includes multimodal training data; a training unit 202, which uses the training data set to train the large language model, wherein during the training process, based on the correlation between the gradient of each modality and the gradient of other modalities, the collaborative adjustment gradient of each modality is determined, which is used to adjust the model parameters for each modality in the large language model in real time, thereby obtaining a trained large language model, wherein the trained large language model is used to recognize medical-related questionnaires.

[0062] It is understandable that the acquisition unit 201 and the training unit 202 can be Figure 3 The processor 1404 in the electronic device 1400 having the functions of these modules or units is implemented.

[0063] The present invention also provides a computer-readable storage medium, which stores instructions that, when executed on a computer, cause the computer to execute Figure 1 The method shown in .

[0064] The present invention also provides a computer program product comprising computer executable instructions, which are executed by the processor 1404 to implement Figure 1 The method shown in .

[0065] Now refer to Figure 3 , Figure 3 An example electronic device 1400 according to an embodiment of the present invention is schematically illustrated. In one embodiment, the electronic device 1400 may include one or more processors 1404, a system control logic unit 1408 coupled to at least one of the processors 1404, a system memory 1412 coupled to the system control logic unit 1408, a non-volatile memory (NVM) 1416 coupled to the system control logic unit 1408, and a network interface 1420 coupled to the system control logic unit 1408.

[0066] In some embodiments, the processor 1404 may include one or more single-core or multi-core processors. In some embodiments, the processor 1404 may include any combination of general-purpose processors and specialized processors (e.g., graphics processors, application processors, baseband processors, etc.). In embodiments where the electronic device 1400 employs an eNB (Evolved Node B) or a RAN (Radio Access Network) controller, the processor 1404 may be configured to execute various embodiments, such as Figure 1 The embodiment shown.

[0067] In some embodiments, system control logic unit 1408 may include any suitable interface controller to provide any suitable interface to at least one of processors 1404 and / or any suitable device or component in communication with system control logic unit 1408 .

[0068] In some embodiments, the system control logic unit 1408 may include one or more memory controllers to provide an interface to the system memory 1412. The system memory 1412 may be used to load and store data and / or instructions. In some embodiments, the system memory 1412 of the electronic device 1400 may include any suitable volatile memory, such as a suitable dynamic random access memory (DRAM).

[0069] The non-volatile memory 1416 may include one or more tangible, non-transitory computer-readable media for storing data and / or instructions. In some embodiments, the non-volatile memory 1416 may include any suitable non-volatile memory such as flash memory and / or any suitable non-volatile storage device, such as at least one of an HDD (Hard Disk Drive), a CD (Compact Disc) drive, and a DVD (Digital Versatile Disc) drive.

[0070] The non-volatile memory 1416 may include a portion of storage resources on the device on which the electronic device 1400 is installed, or it may be accessible to the electronic device but not necessarily a portion of the electronic device. For example, the non-volatile memory 1416 may be accessed via the network interface 1420 over a network.

[0071] In particular, the system memory 1412 and the non-volatile memory 1416 may each include a temporary copy and a permanent copy of the instructions 1424. The instructions 1424 may include instructions that, when executed by at least one of the processors 1404, cause the electronic device 1400 to perform the following operations: Figure 1 In some embodiments, instructions 1424 , hardware, firmware, and / or software components thereof may additionally or alternatively reside in system control logic 1408 , network interface 1420 , and / or processor 1404 .

[0072] The network interface 1420 may include a transceiver for providing a radio interface for the electronic device 1400, thereby communicating with any other suitable devices (such as a front-end module, an antenna, etc.) via one or more networks. In some embodiments, the network interface 1420 may be integrated with other components of the electronic device 1400. For example, the network interface 1420 may be integrated with at least one of the processor 1404, the system memory 1412, the non-volatile memory 1416, and a firmware device (not shown) having instructions. When at least one of the processors 1404 executes the instructions, the electronic device 1400 implements the following. Figure 1 The method shown.

[0073] The network interface 1420 may further include any suitable hardware and / or firmware to provide a multiple-input multiple-output radio interface. For example, the network interface 1420 may be a network adapter, a wireless network adapter, a telephone modem, and / or a wireless modem.

[0074] In one embodiment, at least one of the processors 1404 may be packaged together with logic for one or more controllers of the system control logic unit 1408 to form a system-in-package (SiP). In one embodiment, at least one of the processors 1404 may be integrated on the same die with logic for one or more controllers of the system control logic unit 1408 to form a system-on-chip (SoC).

[0075] Electronic device 1400 may further include input / output (I / O) devices 1432. I / O devices 1432 may include a user interface to enable a user to interact with electronic device 1400; peripheral component interfaces may also be designed to enable peripheral components to interact with electronic device 1400. In some embodiments, electronic device 1400 may also include a sensor for determining at least one of environmental conditions and location information related to electronic device 1400.

[0076] In some embodiments, the user interface may include, but is not limited to, a display (e.g., an LCD display, a touch screen display, etc.), a speaker, a microphone, one or more cameras (e.g., a still image camera and / or a video camera), a flashlight (e.g., an LED flash), and a keyboard.

[0077] The various embodiments of the mechanisms disclosed in this application can be implemented in hardware, software, firmware, or a combination of these implementation methods. The embodiments of this application can be implemented as a computer program or program code executed on a programmable system, which includes at least one processor, a storage system (including volatile and non-volatile memory and / or storage elements), at least one input device, and at least one output device.

[0078] Program code can be applied to input instructions to perform each function described in this application and generate output information. Output information can be applied to one or more output devices in a known manner. For the purposes of this application, a processing system includes any system having a processor such as, for example, a digital signal processor (DSP), a microcontroller, an application specific integrated circuit (ASIC), or a microprocessor.

[0079] Program code can be implemented with a high-level programming language or an object-oriented programming language to communicate with the processing system. Where necessary, program code can also be implemented in assembly language or machine language. In fact, the mechanism described in this application is not limited to the scope of any particular programming language. In either case, the language can be a compiled language or an interpreted language.

[0080] In some cases, the disclosed embodiments may be implemented in hardware, firmware, software, or any combination thereof. The disclosed embodiments may also be implemented as instructions carried by or stored on one or more transitory or non-transitory machine-readable (e.g., computer-readable) storage media, which can be read and executed by one or more processors. For example, the instructions may be distributed over a network or via other computer-readable media. Thus, a machine-readable medium may include any mechanism for storing or transmitting information in a form readable by a machine (e.g., a computer), including but not limited to floppy disks, compact disks, optical discs, read-only memory (CD-ROMs), magneto-optical disks, read-only memory (ROMs), random access memory (RAMs), erasable programmable read-only memory (EPROMs), electrically erasable programmable read-only memory (EEPROMs), magnetic or optical cards, flash memory, or tangible machine-readable storage for transmitting information via the Internet using electrical, optical, acoustic, or other propagated signals (e.g., carrier waves, infrared signals, digital signals, etc.). Thus, a machine-readable medium includes any type of machine-readable medium suitable for storing or transmitting electronic instructions or information in a form readable by a machine (e.g., a computer).

[0081] In the accompanying drawings, some structural or method features may be shown in a particular arrangement and / or order. However, it should be understood that such a particular arrangement and / or order may not be required. Rather, in some embodiments, these features may be arranged in a manner and / or order different from that shown in the illustrative drawings. In addition, the inclusion of a structural or method feature in a particular figure does not imply that such feature is required in all embodiments, and in some embodiments, such features may not be included or may be combined with other features.

[0082] It should be noted that the units / modules mentioned in the various device embodiments of the present application are all logical units / modules. Physically, a logical unit / module can be a physical unit / module, or a part of a physical unit / module, or can be implemented as a combination of multiple physical units / modules. The physical implementation of these logical units / modules themselves is not the most important. The combination of functions implemented by these logical units / modules is the key to solving the technical problems raised by this application. In addition, in order to highlight the innovative part of this application, the above-mentioned device embodiments of this application do not introduce units / modules that are not closely related to solving the technical problems raised by this application. This does not mean that other units / modules do not exist in the above-mentioned device embodiments.

[0083] It should be noted that in the examples and description of the present application, relational terms such as first and second, etc. are merely used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply the existence of any such actual relationship or order between these entities or operations. Moreover, the terms "comprise", "include" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device comprising a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, an element defined by the statement "comprises a" does not exclude the presence of other identical elements in the process, method, article or device comprising the element.

[0084] Although the present application has been shown and described with reference to certain preferred embodiments thereof, it will be understood by those skilled in the art that various changes in form and details may be made therein without departing from the spirit and scope of the application.

Claims

1. A method for training a large language model for an electronic device, characterized in that: The method comprises: An acquisition step of acquiring a training data set, wherein the training data set includes multimodal training data; A training step, using the training data set to train the large language model, wherein during the training process, based on the correlation between the gradient of each modality and the gradient of other modalities, a collaborative adjustment gradient of each modality is determined, which is used to adjust the model parameters for each modality in the large language model in real time, thereby obtaining a trained large language model, The trained large language model is used to identify medical-related questionnaires. The collaborative adjustment gradient of each modality is determined based on the following formula (1) and formula (2): Formula (1): ； Formula (2): ； in, represents the model parameters of the i-th mode, θj represents the model parameters of the j-th mode, represents the gradient of the i-th mode, represents the gradient of the jth mode, is the synergy intensity adjustment coefficient, is the gradient coordination function, represents the inner product of two gradients, represents the norm of the gradient of the i-th mode, represents the norm of the gradient of the jth mode, is the adjustment coefficient, is the sigmoid function, is the collaborative adjustment gradient of the i-th mode.

2. The method according to claim 1, characterized in that The multimodal training data includes visual training data, language training data, and structural training data.

3. The method according to claim 1, characterized in that Use the gradient cooperation function to determine the correlation between the gradient of each modality and the gradient of other modalities, Wherein, the gradient coordination function adopts an adaptive weighting strategy.

4. A device for training a large language model, characterized in that: The device comprises: An acquisition unit, which acquires a training data set, wherein the training data set includes multimodal training data; A training unit trains the large language model using the training data set, wherein during the training process, based on the correlation between the gradient of each modality and the gradients of other modalities, a collaborative adjustment gradient of each modality is determined, and used to adjust the model parameters for each modality in the large language model in real time, thereby obtaining a trained large language model. The trained large language model is used to identify medical-related questionnaires. The collaborative adjustment gradient of each modality is determined based on the following formula (1) and formula (2): Formula (1): ； Formula (2): ； in, represents the model parameters of the i-th mode, θj represents the model parameters of the j-th mode, represents the gradient of the i-th mode, represents the gradient of the jth mode, is the synergy intensity adjustment coefficient, is the gradient coordination function, represents the inner product of two gradients, represents the norm of the gradient of the i-th mode, represents the norm of the gradient of the jth mode, is the adjustment coefficient, is the sigmoid function, is the collaborative adjustment gradient of the i-th mode.

5. A computer-readable storage medium, characterized in that The storage medium stores instructions, which, when executed on a computer, cause the computer to execute the method for training a large language model according to claim 1.

6. An electronic device, characterized in that: include: one or more processors; One or more memories; the one or more memories store one or more programs, and when the one or more programs are executed by the one or more processors, the electronic device executes the method for training a large language model according to claim 1.

7. A computer program product comprising computer executable instructions, characterized in that: The instructions are executed by a processor to implement the method for training a large language model according to claim 1.

Images