Method, device and equipment for training large language model based on thinking chain

Abstract

The embodiment of the invention discloses a method, device and equipment for training a large language model based on a thinking chain. In the embodiment of the invention, multiple pieces of initial sampling data are acquired, and the initial sampling data comprise an image, auxiliary text information of the image and a standard auditing result of the image; generating thinking chain data according to each piece of initially adopted data, and determining a thinking chain data set; performing full fine adjustment on the basic large language model according to the thinking chain data set, and determining an intermediate large language model; iteratively generating multiple pieces of intermediate thinking chain data according to an intermediate large language model and the multiple pieces of initial sampling data; determining a reward value of each piece of intermediate thinking chain data according to a preset reward function; and finally, carrying out reinforcement learning on the middle large language model by adopting a group relative strategy optimization algorithm GRPO, and determining a target large language model. Through the method, the interpretability and the auditing precision of the large language model can be improved.

Description

Technical Field

[0001] This invention relates to the field of computer technology, and more specifically, to a method, apparatus, and device for training large-scale language models based on thought chains. Background Technology

[0002] With the rapid development of artificial intelligence technology, large language models (LLMs) have been widely used in complex business scenarios such as security review and content moderation, playing an important role in improving business processing efficiency.

[0003] In existing technologies, common large language models directly output review results when processing review tasks. When processing complex text and image information, their accuracy and reliability are poor, resulting in low review precision. Furthermore, since the review results are directly output, the basis and reasoning of the large language models in making judgments cannot be explained, resulting in low interpretability. This causes difficulties in optimizing review strategies and tracing risks, and the cost of manual review is also high.

[0004] In conclusion, improving the interpretability and auditing accuracy of large language models is a problem that needs to be solved. Summary of the Invention

[0005] In view of this, embodiments of the present invention provide a method, apparatus, and device for training large language models based on thought chains, which can improve the interpretability and auditing accuracy of large language models.

[0006] In a first aspect, embodiments of the present invention provide a method for training a large language model based on thought chains. The method includes: acquiring multiple initial sampling data, wherein the initial sampling data includes an image, auxiliary text information of the image, and a standard review result of the image; generating thought chain data based on each of the initial sampling data and determining a thought chain data set, wherein the thought chain data set includes multiple thought chain data, the thought chain data including logical reasoning and review results, the logical reasoning including image description and reasoning process; performing full fine-tuning on a basic large language model based on the thought chain data set to determine an intermediate large language model; iteratively generating multiple intermediate thought chain data, wherein the intermediate thought chain data is generated by inputting the initial sampling data into the intermediate large language model; determining a reward value for each intermediate thought chain data based on the multiple intermediate thought chain data and a pre-set reward function; and performing reinforcement learning on the intermediate large language model using the Group Relative Policy Optimization (GRPO) algorithm based on the multiple intermediate thought chain data and the corresponding reward value to determine a target large language model.

[0007] Optionally, acquiring multiple initial sampling data specifically includes: acquiring an initial sampling data set, wherein the initial sampling data set includes multiple initial sampling data, the initial sampling data is divided into white sample sampling data and black sample sampling data, the black sample sampling data is divided into multiple risk categories; and extracting multiple initial sampling data from the initial sampling data set according to a pre-set ratio of black and white samples and a risk category ratio.

[0008] Optionally, generating thought chain data based on each of the initial sampled data specifically includes: inputting the initial sampled data into a visual language model to generate thought chain data.

[0009] Optionally, the step of inputting the initial sampling data into the visual language model to generate thought chain data specifically includes: inputting the initial sampling data into the visual language model to generate candidate thought chain data, wherein the candidate thought chain data includes logical reasoning and review results; and inputting the candidate thought chain data, the image, and the auxiliary text information of the image back into the visual language model to generate the thought chain data.

[0010] Optionally, generating thought chain data based on each of the initial sampled data specifically includes: inputting the image in the initial sampled data into a visual language model to generate an image description; inputting the image description and auxiliary text information in the initial sampled data into a large-scale language reasoning model to generate a reasoning process and an audit result; determining the thought chain data based on the image description, the reasoning process, and the audit result, wherein the image description and the reasoning process determine the logical reasoning in the thought chain data.

[0011] Optionally, determining the thought chain data based on the image description, the reasoning process, and the review result specifically includes: determining candidate thought chain data based on the image description, the reasoning process, and the review result; and inputting the candidate thought chain data, the image, and the auxiliary text information of the image back into the visual language model to generate the thought chain data.

[0012] Optionally, the step of performing full fine-tuning of the basic large-scale language model based on the thought chain data set to determine the intermediate large-scale language model specifically includes: performing full fine-tuning of the basic large-scale language model based on the thought chain data set and the direct question-and-answer data set to determine the intermediate large-scale language model, wherein the direct question-and-answer data set includes multiple direct question data, and each direct question data includes a question and an answer.

[0013] Optionally, determining the reward value of each intermediate thought chain data based on the plurality of intermediate thought chain data and a pre-set reward function specifically includes: inputting each initial sampled data into an intermediate large language model, outputting intermediate thought chain data in a set format, wherein the intermediate thought chain data includes logical reasoning in a set format and review results in a set format; scoring the intermediate thought chain data according to pre-set rules, and determining the score as the reward value corresponding to the reward function.

[0014] Optionally, the method further includes: obtaining current image and text information; inputting the current image and text information into the target large-scale language model to generate a target answer, wherein the target answer includes target logical reasoning and target review results.

[0015] Secondly, embodiments of the present invention provide an apparatus for training a large-scale language model based on thought chains. The apparatus includes: an acquisition unit for acquiring multiple initial sampling data, wherein the initial sampling data includes an image, auxiliary text information of the image, and a standard review result of the image; a generation unit for generating thought chain data based on each of the initial sampling data and determining a thought chain data set, wherein the thought chain data set includes multiple thought chain data, the thought chain data including image description, logical reasoning, and review result, and the logical reasoning including image description and reasoning process; and an adjustment unit for adjusting a basic large-scale language model based on the thought chain data set. A full fine-tuning process is performed to determine an intermediate large language model. The generation unit is further configured to iteratively generate multiple intermediate thought chain data based on the intermediate large language model and multiple initial sampling data, wherein the intermediate thought chain data is generated by inputting the initial sampling data into the intermediate large language model. The determination unit is configured to determine the reward value of each intermediate thought chain data based on the multiple intermediate thought chain data and a pre-set reward function. The update unit is configured to perform reinforcement learning on the intermediate large language model using the Group Relative Policy Optimization (GRPO) algorithm based on the multiple intermediate thought chain data and the corresponding reward values ​​to determine the target large language model.

[0016] Optionally, the acquisition unit is specifically used for: acquiring an initial sampling data set, wherein the initial sampling data set includes multiple initial sampling data, the initial sampling data is divided into white sample sampling data and black sample sampling data, and the black sample sampling data is divided into multiple risk categories; and extracting multiple initial sampling data from the initial sampling data set according to a preset ratio of black and white samples and a risk category ratio.

[0017] Optionally, the generation unit is specifically used to: input the initial sampled data into the visual language model to generate thought chain data.

[0018] Optionally, the generation unit is further configured to: input the initial sampling data into a visual language model to generate candidate thought chain data, wherein the candidate thought chain data includes logical reasoning and review results; and input the candidate thought chain data, the image, and the auxiliary text information of the image back into the visual language model to generate the thought chain data.

[0019] Optionally, the generation unit is specifically used to: input the image in the initial sampling data into a visual language model to generate an image description; input the image description and auxiliary text information in the initial sampling data into a large-scale language reasoning model to generate a reasoning process and an audit result; and determine the thought chain data based on the image description, the reasoning process, and the audit result, wherein the image description and the reasoning process determine the logical reasoning in the thought chain data.

[0020] Optionally, the generation unit is further configured to: determine candidate thought chain data based on the image description, the reasoning process, and the review result; and input the candidate thought chain data, the image, and the auxiliary text information of the image back into the visual language model to generate the thought chain data.

[0021] Optionally, the adjustment unit is specifically used to: perform full fine-tuning on the basic large-scale language model based on the thought chain data set and the direct question-and-answer data set to determine the intermediate large-scale language model, wherein the direct question-and-answer data set includes multiple direct question data, and each direct question data includes a question and an answer.

[0022] Optionally, the determining unit is specifically used to: input each of the initial sampled data into an intermediate large language model, output intermediate thought chain data in a set format, wherein the intermediate thought chain data includes logical reasoning in a set format and review results in a set format; score the intermediate thought chain data according to a pre-set rule, and determine the score as the reward value corresponding to the reward function.

[0023] Optionally, the acquisition unit is further configured to: acquire current image and text information; the generation unit is further configured to: input the current image and text information into the target large-scale language model to generate a target answer, wherein the target answer includes target logical reasoning and target review results.

[0024] Thirdly, embodiments of the present invention provide an electronic device, including a memory and a processor, wherein the memory is used to store one or more computer program instructions, wherein the one or more computer program instructions are executed by the processor to implement the method as described in the first aspect or any one of the possible methods of the first aspect.

[0025] Fourthly, embodiments of the present invention provide a computer-readable storage medium having stored thereon computer program instructions that, when executed by a processor, implement the method as described in the first aspect or any one of the possibilities of the first aspect.

[0026] In this embodiment of the invention, multiple initial sampling data are acquired, including an image, auxiliary text information of the image, and standard review results of the image. Mind chain data is generated based on each initial sampling data, and a mind chain data set is determined. The mind chain data set includes multiple mind chain data, each comprising image description, logical reasoning, and review results. A basic large-scale language model is fully fine-tuned based on the mind chain data set to determine an intermediate large-scale language model. Multiple intermediate mind chain data are iteratively generated based on the intermediate large-scale language model and the multiple initial sampling data, wherein the intermediate mind chain data is generated by inputting the initial sampling data into the intermediate large-scale language model. A reward value for each intermediate mind chain data is determined based on the multiple intermediate mind chain data and a pre-set reward function. Based on the multiple intermediate mind chain data and the corresponding reward values, the Group Relative Policy Optimization (GRPO) algorithm is used to perform reinforcement learning on the intermediate large-scale language model to determine a target large-scale language model. Through the above method, fine-tuning the basic large-scale language model using a mind chain data set and performing reinforcement learning using GRPO can improve the interpretability and review accuracy of the large-scale language model. Attached Figure Description

[0027] The above and other objects, features and advantages of the present invention will become clearer from the following description of embodiments of the invention with reference to the accompanying drawings, in which: Figure 1 This is a flowchart of a method for training a large language model based on a thought chain in an embodiment of the present invention; Figure 2 This is a flowchart of another method for training a large language model based on a thought chain in an embodiment of the present invention; Figure 3 This is a flowchart of another method for training a large language model based on a thought chain in an embodiment of the present invention; Figure 4 This is a flowchart of another method for training a large language model based on a thought chain in an embodiment of the present invention; Figure 5 This is a schematic diagram of a device for training a large language model based on a thought chain in an embodiment of the present invention; Figure 6 This is a schematic diagram of an electronic device according to an embodiment of the present invention. Detailed Implementation

[0028] The present application is described below based on embodiments, but it is not limited to these embodiments. In the detailed description of the present application below, certain specific details are described in detail. Those skilled in the art can fully understand the present application without these details. To avoid obscuring the substance of the present application, well-known methods, processes, flows, elements, and circuits are not described in detail.

[0029] Furthermore, those skilled in the art should understand that the accompanying drawings provided herein are for illustrative purposes only and are not necessarily drawn to scale.

[0030] Unless the context explicitly requires it, words such as "including" or "contains" throughout the application should be interpreted as including rather than exclusive or exhaustive; that is, meaning "including but not limited to".

[0031] In the description of this application, it should be understood that the terms "first," "second," etc., are used for descriptive purposes only and should not be construed as indicating or implying relative importance. Furthermore, in the description of this application, unless otherwise stated, "a plurality of" means two or more.

[0032] In existing technologies, common Large Language Models (LLMs) directly output review results when processing review tasks. For example, inputting a product image and / or its corresponding text information into a LLM will output whether the product image and / or its corresponding text information are compliant or non-compliant. If the product image and / or its corresponding text information are relatively simple, the LLM can generate a relatively accurate review result. However, if the product image and / or its corresponding text information are complex, the accuracy and reliability of the review result generated by the LLM are poor. Because the review result is directly output, the basis and reasoning of the LLM's judgment cannot be explained, resulting in low interpretability. This causes difficulties in optimizing review strategies and tracing risks, and also increases the cost of manual review. Furthermore, in the application field of image and text content security review, the industry currently lacks vertical domain models specifically optimized for such tasks. When using general-purpose LLMs to perform review tasks, the perception and analysis capabilities in the review scenario are weak, resulting in low review accuracy. Therefore, how to improve the interpretability and review accuracy of LLMs is a problem that needs to be solved.

[0033] In this embodiment of the invention, the large language model may also be referred to as a large model or a large language model, etc. The large language model is a deep learning model based on a transformer architecture, which can process and generate natural language text. It is usually trained on a large amount of text data, has the ability to understand and generate language, and is widely used in dialogue systems, text generation and other natural language processing tasks.

[0034] In this embodiment of the invention, to address the aforementioned problems, a method for training large-scale language models based on thought chains is proposed, specifically as follows: Figure 1 As shown, the method includes: Step S101: Obtain multiple initial sampling data.

[0035] Specifically, the initial sampling data includes the image, the auxiliary text information of the image, and the standard review result of the image.

[0036] In one possible implementation, the image is a webpage screenshot, product image, or other image requiring security review. The auxiliary text information of the image is explanatory text or related text. The standard review result of the image can be compliant or non-compliant, qualified or unqualified, etc., determined according to the actual situation. The standard review result is also the factual result. For example, if the standard review result is non-compliant, then the standard review result also includes a risk label corresponding to the image. The risk label is multi-level. The risk label can be a classification of risk type, such as Category I risk, Category II risk, etc., where Category I risk or Category II risk can be prohibited, false advertising, etc., determined according to the actual situation; or it can be a classification of risk level, such as Level I risk, Level II risk, etc., which is only an example here.

[0037] In one possible implementation, acquiring multiple initial sampling data specifically includes: acquiring an initial sampling data set, wherein the initial sampling data set includes multiple initial sampling data, the initial sampling data is divided into white sample sampling data and black sample sampling data, the black sample sampling data is divided into multiple risk categories; and extracting multiple initial sampling data from the initial sampling data set according to a pre-set ratio of black and white samples and a risk category ratio.

[0038] For example, the initial sampling data set consists of data obtained during historical processing. Data that is compliant after review can be called white sample sampling data, and data that is non-compliant can be called black sample sampling data. Suppose that when the initial sampling data set is sampled at a ratio of 1 / 1.2 between black and white samples, and assuming that there are 5 risk types when sampling black samples, then each risk type is sampled according to the principle of balance. This is only an example. Sampling data in the above manner can improve the balance of subsequent training of large-scale language models.

[0039] Step S102: Generate thought chain data based on each of the initial adopted data, and determine the thought chain data set.

[0040] Specifically, the Chain-of-Thought (COT) dataset includes multiple Chain-of-Thought data sets, which include logical reasoning and review results. The logical reasoning includes image descriptions and reasoning processes. The image descriptions refer to text data that describes the content of images in the initial sampled data. The reasoning processes refer to the process of generating review results based on the images in the initial sampled data and the auxiliary text information of the images, i.e., a series of thought processes performed by a large language model before answering questions.

[0041] In one possible implementation, the generation of thought chain data based on each of the initial adopted data includes the following two methods, as detailed below: Method 1: End-to-end one-time generation.

[0042] Specifically, the initial sampled data is input into the Vision-Language Model (VLModel) to generate thought chain data.

[0043] In one possible implementation, the above method is an end-to-end generation method, in which the thought chain data is generated in one go, and the visual language model is a multimodal model that can understand and process image and text information.

[0044] In this embodiment of the invention, the initial sampling data is input into the visual language model to generate thought chain data, specifically as follows: Figure 2 As shown, it includes the following steps: Step 201: Input the initial sampling data into the visual language model to generate candidate thought chain data.

[0045] Specifically, the candidate thought chain data includes logical reasoning and review results.

[0046] Step 202: Input the candidate thought chain data, the image, and the auxiliary text information of the image back into the visual language model to generate the thought chain data.

[0047] Specifically, step 202 above involves rewriting and optimizing the logical reasoning to ensure that the final synthesized thought chain data is logically fluent, concisely expressed, and of high quality.

[0048] Method 2: Step-by-step generation.

[0049] In one possible implementation, the generation of thought chain data based on each of the initial adopted data is specifically as follows: Figure 3 As shown, it includes the following steps: Step 301: Input the image from the initial sampling data into the visual language model to generate an image description.

[0050] Step 302: Input the image description and auxiliary text information in the initial sampling data into the Large Reasoning Model (LRM) to generate the reasoning process and review results.

[0051] Specifically, the large-scale language reasoning model is a language model focused on logical reasoning.

[0052] Step 303: Determine the thought chain data based on the image description, the reasoning process, and the audit results.

[0053] Specifically, the image description and the reasoning process determine the logical reasoning in the thought chain data.

[0054] In one possible implementation, determining the thought chain data based on the image description, the reasoning process, and the review result specifically includes: determining candidate thought chain data based on the image description, the reasoning process, and the review result; inputting the candidate thought chain data, the image, and the auxiliary text information of the image back into the visual language model to generate the thought chain data. The above process also rewrites and optimizes the logical reasoning to ensure that the final synthesized thought chain data is logically fluent, concisely expressed, and of high quality.

[0055] In one possible implementation, the above-mentioned method one, method two, or both methods one and two can be used to generate the mind chain data, depending on the actual situation.

[0056] Step S103: Perform full fine-tuning of the basic large-scale language model based on the aforementioned thought chain data set to determine the intermediate large-scale language model.

[0057] In one possible implementation, the step of performing full fine-tuning of the basic large-scale language model based on the thought chain data set to determine the intermediate large-scale language model specifically includes: performing full fine-tuning of the basic large-scale language model based on the thought chain data set and the direct question-and-answer data set to determine the intermediate large-scale language model, wherein the direct question-and-answer data set includes multiple direct question data, and each direct question data includes a question and an answer.

[0058] By fully fine-tuning the basic large-scale language model using the aforementioned thought chain data set and direct question-and-answer data set, the performance of the basic large-scale language model can be comprehensively improved, generating relatively accurate answers for questions that require or do not require the display of the reasoning process.

[0059] Step S104: Based on the intermediate large language model and the multiple initial sampling data, iteratively generate multiple intermediate thought chain data.

[0060] Specifically, the intermediate thought chain data is generated by inputting the initial sampled data into the intermediate large language model.

[0061] In one possible implementation, each initial sampled data is input into an intermediate large language model, and intermediate thought chain data in a set format is output, wherein the intermediate thought chain data includes logical reasoning in a set format and review results in a set format; the intermediate thought chain data is scored according to a pre-set rule, and the score is determined as the reward value corresponding to the reward function.

[0062] In one possible implementation, the plurality of initial sampled data are unlabeled CoT data.

[0063] Step S105: Determine the reward value for each of the intermediate thought chain data based on the multiple intermediate thought chain data and the pre-set reward function.

[0064] In one possible implementation, the format for the output intermediate thought chain data can be set as follows: [Settings] <think> and< / think> The label, in the <think> and< / think> The tags are generated by a large intermediate language model, and after the logical reasoning, the output is "\judge{compliance}" or "\judge{violation},\cate{multi-level risk tags}". Here, "\judge{compliance}" is a compliance judgment made based on logical reasoning, and "\judge{violation},\cate{multi-level risk tags}" is a violation judgment made based on logical reasoning. The specific content of the "multi-level risk tags" is determined according to the actual situation. The above setting is only an example for illustration.

[0065] In one possible implementation, the intermediate thought chain data is scored according to pre-set rules, wherein the pre-set rules include a reward mechanism and a penalty item, which together constitute the reward function. The reward mechanism can be as follows: if the output is... <think> Starting with and containing tags< / think> At the end, each item will receive 0.01 points. Using the `\judge{}` tag will earn 0.01 points. If the content is either "compliant" or "non-compliant," an additional 0.1 points will be awarded. If only one of "compliant" or "non-compliant" appears, an additional 0.1 points will be awarded. When the conclusion within the `\judge{}` tag is completely consistent with the factual (ground_truth) result, the review result is correctly judged, earning a score of 0.8 points. Based on this, only when both the factual result and the model's judgment are "non-compliant" will the violation classification scoring stage begin: Using the `\cate{}` tag will award 0.05 points. If the content of the classification tag (i.e., multi-level risk tag) is consistent with the category given by the factual result, an additional 0.2 points will be awarded. Furthermore, if no other redundant `\cate{}` tags are included while generating the correct classification, an additional 0.3 points will be awarded. The above reward mechanism is only an example and the specific reward will be determined based on the actual situation.

[0066] In one possible implementation, the penalty is as follows: if the intermediate large language model judges it as "compliant" but incorrectly includes the \cate{} tag which should only be used in violation cases, 0.25 points will be deducted from the total score. The above penalty is only an example and the specific penalty will be determined according to the actual situation.

[0067] Step S106: Based on the multiple intermediate thought chain data and the corresponding reward values, the Group Relative Policy Optimization (GRPO) algorithm is used to perform reinforcement learning on the intermediate large language model to determine the target large language model.

[0068] Specifically, Group Relative Policy Optimization (GRPO) is a model optimization method and a variant of Proximal Policy Optimization.

[0069] In this embodiment of the invention, the advanced optimization of the intermediate large language model using GRPO reinforcement learning can further enhance the review capability and output format compliance of the target large language model.

[0070] In one possible implementation, after training the target large-scale language model, the aforementioned target large-scale language model can be used for image and text review, specifically as follows: Figure 4As shown, it includes the following steps: Step 401: Obtain the current image and text information.

[0071] Specifically, the current image and text information refers to the image and text information to be reviewed.

[0072] Step 402: Input the current image and text information into the target large-scale language model to generate the target answer.

[0073] The target answer includes target logical reasoning and target review results.

[0074] In one possible implementation, the target answer may include <think> 、 < / think> Tags such as \judge{}, \cate{}, etc., can also be used to... <think> 、 < / think> After filtering with tags such as \judge{}, \judge{}, and \cate{}, the specific content is displayed to the user.

[0075] Through the above embodiments, the basic large-scale language model is first fine-tuned using synthesized thought chain data, enabling it to initially grasp the analytical thinking ability of the review scenario. Then, advanced optimization is performed using the GRPO reinforcement learning algorithm to further improve its accuracy and professionalism in specific tasks, thereby constructing a vertical domain model deeply adapted to the security review of text and image content. Because the introduction of thought chains breaks the "black box" characteristic of traditional large-scale language models that only output review results, it can not only output the conclusion of "compliant" or "non-compliant," but also generate the detailed reasoning process leading to the judgment, improving interpretability. Furthermore, during the reinforcement learning phase, a carefully designed reward function forces the target large-scale language model to place the thinking process, final judgment, and violation classification separately. <think>Structured tags such as `\judge{}` and `\cate{}` allow reviewers to quickly understand the basis for their judgments, thereby reducing the difficulty and time cost of manual review. In summary, the above methods improve the accuracy, reliability, and interpretability of large language models in text and image content security review tasks, and reduce manual costs by enhancing the automated processing capabilities and interpretability of large language models.

[0076] In this embodiment of the invention, a device for training large-scale language models based on thought chains is provided, such as... Figure 5 As shown, it specifically includes: an acquisition unit 501, a generation unit 502, an adjustment unit 503, a determination unit 504, and an update unit 505; The acquisition unit 501 is used to acquire multiple initial sampling data, wherein the initial sampling data includes an image, auxiliary text information of the image, and standard review results of the image; the generation unit 502 is used to generate thought chain data based on each initial sampling data and determine a thought chain data set, wherein the thought chain data set includes multiple thought chain data, the thought chain data includes logical reasoning and review results, and the logical reasoning includes image description and reasoning process; the adjustment unit 503 is used to perform full fine-tuning on the basic large-scale language model based on the thought chain data set to determine an intermediate large-scale language model; The generation unit 502 is further configured to iteratively generate multiple intermediate thought chain data based on the intermediate large language model and multiple initial sampling data, wherein the intermediate thought chain data is generated by inputting the initial sampling data into the intermediate large language model; the determination unit 504 is configured to determine the reward value of each intermediate thought chain data based on the multiple intermediate thought chain data and a pre-set reward function; the update unit 505 is configured to perform reinforcement learning on the intermediate large language model using the Group Relative Policy Optimization (GRPO) algorithm based on the multiple intermediate thought chain data and the corresponding reward values ​​to determine the target large language model.

[0077] Further, the acquisition unit is specifically used for: acquiring an initial sampling data set, wherein the initial sampling data set includes multiple initial sampling data, the initial sampling data is divided into white sample sampling data and black sample sampling data, the black sample sampling data is divided into multiple risk categories; and extracting multiple initial sampling data from the initial sampling data set according to a preset ratio of black and white samples and a risk category ratio.

[0078] Furthermore, the generation unit is specifically used to: input the initial sampled data into the visual language model to generate thought chain data.

[0079] Furthermore, the generation unit is specifically used to: input the initial sampling data into the visual language model to generate candidate thought chain data, wherein the candidate thought chain data includes logical reasoning and review results; and input the candidate thought chain data, the image, and the auxiliary text information of the image back into the visual language model to generate the thought chain data.

[0080] Further, the generation unit is specifically used to: input the image in the initial sampling data into a visual language model to generate an image description; input the image description and auxiliary text information in the initial sampling data into a large-scale language reasoning model to generate a reasoning process and an audit result; and determine the thought chain data based on the image description, the reasoning process, and the audit result, wherein the image description and the reasoning process determine the logical reasoning in the thought chain data.

[0081] Furthermore, the generation unit is specifically used to: determine candidate thought chain data based on the image description, the reasoning process, and the review result; and input the candidate thought chain data, the image, and the auxiliary text information of the image back into the visual language model to generate the thought chain data.

[0082] Furthermore, the adjustment unit is specifically used to: perform full fine-tuning on the basic large-scale language model based on the thought chain data set and the direct question-and-answer data set to determine the intermediate large-scale language model, wherein the direct question-and-answer data set includes multiple direct question data, and each direct question data includes a question and an answer.

[0083] Furthermore, the determining unit is specifically used to: input each of the initial sampled data into the intermediate large language model, output intermediate thought chain data in a set format, wherein the intermediate thought chain data includes logical reasoning in a set format and review results in a set format; score the intermediate thought chain data according to a pre-set rule, and determine the score as the reward value corresponding to the reward function.

[0084] Furthermore, the acquisition unit is also used to: acquire current image and text information; the generation unit is also used to: input the current image and text information into the target large-scale language model to generate a target answer, wherein the target answer includes target logical reasoning and target review results.

[0085] Figure 6 This is a schematic diagram of the structure of the electronic device described in an embodiment of the present invention. Figure 6 As shown, it includes a general computer hardware architecture, which includes at least a processor 601 and a memory 602. The processor 601 and the memory 602 are connected via a bus 603. The memory 602 is adapted to store instructions or programs executable by the processor 601. The processor 601 can be a standalone microprocessor or a collection of one or more microprocessors. Thus, the processor 601 executes the instructions stored in the memory 602 to perform the method flow of the embodiments of the present invention as described above, thereby realizing data processing and control of other devices. The bus 603 connects the above-mentioned components together, and also connects the above-mentioned components to a display controller 604, a display device, and an input / output (I / O) device 605. The input / output (I / O) device 605 can be a mouse, keyboard, modem, network interface, touch input device, motion-sensing input device, printer, and other devices known in the art. Typically, the input / output device 605 is connected to the system via an input / output (I / O) controller 606.

[0086] The instructions stored in memory 602 are executed by at least one processor 601 to: acquire multiple initial sampled data; generate thought chain data based on each of the initial sampled data and determine a set of thought chain data; perform full fine-tuning on the basic large language model based on the set of thought chain data to determine an intermediate large language model; iteratively generate multiple intermediate thought chain data based on the intermediate large language model and the multiple initial sampled data; determine the reward value of each intermediate thought chain data based on the multiple intermediate thought chain data and a pre-set reward function; and perform reinforcement learning on the intermediate large language model using the Group Relative Policy Optimization (GRPO) algorithm based on the multiple intermediate thought chain data and the corresponding reward value to determine a target large language model.

[0087] Specifically, the electronic device includes: one or more processors 601 and a memory 602. Figure 6 Take a processor 601 as an example. The processor 601 and the memory 602 can be connected via a bus or other means. Figure 6 Taking a bus connection as an example, memory 602, as a non-volatile computer-readable storage medium, can be used to store non-volatile software programs, non-volatile computer-executable programs, and modules. Processor 601 executes various functional applications and data processing of the device by running the non-volatile software programs, instructions, and modules stored in memory 602, thereby implementing the aforementioned method for determining the training of a large language model based on thought chains.

[0088] Memory 602 may include a program storage area and a data storage area, wherein the program storage area may store the operating system and applications required for at least one function; the data storage area may store an option list, etc. Furthermore, memory 602 may include high-speed random access memory, and may also include non-volatile memory, such as at least one disk storage device, flash memory device, or other non-volatile solid-state storage device. In some embodiments, memory 602 may optionally include memory remotely located relative to processor 601, and these remote memories may be connected to external devices via a network. Examples of such networks include, but are not limited to, the Internet, corporate intranets, local area networks, mobile communication networks, and combinations thereof.

[0089] One or more modules are stored in memory 602, and when executed by one or more processors 601, they execute the method of training a large language model based on thought chain in any of the above method embodiments.

[0090] As those skilled in the art will recognize, various aspects of the embodiments of the present invention can be implemented as a system, method, or computer program product. Therefore, various aspects of the embodiments of the present invention can take the form of a completely hardware implementation, a completely software implementation (including firmware, resident software, microcode, etc.), or an implementation combining software and hardware aspects, which may generally be referred to herein as a "circuit," "module," or "system." Furthermore, various aspects of the embodiments of the present invention can take the form of a computer program product implemented in one or more computer-readable media having computer-readable program code implemented thereon.

[0091] Any combination of one or more computer-readable media can be used. A computer-readable medium can be a computer-readable signal medium or a computer-readable storage medium. A computer-readable storage medium can be, for example, (but not limited to) an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or apparatus, or any suitable combination thereof. More specific examples (not an exhaustive list) of computer-readable storage media will include: an electrical connection having one or more wires, a portable computer floppy disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable optical disc read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination thereof. In the context of embodiments of the present invention, a computer-readable storage medium can be any tangible medium capable of containing or storing a program used by or in conjunction with an instruction execution system, device, or apparatus.

[0092] Computer-readable signal media may include propagated digital signals having computer-readable program code implemented therein, such as in baseband or as part of a carrier wave. Such propagated signals may take any of a variety of forms, including, but not limited to, electromagnetic, optical, or any suitable combination thereof. A computer-readable signal medium may be any computer-readable medium that is not a computer-readable storage medium and can communicate, propagate, or transmit a program used by or in conjunction with an instruction execution system, device, or apparatus.

[0093] Program code implemented on a computer-readable medium may be transmitted using any suitable medium, including but not limited to wireless, wired, fiber optic cable, RF, or any suitable combination thereof.

[0094] Computer program code used to perform operations relating to various aspects of embodiments of the present invention can be written in any combination of one or more programming languages, including: object-oriented programming languages ​​such as Java, Smalltalk, C++, etc.; and conventional procedural programming languages ​​such as the "C" programming language or similar programming languages. The program code can be executed as a standalone software package entirely on the user's computer, partially on the user's computer, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the latter case, the remote computer can be connected to the user's computer via any type of network, including a local area network (LAN) or a wide area network (WAN), or it can be connected to an external computer (e.g., via the Internet provided by an Internet service provider).

[0095] The flowchart illustrations and / or block diagrams of the methods, apparatus (systems), and computer program products according to embodiments of the present invention describe various aspects of the embodiments of the present invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, or other programmable data processing apparatus to produce a machine such that the instructions (executed via the processor of the computer or other programmable data processing apparatus) create means for implementing the functions / actions specified in the flowchart and / or block diagram blocks or blocks.

[0096] These computer program instructions may also be stored in a computer-readable medium that can direct a computer, other programmable data processing apparatus or other means to operate in a particular manner, such that the instructions stored in the computer-readable medium produce an article of writing that includes instructions that implement the functions / actions specified in flowchart and / or block diagram blocks or blocks.

[0097] Computer program instructions may also be loaded onto a computer, other programmable data processing apparatus or other device to cause a series of operable steps to be performed on the computer, other programmable apparatus or other device to produce a computer-implemented process, such that the instructions, which execute on the computer or other programmable apparatus, provide for implementing the functions / actions specified in flowchart and / or block diagram blocks or blocks.

[0098] The above description is merely a preferred embodiment of this application and is not intended to limit this application. Various modifications and variations can be made to this application by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the protection scope of this application.

[0099] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, and displayed data) involved in this application are all information and data authorized by the user or fully authorized by all parties. Furthermore, the collection, use, and processing of such data must comply with the relevant laws, regulations, and standards of the relevant countries and regions, and corresponding access points are provided for users to choose to authorize or refuse processing. A user's refusal to process personal information beyond what is necessary for basic functions will not affect the user's use of basic functions.< / think>

Claims

1. A method for training large-scale language models based on thought chains, characterized in that, The method includes: Acquire multiple initial sampling data, wherein the initial sampling data includes an image, auxiliary text information of the image, and standard review results of the image; Based on each of the initial adopted data, generate thought chain data and determine a thought chain data set, wherein the thought chain data set includes multiple thought chain data, the thought chain data includes logical reasoning and review results, and the logical reasoning includes image description and reasoning process; Based on the aforementioned thought chain dataset, the basic large-scale language model is fully fine-tuned to determine the intermediate large-scale language model. Based on the intermediate large language model and multiple initial sampled data, multiple intermediate thought chain data are iteratively generated, wherein the intermediate thought chain data is generated by inputting the initial sampled data into the intermediate large language model; The reward value for each of the intermediate thought chain data is determined based on the multiple intermediate thought chain data and the pre-set reward function; Based on the multiple intermediate thought chain data and the corresponding reward values, the Group Relative Policy Optimization (GRPO) algorithm is used to perform reinforcement learning on the intermediate large language model to determine the target large language model.

2. The method according to claim 1, characterized in that, The acquisition of multiple initial sampling data specifically includes: Obtain an initial sampling data set, wherein the initial sampling data set includes multiple initial sampling data, the initial sampling data is divided into white sample sampling data and black sample sampling data, and the black sample sampling data is divided into multiple risk categories; Multiple initial sample data are extracted from the initial sample data set according to the pre-set ratio of black and white samples and the ratio of risk categories.

3. The method according to claim 1, characterized in that, The process of generating thought chain data based on each of the initial adopted data specifically includes: The initial sampled data is input into the visual language model to generate thought chain data.

4. The method according to claim 3, characterized in that, The step of inputting the initial sampled data into the visual language model to generate thought chain data specifically includes: The initial sampling data is input into a visual language model to generate candidate thought chain data, wherein the candidate thought chain data includes logical reasoning and review results; The candidate thought chain data, the image, and the auxiliary text information of the image are input again into the visual language model to generate the thought chain data.

5. The method according to claim 1, characterized in that, The process of generating thought chain data based on each of the initial adopted data specifically includes: The images in the initial sampled data are input into a visual language model to generate image descriptions; The image description and auxiliary text information in the initial sampling data are input into a large-scale language reasoning model to generate the reasoning process and review results. The thought chain data is determined based on the image description, the reasoning process, and the review result, wherein the image description and the reasoning process determine the logical reasoning in the thought chain data.

6. The method according to claim 5, characterized in that, The step of determining the thought chain data based on the image description, the reasoning process, and the review result specifically includes: Candidate thought chain data are determined based on the image description, the reasoning process, and the review results; The candidate thought chain data, the image, and the auxiliary text information of the image are input again into the visual language model to generate the thought chain data.

7. The method according to claim 1, characterized in that, The step of performing full fine-tuning of the basic large-scale language model based on the aforementioned thought chain data set to determine the intermediate large-scale language model specifically includes: Based on the aforementioned thought chain data set and direct question-and-answer data set, the basic large-scale language model is fully fine-tuned to determine the intermediate large-scale language model. The direct question-and-answer data set includes multiple direct question data sets, and each direct question data set includes a question and an answer.

8. The method according to claim 1, characterized in that, The step of determining the reward value for each of the intermediate thought chain data based on the multiple intermediate thought chain data and a pre-set reward function specifically includes: Each initial sampled data is input into an intermediate large language model, and intermediate thought chain data in a set format is output, wherein the intermediate thought chain data includes logical reasoning in a set format and review results in a set format. The intermediate thought chain data is scored according to pre-set rules, and the score is determined as the reward value corresponding to the reward function.

9. The method according to claim 1, characterized in that, The method further includes: Get the current image and text information; The current text and image information is input into the target large-scale language model to generate the target answer, wherein the target answer includes target logical reasoning and target review results.

10. A device for training large-scale language models based on thought chains, characterized in that, The device includes: An acquisition unit is used to acquire multiple initial sampling data, wherein the initial sampling data includes an image, auxiliary text information of the image, and standard review results of the image; A generation unit is configured to generate thought chain data based on each of the initial adopted data and determine a set of thought chain data, wherein the set of thought chain data includes multiple sets of thought chain data, and the thought chain data includes logical reasoning and review results, wherein the logical reasoning includes image description and reasoning process; The adjustment unit is used to perform full fine-tuning of the basic large-scale language model based on the thought chain data set to determine the intermediate large-scale language model. The generation unit is further configured to iteratively generate multiple intermediate thought chain data based on the intermediate large language model and multiple initial sampling data, wherein the intermediate thought chain data is generated by inputting the initial sampling data into the intermediate large language model; The determining unit is used to determine the reward value of each of the intermediate thinking chain data based on the plurality of intermediate thinking chain data and a pre-set reward function; The update unit is used to perform reinforcement learning on the intermediate large language model using the Group Relative Policy Optimization (GRPO) algorithm based on the multiple intermediate thought chain data and the corresponding reward values, and to determine the target large language model.

11. An electronic device comprising a memory and a processor, characterized in that, The memory is used to store one or more computer program instructions, wherein the one or more computer program instructions are executed by the processor to implement the method as described in any one of claims 1-9.

12. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, implements the method as described in any one of claims 1-9.

Images