Small sample-based large model optimization method and device, equipment and storage medium

Abstract

This application discloses a method, apparatus, device, and storage medium for optimizing a large model based on a small sample size. The method involves: acquiring initial multimodal data; determining the vertical domain corresponding to the initial multimodal data; optimizing a pre-acquired third-party large language model based on the vertical domain; extracting initial text data from the initial multimodal data and inputting it into the optimized third-party large language model to obtain enhanced text data; extracting initial image data from the initial multimodal data and performing data augmentation on the initial image data to obtain selected images; constructing a hybrid dataset from the initial multimodal data, enhanced text data, and selected images; and optimizing a pre-established base model using the hybrid dataset to obtain an optimized multimodal base model. This application, through the above processing steps, enables a wider range of data coverage and allows the model to adapt to multimodal service tasks.

Description

Technical Field

[0001] This application relates to the field of model optimization technology, specifically to a method, apparatus, device, and storage medium for large model optimization based on small samples. Background Technology

[0002] With the development of technology, digital services on the internet are emerging in an endless stream, such as ride-hailing and internet hospitals. Faced with increasingly complex data service tasks, it is becoming increasingly important to build a model or training framework that can quickly adapt to multimodal service tasks. Traditional machine learning methods often rely on large amounts of labeled data to improve the accuracy and generalization ability of the model. However, in some practical application scenarios, obtaining large-scale datasets is costly and difficult to achieve. Therefore, more and more attention is now shifting to small sample scenarios.

[0003] In scenarios with few samples, the model is prone to overfitting due to insufficient training samples, and cannot cover a wider range of scenarios and situations, making it difficult for the model to adapt to multimodal service tasks. Summary of the Invention

[0004] In view of this, this application provides a method, apparatus, device and storage medium for large model optimization based on small sample size, which is used to solve the problem that in small sample scenarios, due to insufficient training samples, the model is prone to overfitting, cannot cover a wider range of scenarios and situations, and thus the model is difficult to adapt to multimodal service tasks.

[0005] To achieve the above objectives, the following solution is proposed:

[0006] Firstly, a method for optimizing large models based on small samples includes:

[0007] Acquire initial multimodal data;

[0008] Determine the vertical domain corresponding to the initial multimodal data, and optimize the pre-acquired third-party large language model based on the vertical domain;

[0009] Extract the initial text data from the initial multimodal data, and input the initial text data into the optimized third-party large language model to obtain the output enhanced text data;

[0010] Extract the initial image data from the initial multimodal data, and perform data augmentation processing on the initial image data to obtain each selected image;

[0011] A hybrid dataset is constructed from the initial multimodal data, enhanced text data, and selected images;

[0012] The pre-established large-scale pedestal model is optimized using the hybrid dataset to obtain an optimized multimodal large-scale pedestal model.

[0013] Preferably, the optimization of the pre-acquired third-party large language model based on the vertical domain includes:

[0014] Create initial tag text that matches the vertical domain;

[0015] The initial label text is expanded to obtain specific descriptive information;

[0016] Based on the specific description information, construct the loss function;

[0017] The aforementioned loss function is used to optimize the soft prompts of a pre-acquired third-party large language model.

[0018] Preferably, the process by which the optimized third-party large language model processes the initial text data and outputs enhanced text data includes:

[0019] Establish the target problem based on the aforementioned vertical domain;

[0020] The target problem is broken down into its individual steps, resulting in a text document.

[0021] The logical reasoning process is constructed from the text of each step described;

[0022] Generate corresponding prompt text for each step of the text;

[0023] The prompt texts are added to the logical reasoning process according to their corresponding step texts to obtain the thought chain;

[0024] The input initial text data is reasoned according to the aforementioned thought chain to obtain enhanced text data, which is then output.

[0025] Preferably, the step of reasoning through the input initial text data according to the thought chain to obtain enhanced text data and outputting it includes:

[0026] Acquire domain-specific knowledge corresponding to the vertical domain;

[0027] Reasoning is performed on the initial text data according to the aforementioned thought process;

[0028] While reasoning, the output answer corresponding to each step of the text is extracted in real time;

[0029] For each step text, adjust the prompt text for the next step text based on its corresponding output answer;

[0030] While making adjustments, the domain-specific knowledge is added to the prompt text to infer enhanced text data corresponding to the initial text data.

[0031] Preferably, the step of processing and enhancing the initial image data to obtain selected images includes:

[0032] Identify the main object in the initial image data and determine the position, outline, and category of the main object;

[0033] The initial image data is segmented based on the location, contour, and category to obtain the first image data.

[0034] Determine the key tags and text attributes corresponding to the vertical domain, and generate a structured text description based on the key tags and text attributes;

[0035] The structured text description is processed using a pre-trained text-to-image model to obtain each task subject image corresponding to the structured text description;

[0036] Each initial image data point is used to select the main task images to obtain the selected images.

[0037] Preferably, the step of filtering the task subject images corresponding to each initial image data to obtain selected images includes:

[0038] Each of the task subject images is initially screened according to preset clarity threshold, integrity threshold, label matching threshold and background complexity threshold to obtain each coarsely selected image;

[0039] Each of the coarsely selected images and the initial image data are input into a pre-trained expert model, so that the expert model can accurately filter each coarsely selected image according to the initial image data, and output the filtered images as each selected image.

[0040] Preferably, the construction of the hybrid dataset from the initial multimodal data, enhanced text data, and selected images includes:

[0041] Obtain the prompt text and enhanced text data corresponding to the initial text data;

[0042] For each selected image, the prompt text and enhanced text data are associated with that selected image to generate individual image-text data pairs;

[0043] For each selected image, the text data pairs corresponding to each image are scored for authenticity in order to select the target data pairs.

[0044] A set of data pairs is constructed from each of the target data pairs, and the set of data pairs is balanced to obtain each balanced data pair.

[0045] A hybrid dataset is constructed from the initial multimodal data, each of the aforementioned balanced data pairs, and each selected image.

[0046] Secondly, a large model optimization device based on small samples includes:

[0047] The multimodal data acquisition module is used to acquire initial multimodal data;

[0048] The large language model optimization module is used to determine the vertical domain corresponding to the initial multimodal data, and optimize the pre-acquired third-party large language model based on the vertical domain.

[0049] The model processing module is used to extract the initial text data from the initial multimodal data, input the initial text data into the optimized third-party large language model, and obtain the output enhanced text data.

[0050] The data augmentation processing module is used to extract the initial image data from the initial multimodal data and perform data augmentation processing on the initial image data to obtain each selected image;

[0051] A hybrid dataset construction module is used to construct a hybrid dataset from the initial multimodal data, enhanced text data, and selected images;

[0052] The base large model optimization module is used to optimize the pre-established base large model using the hybrid dataset to obtain the optimized multimodal base large model.

[0053] Thirdly, a large model optimization device based on small samples, including memory and processor;

[0054] The memory is used to store programs;

[0055] The processor is configured to execute the program to implement the steps of the small-sample-based large-model optimization method as described in any of the first aspects.

[0056] Fourthly, a storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the small-sample-based large-model optimization method as described in any of the first aspects.

[0057] As can be seen from the above technical solution, this application acquires initial multimodal data; determines the vertical domain corresponding to the initial multimodal data, and optimizes the pre-acquired third-party large language model based on the vertical domain; extracts initial text data from the initial multimodal data, inputs the initial text data into the optimized third-party large language model, and obtains output enhanced text data; extracts initial image data from the initial multimodal data, and performs data augmentation processing on the initial image data to obtain selected images; constructs a hybrid dataset from the initial multimodal data, enhanced text data, and selected images; and optimizes the pre-established base model using the hybrid dataset to obtain the optimized multimodal base model. This application categorizes the initial multimodal data and processes the categorized data separately. It can optimize the third-party large language model based on the vertical domain of the initial multimodal data, making the third-party large language model more targeted. The optimized third-party large language model can then better process the initial text data in the multimodal data to obtain enhanced text data. Furthermore, it performs data augmentation processing on the initial image data in the multimodal data to obtain selected images. Through the above processing, the coverage and scope of the data can be broadened, enabling the model to adapt to multimodal service tasks. Attached Figure Description

[0058] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on the provided drawings without creative effort.

[0059] Figure 1 An optional flowchart of a large model optimization method based on small samples provided for embodiments of this application;

[0060] Figure 2 An optional structural block diagram of a large model optimization method based on small samples provided in an embodiment of this application;

[0061] Figure 3 A schematic diagram of a large model optimization device based on small samples provided in this application embodiment;

[0062] Figure 4 This is a schematic diagram of the structure of a large model optimization device based on small samples, provided in an embodiment of this application. Detailed Implementation

[0063] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0064] With the development of technology, digital services on the internet are emerging in an endless stream, such as ride-hailing and internet hospitals. Faced with increasingly complex data service tasks, building a model or training framework that can quickly adapt to multimodal service tasks is becoming increasingly important. Traditional machine learning methods often rely on large amounts of labeled data to improve the model's accuracy and generalization ability. However, in some practical applications, obtaining large-scale datasets is costly and difficult to achieve. Therefore, more and more attention is now shifting to small-sample scenarios. When faced with a small amount of manually labeled data, efficient training to ensure the model's predictive performance in such scenarios is of great significance. Traditional machine learning methods often rely on large amounts of labeled data to improve the model's accuracy and generalization ability. However, in some practical applications, obtaining large-scale datasets is costly and difficult to achieve. The significance of small-sample training lies in its ability to overcome data volume limitations. Through ingenious model design and innovative algorithms, it can extract effective information from a limited number of samples, improving the model's predictive ability and adaptability. Especially in the field of digital services, many tasks in this field often involve privacy and security, and labeled data is usually scarce and expensive. The method of constructing small samples for training not only helps to improve the model's ability to learn from scarce data, but also promotes the application of data services in small sample scenarios, and drives the popularization and implementation of intelligent technologies in more industries. Exploring and developing small sample training technology has broad application prospects and profound industry significance.

[0065] However, in small sample scenarios, due to insufficient training samples, the model is prone to overfitting and cannot cover a wider range of scenarios and situations, making it difficult for the model to adapt to multimodal service tasks.

[0066] To address the shortcomings of the prior art, this invention provides a large model optimization method based on small samples. This method can be applied to various computer terminals or smart terminals, and its execution entity can be the processor or server of the computer terminal or smart terminal. The flowchart of the method is shown below. Figure 1 As shown, it specifically includes:

[0067] S1: Obtain initial multimodal data.

[0068] The initial multimodal data obtained in this application contains data of various types or structures, such as text data, form data, image data, video frame data, etc., thus covering a wider range.

[0069] After obtaining the initial multimodal data, in order to make the subsequent process more efficient, the multimodal data is preprocessed, including the removal of redundant data. This ensures the alignment of different modal data in the temporal or spatial dimensions. For example, the narration or descriptive text corresponding to video frame data at different times should be timestamped to make the multimodal data more regular.

[0070] The initial multimodal data obtained in this application is a small amount of manually labeled data based on vertical domain tasks.

[0071] S2: Determine the vertical domain corresponding to the initial multimodal data, and optimize the pre-acquired third-party large language model based on the vertical domain.

[0072] In this step, a third-party large language model is introduced, and the third-party large language model is optimized according to the vertical domain corresponding to the initial multimodal data. In this way, the contextual understanding and causal reasoning capabilities of the third-party large language model are used to process the initial multimodal data.

[0073] S3: Extract the initial text data from the initial multimodal data, input the initial text data into the optimized third-party large language model, and obtain the output enhanced text data.

[0074] Since the initial multimodal data contains various types of data, the different types of data are separated and processed separately. The text data and form data in the initial multimodal data are collectively referred to as the initial text data. By inputting them into the optimized third-party large language model, high-quality enhanced text can be generated based on a small amount of labeled initial text data.

[0075] Third-party large language models can generate various instruction texts, including question-and-answer and reasoning, for specific tasks.

[0076] S4: Extract the initial image data from the initial multimodal data, and perform data augmentation processing on the initial image data to obtain each selected image.

[0077] Since the initial multimodal data also contains image data and video frame data, these are extracted and collectively referred to as the initial image data. Then, data augmentation processing is performed on the initial image data to obtain each selected image.

[0078] In this step, a third-party large language model for image and video frame types can also be called. This third-party large language model can be used to alleviate the problem of scarce samples in some categories by generating graphs from text, and to balance data distribution and data bias. This third-party large language model refers to a third-party diffusion model.

[0079] S5: Construct a hybrid dataset from the initial multimodal data, enhanced text data, and each selected image.

[0080] During the construction process, multi-indicator data evaluation, screening, and regeneration can be performed. For example, multimodal data containing duplicate content or irrelevant information can be deleted to ensure the quality and relevance of the dataset. At the same time, the initial multimodal data, enhanced text data, and selected images can be screened based on diversity indicators (such as the range of sample variation and distribution characteristics) and authenticity indicators (such as the generation quality of data and the degree of consistency with the actual task). Low-quality samples or samples that do not meet the task requirements can be removed, thereby constructing a high-quality multimodal dataset that is both rich in diversity and highly reliable.

[0081] S6: Optimize the pre-established large-scale base model using the hybrid dataset to obtain the optimized multimodal large-scale base model.

[0082] Based on the established hybrid dataset, the large-scale data service platform model to be optimized can be fine-tuned using instructions, enabling it to quickly align with different data service task scenarios and generate standardized output results, such as... Figure 2 As shown.

[0083] As can be seen from the above technical solution, this application acquires initial multimodal data; determines the vertical domain corresponding to the initial multimodal data, and optimizes the pre-acquired third-party large language model based on the vertical domain; extracts initial text data from the initial multimodal data, inputs the initial text data into the optimized third-party large language model, and obtains output enhanced text data; extracts initial image data from the initial multimodal data, and performs data augmentation processing on the initial image data to obtain selected images; constructs a hybrid dataset from the initial multimodal data, enhanced text data, and selected images; and optimizes the pre-established base model using the hybrid dataset to obtain the optimized multimodal base model. This application categorizes the initial multimodal data and processes the categorized data separately. It can optimize the third-party large language model based on the vertical domain of the initial multimodal data, making the third-party large language model more targeted. The optimized third-party large language model can then better process the initial text data in the multimodal data to obtain enhanced text data. Furthermore, it performs data augmentation processing on the initial image data in the multimodal data to obtain selected images. Through the above processing, the coverage and scope of the data can be broadened, enabling the model to adapt to multimodal service tasks.

[0084] The method provided in this embodiment of the invention includes a process for optimizing a pre-acquired third-party large language model based on the vertical domain, which is described in detail below:

[0085] Create initial tag text that matches the vertical domain;

[0086] The initial label text is expanded to obtain specific descriptive information;

[0087] Based on the specific description information, construct the loss function;

[0088] The aforementioned loss function is used to optimize the soft prompts of a pre-acquired third-party large language model.

[0089] Specifically, optimizing a third-party large language model based on a vertical domain requires establishing initial label text that matches the vertical domain. This could be a single keyword or an image category name, etc. This initial label text is closely related to the vertical domain, ensuring accuracy and consistency. Then, it needs to be expanded into a more detailed image description with semantic information. In one example, the vertical domain is car driving, and the task is abnormal vehicle driving behavior. Using "Name: XXX, Vehicle Type: XXX, Accident Type: Rear-end Collision" as the initial label text, the expanded result is: "This ride-hailing driver, named XXX, male, driving vehicle type XXX, was involved in a rear-end collision. According to the risk assessment, this accident has potential risks, and further analysis of the cause and liability is needed to assess the safety of his service."

[0090] In this process, background knowledge and task requirements from the vertical domain are combined to generate structured or natural language descriptions, providing richer semantic information support for image generation, task modeling, and subsequent model fine-tuning.

[0091] The loss function constructed based on the specific descriptive information is a function built after freezing the third-party large language model. For this large language model Ψ, a learnable tensor θ (also known as "soft hints") is constructed, and the following loss function is constructed:

[0092] ;

[0093] in, To provide specific information, soft suggestions are optimized by using keywords or descriptions relevant to the vertical domain, rather than modifying the parameters of the entire third-party large language model. This avoids overfitting and parameter inflation. The aforementioned soft suggestion learning is not only applicable to scenarios with scarce data, but also supports cross-modal tasks, ensuring that the third-party large language model generates text data focused on specific scenarios or tasks in vertical domain tasks.

[0094] The following section details the process by which the optimized third-party large language model in this application processes the initial text data and outputs the enhanced text data.

[0095] Establish the target problem based on the aforementioned vertical domain;

[0096] The target problem is broken down into its individual steps, resulting in a text document.

[0097] The logical reasoning process is constructed from the text of each step described;

[0098] Generate corresponding prompt text for each step of the text;

[0099] The prompt texts are added to the logical reasoning process according to their corresponding step texts to obtain the thought chain;

[0100] The input initial text data is reasoned according to the aforementioned thought chain to obtain enhanced text data, which is then output.

[0101] The step of reasoning through the initial input text data according to the aforementioned thought chain to obtain enhanced text data and then outputting it includes:

[0102] Acquire domain-specific knowledge corresponding to the vertical domain;

[0103] Reasoning is performed on the initial text data according to the aforementioned thought process;

[0104] While reasoning, the output answer corresponding to each step of the text is extracted in real time;

[0105] For each step text, adjust the prompt text for the next step text based on its corresponding output answer;

[0106] While making adjustments, the domain-specific knowledge is added to the prompt text to infer enhanced text data corresponding to the initial text data.

[0107] Specifically, the above process can be divided into three steps: learning soft prompts for tasks in a specific vertical domain, generating prompt text based on thought chains, and generating prompt text based on task tags and text attributes in the vertical domain.

[0108] In the above process, the generated prompt text can guide the model step by step to generate content according to the thought chain, thereby improving the ability to solve complex tasks. The prompt text at each step clarifies the part of reasoning or operation that the model needs to complete, making the generation process more systematic. Furthermore, during the generation process, the prompt text of subsequent steps is adjusted in real time according to the output of each step, which can ensure the coherence and accuracy of the thought chain and enhance the control over the generated output answer. Incorporating domain-specific knowledge into the prompt text can improve the understanding and adaptation capabilities of third-party large language models for vertical domain tasks.

[0109] The following example, using abnormal vehicle driving, will be used to illustrate the process of generating the above thought chain to obtain enhanced text data:

[0110] Input: "Suppose we are analyzing driving behavior data of a vehicle, including timestamps, speed, steering wheel angle, and braking intensity. Please list the categories of driving behavior that this data might reflect."

[0111] Model output: "These data can reflect the following driving behavior categories: normal driving, speeding, sudden braking, sudden lane change, fatigue driving, etc."

[0112] Input: "Suppose the following behaviors are detected: speed exceeding 120km / h, and sudden braking occurring 5 times within 10 minutes. Based on these conditions, please determine whether the driving behavior is abnormal."

[0113] Model output: "Abnormal driving behavior: Speeds exceeding 120km / h are considered speeding, and frequent sudden braking is a possible dangerous behavior."

[0114] Input: "Based on the above anomaly, generate a detailed anomaly description for the driver, including the problem and suggestions."

[0115] The model outputs: "Speeding (speed exceeding 120 km / h) and frequent sudden braking (5 sudden brakings within 10 minutes) have been detected. These behaviors may increase the risk of accidents. Drivers are advised to comply with speed limits, maintain smooth driving, and avoid frequent sudden braking."

[0116] In the above process, another effective prompt text can be designed to generate a high-quality dataset by combining the initial label text and text attributes of the vertical domain task. This supports the training of subsequent large-scale models. Based on the needs of the vertical domain task, key labels and text attributes are extracted, and the data distribution is analyzed to identify possible biases. Then, the attribute distribution in the generated data is checked with a trained attribute classifier to identify significant biases and explicitly add attribute constraints to the prompt text. This guides the generation model to output diverse data containing target attributes. By dynamically adjusting the prompt design, it is ensured that the generated data covers a wider range of attributes.

[0117] In one example, also taking abnormal vehicle driving as an example, but unlike simply constructing image descriptions and anomaly identification instructions, this application utilizes key labels and attribute distributions (such as "driver behavior," "driving environment," "distraction level," "potential consequences," and "intervention suggestions") to generate instruction text results based on different attribute values. For example, the attribute "driver behavior" includes "electronic device usage," "interactive behavior," and "other distracting behaviors." In this case, a third-party large language model can be used to construct instructions.

[0118] Electronic device use: "Is the driver using a mobile phone? Please describe in detail whether it is making calls, sending text messages, or using other applications." "Is the driver operating a navigation device? For example, adjusting the destination, viewing a map, etc."

[0119] Interactive Behaviors: "Did the driver talk to passengers in the vehicle? Did this behavior cause the driver's attention to be diverted?" "Did the driver look back? Please describe possible reasons, such as interacting with rear passengers or checking items."

[0120] Other distracting behaviors: "Did the driver eat, drink, or perform other non-driving-related actions? Please list the specific behaviors."

[0121] This ensures the balance of various attribute data, thereby enabling efficient model training and guaranteeing that the model can determine the existence and characteristics of abnormal driving behavior, while also providing improvement suggestions.

[0122] The following embodiments provide a detailed explanation of the steps in this application to process and enhance the initial image data to obtain selected images.

[0123] Identify the main object in the initial image data and determine the position, outline, and category of the main object;

[0124] The initial image data is segmented based on the location, contour, and category to obtain the first image data.

[0125] Determine the key tags and text attributes corresponding to the vertical domain, and generate a structured text description based on the key tags and text attributes;

[0126] The structured text description is processed using a pre-trained text-to-image model to obtain each task subject image corresponding to the structured text description;

[0127] Each initial image data point is used to select the main task images to obtain the selected images.

[0128] The step of filtering the subject images corresponding to each initial image data to obtain selected images includes:

[0129] Each of the task subject images is initially screened according to preset clarity threshold, integrity threshold, label matching threshold and background complexity threshold to obtain each coarsely selected image;

[0130] Each of the coarsely selected images and the initial image data are input into a pre-trained expert model, so that the expert model can accurately filter each coarsely selected image according to the initial image data, and output the filtered images as each selected image.

[0131] In the above scheme, the initial image data can be preprocessed first, such as denoising, deduplication, and removing blurry, unevenly exposed, or poor-quality frames. Then, normalization processing can be performed, such as adjusting the image size, resolution, and color space, to unify the input format. Next, segmentation is achieved by identifying the main object in the initial image data. This can be done by pre-establishing a segmentation scheme so that the first image data obtained is image data with the subject centered and clearly visible.

[0132] For key labels and text attributes corresponding to vertical domains, they often describe the visual concepts and semantic context of image data. Therefore, they can be used to generate structured text descriptions to clarify the requirements of image data. Then, a pre-trained text-based image model (diffusion model or generative adversarial network model) can be used as the basic generation framework. Domain-related data can be loaded or the model can be fine-tuned to adapt to specific tasks. Furthermore, visual label-related conditional prompts, such as color, shape, and scene, can be added to the input data of the text-based image model. This can guide the text-based image model to generate synthetic images that only contain the main content or subject of the task, i.e., task subject images, thereby ensuring the consistency between the image and the task requirements.

[0133] However, considering that the subject images of the task are not yet refined and accurate enough, preliminary screening and precise screening are performed on each subject image. For the preliminary screening, subject images that are below any of the following thresholds can be deleted: sharpness threshold, integrity threshold, label matching threshold, or background complexity threshold. Subsequently, the generated images and real images are input into a pre-trained expert model. This expert model is based on Transformer and uses a distance metric function and screening thresholds to remove abnormal images that are not related to the initial image data. This allows for precise screening to obtain synthetic image data that is similar to the initial image data, which is used to compensate for the common problem of imbalanced multimodal data.

[0134] Furthermore, the process of constructing a hybrid dataset from the initial multimodal data, enhanced text data, and selected images may include the following steps:

[0135] Obtain the prompt text and enhanced text data corresponding to the initial text data;

[0136] For each selected image, the prompt text and enhanced text data are associated with that selected image to generate individual image-text data pairs;

[0137] For each selected image, the text data pairs corresponding to each image are scored for authenticity in order to select the target data pairs.

[0138] A set of data pairs is constructed from each of the target data pairs, and the set of data pairs is balanced to obtain each balanced data pair.

[0139] A hybrid dataset is constructed from the initial multimodal data, each of the aforementioned balanced data pairs, and each selected image.

[0140] Specifically, for the prompt text and enhancement text data, noisy text, such as misspelled text and irrelevant descriptive text, can be removed first to retain high-quality descriptive text that meets semantic requirements. Then, the CLIP model (Contrastive Language–Image Pretraining) is used to calculate the similarity between each image text data and the selected image and the prompt text, i.e., enhancement text data. High-matching image text data pairs are selected, and image text data pairs with large semantic deviations are removed. Finally, the selected image and prompt text and enhancement text data are aligned in time or scene.

[0141] Next, the pre-acquired scoring model can be called to perform authenticity scoring and inference for each image-text data pair. The authenticity index of each image-text data pair, including semantic consistency, label accuracy, and correlation between image and text, can be calculated. A reasonable threshold can be set to select image-text data pairs that are higher than the threshold as target data pairs, thereby ensuring that the data used in the training and inference process has high quality.

[0142] In addition to screening for authenticity scores, it is also necessary to calculate the difference between each target data pair and the entire set of data pairs or a specific category, measure the diversity of the data pair set, calculate the vector distance between target data pairs (such as using cosine similarity in the embedding space) to assess diversity, measure the distribution of each category in the data pair set, ensure that the number of target data pairs for each category or label is balanced, avoid the model being overly biased towards one class, and through multiple rounds of screening and enhancement, ensure that the data pair set covers sufficient categories and attributes, and avoid duplicate samples and overfitting.

[0143] Finally, in the process of constructing the hybrid dataset from the initial multimodal data, each balanced data pair, and each selected image, the proportions of the initial multimodal data, balanced data pairs, and selected images can be reasonably designed to control the impact on the training of the base large model. The proportion of the initial multimodal data can be set to 30% and the proportions of the balanced data pairs and selected images can be set to 70% according to the task requirements. Alternatively, the proportions can be adjusted according to the specific distribution of the real data. After the construction is completed, a quality check should be performed to filter out noisy and low-quality data to ensure the overall quality of the hybrid dataset. The labels can be adjusted or the samples can be relabeled to ensure the effectiveness of the hybrid dataset for model training.

[0144] Therefore, this application enables the large-scale base model to adapt to task requirements in different vertical domains. The text enhancement and image enhancement processes automatically convert initial text data into vertically adapted instruction text, and address common class imbalance issues through text-to-image processing, improving the large-scale base model's rapid adaptation capability in small-sample task scenarios. It is applicable to various data structures and data service tasks. Compared to existing methods that only process single-type data, this application can extract valuable features from different data sources (such as images, text, forms, video frames, etc.), effectively fuse and enhance them, thereby generating synthetic data with higher information richness and stronger generalization ability. This multimodal synthetic data not only covers a wider range of scenarios and situations but also demonstrates superior performance and accuracy in multiple tasks, significantly improving the model's application effect. Furthermore, this application generates text data for specific tasks in vertical domains, including not only traditional descriptive content but also richer and more complex types such as reasoning and decision support. This helps the model better understand task requirements and consider more context and details during execution, effectively solving complex logical reasoning scenarios and task requirements that are difficult to handle in existing technologies, improving the efficiency and accuracy of data processing.

[0145] and Figure 1 Corresponding to the method described above, embodiments of the present invention also provide a large model optimization device based on small samples, used for optimizing large models. Figure 1 The specific implementation of the method, the large model optimization device based on small samples provided in this embodiment of the invention, can be used in computer terminals or various mobile devices, combined with Figure 3 This section introduces a large model optimization device based on a small sample size, such as... Figure 3 As shown, the device may include:

[0146] Multimodal data acquisition module 10 is used to acquire initial multimodal data;

[0147] The large language model optimization module 20 is used to determine the vertical domain corresponding to the initial multimodal data and optimize the pre-acquired third-party large language model based on the vertical domain.

[0148] The model processing module 30 is used to extract the initial text data from the initial multimodal data, input the initial text data into the optimized third-party large language model, and obtain the output enhanced text data.

[0149] The data augmentation processing module 40 is used to extract the initial image data from the initial multimodal data and perform data augmentation processing on the initial image data to obtain each selected image;

[0150] The hybrid dataset construction module 50 is used to construct a hybrid dataset from the initial multimodal data, enhanced text data, and selected images.

[0151] The base large model optimization module 60 is used to optimize the pre-established base large model using the hybrid dataset to obtain the optimized multimodal base large model.

[0152] As can be seen from the above technical solution, this application acquires initial multimodal data; determines the vertical domain corresponding to the initial multimodal data, and optimizes the pre-acquired third-party large language model based on the vertical domain; extracts initial text data from the initial multimodal data, inputs the initial text data into the optimized third-party large language model, and obtains output enhanced text data; extracts initial image data from the initial multimodal data, and performs data augmentation processing on the initial image data to obtain selected images; constructs a hybrid dataset from the initial multimodal data, enhanced text data, and selected images; and optimizes the pre-established base model using the hybrid dataset to obtain the optimized multimodal base model. This application categorizes the initial multimodal data and processes the categorized data separately. It can optimize the third-party large language model based on the vertical domain of the initial multimodal data, making the third-party large language model more targeted. The optimized third-party large language model can then better process the initial text data in the multimodal data to obtain enhanced text data. Furthermore, it performs data augmentation processing on the initial image data in the multimodal data to obtain selected images. Through the above processing, the coverage and scope of the data can be broadened, enabling the model to adapt to multimodal service tasks.

[0153] Furthermore, embodiments of this application provide a large model optimization device based on a small sample size. Optionally, Figure 4 The hardware structure block diagram of the large model optimization device based on small samples is shown. (Refer to...) Figure 4The hardware structure of the model optimization device may include: at least one processor 01, at least one communication interface 02, at least one memory 03, and at least one communication bus 04.

[0154] In this embodiment, the number of processor 01, communication interface 02, memory 03 and communication bus 04 is at least one, and processor 01, communication interface 02 and memory 03 communicate with each other through communication bus 04.

[0155] Processor 01 may be a central processing unit (CPU), an application-specific integrated circuit (ASIC), or one or more integrated circuits configured to implement embodiments of the present invention.

[0156] Memory 03 may include high-speed RAM, and may also include non-volatile memory, such as at least one disk storage device.

[0157] The memory stores a program, which the processor can call to execute. This program is used to perform the following large-model optimization methods based on small samples:

[0158] Acquire initial multimodal data;

[0159] Determine the vertical domain corresponding to the initial multimodal data, and optimize the pre-acquired third-party large language model based on the vertical domain;

[0160] Extract the initial text data from the initial multimodal data, and input the initial text data into the optimized third-party large language model to obtain the output enhanced text data;

[0161] Extract the initial image data from the initial multimodal data, and perform data augmentation processing on the initial image data to obtain each selected image;

[0162] A hybrid dataset is constructed from the initial multimodal data, enhanced text data, and selected images;

[0163] The pre-established large-scale pedestal model is optimized using the hybrid dataset to obtain an optimized multimodal large-scale pedestal model.

[0164] Optionally, the refinement and extension functions of the program can be found in the description of the large model optimization method based on small samples in the method embodiments.

[0165] This application embodiment also provides a storage medium that can store a program suitable for execution by a processor. When the program runs, it controls the device where the storage medium is located to execute the following large model optimization method based on small samples, including:

[0166] Acquire initial multimodal data;

[0167] Determine the vertical domain corresponding to the initial multimodal data, and optimize the pre-acquired third-party large language model based on the vertical domain;

[0168] Extract the initial text data from the initial multimodal data, and input the initial text data into the optimized third-party large language model to obtain the output enhanced text data;

[0169] Extract the initial image data from the initial multimodal data, and perform data augmentation processing on the initial image data to obtain each selected image;

[0170] A hybrid dataset is constructed from the initial multimodal data, enhanced text data, and selected images;

[0171] The pre-established large-scale pedestal model is optimized using the hybrid dataset to obtain an optimized multimodal large-scale pedestal model.

[0172] Specifically, the storage medium can be a computer-readable storage medium, which can be an electronic storage device such as flash memory, EEPROM (Electrically Erasable Programmable Read-Only Memory), EPROM, hard disk, or ROM.

[0173] Optionally, the refinement and extension functions of the program can be found in the description of the large model optimization method based on small samples in the method embodiments.

[0174] Furthermore, the functional modules in the various embodiments of this disclosure can be integrated together to form an independent part, or each module can exist independently, or two or more modules can be integrated to form an independent part. If the function is implemented as a software functional module and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this disclosure, in essence, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, a live streaming device, or a network device, etc.) to execute all or part of the steps of the methods in the various embodiments of this disclosure.

[0175] Finally, it should be noted that in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0176] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on the differences from other embodiments. The same or similar parts between the various embodiments can be referred to each other.

[0177] The above description of the disclosed embodiments enables those skilled in the art to make or use this application. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of this application. Therefore, this application is not to be limited to the embodiments shown herein, but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A method for optimizing a large model based on small samples, characterized in that, include: Acquire initial multimodal data; The process involves identifying a vertical domain corresponding to the initial multimodal data, and optimizing a pre-acquired third-party large language model based on that vertical domain. This includes: establishing initial label text matching the vertical domain; expanding the initial label text to obtain specific descriptive information; constructing a loss function based on the specific descriptive information; and optimizing the soft suggestions of the pre-acquired third-party large language model using the loss function. The process involves extracting initial text data from the initial multimodal data and inputting it into an optimized third-party large language model to obtain enhanced text data. This includes: establishing a target question based on the vertical domain; decomposing the target question to obtain step texts; constructing a logical reasoning process from each step text; generating corresponding prompt texts for each step text; adding the prompt texts to the logical reasoning process according to their corresponding step texts to obtain a thought chain; and reasoning based on the input initial text data according to the thought chain to obtain enhanced text data and output it. The process involves extracting initial image data from the initial multimodal data and performing data augmentation on the initial image data to obtain selected images. This includes: identifying the main object in the initial image data and determining the position, outline, and category of the main object; segmenting the initial image data based on the position, outline, and category to obtain first image data; determining key tags and text attributes corresponding to the vertical domain and generating a structured text description based on the key tags and text attributes; processing the structured text description using a pre-trained text-to-image model to obtain task subject images corresponding to the structured text description; and filtering the task subject images corresponding to each initial image data to obtain selected images. A hybrid dataset is constructed from the initial multimodal data, enhanced text data, and selected images; The pre-established large-scale pedestal model is optimized using the hybrid dataset to obtain an optimized multimodal large-scale pedestal model.

2. The method according to claim 1, characterized in that, The step of reasoning through the input initial text data according to the aforementioned thought chain to obtain enhanced text data and then outputting it includes: Acquire domain-specific knowledge corresponding to the vertical domain; Reasoning is performed on the initial text data according to the aforementioned thought process; While reasoning, the output answer corresponding to each step of the text is extracted in real time; For each step text, adjust the prompt text for the next step text based on its corresponding output answer; While making adjustments, the domain-specific knowledge is added to the prompt text to infer enhanced text data corresponding to the initial text data.

3. The method according to claim 1, characterized in that, The process of filtering the task subject images corresponding to each initial image data to obtain selected images includes: Each of the task subject images is initially screened according to preset clarity threshold, integrity threshold, label matching threshold and background complexity threshold to obtain each coarsely selected image; Each of the coarsely selected images and the initial image data are input into a pre-trained expert model, so that the expert model can accurately filter each coarsely selected image according to the initial image data, and output the filtered images as each selected image.

4. The method according to any one of claims 1 to 3, characterized in that, The hybrid dataset constructed from the initial multimodal data, enhanced text data, and selected images includes: Obtain the prompt text and enhanced text data corresponding to the initial text data; For each selected image, the prompt text and enhanced text data are associated with that selected image to generate image-text data pairs; For each selected image, the authenticity of each image-text data pair is scored to select the target data pairs. A set of data pairs is constructed from each of the target data pairs, and the set of data pairs is balanced to obtain each balanced data pair. A hybrid dataset is constructed from the initial multimodal data, each of the aforementioned balanced data pairs, and each selected image.

5. A large model optimization device based on small samples, characterized in that, include: The multimodal data acquisition module is used to acquire initial multimodal data; The large language model optimization module is used to determine the vertical domain corresponding to the initial multimodal data, and optimize the pre-acquired third-party large language model based on the vertical domain; including: establishing initial label text matching the vertical domain; expanding the initial label text to obtain specific descriptive information; constructing a loss function based on the specific descriptive information; and optimizing the soft prompts of the pre-acquired third-party large language model using the loss function; The model processing module is used to extract initial text data from the initial multimodal data, input the initial text data into the optimized third-party large language model, and obtain the output enhanced text data; including: establishing a target question based on the vertical domain; decomposing the target question to obtain step texts; constructing a logical reasoning process from each step text; generating corresponding prompt texts for each step text; adding the prompt texts to the logical reasoning process according to their corresponding step texts to obtain a thought chain; and reasoning on the input initial text data according to the thought chain to obtain enhanced text data and output it. A data augmentation module is used to extract initial image data from the initial multimodal data and perform data augmentation on the initial image data to obtain selected images. This includes: identifying the main object in the initial image data and determining the position, outline, and category of the main object; segmenting the initial image data based on the position, outline, and category to obtain first image data; determining key tags and text attributes corresponding to the vertical domain and generating a structured text description based on the key tags and text attributes; processing the structured text description using a pre-trained text-to-image model to obtain task subject images corresponding to the structured text description; and filtering the task subject images corresponding to each initial image data to obtain selected images. A hybrid dataset construction module is used to construct a hybrid dataset from the initial multimodal data, enhanced text data, and selected images; The base large model optimization module is used to optimize the pre-established base large model using the hybrid dataset to obtain the optimized multimodal base large model.

6. A large model optimization device based on small samples, characterized in that, Including memory and processor; The memory is used to store programs; The processor is configured to execute the program to implement the steps of the small-sample-based large-model optimization method as described in any one of claims 1-4.

7. A storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the steps of the small-sample-based large-model optimization method as described in any one of claims 1-4.

Images