Dataset creation method, computer device, and storage medium

Abstract

The application discloses a dataset creation method, a computer device and a storage medium, relates to the technical field of data processing, and comprises the following steps: obtaining picture information, inference process data and a logical relationship in initial data, splicing inference step data and the picture information into the inference process data according to the logical relationship; generating a sub-dataset corresponding to a preset data type according to the inference process data; and generating a first target dataset and a second target dataset according to the sub-dataset. The method can solve the problem that a dataset is mainly created in a single text mode, and pictures only appear in questions, which makes it difficult for picture information to assist the thinking of a model, and the interpretability of the thinking process of the model is poor. The method can construct a target dataset in which picture and text alternately appear in the thinking process, and the use of the target dataset to train a model can make the picture assist the thinking of the model, and the interpretability of the thinking process of the model is enhanced, so that the model has the ability to alternately generate pictures and texts in the CoT inference process.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of data processing, and in particular to a data set creation method, a computer device and a storage medium. BACKGROUND

[0002] Multi-modal CoT (Chain of thought) reasoning is an important milestone for artificial intelligence systems to approach human thinking, and multi-modal CoT reasoning can enable large models to understand information in text and pictures, which enables large models to handle complex tasks.

[0003] By creating a CoT data set to train a large model, the large model can perform multi-modal CoT reasoning. Currently, a multi-modal CoT data set is created by obtaining mathematical problems, problem reasoning processes, problem answers, dynamic and static question and answer pairs, and the like. The data used to create the data set is no longer limited to text modalities, generally includes picture data in the question, and includes answers and detailed explanations of the answers. However, the current CoT data set is relatively small, and is mostly single text modalities, with pictures only appearing in the question. In the reasoning process, abstract and complex symbols and text are mostly used, and images are not really added to the reasoning process. Using such a CoT data set to train a model is difficult to let visual information assist the model's thinking, resulting in poor interpretability of the model's thinking process. SUMMARY

[0004] Therefore, the present application provides a data set creation method, a computer device and a storage medium to solve the problem that the created data set is mostly single text modalities, with pictures only appearing in the question, making it difficult for picture information to assist the model's thinking, resulting in poor interpretability of the model's thinking process.

[0005] In a first aspect, the present application provides a data set creation method, which comprises:

[0006] obtaining initial data corresponding to a preset data type, wherein the initial data comprises a plurality of different types of data;

[0007] obtaining reasoning step data, a logical relationship of the reasoning step data, and picture information corresponding to the reasoning step data in the initial data;

[0008] concatenating the reasoning step data and the picture information into reasoning process data according to the logical relationship, and generating a sub-data set corresponding to the preset data type according to the reasoning process data;

[0009] mixing the sub-data sets to generate a first target data set;

[0010] The target sub-data set is obtained in the sub-data set, and the target sub-data set is mixed according to a preset proportion to generate a second target data set.

[0011] In a second aspect, the present application provides a data set creation device, which comprises:

[0012] The first data acquisition module is configured to acquire initial data corresponding to a preset data type, wherein the initial data comprises a plurality of different types of data.

[0013] The second data acquisition module is configured to acquire inference step data, a logical relationship of the inference step data, and picture information corresponding to the inference step data in the initial data.

[0014] The first data set generation module is configured to splice the inference step data and the picture information into inference process data according to the logical relationship, and generate a sub-data set corresponding to the preset data type according to the inference process data.

[0015] The second data set generation module is configured to mix the sub-data sets to generate a first target data set.

[0016] The third data set generation module is configured to obtain a target sub-data set in the sub-data set, mix the target sub-data set according to a preset proportion, and generate a second target data set.

[0017] In a third aspect, the present application provides a computer device, which comprises a memory and a processor, the memory and the processor are communicatively connected with each other, the memory stores computer instructions, and the processor executes the computer instructions to perform the data set creation method of the first aspect or any of the corresponding embodiments thereof.

[0018] In a fourth aspect, the present application provides a computer readable storage medium, which stores computer instructions, and the computer instructions are used to make a computer execute the data set creation method of the first aspect or any of the corresponding embodiments thereof.

[0019] In a fifth aspect, the present application provides a computer program product, which comprises computer instructions, and the computer instructions are used to make a computer execute the data set creation method of the first aspect or any of the corresponding embodiments thereof.

[0020] According to the application, the method obtains picture information, inference process data and logical relationships in initial data, splices inference step data and picture information into inference process data according to the logical relationships, generates a sub-data set corresponding to a preset data type according to the inference process data, and generates a first target data set and a second target data set according to the sub-data set. The method can solve the problem that the creation of a data set is mainly based on a single text mode, and pictures only appear in questions, which makes it difficult for picture information to assist the thinking of a model, and leads to poor explainability of the thinking process of the model. The method can construct a target data set in which pictures and texts appear alternately in the inference process, the pictures in the target data set not only appear in questions, but also have multi-modal information; training a model using the target data set can make picture-assisted thinking of the model, and enhance the explainability of the thinking process of the model, so that the model has the ability to alternately generate pictures and texts in the CoT inference process. BRIEF DESCRIPTION OF DRAWINGS

[0021] In order to more clearly illustrate the technical solutions in the specific embodiments or related art of the present application, the drawings needed to be used in the specific embodiments or related art description will be briefly introduced. Obviously, the drawings in the following description are some embodiments of the present application, and those skilled in the art can also obtain other drawings according to these drawings without creative labor.

[0022] Figure 1 is a flowchart of a data set creation method according to an embodiment of the present application;

[0023] Figure 2 is a flowchart of creating a multi-modal CoT data set according to an embodiment of the present application;

[0024] Figure 3 is a flowchart of another data set creation method according to an embodiment of the present application;

[0025] Figure 4 is a flowchart of extracting picture-text data from video data according to an embodiment of the present application;

[0026] Figure 5 is a structural block diagram of a data set creation device according to an embodiment of the present application;

[0027] Figure 6 is a hardware structure schematic diagram of a computer device according to an embodiment of the present application. DETAILED DESCRIPTION

[0028] In order to make the purposes, technical solutions and advantages of the embodiments of the present application clearer, the technical solutions in the embodiments of the present application will be described clearly and completely below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are some but not all of the embodiments of the present application. Based on the embodiments in the present application, all other embodiments obtained by a person of ordinary skill in the art without creative work fall within the protection scope of the present application.

[0029] The multi-modal CoT reasoning contains a series of multi-step reasoning processes, and the thinking of the latter is based on the thinking of the previous steps. This way is very similar to the reaction of human beings when facing complex problems. However, the current multi-modal CoT reasoning mostly contains pictures in the problem, and mostly uses abstract and complex symbols and texts in the reasoning process. The image is not really added to the reasoning process, and it is difficult to use the image to reduce the difficulty of thinking or explanation in the reasoning process. For example: the problem related to space and motion is a relatively important problem in the real world. Motion in a space is a continuous process. At this time, if there is a picture to explain the motion process, it will be much simpler and easier to understand than just using abstract language to describe. Because the image is not added to the reasoning process, many models will encounter difficulties when facing complex spatial relationships, and it is difficult to handle problems related to continuous motion in a space. In addition, handling geometric problems is another difficult problem for models. Whether it is plane geometry or solid geometry, it is often necessary to add several auxiliary lines or establish a coordinate system to help solve the problem. Since the current model processes geometric problems, the process is detached from the picture, making it difficult for the model to handle geometric problems. In addition to geometric problems, there are many other problems that require drawing to assist in solving problems, such as mathematical application problems, trigonometric functions, and force-related physics problems, and space-related geography problems. The current model is difficult to handle the above problems. In addition to assisting the model in reasoning, adding pictures to the reasoning process can effectively enhance the explainability of the model's reasoning process. When the model directly outputs the answer, the reasoning process is a black box, and it is impossible to understand how the model arrives at the answer. The multi-modal CoT reasoning process outputs text information related to the reasoning process, and from these text information, the model's thinking process can be seen. By introducing pictures into the CoT reasoning process, the reasoning process can be more effectively and intuitively represented.

[0030] Currently, there are relatively few datasets related to CoT, and existing multimodal reasoning datasets are not only far fewer in number than pure language CoT datasets, but also difficult to compare in dataset volume. Fewer datasets have added pictures to the CoT process. Existing datasets such as ScienceQA, Math-vision, CoCoMMR, and SAT datasets, in general, are no longer limited to text modalities. They generally contain picture data in the questions and include answers and detailed explanations of the answers. ScienceQA includes natural sciences, social sciences, and linguistics, with a data volume of 21Kb. The dataset is composed of science-related multiple-choice questions and problem-solving processes from primary to high school levels. The questions contain multimodal backgrounds, correct options, general background knowledge, and specific explanations. Nearly half of the content in ScienceQA contains charts, and 90% contains reasoning processes. It is a relatively comprehensive but small dataset. COCO-MMR has a data volume of 62Kb. The data contains questions, reasoning processes, and answers. The dataset introduces more scenarios related to daily life and contains more open-ended questions. MATH-Vision contains 3Kb of math competition-level data, covering 16 different math fields and 5 difficulty levels. Each question contains visual data, which can be used to comprehensively evaluate the performance of large models in multimodal mathematical reasoning. The SAT dataset contains 218Kb of dynamic and static question-answer pairs, including self-centered motion and object motion, which can enhance the spatial perception and reasoning capabilities of the model. Through training on the above datasets, the model can achieve better performance in scientific reasoning, mathematics, daily reasoning, and spatial perception.

[0031] However, the above datasets have relatively small dataset volumes. The relatively small dataset volume makes it difficult for the model to master the form of multimodal CoT reasoning. The above datasets are mostly limited to one aspect of data, such as only relating to mathematics or science. The above datasets have different standards for establishing CoT reasoning processes. Some are detailed, some are brief, and some are even not in natural language, making it difficult to directly combine the above datasets for model training. Most importantly, in the above datasets, pictures only appear in the questions and do not truly participate in the CoT reasoning process. The part of the dataset that records the CoT reasoning process still only has text as a single modality.

[0032] Based on the above, the embodiment of the application provides a dataset creation method, which collects original data, including videos, exercise sets, existing multi-modal datasets, game-related datasets, etc.; processes video data, each frame of which corresponds to a description, and the processed data is concatenated with certain prompt words to form a dataset A; processes exercise set data, selects questions and answers with pictures to form a dataset B; filters and processes existing multi-modal datasets, and marks the part that needs to introduce pictures in the CoT reasoning process as a dataset C; filters and processes game-related data to form a dataset D; mixes the dataset A, the dataset B, the dataset C and the dataset D to form a first-stage SFT (Supervised Fine-Tuning, supervised fine-tuning) dataset; and mixes the dataset B, the dataset C and the dataset D in a certain proportion to form a second-stage SFT dataset. The conversion of part of the video data into multi-modal CoT data effectively saves the computing resources required during dataset construction; the dataset with picture-text data in the CoT process is constructed; two multi-modal CoT datasets are constructed, which effectively expand the size of the dataset and also unify the form of the prompt words in the training data. Training the model using such a dataset can make the visual information truly assist the thinking of the model and enhance the explainability of the thinking process of the model. In addition, the above dataset fine-tunes a model that already has the ability of picture and text analysis and generation, so that the model has the ability to alternately generate picture-text in the CoT process.

[0033] According to the embodiment of the dataset creation method provided in the application, it should be noted that the steps shown in the flowchart can be executed in a computer system such as a group of computer executable instructions, for example: a computer, a server, etc., and although the logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in a different order from here.

[0034] In this embodiment, a dataset creation method is provided, Figure 1 is a flowchart of the dataset creation method according to the embodiment of the application, as Figure 1 shown, the flow includes the following steps:

[0035] Step S101, obtaining initial data corresponding to a preset data type, wherein the initial data includes a plurality of different types of data.

[0036] Specifically, the preset data type is, for example: video, exercise set, existing dataset and game-related dataset. The initial data corresponding to the preset data type is, for example: video data, exercise set data, game-related data, and data contained in the existing dataset. Therefore, the initial data includes a plurality of different types of data.

[0037] The above process is as shown inFigure 2 The raw data is collected, including videos, problem sets, existing data sets, and some game-related data sets, as shown.

[0038] In step S102, inference step data, logical relationships of the inference step data, and picture information corresponding to the inference step data are obtained from the initial data.

[0039] Specifically, although conventional multi-modal data sets contain different types of data, only questions and answers are included in these data. When a model is trained using a conventional multi-modal data set, the model is difficult to handle complex tasks. The present embodiment creates a multi-modal thinking chain data set by adding inference step data to the data set.

[0040] The inference step data is obtained from the initial data, for example: solving problem X requires 4 inference steps, and the inference step data is inference step x1, inference step x2, and inference step x3. The multi-modal thinking chain data set contains exact logical relationships between the inference steps, and the subsequent steps are derived based on the previous steps. The logical relationships of the inference step data are obtained from the initial data, for example: inference step x2 is derived based on the result of inference step x1, and inference step x3 is derived based on the result of inference step x2. In addition, picture information is used to assist in generating or explaining inference step data during the inference process. The picture information corresponding to the inference step data is obtained from the initial data, for example: inference step x1 corresponds to picture a, and inference step x3 corresponds to picture b.

[0041] In step S103, the inference step data and the picture information are concatenated into inference process data according to the logical relationships, and a sub-data set corresponding to a predetermined data type is generated according to the inference process data.

[0042] Specifically, the logical relationships are, for example: inference step x2 is derived based on the result of inference step x1, and inference step x3 is derived based on the result of inference step x2. The inference step data is concatenated into inference process data, for example: inference step x1, inference step x2, and inference step x3. The inference step data and the picture information are concatenated into inference process data according to the logical relationships, for example: inference step x1, picture a, inference step x2, inference step x3, and picture b.

[0043] To construct a multi-modal data set of thinking chains, multiple predetermined data types are required, such as videos, problem sets, existing data sets, and game-related data sets.

[0044] If the preset data type is video, each frame of the video data corresponds to a description, and each frame contains some image information. The description of each frame contains the corresponding reasoning process data. The frames and descriptions in the video data are concatenated using certain prompts to obtain the reasoning process data. If the preset data type is a set of exercises, questions or answers in the set that contain images are selected, and the images are inserted into the reasoning process data of the questions to obtain the reasoning process data. If the preset data type is an existing dataset, image information is inserted into the reasoning process data of the existing dataset to obtain the reasoning process data. If the preset data type is a game-related dataset, the step diagrams and explanatory texts of the game are obtained from the dataset. The step diagrams contain image information, and the explanatory texts contain reasoning process data. The step diagrams and explanatory texts are concatenated using certain prompts to obtain the reasoning process data.

[0045] Based on the above reasoning process, sub-datasets corresponding to preset data types are generated. Examples of sub-datasets include: dataset A corresponding to videos, dataset B corresponding to exercise sets, dataset C corresponding to existing datasets, and dataset D corresponding to game-related datasets.

[0046] The current thought chain dataset only contains images in the questions or answers, while the reasoning process data uses relatively abstract and complex symbols and text, without incorporating images into the reasoning process. Compared to the current thought chain dataset, the aforementioned subset inserts image information corresponding to the reasoning steps into the reasoning process data, combining them to form the reasoning process data. This incorporates images into the reasoning process, increasing the model's problem-solving accuracy while improving the interpretability of the thought chain.

[0047] The above process is as follows Figure 2 As shown, the video dataset is processed, with each frame corresponding to a description, forming dataset A; the exercise dataset is processed, selecting questions and / or answers containing images, forming dataset B; the existing multimodal dataset is filtered, and the parts where it is necessary to introduce images in the reasoning process are marked as dataset C; the game-related data is filtered and processed, forming dataset D.

[0048] Step S104: Mix the subset datasets to generate the first target dataset.

[0049] Specifically, a first target dataset and a second target dataset are generated based on the subset datasets. These two datasets serve as the SFT datasets for two stages, reducing the amount of data annotation. The first stage of fine-tuning aims to enable the model to generate alternating text and images, while the second stage aims to improve the model's performance on key tasks. The first target dataset is larger in size and has fewer requirements regarding the domain and text length. The second target dataset is smaller and focuses on domains requiring multimodal CoT processing, such as mathematics, physics, and spatial understanding, with stricter requirements for the quality of the dataset annotations. The first preset data type is, for example, video; the second preset data type is, for example, a problem set; the third preset data type is, for example, existing multimodal datasets; and the fourth preset data type is, for example, game-related datasets.

[0050] The subsets are combined to generate the first target dataset. For example, all subsets generated in step S102 include dataset A corresponding to the first preset data type, dataset B corresponding to the second preset data type, dataset C corresponding to the third preset data type, and dataset D corresponding to the fourth preset data type. Dataset A, for example, is video data, where each frame of the video data corresponds to a description. The processed data is concatenated with certain prompts to obtain reasoning process data, and dataset A corresponding to the video is generated based on this reasoning process data. Dataset B, for example, is exercise set data, where questions or answers containing images are selected, or questions and answers containing images are selected as reasoning process data, and dataset B corresponding to the exercise set is generated based on this reasoning process data. Dataset C, for example, is the part of the existing multimodal dataset that needs to have images introduced in the CoT reasoning process, which is marked as reasoning process data, and dataset C corresponding to the existing multimodal dataset is generated based on this reasoning process data. Dataset D, for example, is the step diagrams and explanatory text of the game obtained from the game-related dataset to constitute reasoning process data, and dataset D corresponding to the game-related dataset is generated based on this reasoning process data. Dataset A is large in size and has a good correspondence between images and text, but there is no clear logical reasoning relationship between the data.

[0051] Data sets A, B, C, and D are combined to generate the first target dataset. This first target dataset can serve as the SFT dataset for the first stage, and the majority of data in the first target dataset will originally belong to dataset A. Training a large language model or multimodal language model using the first target dataset enables the model to acquire the ability to alternately generate text from images. The first target dataset is one type of target dataset.

[0052] Step S105: Obtain the target subset from the subset dataset, and mix the target subset according to a preset ratio to generate the second target dataset.

[0053] Specifically, a quantity is set, such as 2, 3, or other values ​​that meet the actual needs. A set number of subsets are selected from the subsets as the target subsets. For example, a set number of subsets are selected as the target subsets, including dataset B, dataset C, and dataset D. Datasets B, C, and D have relatively small data volumes and high data quality, ensuring a progressive reasoning relationship between the text and images.

[0054] Preset ratios, such as 1:1:1, 2:1:1, or other ratios that meet practical needs, are used. Datasets B, C, and D are mixed according to these preset ratios to generate a second target dataset. This second target dataset can serve as the SFT dataset for the second stage. Training a large language model or multimodal language model using this second target dataset enables the model to acquire multimodal reasoning capabilities. The second target dataset is one type of target dataset.

[0055] The above process is as follows Figure 2 As shown, datasets A, B, C, and D are mixed to form the first target dataset; datasets B, C, and D are mixed to form the second target dataset.

[0056] In this implementation, a first dataset and a second dataset are generated based on subsets, effectively expanding the dataset size while unifying the format of prompts in the training data. The larger first dataset covers more domains and is used for the first stage of SFT, allowing the model to gain generality from the large-scale data and learn the format of alternating image and text generation. The smaller second dataset, focused on key tasks, is used for the second stage of SFT, enabling the model to achieve better performance on multimodal CoT tasks that require image-assisted thinking.

[0057] The dataset creation method provided in this embodiment obtains image information, inference process data, and logical relationships from the initial data. Based on the logical relationships, the inference step data and image information are concatenated to form inference process data. Sub-datasets corresponding to preset data types are generated based on the inference process data. A first target dataset and a second target dataset are then generated based on the sub-datasets. This method solves the problem that most datasets are primarily text-based, with images only appearing in the question, making it difficult for image information to assist the model's thinking and resulting in poor interpretability of the model's thinking process. This method can construct a target dataset where images and text alternate during the inference process. The images in the target dataset are not only present in the question but also possess multimodal information. Training the model using the target dataset allows images to assist the model's thinking and enhances the interpretability of the model's thinking process, enabling the model to generate images and text alternately during CoT inference. This solves the problem that most datasets are primarily text-based, with images only appearing in the question, making it difficult for image information to assist the model's thinking and resulting in poor interpretability of the model's thinking process.

[0058] This embodiment provides another method for creating datasets. Figure 3 This is a flowchart of another dataset creation method according to an embodiment of this application, such as... Figure 3 As shown, the process includes the following steps:

[0059] Step S301: Obtain the initial data corresponding to the preset data type.

[0060] Specifically, step S301 includes steps S3011 to S3014.

[0061] Step S3011: Obtain the initial data corresponding to the first preset data type from the preset database according to the preset keywords.

[0062] Step S3012: Obtain a preset exercise set and use the data in the preset exercise set as the initial data corresponding to the second preset data type.

[0063] Step S3013: Obtain a preset dataset and use the data in the preset dataset as the initial data corresponding to the third preset data type.

[0064] Step S3014: Obtain the game dataset related to spatial location, and use the data in the game dataset as the initial data corresponding to the fourth preset data type.

[0065] Specifically, the first preset data type is, for example, video; the second preset data type is, for example, exercise sets; the third preset data type is, for example, existing multimodal datasets; and the fourth preset data type is, for example, game-related datasets.

[0066] Preset keywords include, for example: exercises, code, geometry, mechanics, chess and card games, etc. Preset keywords can be either English or Chinese. For Chinese keywords, the search will be conducted on primarily Chinese video or short video websites; for English keywords, the search will be conducted on primarily English video or short video websites.

[0067] Initial data corresponding to a first preset data type is retrieved from a preset database based on preset keywords. For example, the preset database includes Chinese and English video websites. Videos are collected from both websites using 20 Chinese and English keywords, with video lengths controlled between 2 seconds and 600 seconds. This video data serves as the initial data. The video data includes videos related to code explanations, course explanations, and training. Relevant videos are retrieved through keyword search, and the video aspect ratio is calculated. The original video source will be in a mixed Chinese and English format. The collected information includes not only the video itself but also the video title, subtitles, keywords, subtitles, and video links. To control the video's content, language, and aspect ratio, keyword search is used to retrieve the videos. The initial data corresponding to the first preset data type includes videos related to code explanations, exercise explanations, and course training.

[0068] Obtain a pre-set set of exercises, such as: a set of exercises including elementary school word problems, elementary school math competition problems, plane / solid / analytic geometry problems, topology, graph theory, and physics problems. Physics problems may include various mechanics, thermodynamics, electromagnetism, and optics problems. A small portion of the pre-set set of exercises is selected from existing public datasets, retaining those with images in the questions or answers. Public datasets include, for example, the Common Crawler dataset, which is a publicly accessible database. The majority is selected from answer keys for exercises at various educational levels, retaining those with images in the answers. The set also includes mixed text and image data in both Chinese and English. Use the data from the pre-set set of exercises as the initial data for the second pre-set data type.

[0069] The preset dataset is an existing multimodal question-answering dataset related to reasoning, spatial relationships, and daily life, such as the Math-V dataset, CoCoMMR dataset, SAT dataset, AMC dataset, etc. The data in the preset dataset is used as the initial data corresponding to the third preset data type.

[0070] Obtain a game dataset related to spatial location, including games such as maze problems, ice lake problems, and chess scores. Use the data from this game dataset as the initial data for the fourth preset data type.

[0071] In this implementation, videos, exercise sets, existing multimodal datasets, and game-related datasets are acquired to determine the initial data corresponding to preset data types. Sub-databases are created based on the initial data, increasing the database size and enriching its data types. Furthermore, the initial data contains both text and image information, facilitating the subsequent generation of text and image data.

[0072] Step S302: Obtain the inference step data, the logical relationship of the inference step data, and the image information corresponding to the inference step data from the initial data.

[0073] For details on how to implement this step, please refer to [link / reference]. Figure 1 Step S102 of the illustrated embodiment will not be described again here.

[0074] Step S303: Based on the logical relationship, the reasoning step data and image information are concatenated into reasoning process data, and a subset of data corresponding to the preset data type is generated based on the reasoning process data.

[0075] Specifically, step S303 includes steps A1 to A4.

[0076] Step A1: When the preset data type is the first preset data type, merge video frames with the same subtitle information and use the merged video frame as the new video frame. The video frame and the corresponding subtitle information are obtained from the initial data. The video frame contains image information and there is a logical relationship between the video frames.

[0077] Step A2: Generate the first explanatory information of the video frame and the second explanatory information of the new video frame based on the subtitle information.

[0078] Step A3: Based on the logical relationship, the video frame, the new video frame, the first explanatory information, and the second explanatory information are concatenated into reasoning process data.

[0079] Step A4: Generate the first subset of data based on the inference process data.

[0080] Specifically, the first preset data type is, for example, video. The initial data corresponding to the first preset data type is video data. This embodiment uses video data to generate the first subset of data, which can quickly obtain a large amount of data with alternating text and images, and there is a certain progressive relationship between the alternating text and images. In addition, this embodiment does not intend to actually use the video data, but rather to extract courseware information from the video data, such as static PPTs (PowerPoint presentations), hand-drawn drawings, code, etc., and can obtain frames with a relatively large number of words through video data. Considering the main application areas of multimodal CoT, the video data used in this embodiment is videos related to course explanations, code explanations, etc.

[0081] Retrieve all video frames and subtitle information contained within them from the initial data corresponding to the first preset data type. There are logical relationships between the video frames, such as the order of their timestamps.

[0082] To save computation, this embodiment utilizes subtitle information to compress video frames. Video frames with identical subtitle information are merged, and one frame from these is selected as the merged video frame, known as the video frame sample (F_sample). Only the merged video frame is retained, while other video frames are deleted, thus compressing the video frame rate to F_sample. Video frames covered by the same subtitle information are treated as a single video frame, and the subtitle information is extracted as a description of the video frame. This merged video frame is then used as the new video frame. For example, if a subtitle reads "Okay, in this short video I will introduce what a Monte Carlo tree is and how it works," all video frames containing this subtitle are merged into one video frame, which is then used as the new video frame. For example, the first video frame containing this subtitle is used as the new video frame. All video frames are compressed using this method.

[0083] Based on the subtitle information, first descriptive information for each video frame and second descriptive information for new video frames are generated. For example, 1-N video frames are randomly collected to form a data entry. After each video frame, the corresponding subtitle is used as either the first or second descriptive information. Images and text will appear alternately. Questions related to the video are generated based on the video title, such as "How to use the tool." The value of N can be 4, 5, or other values ​​that meet the actual needs.

[0084] Based on logical relationships, video frames, new video frames, first explanatory information, and second explanatory information are concatenated to form inference process data. In this data, video frames and explanatory information appear alternately, with video frames arranged chronologically. Explanatory information for a video frame appears either before or after the previous video frame. Based on this inference process data, a first subset of data is generated.

[0085] The above process is as follows Figure 4 As shown, video frames covered by the same subtitle are compressed into a single video frame, and subtitle information is extracted. Several data frames are randomly collected and organized according to specific prompts to form a data sample.

[0086] In this embodiment, video frames are compressed using subtitle information, saving computational resources. Based on the video frames, new video frames, first explanatory information, and second explanatory information, a first subset of data consisting of alternating images and text is generated. Converting video data into inference process data effectively conserves computational resources required for subset construction.

[0087] As an optional embodiment, step A3 "generating explanatory information for video frames based on subtitle information" includes steps B1 to B4.

[0088] Step B1: Obtain the proportion of target information in the video frame.

[0089] Step B2: Select video frames with an information ratio greater than the first preset threshold as target video frames, and determine other video frames other than the target video frames.

[0090] Step B3: Save the subtitle information corresponding to other video frames to the target video frame adjacent to the other video frames, and delete the other video frames.

[0091] Step B4: Generate descriptive information for the target video frame based on the subtitle information corresponding to the target video frame.

[0092] Specifically, target information includes, for example, text information and information other than human figures. The proportion of target information within a video frame is obtained. For example, using PaddleOCR, an open-source OCR (Optical Character Recognition) tool that supports both Chinese and English, the text proportion of the video frame can be calculated; using open-source face detection tools, such as InsightFace, can be used to detect faces or upper body proportions within the video frame, thus determining the proportion of information other than human figures; the number of pixels containing target information can also be determined, and divided by the total number of pixels in the video frame to obtain the information proportion.

[0093] Video frames with an information ratio greater than a first preset threshold are designated as target video frames. For example, if the information ratio is the proportion of text, and the first preset threshold is 50%, then if the text ratio of a video frame is greater than 50%, that video frame is designated as the target video frame. Alternatively, if the information ratio is the proportion of information other than the human face, and the first preset threshold is 55%, then if the proportion of information other than the human face is less than 55%, i.e., the proportion of the face or upper body of the frame is greater than 45%, then that video frame is not designated as the target video frame. The specific value of the first preset threshold is set according to actual needs. Other video frames besides the target video frames are then identified.

[0094] Save the subtitle information corresponding to other video frames to the target video frame adjacent to those other video frames. For example, the first target video frame retained after the other video frames. Delete the other video frames. Generate descriptive information for the target video frame based on the subtitle information corresponding to it. For example, randomly collect 1-N target video frames to form a data entry, and use the corresponding subtitle information as the descriptive information after each target video frame. Images and text will appear alternately. Through the above operations, it is possible to achieve the following: If a subtitle reads, "I will give an example in the course, and I will assign 3 homework questions after class, due next week," and the video corresponding to this subtitle shows a speaker standing on a platform, not a blackboard or PPT, then this subtitle will be retained and merged with the subtitle information of a subsequent video frame, such as "We will introduce Monte Carlo trees in four parts, namely…".

[0095] The above process is as follows Figure 4 As shown, the text ratio of a video frame is determined. It is then checked whether the text ratio is greater than a corresponding threshold. If it is, the data frame is retained, and the proportion of the human image within the data frame is determined. If it is less than or equal to the corresponding threshold, the data frame is deleted, and the subtitle information corresponding to the deleted video frame is merged into the first retained video frame. Next, it is checked whether the proportion of the human image is greater than a corresponding threshold. If the proportion is greater, the data frame is deleted, and the subtitle information corresponding to the deleted video frame is merged into the first retained video frame. If the proportion is less than or equal to the corresponding threshold, the data frame is retained, and the retained video frames are compared and filtered.

[0096] In this embodiment, the proportion of target information in a video frame is determined, and the video frames are filtered based on the proportion and a first preset threshold. This retains video frames containing more target information while saving computational resources. Furthermore, descriptive information for the target video frames is generated based on subtitle information, facilitating the generation of a first subset of data where images and text alternate.

[0097] As an optional embodiment, step B5, "generating description information of the target video frame based on the subtitle information corresponding to the target video frame", includes steps C1 to C4.

[0098] Step C1: Determine the adjacent video frames corresponding to the target video frame.

[0099] Step C2: If the target video frame is different from the adjacent video frames, retain the target video frame.

[0100] Step C3: If the target video frame is the same as the adjacent video frame, save the subtitle information corresponding to the target video frame to the adjacent video frame and delete the target video frame.

[0101] Step C4: Generate description information for the target video frame based on the subtitle information corresponding to the retained target video frame.

[0102] Specifically, in this embodiment, the retained target video frames are compared and filtered to ensure that different target video frames are different images. If there are target video frames with the same image, one of the target video frames is deleted, and the subtitle information corresponding to the deleted target video frame is merged into the retained target video frames.

[0103] Identify the adjacent video frames corresponding to the target video frame. Compare the target video frame with the adjacent video frames to determine if their images are identical. If they are different, retain the target video frame; if they are identical, save the corresponding subtitle information to the adjacent video frame. For example, save the subtitle information of the target video frame to the first retained video frame. Delete the target video frame.

[0104] The descriptive information for each target video frame is generated based on the subtitle information corresponding to the retained target video frame. For example, 1-N target video frames are randomly collected to form a data entry, and the corresponding subtitle information is used as the descriptive information for each target video frame, with images and text appearing alternately.

[0105] The above process is as follows Figure 4 As shown, it determines whether a video frame is the same as an adjacent video frame. If they are not the same, the data frame is retained; if they are the same, the data frame is deleted, and the subtitle information corresponding to the deleted video frame is merged into the first retained video frame.

[0106] As an optional embodiment, the method includes steps D1 to D4 before step A1 "merging video frames with the same subtitle information".

[0107] Step D1: Obtain video data based on the initial data.

[0108] Step D2: Determine the first preset number of video scenes included in the video data based on the preset tool.

[0109] Step D3: Segment the video data according to the video scene to obtain the video segments corresponding to the video scene.

[0110] Step D4: Obtain video frames and corresponding subtitle information from the video clip.

[0111] Specifically, the first preset data type is, for example, video. The initial data corresponding to the first preset data type is video data. Therefore, video data can be obtained based on the initial data.

[0112] Preset tools, such as the open-source tool PySceneDetect, can detect video scenes contained in video data. Based on these tools, a first preset number of video scenes are determined from the video data. This first preset number represents one or more scenes, without a specific limit. Examples of video scenes include: a person explaining a problem on a blackboard, or a PowerPoint presentation remaining on the same page.

[0113] The video data is segmented according to the video scene to obtain video segments corresponding to the scene. In video data, a change in video scene generally means a change in the video's theme, such as explaining a problem or presenting a PowerPoint presentation. Therefore, in this embodiment, video segmentation is used to process videos with the same theme each time when generating subsets from the initial data. Video frames and their corresponding subtitle information are then obtained from the video segments. The above process is as follows: Figure 4 As shown, the video is segmented according to the scene.

[0114] As an optional embodiment, the above step A4, "according to logical relationships, concatenating video frames, new video frames, first explanatory information, and second explanatory information into reasoning process data", includes steps E1 to E6.

[0115] Step E1: Generate the start and end identifiers of the data sample.

[0116] Step E2: Determine the prompt words, the first question data, the questioner of the first question data, and the answerer of the data sample based on the first and second explanatory information.

[0117] Step E3 generates a new video frame and image start markers and image end markers for the video frame.

[0118] Step E4: Generate the image field based on the image start identifier, video frame, new video frame, and image end identifier.

[0119] Step E5: Based on the logical relationship, the video frame, the new video frame, and the image field are alternately concatenated to obtain the target field.

[0120] Step E6: Generate reasoning process data based on the start identifier, end identifier, prompt words, first question data, question object, answer object, target field, and preset separator identifier.

[0121] Specifically, in this embodiment, 1-N video frames are randomly collected to form a data sample. The subtitle information following each video frame in the data sample serves as the explanatory information for that video frame, with images and text appearing alternately. Each data sample is generated according to a preset prompt format. In this embodiment, the first and second explanatory information are collectively referred to as explanatory information, and new video frames and video frames are collectively referred to as video frames.

[0122] Preset prompt word format, for example:

[0123] <|begin_of_sentence|>SYSTEM_RPOMPT<|User|> IMAGEQUESTION<|Assistant|> <think> IMAGE ANALYSIS< / think> ANSWER<|end_of_sentence|>, where <|begin_of_sentence|> and <|end_of_sentence|> mark the beginning and end of a data sample; SYSTEM_RPOMPT is the system prompt word in the training data, indicating the type of task to the model, such as "code"; <|User|> and <|Assistant|> mark the questioner and responder, and only one pair will appear for a single-turn dialogue, while multiple pairs will appear multiple times for multi-turn dialogues or multiple rounds of questions; The symbols and mark the beginning and end of the image. In this embodiment, the image is encoded after input and then concatenated with the encoded text. IMAGE represents the image. <think>and< / think> The prompts appear in pairs, with the middle section being the ANALYSIS (analysis) section representing the thought and reasoning process, followed by the ANSWER (answer data), and the QUESTION (question data). Subsequent steps will organize the prompts in this manner for all generated data samples.

[0124] Generate start and end identifiers for the data samples. For example, the start identifier is <|begin_of_sentence|>, and the end identifier is <|end_of_sentence|>.

[0125] Based on the explanatory information, determine the prompt words, first question data, the questioner of the first question data, the answerer, and the reasoning process data for the data sample. The prompt words are SYSTEM_RPOMPT as mentioned above, such as code, exercises, etc. The prompt words can be directly determined from the explanatory information or the video title. If the explanatory information includes explanations of code, the prompt word is "code"; if the video title is "How to handle a certain type of exercise," the prompt word is "exercise." The first question data is the QUESTION. For example, if the explanatory information includes "How to handle a certain type of exercise," use it as the first question data; if the video title is "How to use a certain tool," use it as the first question data. The questioner of the first question data is <|User|>, and the answerer is <|Assistant|>. Generally, <|User|> is the questioner, and <|Assistant|> is the model-generated answer. For multi-turn dialogues, representing the questioner and answerer, based on the existing explanatory information and data annotation operations, the questioner and answerer can be directly determined.

[0126] The reasoning process data is ANALYSIS, which is obtained from explanatory information, such as: identifying the object of force analysis, generating a force analysis diagram; establishing a coordinate system; listing equilibrium or dynamic equations; solving the equations and analyzing the results.

[0127] Generate the start and end markers for the video frame. For example, the start marker is... The image end marker is [image name]. The new video frame and the previous video frame are each used as an image (IMAGE).

[0128] Generate an image field based on the image start marker, video frame, new video frame, and image end marker. The image field is as follows: IMAGE. Based on logical relationships, video frames, new video frames, and image fields are alternately concatenated to obtain the target field. The target field is, for example: <think> IMAGE ANALYSIS IMAGE ANALYSIS< / think> .

[0129] Preset separators, for example: <think>and< / think> Based on the aforementioned start identifier, end identifier, prompt words, first question data, questioner, answerer, reasoning process data, image start identifier, image end identifier, video frame, new video frame, and preset separator, reasoning process data is generated, with each set of reasoning process data serving as a data sample. A first subset is generated based on these data samples, containing multiple data samples. For example, a data sample could be: <|begin_of_sentence|>Physics<|User|> The image X in the question is shown in Figure X. An object of mass m is placed on M on a horizontal surface. If m slides down M at a constant speed while M remains stationary, what are the magnitudes of the pressure and frictional forces exerted by M on the ground? <think>First, use the isolation method to analyze, first isolate m, m is subject to gravity mg, the support force of the slope on it …, as shown in the force analysis diagram Y Force analysis diagram Y because m slides along the slope at a constant speed, from the balance condition, …, then isolate M, M is subject to vertical downward gravity Mg, the ground to it …, build system as shown in Figure Z, Figure Z from Newton's third law, so the ground has no friction on M< / think> Therefore, the pressure of M on the ground is equal to (M+m)g, and the frictional force on the ground is 0.

[0130] In this embodiment, a data sample is generated based on a start identifier, an end identifier, a prompt word, first question data, the questioner, the answerer, reasoning process data, an image start identifier, an image end identifier, video frames, new video frames, and a preset separator. The CoT reasoning process of the data sample includes both text and image data. The subset generated from this data sample contains multimodal data. Training the model using this subset allows images to assist the model's thinking and enhances the interpretability of the model's thought process.

[0131] As an optional embodiment, step S303 above includes steps F1 to F4.

[0132] Step F1: If the preset data type is the second preset data type, generate the label information of the intermediate questions. The intermediate questions are obtained from the initial data. The intermediate questions include the first answer data, reasoning step data and logical relationship. The first answer data includes image information.

[0133] Step F2: If the intermediate question contains a second preset number of image information, move the image information to the first preset position of the intermediate question to obtain the target question. The first preset position is in the reasoning process data.

[0134] Step F3: If the number of image information contained in the intermediate question is greater than the second preset number, move the image information to the second preset position of the intermediate question to obtain the target question. The second preset position is in the reasoning process data.

[0135] Step F4: Based on the logical relationships in the target question, the reasoning step data and image information in the target question are combined into reasoning process data, and a second subset is generated based on the reasoning process data.

[0136] Specifically, the second preset data type is a problem set, and the initial data corresponding to this second preset data type is the problem set data. The problem set data contains multiple problems, which can be elementary school word problems, math problems, physics problems, geography problems, etc. Each problem includes a question, first answer data, reasoning steps, and logical relationships. Problems containing image information in the first answer data are extracted from the problem set data as intermediate problems. The image information represents an image. The process of extracting intermediate problems is as follows: if the problem set data is saved in a PDF file, the Mathpix tool is used to extract the problems contained in that file, and from this, the intermediate problems that meet the requirements are determined. Mathpix is ​​an application that can extract mathematical formulas from images and convert them into an editable format.

[0137] A large language model (LLM) is used to generate labels for intermediate questions. An example large language model is GPT-3.5. The process involves inputting the intermediate questions into the GPT-3.5 model, which then generates labels such as code or exercises. These labels are then placed in the SYSTEM_RPOMPT location within the data sample. The target questions and labels are included in the text and image data.

[0138] The second preset quantity is, for example, 1, 2, or other values ​​that meet actual needs. If the intermediate question contains the second preset quantity of image information, the image information is moved to the first preset position of the intermediate question to obtain the target question. The first preset position is, for example, at the very beginning of the reasoning process data, which is also the very beginning of the intermediate question.

[0139] If the number of images in the intermediate question exceeds the second preset number, the image information is moved to the second preset position in the intermediate question to obtain the target question. The second preset position is, for example, before the text that first mentions the image information in the reasoning process data. The text that first mentions the image information is, for example, if the intermediate question contains "Add auxiliary lines between points A and C, as shown in Figure T," then "Add auxiliary lines between points A and C, as shown in Figure T" is the first text mentioning Figure T. The target question and label information are contained within the image and text data.

[0140] Based on the logical relationships in the target question, the reasoning step data and image information in the target question are concatenated to form reasoning process data. In the reasoning process data, reasoning step data and image information appear alternately, and the order in which different reasoning step data appear satisfies the logical relationship. Each set of reasoning process data is treated as a data sample. A second subset is generated based on these data samples. For example, multiple data samples are generated from the target question and tag information according to the aforementioned preset prompt word format, and a second subset is generated based on these data samples. The process of generating data samples is described in the above embodiment and will not be repeated here.

[0141] In this implementation, the position of the image information in the intermediate questions is adjusted to obtain the target questions. Based on the target questions and tag information, a second subset of data, consisting of alternating images and text, is generated. Furthermore, the size of the second subset of data and the domains covered by the second subset of data are expanded using the exercise set data.

[0142] As an optional embodiment, the above step F2 "generating label information for intermediate questions" includes steps G1 to G6.

[0143] Step G1: Obtain the second question data, second answer data, and image information contained in the intermediate question.

[0144] Step G2: Determine the deviation rate of the second question data, the second answer data, and the image information.

[0145] Step G3: Select intermediate questions with a deviation rate greater than the second preset threshold as questions to be adjusted, and generate target answer data for the questions to be adjusted.

[0146] Step G4: Generate candidate questions based on the questions to be adjusted and the target answer data.

[0147] Step G5: Select intermediate questions with a deviation rate less than or equal to the second preset threshold as candidate questions.

[0148] Step G6: Generate tag information for candidate questions, where candidate questions are included in intermediate questions.

[0149] Specifically, each intermediate question includes second question data, solution process, second answer data, and image information. The process involves retrieving the second question data, second answer data, and image information contained in each intermediate question.

[0150] The sampling inspection checks whether the second question data, second answer data, and image information in the intermediate questions are consistent. The deviation rate of the second question data, second answer data, and image information is determined. Intermediate questions with a deviation rate less than or equal to a second preset threshold are all retained; for intermediate questions with a deviation rate greater than the second preset threshold, answer data needs to be regenerated and adjusted. The second preset threshold is, for example, 20%, 30%, or other values ​​that meet practical needs. The process of calculating the deviation rate is as follows: input the second question data and image information into the large language model, use the large language model to generate reference answer data, and calculate the deviation rate between the reference answer data and the second answer data.

[0151] Intermediate questions with a deviation rate greater than a second preset threshold are designated as questions to be adjusted, and target answer data is generated for these questions. For example, the second question data and image information of intermediate questions are input into a large language model (e.g., Deepseek-R1 model) to generate target answer data. The target answer data is then used to replace the second answer data in the questions to be adjusted, generating candidate questions. Intermediate questions with a deviation rate less than or equal to the second preset threshold are designated as candidate questions.

[0152] Tag information, such as code, exercises, etc. Based on the content of the candidate questions, tag information is generated for each candidate question. For example, if a candidate question contains a lot of code, the tag information for that candidate question is "code"; if a candidate question contains a large amount of data calculation formulas, the tag information for that candidate question is "exercise".

[0153] As an optional embodiment, the above step G6 "generating tag information for candidate questions" includes steps H1 to H4.

[0154] Step H1: Obtain the second question data and the data length of the second question data for the candidate questions;

[0155] Step H2: Select candidate questions with the same data length as the second question data as the questions to be screened;

[0156] Step H3: Retain the questions whose second answer data length is greater than the third preset threshold, and obtain the retained questions to be screened. The retained questions to be screened are included in the candidate questions.

[0157] Step H4: Generate tag information for the retained questions to be screened based on the preset model and the retained questions to be screened.

[0158] Specifically, based on the question data, duplicate candidate questions are deduplicated, retaining those with longer answer data. The question data within the candidate questions is then obtained, which is the second question data, and its length is determined.

[0159] The length of the second question data is used to determine if there are candidate questions with the same second question data. Candidate questions with the same second question data are selected as questions to be filtered. A third preset threshold is determined, for example, the third preset threshold is set to 20 characters or 30 characters, or the length of each second answer data is determined, the lengths are sorted and the second largest length is selected as the third preset threshold. Questions with a second answer data length greater than the third preset threshold are retained to be filtered, and the retained questions are included in the candidate questions.

[0160] Pre-set models include, for example, large language models such as GPT-3.5. These large language models are used to generate label information for the retained questions to be selected. The process of generating label information is as follows: the retained questions are input into the GPT-3.5 model, which generates label information such as code, exercises, etc. When assembling data samples, the label information is placed in the SYSTEM_RPOMPT location of the data samples.

[0161] As an optional embodiment, step S303 above includes steps J1 to J3.

[0162] Step J1: If the preset data type is the third preset data type and the third answer data of the preset dataset contains image information, the preset dataset is used as the intermediate dataset. The preset dataset is obtained based on the initial data, and the intermediate dataset contains logical relationships, reasoning step data, and third answer data.

[0163] In step J2, if the third answer data does not contain image information, and the third question data of the preset dataset contains image information, and if the preset dataset contains data corresponding to the preset task type, the image information in the preset dataset is moved to the third preset position to obtain the intermediate dataset, wherein the third preset position is in the reasoning process data.

[0164] Step J3: Based on the logical relationships in the intermediate dataset, the reasoning step data and image information in the intermediate dataset are concatenated into reasoning process data, and a third sub-dataset is generated based on the reasoning process data.

[0165] Specifically, the third preset data type is, for example, an existing multimodal dataset. The initial data corresponding to the third preset data type is the data contained in the existing multimodal dataset. Existing multimodal datasets include, for example, the Math-V dataset, the CoCoMMR dataset, the SAT dataset, and the AMC dataset. Image information represents images, and containing image information means containing images. The third preset quantity indicates multiple values; no specific quantity limit is specified here.

[0166] Existing multimodal datasets can be categorized as follows: datasets containing images in the answer data, datasets containing images only in the question data, and datasets without images. This embodiment retains datasets that already contain images in the answer data; datasets containing images only in the question data require further filtering; and datasets without images are directly deleted.

[0167] Therefore, if the third answer data in the preset dataset contains image information, then the preset dataset is the dataset that contains images in the answer data, and the preset dataset is used as the intermediate dataset. The intermediate dataset contains logical relationships, reasoning step data, and the third answer data.

[0168] If the third answer data does not contain image information, and the third question data in the preset dataset contains image information, then the preset dataset is a dataset that only contains images in the question data, and further filtering of the preset dataset is required. Determine whether the preset dataset contains data corresponding to a preset task type, such as tasks related to mathematics, physics, geography, or coding. If the preset dataset contains data corresponding to a preset task type, first retain the text portion of the preset dataset. Then move the image information in the preset dataset to a third preset position to obtain an intermediate dataset. The third preset position is, for example, at the very beginning of the CoT thinking process in the reasoning process data, i.e., at the very beginning of all reasoning step data. After moving the image information to the third preset position, connect the image information to the CoT text body in the preset dataset as shown in the figure.

[0169] Based on the logical relationships in the intermediate dataset, the inference step data and image information in the intermediate dataset are concatenated to form inference process data. Inference process data, inference step data and image information appear alternately, and the order in which different inference step data appear satisfies logical relationships. Each set of inference process data is treated as a data sample, and a third subset is generated based on these data samples. For example, multiple data samples are generated from the data in the intermediate dataset according to the aforementioned preset prompt word format, and a third subset is generated based on these data samples. The process of generating data samples is described in the above embodiment and will not be repeated here.

[0170] In this implementation, a third subset of the dataset is created using an existing multimodal dataset, balancing dataset quality and computational efficiency during its construction. Furthermore, the size and scope of the third subset are expanded, and prompts from the dataset are integrated. Training the model using this third subset improves both the model's problem-solving accuracy and the interpretability of its thought process.

[0171] As an optional embodiment, step S303 above includes steps S3031 to S3033.

[0172] Step S3031: When the preset data type is the fourth preset data type, generate explanation information for the step diagram based on the fourth question data and the step diagram. The fourth question data and the step diagram are obtained from the initial data. The explanation information includes reasoning step data, and the step diagram includes image information.

[0173] Step S3032: Based on the step diagram, determine the logical relationship of the reasoning step data in the explanation information.

[0174] Step S3033: Based on the logical relationship, the step diagram and explanation information are combined into reasoning process data, and a fourth subset is generated based on the reasoning process data.

[0175] Specifically, the fourth preset data type is, for example, a game-related dataset. The initial data corresponding to this fourth preset data type is data from the game-related dataset. The games included in this dataset mainly include maze games, board games, and game records, and this dataset contains step diagrams of the game's moves. The step diagrams and fourth question data are obtained from the initial data corresponding to the fourth preset data type. Examples of the fourth question data include: "How to get from the entrance to the exit," "How to checkmate within ten moves," and "Which side, Black or White, can win in this area?"

[0176] The fourth question data and the step diagram are combined and iteratively fed into a multimodal large language model (MLLM) to generate explanatory information for each step diagram. The explanatory information might be: first go left to the first fork, then go up. The fourth question data, the step diagram, and the explanatory information are contained within the text and image data.

[0177] Based on the step diagram, determine the logical relationship between the reasoning step data in the explanation information. For example, reasoning step x2 in the explanation information is derived from the result of reasoning step x1, and reasoning step x3 is derived from the result of reasoning step x2.

[0178] Based on logical relationships, the step diagrams and explanation information are concatenated into reasoning process data. In this data, reasoning steps and image information appear alternately, and the order in which different reasoning steps appear satisfies logical relationships. Each set of reasoning process data is treated as a data sample. A fourth subset is generated based on these data samples. For example, multiple data samples are generated from the fourth question data, step diagrams, and explanation information according to the aforementioned preset prompts. The fourth subset is then generated based on these data samples. The process of generating data samples is described in the above embodiment and will not be repeated here.

[0179] In this implementation, a fourth subset of the dataset is created using game-related datasets, balancing dataset quality and computational efficiency during its construction. Furthermore, the size and scope of the fourth subset are expanded, and hints from the dataset are integrated. Training the model using this fourth subset improves both the model's problem-solving accuracy and the interpretability of its thought process.

[0180] Step S304: Mix the subset datasets to generate the first target dataset.

[0181] For details on how to implement this step, please refer to [link / reference]. Figure 1 Step S104 of the illustrated embodiment will not be described again here.

[0182] Step S305: Obtain the target subset from the subset dataset, and mix the target subset according to a preset ratio to generate the second target dataset.

[0183] For details on how to implement this step, please refer to [link / reference]. Figure 1 Step S105 of the illustrated embodiment will not be described again here.

[0184] The dataset creation method provided in this embodiment obtains initial data corresponding to preset data types; processes the initial data of different preset data types to obtain image and text data; generates sub-datasets corresponding to preset data types based on the image and text data; and selects several sub-datasets from multiple sub-datasets to generate a target dataset. This method can construct a target dataset with image and text data during the reasoning process. The images in the target dataset not only appear in the question but also possess multimodal information. Using the target dataset to train the model allows images to assist the model's thinking and enhances the interpretability of the model's thinking process, enabling the model to alternately generate images and text during CoT inference. This solves the problem that most datasets are primarily single-text modalities, with images only appearing in the question, making it difficult to use image information to assist the model's thinking and resulting in poor interpretability of the model's thinking process.

[0185] As an optional embodiment, the target information is text information, and the above step B1 "obtaining the information ratio of the target information in the video frame" includes steps K1 to K4.

[0186] Step K1: Stack several video frames according to the channel dimension to obtain the first intermediate object to be identified; the number of channels of the first intermediate object to be identified is the sum of the number of channels of the video frames.

[0187] Specifically, a number of video frames can include 4, 5, or other quantities. Stacking by channel dimension means overlapping the video frames to obtain a first intermediate object to be identified with a larger number of channels. The number of channels in the first intermediate object can be equal to the sum of the channels of the video frames. Instead of changing the pixel values, multiple values ​​can be placed into multiple channels. The text region recognition model can determine the text region by analyzing data from more channels. For example, stacking three images of size 224*224*3 yields a first intermediate object of size 224*224*9. Here, 224 represents the number of pixels in the image, and 3 and 9 represent the number of channels.

[0188] Step K2: Use a text detection model to identify the text region of the first intermediate object to be identified, and determine the text region of the first intermediate object to be identified that contains text information.

[0189] Specifically, the text detection model can be a machine model used to detect text regions contained in an image, or a neural network model pre-trained based on training samples. Stacking video frames according to channel dimensions can enhance the information in the image. For example, assuming the image is a 3-channel image, without stacking, the text detection model can use the 3-channel data to determine the text region; however, after stacking according to the method in this embodiment, the text detection model can use the 9-channel data to determine the text region. This allows for the use of more data, which helps to strengthen the text region, reduce the influence of noise, and enables the text detection model to more accurately identify the text regions containing text information in the image.

[0190] Step K3: Based on the location information of the text region, determine the text image corresponding to the text region in the video frame.

[0191] Specifically, a text image can represent image information of a text region within an image; it can be an image region containing text information or image data. The location information of the text region can represent the location information of the region containing text within the first intermediate object to be identified. If the video frame and the first intermediate object to be identified have the same size, the reference point used to represent the location information of the first intermediate object to be identified can be the same as the reference point used to represent the location information of the video frame, such as the lower left corner or lower right corner of the image. This allows the text image corresponding to the text region in the video frame to be determined based on the reference point and location information. If the coordinates or size of the video frame and the first intermediate object to be identified are different, the video frame can be transformed to obtain an image with the same coordinates or size as the first intermediate object to be identified, and then the text image in the video frame can be determined. Alternatively, the location information of the text region can be transformed according to the position or size transformation relationship between the video frame and the first intermediate object to be identified, and the transformed location information can be used to determine the text image corresponding to the text region in the video frame.

[0192] Step K4: Use the ratio of the text image area to the video frame area as the information ratio.

[0193] In this embodiment, multiple frames of images of the object to be identified are stacked according to the channel dimension to obtain a first intermediate object to be identified. A text detection model is then used to identify the first intermediate object to be identified. Since multiple frames of images are stacked according to the channel dimension, the features of the text region in the image can be enhanced, enabling the text detection model to more accurately identify the text region in the object to be identified. This effectively avoids the omission or misidentification of text regions and ensures that a more accurate information ratio is obtained.

[0194] This embodiment also provides a dataset creation apparatus for implementing the above embodiments and preferred embodiments; details already described will not be repeated. As used below, the term "module" can refer to a combination of software and / or hardware that performs a predetermined function. Although the apparatus described in the following embodiments is preferably implemented in software, hardware implementation, or a combination of software and hardware, is also possible and contemplated.

[0195] This embodiment provides a dataset creation device, such as... Figure 5 As shown, it includes:

[0196] The first data acquisition module 501 is used to acquire initial data corresponding to a preset data type, wherein the initial data includes multiple different types of data;

[0197] The second data acquisition module 502 is used to acquire inference step data, logical relationships of inference step data, and image information corresponding to inference step data from the initial data;

[0198] The first dataset generation module 503 is used to concatenate reasoning step data and image information into reasoning process data according to logical relationships, and generate sub-datasets corresponding to preset data types based on the reasoning process data.

[0199] The second dataset generation module 504 is used to mix the sub-datasets to generate the first target dataset;

[0200] The third dataset generation module 505 is used to obtain the target subset from the subset and mix the target subset according to a preset ratio to generate the second target dataset.

[0201] Further functional descriptions of the above modules are the same as those in the corresponding embodiments described above, and will not be repeated here.

[0202] In this embodiment, the dataset creation device is presented in the form of a functional unit. Here, a unit refers to an ASIC (Application Specific Integrated Circuit) circuit, a processor and memory that execute one or more software or fixed programs, and / or other devices that can provide the above functions.

[0203] This application also provides a computer device having the above-described features. Figure 5 The dataset creation device shown.

[0204] Please see Figure 6 , Figure 6 This is a schematic diagram of the structure of a computer device provided in an optional embodiment of this application, such as... Figure 6 As shown, the computer device includes one or more processors 10, memory 20, and interfaces for connecting the components, including high-speed interfaces and low-speed interfaces. The components communicate with each other via different buses and can be mounted on a common motherboard or otherwise installed as needed. The processors can process instructions executed within the computer device, including instructions stored in or on memory to display graphical information of a GUI on external input / output devices (such as display devices coupled to the interfaces). In some alternative implementations, multiple processors and / or multiple buses can be used with multiple memories and multiple memory modules, if desired. Similarly, multiple computer devices can be connected, each providing some of the necessary operations (e.g., as a server array, a group of blade servers, or a multiprocessor system). Figure 6 Take a processor 10 as an example.

[0205] Processor 10 may be a central processing unit, a network processor, or a combination thereof. Processor 10 may further include an integrated circuit, a programmable logic device, or a combination thereof. The programmable logic device may be a complex programmable logic device, a field-programmable gate array (FPGA), a general-purpose array logic (GPA), or any combination thereof.

[0206] The memory 20 stores instructions executable by at least one processor 10 to cause at least one processor 10 to perform the method shown in the above embodiments.

[0207] The memory 20 may include a program storage area and a data storage area. The program storage area may store the operating system and applications required for at least one function; the data storage area may store data created based on the use of the computer device. Furthermore, the memory 20 may include high-speed random access memory and may also include non-transitory memory, such as at least one disk storage device, flash memory device, or other non-transitory solid-state storage device. In some alternative embodiments, the memory 20 may optionally include memory remotely located relative to the processor 10, and these remote memories may be connected to the computer device via a network. Examples of such networks include, but are not limited to, the Internet, intranets, local area networks, mobile communication networks, and combinations thereof.

[0208] The memory 20 may include volatile memory, such as random access memory; the memory may also include non-volatile memory, such as flash memory, hard disk or solid-state drive; the memory 20 may also include a combination of the above types of memory.

[0209] The computer device also includes a communication interface 30 for communicating with other devices or communication networks.

[0210] This application also provides a computer-readable storage medium. The methods described in this application can be implemented in hardware or firmware, or implemented as recordable on a storage medium, or implemented as computer code downloaded over a network and originally stored on a remote storage medium or a non-transitory machine-readable storage medium and subsequently stored on a local storage medium. Thus, the methods described herein can be processed by software stored on a storage medium using a general-purpose computer, a dedicated processor, or programmable or dedicated hardware. The storage medium can be a magnetic disk, optical disk, read-only memory, random access memory, flash memory, hard disk, or solid-state drive, etc.; further, the storage medium can also include combinations of the above types of memory. It is understood that computers, processors, microprocessor controllers, or programmable hardware include storage components capable of storing or receiving software or computer code. When the software or computer code is accessed and executed by the computer, processor, or hardware, the methods shown in the above embodiments are implemented.

[0211] A portion of this application can be applied as a computer program product, such as computer program instructions, which, when executed by a computer, can invoke or provide the methods and / or technical solutions according to this application through the operation of the computer. Those skilled in the art will understand that the forms in which computer program instructions exist in a computer-readable medium include, but are not limited to, source files, executable files, installation package files, etc. Correspondingly, the ways in which computer program instructions are executed by a computer include, but are not limited to: the computer directly executing the instructions, or the computer compiling the instructions and then executing the corresponding compiled program, or the computer reading and executing the instructions, or the computer reading and installing the instructions and then executing the corresponding installed program. Here, the computer-readable medium can be any available computer-readable storage medium or communication medium accessible to a computer.

[0212] Although embodiments of this application have been described in conjunction with the accompanying drawings, those skilled in the art can make various modifications and variations without departing from the spirit and scope of this application, and such modifications and variations all fall within the scope defined by this application.

Claims

1. A method for creating a dataset, characterized in that, The method includes: Obtain initial data corresponding to a preset data type, wherein the initial data includes multiple different types of data; The inference step data, the logical relationship of the inference step data, and the image information corresponding to the inference step data are obtained from the initial data. Based on the logical relationship, the reasoning step data and the image information are concatenated into reasoning process data, and a sub-dataset corresponding to the preset data type is generated based on the reasoning process data; The subset datasets are combined to generate the first target dataset; Obtain a target subset from the subset, and mix the target subset according to a preset ratio to generate a second target dataset.

2. The method according to claim 1, characterized in that, The step of concatenating the reasoning step data and the image information into reasoning process data according to the logical relationship, and generating a sub-dataset corresponding to the preset data type based on the reasoning process data, includes: When the preset data type is the first preset data type, video frames with the same subtitle information are merged, and the merged video frame is used as a new video frame. The video frame and the subtitle information corresponding to the video frame are obtained from the initial data. The video frame contains the image information, and the logical relationship exists between the video frames. Generate first descriptive information for the video frame and second descriptive information for the new video frame based on the subtitle information; Based on the logical relationship, the video frame, the new video frame, the first explanatory information, and the second explanatory information are concatenated to form the reasoning process data; Based on the reasoning process data, a first subset of data is generated.

3. The method according to claim 2, characterized in that, First descriptive information for the video frame is generated based on the subtitle information, including: The proportion of target information in the video frame is obtained; Video frames with an information ratio greater than a first preset threshold are taken as target video frames, and other video frames other than the target video frames are determined. Save the subtitle information corresponding to the other video frames to the target video frame adjacent to the other video frames, and delete the other video frames; The first description information of the target video frame is generated based on the subtitle information corresponding to the target video frame.

4. The method according to claim 3, characterized in that, The step of generating the first descriptive information of the target video frame based on the subtitle information corresponding to the target video frame includes: Determine the adjacent video frames corresponding to the target video frame; If the target video frame is different from the adjacent video frames, the target video frame is retained. If the target video frame is the same as the adjacent video frame, the subtitle information corresponding to the target video frame is saved to the adjacent video frame, and the target video frame is deleted. The first description information of the target video frame is generated based on the subtitle information corresponding to the retained target video frame.

5. The method according to claim 2, characterized in that, Before merging video frames with identical subtitle information, the method further includes: Video data is obtained based on the initial data; The video data contains a first preset number of video scenes, as determined by a preset tool. The video data is segmented according to the video scene to obtain video segments corresponding to the video scene; The video frame and the corresponding subtitle information are obtained from the video segment.

6. The method according to claim 2, characterized in that, The step of concatenating the video frame, the new video frame, the first explanatory information, and the second explanatory information into the reasoning process data according to the logical relationship includes: The start and end identifiers for generating data samples; Based on the first explanatory information and the second explanatory information, determine the prompt words, the first question data, the question object of the first question data, and the answer object of the data sample; Generate the new video frame and the image start identifier and image end identifier of the video frame; Generate an image field based on the image start identifier, the video frame, the new video frame, and the image end identifier; Based on the logical relationship, the video frame, the new video frame, and the image field are alternately concatenated together to obtain the target field; The reasoning process data is generated based on the start identifier, the end identifier, the prompt word, the first question data, the question object, the answer object, the target field, and the preset separator identifier.

7. The method according to claim 1, characterized in that, The step of obtaining the initial data corresponding to the preset data type includes: Retrieve initial data corresponding to the first preset data type from the preset database based on preset keywords; Obtain a preset exercise set, and use the data in the preset exercise set as the initial data corresponding to the second preset data type; Obtain a preset dataset and use the data in the preset dataset as the initial data corresponding to the third preset data type; Obtain a game dataset related to spatial location, and use the data in the game dataset as the initial data corresponding to the fourth preset data type.

8. The method according to claim 1, characterized in that, The step of concatenating the reasoning step data and the image information into reasoning process data according to the logical relationship, and generating a sub-dataset corresponding to the preset data type based on the reasoning process data, includes: When the preset data type is the second preset data type, tag information for intermediate questions is generated. The intermediate questions are obtained from the initial data. The intermediate questions include first answer data, reasoning step data, and logical relationships. The first answer data includes the image information. If the intermediate question contains a second preset number of image information, the image information is moved to a first preset position of the intermediate question to obtain the target question, wherein the first preset position is in the reasoning process data; If the number of image information contained in the intermediate question is greater than the second preset number, the image information is moved to the second preset position of the intermediate question to obtain the target question, wherein the second preset position is in the reasoning process data; Based on the logical relationship in the target question, the reasoning step data and the image information in the target question are concatenated to form the reasoning process data, and a second sub-dataset is generated based on the reasoning process data.

9. The method according to claim 8, characterized in that, The tag information for generating intermediate questions includes: Obtain the second question data, second answer data, and image information contained in the intermediate question; Determine the deviation rate of the second question data, the second answer data, and the image information; Intermediate questions with a deviation rate greater than a second preset threshold are identified as questions to be adjusted, and target answer data for the questions to be adjusted is generated. Based on the questions to be adjusted and the target answer data, candidate questions are generated; The intermediate questions with a deviation rate less than or equal to the second preset threshold are selected as the candidate questions. Generate the tag information of the candidate questions, wherein the candidate questions are included in the intermediate questions.

10. The method according to claim 9, characterized in that, The tag information for generating the candidate questions includes: Obtain the second question data and the data length of the second question data for the candidate questions; Candidate questions with the same data length as the second question data will be selected as the questions to be screened. The remaining questions are selected from those whose second answer data length is greater than a third preset threshold, and the remaining questions are included in the candidate questions. Based on the preset model and the retained questions to be screened, the tag information of the retained questions to be screened is generated.

11. The method according to claim 1, characterized in that, The step of concatenating the reasoning step data and the image information into reasoning process data according to the logical relationship, and generating a sub-dataset corresponding to the preset data type based on the reasoning process data, includes: When the preset data type is the third preset data type, and when the image information is included in the third answer data of the preset dataset, the preset dataset is used as an intermediate dataset. The preset dataset is obtained based on the initial data, and the intermediate dataset includes the logical relationship, the reasoning step data, and the third answer data. If the third answer data does not contain the image information, and the third question data of the preset dataset contains the image information, if the preset dataset contains data corresponding to the preset task type, the image information in the preset dataset is moved to a third preset position to obtain the intermediate dataset, wherein the third preset position is in the reasoning process data; Based on the logical relationships in the intermediate dataset, the reasoning step data and the image information in the intermediate dataset are concatenated to form the reasoning process data, and a third sub-dataset is generated based on the reasoning process data.

12. The method according to claim 1, characterized in that, The step of concatenating the reasoning step data and the image information into reasoning process data according to the logical relationship, and generating a sub-dataset corresponding to the preset data type based on the reasoning process data, includes: When the preset data type is the fourth preset data type, the explanation information of the step diagram is generated based on the fourth question data and the step diagram. The fourth question data and the step diagram are obtained from the initial data. The explanation information includes the reasoning step data, and the step diagram includes the image information. Based on the step diagram, determine the logical relationship of the reasoning step data in the explanation information; Based on the logical relationship, the step diagram and the explanation information are concatenated to form the reasoning process data, and a fourth sub-dataset is generated based on the reasoning process data.

13. A computer device, characterized in that, include: A memory and a processor are communicatively connected, the memory storing computer instructions, and the processor executing the computer instructions to perform the dataset creation method of any one of claims 1 to 12.

14. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer instructions for causing a computer to perform the dataset creation method according to any one of claims 1 to 12.

15. A computer program product, characterized in that, Includes computer instructions for causing a computer to perform the dataset creation method according to any one of claims 1 to 12.

Images