Multi-modal data set construction method, electronic equipment and storage medium

Abstract

The invention provides a multi-modal data set construction method, electronic equipment and a storage medium. The method comprises the following steps: determining an image insertion position according to question and answer statements in a text corpus; determining a retrieval statement according to a target statement where the image insertion position is located; acquiring a plurality of candidate images from an image search engine by utilizing the retrieval statement; screening the plurality of candidate images to obtain a target image which is most matched with the target statement; generating a descriptive abstract of the target image; fusing the descriptive abstract with the target statement to obtain a rewritten statement; and replacing the target statement with the rewritten statement and the target image. According to the technical scheme, the purpose of inserting the image into the question and answer statement in the text corpus according to the context requirement is achieved, the inserted image truly plays an irreplaceable semantic supplement role, and a high-quality training sample is provided for training a multi-modal task model.

Description

Technical Field

[0001] This application relates to the field of data processing technology, specifically to a method for constructing a multimodal dataset, an electronic device, and a storage medium. Background Technology

[0002] With the rapid development of artificial intelligence technology, multimodal research has become one of the important directions in the field of natural language processing. Currently, multimodal dialogue systems mainly pursue two core goals: (1) fully understanding the multimodal information provided by the user; and (2) actively generating or supplementing multimodal information to improve the semantic richness of the response content. However, although large-scale pre-trained models have made significant progress in understanding the image and text content input by users in recent years, such as some studies attempting to convert image content into semantic representations that can be parsed by language models, research on the system's ability to actively provide multimodal information is still relatively scarce.

[0003] Traditional multimodal dialogue systems often rely on explicit user requests, such as "Please show an image," before returning an image. This passive response ignores potential semantic gaps and visual supplementation during dialogue, making it difficult to meet users' information needs in complex contexts. Some research has attempted to improve the multimodal generation capabilities of models by constructing image-text dialogue datasets. Although models can generate responses based on existing images, the images are usually accompanied by detailed text descriptions, meaning that even after removing the images, the system can still maintain basic dialogue effectiveness. This means that image content often exists only as decorative information, without significantly enhancing semantic expression. This phenomenon raises a crucial question: In current multimodal tasks, does image information truly play an irreplaceable role in semantic supplementation? Are images merely redundant supplementary information, or do they provide deeper interpretation and perceptual capabilities in specific contexts? To explore the true value of multimodal information, current research proposes a new direction: allowing models to autonomously determine whether to introduce image information without explicit user requests, and proactively select appropriate images to supplement the semantics of the response based on the dialogue context and question requirements. This research direction not only helps to reveal the true role of multimodal information in dialogue context, but also promotes the evolution of intelligent dialogue systems from passive response to active perception and semantic reasoning.

[0004] In view of the above, this application is hereby submitted. Summary of the Invention

[0005] This application aims to provide a method for constructing a multimodal dataset, an electronic device, and a storage medium. It realizes the purpose of inserting images into question-and-answer statements in a text corpus according to the needs of the context, and makes the inserted images play an irreplaceable semantic supplementary role, providing a deeper level of interpretation and perception in a specific context, rather than existing as redundant additional information. It achieves the effect of deep image-text fusion and provides high-quality training samples for training multimodal task models.

[0006] In a first aspect, embodiments of this application provide a method for constructing a multimodal dataset, including: The image insertion position is determined based on the question-and-answer statements in the text corpus; The search statement is determined based on the target statement where the image is inserted; The search query is used to retrieve multiple candidate images from the image search engine; The candidate images are filtered to obtain the target image that best matches the target statement; Generate a descriptive summary of the target image; The descriptive summary is then merged with the target statement to obtain the rewritten statement; The target statement is replaced using the rewritten statement and the target image.

[0007] According to the technical solution provided in the embodiments of this application, optionally, determining the image insertion position based on question-and-answer statements in a text corpus includes: Each set of question-and-answer statements is divided into multiple text segments; A reference value is determined to characterize the multimodal expressive power of each text segment. The larger the reference value, the stronger the multimodal expressive power of the corresponding text segment. The reference values ​​are sorted in descending order, and the image insertion position is determined based on the text fragments corresponding to the first n reference values.

[0008] According to the technical solution provided in the embodiments of this application, optionally, determining the reference value characterizing the multimodal expressive power of each text segment includes: The probability assessment result of determining the current text segment becoming a potential important word; For each text segment, keywords are extracted and context simulation is performed to generate multiple continuation texts, and the embedding vectors of the multiple continuation texts are determined; the similarity between each keyword and the corresponding embedding vector is calculated respectively; and the normalized summary weight of the current text segment is determined based on the similarity between each keyword and the corresponding embedding vector. Determine the context score of the current text segment. The higher the context score, the stronger the connection between the current text segment and its context. The reference value for the multimodal expressive power of the current text segment is determined based on the probability assessment result of the current text segment becoming a potential important word, the normalized summary weight of the current text segment, and the context score of the current text segment.

[0009] According to the technical solution provided in the embodiments of this application, optionally, the determination of the probability assessment result of the current text segment becoming a potential important word includes: The importance ranking order of each text segment is determined by a large language model; The probability assessment result of the current text segment becoming a potential important word is determined based on the total number of text segments and the importance ranking order of each text segment.

[0010] According to the technical solution provided in the embodiments of this application, optionally, determining the context score of the current text segment includes: A first set of segments that have a contextual relationship with the current text segment is determined from each of the text segments according to semantic and / or temporal relationships; Calculate the cosine similarity between the embedding vector of the current text segment and the embedding vectors of each text segment in the first segment set; Text segments with a cosine similarity greater than a set threshold are identified as the second set of segments that have a contextual relationship with the current text segment. Calculate the semantic similarity between the current text segment and each text segment in the second segment set, and determine the average of the semantic similarity as the context score of the current text segment.

[0011] According to the technical solution provided in the embodiments of this application, optionally, each group of question and answer statements includes a question statement and an answer statement, and the plurality of text fragments include question fragments belonging to question statements and answer fragments belonging to answer statements; Determining the image insertion position based on the text fragments corresponding to the first n reference values ​​includes: For any of the first n reference values, perform the following operation: If the text fragment corresponding to the reference value belongs to the answer fragment, then the position of the text fragment corresponding to the reference value is determined as the image insertion position; If the text fragment corresponding to the reference value belongs to the question fragment, then the position of the answer fragment corresponding to the text fragment corresponding to the reference value is determined as the image insertion position.

[0012] According to the technical solution provided in the embodiments of this application, optionally, the step of filtering the plurality of candidate images to obtain the target image that best matches the target statement includes: Determine the target text segment in the target statement where the image is inserted, and the set of target segments that have a contextual relationship with the target text segment; For each candidate image, perform the following operations: calculate a first similarity between the current candidate image and the target text segment; calculate a second similarity between the current candidate image and the context segment of the target segment; and calculate a first sum of third similarities between the current candidate image and each text segment in the target segment set; determine the first sum, the first similarity, and the second sum of the second similarity; The candidate image whose second sum is the maximum value and whose corresponding first similarity is greater than the set limit is determined as the target image that best matches the target statement.

[0013] According to the technical solution provided in the embodiments of this application, optionally, the step of fusing the descriptive summary with the target statement to obtain the rewritten statement includes: The descriptive summary is fused with the target statement to obtain a specified number of candidate statements; Calculate the similarity between each candidate statement and the target image, and determine the rewritten statement that best matches the target image from the candidate statements based on the similarity.

[0014] Secondly, embodiments of this application also provide an apparatus for constructing a multimodal dataset, comprising: The first determining module is used to determine the image insertion position based on the question-and-answer statements in the text corpus; The second determining module is used to determine the retrieval statement based on the target statement where the image insertion position is located; The acquisition module is used to acquire multiple candidate images from the image search engine using the retrieval statement; The filtering module is used to filter the multiple candidate images to obtain the target image that best matches the target statement; A generation module is used to generate a descriptive summary of the target image; The rewriting module is used to merge the descriptive summary with the target statement to obtain the rewritten statement; The replacement module is used to replace the target statement with the rewritten statement and the target image.

[0015] Thirdly, embodiments of this application also provide an electronic device, the electronic device comprising: Processor and memory; The processor executes the steps of the method for constructing a multimodal dataset as described in any embodiment by invoking programs or instructions stored in the memory.

[0016] Fourthly, embodiments of this application also provide a computer-readable storage medium storing a program or instructions that cause a computer to perform the steps of the method for constructing a multimodal dataset as described in any embodiment.

[0017] In summary, this application proposes a method for constructing a multimodal dataset. First, the image insertion position is determined based on question-and-answer statements in a text corpus. Then, a retrieval statement is determined based on the target statement where the image insertion position is located, and multiple candidate images are obtained from an image search engine using the retrieval statement. Compared to obtaining candidate images from a fixed image library, obtaining candidate images from an image search engine yields a richer selection. Next, the multiple candidate images are filtered to obtain the target image that best matches the target statement. A descriptive summary of the target image is generated. The descriptive summary is fused with the target statement to obtain a rewritten statement. The target statement is then replaced using the rewritten statement and the target image. This method achieves the goal of inserting images into question-and-answer statements in a text corpus according to contextual needs, and ensures that the inserted images truly play an irreplaceable semantic supplementary role, providing a deeper level of explanation and perception in a specific context, rather than existing as redundant additional information. This achieves the effect of deep image-text fusion and provides high-quality training samples for training multimodal task models. Attached Figure Description

[0018] Figure 1 This is a flowchart illustrating a method for constructing a multimodal dataset according to an embodiment of this application; Figure 2 This is a comparative diagram of evaluation results for different datasets provided in an embodiment of this application; Figure 3 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Detailed Implementation

[0019] The present application will now be described in further detail with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative of the invention and not intended to limit it. Furthermore, it should be noted that, for ease of description, only the parts relevant to the invention are shown in the accompanying drawings.

[0020] It should be noted that, unless otherwise specified, the embodiments and features described in this application can be combined with each other. This application will now be described in detail with reference to the accompanying drawings and embodiments.

[0021] Figure 1This is a flowchart illustrating a method for constructing a multimodal dataset according to an embodiment of this application. See also... Figure 1 The method for constructing this multimodal dataset specifically includes the following steps: S110. Determine the image insertion position based on the question-and-answer statements in the text corpus.

[0022] The question-and-answer statements include questions and answers. For example, the question is "What will the weather be like tomorrow?" and the corresponding answer is "Tomorrow will be sunny with a high of 30°C and a low of 25°C".

[0023] Various publicly available question-and-answer statements (i.e., text datasets) can be collected from the internet, covering topics such as business, entertainment, education, and health to reflect real-world online conversation scenarios. To supplement existing question-and-answer statements (which often lack multi-turn casual dialogue), large language models can be used to generate multi-turn dialogue datasets. For example, ChatGPT (a publicly available generative large language model) can be used to generate multi-turn dialogue datasets (containing free-form, disjointed casual conversation content) to cover a wider range of knowledge domains. Finally, these datasets are uniformly formatted to form an initial text corpus, which includes collected or generated question-and-answer statements.

[0024] Subsequently, a series of strategies and algorithms are used to automatically identify the most suitable image insertion point in the text sequence. Unlike previous methods that simply appended images after sentences, this application's method embeds images directly into the text sequence. This finer-grained approach ensures close alignment between multimodal content and corresponding text fragments, making it particularly suitable for longer responses where images can significantly enhance text coherence.

[0025] For example, step S110, "determine the image insertion position based on the question-and-answer statements in the text corpus", includes three sub-steps, namely S111-S113.

[0026] S111. Divide each group of question and answer statements into multiple text segments.

[0027] Each question-and-answer pair refers to a question and its corresponding answer. For example, the original text dataset can be represented as follows: This includes multiple sets of question-and-answer statements. .

[0028] Each set of question and answer statements is divided into multiple text segments. u Text fragments are the smallest units used for subsequent analysis.

[0029] S112. Determine a reference value that characterizes the multimodal expressive ability of each text segment. The larger the reference value, the stronger the multimodal expressive ability of the corresponding text segment.

[0030] The multimodal expressiveness of a text fragment refers to its ability to convey multimodal information, or in other words, its suitability for multimodal expression. For example, the text fragment "The lake surface shimmers under the setting sun" conveys stronger multimodal information, giving the reader a vivid visual experience. This type of text fragment is more suitable for multimodal expression, perhaps accompanied by an image showing the shimmering lake surface under the setting sun. Compared to the text fragment "I am 10 years old this year," the former (the shimmering lake surface under the setting sun) demonstrates a stronger multimodal expressiveness.

[0031] S113. Sort the reference values ​​in descending order, and determine the image insertion position based on the text segments corresponding to the first n reference values.

[0032] For example, the reference value corresponding to the first text fragment u1 is 80, the reference value corresponding to the second text fragment u2 is 70, and the reference value corresponding to the third text fragment u3 is 60. If n is 1, the image insertion position is determined based on the first text fragment u1; if n is 2, one image insertion position is determined based on the first text fragment u1, and one image insertion position is determined based on the second text fragment u2, for a total of two image insertion positions.

[0033] Specifically, determining the image insertion position based on the first text fragment u1 includes: if the first text fragment u1 belongs to an answer fragment (i.e., a fragment extracted from an answer statement), then the position of the first text fragment u1 is determined as the image insertion position; if the first text fragment u1 belongs to a question fragment (i.e., a fragment extracted from a question statement), then the position of the answer fragment corresponding to the first text fragment u1 is determined as the image insertion position. The answer fragment corresponding to the first text fragment u1 can be any text fragment in the answer statement that answers the question statement containing the first text fragment u1.

[0034] In some implementations, step S112 includes the following four sub-steps S11-S14.

[0035] S11. Determine the probability assessment result of the current text segment becoming a potential important word.

[0036] Specifically, the importance ranking order of each text segment is determined by a large language model (e.g., Qwen2.5, a publicly available large language model); the probability assessment result of the current text segment becoming a potential important word is determined based on the total number of text segments and the importance ranking order of each text segment.

[0037] For example, the probability assessment result of the current text segment becoming a potentially important word is determined by the following formula:

[0038] in, Indicates the first i The probability assessment results of a text fragment becoming a potential important word. n Indicates the total number of text fragments. Indicates the first i The order of importance of each text fragment.

[0039] S12. For each text segment, extract keywords and perform context simulation to generate multiple continuation texts, and determine the embedding vectors of the multiple continuation texts; calculate the similarity between each keyword and the corresponding embedding vector; and determine the normalized summary weight of the current text segment based on the similarity between each keyword and the corresponding embedding vector.

[0040] For example, keywords in a text segment can be extracted using TF-IDF (Term Frequency-Inverse Document Frequency). TF-IDF is a weighted technique used in information retrieval and text mining to evaluate the importance of a word to a document in a document collection or corpus. The more important a word is, the more suitable it is as a keyword.

[0041] Furthermore, based on DialoGPT, context simulation is performed on the text fragments to generate multiple continuation texts. DialoGPT is a publicly available large-scale pre-trained dialogue generation model that can simulate the logic and language patterns of human dialogue based on given dialogue context information, generating multiple different subsequent dialogue text contents.

[0042] The generated continuation text is then processed into vector form to obtain embedding vectors, which facilitates subsequent calculations.

[0043] The similarity between each keyword and its corresponding embedding vector is calculated as follows: For example, if the keyword of text fragment u is U, and the set of multiple continuing texts generated based on the context simulation of text fragment u is F, then the similarity between the embedding vector of keyword U and the embedding vector of each continuing text f in F (specifically, cosine similarity) is calculated. Then, the similarity is normalized to obtain the normalized summary weight of text fragment u.

[0044] S13. Determine the context score of the current text segment. The higher the context score, the stronger the connection between the current text segment and the context.

[0045] Specifically, a first set of segments with contextual relationships with the current text segment is determined from each of the text segments according to semantic and / or temporal relationships; the cosine similarity between the embedding vector corresponding to the current text segment and the embedding vectors corresponding to each text segment in the first set of segments is calculated; text segments with cosine similarity greater than a set threshold are determined as a second set of segments with contextual relationships with the current text segment; the semantic similarity between the current text segment and each text segment in the second set of segments is calculated, and the average value of the semantic similarity is determined as the context score of the current text segment.

[0046] S14. Determine a reference value for the multimodal expressive power of the current text segment based on the probability assessment result of the current text segment becoming a potential important word, the normalized summary weight of the current text segment, and the context score of the current text segment.

[0047] For example, the reference value is obtained by weighted summing of the probability assessment result, the normalized summary weight, and the context score.

[0048] S120. Determine the retrieval statement based on the target statement where the image insertion position is located.

[0049] For example, the target statement where the image is inserted can be determined as the search statement, or a synonym of the target statement can be determined as the search statement.

[0050] S130. Use the search statement to obtain multiple candidate images from the image search engine.

[0051] By obtaining multiple candidate images from an image search engine, the novelty and diversity of the candidate images can be guaranteed.

[0052] S140. Filter the multiple candidate images to obtain the target image that best matches the target statement.

[0053] Specifically, multiple candidate images are filtered based on their similarity to the target statement.

[0054] For example, filtering the multiple candidate images to obtain the target image that best matches the target statement includes: determining the target text segment corresponding to the image insertion position in the target statement, and a set of target segments that have a contextual relationship with the target text segment.

[0055] As can be determined from the foregoing embodiments, the image insertion position is determined based on the text fragment. Therefore, the target text fragment corresponding to the image insertion position is known, and the set of target fragments that have a contextual relationship with the target text fragment can also be determined in accordance with the method for determining the second fragment set described in the above embodiments.

[0056] For each candidate image, perform the following operations: Calculate the first similarity between the current candidate image and the target text fragment. This first similarity can be determined using the CLIP (Contrastive Language-Image Pre-training) model. Input the current candidate image i and the target text fragment x into the CLIP model to obtain the first similarity. , can be represented as , This represents the normalization function. The first similarity score represents the basic similarity between the candidate image and the target text fragment.

[0057] Calculate the second similarity between the current candidate image and the context segment of the target segment; this can be represented as... ,in, Indicates the second similarity. This represents the basic similarity between the current candidate image i and the target text segment (i.e., the first similarity mentioned above). This represents the current candidate image i and the concatenated text. The similarity of the concatenated text It is obtained by splicing together the text fragments in the target fragment set.

[0058] And calculate the first sum of the third similarity between the current candidate image and each text segment in the target segment set; determine the second sum of the first sum, the first similarity, and the second similarity; determine the candidate image whose second sum is the maximum value and whose corresponding first similarity is greater than a set limit as the target image that best matches the target sentence.

[0059] S150. Generate a descriptive summary of the target image.

[0060] S160. The descriptive summary is fused with the target statement to obtain the rewritten statement.

[0061] S170, Replace the target statement with the rewritten statement and the target image.

[0062] To avoid disrupting the coherence of the original text by inserting images, this application proposes a method for joint image and text rewriting to naturally integrate image and text content. Specifically, it leverages the generative capabilities of a large language model to rewrite the text: First, the CLIP multimodal model is used to extract the main visual features of the target image and generate a descriptive summary. This descriptive summary is then combined with the target text fragment to generate multiple candidate sentences. The image-text similarity between each candidate sentence and the target image is calculated using the method described in the previous embodiments, and the version with the highest similarity is selected as the final multimodal data sample.

[0063] In general, the descriptive summary is fused with the target statement to obtain a specified number of candidate statements; the similarity between each candidate statement and the target image is calculated, and the rewritten statement that best matches the target image is determined from the candidate statements based on the similarity.

[0064] Ultimately, the goal of inserting images into question-and-answer statements in a text corpus according to contextual needs was achieved. The inserted images truly played an irreplaceable role in semantic supplementation, providing a deeper level of explanation and perception in specific contexts, rather than existing as redundant additional information. This achieved the effect of deep image-text fusion and provided high-quality training samples for training multimodal task models.

[0065] Furthermore, the multimodal dataset constructed in the above embodiments is used to train an active multimodal response dialogue model. The trained dialogue model possesses the ability to generate dialogue responses with active multimodal responses. It can proactively insert appropriate images at suitable text locations. These images do not exist as redundant additional information, but rather play a truly irreplaceable semantic supplementary role, providing a deeper level of interpretation and perception in specific contexts, achieving a deep fusion of text and images. In the specific dialogue effect, multimodal content and text content are presented in real time. Based on the existing text response, key content requiring multimodal information is identified. Finally, the dialogue is reconstructed based on the captured image information to achieve a rich visual effect.

[0066] The dialogue generation model trained using the multimodal dataset provided in this application significantly improves the quality of generated text, the ability to fuse multimodal information, and the effect of emotional expression.

[0067] The significant advantages of the proposed scheme were demonstrated through systematic experiments in multiple aspects. The specific comparison results are shown in Table 1, which shows the performance of the Qwen2-VL model trained on different datasets (including Ours dataset of this application, and two other datasets, namely the publicly available MMDialog and DialogCC).

[0068] Table 1

[0069] For example, the model learning rate was set to 3e-5, the pre-trained model attention heads were set to 12, dropout was set to 0.1, the batch size was set to 16, the fine-tuning epochs were set to 8, the maximum text length was 256, and the LoRa fine-tuning parameters were set to LoRa rank to 8, LoRa alpha to 32, and LoRa dropout to 0.1. The average results of 5 independent training runs for 20 epochs for all models are reported. All models were implemented using PyTorh (a mainstream deep learning framework) and NVIDIA A100-SXM4-80GB (a data center GPU based on the Ampere architecture, designed for AI training, high-performance computing, data analysis, and other scenarios).

[0070] This experiment evaluated the performance of text generation and multimodal fusion using multiple metrics: BLEU-1 and BLEU-2 were used to measure the degree of matching between the generated text and the reference answer at the single-word unigram level and the two-word bigram level, respectively. Our dataset scored 9.76 on BLEU-1, outperforming MMDialog (9.31) and DialogCC (8.53), indicating better word-level matching between the generated text and the reference response. Although the BLEU-2 score was slightly lower due to the greater difficulty of matching at the longer word-level n-grams, our dataset still outperformed the baseline methods.

[0071] ROUGE-L: Evaluates the longest common subsequence between the generated text and the reference text to measure semantic and structural similarity. Our application achieved a dataset score of 12.80, significantly higher than MMDialog (10.12) and DialogCC (8.98), indicating better performance in semantic coherence and context preservation.

[0072] IIT (Image Insertion Timing): This evaluates the model's ability to accurately predict inserted image information in dialogue. Our dataset scores 86.78, significantly outperforming MMDialog (64.78) and DialogCC (69.46), demonstrating a significant advantage in image insertion judgment.

[0073] IR@1, IR@5, and IR@10: These are standard image retrieval metrics, representing the accuracy of the model in retrieving relevant images from the first 1, first 5, and first 10 results, respectively. The scores for this application's dataset are 26.52, 47.63, and 60.41, respectively, demonstrating superior performance in retrieving context-relevant images.

[0074] Furthermore, the diversity of the dataset itself was evaluated. Previous studies have primarily focused on the number of high-quality images available in a single response, while this application's approach shifts to evaluating the frequency of multimodal auxiliary text content used in a response. Table 2 reflects the considerations for the timing of multimodal integration in this application's dataset.

[0075] Where M / U represents the average multimodal insertion timing diversity score for each sample; M / D represents the average multimodal insertion timing diversity score for each response. The results show that the dataset in this application exhibits diverse multimodal usage, effectively utilizing multimodal information to support text content expression.

[0076] Table 2

[0077] Further reference Figure 2 As shown, Figure 2 This paper presents the comparison results of the dataset in this application with DialogCC and MMDialog across multiple evaluation dimensions. Label 210 indicates the evaluation result of the dataset in this application, while labels 220 and 230 indicate the evaluation results of the DialogCC and MMDialog datasets, respectively. It can be observed that the dataset in this application shows a slight improvement over other datasets in some basic multimodal understanding tasks, especially in the emotion domain, where it exhibits a significant enhancement. Specifically, it surpasses the performance of the other two datasets in capturing emotion and atmosphere, indicating that the model trained on the dataset in this application has a greater advantage in handling emotional details and creating atmosphere. This improvement is likely due to a more refined multimodal data fusion method, enabling the model to perform better in handling emotion and atmosphere. Furthermore, there are also improvements in understanding and expression capabilities. Although these improvements are more moderate compared to the significant improvements in the emotion domain, they are still superior to traditional methods, indicating a more balanced overall multimodal understanding capability.

[0078] In summary, the experimental results demonstrate that the dataset presented in this application has significant advantages in enhancing emotion understanding and creating atmosphere, especially suitable for emotion-driven multimodal tasks. Future optimizations can further refine other aspects to achieve balanced improvements across all dimensions, thereby promoting the development of more comprehensive multimodal dialogue systems.

[0079] Figure 3 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. For example... Figure 3 As shown, the electronic device 500 includes one or more processors 501 and memory 502.

[0080] The processor 501 may be a central processing unit (CPU) or other form of processing unit with data processing capabilities and / or instruction execution capabilities, and may control other components in the electronic device 500 to perform desired functions.

[0081] The memory 502 may include one or more computer program products, which may include various forms of computer-readable storage media, such as volatile memory and / or non-volatile memory. The volatile memory may include, for example, random access memory (RAM) and / or cache memory. The non-volatile memory may include, for example, read-only memory (ROM), hard disk, flash memory, etc. One or more computer program instructions may be stored on the computer-readable storage medium, and the processor 501 may execute the program instructions to implement the method for constructing a multimodal dataset according to any embodiment of this application described above, and / or other desired functions. Various contents such as initial extrinsic parameters and thresholds may also be stored in the computer-readable storage medium.

[0082] In one example, the electronic device 500 may further include an input device 503 and an output device 504, these components being interconnected via a bus system and / or other forms of connection mechanisms (not shown). The input device 503 may include, for example, a keyboard, a mouse, etc. The output device 504 may output various information to the outside, including warning messages, braking force, etc. The output device 504 may include, for example, a display, a speaker, a printer, and a communication network and its connected remote output devices, etc.

[0083] Of course, for the sake of simplicity, Figure 3 Only some of the components of the electronic device 500 relevant to this application are shown in this illustration; components such as buses, input / output interfaces, etc., are omitted. In addition, the electronic device 500 may include any other suitable components depending on the specific application.

[0084] In addition to the methods and devices described above, embodiments of this application may also be computer program products, which include computer program instructions that, when executed by a processor, cause the processor to perform the steps of the method for constructing a multimodal dataset provided in any embodiment of this application.

[0085] The computer program product can be written in any combination of one or more programming languages ​​to perform the operations of the embodiments of this application. The programming languages ​​include object-oriented programming languages ​​such as Java and C++, as well as conventional procedural programming languages ​​such as C or similar languages. The program code can be executed entirely on the user's computing device, partially on the user's computing device, as a standalone software package, partially on the user's computing device and partially on a remote computing device, or entirely on a remote computing device or server.

[0086] Furthermore, embodiments of this application may also be computer-readable storage media storing computer program instructions thereon, which, when executed by a processor, cause the processor to perform the steps of the method for constructing a multimodal dataset provided in any embodiment of this application.

[0087] The computer-readable storage medium may be any combination of one or more readable media. A readable medium may be a readable signal medium or a readable storage medium. A readable storage medium may be, for example, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of readable storage media (a non-exhaustive list) include: an electrical connection having one or more wires, a portable disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination thereof.

[0088] It should be noted that the terminology used in this application is for the purpose of describing specific embodiments only and is not intended to limit the scope of this application. As shown in the specification and claims of this application, unless the context clearly indicates otherwise, words such as "a," "an," "an," and / or "the" do not specifically refer to the singular and may also include the plural. The terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, or apparatus. Without further limitations, an element defined by the phrase "comprising an..." does not exclude the presence of other identical elements in the process, method, or apparatus that includes said element.

[0089] It should also be noted that the terms "center," "upper," "lower," "left," "right," "vertical," "horizontal," "inner," and "outer," etc., indicate the orientation or positional relationship based on the orientation or positional relationship shown in the accompanying drawings, and are only for the convenience of describing this application and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation, and therefore should not be construed as a limitation on this application. Unless otherwise expressly specified and limited, the terms "installed," "connected," "linked," etc., should be interpreted broadly. For example, they can refer to a fixed connection, a detachable connection, or an integral connection; they can refer to a mechanical connection or an electrical connection; they can refer to a direct connection or an indirect connection through an intermediate medium; they can refer to the internal communication between two elements. For those skilled in the art, the specific meaning of the above terms in this application can be understood according to the specific circumstances.

[0090] This document uses specific examples to illustrate the principles and implementation methods of this application. The descriptions of the above embodiments are only for the purpose of helping to understand the methods and core ideas of this application. The above descriptions are only preferred embodiments of this application. It should be noted that due to the limitations of written expression, while there are objectively infinite specific structures, those skilled in the art can make several improvements, modifications, or changes without departing from the principles of this invention, and can also combine the above technical features in an appropriate manner. These improvements, modifications, changes, or combinations, or the direct application of the inventive concept and technical solution to other situations without modification, should all be considered within the scope of protection of this application.

Claims

1. A method for constructing a multimodal dataset, characterized in that, The method comprises the following steps: determining an image insertion position according to a question and answer sentence in a text corpus; determining a retrieval sentence according to a target sentence where the image insertion position is located; obtaining multiple candidate images from an image search engine by using the retrieval sentence; obtaining a target image most matched with the target sentence by screening the multiple candidate images; generating a descriptive abstract of the target image; fusing the descriptive abstract with the target sentence to obtain a rewritten sentence; replacing the target sentence with the rewritten sentence and the target image.

2. The method of claim 1, wherein, The method of determining an image insertion position according to a question and answer sentence in a text corpus comprises the following steps: dividing each group of question and answer sentences into multiple text segments; determining a reference value representing the multi-modal expression capability of each text segment; sorting the reference values in descending order, and determining an image insertion position according to the text segments corresponding to the top n reference values.

3. The method of claim 2, wherein, The method of determining a reference value representing the multi-modal expression capability of each text segment comprises the following steps: determining a probability evaluation result of a current text segment becoming a latent important word; for each text segment, extracting a keyword, generating multiple continued texts through context simulation, determining embedding vectors of the multiple continued texts, respectively calculating the similarity between each keyword and the corresponding embedding vector, and determining a normalized abstract weight of the current text segment according to the similarity between each keyword and the corresponding embedding vector; determining a context score of the current text segment; determining a reference value representing the multi-modal expression capability of the current text segment according to the probability evaluation result of the current text segment becoming a latent important word, the normalized abstract weight of the current text segment, and the context score of the current text segment.

4. The method of claim 3, wherein, The method of determining a probability evaluation result of a current text segment becoming a latent important word comprises the following steps: determining an importance ranking order of each text segment by using a large language model; determining a probability evaluation result of a current text segment becoming a latent important word according to the total number of text segments and the importance ranking order of each text segment.

5. The method of claim 3, wherein, The method of determining a context score of a current text segment comprises the following steps: determining a first segment set having a context relationship with the current text segment from each text segment according to semantic relationships and / or temporal relationships; respectively calculating the cosine similarity between the embedding vector corresponding to the current text segment and the embedding vector corresponding to each text segment in the first segment set; determining a second segment set having a context relationship with the current text segment by selecting text segments with a cosine similarity greater than a set threshold value from the first segment set; calculating the semantic similarity between the current text segment and each text segment in the second segment set, and determining the average value of the semantic similarity as the context score of the current text segment.

6. The method of claim 2, wherein, Each group of question and answer sentences comprises a question sentence and an answer sentence, and the multiple text segments comprise a question segment belonging to the question sentence and an answer segment belonging to the answer sentence. The method of determining an image insertion position according to the text segments corresponding to the top n reference values comprises the following steps: The operations performed on any reference value in the first n reference values are as follows: If the text segment corresponding to the reference value belongs to the answer segment, the position of the text segment corresponding to the reference value is determined as the image insertion position; If the text segment corresponding to the reference value belongs to the question segment, the position of the answer segment corresponding to the text segment corresponding to the reference value is determined as the image insertion position.

7. The method of claim 1, wherein, The filtering of the plurality of candidate images to obtain a target image most matching the target sentence comprises: determining a target text segment in the target sentence corresponding to the image insertion position, and a target segment set having a context relationship with the target text segment; for each of the candidate images, calculating a first similarity between the current candidate image and the target text segment, a second similarity between the current candidate image and the context segment of the target segment, and a first sum of third similarities between the current candidate image and each text segment in the target segment set; determining a second sum of the first sum, the first similarity, and the second similarity; the candidate image with the maximum second sum and the first similarity greater than a set limit value is determined as the target image most matching the target sentence.

8. The method of claim 1, wherein, The fusion of the descriptive summary and the target sentence to obtain a rewritten sentence comprises: fusing the descriptive summary and the target sentence to obtain a specified number of candidate sentences; calculating the similarity between each candidate sentence and the target image, and determining the rewritten sentence most matching the target image from the candidate sentences according to the similarity.

9. An electronic device, comprising: The electronic device comprises: a processor and a memory; the processor is configured to execute the steps of the method for constructing a multi-modal data set according to any one of claims 1 to 8 by invoking programs or instructions stored in the memory.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores programs or instructions for causing a computer to execute the steps of the method for constructing a multi-modal data set according to any one of claims 1 to 8.

Images