Multi-modal data generation method and device, computer equipment and storage medium

Abstract

The invention relates to the field of data processing, in particular to a multi-modal data generation method and device, computer equipment and a storage medium, and the method comprises the following steps: carrying out theme extraction on user question data to obtain theme information; performing logic structure analysis on the user question data to obtain logic structure information of the user question data; searching literal data, picture data and video data corresponding to the user question data in a multi-modal database according to the theme information and the edit structure information; carrying out integration processing on the text data, the picture data and the video data to obtain initial multi-modal data; and performing style migration processing on all the target character texts, all the target pictures and all the target videos to obtain target multi-modal data. According to the method, the recommendation information can be displayed more comprehensively in a target multi-modal data form, and the user experience is improved.

Description

Technical Field

[0001] The present invention relates to the field of data processing, and in particular to a method, device, computer equipment and storage medium for generating multimodal data. Background Art

[0002] With the rapid development of the Internet, network data such as text, images, and videos are widely used in information recommendation. When recommending information for an event, the existing technology usually generates corresponding video or image unimodal data through keywords corresponding to the event, or generates corresponding text unimodal data through multimodal data (video and image) corresponding to the event.

[0003] When recommending information involving important knowledge areas (for example, medical, financial, education, etc.), it is usually necessary to combine multimodal data such as text, images, and videos to better interpret the business or product. However, there is currently no method to generate multimodal data based on single-modal data. As a result, when recommending information about a business or product, the generated recommended information data modality is relatively single, and it is impossible to fully display the event from multiple modalities such as text, images, and videos, which reduces the user experience. Summary of the invention

[0004] Based on this, it is necessary to provide a multimodal data generation method, device, computer equipment and storage medium to address the above-mentioned technical problems, so as to solve the problem that the data modality of existing recommendation information is relatively single and events cannot be fully displayed from multiple modalities such as text, images and videos, resulting in a poor user experience.

[0005] A multimodal data generation method, comprising: Get user question data; Extracting the subject of the user's question data to obtain subject information; and performing logical structure analysis on the user's question data to obtain logical structure information of the user's question data; Retrieving text materials, picture materials and video materials corresponding to the user question data in a multimodal database according to the subject information and the editor structure information; Integrate the text data, the picture data and the video data to obtain initial multimodal data; the initial multimodal data includes at least one target text, at least one target picture and at least one target video; Style transfer processing is performed on all the target texts, all the target images, and all the target videos to obtain target multimodal data.

[0006] A multimodal data generating device, comprising: Question module, used to obtain user question data; An analysis module is used to extract the subject of the user question data to obtain subject information; and to perform logical structure analysis on the user question data to obtain logical structure information of the user question data; A retrieval module, used to retrieve text materials, picture materials and video materials corresponding to the user question data in a multimodal database according to the subject information and the editor structure information; An integration module, used for integrating the text data, the picture data and the video data to obtain initial multimodal data; the initial multimodal data includes at least one target text, at least one target picture and at least one target video; The style module is used to perform style migration processing on all the target texts, all the target images and all the target videos to obtain target multimodal data.

[0007] A computer device comprises a memory, a processor and computer-readable instructions stored in the memory and executable on the processor, wherein the processor implements the multimodal data generation method when executing the computer-readable instructions.

[0008] One or more readable storage media storing computer-readable instructions, wherein when the computer-readable instructions are executed by one or more processors, the one or more processors execute the multimodal data generation method as described above.

[0009] The above-mentioned multimodal data generation method, device, computer equipment and storage medium perform topic extraction on the user question data to obtain topic information; and perform logical structure analysis on the user question data to obtain logical structure information of the user question data; according to the topic information and the logical structure information, retrieve the text data, picture data and video data corresponding to the user question data in the multimodal database; integrate the text data, the picture data and the video data to obtain initial multimodal data; the initial multimodal data includes at least one target text, at least one target picture and at least one target video; perform style transfer processing on all the target texts, all the target pictures and all the target videos to obtain target multimodal data. The target multimodal data generated by the present invention includes text, pictures and videos, which realizes a more comprehensive display of recommended information in the form of target multimodal data and improves user experience. At the same time, the target multimodal data undergoes style transfer processing, so that the generated target multimodal data not only retains the original content information, but also incorporates specific style elements, thereby enhancing the expressiveness and attractiveness of the target multimodal data, and is suitable for specific application scenarios, such as artistic creation, advertising or personalized content generation. BRIEF DESCRIPTION OF THE DRAWINGS

[0010] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings required for use in the description of the embodiments of the present invention will be briefly introduced below. Obviously, the accompanying drawings in the following description are only some embodiments of the present invention. For ordinary technicians in this field, other accompanying drawings can be obtained based on these accompanying drawings without paying creative labor.

[0011] Figure 1 is a schematic diagram of an application environment of a multimodal data generation method in an embodiment of the present invention; Figure 2 is a flow chart of a method for generating multimodal data in one embodiment of the present invention; Figure 3 is a structural schematic diagram of a multimodal data generating device in one embodiment of the present invention; Figure 4 is a schematic diagram of a computer device in one embodiment of the present invention. DETAILED DESCRIPTION

[0012] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0013] The multimodal data generation method provided in this embodiment can be applied in Figure 1 In an application environment, a client communicates with a server. The client includes but is not limited to various personal computers, laptops, smart phones, tablet computers, and portable wearable devices. The server can be implemented with an independent server or a server cluster consisting of multiple servers.

[0014] In one embodiment, if Figure 2 As shown, a multimodal data generation method is provided, and the method is applied in Figure 1 The server in the example is used as an example to illustrate the following steps: S10. Obtain user question data.

[0015] It can be understood that user question data refers to questions or demands raised by users. Usually, user question data exists in the form of text, and here, it can also be in the form of voice, picture or video.

[0016] S20, extracting topics from the user question data to obtain topic information; and performing logical structure analysis on the user question data to obtain logical structure information of the user question data.

[0017] Understandably, topic extraction refers to the process of identifying the main topics from the user question data using topic modeling technology to obtain topic information. Among them, topic information refers to the information related to the topic contained in the user question data. For example, when the user question data contains multiple topics, the topic information contains the relevant information of each topic. Specifically, the user question data is vectorized using topic modeling technology to obtain a topic matrix, which includes at least one topic word and a probability value corresponding to the topic word. Logical structure parsing refers to the process of parsing the grammatical and semantic structure of the user question data to obtain the logical structure information of the user question data. Among them, logical structure information refers to the logic and structure information contained in the user question data. Among them, logical information includes causal relationships, conditional relationships, comparative relationships, etc. Structural information includes hierarchical structures (for example, when the problem is complex, the problem is decomposed into sub-problems and the hierarchical relationship between them is determined), logical processes (describe the logic of the information in the problem, for example, from cause to effect), etc.

[0018] Optionally, in step S20, that is, extracting the subject of the user question data to obtain subject information, includes: S201, performing multi-topic identification on the user question data through a topic identification model to obtain a topic matrix; the topic matrix includes at least one topic word and at least one topic probability value; S202: Generate the topic information according to all the topic words and topic probability values.

[0019] Understandably, the topic identification model is a machine learning model, such as LDA (Latent Dirichlet Allocation), which is used to identify and extract topics from user question data to obtain a topic matrix. Specifically, multi-topic identification uses the topic identification model to analyze the word frequency distribution and co-occurrence relationship in the user question data to determine the topic distribution of the user question data and obtain at least one topic word; then, the topic identification model outputs the topic matrix. The topic probability value in the topic matrix represents the probability that the user question data belongs to the corresponding topic word. After determining the topic word and the topic probability value corresponding to the topic word, the topic word is screened according to the topic probability value to generate topic information. Specifically, the topic words with a topic probability value lower than the preset probability value are eliminated, and the topic words with a topic probability value higher than or equal to the preset probability value are retained, and the topic information is generated according to the retained topic words and the corresponding obtained topic probability values. In this way, according to the topic probability values ​​in the topic matrix, several topics with higher probability of topic probability values ​​can be selected as the main topics of the user question data, and the keywords of these topics can be extracted to generate the topic information of the document.

[0020] Optionally, in step S20, that is, performing logical structure analysis on the user question data to obtain logical structure information of the user question data, includes: S203, performing word segmentation and part-of-speech tagging on the user question data to obtain a part-of-speech tag for each word in the user question data; S204, determining the syntactic relationship of the user question data according to the part-of-speech tags; S205: Determine the logical structure information of the user question data according to the syntactic relationship.

[0021] Understandably, word segmentation refers to the process of segmenting user question data into words or phrases. It should be understood that when the user question data is voice, picture or video, the user question data must first be converted into text. Part-of-speech tagging refers to the process of assigning a part of speech (such as noun, verb, adjective, etc.) to each word in the user question data. Syntactic relations refer to the syntactic structural relations of words in sentences in user question data, such as cause-effect relations, subject-predicate relations, verb-object relations, modification relations, transition relations, etc. It should be understood that by analyzing syntactic relations, logical conjunctions and phrases in sentences can be identified, thereby determining the logical structure in the sentence, such as cause-effect relations, transition relations, etc., which helps to deeply understand the meaning of user question data and the intention of the user's question.

[0022] S30. According to the subject information and the editor structure information, text materials, picture materials and video materials corresponding to the user question data are retrieved in a multimodal database.

[0023] Understandably, a multimodal database refers to a database that stores multiple data types (e.g., text, image, video, voice, etc.). After determining the subject information and the editorial structure information, the subject information and the editorial structure information are used as search keywords or query conditions to search for text materials, image materials, and video materials corresponding to the user's question data in the multimodal database. Among them, text materials refer to materials stored in the form of text. Image materials refer to materials stored in the form of images. Video materials refer to materials stored in the form of videos.

[0024] S40, integrating and processing the text data, the picture data and the video data to obtain initial multimodal data; the initial multimodal data includes at least one target text, at least one target picture and at least one target video.

[0025] Understandably, the integration process can combine different types of data to form a unified data set. Here, the integration process refers to the process of unifying text materials, picture materials and video materials to obtain initial multimodal data. Among them, the initial multimodal data includes at least one target text, at least one target picture and at least one target video. The target text refers to the text that is finally determined to correspond to the user's question data. The target picture refers to the picture that is determined to correspond to the target text in the picture material. The target video refers to the video that is determined to correspond to the target text in the video material.

[0026] Optionally, the text material includes at least one initial text; the picture material includes at least one initial picture; the video material includes at least one initial video; it is understandable that the initial text refers to the text material corresponding to the user question data. The initial picture refers to the picture material corresponding to the user question data. The initial video refers to the video material corresponding to the user question data. It should be understood that each topic corresponds to at least one initial text, and when there are multiple topics in the user question data, there are multiple initial text texts in the text material corresponding to the user question data. Similarly, each topic corresponds to at least one initial picture, and when there are multiple topics in the user question data, there are multiple initial pictures in the picture material corresponding to the user question data. Each topic corresponds to at least one initial video, and when there are multiple topics in the user question data, there are multiple initial videos in the video material corresponding to the user question data.

[0027] Furthermore, in step S40, the text data, the picture data and the video data are integrated to obtain initial multimodal data, including: S401, determining at least one target text from all the initial texts; S402, performing semantic recognition on each of the target texts to extract the central intention of each of the target texts; S403, according to the central intention, matching a corresponding target picture in all the initial pictures for each target text; and according to the central intention, matching a corresponding target video in all the initial videos for each target text; S404: Generate the initial multimodal data according to all the target texts, all the target images, and all the target videos.

[0028] Understandably, the target text refers to the text data determined from all the initial texts for further semantic recognition processing, and the target text can be any one of all the initial texts. Among them, semantic recognition refers to the process of identifying the central intention of the target text. The central intention refers to the core meaning or main purpose of the target text. Generally, the central intention is identified by identifying the key sentences, phrases or keywords in the target text. The target picture refers to the picture that matches the central intention of the target text. The target video refers to the video that matches the central intention of the target text. In embodiments S401-S403, the central intention of the target text is used to find the most relevant target pictures and target videos in the initial pictures and initial videos through image recognition and video content analysis technology, so that the determined target text, target video, and target picture correspond to the same central intention, ensuring the matching degree of the target text, target video, and target picture, so that the multimodal data finally generated corresponds to the same central intention of the user's question data, improving the user experience.

[0029] Optionally, the initial multimodal data includes at least one intended multimodal data; Furthermore, in step S404, that is, generating the initial multimodal data according to all the target texts, all the target images and all the target videos includes: S4041. Determine a target central intention among all central intentions; S4042, obtaining a target picture and a target video corresponding to the target center intention; S4043: embed the target image and the target video into a target text corresponding to the target central intent to generate the intent multimodal data.

[0030] It can be understood that the user question data includes at least one topic, each topic corresponds to an initial text, and each target text has its own corresponding central intention. Therefore, the target image, target video, and target text can be integrated and processed based on the target central intention to obtain intention multimodal data, so that the generated intention multimodal data corresponds to the determined target central intention, thereby improving the user experience.

[0031] S50: Perform style transfer processing on all the target texts, all the target images, and all the target videos to obtain target multimodal data.

[0032] Understandably, style transfer is an image processing technology that can apply the style of one image to another image while retaining the content of the latter. Generally, style transfer includes the transfer of artistic style, specific tones, textures or visual elements. The style transfer processing of the target text, the target image and the target video not only retains the original content information, but also incorporates specific style elements, thereby enhancing the expressiveness and attractiveness of the target multimodal data, which is suitable for specific application scenarios, such as artistic creation, advertising or personalized content generation.

[0033] Optionally, in step S50, that is, performing style transfer processing on all the target texts, all the target images and all the target videos to obtain target multimodal data includes: S501, obtaining style information to be transferred of the initial multimodal data; S502, obtaining style features of the style information to be migrated through a style editor; S503: Generate the target multimodal data according to the style feature and the initial multimodal data.

[0034] Understandably, the style information to be transferred refers to the style attributes to be applied to the target text, target image and target video, and the style information to be transferred refers to the style information such as a certain artistic style, a specific tone, texture or visual element. Here, the style information to be transferred can be obtained through user input, a style selection interface or an automatic style recognition technology. The style editor is used to extract and analyze the style information to be transferred and convert it into operable parameters. Specifically, the style information to be transferred is analyzed by the style editor to extract key visual and / or text style features. For example, for the target image and target video, the style features include features such as color, texture, shape and line; for the target text, the style features include features such as vocabulary selection, sentence structure, and tone. After extracting the style features, the target text, target image and target video are style-transferred based on the determined style features, and the stylized target text, target image and target video data are embedded and integrated according to a certain logic and structure to ensure the consistency of the style of the target multimodal data and the coherence of the content. In this way, the generated data target multimodal data presents a highly adaptable or user-specified style in terms of vision and text, while retaining the essence of the original content and improving the user experience.

[0035] In steps S10-S50, the user question data is subjected to topic extraction to obtain topic information; and the user question data is subjected to logical structure analysis to obtain logical structure information of the user question data; according to the topic information and the logical structure information, the text data, picture data and video data corresponding to the user question data are retrieved in the multimodal database; the text data, the picture data and the video data are integrated to obtain initial multimodal data; the initial multimodal data includes at least one target text, at least one target picture and at least one target video; style transfer processing is performed on all the target texts, all the target pictures and all the target videos to obtain target multimodal data. In this embodiment, the generated target multimodal data includes text, pictures and videos, so that the recommended information displayed in the form of target multimodal data is more comprehensive in terms of event display, thereby improving the user experience. At the same time, the target multimodal data undergoes style transfer processing, so that the generated target multimodal data not only retains the original content information, but also incorporates specific style elements, thereby enhancing the expressiveness and attractiveness of the target multimodal data, and is suitable for specific application scenarios, such as artistic creation, advertising or personalized content generation.

[0036] It should be understood that the order of execution of the steps in the above embodiment does not necessarily mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiment of the present invention.

[0037] In one embodiment, a multimodal data generating device is provided, and the multimodal data generating device corresponds one-to-one to the multimodal data generating method in the above embodiment. Figure 3 As shown, the multimodal data generation device includes a questioning module 10, an analysis module 20, a retrieval module 30, an integration module 40 and a style module 50. The functional modules are described in detail as follows: Questioning module 10, used to obtain user question data; The analysis module 20 is used to extract the subject of the user question data to obtain subject information; and to perform logical structure analysis on the user question data to obtain logical structure information of the user question data; A retrieval module 30, configured to retrieve text materials, picture materials and video materials corresponding to the user's question data in a multimodal database according to the subject information and the editor structure information; An integration module 40 is used to integrate the text data, the picture data and the video data to obtain initial multimodal data; the initial multimodal data includes at least one target text, at least one target picture and at least one target video; The style module 50 is used to perform style transfer processing on all the target texts, all the target images and all the target videos to obtain target multimodal data.

[0038] Optionally, the analysis module 20 includes: A topic identification unit, used to perform multi-topic identification on the user question data through a topic identification model to obtain a topic matrix; the topic matrix includes at least one topic word and at least one topic probability value; The topic information unit is used to generate the topic information according to all the topic words and topic probability values.

[0039] Optionally, the analysis module 20 includes: A part-of-speech tag unit, used to perform word segmentation and part-of-speech tagging on the user question data to obtain a part-of-speech tag for each word in the user question data; A syntactic relationship unit, used to determine the syntactic relationship of the user question data according to the part-of-speech tag; The logical structure unit is used to determine the logical structure information of the user question data according to the syntactic relationship.

[0040] Optionally, the text material includes at least one initial text; the picture material includes at least one initial picture; the video material includes at least one initial video; Furthermore, the integration module 40 includes: A target text unit, used to determine at least one target text from all the initial texts; A central intention unit, used to perform semantic recognition on each of the target texts and extract the central intention of each of the target texts; A target data matching unit is used to match a corresponding target picture in all the initial pictures for each target text according to the central intention; and to match a corresponding target video in all the initial videos for each target text according to the central intention; An initial multimodal data unit is used to generate the initial multimodal data according to all the target texts, all the target images and all the target videos.

[0041] Optionally, the initial multimodal data includes at least one intended multimodal data; Furthermore, the initial multimodal data unit includes: A target center intention unit, used to determine a target center intention among all center intentions; A target data acquisition unit, used to acquire a target picture and a target video corresponding to the target center intention; The intention multimodal data unit is used to embed the target image and the target video into the target text corresponding to the target central intention to generate the intention multimodal data.

[0042] Optionally, the style module 50 includes: A style information unit to be transferred, used to obtain the style information to be transferred of the initial multimodal data; A style feature unit, used for obtaining the style features of the style information to be migrated through a style editor; The target multimodal data unit is used to generate the target multimodal data according to the style feature and the initial multimodal data.

[0043] For the specific definition of the multimodal data generating device, please refer to the definition of the multimodal data generating method above, which will not be repeated here. Each module in the above-mentioned multimodal data generating device can be implemented in whole or in part by software, hardware and a combination thereof. The above-mentioned modules can be embedded in or independent of the processor in the computer device in the form of hardware, or can be stored in the memory in the computer device in the form of software, so that the processor can call and execute the operations corresponding to the above modules.

[0044] In one embodiment, a computer device is provided. The computer device may be a server, and its internal structure diagram may be as follows: Figure 4 As shown. The computer device includes a processor, a memory, a network interface and a database connected via a system bus. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a readable storage medium and an internal memory. The readable storage medium stores an operating system, computer-readable instructions and a database. The internal memory provides an environment for the operation of the operating system and computer-readable instructions in the readable storage medium. The database of the computer device is used to store data involved in the multimodal data generation method. The network interface of the computer device is used to communicate with an external terminal via a network connection. When the computer-readable instructions are executed by the processor, a multimodal data generation method is implemented. The readable storage medium provided in this embodiment includes a non-volatile readable storage medium and a volatile readable storage medium.

[0045] In one embodiment, a computer device is provided, including a memory, a processor, and computer-readable instructions stored in the memory and executable on the processor, wherein the processor executes the computer-readable instructions to implement the following steps: Get user question data; Extracting the subject of the user's question data to obtain subject information; and performing logical structure analysis on the user's question data to obtain logical structure information of the user's question data; Retrieving text materials, picture materials and video materials corresponding to the user question data in a multimodal database according to the subject information and the editor structure information; Integrate the text data, the picture data and the video data to obtain initial multimodal data; the initial multimodal data includes at least one target text, at least one target picture and at least one target video; Style transfer processing is performed on all the target texts, all the target images, and all the target videos to obtain target multimodal data.

[0046] In one embodiment, one or more computer-readable storage media storing computer-readable instructions are provided. The readable storage media provided in this embodiment include non-volatile readable storage media and volatile readable storage media. The readable storage media store computer-readable instructions, and when the computer-readable instructions are executed by one or more processors, the following steps are implemented: Get user question data; Extracting the subject of the user's question data to obtain subject information; and performing logical structure analysis on the user's question data to obtain logical structure information of the user's question data; Retrieving text materials, picture materials and video materials corresponding to the user question data in a multimodal database according to the subject information and the editor structure information; Integrate the text data, the picture data and the video data to obtain initial multimodal data; the initial multimodal data includes at least one target text, at least one target picture and at least one target video; Style transfer processing is performed on all the target texts, all the target images, and all the target videos to obtain target multimodal data.

[0047] Those of ordinary skill in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be implemented by instructing related hardware through computer-readable instructions, and the computer-readable instructions can be stored in a non-volatile readable storage medium or a volatile readable storage medium. When the computer-readable instructions are executed, they may include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, storage, database or other media used in the embodiments provided in this application may include non-volatile and / or volatile memory. Non-volatile memory may include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory may include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in many forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM), etc.

[0048] Those skilled in the art can clearly understand that for the convenience and simplicity of description, only the division of the above-mentioned functional units and modules is used as an example. In actual applications, the above-mentioned functions can be distributed and completed by different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above.

[0049] The embodiments described above are only used to illustrate the technical solutions of the present invention, rather than to limit the same. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that the technical solutions described in the aforementioned embodiments may still be modified, or some of the technical features may be replaced by equivalents. Such modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention, and should all be included in the protection scope of the present invention.

Claims

1. A method for generating multimodal data, characterized in that: include: Get user question data; Extracting topics from the user question data to obtain topic information; and performing a logical structure analysis on the user question data to obtain logical structure information of the user question data; Retrieving text materials, picture materials and video materials corresponding to the user question data in a multimodal database according to the subject information and the editor structure information; Integrate the text data, the picture data and the video data to obtain initial multimodal data; the initial multimodal data includes at least one target text, at least one target picture and at least one target video; Style transfer processing is performed on all the target texts, all the target images, and all the target videos to obtain target multimodal data.

2. The multimodal data generation method according to claim 1, characterized in that: The subject extraction of the user question data to obtain subject information includes: Performing multi-topic identification on the user question data through a topic identification model to obtain a topic matrix; the topic matrix includes at least one topic word and at least one topic probability value; The topic information is generated according to all the topic words and topic probability values.

3. The multimodal data generation method according to claim 1, characterized in that: The performing logical structure analysis on the user question data to obtain logical structure information of the user question data includes: Performing word segmentation and part-of-speech tagging on the user question data to obtain a part-of-speech tag for each word in the user question data; Determining the syntactic relationship of the user question data according to the part-of-speech tags; The logical structure information of the user question data is determined according to the syntactic relationship.

4. The multimodal data generation method according to claim 1, characterized in that: The text material includes at least one initial text; the picture material includes at least one initial picture; the video material includes at least one initial video; The integrating and processing the text data, the picture data and the video data to obtain initial multimodal data includes: Determining at least one target text among all the initial texts; Performing semantic recognition on each of the target texts to extract the central intention of each of the target texts; According to the central intention, for each target text, a corresponding target picture is matched in all the initial pictures; and according to the central intention, for each target text, a corresponding target video is matched in all the initial videos; The initial multimodal data is generated according to all the target texts, all the target images and all the target videos.

5. The multimodal data generation method according to claim 4, characterized in that: The initial multimodal data includes at least one intended multimodal data; The generating the initial multimodal data according to all the target texts, all the target images and all the target videos includes: Identify a target central intention among all central intentions; Obtaining a target image and a target video corresponding to the target center intention; The target image and the target video are embedded in a target text corresponding to the target central intent to generate the intent multimodal data.

6. The multimodal data generation method according to claim 1, characterized in that: The performing style transfer processing on all the target texts, all the target images and all the target videos to obtain target multimodal data includes: Acquire the style information to be transferred of the initial multimodal data; Acquiring the style features of the style information to be migrated through a style editor; The target multimodal data is generated according to the style feature and the initial multimodal data.

7. A multimodal data generating device, characterized in that: include: Question module, used to obtain user question data; An analysis module, used to extract topics from the user question data to obtain topic information; and performing a logical structure analysis on the user question data to obtain logical structure information of the user question data; A retrieval module, used to retrieve text materials, picture materials and video materials corresponding to the user question data in a multimodal database according to the subject information and the editor structure information; An integration module, used for integrating the text data, the picture data and the video data to obtain initial multimodal data; the initial multimodal data includes at least one target text, at least one target picture and at least one target video; The style module is used to perform style migration processing on all the target texts, all the target images and all the target videos to obtain target multimodal data.

8. The multimodal data generating device according to claim 7, characterized in that: The analysis module comprises: A topic identification unit, used to perform multi-topic identification on the user question data through a topic identification model to obtain a topic matrix; the topic matrix includes at least one topic word and at least one topic probability value; The topic information unit is used to generate the topic information according to all the topic words and topic probability values.

9. A computer device comprising a memory, a processor, and computer-readable instructions stored in the memory and executable on the processor, characterized in that: When the processor executes the computer-readable instructions, the multimodal data generating method according to any one of claims 1 to 6 is implemented.

10. One or more computer-readable storage media storing computer-readable instructions, characterized in that: When the computer-readable instructions are executed by one or more processors, the one or more processors are caused to perform the multimodal data generating method according to any one of claims 1 to 6.

Images