User subconscious audio and video generation method and system based on multimodal large model

Abstract

The present application provides a method and system for generating user subconscious audio and video based on a multimodal large model, the method comprising: obtaining brain wave fluctuation data within a preset time period; inputting the brain wave fluctuation data into a preset vectorization model, so that the vectorization model extracts corresponding entity names from a preset vector database; determining corresponding entity nodes from a preset multimodal vector knowledge graph according to the entity names; extracting corresponding text, video and picture resources from the multimodal vector knowledge graph according to the entity nodes corresponding to the entity names to obtain a resource set; inputting the resource set into a preset multimodal large model, so that the multimodal large model generates corresponding audio and video clips according to the resource set, and splicing each of the audio and video clips to generate the user subconscious audio and video, providing an efficient user interaction experience.

Description

Technical Field

[0001] The present application relates to the field of artificial intelligence and audio and video generation technology, and in particular to a method and system for generating user subconscious audio and video based on a multimodal large model. Background Art

[0002] Retrieval-Augmented Generation (RAG) is a technique that combines information retrieval and text generation, aiming to improve the accuracy and relevance of large language models (LLMs). RAG technology effectively alleviates the hallucination problem, improves the speed of knowledge updating, and enhances the traceability of content generation by retrieving relevant information from a wide range of document databases before the language model generates answers, and then using this information to guide the generation process. The core of RAG technology includes three main parts: retrieval, augmentation, and generation. First, the system retrieves relevant information from the knowledge base based on the user query; then, the retrieved information is converted into vectors in a high-dimensional space and stored in the vector database; finally, the generation model (such as LLMs) uses the retrieved information to generate text replies. The development of RAG technology has gone through several key stages. From the initial naive RAG to the current modular RAG, the technology has continued to advance, providing more flexibility and functional modules, such as query search engines, fusion of multiple answers, etc. The application range of RAG technology is wide, not only limited to question-answering systems, but also extended to various tasks such as recommendation systems, information extraction, and report generation. The implementation of RAG technology involves multiple steps, including the preparation of knowledge documents, embedding of embedding models, establishment of vector databases, query retrieval, and production of answers. In practical applications, RAG technology can effectively solve the problems faced by LLMs in terms of timeliness, data security, and knowledge updating, and provides an economical, efficient, and data-secure solution. In general, RAG technology is a major trend in the field of AI. It makes large language models more practical and credible in practical applications through retrieval enhancement.

[0003] However, the current technology for RAG mainly stays in the processing of text data, and the data sources are basically text data, audio, video and other data. There is currently no RAG application for brain waves. At the same time, the latest GraphRAG is also for splitting and extracting long texts, and there is no Graph Embedding application directly for human subconscious brain wave signals. In the process of people's creation or thinking, the inspiration generated at a certain moment may contain a lot of useful information, but people often cannot record the inspiration generated in time. Therefore, the method of materializing people's spontaneous and instantaneous "ideas" according to brain wave signals and generating related text and audio and video materials has great practical significance in learning and work. Summary of the invention

[0004] In response to the above technical problems, the present application provides a method and system for generating user subconscious audio and video based on a multimodal large model, which generates corresponding audio and video data according to the user's brain wave data and provides an efficient user interaction experience.

[0005] In a first aspect, the present application provides a method for generating user subconscious audio and video based on a multimodal large model, comprising:

[0006] Obtain brain wave fluctuation data within a preset time period;

[0007] Inputting the brain wave fluctuation data into a preset vectorization model, so that the vectorization model extracts corresponding entity names from a preset vector database according to the brain wave fluctuation data;

[0008] According to the entity names, a corresponding number of entity nodes are determined from a preset multimodal vector knowledge graph, wherein the vector database and the multimodal vector knowledge graph are based on RAG technology and are constructed through multiple observation experiments, the vector database contains the corresponding relationship between each brain wave vector and each entity name, the multimodal vector knowledge graph contains each entity node, and the entity node contains text, video and picture resources;

[0009] According to the entity nodes corresponding to the entity names, corresponding text, video and picture resources are extracted from the multimodal vector knowledge graph to obtain a resource set;

[0010] The resource set is input into a preset multimodal large model, so that the multimodal large model generates a corresponding number of audio and video clips according to the resource set, and each of the audio and video clips is spliced ​​to generate the user's subconscious audio and video.

[0011] The embodiment of the present application provides a method for generating user subconscious audio and video based on a multimodal large model. After obtaining the brain wave fluctuation data, the brain wave fluctuation data is vectorized through a vectorization model, and the corresponding several entity names are extracted from a preset vector database, realizing the conversion of brain wave data to entity names. Then, according to the several entity names, the corresponding several entity nodes are searched from the multimodal vector knowledge base to obtain the text, video and picture resources pre-stored in the entity nodes. Finally, through the multimodal large model, the corresponding several audio and video clips are generated according to the resource set, and spliced ​​into the user's subconscious audio and video, realizing the conversion from entity names to audio and video materials. The present application can generate the inspiration or picture generated by the user at a certain moment according to the brain wave signal, thereby helping all walks of life to further liberate thinking limitations, release innovation capabilities, and provide an efficient user interaction experience.

[0012] Furthermore, the step of obtaining brain wave fluctuation data within a preset time period includes:

[0013] Collecting brain wave fluctuation data that occurs when a user observes an entity through a brain wave collection device;

[0014] The brain wave fluctuation data is subjected to filtering, denoising and jitter signal amplification operations, thereby amplifying the jitter characteristics of the brain wave fluctuation data.

[0015] In an embodiment of the present application, data preprocessing is performed on brain wave fluctuation data through filtering, denoising and jitter signal amplification operations to amplify the jitter characteristics of the brain wave fluctuation data, making the characteristics of entity observation jitter easier to appear, thereby improving the accuracy of subsequent extraction of entity names and the accuracy of generating user subconscious audio and video.

[0016] In a possible implementation, the vectorization model extracts corresponding entity names from a preset vector database according to the brain wave fluctuation data, including:

[0017] identifying a number of brain wave vectors from the brain wave fluctuation data according to jitter characteristics of the brain wave fluctuation data;

[0018] Combining the plurality of brain wave vectors into different fusion vectors by means of permutations and combinations;

[0019] Input each of the fused vectors into the vector database for comparison query to confirm whether there is a corresponding entity name in the vector database, and if so, extract the corresponding entity name from the vector database;

[0020] After the comparison query is completed, several entity names are obtained.

[0021] The embodiment of the present application provides a method for extracting entity names based on brain wave fluctuation data. When a user observes the objective world, he or she does not focus on a single entity node. This process generally involves the parallel observation of multiple entities. At this time, the brain wave fluctuation data will show multiple jitters. For example, observing an apple or a peeler alone may lead to associations between each other. However, if an apple and a peeler are observed at the same time, the brain wave fluctuation data will show a fusion jitter, and this fusion jitter will actually map an entity, such as a fruit salad or apple juice. Therefore, the embodiment of the present application designs a multi-path parallel entity collection mechanism. After identifying a number of brain wave vectors, the brain wave vectors are combined into different fusion vectors by permutations and combinations, and the fusion vectors are input into a vector database for comparison and query, so as to explore as many of the user's subconscious entity mappings as possible, thereby improving the quality of the final audio and video products.

[0022] In a possible implementation, the vector database and the multimodal vector knowledge graph are constructed and obtained through multiple observation experiments based on the RAG technology, including:

[0023] Extracting a number of trimodal data from a preset objective world entity set based on the RDF triple data format to construct an objective world knowledge graph, wherein the objective world knowledge graph includes a number of entity nodes, corresponding entity names, picture resources, and video resources;

[0024] Collect the brain wave data generated by each observer observing each entity to construct a training data set;

[0025] Extracting the jitter frequency, amplitude and phase of each training data in the training data set, and classifying and standardizing the data according to the population labels of the observers corresponding to the data to obtain each standardized feature data;

[0026] Performing two-dimensional vectorization processing on each of the standardized feature data to obtain each vector data corresponding to each entity name in the objective world knowledge graph, and then constructing the vector database;

[0027] Combining each of the vector data with a corresponding entity name and a corresponding crowd label to construct combined data in a label-entity-vector format;

[0028] The combined data is combined with each entity node in the objective world knowledge graph to construct the multimodal vector knowledge graph.

[0029] The embodiment of the present application provides a method for constructing a vector database and a multimodal vector knowledge graph. First, a number of trimodal data are extracted from a preset objective world entity set based on the RDF triple data format to construct an objective world knowledge graph as existing objective world knowledge. Then, the connection between the EEG data and the entities in the objective world is established through multiple observation experiments, wherein the EEG data is classified and standardized according to the observer's population label. This is because even for the same entity, people of different ages, genders, occupations, and educational backgrounds will produce different EEG data when observing the entity. Therefore, it is necessary to classify the acquired EEG data according to the observer's population label to improve the accuracy and rationality of the vector database and the multimodal vector knowledge graph. Finally, the vector database is constructed through the entity names and corresponding vector data in the objective world knowledge graph; the combined data in the label-entity-vector format is combined with the entity nodes in the objective world knowledge graph to construct the multimodal vector knowledge graph, thereby further transforming the existing objective world knowledge, enriching the node attributes of each entity node, improving the accuracy of matching brain wave data with entity nodes, and providing a data basis for subsequent audio and video generation.

[0030] In a possible implementation, after the multimodal macro model generates the user's subconscious audio and video, the user's annotation information is obtained and the multimodal vector knowledge graph or the multimodal macro model is adjusted, including:

[0031] If the annotation information is an entity generation error, adjusting the entity weight in the multimodal large model or adjusting the overall vector representation of the entity generated by the multimodal large model;

[0032] If the annotation information is a scene generation error, performing enhanced training on the multimodal large model;

[0033] If the annotation information is a logical error, the corresponding relationship in the multimodal vector knowledge graph is adjusted or the logical reasoning performance of the multimodal large model is adjusted.

[0034] In an embodiment of the present application, after the model generates the user's subconscious audio and video, the multimodal vector knowledge graph or the multimodal large model is adjusted according to the annotation information fed back by the user to further improve the generation quality and accuracy of the audio and video.

[0035] Furthermore, if the annotation information is an entity generation error, adjusting the entity weight in the multimodal large model or adjusting the overall vector representation of the entity generated by the multimodal large model includes:

[0036] Get the user's annotated entity name;

[0037] Comparing the annotated entity name with the entity name in the user's subconscious audio and video to see whether they are similar;

[0038] If they are similar, increasing the entity weight of the labeled entity name in the multimodal large model;

[0039] If not, the overall vector representation of the entity generated by the multimodal large model is adjusted using data in a preset format.

[0040] Furthermore, if the annotation information is a scene generation error, the multimodal large model is subjected to enhanced training, including:

[0041] Constructing a preset number of question-answer pair data, wherein the question-answer pair data includes a correct scenario and a corresponding correct entity name;

[0042] The multimodal large model is reinforced trained using the question-answer pair data.

[0043] Furthermore, if the annotation information is a logical relationship error, adjusting the corresponding relationship in the multimodal vector knowledge graph or adjusting the logical reasoning performance of the multimodal large model includes:

[0044] Confirm whether the logical relationship in the user's subconscious audio and video is correct in the multimodal knowledge graph, and if not, correct the relationship in the multimodal knowledge graph;

[0045] Inputting each target entity name involved in the logical relationship into the multimodal large model, so that the multimodal large model performs logical reasoning according to each target entity name to generate a logical reasoning result;

[0046] Confirm whether the logical reasoning result is correct. If not, perform intensive training on the logical reasoning part of the multimodal large model.

[0047] In a second aspect, accordingly, the present application provides a user subconscious audio and video generation system based on a multimodal large model, including an acquisition module, an entity extraction module, an entity node determination module, a resource extraction module, and a generation module;

[0048] Wherein, the acquisition module is used to acquire brain wave fluctuation data within a preset time period;

[0049] The entity extraction module is used to input the brain wave fluctuation data into a preset vectorization model, so that the vectorization model extracts corresponding entity names from a preset vector database according to the brain wave fluctuation data;

[0050] The entity node determination module determines a number of corresponding entity nodes from a preset multimodal vector knowledge graph according to the number of entity names, wherein the vector database and the multimodal vector knowledge graph are constructed through multiple observation experiments, the vector database contains the corresponding relationship between each brain wave vector and each entity name, the multimodal vector knowledge graph contains each entity node, and the entity node contains text, video and picture resources;

[0051] The resource extraction module is used to extract corresponding text, video and picture resources from the multimodal vector knowledge graph according to the entity nodes corresponding to the entity names to obtain a resource set;

[0052] The generation module is used to input the resource set into a preset multimodal large model, so that the multimodal large model generates a corresponding number of audio and video clips according to the resource set, and splices each of the audio and video clips to generate the user's subconscious audio and video.

[0053] Furthermore, the vector database and multimodal vector knowledge graph are constructed and obtained through multiple observation experiments based on the RAG technology, including:

[0054] Extracting a number of trimodal data from a preset objective world entity set based on the RDF triple data format to construct an objective world knowledge graph, wherein the objective world knowledge graph includes a number of entity nodes, corresponding entity names, picture resources, and video resources;

[0055] Collect the brain wave data generated by each observer observing each entity to construct a training data set;

[0056] Extracting the jitter frequency, amplitude and phase of each training data in the training data set, and classifying and standardizing the data according to the population labels of the observers corresponding to each data to obtain each standardized feature data;

[0057] Performing two-dimensional vectorization processing on each of the standardized feature data to obtain each vector data corresponding to each entity name in the objective world knowledge graph, and then constructing the vector database;

[0058] Combining each of the vector data with a corresponding entity name and a corresponding crowd label to construct combined data in a label-entity-vector format;

[0059] The combined data is combined with each entity node in the objective world knowledge graph to construct the multimodal vector knowledge graph. BRIEF DESCRIPTION OF THE DRAWINGS

[0060] Figure 1: A flow chart of a method for generating user subconscious audio and video based on a multimodal large model provided in an embodiment of the present application.

[0061] Figure 2 : A flow chart of a multi-channel parallel entity detection mechanism in a method for generating user subconscious audio and video based on a multimodal large model provided in an embodiment of the present application.

[0062] Figure 3 : A schematic diagram of the process of constructing the vector database and the multimodal vector knowledge graph in a method for generating user subconscious audio and video based on a multimodal large model provided in an embodiment of the present application.

[0063] Figure 4 : A flow chart of a method for generating user subconscious audio and video with a user feedback mechanism provided in an embodiment of the present application.

[0064] Figure 5 : A structural diagram of a user subconscious audio and video generation system based on a multimodal large model provided in an embodiment of the present application. DETAILED DESCRIPTION

[0065] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0066] It should be noted that the step numbers in the text are only for the convenience of explanation of the specific embodiments and do not serve to limit the order in which the steps are executed. In the description of this application, the terms "first" and "second" are only used for descriptive purposes and cannot be understood as indicating or implying relative importance or implicitly indicating the number of the indicated technical features. Therefore, the features defined as "first" and "second" may explicitly or implicitly include one or more of the features.

[0067] Embodiment 1:

[0068] like Figure 1 As shown, embodiment 1 provides a method for generating user subconscious audio and video based on a multimodal large model, including steps S1-S5:

[0069] Step S1, obtaining brain wave fluctuation data within a preset time period;

[0070] Step S2, inputting the brain wave fluctuation data into a preset vectorization model, so that the vectorization model extracts corresponding entity names from a preset vector database according to the brain wave fluctuation data;

[0071] Step S3, determining corresponding entity nodes from a preset multimodal vector knowledge graph according to the entity names, wherein the vector database and the multimodal vector knowledge graph are based on RAG technology and are constructed through multiple observation experiments, the vector database contains the corresponding relationship between each brain wave vector and each entity name, the multimodal vector knowledge graph contains each entity node, and the entity node contains text, video and picture resources;

[0072] Step S4: extract corresponding text, video and picture resources from the multimodal vector knowledge graph according to the entity nodes corresponding to the entity names to obtain a resource set;

[0073] Step S5: input the resource set into a preset multimodal large model, so that the multimodal large model generates a corresponding number of audio and video clips according to the resource set, and splices each of the audio and video clips to generate the user's subconscious audio and video.

[0074] The embodiment of the present application provides a method for generating user subconscious audio and video based on a multimodal large model. After obtaining the brain wave fluctuation data, the brain wave fluctuation data is vectorized through a vectorization model, and the corresponding several entity names are extracted from a preset vector database, realizing the conversion of brain wave data to entity names. Then, according to the several entity names, the corresponding several entity nodes are searched from the multimodal vector knowledge base to obtain the text, video and picture resources pre-stored in the entity nodes. Finally, through the multimodal large model, the corresponding several audio and video clips are generated according to the resource set, and spliced ​​into the user's subconscious audio and video, realizing the conversion from entity names to audio and video materials. The present application can generate the inspiration or picture generated by the user at a certain moment according to the brain wave signal, thereby helping all walks of life to further liberate thinking limitations, release innovation capabilities, and provide an efficient user interaction experience.

[0075] Furthermore, in step S1, obtaining brain wave fluctuation data within a preset time period includes:

[0076] Collecting brain wave fluctuation data that occurs when a user observes an entity through a brain wave collection device;

[0077] The brain wave fluctuation data is subjected to filtering, denoising and jitter signal amplification operations, thereby amplifying the jitter characteristics of the brain wave fluctuation data.

[0078] In an embodiment of the present application, data preprocessing is performed on brain wave fluctuation data through filtering, denoising and jitter signal amplification operations to amplify the jitter characteristics of the brain wave fluctuation data, making the characteristics of entity observation jitter easier to appear, thereby improving the accuracy of subsequent extraction of entity names and the accuracy of generating user subconscious audio and video.

[0079] In a possible implementation, in step S2, the vectorized model extracts corresponding entity names from a preset vector database according to the brain wave fluctuation data, including:

[0080] identifying a number of brain wave vectors from the brain wave fluctuation data according to jitter characteristics of the brain wave fluctuation data;

[0081] Combining the plurality of brain wave vectors into different fusion vectors by means of permutations and combinations;

[0082] Input each of the fused vectors into the vector database for comparison query to confirm whether there is a corresponding entity name in the vector database, and if so, extract the corresponding entity name from the vector database;

[0083] After the comparison query is completed, several entity names are obtained.

[0084] The vectorization model may be a BGE model. BGE (BAAI General Embedding) is a general text embedding model developed by Beijing Zhiyuan Artificial Intelligence Research Institute, which can map any text to a low-dimensional dense vector. This model has been widely used in text vectorization tasks and has performed well in the C-MTEB Chinese Ranking and MTEB Ranking.

[0085] In a preferred embodiment, Figure 2 As shown, a multi-way parallel entity detection mechanism is provided. When multiple entities appear, we will design all possible entity associations in parallel by permutations and combinations, and then perform fusion searches. If the fused entity exists in the vector database and the multimodal vector knowledge graph, it will be collected once and removed from the permutations and combinations through the feedback mechanism. Finally, all possible entities and fused entities are extracted in a multi-way parallel manner to form a full vector set. For example, if entities A, B, and C are combined to form a fused entity ABC, then there is no need to test BAC\BCA\CAB\CBA because the composition is the same and the fused entity results are the same. The purpose of the removal is to ensure that the same resources will not be repeatedly extracted when extracting corresponding resources from the multimodal vector knowledge graph, resulting in a high degree of repetitiveness in the final generated results.

[0086] The embodiment of the present application provides a method for extracting entity names based on brain wave fluctuation data. When a user observes the objective world, he or she does not focus on a single entity node. This process generally involves the parallel observation of multiple entities. At this time, the brain wave fluctuation data will show multiple jitters. For example, observing an apple or a peeler alone may lead to associations between each other. However, if an apple and a peeler are observed at the same time, the brain wave fluctuation data will show a fusion jitter, and this fusion jitter will actually map an entity, such as a fruit salad or apple juice. Therefore, the embodiment of the present application designs a multi-path parallel entity collection mechanism. After identifying a number of brain wave vectors, the brain wave vectors are combined into different fusion vectors by permutations and combinations, and the fusion vectors are input into a vector database for comparison and query, so as to explore as many of the user's subconscious entity mappings as possible, thereby improving the quality of the final audio and video products.

[0087] In a possible implementation, in step S3, the vector database and the multimodal vector knowledge graph are constructed through multiple observation experiments based on the RAG technology, including:

[0088] Extracting a number of trimodal data from a preset objective world entity set based on the RDF triple data format to construct an objective world knowledge graph, wherein the objective world knowledge graph includes a number of entity nodes, corresponding entity names, picture resources, and video resources;

[0089] Collect the brain wave data generated by each observer observing each entity to construct a training data set;

[0090] Extracting the jitter frequency, amplitude and phase of each training data in the training data set, and classifying and standardizing the data according to the population labels of the observers corresponding to each data to obtain each standardized feature data;

[0091] Performing two-dimensional vectorization processing on each of the standardized feature data to obtain each vector data corresponding to each entity name in the objective world knowledge graph, and then constructing the vector database;

[0092] Combining each of the vector data with a corresponding entity name and a corresponding crowd label to construct combined data in a label-entity-vector format;

[0093] The combined data is combined with each entity node in the objective world knowledge graph to construct the multimodal vector knowledge graph.

[0094] In a preferred embodiment, the specific process of constructing the vector database and the multimodal vector knowledge graph is as follows: Figure 3As shown. First, according to objective needs, it is necessary to extract RDF triples for objective world knowledge to generate an objective world knowledge graph supported by three modal data types: text, picture, and video. Pictures and videos are stored as node attributes. Among them, the objective world knowledge refers to the RDF open source data provided by Wikipedia, namely entity-relationship-entity. The RDF triple is a basic data structure in the Resource Description Framework (RDF), which is used to describe the relationship between resources. An RDF triple consists of three parts: Subject: represents the resource to be described, which can be a web page, picture, document, or any other type of resource. In RDF, subjects are usually identified by uniform resource identifiers (URIs). Predicate: represents a certain attribute of a subject or its relationship with another resource. A predicate is usually also a URI, which is used to describe the relationship between a subject and an object. Object: represents a resource or data associated with a subject through a predicate. An object can be the URI of another resource or a literal value (such as a string, a number, etc.).

[0095] Secondly, a large amount of training data is needed, that is, people of different ages, genders, occupations, and educational backgrounds are asked to observe the same entity, and their perception of the entity and brain wave data are collected. After accumulating a large amount of observation data, these brain wave data (EEG) need to be preprocessed and feature extracted. The original EEG contains jitter during the observation time, and it is necessary to pay attention to the effective jitter situation to filter and filter, remove noise, and amplify the jitter signal to make the characteristics of the entity observation jitter easier to show.

[0096] The next step is to extract useful features from the preprocessed EEG data, namely the jitter frequency, amplitude, and phase of the EEG data in the time series of the observation moment, and then classify and standardize them according to different populations. Clustering (KMEANS) or SVM algorithms can be used here to collect subtle differences between different populations at the same time of observation.

[0097] After the extraction is completed, the features are two-dimensionally vectorized according to the labels of different groups of people to generate a large amount of label-entity-vector combination data. Then, combined with the pre-stored image and video resources in the objective world knowledge graph, a multimodal vector knowledge graph is reconstructed.

[0098] After completing the construction of the multimodal vector knowledge graph, we can also extract additional influencing factors such as the emotional state and physical condition of different groups of people before observing the entity, thereby further enriching the node attributes of the multimodal vector knowledge graph.

[0099] The embodiment of the present application provides a method for constructing a vector database and a multimodal vector knowledge graph. First, a number of trimodal data are extracted from a preset objective world entity set based on the RDF triple data format to construct an objective world knowledge graph as existing objective world knowledge. Then, the connection between the EEG data and the entities in the objective world is established through multiple observation experiments, wherein the EEG data is classified and standardized according to the observer's population label. This is because even for the same entity, people of different ages, genders, occupations, and educational backgrounds will produce different EEG data when observing the entity. Therefore, it is necessary to classify the acquired EEG data according to the observer's population label to improve the accuracy and rationality of the vector database and the multimodal vector knowledge graph. Finally, the vector database is constructed through the entity names and corresponding vector data in the objective world knowledge graph; the combined data in the label-entity-vector format is combined with the entity nodes in the objective world knowledge graph to construct the multimodal vector knowledge graph, thereby further transforming the existing objective world knowledge, enriching the node attributes of each entity node, improving the accuracy of matching brain wave data with entity nodes, and providing a data basis for subsequent audio and video generation.

[0100] In a possible implementation, after the multimodal macro model generates the user's subconscious audio and video, the user's annotation information is obtained and the multimodal vector knowledge graph or the multimodal macro model is adjusted, including:

[0101] If the annotation information is an entity generation error, adjusting the entity weight in the multimodal large model or adjusting the overall vector representation of the entity generated by the multimodal large model;

[0102] If the annotation information is a scene generation error, performing enhanced training on the multimodal large model;

[0103] If the annotation information is a logical error, the corresponding relationship in the multimodal vector knowledge graph is adjusted or the logical reasoning performance of the multimodal large model is adjusted.

[0104] In an embodiment of the present application, after the model generates the user's subconscious audio and video, the multimodal vector knowledge graph or the multimodal large model is adjusted according to the annotation information fed back by the user to further improve the generation quality and accuracy of the audio and video.

[0105] In a preferred embodiment, in steps S3-S5, after all possible entities and fused entities are extracted in a multi-path parallel manner to form a full vector set, based on the full vector set, the text information, picture information, and video information of each entity node can be correspondingly extracted, and due to the mechanism of multi-path parallel entity detection, many hidden entities are also excavated, and the text, pictures, and videos of all entities are collected and then injected into the open source base large model. For the selection of the large model, it should be a large model with a parameter amount of not less than 70b and strong Chinese ability. The large model will construct a set of reasonable text and video collections based on the provided text content, picture content, and video content. In the early stage of model use, due to the small amount of such data injected and the large model has not been fine-tuned, the effect will be unsatisfactory. Therefore, this embodiment provides a user feedback mechanism, such as Figure 4 As shown, with Figure 4 As user annotations in the dataset continue to accumulate, the model will have a deeper understanding of the task objectives of subconscious recognition and audio and video content generation, providing better quality creative results.

[0106] Furthermore, if the annotation information is an entity generation error, adjusting the entity weight in the multimodal large model or adjusting the overall vector representation of the entity generated by the multimodal large model includes:

[0107] Get the user's annotated entity name;

[0108] Comparing the annotated entity name with the entity name in the user's subconscious audio and video to see whether they are similar;

[0109] If they are similar, increasing the entity weight of the labeled entity name in the multimodal large model;

[0110] If not, the overall vector representation of the entity generated by the multimodal large model is adjusted using data in a preset format.

[0111] If the entity is generated incorrectly, it is necessary to compare the annotated entity and the generated entity to see if they are similar. If they are similar, the weight of the annotated entity is increased to improve the probability of RAG retrieval. If they are not similar, it is necessary to construct data in the format of {query, pos, neg} through llm to fine-tune the overall vector representation of the generated entity.

[0112] Furthermore, if the annotation information is a scene generation error, the multimodal large model is subjected to enhanced training, including:

[0113] Constructing a preset number of question-answer pair data, wherein the question-answer pair data includes a correct scenario and a corresponding correct entity name;

[0114] The multimodal large model is reinforced trained using the question-answer pair data.

[0115] Scene generation errors. This type of problem is generally a problem with the generation of LLM. For scene generation errors, we need to construct QA question-and-answer data to fine-tune the large model, that is, the correct scenes and correct entities, construct no less than 1,000 question-and-answer data, and retrain the LLM.

[0116] Furthermore, if the annotation information is a logical relationship error, adjusting the corresponding relationship in the multimodal vector knowledge graph or adjusting the logical reasoning performance of the multimodal large model includes:

[0117] Confirm whether the logical relationship in the user's subconscious audio and video is correct in the multimodal knowledge graph, and if not, correct the relationship in the multimodal knowledge graph;

[0118] Inputting each target entity name involved in the logical relationship into the multimodal large model, so that the multimodal large model performs logical reasoning according to each target entity name to generate a logical reasoning result;

[0119] Confirm whether the logical reasoning result is correct. If not, perform intensive training on the logical reasoning part of the multimodal large model.

[0120] If the logical relationship is wrong, fine-tuning is required from two dimensions. First, confirm whether the relationship is correct in the multimodal knowledge graph. If it is consistent with general facts, it is not recommended to change it. If it is not consistent, the relationship in the knowledge graph needs to be corrected. Secondly, it is necessary to confirm whether the logical relationship is correct in the process of generating the large model. Other influencing factors can be eliminated, and then the target entity pair is taken alone to let the large model perform logical inference. If it is wrong, it needs to be fine-tuned. If it is correct, no processing is required.

[0121] In a preferred embodiment, the present application can be applied to dream reproduction. On the basis of the gradual improvement of the above-mentioned large model, since EEG data not only appears when a person is awake, but also has very intense brain wave fluctuations after falling asleep. Through daily links to brain wave data and entities, we can monitor the brain waves of sleeping people, and by collecting the fluctuations of human brain waves during sleep at night, we can use the above process to reproduce human dreams in audio and video mode. It should be noted that people cannot give timely feedback when they fall asleep, so it is best to use a model that has been fine-tuned and experimented to process the sleeping scene; people's dreams often have counterintuitive phenomena, that is, things that cannot happen in the objective world. If it is for dream generation, it is necessary to contact the logical restrictions of the fusion entity to allow unrelated entities to merge with each other.

[0122] In a preferred embodiment, the present application can be applied to personalized customization. After the model is perfected, since the model is mainly trained on the objective world and most phenomena, we can enrich the basic vector knowledge base by giving some personal observation experience, thereby updating a vector knowledge base and private knowledge graph that belongs only to ourselves. The vectorization technology itself is an end-to-end encryption method, and the data storage locally can fully guarantee the security of the data, thereby realizing the customization of the personalized audio and video generation system.

[0123] The embodiments of the present application capture and analyze brain waves to deeply understand the subconscious needs of users, generate highly personalized audio and video content, and provide an efficient and non-intrusive user interaction experience. It can not only accurately identify the user's emotional state and enhance the emotional resonance of the content, but also make intelligent recommendations based on the knowledge graph while paying attention to user privacy protection. This innovative technology combines cutting-edge brain science and artificial intelligence, pushing the boundaries of personalized services and intelligent recommendations, and providing users with an unprecedented intelligent experience.

[0124] Embodiment 2:

[0125] like Figure 5 As shown, embodiment 2 provides a user subconscious audio and video generation system based on a multimodal large model, including an acquisition module 10, an entity extraction module 20, an entity node determination module 30, a resource extraction module 40 and a generation module 50;

[0126] Wherein, the acquisition module 10 is used to acquire brain wave fluctuation data within a preset time period;

[0127] The entity extraction module 20 is used to input the brain wave fluctuation data into a preset vectorization model, so that the vectorization model extracts corresponding entity names from a preset vector database according to the brain wave fluctuation data;

[0128] The entity node determination module 30 determines corresponding entity nodes from a preset multimodal vector knowledge graph according to the entity names, wherein the vector database and the multimodal vector knowledge graph are constructed through multiple observation experiments, the vector database contains the corresponding relationship between each brain wave vector and each entity name, the multimodal vector knowledge graph contains each entity node, and the entity node contains text, video and picture resources;

[0129] The resource extraction module 40 is used to extract corresponding text, video and picture resources from the multimodal vector knowledge graph according to the entity nodes corresponding to the entity names to obtain a resource set;

[0130] The generation module 50 is used to input the resource set into a preset multimodal large model, so that the multimodal large model generates a corresponding number of audio and video clips according to the resource set, and splices each of the audio and video clips to generate the user's subconscious audio and video.

[0131] Furthermore, the vector database and multimodal vector knowledge graph are constructed and obtained through multiple observation experiments based on the RAG technology, including:

[0132] Extracting a number of trimodal data from a preset objective world entity set based on the RDF triple data format to construct an objective world knowledge graph, wherein the objective world knowledge graph includes a number of entity nodes, corresponding entity names, picture resources, and video resources;

[0133] Collect the brain wave data generated by each observer observing each entity to construct a training data set;

[0134] Extracting the jitter frequency, amplitude and phase of each training data in the training data set, and classifying and standardizing the data according to the population labels of the observers corresponding to each data to obtain each standardized feature data;

[0135] Performing two-dimensional vectorization processing on each of the standardized feature data to obtain each vector data corresponding to each entity name in the objective world knowledge graph, and then constructing the vector database;

[0136] Combining each of the vector data with a corresponding entity name and a corresponding crowd label to construct combined data in a label-entity-vector format;

[0137] The combined data is combined with each entity node in the objective world knowledge graph to construct the multimodal vector knowledge graph.

[0138] The embodiment of the present application provides a user subconscious audio and video generation system based on a multimodal large model. After obtaining the brain wave fluctuation data, the brain wave fluctuation data is vectorized through a vectorization model, and the corresponding several entity names are extracted from a preset vector database, realizing the conversion of brain wave data to entity names. Then, according to the several entity names, the corresponding several entity nodes are searched from the multimodal vector knowledge base to obtain the text, video and picture resources pre-stored in the entity nodes. Finally, through the multimodal large model, the corresponding several audio and video clips are generated according to the resource set, and spliced ​​into the user's subconscious audio and video, realizing the conversion from entity names to audio and video materials. The present application can generate inspiration or pictures generated by the user at a certain moment according to brain wave signals, thereby helping all walks of life to further liberate thinking limitations, release innovation capabilities, and provide efficient user interaction experience.

[0139] The more detailed working principle and step flow of this embodiment can refer to, but are not limited to, the relevant records of Embodiment 1.

[0140] The specific embodiments described above further describe the purpose, technical solutions and beneficial effects of the present application in detail. It should be understood that the above description is only a specific embodiment of the present application and is not intended to limit the scope of protection of the present application. It is particularly pointed out that for those skilled in the art, any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application should be included in the scope of protection of the present application.

Claims

1. A method for generating user subconscious audio and video based on a multimodal large model, characterized in that: include: Obtain brain wave fluctuation data within a preset time period; Inputting the brain wave fluctuation data into a preset vectorization model, so that the vectorization model extracts corresponding entity names from a preset vector database according to the brain wave fluctuation data; According to the entity names, a corresponding number of entity nodes are determined from a preset multimodal vector knowledge graph, wherein the vector database and the multimodal vector knowledge graph are based on RAG technology and are constructed through multiple observation experiments, the vector database contains the corresponding relationship between each brain wave vector and each entity name, the multimodal vector knowledge graph contains each entity node, and the entity node contains text, video and picture resources; According to the entity nodes corresponding to the entity names, corresponding text, video and picture resources are extracted from the multimodal vector knowledge graph to obtain a resource set; Inputting the resource set into a preset multimodal large model, so that the multimodal large model generates a corresponding number of audio and video clips according to the resource set, and splicing the audio and video clips to generate the user's subconscious audio and video; The RAG technology is based on constructing the vector database and the multimodal vector knowledge graph through multiple observation experiments, including: extracting a number of trimodal data from a preset objective world entity set based on the RDF triple data format to construct an objective world knowledge graph, wherein the objective world knowledge graph contains a number of entity nodes, corresponding entity names, picture resources and video resources; collecting various brain wave data generated by each observer observing each entity to construct a training data set; extracting the jitter frequency, amplitude and phase of each training data in the training data set, and classifying and standardizing them according to the crowd labels of the observers corresponding to each data to obtain various standardized feature data; performing two-dimensional vectorization processing on each of the standardized feature data to obtain various vector data corresponding to each entity name in the objective world knowledge graph, and then constructing the vector database; combining each of the vector data with the corresponding entity name and the corresponding crowd label to construct combined data in the label-entity-vector format; combining the combined data with each entity node in the objective world knowledge graph to construct the multimodal vector knowledge graph.

2. A method for generating user subconscious audio and video based on a multimodal large model as claimed in claim 1, characterized in that: The step of obtaining brain wave fluctuation data within a preset time period includes: Collecting brain wave fluctuation data that occurs when a user observes an entity through a brain wave collection device; The brain wave fluctuation data is subjected to filtering, denoising and jitter signal amplification operations, thereby amplifying the jitter characteristics of the brain wave fluctuation data.

3. The method for generating user subconscious audio and video based on a multimodal large model as claimed in claim 1, characterized in that: The vectorized model extracts corresponding entity names from a preset vector database according to the brain wave fluctuation data, including: identifying a number of brain wave vectors from the brain wave fluctuation data according to jitter characteristics of the brain wave fluctuation data; Combining the plurality of brain wave vectors into different fusion vectors by means of permutations and combinations; Input each of the fused vectors into the vector database for comparison query to confirm whether there is a corresponding entity name in the vector database, and if so, extract the corresponding entity name from the vector database; After the comparison query is completed, several entity names are obtained.

4. A method for generating user subconscious audio and video based on a multimodal large model as described in any one of claims 1 to 3, characterized in that: After the multimodal large model generates the user's subconscious audio and video, the user's annotation information is obtained and the multimodal vector knowledge graph or the multimodal large model is adjusted, including: If the annotation information is an entity generation error, adjusting the entity weight in the multimodal large model or adjusting the overall vector representation of the entity generated by the multimodal large model; If the annotation information is a scene generation error, performing enhanced training on the multimodal large model; If the annotation information is a logical error, the corresponding relationship in the multimodal vector knowledge graph is adjusted or the logical reasoning performance of the multimodal large model is adjusted.

5. A method for generating user subconscious audio and video based on a multimodal large model as claimed in claim 4, characterized in that: If the annotation information is an entity generation error, adjusting the entity weight in the multimodal large model or adjusting the overall vector representation of the entity generated by the multimodal large model includes: Get the user's annotated entity name; Comparing the annotated entity name with the entity name in the user's subconscious audio and video to see whether they are similar; If they are similar, increasing the entity weight of the labeled entity name in the multimodal large model; If not, the overall vector representation of the entity generated by the multimodal large model is adjusted using data in a preset format.

6. A method for generating user subconscious audio and video based on a multimodal large model as claimed in claim 4, characterized in that: If the annotation information is a scene generation error, the multimodal large model is subjected to enhanced training, including: Constructing a preset number of question-answer pair data, wherein the question-answer pair data includes a correct scenario and a corresponding correct entity name; The multimodal large model is reinforced trained using the question-answer pair data.

7. The method for generating user subconscious audio and video based on a multimodal large model as claimed in claim 4, characterized in that: If the annotation information is a logical relationship error, adjusting the corresponding relationship in the multimodal vector knowledge graph or adjusting the logical reasoning performance of the multimodal large model includes: Confirm whether the logical relationship in the user's subconscious audio and video is correct in the multimodal knowledge graph, and if not, correct the relationship in the multimodal knowledge graph; Inputting each target entity name involved in the logical relationship into the multimodal large model, so that the multimodal large model performs logical reasoning according to each target entity name to generate a logical reasoning result; Confirm whether the logical reasoning result is correct. If not, perform intensive training on the logical reasoning part of the multimodal large model.

8. A user subconscious audio and video generation system based on a multimodal large model, characterized in that: It includes an acquisition module, an entity extraction module, an entity node determination module, a resource extraction module and a generation module; Wherein, the acquisition module is used to acquire brain wave fluctuation data within a preset time period; The entity extraction module is used to input the brain wave fluctuation data into a preset vectorization model, so that the vectorization model extracts corresponding entity names from a preset vector database according to the brain wave fluctuation data; The entity node determination module determines corresponding entity nodes from a preset multimodal vector knowledge graph according to the entity names, wherein the vector database and the multimodal vector knowledge graph are constructed based on RAG technology through multiple observation experiments, the vector database contains the corresponding relationship between each brain wave vector and each entity name, the multimodal vector knowledge graph contains each entity node, and the entity node contains text, video and picture resources; The resource extraction module is used to extract corresponding text, video and picture resources from the multimodal vector knowledge graph according to the entity nodes corresponding to the entity names to obtain a resource set; The generation module is used to input the resource set into a preset multimodal large model, so that the multimodal large model generates a corresponding number of audio and video clips according to the resource set, and splices each of the audio and video clips to generate the user's subconscious audio and video; The RAG technology is based on constructing the vector database and the multimodal vector knowledge graph through multiple observation experiments, including: extracting a number of trimodal data from a preset objective world entity set based on the RDF triple data format to construct an objective world knowledge graph, wherein the objective world knowledge graph contains a number of entity nodes, corresponding entity names, picture resources and video resources; collecting various brain wave data generated by each observer observing each entity to construct a training data set; extracting the jitter frequency, amplitude and phase of each training data in the training data set, and classifying and standardizing them according to the crowd labels of the observers corresponding to each data to obtain various standardized feature data; performing two-dimensional vectorization processing on each of the standardized feature data to obtain various vector data corresponding to each entity name in the objective world knowledge graph, and then constructing the vector database; combining each of the vector data with the corresponding entity name and the corresponding crowd label to construct combined data in the label-entity-vector format; combining the combined data with each entity node in the objective world knowledge graph to construct the multimodal vector knowledge graph.

Images