Multi-modal street dance knowledge question and answer method and device based on large language model and medium

Abstract

The invention discloses a multi-modal street dance knowledge question-answering method and device based on a large language model and a medium. The method comprises the steps of collecting original text data related to street dance, preprocessing and labeling the original text data of the street dance, constructing a pre-training language model, performing classification fine adjustment, feature extraction and feature vector classification on a street dance data set with a label, and establishing a database; obtaining different question and answer sample sets; constructing a knowledge base to store and retrieve data, supporting an answer generation engine to generate accurate answers, constructing the answer generation engine, and generating a question and answer model output result according to a retrieval result of the knowledge base. According to the method, the multi-modal street dance knowledge question-answering system is constructed, the text and video data are integrated, and the query accuracy and the user experience are improved by utilizing a large language model and an efficient retrieval tool. And the system supports complex query, has flexibility and expandability, can adapt to constantly changing requirements, and can solve the problems of dispersed street dance knowledge and difficult user query under the real condition.

Description

Technical Field

[0001] The present invention relates to the technical field of question-answering models, and in particular to a multimodal street dance knowledge question-answering method, device, and medium based on a large language model. Background Art

[0002] In recent years, breakthroughs have been made in large-scale models. These are complex artificial neural network models with extremely large parameters, typically comprising hundreds of millions to trillions of parameters. Due to their powerful learning and generalization capabilities, large models have demonstrated broad application potential in numerous fields. For example, DeepSeek's large-scale model can generate high-quality text content such as articles, stories, and news reports. Building on this potential, multimodal large models have been further developed. These models can process not only text data but also information from multiple modalities, such as images, videos, and music.

[0003] In reality, street dance knowledge is fragmented and difficult to search for. Dancers and street dance enthusiasts need to search through multiple channels to get satisfactory answers, and the information they find is often fragmented and unsystematic. No single search system can solve these problems in street dance knowledge search in one place. Summary of the Invention

[0004] The technical problem to be solved by this invention is that in the real world, street dance knowledge is fragmented, making it difficult for users to query and obtain accurate answers. The purpose of this invention is to provide a multimodal street dance knowledge question-and-answer method, device, and medium based on a large language model. By building a multimodal street dance knowledge question-and-answer system that integrates text, video, and other data, and utilizing a large language model and efficient search tools, the accuracy of queries and user experience are improved. The system also supports complex queries, promotes the dissemination of street dance culture, and possesses flexibility and scalability to adapt to ever-changing needs. It can address the problem of fragmented street dance knowledge and difficult user queries in the real world, providing a continuously updated platform for street dance learning.

[0005] The present invention is achieved through the following technical solutions:

[0006] The first aspect of the present invention provides a multimodal street dance knowledge question answering method based on a large language model, comprising the following specific steps:

[0007] Collect original text data related to street dance, clean and format the original text data of street dance, and obtain the street dance dataset;

[0008] Label each text in the street dance dataset;

[0009] Build a pre-trained language model and fine-tune the classification of the annotated street dance dataset;

[0010] Feature extraction of the fine-tuned classified street dance dataset based on semantic understanding;

[0011] Based on the extracted features, feature vectors are classified to obtain different question and answer sample sets;

[0012] Build a knowledge base and an answer generation engine. The knowledge base includes: a structured text retrieval library, a vector-based retrieval database, and a multimodal processing database. The knowledge base is used to store and retrieve data, supporting the answer generation engine to generate accurate answers.

[0013] The answer generation engine is used to retrieve data from a structured text retrieval library, a vector retrieval database, and a multimodal processing database based on the text input of the question-answering model; perform data splicing based on the retrieval results to form a context and generate the question-answering model output result.

[0014] Furthermore, the construction of the pre-trained language model and the classification fine-tuning of the annotated street dance dataset specifically include:

[0015] Build a BERT-base model and add a fully connected layer and a softmax layer on top of the BERT-base model. The fully connected layer is used to reduce the dimension and classify the input data of the pre-trained language model, and map the feature vectors output by BERT to classification labels. The softmax layer is used to output the probability distribution of the categories of the street dance dataset.

[0016] Setting hyperparameters for the pre-trained language model and training the pre-trained language model to obtain an optimized pre-trained language model;

[0017] Based on the optimized pre-trained language model, the classification of the labeled street dance dataset is fine-tuned.

[0018] Furthermore, the data types of the structured text retrieval library are basic knowledge of street dance and dancer resumes; the structured text retrieval library is used to perform keyword matching and score sorting on the acquired data based on the BM25 algorithm, wherein the keyword matching specifically includes:

[0019] Extract data from the structured text retrieval database and perform word segmentation on the data;

[0020] Convert the query to a Boolean query, matching the fields name and awards;

[0021] The search returns the document with the highest score and extracts the relevant fields.

[0022] Furthermore, the data types of the vector-based search database are street dance professional terms and competition rules; the vector-based search database is used to:

[0023] Use Sentence-BERT to encode the text definition of the acquired data into an n-dimensional vector and then store it in FAISS;

[0024] When the question-answering model inputs a user question, it encodes the question into a vector;

[0025] An approximate nearest neighbor search algorithm based on a hierarchical graph structure performs similarity search in a vector database;

[0026] Calculate the cosine similarity between the vector and the vectors in the database based on FAISS, and return the k results with the highest probability of similarity.

[0027] Furthermore, the data types of the multimodal processing database are video motion and audio data; the multimodal processing database is used to:

[0028] Extract dance movement features from the video, train a time series model for classification, and encode them into vector storage;

[0029] Extract keyframes based on vectors;

[0030] Multimodal processing of actions based on key frame extraction and key point detection;

[0031] Annotate datasets based on multimodal processed data;

[0032] Build an LSTM model and use the labeled dataset to train the LSTM model for action classification;

[0033] Use CLIP to encode the key frame into an m-dimensional vector and store it in FAISS;

[0034] After the user asks an encoded question, FAISS retrieves the matching action vectors and returns a video clip or a decomposed description.

[0035] Furthermore, when generating the output result of the question-answering model, the knowledge base is also updated. The updating process includes:

[0036] Monitor the output of the question-answering model and extract any erroneous data if detected.

[0037] Capturing triples of input text, output answers, and user-corrected answers for question-answering models that capture erroneous data;

[0038] Extracting the conversation history and user identity of the erroneous data based on the triple record, storing the conversation history and user identity in a temporary database, and marking the status as pending review;

[0039] Generating data that will be tagged with the status will automatically classify the error type and assign the error to different review queues;

[0040] Get the audit results of the audit queue:

[0041] When the corrected answer is directly adopted, the knowledge base update is triggered directly;

[0042] When a rejection occurs, the user is given the reason for the rejection;

[0043] When disputes are flagged, the issue is referred to domain experts for resolution.

[0044] Furthermore, the method further includes performing security filtering on the input text of the question-answering model, wherein the filtering step includes:

[0045] Build a violation word library and update it in real time;

[0046] Based on the illegal word library, regular matching is used to detect whether there are illegal words in the input text of the question-answering model;

[0047] Mark the sentences containing illegal words;

[0048] Use the HanLP dependency parser to build a sentence dependency tree for the marked fields and determine the semantic role of the offending words;

[0049] If the semantic role of the offending word is a non-positive expression, it is considered compliant, otherwise it is intercepted.

[0050] Furthermore, the output of the question-answering model is intercepted, and the interception step includes:

[0051] Get the raw score Logits of each text token before the output layer of the question-answering model;

[0052] Calculate the probability distribution of the last token and extract the highest probability as the confidence level;

[0053] If the confidence level is less than the set threshold, it is judged as a low-quality answer and the user is prompted to regenerate the question.

[0054] The second aspect of the present invention provides an electronic device, including a memory, a processor, and a computer program stored in the memory and runnable on the processor, wherein when the processor executes the program, a multimodal street dance knowledge question-answering method based on a large language model is implemented.

[0055] A third aspect of the present invention provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements a multimodal street dance knowledge question-answering method based on a large language model.

[0056] Compared with the prior art, the present invention has the following advantages and beneficial effects:

[0057] By building a multimodal street dance knowledge question-and-answer system, we effectively integrate multimodal data related to street dance, including text and videos, to provide users with a unified and easily accessible knowledge base. This system not only greatly improves the convenience of users' street dance knowledge searches, but also significantly enhances the accuracy and relevance of query results by leveraging large language models and multimodal data processing technologies. Users can easily access street dance knowledge that best matches their query, including definitions of professional terms, action videos, and instructional texts, greatly enhancing their learning experience and satisfaction.

[0058] Furthermore, the system supports complex query requirements and can handle multimodal data retrieval and semantic-based similarity search, allowing users to obtain the required street dance knowledge more flexibly and accurately. This not only promotes the spread and learning of street dance culture, allowing more people to understand and learn street dance, but also improves the accessibility and learning efficiency of street dance knowledge by providing an efficient and accurate knowledge acquisition platform.

[0059] In terms of technical implementation, this invention utilizes efficient data processing and retrieval tools such as Elasticsearch and FAISS to rapidly process and retrieve large amounts of street dance data, thereby improving the system's response speed and processing capabilities. This efficient data processing and retrieval mechanism ensures that users can obtain fast and accurate query results even when faced with large amounts of data.

[0060] The design of this invention also offers excellent flexibility and scalability, allowing for the easy addition of new data sources and search capabilities to adapt to the ever-changing needs of users and the development of the street dance field. This design enables the system to continuously evolve and meet new user demands for street dance knowledge, while also providing a platform for continuous updating and development for the dissemination and learning of street dance culture.

[0061] By integrating multimodal street dance data and leveraging advanced search technology, we provide an efficient, accurate, and easy-to-use street dance knowledge question-answering system. This system not only improves the accessibility and learning efficiency of street dance knowledge, promoting the dissemination and development of street dance culture, but also provides development for future street dance knowledge acquisition and learning through its flexibility and scalability. BRIEF DESCRIPTION OF THE DRAWINGS

[0062] In order to more clearly illustrate the technical solutions of the exemplary embodiments of the present invention, the following briefly introduces the drawings required for use in the examples. It should be understood that the following drawings only illustrate certain embodiments of the present invention and should not be considered as limiting the scope. A person of ordinary skill in the art can also derive other relevant drawings based on these drawings without inventive effort. In the drawings:

[0063] Figure 1 This is the question-answering process in an embodiment of the present invention. DETAILED DESCRIPTION

[0064] In order to make the objectives, technical solutions and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with examples and drawings. The exemplary embodiments of the present invention and their descriptions are only used to explain the present invention and are not intended to limit the present invention.

[0065] As a possible implementation, Figure 1 As shown, this embodiment provides a multimodal street dance knowledge question and answer method based on a large language model, including the following specific steps: collecting original text data related to street dance, cleaning and formatting the original street dance text data to obtain a street dance data set; annotating each text in the street dance data set; building a pre-trained language model, classifying and fine-tuning the annotated street dance data set; extracting features from the fine-tuned classified street dance data set based on semantic understanding; classifying feature vectors based on the extracted features to obtain different question and answer sample sets; building a knowledge base and an answer generation engine, the knowledge base including: a structured text retrieval library, a vector retrieval database and a multimodal processing database; the knowledge base is used to store and retrieve data, supporting the answer generation engine to generate accurate answers; the answer generation engine is used to retrieve data from the structured text retrieval library, the vector retrieval database and the multimodal processing database according to the question and answer model input text; data splicing is performed according to the retrieval results to form a context, and the question and answer model output result is generated.

[0066] In this embodiment, a structured street dance dataset was constructed by collecting, cleaning, and formatting raw text data related to street dance. By annotating and classifying each text in the street dance dataset, automated data processing was achieved, improving both efficiency and accuracy. By constructing a pre-trained language model and fine-tuning the street dance dataset for classification, the model's understanding of semantic features specific to the street dance domain was enhanced, thereby improving the accuracy of classification and feature extraction. Feature extraction and feature vector classification were performed on the fine-tuned classified street dance dataset based on semantic understanding, providing a rich feature representation for the question-answering system and helping to improve the accuracy and relevance of questions and answers. A knowledge base, including a structured text retrieval library, a vector-based retrieval database, and a multimodal processing database, was constructed, enabling efficient data storage and retrieval, providing strong data support for answer generation. By searching the knowledge base for data and concatenating the retrieval results to form a context, which was then matched with the question-answering sample set to generate the question-answering model output, automated and intelligent answer generation was achieved. Through steps such as data integration, model optimization, feature extraction, and knowledge base construction, the entire solution significantly improved the performance of the question-answering system, including accuracy, relevance, and response speed. By building an efficient street dance knowledge question and answer system, the dissemination and learning of street dance culture has been promoted, allowing more people to easily acquire and learn street dance knowledge.

[0067] In some possible implementations, data preprocessing is a key step in natural language processing and machine learning projects, and a series of technical means are used to convert raw data into a format suitable for model training. In this question-answering model, the question classifier is an important component of the data preprocessing stage. It can semantically understand and categorize the input questions, providing guidance for subsequent processing. The question classifier mainly undertakes the following functions in data preprocessing: 1. Semantic understanding; 2. Routing decision; 3. Feature extraction; 4. Efficiency optimization. The data to be preprocessed in this embodiment includes: data related to street dance that is written by oneself and collected from online uploads. For example, text containing descriptions of street dance (such as "What is the origin of a certain type of street dance?", "What are the rhythm characteristics of this street dance?") or subtitles of short videos.

[0068] In some possible implementations, the process of pre-processing data using the BERT-based question classifier is as follows:

[0069] (1) Data collection and initial cleaning

[0070] First, collect raw text data related to street dance, which may include user questions, comments, or video subtitles. This data often contains noise (such as spelling errors, special symbols, or non-standard expressions). Through text normalization methods, such as uniform encoding to UTF-8, removing unnecessary punctuation or emoticons, and ensuring case consistency, the raw text is converted into a clean format. For example, "what's the origin of Breaking???" can be normalized to "What is the origin of Breaking?"

[0071] (2) Data annotation

[0072] The collected street dance data is categorized and annotated, identifying the intent (e.g., "asking about origins," "asking about techniques") and domain (e.g., "breaking," "locking") of each text. For example, "What is the origin of breaking?" is annotated as {intent: asking about origins, domain: breaking}. This step provides training samples for supervised learning.

[0073] (3) Semantic understanding and feature extraction

[0074] The normalized text is fed into the BERT-base model. BERT uses its 12-layer Transformer encoder (768-dimensional hidden layers, 12 attention heads, and approximately 110 million parameters) to perform deep semantic encoding on the text, generating context-sensitive feature vectors. For example, for the question "What are the rhythmic characteristics of Breaking?" the model captures the semantic association between "rhythmic characteristics" and "Breaking," outputting a high-dimensional feature representation.

[0075] (4) Question classification and sample generation

[0076] A fully connected layer and a softmax layer are added on top of BERT to classify the feature vector and output the question category (such as "rhythm-related questions"). Based on the classification results, the data is divided into different sets of question-answering samples, such as those related to "the origins of street dance," "street dance techniques," or "street dance styles." These samples can be used to build the knowledge base or train models for the subsequent question-answering system.

[0077] In some possible implementations, the model fine-tuning process specifically includes:

[0078] In order to adapt the BERT-base model to the classification task of dance data, it needs to be fine-tuned. The specific steps are as follows:

[0079] Build a pre-trained BERT-base model and add a fully connected layer and softmax layer on top of the pre-trained BERT-base model to map the feature vectors output by BERT to classification labels (such as "origin" or "technique" in street dance). The fully connected layer performs dimensionality reduction and classification, while the softmax layer outputs the probability distribution of each category.

[0080] Hyperparameter settings, including:

[0081] Learning rate: Set it between 2e-5 and 5e-5 to avoid excessive learning rate from destroying pre-trained weights.

[0082] Batch size: Choose 16 or 32 to balance training speed and video memory requirements.

[0083] Training rounds: Set to 2 to 4 epochs to avoid overfitting.

[0084] Optimization technique: Randomly mask some tokens in the input (e.g., replacing "tango" with [MASK]) to enhance the model's robustness to context. Use a smaller learning rate (e.g., 1e-5) for the bottom Transformer layer to preserve general language knowledge; use a larger learning rate (e.g., 3e-5) for the top classification layer to accelerate task adaptation.

[0085] Training and Evaluation:

[0086] Use labeled data for supervised learning, optimize the cross-entropy loss function, and evaluate the classification accuracy (such as F1 score) on the validation set.

[0087] BERT-base is a general pre-trained model that learns rich language representations on a large corpus, but its adaptability to specific domains (such as street dance question classification) is insufficient. Fine-tuning further trains the model on task-specific data to enable it to understand the semantic characteristics and user intent of the street dance domain. For example, an untuned BERT might not accurately distinguish the subtle differences between "Breaking's rhythm" and "Breaking's origin," while a fine-tuned model can better capture these semantic differences.

[0088] As a result, the fine-tuned BERT-base model achieves significant improvements in classification accuracy and generalization. For example, in a street dance question classification task, the F1 score of the fine-tuned model can increase from 70% of the pre-trained score to over 90%. Compared to existing technologies (such as TF-IDF or traditional RNN-based classifiers), the advantages of fine-tuning BERT are:

[0089] Stronger semantic understanding ability: The Transformer-based attention mechanism can capture long-distance dependencies, while traditional methods rely on manual features and have limited effectiveness.

[0090] More adaptable: The fine-tuned model is optimized for the dance domain, while general models or non-fine-tuned methods may perform poorly in specific scenarios.

[0091] Improved robustness: Techniques such as dynamic masking make the model more tolerant to noisy data (such as non-standard expressions of user input).

[0092] In summary, through the above data processing and fine-tuning process, the BERT-based question classifier can efficiently convert street dance-related raw data into structured question and answer samples, providing high-quality input for subsequent systems. At the same time, it surpasses traditional technologies in performance and demonstrates greater practicality and flexibility.

[0093] In some possible implementations, the intelligent question-answering model specifically includes:

[0094] The intelligent question-answering system is mainly divided into a knowledge base and an answer generation engine.

[0095] The knowledge base is used to store and retrieve data, supporting the answer generation engine to generate accurate answers. User questions are classified by the question classifier and then routed to the knowledge base, which consists of three parts: a structured text retrieval database, a vector-based retrieval database, and a multimodal database.

[0096] In some possible implementations, the structured text retrieval library specifically includes:

[0097] The data types include: basic knowledge (such as street dance history and style classification) and dancer resumes (such as name and award records). We use Elasticsearch (ES) to build a full-text search library, and perform keyword matching and score sorting based on the BM25 algorithm. Example:

[0098] Suppose a user asks "Poppin John's resume".

[0099] Resume data is stored in JSON format, such as {"name":"Poppin John","birth":"1984-08-18","awards":["World of Dance 2019"]}.

[0100] The query is segmented into "Poppin John" and "resume". ES converts the query into a Boolean query, matching the fields name and awards.

[0101] The search returns the highest-scoring documents and extracts the relevant fields.

[0102] Since street dance basics and resumes are in a fixed format, they are suitable for keyword matching, and ES supports high-concurrency queries, which is suitable for real-time question-and-answer scenarios; the BM25 algorithm ensures high relevance and reduces irrelevant results. Therefore, by building a structured text retrieval library, structured information can be quickly retrieved, with a response time of <100ms and an accuracy rate of >95% (based on the test set).

[0103] In some possible implementations, the vector-based search database specifically includes:

[0104] The data types are: professional terms (such as "Toprock" and "Freeze"), and competition rules (such as "Juste Debout rules").

[0105] FAISS is used to build a vector database to store semantically encoded vectors, and the HNSW algorithm is used for efficient similarity search.

[0106] Example: 1: User asks "What is Freeze?"

[0107] The term "Freeze" is defined as "a pause in movement in street dance, usually in a balanced position."

[0108] Use Sentence-BERT to define the encoding as a 768-dimensional vector and store it in FAISS.

[0109] User questions are encoded as vectors, and FAISS calculates cosine similarity and returns the Top-K results (as defined in “Freeze”).

[0110] Example 2: The rules of a competition are such as "Juste Debout requires two-person teams and no contact is allowed." The rules text is also encoded as a vector and stored in FAISS.

[0111] Effect: Semantic retrieval accuracy is >90%, and fuzzy matching is supported (for example, "pause action" can match "Freeze").

[0112] Design reasons and advantages:

[0113] Because terms and rules require semantic understanding, keyword matching can easily fail. Therefore, FAISS can efficiently process high-dimensional vectors, while the HNSW algorithm offers fast search speeds (<50ms). Furthermore, vector retrieval supports semantic similarity and is adaptable to diverse queries.

[0114] In some possible implementations, the multimodal processing database specifically includes:

[0115] The data types are: video action (such as Breaking's Windmill), audio (rhythm analysis).

[0116] Extract video action features, train temporal model classification, and encode them into vector storage.

[0117] Example: User asks "How do I do the Windmill action?"

[0118] Keyframe extraction: Extract keyframes from Windmill teaching videos based on scene change detection (such as YOLOv5 detecting human body boundaries), at 2-3 frames per second.

[0119] Keypoint detection: Use MediaPipe to extract the coordinates of 25 keypoints (such as shoulders, hips, and knees) and generate time series data.

[0120] Action classification: Labeled dataset (1000 Windmill videos), trained LSTM model, classification accuracy >85%.

[0121] Vector encoding: Use CLIP to encode key frames into 512-dimensional vectors and store them in FAISS.

[0122] After the user asks an encoded question, FAISS retrieves the matching action vectors and returns a video clip or a decomposed description.

[0123] Since movements are the most important part of street dance knowledge, street dance movements require visual analysis, and text descriptions are insufficient to express the details. In this embodiment, key point detection and LSTM are used to capture movement dynamics, and CLIP vectors support cross-modal matching. Multimodal processing enriches the answer form (such as video + text), thereby building a multimodal processing database, which can make the movement recognition accuracy >85%, the retrieval response time <200ms, and support movement teaching decomposition.

[0124] In some possible implementations, the answer generation engine specifically includes:

[0125] The core model adopts DeepSeek-R1:7b, combined with Retrieval-Augmented Generation (RAG) technology.

[0126] The RAG process includes:

[0127] Retrieval phase: Retrieve structured text (such as resumes) from ES.

[0128] Retrieve semantic information (e.g., term definitions) and multimodal data (e.g., action videos) from FAISS.

[0129] Example: Asking "How do I learn Windmill?" will return the action definition, breakdown video, and instructional text.

[0130] Enhancement stage: Splicing search results, engineered prompt words (such as "provide concise teaching steps") and original questions to form a context.

[0131] Example: The context is "Windmill: Rotational movement, requires back support. Video clip: Key point sequence. Hint: Break down the steps. Question: How do you learn the Windmill?"

[0132] Generation phase: DeepSeek-R1:7b generates answers based on the context, such as “Windmill learning steps: 1. Lie flat on the ground and bend your knees; 2. Use your back to support and rotate your hips; 3. Maintain arm balance. Reference video: XXX.”

[0133] Since a single generative model is prone to hallucinations and requires retrieval to provide factual evidence, this embodiment uses RAG to combine retrieval and generation to ensure the professionalism and credibility of the answers. DeepSeek-R1:7b efficiently processes long contexts and is suitable for multimodal input, achieving an answer accuracy rate of >90%, a hallucination rate of <5%, and a generation time of <1 second.

[0134] In some possible implementations, the question-answering model supports multi-round conversations, uses Redis to cache session states, and employs windowed memory to retain the most recent five rounds of conversations.

[0135] Example: The user asks "What is Windmill?" in round 1 and "How to practice it?" in round 2.

[0136] Redis stores the round 1 answers (e.g. "Windmill is a rotating action") and the questions.

[0137] Round 2 combines the cached context to generate coherent answers (e.g., “Windmill exercise: Starting with basic rotations…”).

[0138] Redis high-performance cache is used to ensure real-time response; windowed memory balances memory usage and context integrity. By combining context, it can avoid repeated or contradictory answers, achieve conversation coherence >95%, and cache query time <10ms.

[0139] In some possible implementations, when the question-answering model obtains input questions and outputs results, a security filter module is set to ensure that the content generated by the intelligent question-answering system is compliant and legal, intercept illegal or low-quality answers, and maintain platform specifications and user experience.

[0140] The module is divided into two parts: content compliance checking and low-confidence answer blocking.

[0141] Compliance check: Block answers containing illegal content (such as violence, discrimination, and political sensitivity).

[0142] Regular expression matching: We maintain a database of illegal terms (such as "violence" and "discriminatory terms") containing over 5,000 high-risk terms, which is updated regularly. We perform regular expression matching on generated answers and flag potential illegal terms.

[0143] Example: The answer to "Is there violence in street dance?" contains the word "violence." The regular expression violence|violence triggers a match.

[0144] Syntactic context analysis: Use the HanLP dependency parser to build a sentence dependency tree and determine the semantic role of the offending word.

[0145] Example: In “oppose violence”, “violence” is the object of “oppose”, which is not a positive expression and is judged to be compliant.

[0146] In “advocating violence”, “violence” is the object of “advocating”, which is a positive expression and is judged to be a violation.

[0147] Intercept processing: The illegal answer is replaced with a fixed reply: "This question involves sensitive content and cannot be answered."

[0148] Simple regular matching is prone to misjudgment (such as "oppose violence"), and semantic analysis is needed to improve accuracy. Therefore, dependency tree analysis captures contextual semantics and reduces false positives. Regular matching is efficient and suitable for preliminary screening. Regular updates to the vocabulary ensure coverage of emerging sensitive words, which can achieve an accuracy rate of >98% in blocking illegal content and a false positive rate of <2% (based on 10,000 test samples).

[0149] In some possible implementations, a low-confidence answer interception is used to intercept low-quality or unreliable answers output by the generation model.

[0150] Confidence calculation: When generating answers using DeepSeek-R1:7b, get the logits of the output token.

[0151] Calculate the probability distribution of the last token and extract the highest probability as the confidence.

[0152] Example: Generate a sequence to answer the question "Windmill action decomposition". Logits are calculated by softmax to obtain a probability distribution.

[0153] If the probability of the last token (such as "step") is 0.6, the confidence level = 0.6.

[0154] Intercept threshold: If the confidence level is less than 0.7, the answer is considered low-quality and replaced with a fixed response: "Unable to answer at this time. Please provide more description or ask again."

[0155] Example: When the question "What are the future trends of street dance" is asked, the generated answer is semantically ambiguous, with a confidence level of 0.65, triggering an interception.

[0156] Generative models may output unreliable answers due to insufficient data or complex questions, necessitating quantitative evaluation. Therefore, we utilize the model's own logits to calculate confidence, eliminating the need for additional models and ensuring efficiency and lightweightness. A threshold of 0.7 balances sensitivity and user experience, avoiding excessive interception. This approach achieves an interception rate of >90% for low-quality answers, effectively reducing ambiguous or inaccurate responses, while maintaining computation time of <10ms and negligible latency.

[0157] In some possible implementations, this implementation includes user feedback and knowledge base updates during the knowledge question answering process.

[0158] This question-answering model has an automatic learning function. When the user clicks the "Correct" button or a negative answer (such as "This answer is incorrect") appears in the dialogue, the system triggers the following process:

[0159] 1. Error data capture: First, record the triple (original question, incorrect answer, user-corrected answer). Then, add context such as conversation history and user identity. Finally, store the data in a temporary repository and mark it as pending review.

[0160] 2. Annotation task generation: Annotation task generation will automatically classify error types (such as factual errors, unclear statements, and outdated information) and assign errors to different review queues (urgent errors are prioritized).

[0161] 3. Manual background review. The review interface displays: problem comparison, user correction content, and related knowledge point map

[0162] When the corrected answer is directly adopted, the knowledge base update is triggered directly; when it is rejected, the reason for rejection is fed back to the user; when a dispute is marked, the issue is transferred to the domain expert for processing.

[0163] 4. Knowledge base updates are mainly done through ES index updates and vector library updates. When updating the vector library, the answers are first re-encoded and corrected, and then the FAISS index is incrementally updated.

[0164] As a possible implementation method, this embodiment provides an electronic device, including a memory, a processor, and a computer program stored in the memory and runnable on the processor. When the processor executes the program, a multimodal street dance knowledge question-and-answer method based on a large language model is implemented.

[0165] As a possible implementation, this embodiment provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements a multimodal street dance knowledge question-answering method based on a large language model.

[0166] The specific implementation methods described above further illustrate the objectives, technical solutions and beneficial effects of the present invention in detail. It should be understood that the above description is only a specific implementation method of the present invention and is not intended to limit the scope of protection of the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the scope of protection of the present invention.

Claims

1. A multimodal street dance knowledge question answering method based on a large language model, characterized by: The specific steps include: Collect original text data related to street dance, clean and format the original text data of street dance, and obtain the street dance dataset; Label each text in the street dance dataset; Build a pre-trained language model and fine-tune the classification of the annotated street dance dataset; Feature extraction of the fine-tuned classified street dance dataset based on semantic understanding; Based on the extracted features, feature vectors are classified to obtain different question and answer sample sets; Building a knowledge base and an answer generation engine, wherein the knowledge base includes: a structured text retrieval database, a vector-type retrieval database, and a multimodal processing database; The knowledge base is used to store and retrieve data, supporting the answer generation engine to generate accurate answers; The answer generation engine is used to retrieve data from a structured text retrieval library, a vector retrieval database, and a multimodal processing database based on the text input of the question-answering model; perform data splicing based on the retrieval results to form a context and generate the question-answering model output result.

2. The multimodal street dance knowledge question-answering method based on a large language model according to claim 1 is characterized in that: The construction of the pre-trained language model and fine-tuning of the classification of the annotated street dance dataset specifically includes: Build a BERT-base model and add a fully connected layer and a softmax layer on top of the BERT-base model. The fully connected layer is used to reduce the dimension and classify the input data of the pre-trained language model, and map the feature vectors output by BERT to classification labels. The softmax layer is used to output the probability distribution of the categories of the street dance dataset. Setting hyperparameters for the pre-trained language model and training the pre-trained language model to obtain an optimized pre-trained language model; Based on the optimized pre-trained language model, the classification of the labeled street dance dataset is fine-tuned.

3. The multimodal street dance knowledge question answering method based on a large language model according to claim 1 is characterized in that: The data types of the structured text retrieval library are basic knowledge of street dance and dancer resumes; The structured text retrieval library is used to perform keyword matching and score sorting on the acquired data based on the BM25 algorithm, wherein the keyword matching specifically includes: Extract data from the structured text retrieval database and perform word segmentation on the data; Convert the query to a Boolean query, matching the fields name and awards; The search returns the document with the highest score and extracts the relevant fields.

4. The multimodal street dance knowledge question answering method based on a large language model according to claim 1 is characterized in that: The data types of the vector-based search database are street dance professional terms and competition rules; the vector-based search database is used to: Use Sentence-BERT to encode the text definition of the acquired data into an n-dimensional vector and then store it in FAISS; When the question-answering model inputs a user question, it encodes the question into a vector; An approximate nearest neighbor search algorithm based on a hierarchical graph structure performs similarity search in a vector database; Calculate the cosine similarity between the vector and the vectors in the database based on FAISS, and return the k results with the highest probability of similarity.

5. The multimodal street dance knowledge question answering method based on a large language model according to claim 1 is characterized in that: The data types of the multimodal processing database are video motion and audio data; the multimodal processing database is used to: Extract dance movement features from the video, train a time series model for classification, and encode them into vector storage; Extract keyframes based on vectors; Multimodal processing of actions based on key frame extraction and key point detection; Annotate datasets based on multimodal processed data; Build an LSTM model and use the labeled dataset to train the LSTM model for action classification; Use CLIP to encode the key frame into an m-dimensional vector and store it in FAISS; After the user asks an encoded question, FAISS retrieves the matching action vectors and returns a video clip or a decomposed description.

6. The multimodal street dance knowledge question answering method based on a large language model according to claim 1 is characterized in that: When generating the output result of the question-answering model, the knowledge base is also updated. The updating process includes: Monitor the output of the question-answering model and extract any erroneous data if detected. Capturing triples of input text, output answers, and user-corrected answers for question-answering models that capture erroneous data; Extracting the conversation history and user identity of the erroneous data based on the triple record, storing the conversation history and user identity in a temporary database, and marking the status as pending review; Generating data that will be tagged with the status will automatically classify the error type and assign the error to different review queues; Get the audit results of the audit queue: When the corrected answer is directly adopted, the knowledge base update is triggered directly; When a rejection occurs, the user is given the reason for the rejection; When disputes are flagged, the issue is referred to domain experts for resolution.

7. The multimodal street dance knowledge question answering method based on a large language model according to claim 1 is characterized in that: The method also includes performing security filtering on the input text of the question-answering model, wherein the filtering step includes: Build a violation word library and update it in real time; Based on the illegal word library, regular matching is used to detect whether there are illegal words in the input text of the question-answering model; Mark the sentences containing illegal words; Use the HanLP dependency parser to build a sentence dependency tree for the marked fields and determine the semantic role of the offending words; If the semantic role of the offending word is a non-positive expression, it is considered compliant, otherwise it is intercepted.

8. The multimodal street dance knowledge question-answering method based on a large language model according to claim 7 is characterized in that: The method further includes intercepting the output of the question-answering model, wherein the intercepting step includes: Get the raw score Logits of each text token before the output layer of the question-answering model; Calculate the probability distribution of the last token and extract the highest probability as the confidence level; If the confidence level is less than the set threshold, it is judged as a low-quality answer and the user is prompted to regenerate the question.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the program, it implements the multimodal street dance knowledge question-answering method based on a large language model as described in any one of claims 1 to 8.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, it implements the multimodal street dance knowledge question-answering method based on a large language model as described in any one of claims 1 to 8.

Images