Cultural relic explanation method based on multi-modal user intention recognition and alignment

Abstract

The invention relates to the technical field of natural language processing, in particular to a cultural relic explanation method applied to natural language processing NLP and artificial intelligence and based on multi-modal user intention recognition and alignment. The method comprises the following steps: firstly, receiving various inputs provided by a user through a multi-modal alignment module, such as voice, characters, pictures, geographic positions and the like, and outputting two values, namely a final dialogue and an object main body; then, when the object main body exists, the content and the main body noun are combined and optimized into a prompt dialogue through a prompt project, and the prompt dialogue is used as input of an intention recognition model; thirdly, performing intention recognition on the prompt dialogue by using an intention recognition model, and performing intention division according to a recognition result; meanwhile, a large amount of data related to cultural relics are collected, the data are cleaned, preprocessed, labeled and classified, the labeled data are used for model training, and parameters are adjusted to achieve the optimal intention classification and named subject recognition effects. And providing services for the user by combining contents of intention recognition and named subject recognition.

Description

Technical Field

[0001] The present invention relates to the technical field of natural language processing, and is applied to natural language processing (NLP) and artificial intelligence, and is a cultural relics interpretation method based on multi-modal user intention recognition and alignment. Background Art

[0002] In the field involved in the present invention, existing technical solutions usually adopt a relatively single mode to identify user intentions. For example, only text information is relied on for analysis and understanding, and other modal data such as audio and pictures are insufficiently utilized or basically not utilized.

[0003] During the processing, the input text may simply be subjected to conventional semantic analysis, lacking comprehensive consideration and integration of multiple data types. The use of decoders is also relatively limited, and there may only be simple decoders for text, which cannot be flexibly scheduled and effectively aligned according to different data types. Data preprocessing may also be relatively simple and rough, and the generated text data cannot be well converted into multimodal data and accurately represented by vectors. Moreover, existing technologies often cannot well integrate multimodal data and feed it back to users in a rich form (such as text plus voice), and there are certain deficiencies in recognition accuracy and user experience.

[0004] Therefore, it is necessary to integrate multiple different types of data such as text, voice, and pictures to diversify the way users input data and no longer be limited to text. Based on this, a new multimodal recognition and alignment method is improved and innovated to better meet tourists' needs for cultural and museum explanations. Summary of the invention

[0005] The present invention provides a user intention recognition method based on multimodal applications, which can effectively align various user inputs, recognize and optimize user intentions, significantly improve the accuracy of intention recognition for user questions, and greatly improve the efficiency of answering. The method focuses on cultural and museum scenes, and aims to provide tourists with more accurate introductions and explanations of knowledge about cultural relics, exhibition halls, and museums, and can meet various questions raised by users to the greatest extent.

[0006] The present invention relates to a method for identifying museum cultural relics based on user intentions and a method for storing and calling based on multimodality, the core goal of which is to properly handle the problem that the current model cannot accurately identify user intentions with the help of various types of information such as text, audio, and pictures of cultural and museum scenes input by users. Based on this purpose, the present invention's method for identifying cultural relics based on user intentions based on multimodality specifically covers the following contents: according to the input data type, the corresponding decoders are reasonably scheduled to achieve the alignment effect; the text is returned through the corresponding decoders, and then the generated text data is preprocessed to obtain multimodal data; and then these multimodal data are input into a pre-trained neural network. Moreover, this method has the ability to fuse multimodal data and can provide feedback to users in the form of a combination of text and voice.

[0007] A method for explaining cultural relics based on multimodal user intention recognition and alignment includes the following steps: S1. First, align the multimodal input content, which includes voice, text, pictures, and locations; deploy the Qwen-Audio large model with voice recognition capabilities; realize image recognition and select yolov8 for target detection model training; S2. Identify through the intent recognition model; build the intent subject recognition service, train the intent recognition model based on BERT, and train the named subject recognition model based on NNCRF-NER; Data collection: collect text data on issues in museums and cultural relics in various dimensions. The collection method is mainly web crawlers and questionnaires, supplemented by manual supplementation. Web crawlers are used to automatically collect cultural relic-related information from online platforms such as Sogou. Questionnaires are used to collect users' interests and frequently asked questions about cultural relics at the museum. Finally, some missing data points are supplemented manually, and a csv plain text file with one line of text data is delivered. Data cleaning: delete stop words, special symbols, and duplicate data, delete blanks in the text, standardize the text to convert it into a unified format, annotate and classify the cleaned data according to different dimensions of cultural relic information, and finally deliver a csv plain text file with one line of text data after filtering. The first step in labeling preparation is to divide the intent into 13 categories: basic introduction of cultural relics, basic information, craftsmanship and materials, historical background, legends, related tasks, related deeds, digital restoration, internal structure, archaeological excavation, historical origin, video recommendation, and others. Different codes are given to the intent, and then the naming subject is determined. It is necessary to clearly define all the keywords that need to be included in the service, such as the name of the cultural relic, the name of the museum, and the name of the exhibition. Two TXT files are delivered, one is the intent coding TXT file separated by line breaks, and the other is the keyword TXT file separated by line breaks. Data annotation, first of all, explains the specific format of subject annotation and intent annotation for a piece of text data. The annotated data can be used for intent training as well as naming subject training. The specific format of annotation is “[a piece of text data] <sep>[token label column] <sep>[Intent code]", "[token label column]" starts with B-LOC as the keyword, ends with E-LOC as the keyword, the middle is represented by M-LOC, and the token that is not a keyword is represented by O. At the end of the text, replace the "[Intent code]" mark with the intent code; Model training: use the pre-trained BERT model for intent recognition training and NNCRF-NER for named entity recognition model training; Model deployment, using network services to provide model capabilities to the outside world. In the API, it is necessary to process requests from clients, including verifying the format of input data, performing necessary preprocessing, and passing the preprocessed data to the model for prediction, and finally returning the prediction results; S3: Provide services by combining content with intent recognition and named subject recognition, build a prompt library that contains diversity and covers different user intentions, obtain user intent encoding and named entities through S2, select prompts for the intent and named entities from the prompt library, and combine with personalized recommendation algorithms to provide users with customized cultural relics recommendation and explanation services based on user preferences and historical interaction behaviors.

[0008] The S1 is divided into: S101, collect pictures of cultural relics in actual scenes by on-site shooting, and train them by data annotation, and finally form a target detection model; Data collection: first determine the list of cultural relics to be image-recognized, and generate labels for each cultural relic according to the requirements of [museum code], [exhibition hall code], and [cultural relic code]; record a video; extract frames from the cultural relic video and use the images obtained as training data; data labeling: label the cultural relics according to the cultural relic labels; after labeling, a labeling file will be generated, and a txt format of labeling data will be produced; after the data is labeled, the data will be used to train the target detection model.

[0009] S102. Using indoor positioning of Bluetooth beacons, select the vicinity of cultural relics where tourists gather and key cultural relics at the museum site, collect the tag information of the Bluetooth beacon locations and cultural relics near the locations; edit the major value and minor value of the Bluetooth beacon and associate them with the cultural relics list. Multimodality only needs to build a database association table to provide the ability to query based on the major value followed by the minor value; when the Bluetooth device receives the frequency of nearby Bluetooth signals and detects the same beacon code, it can be determined that the user is staying at the location; concatenate the major and minor codes of the Bluetooth beacon as the Bluetooth beacon ID and pass it to the multimodality; S103: Language recognition capability, deploying the open source large model Qwen-Audio large language model to implement, and developing interfaces to input audio and return text; S104: After processing the incoming information, the multimodal will output two values. The first value represents the text converted from the language or the text directly input as parameter A, and the second value is the list of entity names obtained by image recognition and position detection as parameter B. After receiving these two values, different types of data will get different outputs. For pictures and audio, they need to be uploaded to the object storage first. The specific situation is as follows: Language input and output: A: [the result of speech conversion is a text], B: []; Image input and output: A:[], B:[list of cultural relics names]; Word input and output: A: [incoming text], B: []; Bluetooth beacon ID incoming output: A: [], B: [artifact name list].

[0010] The beneficial effects of the present invention are: The present invention discloses a method for aligning the intent of user questions based on a multimodal alignment module framework. The method first receives various inputs provided by the user, such as voice, text, pictures, geographic location, etc., through a multimodal alignment module, and outputs two values: the final dialogue and the object subject. Then, when the object subject exists, the content and the subject noun are combined and optimized into a prompt dialogue through a prompt project, which is used as the input of the intent recognition model. Next, the intent recognition model is used to recognize the intent of the prompt dialogue, and the intent is divided according to the recognition result. At the same time, a large amount of data related to cultural relics is collected, the data is cleaned, preprocessed, labeled and classified, and the labeled data is used to train the model, and the parameters are adjusted to achieve the best intent classification and named subject recognition effect. The content of intent recognition and named subject recognition is combined to provide services to users.

[0011] The beneficial effects of the present invention are as follows: through the multimodal alignment module and the intention recognition model, the user's intention can be accurately understood, and accurate answers can be provided according to the user's questions; through the named entity recognition model, the conversation subject can be identified, so as to provide more personalized services; through the collection and processing of a large amount of cultural relics related data, the accuracy and generalization ability of the model can be improved; through the training and optimization of the model, the performance and efficiency of the model can be improved. BRIEF DESCRIPTION OF THE DRAWINGS

[0012] Figure 1 The present invention is a flowchart of intention alignment based on multimodality; Figure 2 Schematic diagram of building a flow for the intent identification and name subject identification models. DETAILED DESCRIPTION

[0013] In order to achieve the above purpose, the present invention provides a method for aligning the intent of user questions based on the framework of a multimodal alignment module. After the user provides a variety of inputs (voice, text, pictures, locations, etc.), the multimodal alignment module is entered, and the alignment of various types of questions is unified through the multimodal module. The aligned text is passed to the intent recognition module, and the intent recognition module is entered to determine the user's intent and the subject of the question, and the user's intent and subject are passed to the large model module for answering. In the communication process between the above modules, different prompt projects are constructed to further process and integrate the aligned information to form a clear prompt so that it can be accurately understood and processed.

[0014] A method for explaining cultural relics based on multimodal user intention recognition and alignment includes the following steps: S1. First, align the multimodal input content, which includes voice, text, pictures, and locations; deploy the Qwen-Audio large model with voice recognition capabilities; realize image recognition and select yolov8 for target detection model training; S2. Identify through the intent recognition model; build the intent subject recognition service, train the intent recognition model based on BERT, and train the named subject recognition model based on NNCRF-NER; Data collection: collect text data on issues in museums and cultural relics in various dimensions. The collection method is mainly web crawlers and questionnaires, supplemented by manual supplementation. Web crawlers are used to automatically collect cultural relic-related information from online platforms such as Sogou. Questionnaires are used to collect users' interests and frequently asked questions about cultural relics at the museum. Finally, some missing data points are supplemented manually, and a csv plain text file with one line of text data is delivered. Data cleaning: delete stop words, special symbols, and duplicate data, delete blanks in the text, standardize the text to convert it into a unified format, annotate and classify the cleaned data according to different dimensions of cultural relic information, and finally deliver a csv plain text file with one line of text data after filtering. The first step in labeling preparation is to divide the intent into 13 categories: basic introduction of cultural relics, basic information, craftsmanship and materials, historical background, legends, related tasks, related deeds, digital restoration, internal structure, archaeological excavation, historical origin, video recommendation, and others. Different codes are given to the intent, and then the naming subject is determined. It is necessary to clearly define all the keywords that need to be included in the service, such as the name of the cultural relic, the name of the museum, and the name of the exhibition. Two TXT files are delivered, one is the intent coding TXT file separated by line breaks, and the other is the keyword TXT file separated by line breaks. Data annotation, first of all, the specific format of subject annotation and intent annotation for a piece of text data is explained. The annotated data can be used for intent training as well as naming subject training. The specific format of annotation is "[a piece of text data] <sep>[token label column] <sep>[Intent code]", "[token label column]" starts with B-LOC as the keyword, ends with E-LOC as the keyword, the middle is represented by M-LOC, and the token that is not a keyword is represented by O. At the end of the text, replace the "[Intent code]" mark with the intent code; Model training: use the pre-trained BERT model for intent recognition training and NNCRF-NER for named entity recognition model training; Model deployment, using network services to provide model capabilities to the outside world. In the API, it is necessary to process requests from clients, including verifying the format of input data, performing necessary preprocessing, and passing the preprocessed data to the model for prediction, and finally returning the prediction results; S3: Provide services by combining content with intent recognition and named subject recognition, build a prompt library that contains diversity and covers different user intentions, obtain user intent encoding and named entities through S2, select prompts for the intent and named entities from the prompt library, and combine with personalized recommendation algorithms to provide users with customized cultural relics recommendation and explanation services based on user preferences and historical interaction behaviors.

[0015] The S1 is divided into: S101, collect pictures of cultural relics in actual scenes by on-site shooting, and train them by data annotation, and finally form a target detection model; Data collection: first determine the list of cultural relics to be image-recognized, and generate labels for each cultural relic according to the requirements of [museum code], [exhibition hall code], and [cultural relic code]; record a video; extract frames from the cultural relic video and use the images obtained as training data; data labeling: label the cultural relics according to the cultural relic labels; after labeling, a labeling file will be generated, and a txt format of labeling data will be produced; after the data is labeled, the data will be used to train the target detection model.

[0016] S102. Using indoor positioning of Bluetooth beacons, select the vicinity of cultural relics where tourists gather and key cultural relics at the museum site, collect the tag information of the Bluetooth beacon locations and cultural relics near the locations; edit the major value and minor value of the Bluetooth beacon and associate them with the cultural relics list. Multimodality only needs to build a database association table to provide the ability to query based on the major value followed by the minor value; when the Bluetooth device receives the frequency of nearby Bluetooth signals and detects the same beacon code, it can be determined that the user is staying at the location; concatenate the major and minor codes of the Bluetooth beacon as the Bluetooth beacon ID and pass it to the multimodality; S103: Language recognition capability, deploying the open source large model Qwen-Audio large language model to implement, and developing interfaces to input audio and return text; S104: After processing the incoming information, the multimodal will output two values. The first value represents the text converted from the language or the text directly input as parameter A, and the second value is the list of entity names obtained by image recognition and position detection as parameter B. After receiving these two values, different types of data will get different outputs. For pictures and audio, they need to be uploaded to the object storage first. The specific situation is as follows: Language input and output: A: [the result of speech conversion is a text], B: []; Image input and output: A:[], B:[list of cultural relics names]; Word input and output: A: [incoming text], B: []; Bluetooth beacon ID incoming output: A: [], B: [artifact name list].

[0017] The present invention is composed of multimodal alignment and intention subject recognition, which is subdivided into five specific modules: image recognition, speech recognition, position detection, intention recognition and named subject recognition.

[0018] S1: The multimodality mentioned in this article is an interface system that integrates the target detection model and the speech recognition model and has a Bluetooth association relationship. The first thing to do is to align the output content of the multimodality. The multimodality here mainly receives a variety of inputs provided by users, including voice, text, and pictures, and the location of the venue. The speech recognition capability is achieved by deploying the open source Qwen-Audio model. Image recognition selects yolov8 for target detection model training. Here we mainly introduce two implementation methods: cultural relic image recognition and indoor positioning.

[0019] S101: Image recognition capability, which is supported by trained object detection algorithms. This capability can be achieved by collecting images of cultural relics in actual scenes through on-site photography, and training through data annotation to eventually form an object detection model.

[0020] 1) Data collection: The first step is to determine the list of cultural relics for image recognition, and specify the label of each cultural relic according to the requirements of [museum code] + [exhibition hall code] + [cultural relic code]. For example, if the museum code is M001, the exhibition hall code is E101, and the cultural relic code is 001, then the cultural relic label is M001-E101-001; the second step is to use a mobile phone to record videos in actual scenes. One cultural relic provides two videos shot with a mobile phone in vertical mode. In the first video, the mobile phone shoots horizontally from left to right with the vertical screen around the cultural relic, and in the second video, the mobile phone shoots vertically from top to bottom with the vertical screen around the cultural relic; the third step is to extract frames from the cultural relic video to obtain pictures as training data.

[0021] 2) Data labeling: Label the cultural relics images. The cultural relics labels have been specified in the previous step. Label the cultural relics according to the labels. Labeling is strictly in accordance with the target detection requirements as follows: 1. Edge rule: The annotation box should be close to the edge of the cultural relic target. Avoid making the box too large or too small, and ensure that the box boundary is as close to the actual edge of the cultural relic as possible.

[0022] 2. Completeness: The annotation box should completely surround the cultural relic target without any missing or omitted parts. Make sure that the annotation box does not cut off any important part of the cultural relic.

[0023] 3. Angles and perspectives: Annotate the artifacts according to their different angles and perspectives. Multiple angles may be required to fully capture the shape and features of the artifacts.

[0024] 4. Occlusion processing: If the cultural relic is partially obscured by other objects or obstructions, the annotation box should depict the outline of the visible part as accurately as possible.

[0025] After the annotation is completed, a annotation file will be generated, which will produce annotation data in txt format, and the annotation information of a single border:<class_id><x_center><y_center> <width> <height>,<class_id> is the identifier of the detected object category, usually an integer corresponding to the category in the dataset.<x_center> ,<y_center> , <width> , <height>The center coordinates, width and height of the bounding box are respectively, and are usually normalized relative to the size of the entire image. Annotation data example "0 0.5 0.22411894273127722 0.289280469897209740.16409691629955947" 3) Training the target detection model: After the data is labeled, it is divided into training set, validation set and test set in a ratio of 0.8:0.2:0.0 for target detection model training.

[0026] 4) Model optimization and evaluation: After training, the model needs to be evaluated and optimized. The evaluation process mainly looks at the two indicators of training mAP50 and mAP50-95. mAP50 is the abbreviation of mean Average Precision, which indicates the average accuracy over multiple categories. mAP50-95: This is a more stringent evaluation indicator used to evaluate the model's target detection performance. This can more accurately evaluate the performance of the model. The first step in the evaluation process is to look at the mAP and mAP50-95 indicators of the training results. The higher the two indicators, the better the model accuracy. The second step is to identify the cultural relics in the actual scene by taking pictures of various angles. It is necessary to supplement the cultural relics picture data of angles that cannot be identified, improve the quality of data annotation, and repeatedly train to obtain a good model. The model mAP50 and mAP50-95 both reach above 0.9 to meet the use requirements, and the model is integrated into the multi-modal reasoning for image recognition.

[0027] S102: Indoor positioning capabilities are mainly achieved through Bluetooth beacons.

[0028] Collecting distribution point information: Select the vicinity of cultural relics and key cultural relics where tourists gather at the museum site, collect information on the location of Bluetooth beacons and cultural relics near the distribution point, and mainly determine the names of cultural relics. Data association: Edit the major value and minor value of the Bluetooth beacon and associate them with the list of cultural relics. Multimodality only needs to build a database association table to provide the ability to query based on the major value followed by the minor value. Bluetooth beacon: Team the numbered Bluetooth beacons. Positioning: The positioning capability is mainly determined by the frequency at which the Bluetooth device receives nearby Bluetooth signals. Take the strongest beacon signal among all beacons. When all 6 detections are for the same beacon code, it can be determined that the user is staying at that location. After confirming the user's stay location, the major and minor codes of the Bluetooth beacon are spliced ​​and passed into the multimodality as the Bluetooth beacon ID.

[0029] S103: Language recognition capability, mainly deployed by the open source large model Qwen-Audio large language model, and interface development can input audio and return text.

[0030] S104: After multimodality processes the incoming information, it will output two values. The first value represents the text converted from the language or the text directly input, which is defined as parameter A in the following example. The second value is the list of entity names obtained by image recognition and location detection, which is defined as parameter B in the following example. After receiving these two values, different types of data will get different outputs. For pictures and audio, they need to be uploaded to the object storage first. The specific situation is as follows: 1) Language input and output: A: [the result of speech conversion is a text], B: []; 2) Image input and output: A:[], B:[list of cultural relics names]; 3) Text input and output: A: [input text], B: []; 4) Bluetooth beacon ID input and output: A: [], B: [artifact name list].

[0031] S2: Use the intent recognition model to identify the intent and named topic of the first data passed in by S1. Perform intent recognition on the second data passed in by S1.

[0032] S201: Build the intent subject recognition service, train the intent recognition model based on BERT, and train the named subject recognition model based on NNCRF-NER. Here, BERT and NNCRF-NER (NNCRF-NER is a conditional random field model based on a neural network) are used as an algorithm to participate in the construction of the entire system. The model construction is mainly divided into the following key steps: data collection and cleaning, data labeling, model training, model evaluation and optimization, and model deployment.

[0033] Before implementation begins, the scope of implementation needs to be clarified. This scope mainly refers to a certain museum or an exhibition within a museum. Once the scope is clarified, all subsequent steps will be based on the current scope. Below, a museum will be used as the implementation target.

[0034] 1) Data collection: Based on a certain museum, we collected text data on issues in various dimensions of the museum and its cultural relics. The collection method was mainly based on web crawlers and questionnaires, supplemented by manual supplementation. We used web crawlers to automatically collect cultural relic-related information from online platforms such as Sogou, and used questionnaires to collect users' interests and frequently asked questions about cultural relics at the museum. Finally, we manually supplemented some missing data points and delivered a csv plain text file with one line of text data.

[0035] Data cleaning, delete stop words, special symbols and duplicate data, delete blanks in the text, standardize the text and convert it into a unified format to improve data quality. The cleaned data is labeled and classified according to different cultural relics information dimensions, and finally delivered as a csv plain text file with each line of text data filtered out.

[0036] 2) Labeling preparation, first of all, the intent is divided into 13 categories, including basic introduction of cultural relics, basic information, craftsmanship and materials, historical background, legends, related tasks, related deeds, digital restoration, internal structure, archaeological excavation, historical origin, video recommendation, and others. Different codes are used for the intents. For example, the basic introduction is "introduce" and the remaining 12 intents are coded with corresponding English names. Then determine the naming subject. It is necessary to clarify all the keywords that need to be included in the service, such as the name of the cultural relic, the name of the museum, and the name of the exhibition. Two TXT files are delivered, one is the intent coding TXT file separated by line breaks, and the other is the keyword TXT file separated by line breaks.

[0037] 3) Data annotation. First, the specific format of subject annotation and intent annotation for a piece of text data is explained. The annotated data can be used for both intent training and naming subject training. The specific format of annotation is "[a piece of text data] <sep>[token label column] <sep>[Intent code]", "[token label column]" starts with B-LOC as the keyword, ends with E-LOC as the keyword, the middle is represented by M-LOC, and tokens that are not keywords are represented by O. Replace "[Intent code]" with the intention code at the end of the text. The following is an example of "Introduction to Cultural Relics A" <sep>B-LOC M-LOC E-LOC OO <sep>[introduce]", the intent represented by this data is "introduce", i.e. basic introduction, and the keyword is "Cultural Relic A". The annotation method used next is based on the keyword list and annotation format confirmed in the previous step. We can use the program to complete the text keyword annotation with one click, and manually perform intent recognition annotation based on the divided intent.

[0038] Finally, write the code to divide the data into training set, validation set and test set in a ratio of 8:1:1, which is helpful for model training, tuning and evaluation. In this step, you need to deliver the training set, validation set and test set separately, each as a CSV plain text file with one line of text data.

[0039] 4) Model training: Based on the training set, the pre-trained BERT model is used for intent recognition training, and the NNCRF-NER model is used for named entity recognition model training. The evaluation of the intent recognition model is mainly based on the accuracy of the test set, and the accuracy, recall rate (measurement of the proportion of positive examples predicted correctly by the model to all actual positive examples) and F1 score (a metric used in statistics to measure the accuracy of binary classification models. It takes into account both the precision and recall rate of the classification model and can be regarded as the harmonic mean of these two indicators) are used as the measurement standards. The intent recognition model with a test accuracy of more than 90% and the named subject recognition model with an F1 score of more than 0.9 are applied to the actual system. At this stage, two models need to interact, a BERT model, which predicts the intent of the input text content, and a NNCRF-NER model, which predicts the named subject contained in the input text.

[0040] 5) Model deployment. The intention recognition and naming subject recognition models for cultural relics obtained based on the above steps need to be applied to actual scenario predictions. We use network services to provide model capabilities to the outside world. In the API, we need to process requests from the client, including verifying the format of the input data, performing necessary preprocessing, and passing the preprocessed data to the model for prediction, and finally returning the prediction results.

[0041] 6) Build a feedback loop mechanism to collect user feedback and the actual performance data of the system, and record the actual prediction data and user feedback in the database respectively. These data can be used as the basis for data source, data adjustment and model optimization.

[0042] S3: Provide services by combining the content of intent recognition and named subject recognition, optimize the final intent through prompt engineering, and build a prompt library that contains diversity and covers different user intentions. The user intent encoding and named entities have been obtained through S2, and prompts for the intent and named entities are selected from the prompt library. Combined with personalized recommendation algorithms, customized cultural relics recommendation and explanation services are provided to users based on user preferences and historical interaction behaviors.

[0043] This patent solution can achieve: meet user needs. In the cultural and museum scene, tourists have various needs. They may be interested in the historical background, production process, and artistic value of cultural relics. They may also want to know the layout and theme of the exhibition hall, or ask questions about the opening hours and tour routes of the museum. The present invention can effectively align, identify and optimize user intentions for various user inputs, provide tourists with more accurate knowledge introduction and explanations, and meet various user questions to the greatest extent. Improve service efficiency so that tourists can obtain the required information in time during the visit and will not feel impatient because of long waiting for answers. Especially in museums with large traffic, efficient answers can improve the fluency of services, reduce tourists' waiting time, and enhance the overall visiting experience. In terms of technology, this method uses an open source voice large model, combined with target detection algorithm and Bluetooth indoor positioning. It has been verified in actual scenarios, and can accurately obtain user questions, perform intention recognition, and provide relevant services. It has great feasibility in cultural and museum scenes.

[0044] Possible future application scenarios: The user intention recognition method based on multimodal applications introduced in this article has wide applicability and can play an important role in various scenarios involving the explanation of proprietary knowledge. In cultural and museum scenarios, this method can provide tourists with extremely rich and accurate cultural relics knowledge explanations. At tourist attractions, tourists can use the system to obtain introductions to historical buildings, cultural sites, etc. For example, by taking photos of ancient buildings, the system identifies and explains their historical background and architectural features.

[0045] Glossary Natural language processing: is an important branch of computer science and artificial intelligence. It is dedicated to enabling computers to understand and process human natural language, including text and speech. Its main tasks include but are not limited to: text classification, sentiment analysis, machine translation, question-answering system, information extraction, speech recognition, semantic understanding, text generation, etc. By using various technologies and algorithms, such as lexical analysis, syntactic analysis, semantic analysis, deep learning models (such as recurrent neural networks, long short-term memory networks, etc.), natural language processing continuously improves the computer's ability to understand and process natural language. Natural language processing has a wide range of applications and plays an important role in many fields such as intelligent customer service, intelligent writing, voice assistants, information retrieval, machine translation, etc. It has greatly changed the way people interact with computers and promoted the development and application of artificial intelligence.

[0046] Multimodality: refers to a mode that integrates multiple ways of expressing information or sources of information. In multimodality, it usually includes multiple different modes, such as text, images, audio, video, etc. These different modes can be combined, complemented and work together to provide richer, more comprehensive and in-depth information expression and understanding. Multimodal technology is widely used in many fields. For example, in the field of artificial intelligence, by combining image recognition and natural language processing, accurate description and understanding of image content can be achieved; in the field of education, multimodal resources can be used to improve teaching effectiveness and learning experience; in the entertainment industry, multimodal interaction can bring a more immersive experience, etc. The research and application of multimodality helps to break through the limitations of a single mode, better simulate the way humans perceive and understand the world, and promote the development of related technologies and fields.

[0047] User intent: refers to the purpose or desired goal behind a user's behavior when interacting with a system (such as software, website, etc.). Understanding user intent is the key to providing quality services and a good user experience. By analyzing the user's input content (such as text, operations, etc.), it is possible to infer different intentions of the user, such as wanting to query information, perform a task, seek help, express emotions, etc. Accurately grasping user intent helps the system make appropriate responses and better meet user needs. For example, in natural language processing, the analysis of user intent can enable intelligent customer service to answer questions or provide relevant suggestions more accurately; in search engines, it can help present search results that better meet user needs, etc. In short, in-depth exploration and understanding of user intent is of great significance to improving the performance of various applications and systems.

[0048] Decoder: Process and analyze the content containing voice information or image information, and convert the voice into corresponding text through specific algorithms and technologies. This is more common in speech recognition technology; there are also some image recognition technologies that can extract relevant text information from pictures. This process of converting information of different modes into text helps to achieve unified processing of information and further understanding and analysis, and plays an important role in many intelligent applications.

[0049] Object detection: It is an important task in the field of computer vision, which aims to identify and locate the positions of multiple different objects in images or videos.

[0050] Neural network: It is a computational model that imitates the structure and function of biological neural networks. It is composed of nodes (neurons) that are interconnected. After the information is passed to the input layer, it is processed by multiple hidden layers and then passed to the output layer for final processing. It has strong nonlinear fitting ability, self-learning ability and adaptability. It can automatically learn patterns and features in data through training, and predict and classify new data.

[0051] BERT: A pre-trained language model that learns universal language representations from text data through large-scale unsupervised training. The main innovation of BERT is to use the Transformer architecture for bidirectional (i.e., bi-context) pre-training, which can better understand the semantics and grammar in the context of text.

[0052] NNCRF-NER: is a sequence labeling model that combines neural networks and conditional random fields, specifically for named entity recognition. NER is a key task in natural language processing, and its goal is to identify named entities with specific meanings in text, such as names of people, places, and organizations.

[0053] The ‌major value and ‌minor value of Bluetooth beacons are two broadcast parameters of Bluetooth beacons that can be customized and modified. These values ​​are mainly used to identify and distinguish different beacon devices, helping users to configure and modify them through specific mobile phone applications.

[0054] Qwen-Audio: It is a large speech understanding model open sourced by Alibaba. It can effectively perceive audio signals, take a variety of audio (including speaker voice, natural sound, music, singing) and text as input, and output text.< / sep> < / sep> < / sep> < / sep> < / height> < / width> < / height> < / width> < / sep> < / sep> < / sep> < / sep>

Claims

1. A cultural relic explanation method based on multimodal user intention recognition and alignment, characterized in that The steps include: S1. First, align the multimodal input content, which includes voice, text, pictures, and locations; deploy the Qwen-Audio large model with voice recognition capabilities; realize image recognition and select yolov8 for target detection model training; S2. Identify through the intent recognition model; build the intent subject recognition service, train the intent recognition model based on BERT, and train the named subject recognition model based on NNCRF-NER; Data collection: collect text data on issues in museums and cultural relics in various dimensions. The collection method is mainly web crawlers and questionnaires, supplemented by manual supplementation. Web crawlers are used to automatically collect cultural relic-related information from online platforms such as Sogou. Questionnaires are used to collect users' interests and frequently asked questions about cultural relics at the museum. Finally, some missing data points are supplemented manually, and a csv plain text file with one line of text data is delivered. Data cleaning: delete stop words, special symbols, and duplicate data, delete blanks in the text, standardize the text to convert it into a unified format, annotate and classify the cleaned data according to different dimensions of cultural relic information, and finally deliver a csv plain text file with one line of text data after filtering. The first step in labeling preparation is to divide the intent into 13 categories: basic introduction of cultural relics, basic information, craftsmanship and materials, historical background, legends, related tasks, related deeds, digital restoration, internal structure, archaeological excavation, historical origin, video recommendation, and others. Different codes are given to the intent, and then the naming subject is determined. It is necessary to clearly define all the keywords that need to be included in the service, such as the name of the cultural relic, the name of the museum, and the name of the exhibition. Two TXT files are delivered, one is the intent coding TXT file separated by line breaks, and the other is the keyword TXT file separated by line breaks. Data annotation, first of all, the specific format of subject annotation and intent annotation for a piece of text data is explained. The annotated data can be used for intent training as well as naming subject training. The specific format of annotation is "[a piece of text data] <sep>[token label column] <sep> [Intent code]", "[token label column]" starts with B-LOC as the keyword, ends with E-LOC as the keyword, the middle is represented by M-LOC, and the token that is not a keyword is represented by O. At the end of the text, replace the "[Intent code]" mark with the intent code;< / sep> < / sep> Model training: use the pre-trained BERT model for intent recognition training and NNCRF-NER for named entity recognition model training; Model deployment, using network services to provide model capabilities to the outside world. In the API, it is necessary to process requests from clients, including verifying the format of input data, performing necessary preprocessing, and passing the preprocessed data to the model for prediction, and finally returning the prediction results; S3: Provide services by combining content with intent recognition and named subject recognition, build a prompt library that contains diversity and covers different user intentions, obtain user intent encoding and named entities through S2, select prompts for the intent and named entities from the prompt library, and combine with personalized recommendation algorithms to provide users with customized cultural relics recommendation and explanation services based on user preferences and historical interaction behaviors.

2. According to claim 1, a method for explaining cultural relics based on multimodal user intention recognition and alignment is characterized in that The S1 is divided into: S101, collect pictures of cultural relics in actual scenes by on-site shooting, and train them by data annotation, and finally form a target detection model; Data collection: First determine the list of cultural relics to be image-recognized, and generate labels for each cultural relic according to the requirements of [museum code], [exhibition hall code], and [cultural relic code]; Record video; extract frames from the cultural relic video and use the images obtained as training data; Data labeling: label the cultural relics according to the cultural relic labels; after labeling, a labeling file will be generated, and a txt format of labeling data will be generated; After the data is labeled, the data is used to train the target detection model; S102. Using indoor positioning of Bluetooth beacons, select the vicinity of cultural relics where tourists gather and key cultural relics at the museum site, and collect the tag information of the Bluetooth beacon locations and cultural relics near the locations; edit the major value and minor value of the Bluetooth beacon and associate them with the cultural relic list. Multimodality only needs to build a database association table to provide the ability to query based on the major value followed by the minor value; The frequency of nearby Bluetooth signals received by the Bluetooth device, when detecting the same beacon code, can determine that the user is staying at that location; Concatenate the major and minor codes of the Bluetooth beacon and pass them into the multi-mode as the Bluetooth beacon ID; S103: Language recognition capability, deploying the open source large model Qwen-Audio large language model to implement, and developing interfaces to input audio and return text; S104: After processing the incoming information, the multimodal will output two values. The first value represents the text converted from the language or the text directly input as parameter A, and the second value is the list of entity names obtained by image recognition and position detection as parameter B. After receiving these two values, different types of data will get different outputs. For pictures and audio, they need to be uploaded to the object storage first. The specific situation is as follows: Language input and output: A: [the result of speech conversion is a text], B: []; Image input and output: A:[], B:[list of cultural relics names]; Word input and output: A: [incoming text], B: []; Bluetooth beacon ID incoming output: A: [], B: [artifact name list].

Images