Science popularization venue audience portrait generation and personalized resource recommendation system based on multi-modal data

Abstract

The invention relates to the technical field of science popularization venue service, and particularly discloses a science popularization venue audience portrait generation and personalized resource recommendation system based on multi-modal data. Comprising a science popularization venue audience portrait multi-modal data collection unit, a science popularization venue audience portrait model construction unit based on a neural network and a data-driven science popularization venue audience portrait recommendation unit. The science popularization venue audience portrait multi-modal data collection unit comprises a science popularization venue audience portrait multi-modal data collection module and a data processing module, and a science popularization venue audience portrait model construction unit based on a neural network. Comprising a training module and a testing module, a data-driven science popularization venue audience portrait recommendation unit comprises a personalized exhibit recommendation module and a visiting route recommendation module, customized science popularization contents and visiting routes can be provided according to specific requirements and interest points of each audience, and transformation from extensive science popularization venue service to precise science popularization venue service is achieved.

Description

Technical Field

[0001] The present invention belongs to the technical field of science popularization venue service, and specifically relates to a science popularization venue audience portrait generation and personalized resource recommendation system based on multimodal data. Background Art

[0002] With the rapid development of information technology and the Internet, science and technology venues are facing the challenge of transforming from traditional exhibitions to digital and intelligent ones. In this transformation process, how to use multimodal data (including but not limited to audience behavior data, preference information, interaction records, etc.) to build audience portraits and provide personalized resource recommendations based on these portraits has become the key to improving user experience and venue service levels.

[0003] At present, there is a relative lack of research on the generation and application of audience portraits in science and technology museums in China. By searching with the keywords "science and technology museum" and "personalized recommendation" in CNKI, only one relevant paper was found, and there was no record of patents. This paper mainly discusses the limitations of traditional collaborative filtering algorithms, and improves on this basis, proposing a user-exhibit evaluation matrix method based on user ratings of exhibits to achieve personalized recommendations. Further searches with the keywords "science and technology museum" and "audience portrait" retrieved a total of 2 papers, and there were also no patent documents. Among them, the first paper adopted a direct "headcount" survey method to objectively and truly collect audience data, providing practical guidance for the strategy formulation of science and technology museums. The second paper only explored the potential and possibility of using big data technology for audience portrait analysis at the theoretical level, but has not yet been implemented.

[0004] Some studies focus on the generation of audience portraits in museum settings, using big data technology to collect and label audience data, and using data mining technology to analyze audience behavior characteristics and interest preferences, discovering the commonalities and differences between audience behaviors, thereby constructing audience portraits and dividing audiences into different types. However, most of these studies only stay at the stage of constructing audience portraits, and have not yet fully utilized the neural network technology in deep learning to develop and utilize data at a deeper level, and lack in-depth discussion of audience portraits at the specific application level.

[0005] At present, China's science and technology venues, such as the China Science and Technology Museum and the Shanghai Science and Technology Museum, have not yet been able to provide deeply personalized exhibit recommendation services and tour route planning. These venues mainly rely on web page information provided by the official website to introduce each exhibition, and route recommendations are mainly based on traditional classifications, such as fixed classifications based on visit duration or visitor groups. Although these methods have met the basic needs of the public to a certain extent, there is still a clear gap in the application of personalized resource recommendations. This situation limits the potential of science and technology venues in providing customized visitor experiences and misses the opportunity to use advanced technologies to improve visitor satisfaction and engagement.

[0006] In practical applications, how to effectively integrate multimodal data, build accurate audience portraits, and achieve personalized resource recommendations on this basis still faces technical challenges. Therefore, the present invention aims to propose a science and technology museum audience portrait generation and personalized resource recommendation system based on multimodal data to address the deficiencies in the existing technology, promote the digital transformation of science and technology museum services, and improve audience experience and satisfaction. Summary of the invention

[0007] The purpose of the present invention is to provide a science museum audience portrait generation and personalized resource recommendation system based on multimodal data to solve the problems raised in the above background technology.

[0008] To achieve the above object, the present invention provides the following technical solutions:

[0009] A science and technology venue audience portrait generation and personalized resource recommendation system based on multimodal data, including a science and technology venue audience portrait multimodal data collection unit, a science and technology venue audience portrait model building unit based on a neural network, and a science and technology venue audience portrait recommendation unit driven by data;

[0010] Process 1: The multimodal data collection unit for the portrait of the audience of the science and technology venue includes a multimodal data collection module and a data processing module for the portrait of the audience of the science and technology venue. The data collection module includes basic attribute data and visiting behavior data. The basic attribute data includes gender, age, native place, education level, major, and occupation type. The basic attribute data is collected through the reservation service system and stored in a structured form. The visiting behavior data includes visiting routes, dwelling time, and expressions of visiting exhibits. The visiting behavior data comes from the positioning system, the guide app, and the exhibition hall camera. The visiting route records the timestamp and location coordinates of the audience when moving in the venue, and the dwelling time records the start and end time of the audience's stay at each exhibit. The expressions of visiting exhibits are stored in the form of images.

[0011] Before inputting the data into the neural network, the data processing module cleans and preprocesses the raw data. Multimodal data is mainly divided into text data and image data according to the data type. For text data, it is necessary to delete information and special symbols irrelevant to the text analysis task, and fill or delete missing values. For image data, it is necessary to perform image cropping, format conversion, denoising, and normalization operations, and delete low-quality and non-standard images to provide an accurate basis for subsequent analysis;

[0012] Process 2: The model building unit of the science and technology museum audience portrait based on the neural network includes a training module and a testing module:

[0013] The training module uses the basic attribute data collected in the early stage as text data, and the expressions of visitors to the exhibits as image data, and makes a one-to-one correspondence between the text data and image data of the same visitor to form a data pair with the visitor as the basic unit, which is used as the input of the text classification network and the image classification network. The visiting route, stay time, and the expression data of the visitors to the exhibits are used as the basis for whether the visitors are interested in the exhibit type, and the degree of interest is scored as the output true value of the neural network. The specific formula is:

[0014] s=αS route +βS duration +γS expression

[0015] Among them, S is the final interest score, that is, the true output value of the neural network, S route , S duration and expression are the scores of the visiting route, the stay time and the visiting expression, respectively, and α, β and γ are the corresponding weights;

[0016] The two networks are trained separately to obtain their own prediction results. The prediction results of the two network branches are weighted averaged to obtain the final prediction result after multimodal fusion, that is, the interest level score of the same visitor for different exhibit types;

[0017] Compare the final prediction result of the model with the actual true value, perform network back propagation and adjust parameters; in this process, the rich information contained in the text will be transferred to the image representation through the bridge of fusion representation, and the information contained in the image will also be transferred to the text representation through the bridge; in this way, data information of different modes will assist each other, thereby improving the prediction effect of the multimodal network;

[0018] The test module has been fully trained and parameterized for the network, and has learned the interest characteristics of visitors with different labels for the types of exhibits in the science and technology museum. In the test phase, the basic attribute data of the visitors is provided as input, but the expression data of the visitors is no longer provided. The network is required to make predictions based on single modal data to obtain the interest scores of the visitors for different types of exhibits.

[0019] Process 3: The data-driven recommendation unit for the visitor portrait of science and technology museums includes a personalized exhibit recommendation module and a tour route recommendation module:

[0020] Personalized exhibit recommendation module: For visitors who visit science and technology museums through the reservation service system, they can choose whether to need personalized recommendation services according to their personal wishes. For those who do not need personalized recommendation services, only the most basic personal information is collected for venue reservation. For those who need personalized recommendation services, additional data such as place of origin, education level, major, and occupation type are collected as input data. The network predicts interests based on the learned features, scores different types of exhibits, and sorts and intelligently recommends them based on the scores. When recommending exhibits, it can combine user interests and use collaborative filtering algorithms and keyword retrieval methods to recommend extended resources or supplementary knowledge to popularize exhibit-related knowledge and promote the audience's embodied experience and learning effect. The specific formula is:

[0021]

[0022] Among them, i k and i l They are two different exhibits. is the audience j's attitude towards exhibit i k Interest score, It is an exhibit k The average score of the two exhibits, J is the set of visitors who have scored both exhibits. According to this formula, the similarity between exhibits can be calculated to recommend exhibits of the same type.

[0023] Tour route recommendation module: Based on the expected time schedule for visiting science and technology venues, the audience is roughly divided into two categories, namely, visitors and learners. The expected length of time for the audience to visit the science and technology venue is used as a constraint, and a personalized tour route is formulated in combination with the path planning algorithm, so that the audience's interest score in visiting exhibits is maximized while meeting the time constraint.

[0024] Preferably, the network model selected for the text information recognition processing includes but is not limited to the Word2Vec model and the BERT model, and recognizes and classifies the basic attribute data of the audience collected through the reservation service system, the official website, etc.;

[0025] The model can map words into a continuous vector space, so that words with similar semantics are close in distance in the vector space, thereby mining the common features of audiences with similar basic attribute data. The specific formula is:

[0026] E:υ→R d

[0027]

[0028] Among them, υ is the vocabulary, which contains the set of all words, E is the embedding function, which maps each word to a d-dimensional vector space, and R d Represents a d-dimensional real vector space. For two words w1 and w2, the Euclidean distance is used to measure their distance in the vector space.

[0029] Preferably, the image information recognition processing is to obtain the facial expression images of the audience when visiting the exhibits through the exhibition hall camera, and the selected model includes but is limited to the FaceNet model and the ArcFace model to recognize and classify the facial expressions;

[0030] After the facial expression image is input into the network, it undergoes a series of convolution and pooling operations to obtain a fixed-dimensional facial feature vector. The specific formula is:

[0031] H P =Pooling(ReLU(W c *I+b c ))

[0032] Among them, Pooling is the pooling operation, H P is an important feature obtained after the pooling operation, W c and b c are the weights and biases of the convolutional layer, * represents the convolution operation, and I is the input image. The neural network model analyzes the emotions contained in these expressions and scores the degree of interest based on positivity, neutrality, and negativity, thereby making personalized recommendations.

[0033] Preferably, the text classification network and the image classification network are trained independently to obtain their respective prediction outputs. In the final decision stage of the network, the prediction results of the two branches are combined by weighted fusion to achieve information complementarity and sharing between different modalities. The specific formula is:

[0034] rf=ωy text +(1-ω)y image

[0035] Among them, rf is the final interest score vector, y text and imageare the outputs of the text classification network and the image classification network respectively. ω is the weight parameter used to balance the influence of the two network branches. The fused prediction results are compared with the actual labels to evaluate the performance of the model.

[0036] Compared with the prior art, the present invention has the following beneficial effects:

[0037] Promoting the digital transformation of science and technology venues: The system can provide customized popular science content and tour routes based on the specific needs and interests of each visitor, realizing the transformation of popular science venue services from extensive to precise. At the same time, through the analysis of visit data, the system can identify which exhibits are more popular and which areas require more maintenance or updates, thereby guiding the rational allocation of venue resources and the updating of exhibits, improving resource utilization and the overall service level of the venue, and providing data support and decision-making basis for the long-term development of the venue.

[0038] 2. Improve the scientific literacy of the whole people: The system can stimulate the public's interest and curiosity in science by accurately pushing popular science information that the audience is interested in, thereby increasing their enthusiasm for actively learning and exploring scientific knowledge. This personalized service method helps to improve the pertinence and appeal of popular science education, allowing more people to access scientific knowledge. At the same time, it also improves the actual effect of popular science education, making the dissemination of scientific knowledge more efficient and in-depth, and achieving the goal of improving the scientific literacy of the whole people. BRIEF DESCRIPTION OF THE DRAWINGS

[0039] Figure 1 It is the audience portrait data framework diagram of the present invention;

[0040] Figure 2 is a system architecture diagram of the present invention;

[0041] Figure 3 It is a neural network framework diagram of the present invention;

[0042] Figure 4 This is a system reservation process framework diagram of the present invention. DETAILED DESCRIPTION

[0043] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0044] Embodiment 1:

[0045] System preparation and data collection initialization

[0046] Hardware and software deployment

[0047] Install high-performance computing equipment in the science venue’s server room, including installing and configuring relevant software, including database management systems (such as MySQL or PostgreSQL) for storing audience data, and deep learning frameworks (such as TensorFlow or PyTorch) for model building and training.

[0048] Improve the reservation service system, and clearly require filling in detailed basic attribute data during the user registration and reservation process, including gender, age, place of origin, education level, major, occupation type, etc., and perform data format verification to ensure the accuracy and completeness of the data.

[0049] Reasonably arrange positioning system equipment in the science and technology venues to ensure accurate tracking of visitors' movements in the venue. Optimize the functions of the guide app so that it can accurately record the time visitors stay at each exhibit and synchronize data with the server in real time. Install high-definition cameras to cover each exhibit area in the venue to ensure clear images of visitors' facial expressions when visiting the exhibits.

[0050] 2. Multimodal Data Collection

[0051] Basic attribute data collection example

[0052] Visitor Xiao Li made an appointment to visit the science museum through its official website. When filling in the appointment information, he entered his name as "Li Hua", gender as "male", age as "28 years old", place of origin as "Shenzhen, Guangdong", education as "bachelor", major as "electronic information engineering", and occupation as "hardware engineer". This information was encrypted, transmitted, and stored in the database of the reservation service system, forming BasicInfo{male, 28, Shenzhen, Guangdong, bachelor, electronic information engineering, hardware engineer}.

[0053] Visitor behavior data collection

[0054] When Xiao Li entered the science museum, the positioning system began to track his location. He first entered the museum gate at 10:00, then arrived at the exhibit "Optical Principle Display Stand" at 10:10, and stayed there until 10:25. The tour guide app recorded this stay time information. At the same time, the exhibition hall camera captured an image of his facial expression when he was looking at the exhibit, and the image showed that he was focused and curious. These tour routes, stay time, and exhibition expression data are collected and stored as Behavior{tour route record (including time and location information), stay time (10:10-10:25), exhibition expression image (image at the optical principle display stand)}.

[0055] 3. Data Cleaning and Preprocessing

[0056] Text data cleaning

[0057] Extract basic attribute data of all viewers from the database. For each record, check whether there is duplicate data, such as by comparing key information such as name and ID number (if collected) to identify duplicate records, and delete duplicate entries.

[0058] Clean up special symbols and irrelevant characters in text data. For example, if an incorrect input such as "engineer&" appears in the occupation type, delete the "&" symbol. At the same time, check whether there are missing values. If it is found that the native place information of a certain audience is missing, try to make reasonable inferences and fill it in based on the place of origin of their mobile phone number or other relevant information; if it cannot be filled in, decide whether to delete the record based on the proportion of missing data.

[0059] Image data preprocessing

[0060] Get all the facial expression images of visitors to exhibits from the data storage. Crop the images to remove the redundant background around the images, such as cropping out irrelevant scenery around the heads of visitors and only retaining the facial expression area.

[0061] Convert the image format to the commonly used .jpg format for subsequent processing. Use image denoising algorithms (such as median filtering or Gaussian filtering) to remove noise in the image and improve image quality. Finally, normalize the image and map the pixel value to the [0,1] interval using the formula: Normalized pixel value = (original pixel value - minimum value) / (maximum value - minimum value). For low-quality images that are blurred and cannot recognize expressions, judge and delete them according to the set clarity threshold.

[0062] 4. Model construction based on neural network - training stage

[0063] Data matching and annotation

[0064] The cleaned and preprocessed basic attribute data is used as text data, and the expression of the visitor when visiting the exhibits is used as image data. The text data and image data of the same visitor are matched one by one according to the visitor ID to form a data pair. For example, the basic attribute data of Xiao Li is matched with the image data of his expression in front of each exhibit.

[0065] Based on Xiao Li's tour route, stay time, and expression when visiting exhibits, his interest in different types of exhibits is marked and scored. Assuming that he stays for a long time at the "Optical Principle Display Stand" and his expression is focused, he is given a higher interest score, such as 4 points; while when he passes by the "Simple Mechanical Transmission Device", he stays for a short time and his expression is dull, so he is given 2 points. These scores are used as the true output values ​​of the neural network and are used together with the corresponding input data pairs for model training.

[0066] Model selection and training

[0067] Select a suitable text classification network model, such as the BERT model, and initialize the parameters. Input the paired text data into the BERT model for training. By adjusting the weights and biases of the model, the model can learn the relationship between different basic attribute data and audience interests, such as learning that audiences majoring in electronic information engineering may be more interested in electronic exhibits.

[0068] At the same time, the FaceNet model was selected as the image classification network model to train the facial expression images of visitors to the exhibits. The model extracts features from the images through convolutional layers and pooling layers, converts the facial expression images into fixed-dimensional facial feature vectors, and scores the audience's interest level based on these feature vectors. For example, the feature vector corresponding to Xiao Li's focused expression at the "Optical Principle Display Stand" is identified and mapped to a higher interest score range.

[0069] During the training process, the two networks are trained separately to obtain prediction results. For example, the BERT model predicts that Xiao Li's interest score for electronic exhibits is 3.5 points, and the FaceNet model predicts 4 points. Then, according to the set weights (such as the text classification network weight is 0.4, and the image classification network weight is 0.6), the prediction results of the two network branches are weighted averaged to obtain the final prediction result after multimodal fusion, that is, Xiao Li's final interest score for the "Optical Principle Display Stand": points. This final prediction result is compared with the actual annotated true value (4 points), the loss function (such as the mean square error loss function) is calculated, and the parameters of the model are adjusted through the back-propagation algorithm to continuously optimize the performance of the model.

[0070] 5. Neural Network-Based Model Construction-Testing Phase

[0071] Single modal data testing

[0072] After the model is fully trained, it enters the testing phase. A new batch of visitor data is selected, such as the data of visitor Xiao Wang. Only Xiao Wang's basic attribute data (gender "female", age "35 years old", native place "Nanjing, Jiangsu", education level "Master", major "bioengineering", occupation type "scientific researcher") is provided as input to the trained model, and his facial expression data when visiting the exhibits is not provided.

[0073] The model uses the learned features to predict the types of exhibits that Xiao Wang may be interested in and derive an interest score. For example, the model predicts that Xiao Wang's interest score for biological exhibits is 4.2 points, and his interest score for physical exhibits is 3 points, etc. By comparing the predicted results with the actual situation (which can be verified through subsequent visitor feedback or actual visit behavior data), the accuracy and generalization ability of the model in the case of single-modal data are evaluated.

[0074] Data-driven personalized exhibit recommendations

[0075] Audience demand response and data acquisition

[0076] Visitor Xiao Zhao made an appointment to visit the science museum through the mobile phone application and selected the personalized recommendation service. The system obtained Xiao Zhao's basic information, including ID number "44010119900101XXXX", mobile phone number "13888888888", and additional data such as native place "Changsha, Hunan", education level "PhD", major "Chemical Engineering and Technology", and occupation type "college teacher".

[0077] Interest prediction and recommendation generation

[0078] Input Xiao Zhao's input data into the trained model, and the model predicts and scores the interest of different types of exhibits based on the learned features. Suppose the model predicts that Xiao Zhao has the highest interest score of 4.5 for chemistry exhibits, and 3.8 for materials exhibits, etc.

[0079] The exhibits are sorted according to the scores, and exhibits with higher scores are recommended to Xiao Zhao, such as "New Catalyst Display" and "Polymer Material Synthesis Process Demonstration". At the same time, combined with the collaborative filtering algorithm, other exhibits that have been viewed or rated highly by other viewers with similar interests as Xiao Zhao are found for supplementary recommendations. For example, it was found that several college teachers majoring in chemistry had a high evaluation of the "Nanomaterial Performance Test Device", and they were also recommended to Xiao Zhao. In addition, using keyword retrieval technology, popular science articles, video lectures and other extended resources related to the recommended exhibits are searched from the knowledge base of the science museum. For example, an article on "Progress in the Research and Development of New Catalysts" and a video on "Application of Polymer Materials in Daily Life" are found and recommended to Xiao Zhao to enrich his visit and learning content.

[0080] 7. Data-driven tour route recommendations

[0081] Audience classification and time analysis

[0082] When making an appointment for the visit, visitor Xiao Qian indicated that his expected visit time was 3.5 hours. Based on this time information and his basic attribute data (gender "male", age "40 years old", native place "Beijing", education level "undergraduate", major "mechanical engineering", occupation type "engineer"), the system determined that he was a learning visitor, because he had relatively ample time for the visit and his occupation was related to mechanical engineering, so he might focus more on in-depth learning of the exhibits.

[0083] Route planning and optimization

[0084] The system combines the path planning algorithm to develop a tour route based on Xiao Qian's interest score in the exhibits. For example, the model predicts that Xiao Qian has a high interest score in mechanical exhibits and a general interest in electronic exhibits. There are mechanical exhibition area A, electronic exhibition area B, and comprehensive technology exhibition area C in the venue. The system first plans to visit the key exhibits in the mechanical exhibition area A, such as "Advanced CNC Machining Center Demonstration" and "New Engine Principle Display", and arranges the order reasonably according to the distance between the exhibits and the visiting time. Then, considering that there are also some interdisciplinary exhibits related to mechanical engineering in the comprehensive technology exhibition area C, such as "Intelligent Control System Fusion of Mechanics and Electronics", it is also included in the tour route. During the planning process, ensure that the total time of the entire tour route does not exceed 3.5 hours, and at the same time try to maximize Xiao Qian's exposure to the exhibits he is interested in during the visit, so as to improve the visiting experience and learning effect.

[0085] Among them, the recommended tour route: according to the expected time arrangement for visiting science and technology venues, the audience can be roughly divided into two categories, visiting audience and learning audience. The visiting audience mostly visits for the purpose of tourism and punching in. They have a low understanding of science and technology venues before visiting. Due to the full schedule, the time reserved for visiting science and technology venues is relatively short. Learning audiences are more interested in the exhibition itself, and mostly aim to increase their own knowledge reserves. They have a higher degree of disposable time for visiting science and technology venues. Here, the expected length of time for the audience to visit the science and technology venue is used as a constraint condition, and a personalized tour route is formulated in combination with the path planning algorithm. Under the premise of meeting the time constraint, the audience's interest score in visiting the exhibits is maximized, thereby improving the audience's experience and satisfaction and meeting the personalized needs of different types of audiences. Some codes are shown below.

[0086]

[0087] The text data reference table of the present invention is as follows:

[0088] Basic attribute data table

[0089]

[0090] Tour route data sheet

[0091]

[0092] Dwell time data

[0093]

[0094] It will be appreciated that in the development of any actual implementation, as in any engineering or design project, numerous implementation-specific decisions may be made. Such a development effort may be complex and time-consuming, but will be a routine task of design, fabrication, and production for those of ordinary skill having the benefit of this disclosure without undue experimentation.

[0095] It should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention rather than to limit it. Although the present invention has been described in detail with reference to the preferred embodiments, those skilled in the art should understand that the technical solutions of the present invention may be modified or replaced by equivalents without departing from the spirit and scope of the technical solutions of the present invention, which should all be included in the scope of the claims of the present invention.

Claims

1. A science museum audience portrait generation and personalized resource recommendation system based on multimodal data, characterized by: It includes a multimodal data collection unit for the profile of the audience of science and technology venues, a model building unit for the profile of the audience of science and technology venues based on a neural network, and a data-driven recommendation unit for the profile of the audience of science and technology venues; Process 1: The multimodal data collection unit for the portrait of the audience of the science and technology venue includes a multimodal data collection module and a data processing module for the portrait of the audience of the science and technology venue. The data collection module includes basic attribute data and visiting behavior data. The basic attribute data includes gender, age, native place, education level, major, and occupation type. The basic attribute data is collected through the reservation service system and stored in a structured form. The visiting behavior data includes the visiting route, the stay time, and the expression of visiting exhibits. The visiting behavior data comes from the positioning system, the guide app and the exhibition hall camera. The visiting route records the timestamp and location coordinates of the audience when moving in the venue, and the stay time records the start and end time of the audience's stay at each exhibit. The expression of visiting exhibits is stored in the form of images; Before inputting the data into the neural network, the data processing module cleans and preprocesses the raw data. Multimodal data is mainly divided into text data and image data according to the data type. For text data, it is necessary to delete information and special symbols irrelevant to the text analysis task, and fill or delete missing values. For image data, it is necessary to perform image cropping, format conversion, denoising, and normalization operations, and delete low-quality and non-standard images to provide an accurate basis for subsequent analysis; Process 2: The model building unit of the science and technology museum audience portrait based on the neural network includes a training module and a testing module: The training module uses the basic attribute data collected in the early stage as text data, and the expressions of visitors to the exhibits as image data, and makes a one-to-one correspondence between the text data and image data of the same visitor to form a data pair with the visitor as the basic unit, which is used as the input of the text classification network and the image classification network. The visiting route, stay time, and the expression data of the visitors to the exhibits are used as the basis for whether the visitors are interested in the exhibit type, and the degree of interest is scored as the output true value of the neural network. The specific formula is: s=αS route +βS duration +γS expression Among them, S is the final interest score, that is, the true output value of the neural network, S route , S duration and expression are the scores of the visiting route, the stay time and the visiting expression, respectively, and α, β and γ are the corresponding weights; The two networks are trained separately to obtain their own prediction results. The prediction results of the two network branches are weighted averaged to obtain the final prediction result after multimodal fusion, that is, the interest level score of the same visitor for different exhibit types; Compare the final prediction result of the model with the actual true value, perform network back propagation and adjust parameters; in this process, the rich information contained in the text will be transferred to the image representation through the bridge of fusion representation, and the information contained in the image will also be transferred to the text representation through the bridge; in this way, data information of different modes will assist each other, thereby improving the prediction effect of the multimodal network; The test module has been fully trained and parameterized for the network, and has learned the interest characteristics of visitors with different labels for the types of exhibits in the science and technology museum. In the test phase, the basic attribute data of the visitors is provided as input, but the expression data of the visitors is no longer provided. The network is required to make predictions based on single modal data to obtain the interest scores of the visitors for different types of exhibits. Process 3: The data-driven recommendation unit for the visitor portrait of science and technology museums includes a personalized exhibit recommendation module and a tour route recommendation module: Personalized exhibit recommendation module: For visitors who visit science and technology museums through the reservation service system, they can choose whether to need personalized recommendation services according to their personal wishes. For those who do not need personalized recommendation services, only the most basic personal information is collected for venue reservation. For those who need personalized recommendation services, additional data such as place of origin, education level, major, and occupation type are collected as input data. The network predicts interests based on the learned features, scores different types of exhibits, and sorts and intelligently recommends them based on the scores. When recommending exhibits, it can combine user interests and use collaborative filtering algorithms and keyword retrieval methods to recommend extended resources or supplementary knowledge to popularize exhibit-related knowledge and promote the audience's embodied experience and learning effect. The specific formula is: Among them, i k and i l They are two different exhibits. is the audience j's attitude towards exhibit i k Interest score, It is an exhibit k The average score of the two exhibits, J is the set of visitors who have scored both exhibits. According to this formula, the similarity between exhibits can be calculated to recommend exhibits of the same type; Tour route recommendation module: Based on the expected time schedule for visiting science and technology venues, the audience is roughly divided into two categories, namely, visitors and learners. The expected length of time for the audience to visit the science and technology venue is used as a constraint, and a personalized tour route is formulated in combination with the path planning algorithm, so that the audience's interest score in visiting exhibits is maximized while meeting the time constraint.

2. The system for generating audience portraits and recommending personalized resources for science museums based on multimodal data according to claim 1, characterized in that: The network model selected for text information recognition processing includes but is not limited to Word2Vec model and BERT model, which recognizes and classifies the basic attribute data of the audience collected through the reservation service system, official website, etc.; The model can map words into a continuous vector space, so that words with similar semantics are close in distance in the vector space, thereby mining the common features of audiences with similar basic attribute data. The specific formula is: E:υ→R d Among them, υ is the vocabulary, which contains the set of all words, E is the embedding function, which maps each word to a d-dimensional vector space, and R d Represents a d-dimensional real vector space. For two words w1 and w2, the Euclidean distance is used to measure their distance in the vector space.

3. The system for generating audience portraits and recommending personalized resources for science museums based on multimodal data according to claim 1, characterized in that: The image information recognition processing is to obtain the facial expression images of the visitors when visiting the exhibits through the exhibition hall camera, and the selected models include but are limited to the FaceNet model and the ArcFace model to recognize and classify the facial expressions; After the facial expression image is input into the network, it undergoes a series of convolution and pooling operations to obtain a fixed-dimensional facial feature vector. The specific formula is: A P =Pooling(ReLU(W c *I+b c )) Among them, Pooling is the pooling operation, H P is an important feature obtained after the pooling operation, W c and b c are the weights and biases of the convolutional layer, * represents the convolution operation, and I is the input image. The neural network model analyzes the emotions contained in these expressions and scores the degree of interest based on positivity, neutrality, and negativity, thereby making personalized recommendations.

4. The system for generating audience portraits and recommending personalized resources for science popularization venues based on multimodal data according to claim 1, characterized in that: The text classification network and the image classification network are trained independently to obtain their respective prediction outputs. In the final decision stage of the network, the prediction results of the two branches are combined by weighted fusion to achieve information complementarity and sharing between different modalities. The specific formula is: rf=ωy text +(1-ω)y image Among them, rf is the final interest score vector, y text and image are the outputs of the text classification network and the image classification network respectively. ω is the weight parameter used to balance the influence of the two network branches. The fused prediction results are compared with the actual labels to evaluate the performance of the model.

Images