An intelligent comprehensive service system based on multi-modal data fusion

Abstract

The present application relates to the technical field of intelligent integrated service system, specifically relates to an intelligent integrated service system based on multi-modal data fusion, aiming to meet the growing demand for intelligent in various fields, break through the limitations of existing intelligent devices and services. The system integrates a variety of cutting-edge technologies, realizing a series of rich and practical functions. The system has powerful voice interaction function, based on advanced natural language processing technology, supports multi-local dialect and colloquial expression, can accurately understand user intention and make intelligent response. In terms of entertainment, it has a large number of multimedia resource library such as songs, songs, etc. Through deep learning algorithm analysis of user preference, personalized content recommendation is realized. The health monitoring function can recognize posture, detect snoring sound, detect blood oxygen and basic human function with various sensors, collect and analyze data in real time, and provide comprehensive protection for user health.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of intelligent integrated service systems, in particular to an intelligent integrated service system based on multi-modal data fusion. BACKGROUND

[0002] In today's rapidly developing information age, the importance of intelligent integrated service systems is increasingly prominent, from smart home control to personal health management, its application scenarios continue to expand. Especially in complex indoor environments or personal privacy space, there are challenges of inaccurate data collection and inaccurate service provision, so that intelligent integrated service systems must rely on more advanced technology to ensure their reliability and practicality. Although the traditional service system meets the needs of users to a certain extent, there are still obvious shortcomings in data processing and service personalization. Single data source and simple data processing method make it difficult for the service system to accurately understand user needs. This reality has driven the intelligent service system to transform towards multi-modal data fusion and deep analysis, reducing the dependence on single data and simple algorithms. In the process of building an efficient intelligent integrated service system, accurately collecting and processing multi-modal data is a key task. It not only allows the service system to analyze user needs, but also solves the problem of personalized service provision. Traditionally, service systems mainly rely on a single type of data to provide services, which is relatively easy to achieve in simple scenarios and standard user needs. But the diversity and complexity of user needs in real life require service systems to include more data types and more advanced data processing methods. Therefore, intelligent integrated service systems need to integrate data from multiple sensors and data sources to capture user behavior details and characteristics. This multi-modal data fusion becomes a solution to ensure that the service system can accurately provide personalized services in complex environments and diverse user needs. In order to improve the service quality and user satisfaction of intelligent integrated service systems, multiple data types need to be combined when processing user data to obtain more comprehensive user characteristics and demand information. SUMMARY

[0003] In view of this, the present application proposes an intelligent integrated service system based on multi-modal data fusion to solve at least one of the problems in the above background technology.

[0004] The present application provides an intelligent integrated service system based on multi-modal data fusion, comprising: a data acquisition and knowledge base construction module, the data acquisition and knowledge base construction module is used for collecting human multi-modal data, and constructing a comprehensive knowledge base of song resources, health knowledge, and knowledge in the field of dietary nutrition;

[0005] The data structuring and model simplification module is used to apply natural language processing algorithms to convert user commands in voice interaction into semantic representations that can be understood by computers, and to use feature extraction algorithms to convert the human multimodal data into structured data. The structured human multimodal data is then analyzed and simplified into model equations that are easy to process using the Lagrange multiplier method.

[0006] The model update and optimization module uses an iterative optimization algorithm to solve the model equations based on the real-time collected human multimodal data and user feedback information, and applies an adaptive adjustment strategy to update the comprehensive knowledge base and related algorithm models.

[0007] The position and status determination module determines the system's spatial position relative to the user and the user's current status by using a built-in positioning sensor and related algorithms, combined with user operating habits and environmental information.

[0008] The service path planning module generates personalized service paths based on the updated model, the user's spatial location, status, and information in the comprehensive knowledge base, using intelligent planning algorithms.

[0009] The multi-source data acquisition and integration module coordinates with the voice interaction and health detection function modules according to the generated service path to collect relevant user data again. The relevant user data collected again includes further voice commands and real-time health data changes. The newly collected multi-source data is initially integrated to make it consistent in format and semantics.

[0010] The data calibration and feature representation module performs mutual calibration on the integrated multi-source data to eliminate data bias and error. The calibrated data is then represented by feature vectors or feature matrices through feature extraction and transformation algorithms, and the confidence level of the feature representation is calculated to evaluate the reliability of the data.

[0011] The demand identification and service provision module uses classification and recognition algorithms to accurately identify users' health needs, lifestyle needs, and entertainment needs based on the confidence levels represented by user health data feature vectors and lifestyle habit data feature vectors. Based on the identified needs, it calls upon a comprehensive knowledge base to provide users with accurate health management advice, personalized entertainment content, and scientific dietary recommendations.

[0012] In some embodiments, the data acquisition and knowledge base construction module includes:

[0013] Microphones are configured on the intelligent integrated service system equipment for voice data collection, and posture sensors, snoring sensors, blood oxygen detection modules, and basic human function detection equipment are deployed.

[0014] The system uses a microphone to collect users’ voice commands in real time, a posture sensor to obtain users’ posture information, a snoring sensor to record snoring data, and a blood oxygen detection module and a basic human function detection device to collect blood oxygen and basic human function data respectively.

[0015] By using web crawler technology and data interfaces, relevant information is obtained from various song websites, health and medical databases, and nutrition and diet knowledge bases. After data cleaning, classification, and organization, a comprehensive knowledge base covering song resources, health knowledge, and dietary nutrition knowledge is constructed.

[0016] In some embodiments, the data structuring and model simplification module includes:

[0017] Structuring voice commands: This method utilizes lexical analysis, syntactic analysis, and semantic understanding algorithms from natural language processing to parse the user's voice commands. Let the voice command be... The word vector set is obtained through lexical analysis. Then, a syntax tree is constructed through syntactic analysis. Finally, it is transformed into a semantic representation by combining semantic understanding algorithms. ,Right now ,in This represents a semantic understanding function;

[0018] Other data structuring: For pose data A feature extraction algorithm is used to transform it into a pose feature vector. For snoring data Using signal processing algorithms, it is converted into a snoring feature vector. Regarding blood oxygen and basic human function data Through standardization and normalization, it is transformed into the corresponding feature vector. ;

[0019] Model simplification: Assume that the semantic representation of speech commands is taken into account. Pose feature vector snoring feature vector Blood oxygen and basic human functional characteristic vectors The function is Using the Lagrange multiplier method, for the function To simplify, based on the constraints:

[0020] Construct the Lagrange function ,right Taking the partial derivatives with respect to each variable and setting them to zero, we obtain the simplified model equations. ,Right now ,in It is a Lagrange multiplier.

[0021] In some embodiments, the model update and optimization module includes:

[0022] Real-time data processing: Let the new voice commands acquired in real time be... The new posture data is New snoring data is New blood oxygen and basic human function data are The data is then transformed into corresponding structured data according to the methods in the data structuring and model simplification module. , , , ;

[0023] Model update algorithm: Stochastic gradient descent algorithm is used to update the simplified model equations. To update, let the model parameters be... The loss function is This represents the difference between the model's predictions and the actual data. In each iteration, the loss function is calculated with respect to the parameters. gradient

[0024] According to the formula Update parameters ,in The learning rate;

[0025] Knowledge base updates: The comprehensive knowledge base is updated based on new knowledge, new needs, and model update results reflected in real-time data. This includes: if users show interest in a new song type, relevant song resources are obtained through online searches and added to the song knowledge base; if health data shows a new health trend or need, the relevant content in the health knowledge and dietary nutrition knowledge base is updated.

[0026] In some embodiments, the position and state determination module includes:

[0027] Location Determination: The intelligent integrated service system incorporates a GPS module and indoor positioning sensors, including Bluetooth positioning and Wi-Fi positioning. Let the geographical coordinates obtained by the GPS module be... Indoor positioning sensors obtain relative coordinates within the indoor environment by receiving ambient signal strength information and using trilateration or fingerprint positioning algorithms. The system automatically switches its positioning mode based on the environment. When outdoors, the system's location coordinates are... When indoors, the final location coordinates are obtained by combining GPS coordinates and indoor positioning coordinates. Through weighted average method , ,in and The weights are determined based on the actual environment and positioning accuracy.

[0028] State determination: combining user posture data Sentiment analysis results in voice commands and current time information Determine the user state, and let the state function be... If the posture data indicates that the user is in a sedentary state, the sentiment analysis of the voice command indicates fatigue, and the current time is evening, the state function... The output indicates a state of fatigue and rest at night.

[0029] In some embodiments, the service path planning module includes:

[0030] Requirements Analysis: Based on the updated model, user location, and status information, a deep analysis of user requirements is conducted. Let the user requirement function be... ,in For the updated model, For user location, Given the user's state, if the user is in a tired resting state at night and is in their bedroom at home, the demand function analysis indicates that the user needs services that help them sleep. These sleep-aiding services include playing soft music or providing relaxing health advice.

[0031] Resource matching: Based on the results of the requirements analysis The system searches for matching resources in a comprehensive knowledge base, and for song playback needs, it filters a list of songs that meet relaxation requirements from the song resource library. For health advice requests, relevant sleep health advice can be extracted from the health knowledge base. For dietary recommendations, based on the user's health data and current status, suitable dietary plans are searched from the dietary nutrition knowledge base. ;

[0032] Path planning algorithm: This algorithm uses A* and Dijkstra's algorithm to plan service paths, taking the efficiency and quality of resource acquisition and service provision as evaluation indicators. Let the service path be... The algorithm calculates the optimal service path from the current state to meeting the user's needs based on the relationships between different resources and their acquisition difficulty. ,in Indicates possible service paths, The evaluation function takes into account path length and resource matching.

[0033] In some embodiments, the multi-source data acquisition and integration includes:

[0034] Data Collection: Based on the planned service path, the relevant functional modules are restarted to collect data, including: if the service path involves further understanding the user's song preferences, the user's feedback on specific songs is solicited through the voice interaction module to obtain new voice data. If it is necessary to monitor changes in a user's physical condition in real time after receiving health advice, the health detection module can be used to collect new posture data. snoring data Blood oxygen and basic human function data ;

[0035] Data format unification: The newly collected multi-source data is processed to unify its format, including voice data. This converts the text into the same format as the previous structured processing of voice commands, i.e., a set of word vectors is obtained after preprocessing. syntax tree and semantic representation For pose data snoring data Blood oxygen and basic human function data Each feature vector is transformed into a feature vector with the same dimensions and format as the previous feature vectors, using the same feature extraction algorithm as before. , , ;

[0036] Data integration: Integrating multi-source data after standardizing the format, let the integrated dataset be... By semantic representation eigenvectors , , Combine them in order to obtain This provides a consistent data structure for subsequent data calibration and processing.

[0037] In some embodiments, the data calibration and feature representation module includes:

[0038] Data calibration: for integrated multi-source data A calibration method based on a statistical model is adopted, assuming the speech semantic representation Pose feature vector snoring feature vector Blood oxygen and basic human functional characteristic vectors These features originate from different sensors or processing modules, and therefore exhibit varying degrees of error and bias, affecting the pose feature vector. Assuming its error follows a normal distribution By statistically analyzing historical posture data, the mean was estimated. and variance Then use the formula The calibration is performed to obtain the calibrated pose feature vector. Similarly, other feature vectors are calibrated to obtain calibrated snoring feature vectors. Blood oxygen and basic human functional characteristic vectors and speech semantic representation .

[0039] Feature representation transformation: Principal component analysis (PCA) algorithm is used to transform the calibrated feature vectors. , , Dimensionality reduction is performed to obtain new feature vectors. , Simultaneously, the speech semantic representation These new feature vectors are then fused to form a unified feature matrix. ,Right now

[0040] ;

[0041] Confidence calculation: The feature matrix is ​​calculated using Bayesian inference methods. The confidence level, let the feature matrix be... Each feature vector and semantic representation in the data is a random variable. Based on historical data and prior knowledge, determine the prior probability of each random variable. Meanwhile, by analyzing actual data, the conditional probabilities among these random variables under different conditions are determined. Then, according to Bayes' theorem:

[0042] Calculate the characteristic matrix The joint probability, as a measure of confidence, indicates the reliability of the data.

[0043] In some embodiments, the demand identification and service provision module includes:

[0044] Demand identification algorithm: The support vector machine (SVM) algorithm is used to analyze the feature matrix. Classify users to identify different types of needs, and set up a feature matrix. For the input sample, the user requirement type is category label. Build an SVM model ,in For Lagrange multipliers, A kernel function is used to map input data to a high-dimensional space. As a bias term, the model parameters are adjusted by training on historical data with category labels. and ;

[0045] Service matching and provision: If the system identifies that the user has entertainment needs, specifically music appreciation needs, it selects songs with high confidence and that match the user's preferences from the song resource library and plays them for the user through the audio playback module of the intelligent integrated service system. If the system identifies that the user has health needs, such as the need to improve sleep quality, it extracts professional sleep health advice from the health knowledge base and provides it to the user in voice or text form. For dietary needs, based on the identification results and the user's health data, it generates a detailed dietary recommendation plan from the dietary nutrition knowledge base and displays it to the user.

[0046] Compared with the prior art, the beneficial effects of the present invention are as follows:

[0047] 1. This invention utilizes multimodal data fusion technology to integrate data from multiple sensors and data sources, enabling a comprehensive and in-depth understanding of user needs and status, thereby achieving precise and personalized service provision to meet users' complex needs in various aspects of life, health, and entertainment.

[0048] 2. This invention improves the system's accuracy in analyzing and predicting user needs by structuring and simplifying multimodal data and by using algorithms such as stochastic gradient descent for model updating and optimization. It also enhances the system's adaptability and learning ability, enabling it to continuously optimize services as user behavior and needs change.

[0049] 3. This invention collects and processes user data in real time, and combined with location and status determination functions, it can provide targeted services based on the user's real-time context. For example, based on the user's location and current status, it can recommend nearby service locations or provide health advice that matches the current status, improving the timeliness and practicality of the service.

[0050] 4. This invention employs multiple algorithms for demand identification and service path planning. By accurately matching resources in a comprehensive knowledge base, it can efficiently provide users with the services they need. Whether it's music playback, health advice, or dietary recommendations, it can quickly and accurately meet user needs, improving service efficiency and user satisfaction.

[0051] 5. This invention can achieve beneficial effects such as efficient processing of multimodal data, accurate demand identification, personalized service provision, and dynamic updating of the knowledge base.

[0052] The above general description and the following detailed description are exemplary and explanatory only, and are not intended to limit this disclosure.

[0053] Other features and aspects of this disclosure will become clearer from the following detailed description of exemplary embodiments with reference to the accompanying drawings. Attached Figure Description

[0054] To more clearly illustrate the specific embodiments of the present invention or the technical solutions in the prior art, the drawings used in the description of the specific embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0055] Figure 1 This is a block diagram of an intelligent integrated service system based on multimodal data fusion, provided for an embodiment of the present invention. Detailed Implementation

[0056] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of this application.

[0057] In the description of this application, it should be understood that the terms "center", "upper", "lower", "front", "rear", "left", "right", "vertical", "horizontal", "top", "bottom", "inner", "outer", etc., indicate the orientation or positional relationship based on the orientation or positional relationship shown in the accompanying drawings. They are only for the convenience of describing this application and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation. Therefore, they should not be construed as limitations on this application.

[0058] The terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features indicated. Therefore, a feature defined as "first" or "second" may explicitly or implicitly include one or more of that feature. In the description of this application, unless otherwise stated, "a plurality of" means two or more.

[0059] In the description of this application, it should be noted that, unless otherwise expressly specified and limited, the terms "installation," "connection," and "linking" should be interpreted broadly. For example, they can refer to a fixed connection, a detachable connection, or an integral connection; they can refer to a mechanical connection or an electrical connection; they can refer to a direct connection or an indirect connection through an intermediate medium; and they can refer to the internal connection between two components. Those skilled in the art can understand the specific meaning of the above terms in this application based on the specific circumstances.

[0060] See Figure 1 As shown, an intelligent integrated service system based on multimodal data fusion according to an embodiment of this application includes:

[0061] The data acquisition and knowledge base construction module is used to collect human multimodal data and construct a comprehensive knowledge base covering song resources, health knowledge, and dietary nutrition.

[0062] The data structuring and model simplification module is used to apply natural language processing algorithms to convert user commands in voice interaction into semantic representations that can be understood by computers, and to use feature extraction algorithms to convert the human multimodal data into structured data. The structured human multimodal data is then analyzed and simplified into model equations that are easy to process using the Lagrange multiplier method.

[0063] The model update and optimization module uses an iterative optimization algorithm to solve the model equations based on the real-time collected human multimodal data and user feedback information, and applies an adaptive adjustment strategy to update the comprehensive knowledge base and related algorithm models.

[0064] The position and status determination module determines the system's spatial position relative to the user and the user's current status by using a built-in positioning sensor and related algorithms, combined with user operating habits and environmental information.

[0065] The service path planning module generates personalized service paths based on the updated model, the user's spatial location, status, and information in the comprehensive knowledge base, using intelligent planning algorithms.

[0066] The multi-source data acquisition and integration module coordinates with the voice interaction and health detection function modules according to the generated service path to collect relevant user data again. The relevant user data collected again includes further voice commands and real-time health data changes. The newly collected multi-source data is initially integrated to make it consistent in format and semantics.

[0067] The data calibration and feature representation module performs mutual calibration on the integrated multi-source data to eliminate data bias and error. The calibrated data is then represented by feature vectors or feature matrices through feature extraction and transformation algorithms, and the confidence level of the feature representation is calculated to evaluate the reliability of the data.

[0068] The demand identification and service provision module uses classification and recognition algorithms to accurately identify users' health needs, lifestyle needs, and entertainment needs based on the confidence levels represented by user health data feature vectors and lifestyle habit data feature vectors. Based on the identified needs, it calls upon a comprehensive knowledge base to provide users with accurate health management advice, personalized entertainment content, and scientific dietary recommendations.

[0069] In some specific embodiments, the data acquisition and knowledge base construction module includes:

[0070] Microphones are configured on the intelligent integrated service system equipment for voice data collection, and posture sensors, snoring sensors, blood oxygen detection modules, and basic human function detection equipment are deployed.

[0071] The system uses a microphone to collect users’ voice commands in real time, a posture sensor to obtain users’ posture information, a snoring sensor to record snoring data, and a blood oxygen detection module and a basic human function detection device to collect blood oxygen and basic human function data respectively.

[0072] By using web crawler technology and data interfaces, relevant information is obtained from various song websites, health and medical databases, and nutrition and diet knowledge bases. After data cleaning, classification, and organization, a comprehensive knowledge base covering song resources, health knowledge, and dietary nutrition knowledge is constructed.

[0073] In some specific embodiments, the data structuring and model simplification module includes:

[0074] Structuring voice commands: This method utilizes lexical analysis, syntactic analysis, and semantic understanding algorithms from natural language processing to parse the user's voice commands. Let the voice command be... The word vector set is obtained through lexical analysis. Then, a syntax tree is constructed through syntactic analysis. Finally, it is transformed into a semantic representation by combining semantic understanding algorithms. ,Right now ,in This represents a semantic understanding function;

[0075] Other data structuring: For pose data A feature extraction algorithm is used to transform it into a pose feature vector. For snoring data Using signal processing algorithms, it is converted into a snoring feature vector. Regarding blood oxygen and basic human function data Through standardization and normalization, it is transformed into the corresponding feature vector. ;

[0076] Model simplification: Assume that the semantic representation of speech commands is taken into account. Pose feature vector snoring feature vector Blood oxygen and basic human functional characteristic vectors The function is Using the Lagrange multiplier method, for the function To simplify, based on the constraints:

[0077] Construct the Lagrange function:

[0078] ;

[0079] right Taking the partial derivatives with respect to each variable and setting them to zero, we obtain the simplified model equations. ,Right now ,in It is a Lagrange multiplier.

[0080] A smart integrated service system is deployed in the user's home. A high-sensitivity microphone captures the user saying in real time, "I haven't been sleeping well lately, I feel very tired, do you have any good suggestions, and could you recommend some relaxing songs?" Simultaneously, a posture sensor detects the user lying in bed and processes data from accelerometers and gyroscopes to obtain posture feature vectors. A snoring sensor records the user's recent snoring data, which is then processed to generate snoring feature vectors. A blood oxygen detection module and basic human function monitoring equipment measure the user's blood oxygen content, heart rate, blood pressure, and other data, which are standardized and normalized to obtain corresponding feature vectors. The system uses web crawlers to retrieve relaxing songs from websites. It collects knowledge about sleep problems and fatigue relief from professional health and medical databases, such as the dangers of sleep deprivation and methods to relieve fatigue. It also obtains information on foods that can improve sleep and relieve fatigue from a nutritional knowledge base, such as the nutritional components and consumption suggestions for oats and walnuts. After data cleaning, classification, and organization, a comprehensive knowledge base is constructed. Lexical analysis is performed on the collected voice commands to obtain a set of word vectors, such as {"recently", "sleep", "not good", "feeling", "very tired", "suggestion", "recommendation", "relaxing", "song"}. A syntax tree is constructed through syntactic analysis to clarify the grammatical relationships between words. Semantic understanding algorithms are then used to transform these into semantic representations, indicating that the user expressed poor sleep, fatigue, and a desire for improvement suggestions and relaxing song recommendations.

[0081] Afterwards, the microphone picked up a new voice command: "I'm not really interested in the song you just recommended. Do you have one with a more relaxed tempo?" Simultaneously, new posture data indicated that the user was turning over in bed, and new snoring, blood oxygen levels, and basic bodily function data also showed corresponding changes. This data was then converted into structured data using the previous method.

[0082] In some specific embodiments, the model update and optimization module includes:

[0083] Real-time data processing: Let the new voice commands acquired in real time be... The new posture data is New snoring data is New blood oxygen and basic human function data are The data is then transformed into corresponding structured data according to the methods in the data structuring and model simplification module. , , , ;

[0084] Model update algorithm: Stochastic gradient descent algorithm is used to update the simplified model equations. To update, let the model parameters be... The loss function is This represents the difference between the model's predictions and the actual data. In each iteration, the loss function is calculated with respect to the parameters. gradient

[0085] According to the formula Update parameters ,in The learning rate;

[0086] Knowledge base updates: The comprehensive knowledge base is updated based on new knowledge, new needs, and model update results reflected in real-time data. This includes: if users show interest in a new song type, relevant song resources are obtained through online searches and added to the song knowledge base; if health data shows a new health trend or need, the relevant content in the health knowledge and dietary nutrition knowledge base is updated.

[0087] The simplified model equations are processed using the stochastic gradient descent algorithm. Update the model. Let the model parameters be... The loss function is The model's predictions are measured against the actual data. In each iteration, the loss function is calculated with respect to the parameters. gradient According to the formula Update parameters In this embodiment, the learning rate Set it to 0.001.

[0088] In some specific embodiments, the position and state determination module includes:

[0089] Location Determination: The intelligent integrated service system incorporates a GPS module and indoor positioning sensors, including Bluetooth positioning and Wi-Fi positioning. Let the geographical coordinates obtained by the GPS module be... Indoor positioning sensors obtain relative coordinates within the indoor environment by receiving ambient signal strength information and using trilateration or fingerprint positioning algorithms. The system automatically switches its positioning mode based on the environment. When outdoors, the system's location coordinates are... When indoors, the final location coordinates are obtained by combining GPS coordinates and indoor positioning coordinates. Through weighted average method , ,in and The weights are determined based on the actual environment and positioning accuracy.

[0090] State determination: combining user posture data Sentiment analysis results in voice commands and current time information Determine the user state, and let the state function be... If the posture data indicates that the user is in a sedentary state, the sentiment analysis of the voice command indicates fatigue, and the current time is evening, the state function... The output indicates a state of fatigue and rest at night.

[0091] The system's built-in GPS module failed to acquire valid outdoor positioning information. The indoor positioning sensor determined the user's indoor coordinates using trilateration by receiving ambient signal strength and combining this with system settings to determine the user's location in the bedroom. This data was then used in conjunction with user posture data (turning over in bed), voice command emotion analysis results (fatigue, dissatisfaction), and current time information, through a state function... The user's condition is determined to be "poor sleep at night and fatigued and irritable".

[0092] In some specific embodiments, the service path planning module includes:

[0093] Requirements Analysis: Based on the updated model, user location, and status information, a deep analysis of user requirements is conducted. Let the user requirement function be... ,in For the updated model, For user location, Given the user's state, if the user is in a tired resting state at night and is in their bedroom at home, the demand function analysis indicates that the user needs services that help them sleep. These sleep-aiding services include playing soft music or providing relaxing health advice.

[0094] Resource matching: Based on the results of the requirements analysis The system searches for matching resources in a comprehensive knowledge base, and for song playback needs, it filters a list of songs that meet relaxation requirements from the song resource library. For health advice requests, relevant sleep health advice can be extracted from the health knowledge base. For dietary recommendations, based on the user's health data and current status, suitable dietary plans are searched from the dietary nutrition knowledge base. ;

[0095] Path planning algorithm: This algorithm uses A* and Dijkstra's algorithm to plan service paths, taking the efficiency and quality of resource acquisition and service provision as evaluation indicators. Let the service path be... The algorithm calculates the optimal service path from the current state to meeting the user's needs based on the relationships between different resources and their acquisition difficulty. ,in Indicates possible service paths, The evaluation function takes into account path length and resource matching.

[0096] Based on the updated model, user location (bedroom), and state (poor sleep and fatigued / irritable state at night), the user demand function is used. The analysis revealed user needs for services that help relax the mind and body and improve sleep, such as playing soothing music, providing sleep-aiding health advice, and recommending sleep-inducing foods. For music playback requests, a list of more soothing songs was selected from the music library. For health advice requests, relevant content such as "taking a hot bath before bed" and "practicing simple deep breathing relaxation exercises" was extracted from the health knowledge base. For dietary recommendations, information on sleep-aiding foods such as milk and bananas was found from the dietary nutrition knowledge base, and simple consumption suggestions were generated.

[0097] In some specific embodiments, the multi-source data acquisition and integration includes:

[0098] Data Collection: Based on the planned service path, the relevant functional modules are restarted to collect data, including: if the service path involves further understanding the user's song preferences, the user's feedback on specific songs is solicited through the voice interaction module to obtain new voice data. If it is necessary to monitor changes in a user's physical condition in real time after receiving health advice, the health detection module can be used to collect new posture data. snoring data Blood oxygen and basic human function data ;

[0099] Data format unification: The newly collected multi-source data is processed to unify its format, including voice data. This converts the text into the same format as the previous structured processing of voice commands, i.e., a set of word vectors is obtained after preprocessing. syntax tree and semantic representation For pose data snoring data Blood oxygen and basic human function data Each feature vector is transformed into a feature vector with the same dimensions and format as the previous feature vectors, using the same feature extraction algorithm as before. , , ;

[0100] Data integration: Integrating multi-source data after standardizing the format, let the integrated dataset be... By semantic representation eigenvectors , , Combine them in order to obtain This provides a consistent data structure for subsequent data calibration and processing.

[0101] In some specific embodiments, the data calibration and feature representation module includes:

[0102] Data calibration: for integrated multi-source data A calibration method based on a statistical model is adopted, assuming the speech semantic representation Pose feature vector snoring feature vector Blood oxygen and basic human functional characteristic vectors These features originate from different sensors or processing modules, and therefore exhibit varying degrees of error and bias, affecting the pose feature vector. Assuming its error follows a normal distribution By statistically analyzing historical posture data, the mean was estimated. and variance Then use the formula The calibration is performed to obtain the calibrated pose feature vector. Similarly, other feature vectors are calibrated to obtain calibrated snoring feature vectors. Blood oxygen and basic human functional characteristic vectors and speech semantic representation .

[0103] Feature representation transformation: Principal component analysis (PCA) algorithm is used to transform the calibrated feature vectors. , , Dimensionality reduction is performed to obtain new feature vectors. , Simultaneously, the speech semantic representation These new feature vectors are then fused to form a unified feature matrix. ,Right now

[0104] ;

[0105] Confidence calculation: The feature matrix is ​​calculated using Bayesian inference methods. The confidence level, let the feature matrix be... Each feature vector and semantic representation in the data is a random variable. Based on historical data and prior knowledge, determine the prior probability of each random variable. Meanwhile, by analyzing actual data, the conditional probabilities among these random variables under different conditions are determined. Then, according to Bayes' theorem:

[0106] Calculate the characteristic matrix The joint probability, as a measure of confidence, indicates the reliability of the data.

[0107] In some specific embodiments, the demand identification and service provision module includes:

[0108] Demand identification algorithm: The support vector machine (SVM) algorithm is used to analyze the feature matrix. Classify users to identify different types of needs, and set up a feature matrix. For the input sample, the user requirement type is category label. Build an SVM model ,in For Lagrange multipliers, A kernel function is used to map input data to a high-dimensional space. As a bias term, the model parameters are adjusted by training on historical data with category labels. and ;

[0109] Service matching and provision: If the system identifies that the user has entertainment needs, specifically music appreciation needs, it selects songs with high confidence and that match the user's preferences from the song resource library and plays them for the user through the audio playback module of the intelligent integrated service system. If the system identifies that the user has health needs, such as the need to improve sleep quality, it extracts professional sleep health advice from the health knowledge base and provides it to the user in voice or text form. For dietary needs, based on the identification results and the user's health data, it generates a detailed dietary recommendation plan from the dietary nutrition knowledge base and displays it to the user.

[0110] The system uses a voice interaction module to ask the user, "Does this recommended song's tempo suit your needs?" to acquire new voice data. Simultaneously, a health monitoring module collects user posture data, snoring data, blood oxygen saturation, and basic bodily function data after listening to the song. The new voice data is preprocessed to obtain a word vector set, a syntax tree, and a semantic representation. The new posture, snoring, blood oxygen, and basic bodily function data are then transformed into data with the same dimensions and format as the previous feature vectors using the same feature extraction algorithm. For posture feature vectors... Assuming its error follows a normal distribution Through statistical analysis of a large amount of historical posture data, the mean was estimated. and variance Using formula Calibration is then performed. Similarly, other feature vectors are calibrated. Principal Component Analysis (PCA) is used to reduce the dimensionality of the calibrated feature vectors, resulting in new feature vectors. The speech semantic representation is then fused with these new feature vectors to form a unified feature matrix. The feature matrix is ​​calculated using Bayesian inference methods. The confidence level. Let the characteristic matrix be... Each element in the array is a random variable, and the prior probability of each random variable is determined based on historical data and prior knowledge. By analyzing a large amount of real-world data, the conditional probabilities between random variables under different conditions are determined. Calculate the characteristic matrix according to Bayes' theorem. The joint probability of the feature matrix is ​​used as the confidence level. A Support Vector Machine (SVM) algorithm is employed to analyze the feature matrix. The system categorizes users and trains them on a large amount of historical data with category labels, adjusting model parameters to enable the model to accurately identify user needs based on the feature matrix. It identifies entertainment, health, and dietary needs. Songs with high confidence and matching user preferences are selected from the song library and played to the user via the audio playback module. Sleep-aiding health suggestions extracted from the health knowledge base are provided to the user in voice format. Sleep-aiding food recommendations generated from the dietary nutrition knowledge base are displayed to the user on a screen, completing the personalized service delivery.

[0111] Obviously, those skilled in the art can make various modifications and variations to this invention without departing from its spirit and scope. Therefore, if these modifications and variations fall within the scope of the claims of this invention and their equivalents, this invention also intends to include these modifications and variations.

Claims

1. An intelligent integrated service system based on multimodal data fusion, characterized in that, include: The data acquisition and knowledge base construction module is used to collect human multimodal data and construct a comprehensive knowledge base covering song resources, health knowledge, and dietary nutrition. The data structuring and model simplification module is used to apply natural language processing algorithms to convert user commands in voice interaction into semantic representations that can be understood by computers, and to use feature extraction algorithms to convert the human multimodal data into structured data. The structured human multimodal data is then analyzed and simplified into model equations that are easy to process using the Lagrange multiplier method. The model update and optimization module uses an iterative optimization algorithm to solve the model equations based on the real-time collected human multimodal data and user feedback information, and applies an adaptive adjustment strategy to update the comprehensive knowledge base and related algorithm models. The position and status determination module determines the system's spatial position relative to the user and the user's current status by using a built-in positioning sensor and related algorithms, combined with user operating habits and environmental information. The service path planning module generates personalized service paths based on the updated model, the user's spatial location, status, and information in the comprehensive knowledge base, using intelligent planning algorithms. The multi-source data acquisition and integration module coordinates with the voice interaction and health detection function modules according to the generated service path to collect relevant user data again. The relevant user data collected again includes further voice commands and real-time health data changes. The newly collected multi-source data is initially integrated to make it consistent in format and semantics. The data calibration and feature representation module performs mutual calibration on the integrated multi-source data to eliminate data bias and error. The calibrated data is then represented by feature vectors or feature matrices through feature extraction and transformation algorithms, and the confidence level of the feature representation is calculated to evaluate the reliability of the data. The demand identification and service provision module uses classification and recognition algorithms to accurately identify users' health needs, lifestyle needs, and entertainment needs based on the confidence levels represented by user health data feature vectors and lifestyle habit data feature vectors. The service path planning module includes: Requirements Analysis: Based on the updated model, user location, and status information, a deep analysis of user requirements is conducted. Let the user requirement function be... ,in For the updated model, For user location, Given the user's state, if the user is in a tired resting state at night and is in their bedroom at home, the demand function analysis indicates that the user needs services that help them sleep. These sleep-aiding services include playing soft music or providing relaxing health advice. Resource matching: Based on the results of the requirements analysis The system searches for matching resources in a comprehensive knowledge base, and for song playback needs, it filters a list of songs that meet relaxation requirements from the song resource library. For health advice requests, relevant sleep health advice can be extracted from the health knowledge base. For dietary recommendations, based on the user's health data and current status, suitable dietary plans are searched from the dietary nutrition knowledge base. ; Path planning algorithm: This algorithm uses A* and Dijkstra's algorithm to plan service paths, taking the efficiency and quality of resource acquisition and service provision as evaluation indicators. Let the service path be... The algorithm calculates the optimal service path from the current state to meeting the user's needs based on the relationships between different resources and their acquisition difficulty. ,in Indicates possible service paths, The evaluation function takes into account path length and resource matching.

2. The intelligent integrated service system based on multimodal data fusion according to claim 1, characterized in that, The data acquisition and knowledge base construction module includes: Microphones are configured on the intelligent integrated service system equipment for voice data collection, and posture sensors, snoring sensors, blood oxygen detection modules, and basic human function detection equipment are deployed. The system uses a microphone to collect users’ voice commands in real time, a posture sensor to obtain users’ posture information, a snoring sensor to record snoring data, and a blood oxygen detection module and a basic human function detection device to collect blood oxygen and basic human function data respectively. By using web crawler technology and data interfaces, relevant information is obtained from various song websites, health and medical databases, and nutrition and diet knowledge bases. After data cleaning, classification, and organization, a comprehensive knowledge base covering song resources, health knowledge, and dietary nutrition knowledge is constructed.

3. The intelligent integrated service system based on multimodal data fusion according to claim 2, characterized in that, The data structuring and model simplification module includes: Structuring voice commands: This method utilizes lexical analysis, syntactic analysis, and semantic understanding algorithms from natural language processing to parse the user's voice commands. Let the voice command be... The word vector set is obtained through lexical analysis. Then, a syntax tree is constructed through syntactic analysis. Finally, it is transformed into a semantic representation by combining semantic understanding algorithms. ,Right now ,in This represents a semantic understanding function; Other data structuring: For pose data A feature extraction algorithm is used to transform it into a pose feature vector. For snoring data Using signal processing algorithms, it is converted into a snoring feature vector. Regarding blood oxygen and basic human function data Through standardization and normalization, it is transformed into the corresponding feature vector. ; Model simplification: Assume that the semantic representation of speech commands is taken into account. Pose feature vector snoring feature vector Blood oxygen and basic human functional characteristic vectors The function is Using the Lagrange multiplier method, for the function To simplify, based on the constraints: Construct the Lagrange function: ;right Taking the partial derivatives with respect to each variable and setting them to zero, we obtain the simplified model equations. ,Right now ,in It is a Lagrange multiplier.

4. The intelligent integrated service system based on multimodal data fusion according to claim 3, characterized in that, The model update and optimization module includes: Real-time data processing: Let the new voice commands acquired in real time be... The new posture data is New snoring data is New blood oxygen and basic human function data are The data is then transformed into corresponding structured data according to the methods in the data structuring and model simplification module. , , , ; Model update algorithm: Stochastic gradient descent algorithm is used to update the simplified model equations. To update, let the model parameters be... The loss function is This represents the difference between the model's predictions and the actual data. In each iteration, the loss function is calculated with respect to the parameters. gradient According to the formula Update parameters ,in The learning rate; Knowledge base updates: The comprehensive knowledge base is updated based on new knowledge, new needs, and model update results reflected in real-time data. This includes: if users show interest in a new song type, relevant song resources are obtained through online searches and added to the song knowledge base; if health data shows a new health trend or need, the relevant content in the health knowledge and dietary nutrition knowledge base is updated.

5. The intelligent integrated service system based on multimodal data fusion according to claim 4, characterized in that, The position and status determination module includes: Location Determination: The intelligent integrated service system incorporates a GPS module and indoor positioning sensors, including Bluetooth positioning and Wi-Fi positioning. Let the geographical coordinates obtained by the GPS module be... Indoor positioning sensors obtain relative coordinates within the indoor environment by receiving ambient signal strength information and using trilateration or fingerprint positioning algorithms. The system automatically switches its positioning mode based on the environment. When outdoors, the system's location coordinates are... When indoors, the final location coordinates are obtained by combining GPS coordinates and indoor positioning coordinates. Through weighted average method , ,in and The weights are determined based on the actual environment and positioning accuracy. State determination: combining user posture data Sentiment analysis results in voice commands and current time information Determine the user state, and let the state function be... If the posture data indicates that the user is in a sedentary state, the sentiment analysis of the voice command indicates fatigue, and the current time is evening, the state function... The output indicates a state of fatigue and rest at night.

6. The intelligent integrated service system based on multimodal data fusion according to claim 5, characterized in that, The multi-source data acquisition and integration includes: Data Collection: Based on the planned service path, the relevant functional modules are restarted to collect data, including: if the service path involves further understanding the user's song preferences, the user's feedback on specific songs is solicited through the voice interaction module to obtain new voice data. If it is necessary to monitor changes in a user's physical condition in real time after receiving health advice, the health detection module can be used to collect new posture data. snoring data Blood oxygen and basic human function data ; Data format unification: The newly collected multi-source data is processed to unify its format, including voice data. This converts the text into the same format as the previous structured processing of voice commands, i.e., a set of word vectors is obtained after preprocessing. syntax tree and semantic representation For pose data snoring data Blood oxygen and basic human function data Each feature vector is transformed into a feature vector with the same dimensions and format as the previous feature vectors, using the same feature extraction algorithm as before. , , ; Data integration: Integrating multi-source data after standardizing the format, let the integrated dataset be... By semantic representation eigenvectors , , Combine them in order to obtain This provides a consistent data structure for subsequent data calibration and processing.

7. The intelligent integrated service system based on multimodal data fusion according to claim 1, characterized in that, The data calibration and feature representation module includes: Data calibration: for integrated multi-source data A calibration method based on a statistical model is adopted, assuming the speech semantic representation Pose feature vector snoring feature vector Blood oxygen and basic human functional characteristic vectors These features originate from different sensors or processing modules, and therefore exhibit varying degrees of error and bias, affecting the pose feature vector. Assuming its error follows a normal distribution By statistically analyzing historical posture data, the mean was estimated. and variance Then use the formula The calibration is performed to obtain the calibrated pose feature vector. Similarly, other feature vectors are calibrated to obtain calibrated snoring feature vectors. Blood oxygen and basic human functional characteristic vectors and speech semantic representation ; Feature representation transformation: Principal component analysis (PCA) algorithm is used to transform the calibrated feature vectors. , , Dimensionality reduction is performed to obtain new feature vectors. , Simultaneously, the speech semantic representation These new feature vectors are then fused to form a unified feature matrix. ,Right now ； Confidence calculation: The feature matrix is ​​calculated using Bayesian inference methods. The confidence level, let the feature matrix be... Each feature vector and semantic representation in the data is a random variable. Based on historical data and prior knowledge, determine the prior probability of each random variable. Meanwhile, by analyzing actual data, the conditional probabilities among these random variables under different conditions are determined. Then, according to Bayes' theorem: Calculate the characteristic matrix The joint probability, as a measure of confidence, indicates the reliability of the data.

8. The intelligent integrated service system based on multimodal data fusion according to claim 1, characterized in that, The demand identification and service provision module includes: Demand identification algorithm: The support vector machine (SVM) algorithm is used to analyze the feature matrix. Classify users to identify different types of needs, and set up a feature matrix. For the input sample, the user requirement type is category label. Build an SVM model ,in For Lagrange multipliers, A kernel function is used to map input data to a high-dimensional space. As a bias term, the model parameters are adjusted by training on historical data with category labels. and ; Service matching and provision: If the system identifies that the user has entertainment needs, specifically music appreciation needs, it selects songs with high confidence and that match the user's preferences from the song resource library and plays them for the user through the audio playback module of the intelligent integrated service system. If the system identifies that the user has health needs, such as the need to improve sleep quality, it extracts professional sleep health advice from the health knowledge base and provides it to the user in voice or text form. For dietary needs, based on the identification results and the user's health data, it generates a detailed dietary recommendation plan from the dietary nutrition knowledge base and displays it to the user.

Images