A robot operating system and operating method supporting multimodal interaction

Abstract

The present invention discloses a robot operating system and operating method supporting multimodal interaction, relating to the field of intelligent robotics. The system comprises: a data receiving module, a target object recognition module, an analysis and decision module, and a response module. The data receiving module is configured to preprocess acquired multimodal heterogeneous data to obtain sensory data, wherein the multimodal heterogeneous data includes at least one of visual data, auditory data, tactile data, and semantic data. The target object recognition module is configured to create a corresponding user profile for the target object based on the sensory data. The analysis and decision module is configured to perform preference analysis based on the user profile and derive a response strategy based on a deep learning model. The response module is configured to issue an operation instruction based on the response strategy. The present invention achieves the ability to better understand and respond to user needs based on individual user characteristics.

Description

Technical Field

[0001] The present invention relates to the technical field of intelligent robots, and in particular to a robot operating system and an operating method supporting multimodal interaction. Background Art

[0002] Currently, robots have limited capabilities in understanding user behaviors, intentions, and emotions, and are unable to accurately capture subtle clues in user communications. In long-term interactions with users, they lack growth potential, find it difficult to truly understand user needs, and are unable to provide intelligent and customized services. Summary of the Invention

[0003] Based on this, the purpose of the present invention is to provide a robot operating system and operating method that supports multimodal interaction, aiming to better understand the user's needs and respond based on the user's individual characteristics.

[0004] In a first aspect, the present application provides a robot operating system supporting multimodal interaction, comprising:

[0005] Data receiving module, target object recognition module, analysis and decision module and response module;

[0006] The data receiving module is used to pre-process the acquired multimodal heterogeneous data to obtain sensory data; wherein the multimodal heterogeneous data includes at least one of visual data, auditory data, tactile data and semantic data;

[0007] The target object recognition module is used to create a corresponding user profile for the target object based on sensory data;

[0008] The analysis and decision-making module is used to analyze preferences based on user portraits and derive response strategies based on deep learning models;

[0009] The response module is used to issue operation instructions based on the response strategy.

[0010] In one embodiment, the target object recognition module includes a multimodal data fusion unit, a user profiling unit, and a user classification unit;

[0011] The multimodal data fusion unit is used to extract features of the sensory data based on the multimodal data fusion model, where the features include at least one of age, gender, language habits, knowledge level, and disability;

[0012] The user portrait unit is used to create a user portrait based on sensory data from the same user, and store and manage the user portrait based on voiceprint features as an index;

[0013] The user classification unit is used to classify group characteristics based on user portraits for training deep learning models.

[0014] In one embodiment, the analysis and decision module includes an input submodule and an output submodule:

[0015] The input submodule is used to obtain the user's historical behavior sequence data, assign weights to each data point through the attention mechanism and input it into the long short-term memory network, where the attention energy The calculation formula is as follows:

[0016]

[0017] in, is the attention energy, is the number of time steps of the historical behavior sequence data, is the attention vector, is the weight matrix, is the bias vector, is the i-th input data, is the hyperbolic tangent activation function;

[0018] Use the attention conversion formula to convert attention energy Convert to attention weight , where the attention conversion formula is as follows:

[0019]

[0020] in, is the attention weight, exp is the exponential function, is the sum of the exponential operations on the attention energy at all time steps;

[0021] Multiply the input sequence by the attention weight to obtain the weighted input sequence, where the calculation formula of the weighted input sequence is as follows:

[0022]

[0023] in, is the t-th input after attention weighting, is the attention weight of the ith input at the tth time step, is the i-th input data;

[0024] The output submodule is used to:

[0025] Computing preference representations based on weighted input sequences , where preference represents The calculation formula is as follows:

[0026]

[0027] in, is the weight matrix of the fully connected layer, is the sum of the weighted input sequence, is the rectified linear unit activation function, is the bias vector of the fully connected layer;

[0028] Generating response strategies based on preference representation , where the response strategy The calculation formula is as follows:

[0029]

[0030] in, is the preference representation, is the weight matrix of the fully connected layer, is the bias vector, is the activation function.

[0031] Furthermore, the response module includes:

[0032] A real-time contextual awareness unit, which identifies the current interaction context based on sensory data, user profiles, and operational behaviors, and formulates corresponding output methods. User profiles include preference information and emotional states.

[0033] The real-time situational awareness unit also includes:

[0034] A preference matching subunit, which is used to establish a mapping relationship between preference information and output mode based on a machine learning algorithm;

[0035] The dynamic decision-making subunit is used to continuously monitor the interaction scenario and emotional state and extract feature vectors during the interaction process. When the Euclidean distance between the feature vector of the interaction scenario and / or emotional state within a preset time period and the feature vector at the previous moment exceeds a preset threshold, the output method is re-formulated.

[0036] Preferably, the system also includes a group user portrait module and an interactive resonance module:

[0037] The group user portrait module is used to extract representative features from multiple user portraits and perform cluster analysis based on the representative features to obtain multiple groups and corresponding group user portraits; the representative features include at least one of self-identity, geographical distribution, age and education level;

[0038] The interactive resonance module is used to conduct data mining on group user portraits to obtain personalized interaction strategies for each group. Personalized interaction strategies are used to indicate operational instructions.

[0039] Preferably, the system further includes an index module, a cache module and an interest update module:

[0040] The index module is used to perform hash operations on the voiceprint features to obtain a hash value used to determine the storage location;

[0041] The cache module is used to store the hash value of the voiceprint feature as a key and the user's interest information as a value in the cache, where the interest information is information whose access times exceed a preset threshold within a preset period of time;

[0042] The interest update module is used to perform cluster analysis on interest information and tag it with corresponding interest tags. Based on the interest tags, it uses web crawler technology to regularly crawl interest-related information from selected information sources. The interest-related information is used to update the interest information.

[0043] Preferably, the response module further comprises: an active interaction unit and a feedback receiving unit;

[0044] The active interaction unit is used to start the active interaction program;

[0045] The feedback receiving unit is used to obtain multimodal heterogeneous data again after issuing an operation instruction and calculate the satisfaction of the target object based on artificial intelligence. If the satisfaction is lower than the threshold, the active interaction unit is called to start the active interaction program.

[0046] In a second aspect, the present application further provides a robot operation method supporting multimodal interaction, comprising:

[0047] Acquiring multimodal heterogeneous data and preprocessing the multimodal heterogeneous data to obtain sensory data; wherein the multimodal heterogeneous data includes at least one of visual data, auditory data, tactile data, and semantic data;

[0048] Create a corresponding user profile for the target object based on sensory data;

[0049] Conduct preference analysis based on user profiles and derive response strategies based on deep learning models;

[0050] Issue action instructions based on the response strategy.

[0051] In a third aspect, the present application further provides a computer device comprising a memory and a processor, wherein the memory stores a computer program, and the processor implements the aforementioned method when executing the computer program.

[0052] In a fourth aspect, the present application further provides a computer-readable storage medium on which a computer program is stored, and when the computer program is executed by a processor, it is used to implement the method as described above.

[0053] The present invention integrates a data receiving module, a target object recognition module, an analysis and decision module, and a response module. The data receiving module converts acquired multimodal heterogeneous data into sensory data through preprocessing. The multimodal heterogeneous data includes at least one of visual data, auditory data, tactile data, and semantic data. The target object recognition module creates a corresponding user profile for the target object based on this sensory data. The analysis and decision module uses a deep learning model to perform preference analysis on the user profile and derives a response strategy based on this. The response module issues operation instructions based on the response strategy determined by the analysis and decision module, enabling the system to respond quickly and effectively to user needs, improving the system's response speed and efficiency and meeting users' complex and diverse personalized needs. This solves the problem in existing technologies where robots have difficulty truly understanding user needs and are unable to provide intelligent and customized services. BRIEF DESCRIPTION OF THE DRAWINGS

[0054] In order to more clearly illustrate the technical solutions in the embodiments of the present application or related technologies, the following briefly introduces the drawings required for use in the embodiments or related technical descriptions. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0055] Figure 1 A schematic diagram of a robot operating system supporting multimodal interaction provided by an embodiment of the present invention;

[0056] Figure 2 A schematic flow chart of a robot operation method supporting multimodal interaction provided by an embodiment of the present invention.

[0057] The following specific embodiments will further illustrate the present invention in conjunction with the above-mentioned drawings. DETAILED DESCRIPTION

[0058] To facilitate understanding of the present invention, the present invention will be described more fully below with reference to the accompanying drawings. The drawings illustrate several embodiments of the present invention. However, the present invention may be implemented in many different forms and is not limited to the embodiments described herein. Rather, these embodiments are provided to provide a more thorough and comprehensive understanding of the present invention.

[0059] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one skilled in the art to which this invention pertains. The terms used in this specification of the present invention are for the purpose of describing specific embodiments only and are not intended to limit the present invention. The term "and / or" as used herein includes any and all combinations of one or more of the associated listed items.

[0060] First, a brief introduction is given to the terms involved in the embodiments of this application.

[0061] Multimodal data refers to data that contains multiple different modes or types, and each mode can be a different type of data, such as text, images, audio, video, etc.; heterogeneous data refers to a collection of data with different types or structures; multimodal heterogeneous data is diverse, correlated and complex. In the intelligent robot interactive operating system, processing multimodal heterogeneous data can enhance the robot's perception and understanding capabilities, enabling it to better adapt to complex and changing environments.

[0062] User portrait is a model built based on the multi-dimensional characteristics of users, used to describe and predict user behavior, preferences and needs. It collects and analyzes multi-source data such as user personal information, behavioral data, interaction records, etc., and uses machine learning and data analysis techniques to extract and model user characteristics.

[0063] The Long Short-Term Memory (LSTM) network is a special type of recurrent neural network (RNN) that is primarily used to process and predict time series data. When processing historical user behavior sequence data, it effectively captures long-range dependencies by introducing a special gating mechanism, including a forget gate, an input gate, and an output gate. The forget gate determines which past information to retain or discard; the input gate controls the input of new information; and the output gate determines the output content. This allows the LSTM to remember important information from long periods of time while also updating its state based on current input, effectively capturing both long-term dependencies and short-term changes in historical behavior sequences.

[0064] Based on the above explanation of terms, the implementation environment of a robot operating system supporting multimodal interaction provided in an embodiment of the present application is described. Schematically, the implementation environment includes: sensor devices, processors, storage devices, and output devices. The sensors may be visual sensors, auditory sensors, tactile sensors, or semantic data acquisition devices; visual sensors include, but are not limited to, optical cameras, depth cameras, and infrared cameras; auditory sensors may be multi-microphone arrays; tactile sensors may be pressure sensors, strain sensors, capacitive tactile sensors, and the like, distributed at the operating parts of the robot; semantic data acquisition devices may receive data from external devices (such as a user's mobile device, computer, etc.) via a network interface, or be equipped with dedicated input devices, such as a keyboard, touch screen, etc., to enable users to input text information, which is not limited here.

[0065] In combination with the above-mentioned explanations of terms and implementation environment, the application scenarios of the embodiments of the present application are explained.

[0066] The robot operating system supporting multimodal interaction provided in the embodiments of the present application can be applied in scenarios including but not limited to the following:

[0067] In the multimodal interactive robot system of smart home, the system can be used to collect visual, auditory, tactile and semantic information in the room, build a user profile for the user, and automatically adjust the operating instructions of smart furniture according to the user's appliance usage habits and preferences. For example, it can automatically adjust the brightness of lights according to user habits, turn on appliances in advance, etc., to improve the user experience.

[0068] In intelligent tour guide applications, the system can be used to provide tour guide services to tourists. Robots equipped with the system can identify exhibits, crowds, and environmental layouts, and receive questions from tourists for interaction. At the same time, user profiles of tourists are established based on multimodal heterogeneous data. For example, if it is found that tourists are more interested in historical exhibits, stay longer at certain exhibits, or frequently ask specific types of questions, this information will be incorporated into the user profile. Using a deep learning model, the user profile of the tourist will be analyzed, and operational instructions will be issued based on the strategy derived from the analysis to provide personalized services to tourists.

[0069] Illustratively, the robot operating system supporting multimodal interaction provided in the embodiments of the present application can also be applied to other application scenarios. This is only provided as an example and is not limited to specific application scenarios.

[0070] In an exemplary embodiment, Figure 1 As shown, a robot operating system 10 supporting multimodal interaction is provided. This system is described using the aforementioned processor as an example. It is understood that the system can also be implemented through interaction between the robot's processor and other processors / controllers / servers. In this embodiment, the system includes the following: a data receiving module 11, a target object recognition module 12, an analysis and decision module 13, and a response module 14. Among them:

[0071] The data receiving module 11 is used to pre-process the acquired multimodal heterogeneous data to obtain sensory data; wherein the multimodal heterogeneous data includes at least one of visual data, auditory data, tactile data and semantic data.

[0072] For example, the system can receive data from multiple modalities, including visual, auditory, tactile, and semantic data. This allows the system to acquire information from different perspectives, thereby gaining a more comprehensive understanding of the user and their environment. For example, not only can the user's appearance and movements be seen through visual data, but the user's voice and ambient sounds can also be heard through auditory data, and the user's operating force can be perceived through tactile data, providing a rich information foundation for subsequent analysis.

[0073] The target object recognition module 12 is used to establish a corresponding user portrait of the target object based on the sensory data.

[0074] For example, building a user profile based on multimodal sensory data can provide personalized services for users. By analyzing multiple aspects of user information, such as appearance, voice characteristics, and operating habits, a detailed user profile is constructed to better understand the user's characteristics and needs, making the interaction between the system and the user more tailored to the user's actual situation.

[0075] The analysis and decision-making module 13 is used to perform preference analysis based on the user portrait and derive a response strategy based on a deep learning model.

[0076] For example, a deep learning model is used to perform preference analysis and derive response strategies. Deep learning has powerful feature learning and representation capabilities, and can automatically mine deep user preference information from user portraits. The deep learning model can generate highly adaptable response strategies based on different user portraits and a large amount of historical data.

[0077] The response module 14 is configured to issue an operation instruction based on the response strategy.

[0078] For example, different response strategies can be used to issue diverse operational instructions to meet the needs of different users in different scenarios. For example, based on user preferences, operational instructions might be issued to play the user's favorite music, recommend products of interest to the user, or provide the user with information they need, thereby diversifying system functionality. Different combinations of output modes can also be selected based on different response strategies, enhancing the system's adaptability to complex environments and user behaviors.

[0079] The above-mentioned robot operating system supporting multimodal interaction integrates a data receiving module, a target object recognition module, an analysis and decision module, and a response module. The data receiving module converts the acquired multimodal heterogeneous data into sensory data through preprocessing. The multimodal heterogeneous data includes at least one of visual data, auditory data, tactile data and semantic data. The target object recognition module establishes a corresponding user portrait for the target object based on these sensory data. The analysis and decision module uses a deep learning model to perform preference analysis on the user portrait and derives a response strategy based on this. The response module issues operation instructions based on the response strategy determined by the analysis and decision module, so that the system can respond to user needs quickly and effectively, improve the response speed and efficiency of the system, and meet the complex and diverse personalized needs of users.

[0080] In one embodiment, the target object recognition module 12 may further include a multimodal data fusion unit, a user profiling unit, and a user classification unit.

[0081] The multimodal data fusion unit is used to extract features of the sensory data based on the multimodal data fusion model, where the features include at least one of age, gender, language habits, knowledge level, and disability.

[0082] Specifically, the system collects multimodal data, including visual, auditory, and tactile data, and applies a multimodal data fusion model to extract features from this integrated data. For example, facial recognition technology can be used to extract age and gender information from visual data; natural language processing technology can be used to analyze voice data to identify language habits and knowledge levels; and interactive behavior and image recognition can be used to determine whether a user is disabled. By integrating visual, auditory, and other information, the multimodal data fusion unit extracts more comprehensive and accurate features, thereby constructing a more precise user profile.

[0083] The user portrait unit is used to create a user portrait based on sensory data from the same user, and store and manage the user portrait based on voiceprint features as an index.

[0084] Specifically, various sensory data features from the same user are aggregated to construct a user portrait. Voiceprint recognition technology is used to extract the user's voiceprint features and use them as index identifiers. Each user portrait is associated with the corresponding voiceprint features and stored in the database to ensure that each user's portrait is unique and traceable.

[0085] The user classification unit is used to classify group characteristics based on user portraits and to train deep learning models.

[0086] Specifically, we conduct in-depth analysis of various features within user profiles to uncover commonalities and differences between different users. Based on the analysis results, we categorize users into different groups. This categorized user group data serves as the training set for deep learning models. By continuously adjusting model parameters, we enable the model to accurately identify the characteristics of different groups, improving training efficiency and accuracy.

[0087] In one embodiment, the analysis and decision module 13 may further include an input submodule and an output submodule.

[0088] The input submodule is used to obtain the user's historical behavior sequence data, assign weights to each data point through the attention mechanism and input it into the long short-term memory network, where the attention energy The calculation formula is as follows:

[0089]

[0090] in, is the attention energy, is the number of time steps of the historical behavior sequence data, is the attention vector, is the weight matrix, is the bias vector, is the i-th input data, is the hyperbolic tangent activation function.

[0091] Focus your energy Convert to attention weight , as shown in the formula:

[0092]

[0093] in, is the attention weight, exp is the exponential function, is the sum of the exponential operations on the attention energy at all time steps;

[0094] The weighted input sequence is obtained by multiplying the input sequence with the attention weight using the following formula to obtain the weighted input sequence :

[0095]

[0096] in, is the t-th input after attention weighting, is the attention weight of the ith input at the tth time step, is the i-th input data.

[0097] Specifically, the attention mechanism assigns weights to each data point, effectively distinguishing the importance of each data point in historical behavioral sequence data. In real-world applications, different user behaviors have varying degrees of impact on their current preferences and needs. The attention mechanism automatically learns these differences, allowing the model to focus more on important data points when performing preference analysis, thereby improving the accuracy and effectiveness of the analysis.

[0098] The output submodule is used to:

[0099] The preference representation is calculated based on the weighted input sequence using the following formula :

[0100]

[0101] in, is the weight matrix of the fully connected layer, is the sum of the weighted input sequence, is the rectified linear unit activation function, is the bias vector of the fully connected layer;

[0102] Generate a response strategy based on the preference representation using the following formula :

[0103]

[0104] in, is the preference representation, is the weight matrix of the fully connected layer, is the bias vector, is the activation function.

[0105] For example, the use of fully connected layers in the preference representation calculation and response strategy generation process makes the model highly flexible and scalable. Fully connected layers can adapt to different tasks and data characteristics by adjusting the weight matrix and bias vector. This also facilitates further optimization and improvement of the model, such as adding more hidden layers and adjusting the network structure, to improve its performance and adaptability.

[0106] The output submodule generates a user preference representation by processing the weighted input sequence, and generates a response strategy based on this. It can generate a highly personalized response strategy based on the unique historical behavior of each user. This system can provide users with product recommendations and service suggestions that are more in line with their needs based on their preferences, greatly improving user experience and satisfaction and enhancing the competitiveness of the system.

[0107] In one embodiment, the response module 14 may include:

[0108] The real-time situational awareness unit is used to identify the current interaction scenario based on sensory data, user profiles, and operational behaviors in order to formulate corresponding output methods; among them, user profiles include preference information and emotional state.

[0109] For example, this system extracts environmental scene information based on sensory data and identifies specific interaction scenarios based on user profiles and operational behaviors to develop appropriate output methods. The output methods can be a combination of various output methods, such as voice, vibration feedback, projection, or image text, etc., which can convey information from different dimensions, meet the user's diverse reception needs, enhance user participation and satisfaction, and adapt to different usage scenarios. This application does not limit this. The source of preference information in the user profile can be the user's registration information, historical interaction records, questionnaire feedback, etc.; the source of emotional state can be analyzed from the user's tone, voice, and word choice.

[0110] The real-time situational awareness unit also includes:

[0111] The preference matching subunit is used to establish a mapping relationship between preference information and output mode based on a machine learning algorithm.

[0112] For example, based on the data characteristics and the nature of the problem, an appropriate machine learning algorithm, such as a decision tree, support vector machine (SVM), or neural network, is selected. The model is trained using the compiled preference information and output method data. By continuously adjusting the model parameters, the model accurately learns the mapping relationship between preference information and output methods. During the training process, cross-validation and other methods are used to evaluate the model's performance to ensure its accuracy and generalization ability.

[0113] The dynamic decision-making subunit is used to continuously monitor the interaction scenario and emotional state and extract feature vectors during the interaction process. When the Euclidean distance between the feature vector of the interaction scenario and / or emotional state within a preset time period and the feature vector at the previous moment exceeds a preset threshold, the output method is re-formulated.

[0114] For example, during the interaction process, sensory data, user profile data, and operational behavior data are continuously collected. Corresponding feature vectors are extracted for the interaction scenario and emotional state. For the interaction scenario, environmental features (such as light intensity and noise level) and device status (battery level, network connection) can be extracted. For the emotional state, facial expression features (such as the angle of the mouth corners raised, changes in eye contact) and speech features (such as speech rate, intonation, and volume changes) can be extracted through techniques such as facial expression analysis and voice emotion analysis. At preset intervals, feature vectors for the current interaction scenario and emotional state are periodically extracted and compared with the feature vectors from the previous moment. The Euclidean distance between the two is calculated. When the Euclidean distance exceeds a preset threshold, it indicates a significant change in the interaction scenario and / or emotional state. At this point, the output mode is reconfigured based on the new interaction scenario and user emotional state, combined with the mapping relationship between preference information and output mode. For example, if the user's mood changes from calm to excited, and the current ambient light is dim, the output mode may be adjusted from primarily text display to primarily voice output, and the voice volume may be appropriately increased based on the mapping relationship.

[0115] Furthermore, a robot operating system supporting multimodal interaction provided in an embodiment of the present application may also include a group user portrait module and an interactive resonance module.

[0116] The group user portrait module is used to extract representative features from multiple user portraits and perform cluster analysis based on the representative features to obtain multiple groups and corresponding group user portraits; the representative features include at least one of self-identity, geographical distribution, age and education level.

[0117] Specifically, a large amount of individual user profile data is extracted from the system's existing user database. This data covers basic user information, preferences, behavioral data, and other aspects. Representative features are then extracted for each user profile. For example, self-identification tendencies are analyzed from users' social interaction data and self-evaluations; geographical distribution information is obtained from registration information or location data; age information is obtained from the age field; and educational level information is extracted from educational background information. For ambiguous or missing data, data cleaning and supplementation algorithms are used to ensure feature validity. Clustering algorithms, such as K-Means, are used to group user profiles with similar representative features into the same cluster. For each cluster, the characteristics of its members are statistically analyzed and summarized. The group's average age, primary geographical distribution, concentrated educational level, and prevalent self-identification type are calculated. Further analysis can be performed to analyze the shared preferences and behavioral patterns of users within the group, thereby constructing a detailed group user profile.

[0118] The interactive resonance module is used to conduct data mining on group user portraits to obtain personalized interaction strategies for each group. Personalized interaction strategies are used to indicate operational instructions.

[0119] Specifically, by analyzing the behavioral patterns, preference information, and interaction history of group users with the system, we find interaction methods and strategies that can resonate with the group and use them to indicate operational instructions.

[0120] Furthermore, the robot operating system supporting multimodal interaction provided by the embodiment of the present application may also include an index module, a cache module and an interest update module.

[0121] The index module is used to perform hash operations on the voiceprint features to obtain a hash value used to determine the storage location.

[0122] Specifically, voiceprint recognition technology is used to extract voiceprint features from the user's voice data, such as the Mel-Frequency Cepstral Coefficient (MFCC) algorithm. After preprocessing, framing, windowing and other operations on the voice signal, a representative voiceprint feature vector is calculated. A hash operation is performed on the extracted voiceprint feature vector to convert the voiceprint feature vector into a hash value of a fixed length. The hash value is unique and deterministic and can uniquely identify the corresponding voiceprint feature. The storage location in the storage system is determined according to the hash value. For example, the address space of the storage system can be modulo-operated through the hash value, and the result is used as the index of the storage address to determine the storage location of the voiceprint feature-related data. The storage location can be quickly located through the hash value, which can respond quickly and improve the data retrieval speed.

[0123] The cache module is used to store the hash value of the voiceprint feature as the key and the user's interest information as the value in the cache. The interest information is information that has been accessed more than a preset threshold number of times within a preset time period.

[0124] Specifically, after obtaining the user's voiceprint feature hash value and corresponding interest information, the key-value pair is stored in the cache using the API provided by the cache system. Based on the cache capacity and data access frequency, an appropriate cache replacement strategy is adopted, such as least recently used (LRU) or most recently used (LFU). When the cache is full and new data needs to be stored, some old data is eliminated according to the replacement strategy to ensure efficient cache operation. Frequently used user interest information is stored in the cache. When the system needs to obtain this information, it can be read directly from the cache, avoiding reading data from slower storage media (such as hard disks). This improves data access speed, enhances user experience, reduces system load, and improves system performance and stability.

[0125] The interest update module is used to perform cluster analysis on interest information and tag it with corresponding interest tags. Based on the interest tags, it uses web crawler technology to regularly crawl interest-related information from selected information sources. The interest-related information is used to update the interest information.

[0126] Specifically, cluster analysis is regularly performed on user interest information. Clustering algorithms are used to group similar interest information into the same category and label them with interest tags. Based on the user's interest tags, appropriate information sources are selected, such as relevant news websites, professional forums, social media accounts, etc. Web crawler technology is used to crawl interest-related information from the selected information sources according to set rules and frequencies. The crawled new information is compared and updated with the interest information in the cache. New interest-related information is added to the cache, and the corresponding interest tags and visit counts are updated. Existing interest information is updated if there is any new content, ensuring that the user's interest information is always up to date and accurate. Continuously updating interest information and labeling it with accurate interest tags helps the system more accurately understand the user's interest preferences, thereby providing users with content and services that better suit their interests, improving the quality and satisfaction of user interaction with the system.

[0127] Preferably, the response module 14 may further include: an active interaction unit and a feedback receiving unit;

[0128] The active interaction unit is used to start the active interaction program.

[0129] The feedback receiving unit is used to obtain multimodal heterogeneous data again after issuing an operation instruction and calculate the satisfaction of the target object based on artificial intelligence. If the satisfaction is lower than the threshold, the active interaction unit is called to start the active interaction program.

[0130] Specifically, after the system issues an operation instruction, it collects multimodal, heterogeneous data again through various sensors and data interfaces. For example, a camera is used to capture the user's facial expressions after receiving the operation instruction, and a microphone is used to record the user's possible comments. Using artificial intelligence technology, a large amount of user feedback data and corresponding actual satisfaction annotations (which can be obtained through user-initiated reviews, questionnaires, etc.) are collected to train the model. The collected multimodal, heterogeneous data is input into the constructed satisfaction calculation model, and the satisfaction score of the target object is calculated in real time. A preset satisfaction threshold is set and the calculated satisfaction score is compared with the threshold. If the satisfaction level falls below the threshold, it indicates that the user may be dissatisfied with the execution result of the system's operation instruction. In this case, the feedback receiving module immediately calls the active interaction module and initiates the active interaction program. Upon receiving the call signal from the feedback receiving module, the active interaction module selects an appropriate active interaction strategy based on the current situation and initiates the corresponding interaction program. By collecting multimodal data in real time and calculating user satisfaction, the system can comprehensively and accurately monitor user feedback on the operation instruction. When low user satisfaction is detected, the active interaction module can automatically call the active interaction module to intervene, enabling the system to adjust its strategy based on actual user feedback, improving the system's adaptability and flexibility and promptly meeting user needs.

[0131] To summarize, the robot operating system supporting multimodal interaction provided in the embodiment of the present application is responsible for collecting multimodal heterogeneous data such as vision, hearing, touch and semantics through the data receiving module, and preprocessing it into sensory data; the target object recognition module extracts the features of the sensory data, such as age and gender, based on the multimodal data fusion unit; the user portrait unit establishes a user portrait and manages it with a voiceprint feature index, and the user classification unit classifies group features based on the portrait to provide data for deep learning model training; the input submodule of the analysis and decision-making module uses the attention mechanism to process the user's historical behavior sequence data and then inputs it into the long short-term memory network, and the output submodule calculates the preference representation based on this and generates a response strategy. In addition to issuing operational instructions based on the response strategy, the response module can also include a real-time contextual awareness unit. This unit establishes a mapping between preferences and output methods through a preference matching subunit, and a dynamic decision-making subunit monitors contextual and emotional changes and adjusts the output method accordingly. The system also features a group user portrait module and an interactive resonance module. This module clusters and analyzes multiple user portraits to derive a group portrait, which is then used to develop personalized interaction strategies. The indexing module, cache module, and interest update module work together to hash voiceprint features and cache user interest information, regularly updating this information through web crawler technology. After issuing instructions, the feedback receiving unit obtains multimodal data to calculate user satisfaction. If the user satisfaction score is not met, the active interaction unit is invoked to initiate the interaction process. Through the integrated operation of various modules, the above-mentioned solution formulates diverse interaction strategies based on different application scenarios and user group characteristics. This allows for a quick and effective response to users' personalized needs, improving the system's response speed and efficiency and meeting the complex and diverse personalized needs of users.

[0132] It should be understood that, although the various steps in the flowcharts involved in the various embodiments described above are displayed in sequence according to the instructions of the arrows, these steps are not necessarily executed in sequence in the order indicated by the arrows. Unless otherwise specified herein, there is no strict order restriction on the execution of these steps, and these steps can be executed in other orders. Moreover, at least a portion of the steps in the flowcharts involved in the various embodiments described above can include multiple steps or multiple stages, and these steps or stages are not necessarily executed and completed at the same time, but can be executed at different times, and the execution order of these steps or stages is not necessarily to be carried out in sequence, but can be executed in turn or alternately with other steps or at least a portion of steps or stages in other steps.

[0133] Based on the same inventive concept, the present application also provides a method for implementing the aforementioned method of operating a robot supporting multimodal interaction. The solution provided by this method is similar to the solution described in the aforementioned method. Therefore, the specific limitations of one or more embodiments of the method for operating a robot supporting multimodal interaction provided below can be found in the above-mentioned limitations on the robot operating system supporting multimodal interaction, and will not be repeated here.

[0134] In an exemplary embodiment, Figure 2 As shown, a robot operation method supporting multimodal interaction is provided, comprising the following steps:

[0135] Step 201: Acquire multimodal heterogeneous data and preprocess the multimodal heterogeneous data to obtain sensory data; wherein the multimodal heterogeneous data includes at least one of visual data, auditory data, tactile data, and semantic data;

[0136] Step 202: Create a corresponding user profile for the target object based on the sensory data;

[0137] Step 203: Perform preference analysis based on the user profile and derive a response strategy based on the deep learning model;

[0138] Step 204: Issue an operation instruction based on the response strategy.

[0139] In one embodiment, step 202 of the method may further include the following sub-steps:

[0140] Features of sensory data are extracted based on a multimodal data fusion model, where the features include at least one of age, gender, language habits, knowledge level, and disability.

[0141] Create a user profile based on sensory data from the same user, and use voiceprint features as an index to store and manage the user profile.

[0142] Group characteristics are classified according to user portraits for training deep learning models.

[0143] In one embodiment, step 203 of the method may further include the following sub-steps:

[0144] Obtain the user's historical behavior sequence data, assign weights to each data point through the attention mechanism and input it into the long short-term memory network, where the attention energy The calculation formula is as follows:

[0145]

[0146] in, is the attention energy, is the number of time steps of the historical behavior sequence data, is the attention vector, is the weight matrix, is the bias vector, is the i-th input data, is the hyperbolic tangent activation function.

[0147] Focus your energy Convert to attention weight , as shown in the formula:

[0148]

[0149] in, is the attention weight, exp is the exponential function, is the sum of the exponential operations on the attention energy at all time steps;

[0150] The weighted input sequence is obtained by multiplying the input sequence with the attention weight using the following formula to obtain the weighted input sequence:

[0151]

[0152] in, is the t-th input after attention weighting, is the attention weight of the ith input at the tth time step, is the i-th input data.

[0153] The preference representation is calculated based on the weighted input sequence using the following formula :

[0154]

[0155] in, is the weight matrix of the fully connected layer, is the sum of the weighted input sequence, is the rectified linear unit activation function, is the bias vector of the fully connected layer;

[0156] Generate a response strategy based on the preference representation using the following formula :

[0157]

[0158] in, is the preference representation, is the weight matrix of the fully connected layer, is the bias vector, is the activation function.

[0159] Furthermore, step 204 of the method may further include the following sub-steps:

[0160] Identify the current interaction scenario based on sensory data, user profile, and operational behavior to develop a corresponding output method; among them, the user profile includes preference information and emotional state.

[0161] The above-mentioned step of identifying the current interaction scenario based on sensory data, user profile, and operation behavior to formulate a corresponding output method, wherein the user profile includes preference information and emotional state, may also include the following sub-steps:

[0162] A mapping relationship between preference information and output methods is established based on machine learning algorithms.

[0163] During the interaction process, the interaction scenario and emotional state are continuously monitored and feature vectors are extracted. When the Euclidean distance between the feature vector of the interaction scenario and / or emotional state within a preset period and the feature vector at the previous moment exceeds a preset threshold, the output method is re-formulated.

[0164] Preferably, the method may further comprise the following steps:

[0165] Extract representative features from multiple user portraits and perform cluster analysis based on the representative features to obtain multiple groups and corresponding group user portraits; the representative features include at least one of self-identity, geographical distribution, age, and education level;

[0166] Data mining is performed on group user portraits to obtain personalized interaction strategies for each group. Personalized interaction strategies are used to indicate operation instructions.

[0167] Preferably, the method may further comprise the following steps:

[0168] Perform a hash operation on the voiceprint feature to obtain a hash value used to determine the storage location.

[0169] The hash value of the voiceprint feature is used as the key and the user's interest information is stored in the cache as the value. The interest information is the information that has been accessed more than a preset threshold number of times within a preset period of time;

[0170] The interest information is clustered and tagged with corresponding interest tags. Based on the interest tags, web crawler technology is used to regularly crawl interest-related information from selected information sources, and the interest-related information is used to update the interest information.

[0171] Preferably, step 204 of the method may further include the following steps:

[0172] Start active interaction program;

[0173] After issuing the operation instruction, multimodal heterogeneous data is obtained again and the satisfaction of the target object is calculated based on artificial intelligence. If the satisfaction is lower than the threshold, the active interaction unit is called to start the active interaction program.

[0174] In one embodiment, a computer device is provided, including a memory and a processor, wherein the memory stores a computer program, and the processor implements the steps of the method in each of the above embodiments when executing the computer program.

[0175] In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the steps of the method in the above embodiments are implemented.

[0176] The technical features of the above embodiments can be combined arbitrarily. In order to make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0177] Those skilled in the art will appreciate that the logic and / or steps represented in the flowcharts or otherwise described herein, for example, can be considered as a sequenced list of executable instructions for implementing logical functions, and can be embodied in any computer-readable storage medium for use by an instruction execution system, apparatus, or device (e.g., a computer-based system, a system including a processor, or other system that can fetch and execute instructions from an instruction execution system, apparatus, or device), or for use in conjunction with such instruction execution system, apparatus, or device. For purposes of this specification, a "computer-readable storage medium" can be any device that can contain, store, communicate, propagate, or transmit a program for use by an instruction execution system, apparatus, or device, or in conjunction with such instruction execution system, apparatus, or device.

[0178] More specific examples (a non-exhaustive list) of computer-readable storage media include the following: an electrical connection with one or more wires (electronic device), a portable computer disk cartridge (magnetic device), a random access memory (RAM), a read-only memory (ROM), an erasable and programmable read-only memory (EPROM or flash memory), a fiber optic device, and a portable compact disk read-only memory (CDROM). In addition, the computer-readable storage medium may even be paper or other suitable medium on which the program is printed, since the program may be obtained electronically, for example, by optically scanning the paper or other medium, followed by editing, deciphering, or processing in another suitable manner as necessary, and then stored in a computer memory.

[0179] It should be understood that various components of the present invention may be implemented using hardware, software, firmware, or a combination thereof. In the aforementioned embodiments, multiple steps or methods may be implemented using software or firmware stored in a memory and executed by a suitable instruction execution system. For example, if implemented using hardware, as in another embodiment, any one or a combination of the following technologies known in the art may be used: a discrete logic circuit having logic gate circuits for implementing logic functions on data signals, an application-specific integrated circuit having suitable combinational logic gate circuits, a programmable gate array (PGA), a field-programmable gate array (FPGA), etc.

[0180] Throughout this specification, reference to terms such as "one embodiment," "some embodiments," "examples," "specific examples," or "some examples" means that a specific feature, structure, material, or characteristic described in conjunction with that embodiment or example is included in at least one embodiment or example of the present invention. In this specification, schematic representations of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in any one or more embodiments or examples.

[0181] The above-described embodiments merely illustrate several implementations of the present invention, and while their descriptions are relatively specific and detailed, they should not be construed as limiting the scope of the present invention. It should be noted that a person skilled in the art would be able to make numerous variations and improvements without departing from the spirit of the present invention, all of which fall within the scope of protection of the present invention. Therefore, the scope of protection of the present invention shall be determined by the appended claims.

Claims

1. A robot operating system supporting multimodal interaction, characterized in that: The system comprises: Data receiving module, target object recognition module, analysis and decision module and response module; The data receiving module is used to preprocess the acquired multimodal heterogeneous data to obtain sensory data; wherein the multimodal heterogeneous data includes two or more of visual data, auditory data, tactile data and semantic data; The target object recognition module is used to establish a corresponding user portrait of the target object based on the sensory data; The analysis and decision-making module is used to use a deep learning model to perform preference analysis based on the user portrait and to derive a response strategy based on the preference analysis; The response module is used to issue an operation instruction based on the response strategy; The target object recognition module includes a multimodal data fusion unit, a user profiling unit and a user classification unit; The multimodal data fusion unit is used to extract features of the sensory data based on a multimodal data fusion model, where the features include at least one of age, gender, language habits, knowledge level, and disability; The user portrait unit is used to create the user portrait based on the sensory data from the same user, and store and manage the user portrait based on the voiceprint feature as an index; The analysis and decision module includes an input submodule and an output submodule: The input submodule is used to extract the user's historical behavior sequence data from the user portrait, assign weights to each data point through the attention mechanism and input them into the long short-term memory network, where the attention energy The calculation formula is as follows: in, is the attention energy, is the number of time steps of the historical behavior sequence data, is the attention vector, is the weight matrix, is the bias vector, is the i-th input data, is the hyperbolic tangent activation function; Use the attention conversion formula to convert attention energy Convert to attention weight , where the attention conversion formula is as follows: in, is the attention weight, exp is the exponential function, is the sum of the exponential operations on the attention energy at all time steps; Multiply the input sequence by the attention weight to obtain the weighted input sequence, where the calculation formula of the weighted input sequence is as follows: in, is the t-th input after attention weighting, is the attention weight of the ith input at the tth time step, is the i-th input data; The output submodule is used for: Computing a preference representation based on the weighted input sequence , where preference represents The calculation formula is as follows: in, is the weight matrix of the fully connected layer, is the sum of the weighted input sequence, is the rectified linear unit activation function, is the bias vector of the fully connected layer; Generating response strategies based on preference representation , where the response strategy The calculation formula is as follows: in, is the preference representation, is the weight matrix of the fully connected layer, is the bias vector, is the activation function; The response module includes: A real-time context perception unit, configured to identify the current interaction context based on the sensory data, the user profile, and the operation behavior, so as to formulate a corresponding output method; wherein the user profile includes preference information and emotional state, and the output method is a specific form of the operation instruction; A dynamic decision-making subunit is used to continuously monitor the interaction scenario and the emotional state during the interaction process and extract feature vectors, and when the Euclidean distance between the feature vector of the interaction scenario and / or the emotional state within a preset time period and the feature vector at the previous moment exceeds a preset threshold, the output method is re-formulated.

2. The robot operating system supporting multimodal interaction according to claim 1, characterized in that: The system further includes an index module, a cache module and an interest update module; The index module is used to perform a hash operation on the voiceprint feature to obtain a hash value for determining a storage location; The cache module is used to store the hash value of the voiceprint feature as a key and the user's interest information as a value in the cache, wherein the interest information is information whose access times exceed a preset threshold within a preset time period; The interest updating module is used to perform cluster analysis on the interest information and add corresponding interest tags, and use web crawler technology to regularly crawl interest-related information from selected information sources based on the interest tags. The interest-related information is used to update the interest information.

3. The robot operating system supporting multimodal interaction according to claim 1, characterized in that: The response module further includes: an active interaction unit and a feedback receiving unit; The active interaction unit is used to start the active interaction program; The feedback receiving unit is used to obtain the multimodal heterogeneous data again after issuing an operation instruction and calculate the satisfaction of the target object based on artificial intelligence. If the satisfaction is lower than a threshold, the active interaction unit is called to start the active interaction program.

4. A method for operating a robot supporting multimodal interaction, characterized in that: The robot operating system supporting multimodal interaction according to claim 1 is implemented, comprising: Acquiring multimodal heterogeneous data and preprocessing the multimodal heterogeneous data to obtain sensory data; wherein the multimodal heterogeneous data includes two or more of visual data, auditory data, tactile data, and semantic data; Creating a corresponding user profile for the target object based on the sensory data; Using a deep learning model to perform preference analysis based on the user profile, and deriving a response strategy based on the preference analysis; An operation instruction is issued based on the response strategy.

5. A computer device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the method according to claim 4 is implemented.

6. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the method according to claim 4 is implemented.

Images