Robot operating system supporting multi-modal interaction and operating method

Abstract

The invention discloses a robot operation system and operation method supporting multi-modal interaction, and relates to the technical field of intelligent robots, the system comprises a data receiving module, a target object recognition module, an analysis decision module and a response module; the data receiving module is used for preprocessing the obtained multi-modal heterogeneous data to obtain sensory data; wherein the multi-modal heterogeneous data comprises at least one of visual data, auditory data, tactile data and semantic data; the target object recognition module is used for establishing a corresponding user portrait for the target object according to the sensory data; the analysis decision module is used for performing preference analysis according to the user portrait and obtaining a response strategy based on a deep learning model; the response module is used for sending an operation instruction based on the response strategy. According to the method, the requirements of the user can be better understood and the response can be made based on the individual characteristics of the user.

Description

Technical Field

[0001] The present invention relates to the technical field of intelligent robots, and in particular to a robot operating system and an operating method supporting multi-modal interaction. Background Art

[0002] Currently, robots have limited capabilities in understanding user behaviors, intentions, and emotions, and are unable to accurately capture subtle clues in user communications. In long-term interactions with users, they lack growth, find it difficult to truly understand user needs, and are unable to provide intelligent and customized services. Summary of the invention

[0003] Based on this, the purpose of the present invention is to provide a robot operating system and operating method that supports multimodal interaction, aiming to better understand the user's needs and respond to them based on the user's individual characteristics.

[0004] In a first aspect, the present application provides a robot operating system supporting multimodal interaction, comprising: Data receiving module, target object recognition module, analysis and decision module and response module; The data receiving module is used to pre-process the acquired multimodal heterogeneous data to obtain sensory data; wherein the multimodal heterogeneous data includes at least one of visual data, auditory data, tactile data and semantic data; The target object recognition module is used to establish a corresponding user portrait of the target object based on the sensory data; The analysis and decision-making module is used to analyze preferences based on user portraits and derive response strategies based on deep learning models; The response module is used to issue operation instructions based on the response strategy.

[0005] In one of the embodiments, the target object recognition module includes a multimodal data fusion unit, a user profiling unit, and a user classification unit; The multimodal data fusion unit is used to extract features of the sensory data based on the multimodal data fusion model, where the features include at least one of age, gender, language habits, knowledge level, and whether or not the person is disabled; The user portrait unit is used to create a user portrait based on sensory data from the same user, and store and manage the user portrait based on the voiceprint feature as an index; The user classification unit is used to classify group characteristics based on user portraits for training deep learning models.

[0006] In one embodiment, the analysis and decision module includes an input submodule and an output submodule: The input submodule is used to obtain the user's historical behavior sequence data, assign weights to each data point through the attention mechanism and input it into the long short-term memory network, where the attention energy The calculation formula is as follows:

[0007] in, is the attention energy, is the number of time steps of the historical behavior sequence data, is the attention vector, is the weight matrix, is the bias vector, is the i-th input data, is the hyperbolic tangent activation function; Use the attention conversion formula to convert attention energy Convert to attention weight , where the attention conversion formula is as follows:

[0008] in, is the attention weight, exp is the exponential function, is the sum of the exponential operations on the attention energy at all time steps; Multiply the input sequence by the attention weight to get the weighted input sequence, where the calculation formula of the weighted input sequence is as follows:

[0009] in, is the t-th input after attention weighting, is the attention weight of the ith input at the tth time step, is the i-th input data; The output submodule is used to: Computing preference representations based on weighted input sequences , where preference represents The calculation formula is as follows:

[0010] in, is the weight matrix of the fully connected layer, is the sum of the weighted input sequence, is the rectified linear unit activation function, is the bias vector of the fully connected layer; Generating response strategies based on preference representation , where the response strategy The calculation formula is as follows:

[0011] in, is the preference representation, is the weight matrix of the fully connected layer, is the bias vector, is the activation function.

[0012] Furthermore, the response module includes: A real-time situational awareness unit, which is used to identify the current interaction situation based on sensory data, user profiles, and operation behaviors, so as to formulate corresponding output methods; wherein the user profiles include preference information and emotional states; The real-time situational awareness unit also includes: A preference matching subunit, used to establish a mapping relationship between preference information and output mode based on a machine learning algorithm; The dynamic decision subunit is used to continuously monitor the interaction scenario and emotional state and extract feature vectors during the interaction process. When the Euclidean distance between the feature vector of the interaction scenario and / or emotional state within a preset time period and the feature vector at the previous moment exceeds a preset threshold, the output method is re-formulated.

[0013] Preferably, the system also includes a group user portrait module and an interactive resonance module: The group user portrait module is used to extract representative features from multiple user portraits, and perform cluster analysis based on the representative features to obtain multiple groups and corresponding group user portraits; the representative features include at least one of self-identification, geographical distribution, age and education level; The interactive resonance module is used to conduct data mining on group user portraits to obtain personalized interaction strategies for each group. The personalized interaction strategies are used to indicate operation instructions.

[0014] Preferably, the system further includes an index module, a cache module and an interest update module: The index module is used to perform hash operations on the voiceprint features to obtain a hash value used to determine the storage location; The cache module is used to store the hash value of the voiceprint feature as a key and the user's interest information as a value in the cache, where the interest information is information whose access times exceed a preset threshold within a preset period of time; The interest update module is used to perform cluster analysis on interest information and add corresponding interest tags. Based on the interest tags, the web crawler technology is used to regularly crawl interest-related information from the selected information source, and the interest-related information is used to update the interest information.

[0015] Preferably, the response module further includes: an active interaction unit and a feedback receiving unit; The active interaction unit is used to start the active interaction program; The feedback receiving unit is used to obtain multimodal heterogeneous data again and calculate the satisfaction of the target object based on artificial intelligence after issuing an operation instruction. If the satisfaction is lower than the threshold, the active interaction unit is called to start the active interaction program.

[0016] In a second aspect, the present application also provides a robot operation method supporting multimodal interaction, comprising: Acquire multimodal heterogeneous data and preprocess the multimodal heterogeneous data to obtain sensory data; wherein the multimodal heterogeneous data includes at least one of visual data, auditory data, tactile data and semantic data; Create a corresponding user portrait for the target object based on sensory data; Conduct preference analysis based on user portraits and derive response strategies based on deep learning models; Issue action instructions based on the response strategy.

[0017] In a third aspect, the present application further provides a computer device, including a memory and a processor, wherein the memory stores a computer program, and the processor implements the above-mentioned method when executing the computer program.

[0018] In a fourth aspect, the present application also provides a computer-readable storage medium on which a computer program is stored, and when the computer program is executed by a processor, it is used to implement the method as described above.

[0019] The present invention integrates a data receiving module, a target object recognition module, an analysis and decision module, and a response module. The data receiving module converts the acquired multimodal heterogeneous data into sensory data through preprocessing. The multimodal heterogeneous data includes at least one of visual data, auditory data, tactile data, and semantic data. The target object recognition module establishes a corresponding user portrait for the target object based on these sensory data. The analysis and decision module uses a deep learning model to perform preference analysis on the user portrait and derives a response strategy based on this. The response module issues an operation instruction based on the response strategy determined by the analysis and decision module, so that the system can respond to user needs quickly and effectively, improve the response speed and efficiency of the system, and meet the complex and diverse personalized needs of users. The problem that robots in the prior art are difficult to truly understand user needs and cannot provide intelligent and customized services is solved. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the related technologies, the drawings required for use in the embodiments or the related technical descriptions are briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0021] Figure 1 A schematic diagram of a robot operating system supporting multimodal interaction provided by an embodiment of the present invention; Figure 2 A schematic flow chart of a robot operation method supporting multimodal interaction provided by an embodiment of the present invention.

[0022] The following specific implementation manner will further illustrate the present invention in conjunction with the above-mentioned drawings. DETAILED DESCRIPTION

[0023] In order to facilitate the understanding of the present invention, the present invention will be described more fully below with reference to the relevant drawings. Several embodiments of the present invention are given in the drawings. However, the present invention can be implemented in many different forms and is not limited to the embodiments described herein. On the contrary, the purpose of providing these embodiments is to make the disclosure of the present invention more thorough and comprehensive.

[0024] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by those skilled in the art to which the present invention belongs. The terms used herein in the specification of the present invention are only for the purpose of describing specific embodiments and are not intended to limit the present invention. The term "and / or" used herein includes any and all combinations of one or more of the related listed items.

[0025] First, a brief introduction is given to the terms involved in the embodiments of the present application.

[0026] Multimodal data refers to data that contains multiple different modes or types, and each mode can be a different type of data, such as text, image, audio, video, etc.; heterogeneous data refers to a collection of data with different types or structures; multimodal heterogeneous data is diverse, correlated and complex. In the intelligent robot interactive operating system, processing multimodal heterogeneous data can enhance the robot's perception and understanding capabilities, enabling it to better adapt to complex and changing environments.

[0027] User portrait is a model built based on the multi-dimensional characteristics of users. It is used to describe and predict user behaviors, preferences and needs. It collects and analyzes multi-source data such as user personal information, behavioral data, interaction records, etc., and uses machine learning and data analysis techniques to extract and model user characteristics.

[0028] Long Short-Term Memory (LSTM) is a special type of recurrent neural network (RNN) that is mainly used to process and predict time series data. When processing user historical behavior sequence data, it can effectively capture long-distance dependencies by introducing special gating mechanisms, including forget gate, input gate, and output gate. The forget gate determines which past information to keep or discard; the input gate controls the input of new information; and the output gate determines the output content, so that LSTM can remember important information from a long time ago, and can also update the state according to the current input, thereby well capturing long-term dependencies and short-term change information in the historical behavior sequence.

[0029] According to the above explanation of terms, the implementation environment of a robot operating system supporting multimodal interaction provided in an embodiment of the present application is explained. Schematically, the implementation environment includes: sensor devices, processors, storage devices and output devices. Among them, the sensor can be a visual sensor, an auditory sensor, a tactile sensor or a semantic data acquisition device, etc.; the visual sensor includes but is not limited to an optical camera, a depth camera and an infrared camera, etc.; the auditory sensor can be a multi-microphone array; the tactile sensor can be a pressure sensor, a strain sensor, a capacitive tactile sensor, etc., distributed in the operating part of the robot; the semantic data acquisition device can receive from an external device (such as a user's mobile device, computer, etc.) through a network interface, or be equipped with a special input device, such as a keyboard, a touch screen, etc., so that the user can enter text information, which is not limited here.

[0030] In combination with the above-mentioned explanations of terms and implementation environment, the application scenarios of the embodiments of the present application are explained.

[0031] The robot operating system supporting multimodal interaction provided in the embodiments of the present application can be applied to the following scenarios including but not limited to: In the multimodal interactive robot system of smart home, the system can be used to collect visual, auditory, tactile and semantic information in the room, build a user portrait for the user, and automatically adjust the operating instructions of smart furniture according to the user's appliance usage habits and preferences. For example, it can automatically adjust the brightness of the lights according to the user's habits, turn on the appliances in advance, etc., to enhance the user experience.

[0032] In smart tour guide applications, the system can be used to provide tour guide services to tourists. Robots equipped with the system can identify exhibits, crowds, and environmental layouts, and receive questions from tourists for interaction. At the same time, user profiles of tourists are established based on multimodal heterogeneous data. For example, if it is found that tourists are more interested in historical exhibits, stay longer at certain exhibits, or often ask specific types of questions, this information will be incorporated into the user profile. The deep learning model will be used to analyze the user profile of tourists, and operational instructions will be issued based on the strategies derived from the analysis to provide personalized services for tourists.

[0033] Illustratively, the robot operating system supporting multimodal interaction provided in the embodiments of the present application can also be applied to other application scenarios, which are only illustrated here and are not limited to specific application scenarios.

[0034] In an exemplary embodiment, Figure 1 As shown, a robot operating system 10 supporting multimodal interaction is provided. The system is applied to the aforementioned processor as an example for explanation. It can be understood that the system can also be implemented through the interaction between the processor mounted on the robot and other processors / controllers / servers. In this embodiment, the system includes the following data receiving module 11, target object recognition module 12, analysis and decision module 13 and response module 14. Among them: The data receiving module 11 is used to pre-process the acquired multimodal heterogeneous data to obtain sensory data; wherein the multimodal heterogeneous data includes at least one of visual data, auditory data, tactile data and semantic data.

[0035] Exemplarily, the system can receive data in multiple modalities including visual data, auditory data, tactile data, and semantic data, which enables the system to obtain information from different angles, thereby more comprehensively understanding the user and the environment he or she is in. For example, not only can the user's appearance and movements be seen through visual data, but the user's voice and environmental sounds can also be heard through auditory data, and the user's operating force can be sensed through tactile data, etc., providing a rich information basis for subsequent analysis.

[0036] The target object recognition module 12 is used to establish a corresponding user portrait for the target object according to the sensory data.

[0037] For example, building a user portrait based on multimodal sensory data can provide personalized services for users. By analyzing various aspects of user information, such as the user's appearance, voice characteristics, operating habits, etc., a detailed user portrait can be built to better understand the user's characteristics and needs, making the interaction between the system and the user more in line with the user's actual situation.

[0038] The analysis and decision-making module 13 is used to perform preference analysis according to the user portrait and derive a response strategy based on a deep learning model.

[0039] Exemplarily, a deep learning model is used to perform preference analysis and derive response strategies. Deep learning has powerful feature learning and representation capabilities, and can automatically mine deep user preference information from user portraits. The deep learning model can generate highly adaptable response strategies based on different user portraits and a large amount of historical data.

[0040] The response module 14 is used to issue an operation instruction based on the response strategy.

[0041] For example, various operation instructions can be issued according to different response strategies to meet the needs of different users in different scenarios. For example, according to the user's preferences, operation instructions such as playing the user's favorite music, recommending the user's interested products, and providing the user's required information may be issued to achieve the diversification of system functions. Different output mode combinations can also be selected according to different response strategies to enhance the system's adaptability to complex environments and complex user behaviors.

[0042] The above-mentioned robot operating system supporting multimodal interaction integrates a data receiving module, a target object recognition module, an analysis and decision module, and a response module. The data receiving module converts the acquired multimodal heterogeneous data into sensory data through preprocessing. The multimodal heterogeneous data includes at least one of visual data, auditory data, tactile data and semantic data. The target object recognition module establishes a corresponding user portrait for the target object based on these sensory data. The analysis and decision module uses a deep learning model to perform preference analysis on the user portrait and derives a response strategy based on this. The response module issues operation instructions based on the response strategy determined by the analysis and decision module, so that the system can respond to user needs quickly and effectively, improve the response speed and efficiency of the system, and meet the complex and diverse personalized needs of users.

[0043] In one of the embodiments, the target object identification module 12 may also include a multimodal data fusion unit, a user profiling unit and a user classification unit.

[0044] The multimodal data fusion unit is used to extract features of the sensory data based on the multimodal data fusion model, where the features include at least one of age, gender, language habits, knowledge level, and whether or not the person is disabled.

[0045] Specifically, this system collects multimodal data such as vision, hearing, and touch, and uses a multimodal data fusion model to extract features from the integrated data. For example, it uses face recognition technology to obtain age and gender information from visual data; uses natural language processing technology to analyze voice data to identify language habits and knowledge levels; and uses interactive behavior and image recognition to comprehensively analyze whether a person is disabled. The multimodal data fusion unit extracts more comprehensive and accurate features by integrating visual, auditory and other information, thereby building a more accurate user portrait.

[0046] The user portrait unit is used to create a user portrait based on sensory data from the same user, and store and manage the user portrait based on voiceprint features as an index.

[0047] Specifically, various sensory data features from the same user are summarized to construct a user portrait, and the user's voiceprint features are extracted using voiceprint recognition technology and used as index identifiers. Each user portrait is associated with the corresponding voiceprint features and stored in the database to ensure that each user's portrait is unique and traceable.

[0048] The user classification unit is used to classify group characteristics based on user portraits and to train deep learning models.

[0049] Specifically, we conduct in-depth analysis of various features in user portraits, explore the commonalities and differences between different users, and classify users into different groups based on the analysis results. We use the classified user group data as a training set to train deep learning models. By continuously adjusting model parameters, the model can accurately identify the characteristics of different groups, improving the training efficiency and accuracy of the model.

[0050] In one embodiment, the analysis and decision module 13 may further include an input submodule and an output submodule.

[0051] The input submodule is used to obtain the user's historical behavior sequence data, assign weights to each data point through the attention mechanism and input it into the long short-term memory network, where the attention energy The calculation formula is as follows:

[0052] in, is the attention energy, is the number of time steps of the historical behavior sequence data, is the attention vector, is the weight matrix, is the bias vector, is the i-th input data, is the hyperbolic tangent activation function.

[0053] Focus energy Convert to attention weight , as shown in the formula:

[0054] in, is the attention weight, exp is the exponential function, is the sum of the exponential operations on the attention energy at all time steps; Use the following formula to multiply the input sequence by the attention weight to get the weighted input sequence. :

[0055] in, is the t-th input after attention weighting, is the attention weight of the ith input at the tth time step, is the i-th input data.

[0056] Specifically, by assigning weights to each data point through the attention mechanism, the importance of each data point in the historical behavior sequence data can be effectively distinguished. In practical applications, different behaviors of users have different impacts on their current preferences and needs. The attention mechanism can automatically learn these differences, allowing the model to pay more attention to important data points when performing preference analysis, thereby improving the accuracy and effectiveness of the analysis.

[0057] The output submodule is used to: The preference representation is calculated based on the weighted input sequence using the following formula :

[0058] in, is the weight matrix of the fully connected layer, is the sum of the weighted input sequence, is the rectified linear unit activation function, is the bias vector of the fully connected layer; Generate a response strategy based on the preference representation using the following formula :

[0059] in, is the preference representation, is the weight matrix of the fully connected layer, is the bias vector, is the activation function.

[0060] For example, the fully connected layer is used in the preference representation calculation and response strategy generation process, so that the model has high flexibility and scalability. The fully connected layer can adapt to different tasks and data characteristics by adjusting the weight matrix and bias vector. At the same time, it is also convenient to further optimize and improve the model, such as adding more hidden layers, adjusting the network structure, etc., to improve the performance and adaptability of the model.

[0061] The output submodule generates a user preference representation by processing the weighted input sequence, and generates a response strategy based on this. It can generate a highly personalized response strategy based on the unique historical behavior of each user. This system can provide users with product recommendations and service suggestions that better meet their needs based on their preferences, greatly improving user experience and satisfaction and enhancing the competitiveness of the system.

[0062] In one embodiment, the response module 14 may include: The real-time situational awareness unit is used to identify the current interaction situation based on sensory data, user profile and operation behavior to formulate a corresponding output method; among which, the user profile includes preference information and emotional state.

[0063] Exemplarily, this system extracts environmental scene information based on sensory data, and identifies specific interaction scenarios in combination with user portraits and operation behaviors to develop appropriate output methods. The output methods can be a combination of multiple output methods, such as voice, vibration feedback, projection, or image text, etc., which can transmit information from different dimensions, meet the diverse reception needs of users, enhance user participation and satisfaction, and adapt to different usage scenarios. This application does not limit this. The source of preference information in the user portrait can be the user's registration information, historical interaction records, questionnaire feedback, etc.; the source of emotional state can be analyzed from the user's tone, voice, and words.

[0064] The real-time situation awareness unit also includes: The preference matching subunit is used to establish a mapping relationship between preference information and output mode based on a machine learning algorithm.

[0065] For example, according to the characteristics of the data and the nature of the problem, select a suitable machine learning algorithm, such as a decision tree, support vector machine (SVM) or neural network. Use the sorted preference information and output mode data to train the model, and by continuously adjusting the model parameters, the model can accurately learn the mapping relationship between the preference information and the output mode. During the training process, use methods such as cross-validation to evaluate the performance of the model to ensure the accuracy and generalization ability of the model.

[0066] The dynamic decision subunit is used to continuously monitor the interaction scenario and emotional state and extract feature vectors during the interaction process. When the Euclidean distance between the feature vector of the interaction scenario and / or emotional state within a preset time period and the feature vector at the previous moment exceeds a preset threshold, the output method is re-formulated.

[0067] Exemplarily, during the interaction process, the user's sensory data, user portrait data, and operation behavior data are continuously collected, and the corresponding feature vectors are extracted for the interaction scenario and emotional state. For the interaction scenario, environmental features (such as light intensity, noise level), device status (battery, network connection status) and other features can be extracted; for the emotional state, facial expression features (such as the angle of the corner of the mouth, changes in eyes), voice features (speech speed, intonation, volume changes), etc. can be extracted through facial expression analysis, voice emotion analysis and other technologies. According to the preset time period, the feature vectors of the current interaction scenario and emotional state are regularly extracted and compared with the feature vectors of the previous moment. The Euclidean distance between the two is calculated. When the Euclidean distance exceeds the preset threshold, it indicates that the interaction scenario and / or emotional state has changed significantly. At this time, according to the new interaction scenario and the user's emotional state, combined with the mapping relationship between preference information and output mode, the output mode is re-formulated. For example, if it is detected that the user's emotion changes from calm to excited, and the current ambient light is dim, according to the mapping relationship, the output mode may be adjusted from text display to voice output, and the voice volume may be appropriately increased.

[0068] Furthermore, a robot operating system supporting multimodal interaction provided in an embodiment of the present application may also include a group user portrait module and an interactive resonance module.

[0069] The group user portrait module is used to extract representative features from multiple user portraits, and perform cluster analysis based on the representative features to obtain multiple groups and corresponding group user portraits; the representative features include at least one of self-identity, geographical distribution, age and education level.

[0070] Specifically, a large amount of individual user portrait data is extracted from the existing user database of the system. These data cover multiple aspects such as the user's basic information, preference information, and behavior data. For each user portrait, its representative features are extracted. For example, the self-identification tendency of the user is analyzed from the user's social interaction data, self-evaluation and other information; the regional distribution information is obtained from the registration information or positioning data; the age information is obtained from the age field; and the education level information is extracted from the education background related information. For some fuzzy or missing data, data cleaning and supplementary algorithms can be used to process them to ensure the validity of the features. Clustering algorithms, such as the K-Means clustering algorithm, are used to divide user portraits with similar representative features into the same category. For each cluster group, the characteristics of its members are counted and summarized. The average age, main regional distribution, concentrated education level, and common self-identification type of the group are calculated, and the common preference information and behavior patterns of users in the group can be further analyzed to construct a detailed group user portrait.

[0071] The interactive resonance module is used to conduct data mining on group user portraits to obtain personalized interaction strategies for each group. The personalized interaction strategies are used to indicate operation instructions.

[0072] Specifically, by analyzing the behavioral patterns, preference information, and interaction history of group users with the system, we find out the interaction methods and strategies that can resonate with the group and use them to indicate operational instructions.

[0073] Furthermore, a robot operating system supporting multimodal interaction provided in an embodiment of the present application may also include an index module, a cache module and an interest update module.

[0074] The index module is used to perform hash operations on the voiceprint features to obtain a hash value used to determine the storage location.

[0075] Specifically, voiceprint features are extracted from the user's voice data using voiceprint recognition technology, such as the Mel-frequency cepstral coefficient (MFCC) algorithm. After preprocessing, framing, windowing and other operations on the voice signal, a representative voiceprint feature vector is calculated. The extracted voiceprint feature vector is hashed to convert the voiceprint feature vector into a hash value of a fixed length. The hash value is unique and deterministic and can uniquely identify the corresponding voiceprint feature. The storage location in the storage system is determined based on the hash value. For example, the address space of the storage system can be modulo-operated using the hash value, and the result is used as the index of the storage address to determine the storage location of the voiceprint feature-related data. The storage location can be quickly located using the hash value, which can respond quickly and improve the data retrieval speed.

[0076] The cache module is used to use the hash value of the voiceprint feature as the key and the user's interest information as the value to store in the cache, where the interest information is information whose access times exceed a preset threshold within a preset time period.

[0077] Specifically, after obtaining the user's voiceprint feature hash value and the corresponding interest information, the key-value pair is stored in the cache using the API provided by the cache system. According to the cache capacity and the frequency of data access, an appropriate cache replacement strategy is adopted, such as the least recently used (LRU), most recently used (LFU), etc. When the cache is full and new data needs to be stored, some old data is eliminated according to the replacement strategy to ensure the efficient operation of the cache, and the commonly used user interest information is stored in the cache. When the system needs to obtain this information, it can be read directly from the cache, avoiding reading data from slower storage media (such as hard disks), improving data access speed, improving user experience, reducing system load, and improving system performance and stability.

[0078] The interest update module is used to perform cluster analysis on interest information and add corresponding interest tags. Based on the interest tags, the web crawler technology is used to regularly crawl interest-related information from the selected information source, and the interest-related information is used to update the interest information.

[0079] Specifically, cluster analysis is performed regularly on user interest information, and clustering algorithms are used to classify similar interest information into the same category and label them with interest tags. Based on the user's interest tags, appropriate information sources are selected, such as relevant news websites, professional forums, social media accounts, etc., and web crawler technology is used to crawl interest-related information from the selected information sources according to the set rules and frequency. The crawled new information is compared and updated with the interest information in the cache. For newly emerging interest-related information, it is added to the cache, and the corresponding interest tags and access times are updated; for existing interest information, if there is updated content, it is updated to ensure that the user's interest information is always up to date and accurate. Continuously updating interest information and labeling accurate interest tags helps the system understand the user's interest preferences more accurately, thereby providing users with content and services that are more in line with their interests, and improving the quality and satisfaction of user interaction with the system.

[0080] Preferably, the response module 14 may further include: an active interaction unit and a feedback receiving unit; The active interaction unit is used to start the active interaction program.

[0081] The feedback receiving unit is used to obtain multimodal heterogeneous data again and calculate the satisfaction of the target object based on artificial intelligence after issuing an operation instruction. If the satisfaction is lower than the threshold, the active interaction unit is called to start the active interaction program.

[0082] Specifically, after the system issues an operation instruction, it collects multimodal heterogeneous data again through a variety of sensors and data interfaces. For example, it uses a camera to capture the changes in the user's facial expressions after receiving the operation instruction, and uses a microphone to record the evaluation voice that the user may issue. Using artificial intelligence technology, it collects a large amount of user feedback data and corresponding actual satisfaction annotations (which can be obtained through user active evaluation, questionnaires, etc.), trains the model, and inputs the collected multimodal heterogeneous data into the constructed satisfaction calculation model. The satisfaction score of the target object is calculated in real time. A satisfaction threshold is preset, and the calculated satisfaction score is compared with the threshold. If the satisfaction is lower than the threshold, it means that the user may be dissatisfied with the execution result of the system's operation instruction. At this time, the feedback receiving module immediately calls the active interaction module and starts the active interaction program. After receiving the call signal of the feedback receiving module, the active interaction module selects a suitable active interaction strategy according to the current situation and starts the corresponding interaction program. By collecting multimodal data in real time and calculating user satisfaction, the system can comprehensively and accurately monitor the user's feedback on the operation instruction. When it is detected that the user's satisfaction is low, the active interaction module can be automatically called to intervene, so that the system can adjust the strategy in time according to the user's actual feedback, improve the system's adaptability and flexibility, and meet the user's needs in time.

[0083] To summarize, the robot operating system supporting multimodal interaction provided in the embodiment of the present application is responsible for collecting multimodal heterogeneous data such as vision, hearing, touch and semantics through the data receiving module, and preprocessing it into sensory data; the target object recognition module extracts the characteristics of the sensory data, such as age, gender, etc., based on the multimodal data fusion unit; the user portrait is established through the user portrait unit and managed with the voiceprint feature index, and the user classification unit classifies the group characteristics based on the portrait to provide data for deep learning model training; the input submodule of the analysis and decision-making module uses the attention mechanism to process the user's historical behavior sequence data and then inputs it into the long short-term memory network, and the output submodule calculates the preference representation based on this and generates a response strategy. In addition to issuing operation instructions based on the response strategy, the response module can also include a real-time situation perception unit, establish a mapping between preferences and output methods through a preference matching subunit, and a dynamic decision subunit monitors situation and emotion changes and adjusts the output method in a timely manner; the system also has a group user portrait module and an interactive resonance module, which obtains a group portrait through cluster analysis of multiple user portraits, and mines personalized interaction strategies based on this; the index module, cache module and interest update module work together to perform hash operations on voiceprint features and cache user interest information, and regularly update user interest information through web crawler technology; the feedback receiving unit obtains multimodal data to calculate user satisfaction after issuing instructions, and calls the active interaction unit to start the interaction program if it does not meet the standards. The above scheme, through the integrated operation of various modules, formulates a variety of interaction strategies according to different application scenarios and user group characteristics, can respond to users' personalized needs quickly and effectively, improve the response speed and efficiency of the system, and meet users' complex and diverse personalized needs.

[0084] It should be understood that, although the various steps in the flowcharts involved in the above-mentioned embodiments are displayed in sequence according to the indication of the arrows, these steps are not necessarily executed in sequence according to the order indicated by the arrows. Unless there is a clear explanation in this article, the execution of these steps does not have a strict order restriction, and these steps can be executed in other orders. Moreover, at least a part of the steps in the flowcharts involved in the above-mentioned embodiments can include multiple steps or multiple stages, and these steps or stages are not necessarily executed at the same time, but can be executed at different times, and the execution order of these steps or stages is not necessarily to be carried out in sequence, but can be executed in turn or alternately with other steps or at least a part of the steps or stages in other steps.

[0085] Based on the same inventive concept, the embodiment of the present application also provides a method for implementing the above-mentioned robot operation method supporting multimodal interaction. The implementation scheme for solving the problem provided by the method is similar to the implementation scheme recorded in the above-mentioned method, so the specific limitations in the embodiments of one or more robot operation methods supporting multimodal interaction provided below can be referred to the limitations of the robot operation system supporting multimodal interaction above, and will not be repeated here.

[0086] In an exemplary embodiment, Figure 2 As shown, a robot operation method supporting multimodal interaction is provided, comprising the following steps: Step 201, acquiring multimodal heterogeneous data and preprocessing the multimodal heterogeneous data to obtain sensory data; wherein the multimodal heterogeneous data includes at least one of visual data, auditory data, tactile data and semantic data; Step 202, establishing a corresponding user portrait for the target object based on the sensory data; Step 203, performing preference analysis based on the user portrait, and deriving a response strategy based on a deep learning model; Step 204: Issue an operation instruction based on the response strategy.

[0087] In one embodiment, step 202 of the method may further include the following sub-steps: Features of sensory data are extracted based on a multimodal data fusion model, where the features include at least one of age, gender, language habits, knowledge level, and disability.

[0088] A user portrait is created based on sensory data from the same user, and the voiceprint features are used as an index to store and manage the user portrait.

[0089] Group characteristics are classified according to user portraits for training deep learning models.

[0090] In one embodiment, step 203 of the method may further include the following sub-steps: Obtain the user's historical behavior sequence data, assign weights to each data point through the attention mechanism and input it into the long short-term memory network, where the attention energy The calculation formula is as follows:

[0091] in, is the attention energy, is the number of time steps of the historical behavior sequence data, is the attention vector, is the weight matrix, is the bias vector, is the i-th input data, is the hyperbolic tangent activation function.

[0092] Focus energy Convert to attention weight , as shown in the formula:

[0093] in, is the attention weight, exp is the exponential function, is the sum of the exponential operations on the attention energy at all time steps; The weighted input sequence is obtained by multiplying the input sequence with the attention weight using the following formula to obtain the weighted input sequence:

[0094] in, is the t-th input after attention weighting, is the attention weight of the ith input at the tth time step, is the i-th input data.

[0095] The preference representation is calculated based on the weighted input sequence using the following formula :

[0096] in, is the weight matrix of the fully connected layer, is the sum of the weighted input sequence, is the rectified linear unit activation function, is the bias vector of the fully connected layer; Generate a response strategy based on the preference representation using the following formula :

[0097] in, is the preference representation, is the weight matrix of the fully connected layer, is the bias vector, is the activation function.

[0098] Furthermore, step 204 of the method may also include the following sub-steps: Identify the current interaction scenario based on sensory data, user portraits, and operational behaviors to develop corresponding output methods; among them, user portraits include preference information and emotional states.

[0099] The above step of identifying the current interaction scenario based on sensory data, user portrait and operation behavior to formulate a corresponding output method; wherein the user portrait includes preference information and emotional state may also include the following sub-steps: A mapping relationship between preference information and output methods is established based on machine learning algorithms.

[0100] During the interaction process, the interaction scenario and emotional state are continuously monitored and feature vectors are extracted. When the Euclidean distance between the feature vector of the interaction scenario and / or emotional state within a preset period and the feature vector at the previous moment exceeds a preset threshold, the output method is re-formulated.

[0101] Preferably, the method may further comprise the following steps: Extract representative features from multiple user portraits, and perform cluster analysis based on the representative features to obtain multiple groups and corresponding group user portraits; the representative features include at least one of self-identity, geographical distribution, age and education level; Data mining is performed on group user portraits to obtain personalized interaction strategies for each group. Personalized interaction strategies are used to indicate operation instructions.

[0102] Preferably, the method may further comprise the following steps: Perform a hash operation on the voiceprint feature to obtain a hash value used to determine the storage location.

[0103] The hash value of the voiceprint feature is used as the key, and the user's interest information is stored in the cache as the value, where the interest information is the information that has been accessed more than a preset threshold within a preset period of time; The interest information is clustered and tagged with corresponding interest tags. Based on the interest tags, web crawler technology is used to regularly crawl interest-related information from selected information sources, and the interest-related information is used to update the interest information.

[0104] Preferably, step 204 of the method may further include the following steps: Start active interaction program; After issuing the operation instruction, multimodal heterogeneous data is obtained again and the satisfaction of the target object is calculated based on artificial intelligence. If the satisfaction is lower than the threshold, the active interaction unit is called to start the active interaction program.

[0105] In one embodiment, a computer device is provided, including a memory and a processor, wherein the memory stores a computer program, and the processor implements the steps of the method in each of the above embodiments when executing the computer program.

[0106] In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the steps of the method in the above embodiments are implemented.

[0107] The technical features of the above embodiments can be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0108] Those skilled in the art will appreciate that the logic and / or steps represented in the flowchart or otherwise described herein, for example, may be considered as an ordered list of executable instructions for implementing logical functions, and may be specifically implemented in any computer-readable storage medium for use by an instruction execution system, device or apparatus (such as a computer-based system, a system including a processor, or other system that can fetch instructions from an instruction execution system, device or apparatus and execute instructions), or in conjunction with such instruction execution systems, devices or apparatuses. For purposes of this specification, a "computer-readable storage medium" may be any device that can contain, store, communicate, propagate or transmit a program for use by an instruction execution system, device or apparatus, or in conjunction with such instruction execution systems, devices or apparatuses.

[0109] More specific examples (a non-exhaustive list) of computer-readable storage media include the following: an electrical connection with one or more wires (electronic device), a portable computer disk case (magnetic device), a random access memory (RAM), a read-only memory (ROM), an erasable and programmable read-only memory (EPROM or flash memory), an optical fiber device, and a portable compact disk read-only memory (CDROM). In addition, the computer-readable storage medium may even be a paper or other suitable medium on which the program is printed, since the program may be obtained electronically, for example, by optically scanning the paper or other medium, followed by editing, deciphering or, if necessary, processing in another suitable manner, and then stored in a computer memory.

[0110] It should be understood that the various parts of the present invention can be implemented by hardware, software, firmware or a combination thereof. In the above-mentioned embodiments, multiple steps or methods can be implemented by software or firmware stored in a memory and executed by a suitable instruction execution system. For example, if implemented by hardware, as in another embodiment, it can be implemented by any one of the following technologies known in the art or a combination thereof: a discrete logic circuit having a logic gate circuit for implementing a logic function for a data signal, a dedicated integrated circuit having a suitable combination of logic gate circuits, a programmable gate array (PGA), a field programmable gate array (FPGA), etc.

[0111] In the description of this specification, the description with reference to the terms "one embodiment", "some embodiments", "examples", "specific examples", or "some examples" means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present invention. In this specification, the schematic representation of the above terms does not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described may be combined in any one or more embodiments or examples in a suitable manner.

[0112] The above-mentioned embodiments only express several implementation methods of the present invention, and the description thereof is relatively specific and detailed, but it cannot be understood as limiting the scope of the patent of the present invention. It should be pointed out that, for ordinary technicians in this field, several variations and improvements can be made without departing from the concept of the present invention, which all belong to the protection scope of the present invention. Therefore, the protection scope of the patent of the present invention shall be subject to the attached claims.

Claims

1. A robot operating system supporting multimodal interaction, characterized in that: The system comprises: Data receiving module, target object recognition module, analysis and decision module and response module; The data receiving module is used to pre-process the acquired multimodal heterogeneous data to obtain sensory data; wherein the multimodal heterogeneous data includes at least one of visual data, auditory data, tactile data and semantic data; The target object recognition module is used to establish a corresponding user portrait for the target object according to the sensory data; The analysis and decision-making module is used to perform preference analysis according to the user portrait and derive a response strategy based on a deep learning model; The response module is used to issue an operation instruction based on the response strategy.

2. The robot operating system supporting multimodal interaction according to claim 1, characterized in that: The target object recognition module includes a multimodal data fusion unit, a user profiling unit and a user classification unit; The multimodal data fusion unit is used to extract features of the sensory data based on a multimodal data fusion model, wherein the features include at least one of age, gender, language habits, knowledge level, and disability; The user portrait unit is used to establish the user portrait according to the sensory data from the same user, and store and manage the user portrait according to the voiceprint feature as an index; The user classification unit is used to perform group characteristic classification according to the user portrait for training the deep learning model.

3. The robot operating system supporting multimodal interaction according to claim 2, characterized in that: The analysis and decision module includes an input submodule and an output submodule: The input submodule is used to obtain the user's historical behavior sequence data, assign weights to each data point through the attention mechanism and input them into the long short-term memory network, where the attention energy The calculation formula is as follows: in, is the attention energy, is the number of time steps of the historical behavior sequence data, is the attention vector, is the weight matrix, is the bias vector, is the i-th input data, is the hyperbolic tangent activation function; Use the attention conversion formula to convert attention energy Convert to attention weight , where the attention conversion formula is as follows: in, is the attention weight, exp is the exponential function, is the sum of the exponential operations on the attention energy at all time steps; Multiply the input sequence by the attention weight to get the weighted input sequence, where the calculation formula of the weighted input sequence is as follows: in, is the t-th input after attention weighting, is the attention weight of the ith input at the tth time step, is the i-th input data; The output submodule is used for: Compute a preference representation based on the weighted input sequence , where preference represents The calculation formula is as follows: in, is the weight matrix of the fully connected layer, is the sum of the weighted input sequence, is the rectified linear unit activation function, is the bias vector of the fully connected layer; Generating response strategies based on preference representation , where the response strategy The calculation formula is as follows: in, is the preference representation, is the weight matrix of the fully connected layer, is the bias vector, is the activation function.

4. The robot operating system supporting multimodal interaction according to claim 2, characterized in that: The response module comprises: A real-time situation perception unit, used to identify the current interaction situation based on the sensory data, the user portrait and the operation behavior, so as to formulate a corresponding output mode; wherein the user portrait includes preference information and emotional state; The real-time situation awareness unit also includes: A preference matching subunit, used to establish a mapping relationship between the preference information and the output mode based on a machine learning algorithm; The dynamic decision subunit is used to continuously monitor the interaction scenario and the emotional state and extract feature vectors during the interaction process, and to reformulate the output mode when the Euclidean distance between the feature vector of the interaction scenario and / or the emotional state within a preset time period and the feature vector at the previous moment exceeds a preset threshold.

5. The robot operating system supporting multimodal interaction according to claim 2, characterized in that: The system also includes a group user portrait module and an interactive resonance module: The group user portrait module is used to extract representative features from the multiple user portraits, and perform cluster analysis based on the representative features to obtain multiple groups and corresponding group user portraits; the representative features include at least one of self-identification, geographical distribution, age and education level; The interactive resonance module is used to perform data mining on the group user portraits to obtain personalized interaction strategies for each of the groups, and the personalized interaction strategies are used to indicate the operation instructions.

6. The robot operating system supporting multimodal interaction according to claim 2, characterized in that: The system also includes an index module, a cache module and an interest update module: The index module is used to perform a hash operation on the voiceprint feature to obtain a hash value for determining a storage location; The cache module is used to store the hash value of the voiceprint feature as a key and the user's interest information as a value in the cache, wherein the interest information is information whose access times exceed a preset threshold within a preset period of time; The interest updating module is used to perform cluster analysis on the interest information and add corresponding interest tags, and use web crawler technology to regularly crawl interest-related information from selected information sources based on the interest tags, and the interest-related information is used to update the interest information.

7. The robot operating system supporting multimodal interaction according to claim 1, characterized in that: The response module also includes: an active interaction unit and a feedback receiving unit; The active interaction unit is used to start the active interaction program; The feedback receiving unit is used to obtain the multimodal heterogeneous data again and calculate the satisfaction of the target object based on artificial intelligence after issuing an operation instruction. If the satisfaction is lower than a threshold, the active interaction unit is called to start the active interaction program.

8. A method for operating a robot supporting multimodal interaction, characterized in that: include: Acquire multimodal heterogeneous data and preprocess the multimodal heterogeneous data to obtain sensory data; wherein the multimodal heterogeneous data includes at least one of visual data, auditory data, tactile data and semantic data; Establishing a corresponding user portrait for the target object according to the sensory data; Perform preference analysis based on the user portrait and derive a response strategy based on a deep learning model; An operation instruction is issued based on the response strategy.

9. A computer device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the method according to claim 8 is implemented.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the method according to claim 8 is implemented.

Images