A multi-modal data processing method and system for a robot

Abstract

The present application relates to the field of data processing, in particular to a kind of multimodal data processing method and system for robot, comprising the multimodal data that robot end side sensor gathers, respectively with different algorithm processing and store feature data, using unified packaging format, lightweight compression and quantization coding, also carry out storage access optimization;System is deployed in robot end side, contain multimodal data acquisition, special storage, interactive decision, execution, storage management, performance monitoring and fault-tolerant module.This application achieves the technical effect that can efficiently process multimodal data, realize robot wake-up free active interaction, optimize storage access, ensure data integrity and system performance.

Description

Technical Field

[0001] This invention relates to the field of data processing technology, and in particular to a multimodal data processing method and system for robots. Background Technology

[0002] With the continuous advancement of technology, robotics has developed rapidly, playing an increasingly important role in numerous fields such as service, education, and entertainment. Robots can interact with humans, providing various conveniences and services, greatly improving the efficiency of life and work. In particular, robots with multimodal data analysis and interaction capabilities can more accurately understand user intentions and provide personalized services, which is of great significance for improving the human-computer interaction experience and expanding the application scenarios of robots. Multimodal data analysis can comprehensively consider various information, such as facial features, facial expressions, and directional perception, enabling robots to more comprehensively understand the user's state, thereby achieving more intelligent interaction.

[0003] In existing technologies, various methods are typically employed to enable robots to understand user intentions and interact. For facial recognition, the common approach is to first perform simple preprocessing on the acquired facial images to remove noise and interference, then input the preprocessed image into a general facial recognition algorithm to extract facial features, and finally store the extracted features in a standard data storage area. However, this storage method lacks specificity and efficiency. For expression analysis, facial images are analyzed to extract some simple expression features, which are then directly stored in a standard storage area without considering the differences and characteristics of different facial expression regions. For orientation perception, directional information is acquired through sensors and then simply recorded and stored without effective processing or optimized storage. These conventional methods can enable robots to analyze and understand some user information to a certain extent, but they are significantly insufficient in multimodal data processing.

[0004] However, existing technologies have a significant drawback: the memory cache capacity cannot meet the concurrent storage requirements of face recognition algorithms, expression analysis algorithms, and orientation perception algorithms. When processing multimodal data simultaneously, this leads to multimodal data processing delays, which seriously affect the real-time performance and accuracy of robot-user interaction. As a result, the robot cannot respond to users in a timely and accurate manner, reducing the quality and efficiency of human-computer interaction. Summary of the Invention

[0005] This invention solves the problem of latency in multimodal data processing and proposes a multimodal data processing method and system for robots.

[0006] To achieve the above objectives, the following technical solution is proposed: A multimodal data processing method for robots includes: S1, acquiring multimodal data collected by sensors on the robot's end-user side, the multimodal data including face image data, expression image data, and orientation perception data; S2, processing the face image data based on a face recognition algorithm, and storing the extracted face feature data in a dedicated face recognition storage module; S3, processing the expression image data based on an expression analysis algorithm, and storing the extracted expression feature data hierarchically according to the influence degree of the action area in a dedicated expression analysis storage module; S4, processing the orientation perception data based on an orientation perception algorithm, and storing the extracted spatial location feature data according to coordinate direction in a dedicated orientation perception storage module; S5, generating user tags, intentions, and spatial positioning information based on the concurrent access results of each dedicated storage module, and generating differentiated interaction commands based on the user tags, intentions, and spatial positioning information to control the robot to perform wake-up-free active interaction.

[0007] By adopting the above technical solution, multimodal data collected by the robot's end-side sensors can be acquired. Face, expression, and orientation perception data are processed and stored in corresponding dedicated storage modules. User tags, intentions, and spatial positioning information are generated based on the concurrent access results of each storage module, thereby generating differentiated interaction commands and realizing the robot's wake-up-free active interaction. This solves the problem of multimodal data processing delay caused by the insufficient memory cache capacity to meet the concurrent storage requirements of multiple algorithms.

[0008] Preferably, storing the extracted facial feature data in a dedicated facial recognition storage module includes: encapsulating the facial image features into a preset data structure using a feature encapsulation data structure before storage, wherein the preset data structure includes feature extraction algorithm model identification information, custom attribute information, and verification information; storing the extracted facial expression feature data hierarchically in a dedicated facial expression analysis storage module includes: dividing the facial expression features into a first action region and a second action region, wherein the data of the first action region is stored in a high-speed cache, and the data of the second action region is stored in a standard cache; storing the extracted spatial location feature data according to coordinate directions in a dedicated direction-aware storage module includes: pooling the features of the spatial location information along the X and Y directions to generate attention maps, and storing them according to a preset feature map format.

[0009] By adopting the above technical solutions, facial image features are encapsulated into a preset data structure containing feature extraction algorithm model identification information, custom attribute information, and verification information. This avoids repeatedly storing feature data from different versions of the model and facilitates feature data recognition. Facial expression features are stored hierarchically according to the degree of influence of the action region. Data from the first action region is cached at high speed, while data from the second action region is cached at a standard speed, which improves the storage and access efficiency of facial expression feature data. Spatial location information is pooled along the X and Y directions to generate attention maps and stored in a preset feature map format, which helps to store and analyze orientation-aware data more accurately.

[0010] Preferably, the method further includes a storage access optimization step: establishing an index system based on feature extraction algorithm model identifiers; when the algorithm model is upgraded, determining whether features need to be re-extracted based on the identifier information; employing a computing node mapping strategy to allocate storage requests to sub-regions of each dedicated storage module to achieve concurrent access control for face recognition algorithms, expression analysis algorithms, and orientation awareness algorithms, wherein the storage access of a single algorithm does not exceed 60% of the total bandwidth; and managing cached data using an improved cache replacement strategy based on algorithm priority and access frequency, with face recognition algorithms having the highest priority, expression analysis algorithms second, and orientation awareness algorithms having the lowest priority.

[0011] By adopting the above technical solutions, an indexing system based on feature extraction algorithm model identifiers can be established to achieve rapid retrieval of feature-encapsulated data. When the algorithm model is upgraded, the system can determine whether features need to be re-extracted based on the identifier information, avoiding full data reprocessing. By using a computing node mapping strategy to allocate storage requests to sub-regions of each dedicated storage module, concurrent access control for the three algorithms can be achieved, and the storage access of a single algorithm does not exceed 60% of the total bandwidth, ensuring the rationality of concurrent storage. An improved cache replacement strategy based on algorithm priority and access frequency manages cached data, which can effectively utilize cache space, prioritize the operation needs of high-priority algorithms, improve the efficiency of multimodal data processing, and reduce data processing latency.

[0012] Preferably, the improved cache replacement strategy is as follows: when the cache space is insufficient, priority is given to replacing low-priority data blocks with low access frequency.

[0013] By adopting the above technical solution, multimodal data collected by the robot's end-side sensors is acquired. After processing by corresponding algorithms, facial, expression, and spatial location feature data are stored in dedicated storage modules. Storage is optimized by using feature encapsulation data structures, hierarchical storage, and coordinate direction storage. An indexing system is established, and a computing node mapping strategy and an improved cache replacement strategy are used to optimize storage access. When the cache space is insufficient, low-priority and infrequently accessed data blocks are replaced first, which avoids full data reprocessing, realizes concurrent access control, optimizes cache management, effectively improves storage resource utilization, reduces multimodal data processing latency, and generates user tags, intents, and spatial positioning information based on concurrent access results to control the robot to perform wake-free active interactions.

[0014] Preferably, when storing the facial feature data, expression feature data, and spatial location feature data, a unified data encapsulation format is adopted. The format includes header information and feature package. The header information contains data type, algorithm identifier, and timestamp metadata. The feature package contains feature vector and verification information.

[0015] By adopting the above technical solution, multimodal data collected by the robot's end-side sensors is acquired, including facial images, facial expression images, and orientation perception data. Facial feature data is stored in a dedicated facial recognition storage module, facial expression feature data is stored hierarchically according to the degree of influence of the action area in a dedicated facial expression analysis storage module, and spatial location feature data is stored separately according to coordinate direction in a dedicated orientation perception storage module. User tags, intents, and spatial positioning information are generated based on concurrent access results to control the robot to perform wake-free active interaction. At the same time, a unified data encapsulation format is used to store the three types of feature data. The header information includes data type, algorithm identifier, and timestamp metadata, and the features include feature vectors and verification information, which facilitates memory page alignment and fast access. The verification information can also be used to detect data integrity.

[0016] Preferably, before storage, the feature data is subjected to lightweight compression and quantization encoding to convert floating-point features into fixed-point representations, with the compression ratio controlled at 60%-70%, and ensuring that the algorithm's recognition accuracy is not less than 95% of that before compression.

[0017] By adopting the above technical solution, multimodal data collected by the robot's end-side sensors, including face, expression, and orientation perception data, is acquired. Facial feature data is stored in a dedicated face recognition storage module, expression feature data is stored hierarchically according to the influence of the action region in a dedicated expression analysis storage module, and spatial location feature data is stored according to coordinate direction in a dedicated orientation perception storage module. User tags, intents, and spatial positioning information are generated based on the concurrent access results of each dedicated storage module to control the robot's wake-up-free active interaction. A feature encapsulation data structure is used to encapsulate facial image features, and expression features are divided according to action regions and stored in a high-speed cache and a standard cache respectively. Spatial location information is generated into an attention map according to coordinate direction features and stored. A unified data encapsulation format is used to store feature data. Lightweight compression and quantization encoding are performed on the feature data, converting floating-point features into fixed-point representations. The compression ratio is controlled at 60%-70%, ensuring that the algorithm's recognition accuracy is not less than 95% of that before compression. This solves the problem that the memory cache capacity cannot meet the concurrent storage requirements of multimodal data processing, reduces data processing latency, and reduces storage space while ensuring the algorithm's recognition accuracy.

[0018] A multimodal data processing system for robots, applicable to the multimodal data storage and interaction method for robots as described in any one of claims 1-6, the system being deployed on the robot's end-user side, comprising: a multimodal data acquisition module configured to acquire multimodal data via sensors, the multimodal data including at least facial image data, expression image data, and orientation perception data; a dedicated storage module, comprising: a facial recognition dedicated storage module configured to store facial feature data extracted by a facial recognition algorithm; an expression analysis dedicated storage module configured to hierarchically store expression feature data extracted by an expression analysis algorithm according to the degree of influence of the action region; an orientation perception dedicated storage module configured to store spatial location feature data extracted by an orientation perception algorithm according to coordinate directions; an interaction decision module configured to generate user tags, intentions, and spatial positioning information based on the concurrent access results of the dedicated storage modules, and generate differentiated interaction instructions accordingly; and an execution module configured to execute the interaction instructions to achieve wake-free active interaction of the robot.

[0019] By adopting the above technical solution, multimodal data from the robot's edge can be collected, including face, expression, and orientation perception data. The features of different types of data are stored in dedicated storage modules. User tags, intentions, and spatial positioning information are generated based on concurrent access results, thereby generating differentiated interaction commands and enabling robot wake-up-free active interaction. This effectively solves the problem of multimodal data processing delay caused by insufficient memory cache capacity to meet the concurrent storage requirements of multiple algorithms.

[0020] Preferably, the face recognition dedicated storage module uses a feature encapsulation data structure to store face feature data, the data structure including feature extraction algorithm model identification information, custom attribute information and verification information; the expression analysis dedicated storage module is configured to divide expression features into a first action region and a second action region, and store them using a high-speed cache and a standard cache respectively; the direction perception dedicated storage module is configured to integrate a coordinate attention mechanism to store attention mapping maps generated according to X and Y direction features.

[0021] By adopting the above technical solution, the dedicated face recognition storage module uses a feature encapsulation data structure that includes feature extraction algorithm model identification information, custom attribute information, and verification information to store face feature data, thus establishing a... Figure 1 The file association mechanism avoids the duplication of storing feature data from different versions of the model; the dedicated expression analysis storage module divides expression features into the first action region and the second action region and stores them using high-speed cache and standard cache respectively, which can control the access latency of different action regions; the dedicated direction perception storage module integrates a coordinate attention mechanism, stores attention maps generated by features in the X and Y directions, and can store spatial location feature data through a specific storage format.

[0022] Preferably, the system further includes a storage management module, which includes: an indexing unit configured to establish an indexing system based on feature extraction algorithm model identifiers; a concurrency control unit configured to use a computing node mapping strategy to allocate storage requests to sub-regions of each dedicated storage module to achieve concurrent access control; and a cache replacement unit configured to execute an improved cache replacement strategy based on algorithm priority and access frequency, with the face recognition algorithm having the highest priority.

[0023] By adopting the above technical solutions, an indexing system based on feature extraction algorithm model identifiers can be established to achieve rapid retrieval of feature-encapsulated data. When the algorithm model is upgraded, the identifier information can be used to determine whether features need to be re-extracted, avoiding full data reprocessing. By using a computing node mapping strategy to allocate storage requests to sub-regions of each dedicated storage module, concurrent access control for face recognition algorithms, expression analysis algorithms, and orientation awareness algorithms can be achieved, ensuring that the storage access of a single algorithm does not exceed 60% of the total bandwidth. An improved cache replacement strategy based on algorithm priority and access frequency can be implemented, which can prioritize the replacement of low-priority and low-access-frequency data blocks when cache space is insufficient, rationally allocate cache resources, improve the efficiency of multimodal data processing, and reduce processing latency.

[0024] Preferably, the system further includes a performance monitoring and fault tolerance module, configured to: monitor the access latency and hit rate of each dedicated storage module in real time; trigger a cache preloading or storage reallocation mechanism when the access latency exceeds a threshold; and detect data integrity through verification information and trigger a recovery process when data is corrupted.

[0025] By adopting the above technical solutions, the access latency and hit rate of each dedicated storage module can be monitored in real time. When the access latency exceeds the threshold, a cache preloading or storage reallocation mechanism is triggered to ensure storage performance. By verifying information to detect data integrity, a recovery process is triggered when data is corrupted, which can ensure the stable operation of the system.

[0026] The beneficial effects of this invention are: 1. A dedicated storage module architecture is adopted, in which multimodal data is stored in corresponding dedicated storage modules, and targeted data processing and storage optimization are performed to meet the concurrent storage requirements of face recognition, expression analysis and orientation perception algorithms, and solve the problem of multimodal data processing latency.

[0027] 2. By optimizing storage access strategies, such as feature encapsulation and indexing mechanisms, load balancing and concurrency control, and cache replacement strategies, fast retrieval of stored data and concurrent access of the three algorithms were achieved, improving the efficiency of multimodal data processing.

[0028] 3. By employing data structure and storage format optimization techniques such as unified data encapsulation format, compression, and encoding optimization, it is easier to align memory pages and quickly access feature data. At the same time, feature data is compressed while ensuring the accuracy of algorithm recognition, thus saving storage resources. Attached Figure Description

[0029] Figure 1 This is a flowchart of the processing method of the present invention.

[0030] Figure 2 This is a diagram showing the relationship between the system modules of the present invention. Detailed Implementation

[0031] The technical solutions in the embodiments of the present invention will be further described in detail below with reference to the accompanying drawings. The described embodiments are only possible technical implementations of the present invention, but are not limited thereto. Other embodiments obtained by those skilled in the art in conjunction with the embodiments of the present invention without creative effort are also within the protection scope of the present invention.

[0032] This application mainly employs a dedicated storage module and optimization strategy to process multimodal data, thereby improving the efficiency and real-time performance of robot multimodal data processing. The following is a further detailed description of this application. Example

[0033] This application provides a method for multimodal data processing for robots, comprising: S1. Acquire multimodal data collected by sensors on the robot's end, including facial image data, facial expression image data, and orientation perception data.

[0034] Let the raw data collected by the sensor at time stamp t be Draw(t) = {D face D expression D direction}, which correspond to face images, expression images, and orientation-aware data streams, respectively.

[0035] Image data captured by a camera can be represented as a three-dimensional tensor I∈R H×W×C H, W, and C represent the height, width, and number of channels (e.g., RGB three channels) of the image, respectively.

[0036] Orientation-aware data (such as from infrared sensors or depth cameras) can be represented as point clouds. Or depth map D∈R H×W .

[0037] Image data is normalized in grayscale and standardized in size using the following formula:

[0038] Where μ and σ are the mean and standard deviation of the dataset, used to adjust for the effects of illumination; the Resize function (such as bilinear interpolation) scales the image to a standard size (H). std W std ).

[0039] For orientation-aware data, coordinate system unification and noise filtering are performed. For example, point cloud data is transformed to the robot's base coordinate system: in This is the transformation matrix from the sensor to the robot's base coordinate system. A statistical outlier removal filter can be used to reduce noise: P filtered ={p i ∈P∣distance(p i ,neighbors)<τ}, where τ is the distance threshold.

[0040] Because multimodal sensors acquire data at different frequencies, timestamp alignment is necessary. For non-image modal data, if the acquisition frequency is lower than that of image data, interpolation (such as linear interpolation) can be performed at the image timestamp to synchronize all modal data.

[0041] S2. The face image data is processed based on the face recognition algorithm, and the extracted face feature data is stored in a dedicated face recognition storage module.

[0042] Face detection and alignment: Face detection is performed using a cascaded classifier based on Haar or HOG features. Haar features are calculated as the difference between the sums of pixels within a rectangular region of the image.

[0043] After detecting a face, facial key points (such as 68-point or 118-point models) are used to align the face and normalize the face image to a standard pose.

[0044] Feature extraction: Principal component analysis (PCA, Eigenface) was used for feature dimensionality reduction and extraction. Calculate the average face value of the face images in the training set: .

[0045] Calculate the covariance matrix: .

[0046] Extract the eigenfaces corresponding to the k largest eigenvalues ​​as the projection basis: E=[e1,e2,...,e k ].

[0047] The feature vector of the new face image is: .

[0048] Deep learning methods (such as CNNs) can replace PCA, where the output of the last fully connected layer can be used as a feature vector.

[0049] Feature encapsulation and storage: The extracted feature vector v is encapsulated into a preset data structure: DataStruct={ModelID,Attributes,Checksum}.

[0050] ModelID: Identifies the version of the feature extraction algorithm.

[0051] Attributes: Contains custom attributes (such as timestamp t, feature vector v).

[0052] Checksum: The checksum, for example, CRC32: checksum=CRC32(v).

[0053] Establish "one Figure 1 The "file" index has a key that is a feature vector hash (such as MD5) and a value that is the storage address to avoid duplicate storage.

[0054] S3. The facial expression image data is processed based on the facial expression analysis algorithm, and the extracted facial expression feature data is stored in a dedicated facial expression analysis storage module in layers according to the degree of influence of the action area.

[0055] Facial feature extraction: Region segmentation: The face is divided into a primary motion region (high-impact areas, such as the eyes and mouth) and a secondary motion region (low-impact areas, such as the cheeks and forehead). Region segmentation can be based on facial key point localization.

[0056] Feature extraction: For the first action region, extract the Local Binary Pattern (LBP) texture features. The LBP operator is: , where s(x) = 1 if x ≥ 0, otherwise 0, g c g represents the grayscale value of the center pixel. p This represents the value of the neighboring pixels.

[0057] For the second action region, Gabor wavelet features can be extracted: , where x′=xcosθ+ysinθ, y′=−xsinθ+ycosθ, γ, σ, λ, θ are parameters.

[0058] Deep learning methods (such as CNNs) can extract global and local features end-to-end.

[0059] Tiered storage strategy: First action area feature F high Stored in a cache (such as SRAM), target access latency Δt high ≤15ms.

[0060] The second action region feature Flow is stored in a standard cache (such as DRAM), and the target access latency Δt low ≤40ms.

[0061] Cache mapping function: Address high =Hash(F high )modM high Address low =Hash(F low )modM low M high M low This is the cache size.

[0062] S4. The direction-aware data is processed based on the direction-aware algorithm, and the extracted spatial location feature data is stored in the direction-aware dedicated storage module according to the coordinate direction.

[0063] This step processes spatial location information and uses a coordinate attention mechanism to highlight key directional features.

[0064] Spatial feature extraction: Using coordinate attention mechanism: Input feature map U∈R H×W×C Pooling (max pooling and average pooling) is performed along the X and Y directions respectively to obtain direction-aware features:

[0065] The pooling results are concatenated and then used to generate an attention weight map A through convolution and an activation function (such as sigmoid). x A y .

[0066] Output weighted feature map: Vc=u c ×A x ×A y .

[0067] Feature map storage: The generated attention map is stored in a 22×22×32 feature map format (i.e., height 22, width 22, number of channels 32). Features for each direction (X, Y) are stored separately and retrieved quickly using spatial indexes (such as quadtrees or grid indexes).

[0068] Lightweight compression is performed before storage: quantization encoding converts floating-point features into 8-bit fixed-point representations, achieving a compression ratio of [missing information]. Ensure that the loss in recognition accuracy is ≤5%.

[0069] S5. Based on the concurrent access results of each dedicated storage module, generate user tags, intentions and spatial positioning information. Based on the user tags, intentions and spatial positioning information, generate differentiated interaction instructions and control the robot to perform wake-up-free active interaction.

[0070] This step integrates multimodal features to generate user profiles and trigger proactive interactions. The core of this step is collaborative decision-making based on multimodal information.

[0071] Multimodal feature fusion: Feature-level fusion is performed on the results of concurrent access to each dedicated storage module. Let the facial feature be v. f The facial expression feature is v e The directional feature is v d The fused feature vector is:

[0072] Among them W f W e W dThe weights for each modality can be dynamically calculated using an attention mechanism: .

[0073] Alternatively, a retrieval-enhanced generation (RAG) approach can be used to match fused features with a knowledge base (such as a user profile database) to generate scene description information.

[0074] User tags and intent recognition: User tag (gender, age) classification: using Softmax regression, , where w k Let be the weight vector for category k.

[0075] Facial expression intent recognition: As a multi-classification problem, the Support Vector Machine (SVM) decision function f(v) can be used. e )=sign(w T v e +b).

[0076] Interactive command generation: Based on user tags, intents, and spatial location information, differentiated interaction instructions are generated through a rule engine or a lightweight decision tree. For example, IF (age = "child" AND mood = "happy" AND distance < 1m) THEN execute "play welcome animation".

[0077] The instruction generation response time must meet t response Real-time requirement of ≤20ms.

[0078] By using dedicated storage modules and hierarchical storage to process and store data of different modalities, the efficiency and real-time performance of multimodal data processing are improved, and the problem of data processing delay caused by insufficient memory cache capacity is solved.

[0079] Specifically, in the process of acquiring multimodal data, sensors include cameras and infrared sensors. Cameras can clearly capture facial and facial expression data, while infrared sensors can effectively collect orientation perception data. These sensors can be replaced with other devices with similar functions, such as depth cameras. The acquired multimodal data is initially processed and transmitted using these sensors, preparing it for subsequent processing.

[0080] For the step of processing facial image data based on a facial recognition algorithm and storing it in a dedicated facial recognition storage module, the facial recognition algorithm can employ convolutional neural network algorithms, etc. During storage, a feature encapsulation data structure is used to encapsulate the facial image features into a preset data structure. This preset data structure includes feature extraction algorithm model identification information, custom attribute information, and verification information. The feature extraction algorithm model identification information facilitates the identification of different facial recognition algorithm models; the custom attribute information can add additional descriptive information according to actual needs; and the verification information is used to ensure the accuracy of the data. This data structure can be replaced with other similar data structures containing key information. Storing facial feature data in this way establishes a... Figure 1 The file association mechanism avoids duplicate storage of feature data from different versions of the model, thus improving storage efficiency.

[0081] The steps involve processing facial expression image data using an expression analysis algorithm and storing it hierarchically in a dedicated expression analysis storage module. This algorithm can employ deep learning-based expression recognition algorithms, among others. Facial features are divided into a first action region and a second action region. The first action region, representing high-influence areas such as the eyes and mouth, significantly impacts the intended meaning and emotion; its data is cached to minimize access latency and ensure rapid response to data access requests. The second action region, representing low-influence areas, uses a standard cache. This hierarchical storage method can be replaced with other methods based on data importance or access frequency. Hierarchical storage allows for more efficient use of storage resources and improves the access efficiency of facial expression feature data.

[0082] The steps involve processing direction-aware data using a direction-aware algorithm and storing it separately according to coordinate directions in a dedicated direction-aware storage module. This algorithm can employ sensor fusion-based direction-awareness methods. An integrated coordinate attention mechanism stores spatial location information separately for the X and Y directions. Each directional feature is processed using max pooling and average pooling to generate an attention map. The storage structure uses a specific feature map format to better represent spatial location information. This processing and storage method can be replaced with other methods for effectively processing and storing direction information. This processing and storage improves the efficiency of direction-aware data storage and processing.

[0083] In the step of generating interactive commands and controlling robot interaction, user tags, intentions, and spatial positioning information are generated based on the results of concurrent access to various dedicated storage modules. For example, user group tags such as gender and age are generated based on facial feature data, the user's expressive intentions and emotions are determined based on facial expression feature data, and the user's spatial location is determined based on orientation perception data. Then, differentiated interactive commands are generated based on this information to control the robot to perform wake-free proactive interactions. This allows the robot to more accurately understand user needs, respond promptly, and improve the quality and efficiency of human-computer interaction.

[0084] Taking a specific intelligent robot station as an example, this paper demonstrates how the above-mentioned multimodal data processing methods can play a role in practical applications, especially how the dedicated edge storage and optimization strategies support its efficient and secure operation.

[0085] Scene setting User: A man of about 30 years old, with a pleasant expression (upturned corners of his mouth and slight wrinkles around his eyes), walks towards the robot from 2 meters in front of him at a speed of 0.5 meters per second.

[0086] Robot task: Identify user identity and emotions, determine their intentions (such as proactive greetings or product recommendations), and prepare for interaction.

[0087] 1. Data Acquisition and Preprocessing: Face / expression image acquisition: The camera captures a frame of RGB image I∈R 512×512×3 .

[0088] Normalization: Scales pixel values ​​from [0, 255] to [0, 1]:

[0089] Size standardization: If the original image is 640×480, it needs to be scaled to 512×512 using bilinear interpolation.

[0090] Orientation-aware data acquisition: Depth camera acquires point cloud The user's center point coordinates are (1.5, 0.2, 2.1) (relative to the robot coordinate system).

[0091] Coordinate unification: Transform the point cloud to the robot's base coordinate system.

[0092] in This is the transformation matrix (including rotation and translation) from the camera to the robot base.

[0093] 2. Facial recognition feature extraction and storage: Face detection and alignment: Haar feature calculation: A sliding window is used in the image to calculate the difference between pixels in a rectangular region, for example:

[0094] The cascaded classifier identifies the region as a face.

[0095] Feature extraction (PCA method): Project the face image onto the pre-computed Eigenfaces space (assuming the first 100 principal components):

[0096] Where E∈R 100×100 Here is the eigenface matrix, and represents the average face. The extracted feature vector v f It is 100-dimensional.

[0097] Feature encapsulation and storage: Generate data structures: DataStruct = { ModelID: "FaceNet_v2", Attributes: { timestamp: t, feature: v_f}, Checksum: CRC32(v_f) } Stored in a dedicated face cache (accounting for 40% of the total), and a hash index (such as MD5(v_f)) is created.

[0098] 3. Layered storage of facial expression analysis: Region segmentation and feature extraction: First action region (eyes, mouth): Extract LBP texture features.

[0099] Taking the eye region as an example, calculate the LBP encoding: , thus obtaining the feature vector v e -high.

[0100] Second action region (cheek): Gabor filter features are extracted to obtain v e-low .

[0101] Tiered storage: V e-high Store in cache (target latency ≤ 15ms), v e-low Store in standard cache (latency ≤ 40ms).

[0102] Emotion classification: Emotion scores, such as happiness scores, are calculated using SVM. .

[0103] 4. Orientation-aware processing and storage Spatial attention mechanism: Generating feature maps U∈R from depth images 22×22×32 .

[0104] Pooling is performed along the X and Y directions respectively to generate attention weights:

[0105] Where σ is the Sigmoid function, f x ,f y This is a convolution operation.

[0106] Location calculation: User distance: .

[0107] Movement speed: .

[0108] 5. Multimodal decision-making and interaction generation: Feature fusion: Weighted fusion of multimodal features (weight assumptions: face 0.5, expression 0.3, orientation 0.2): v fusion = 0.5 v f +0.3 v e +0.2 v d Intent judgment: Rule engine logic: If the user is a member, has a sentiment score > 0.8, and is less than 3 meters away... THEN Generate personalized greeting instructions Response time verification: Total latency = face processing (20ms) + expression processing (15ms) + orientation processing (10ms) + decision making (5ms) ≈ 50ms < the 100ms threshold required by the robot.

[0109] Through the above calculation examples, this solution achieves efficient processing of multimodal data on the robot's edge, solves the latency problem caused by insufficient cache capacity, and provides real-time and accurate decision-making basis for proactive interaction. The implementation principle of this embodiment is as follows: By using dedicated storage modules and hierarchical storage to process and store data from different modalities, this embodiment avoids interference between data from different algorithms and improves the utilization efficiency of storage resources. Simultaneously, by establishing an indexing system, employing load balancing and concurrency control strategies, and cache replacement strategies, data storage and access are further optimized. This solves the data processing latency problem caused by insufficient memory cache capacity, improves the efficiency and real-time performance of robot multimodal data processing, and thus enhances the quality and efficiency of human-computer interaction.

[0110] Example 2 This embodiment is based on Embodiment 1, but differs from the previous embodiments in that: the indexing unit in the storage access optimization step establishes an indexing system based on the feature extraction algorithm model identifier. This indexing system can employ methods such as hash indexing. Through this indexing system, when the algorithm model is upgraded, it can quickly determine whether features need to be re-extracted based on the identifier information, avoiding full data reprocessing and keeping index lookup time within a short period. The concurrency control unit adopts a computing node mapping strategy, dividing the storage area into multiple sub-regions. Each sub-region is configured with feature extraction operators and retrieval operators. Storage requests are allocated through round-robin or random methods to achieve concurrent access for the three algorithms, ensuring that the storage access of a single algorithm does not exceed a certain proportion of the total bandwidth. The cache replacement unit executes an improved cache replacement strategy based on algorithm priority and access frequency. The face recognition algorithm has the highest priority; when cache space is insufficient, lower-priority data blocks with low access frequency are replaced first.

[0111] The implementation principle of this embodiment is as follows: by using storage access optimization measures such as indexing system, concurrency control and cache replacement strategy, the storage and access efficiency of multimodal data is further improved, data processing latency is reduced, and stability and efficiency are ensured when multiple algorithms access concurrently, thereby improving the overall performance of robot multimodal data processing.

[0112] Example 3 This embodiment is based on Embodiment 1. The multimodal data processing system for robots provided in this application includes a multimodal data acquisition module, a dedicated storage module, an interactive decision-making module, and an execution module. The modules cooperate with each other to achieve wake-up-free active interaction of the robot by acquiring, processing, storing, and analyzing multimodal data, thereby improving the quality and efficiency of human-computer interaction.

[0113] Specifically, the multimodal data acquisition module includes sensors such as cameras and infrared sensors. Cameras can quickly and accurately acquire facial and facial expression data, while infrared sensors effectively acquire orientation perception data. These sensors can be replaced with other devices with similar functions, such as LiDAR. The acquired multimodal data is then transmitted to subsequent modules for processing.

[0114] The dedicated storage modules include a face recognition module, an expression analysis module, and a direction-aware module. The face recognition module uses a feature encapsulation data structure to store face feature data. This structure includes feature extraction algorithm model identification information, custom attribute information, and verification information, facilitating the management and recognition of face feature data. The expression analysis module divides expression features into first and second action regions, storing them using high-speed caching and standard caching respectively, improving the access efficiency of expression feature data. The direction-aware module integrates a coordinate attention mechanism, storing attention maps generated based on X and Y direction features, which better represents spatial location information.

[0115] The interaction decision module receives data from dedicated storage modules and, based on the results of concurrent access to each dedicated storage module, generates user tags, intents, and spatial location information. For example, it generates user group tags based on facial feature data, determines user emotions and intents based on facial expression feature data, and determines user spatial location based on orientation perception data. Then, it generates differentiated interaction commands based on this information.

[0116] The execution module receives the interactive instructions generated by the interactive decision module and controls the robot to perform wake-up-free proactive interactions, enabling the robot to respond to users in a timely and accurate manner, thereby improving the quality and efficiency of human-computer interaction.

[0117] The implementation principle of this embodiment is as follows: Through the collaborative work of various modules, the system comprehensively collects, processes, stores, and analyzes multimodal data, realizing wake-free active interaction of the robot. The design of the dedicated storage module and the hierarchical storage method improve the processing and storage efficiency of multimodal data, solve the problem of insufficient memory cache capacity, and thus improve the quality and efficiency of human-computer interaction.

[0118] Example 4 This embodiment is based on Embodiment 1, but differs from the previous embodiments in that the system further includes a storage management module, which comprises an indexing unit, a concurrency control unit, and a cache replacement unit. The indexing unit establishes an indexing system based on feature extraction algorithm model identifiers, employing methods such as B-tree indexes for fast and accurate data retrieval. The concurrency control unit uses a computing node mapping strategy to allocate storage requests to sub-regions of each dedicated storage module, achieving concurrent access control and ensuring efficient access to the storage modules by each algorithm. The cache replacement unit executes an improved cache replacement strategy based on algorithm priority and access frequency, with the face recognition algorithm having the highest priority, ensuring priority storage and access for important data.

[0119] The implementation principle of this embodiment is as follows: the addition of the storage management module further optimizes the storage and access performance of the system. Through the indexing system, concurrency control and cache replacement strategy, the storage and access efficiency of multimodal data is improved, data processing latency is reduced, and the stability and efficiency of the system are ensured when multiple algorithms access concurrently.

[0120] Example 5 This embodiment is based on Embodiment 1, but differs from the previous embodiments in that it further includes a performance monitoring and fault tolerance module. This module is configured to monitor the access latency and hit rate of each dedicated storage module in real time. For example, monitoring software is used to monitor various indicators of the storage modules in real time. When the access latency exceeds a threshold, a cache preloading or storage reallocation mechanism is triggered to improve data access speed. Data integrity is checked by verifying information, and a recovery process is triggered when data is corrupted, such as restoring data from backup storage or recalculating data, to ensure stable system operation.

[0121] The implementation principle of this embodiment is as follows: the setting of the performance monitoring and fault tolerance module can promptly detect problems in the operation of the system and take corresponding measures to optimize and repair them, ensuring the stability and reliability of the system and further improving the overall performance of the robot multimodal data processing system.

[0122] The above are all preferred embodiments of this application, and are not intended to limit the scope of protection of this application. Therefore, all equivalent changes made in accordance with the structure, shape and principle of this application should be covered within the scope of protection of this application.

Claims

1. A method for multimodal data processing for robots, characterized in that, The method includes: S1. Acquire multimodal data collected by sensors on the robot's end, including facial image data, facial expression image data, and orientation perception data; S2. Process the face image data based on the face recognition algorithm, and store the extracted face feature data in a dedicated face recognition storage module; S3. The facial expression image data is processed based on the facial expression analysis algorithm, and the extracted facial expression feature data is stored in a dedicated facial expression analysis storage module in layers according to the degree of influence of the action area. S4. The direction-aware data is processed based on the direction-aware algorithm, and the extracted spatial location feature data is stored in the direction-aware dedicated storage module according to the coordinate direction. S5. Based on the concurrent access results of each dedicated storage module, generate user tags, intentions and spatial positioning information. Based on the user tags, intentions and spatial positioning information, generate differentiated interaction instructions and control the robot to perform wake-up-free active interaction.

2. The multimodal data processing method for robots according to claim 1, characterized in that, The step of storing the extracted facial feature data in a dedicated facial recognition storage module includes: encapsulating the facial image features into a preset data structure using a feature encapsulation data structure and then storing it. The preset data structure includes feature extraction algorithm model identification information, custom attribute information, and verification information. The step of storing the extracted facial expression feature data in a dedicated facial expression analysis storage module in layers includes: dividing the facial expression features into a first action region and a second action region, wherein the data of the first action region is stored in a high-speed cache and the data of the second action region is stored in a standard cache. The step of storing the extracted spatial location feature data in a dedicated direction-aware storage module according to the coordinate direction includes: pooling the features of the spatial location information along the X and Y directions to generate attention mapping maps, and storing them according to a preset feature map format.

3. The multimodal data processing method for robots according to claim 2, characterized in that, The method also includes a storage access optimization step: establishing an index system based on the feature extraction algorithm model identifier, and determining whether features need to be re-extracted based on the identifier information when the algorithm model is upgraded; A computing node mapping strategy is adopted to allocate storage requests to sub-regions of each dedicated storage module, so as to achieve concurrent access control for face recognition algorithm, expression analysis algorithm and orientation awareness algorithm, wherein the storage access of a single algorithm does not exceed 60% of the total bandwidth; Furthermore, based on algorithm priority and access frequency, an improved cache replacement strategy is adopted to manage cached data, with face recognition algorithms having the highest priority, expression analysis algorithms second, and orientation perception algorithms having the lowest priority.

4. The multimodal data processing method for robots according to claim 3, characterized in that, The improved cache replacement strategy is as follows: when cache space is insufficient, prioritize replacing low-priority data blocks that are accessed infrequently.

5. A multimodal data processing method for robots according to claim 2, characterized in that, When storing the facial feature data, expression feature data, and spatial location feature data, a unified data encapsulation format is adopted. The format includes header information and feature package. The header information contains data type, algorithm identifier, and timestamp metadata. The feature package contains feature vector and verification information.

6. The multimodal data processing method for robots according to claim 5, characterized in that, Before storage, the feature data is lightly compressed and quantized to convert floating-point features into fixed-point representations. The compression ratio is controlled at 60%-70%, and the algorithm's recognition accuracy is ensured to be no less than 95% of that before compression.

7. A multimodal data processing system for robots, applicable to the multimodal data storage and interaction method for robots as described in any one of claims 1-6, characterized in that, The system is deployed on the robot's end face and includes: A multimodal data acquisition module is configured to acquire multimodal data through sensors, wherein the multimodal data includes at least facial image data, facial expression image data, and orientation perception data; A dedicated storage module includes: a dedicated storage module for face recognition, configured to store facial feature data extracted by a face recognition algorithm; A dedicated storage module for facial expression analysis is configured to store facial feature data extracted by facial expression analysis algorithms in layers according to the degree of influence of the action area; a dedicated storage module for direction perception is configured to store spatial location feature data extracted by direction perception algorithms according to coordinate direction. The interactive decision module is configured to generate user tags, intents, and spatial location information based on the concurrent access results of the dedicated storage module, and generate differentiated interactive instructions accordingly. The execution module is configured to execute the interactive instructions to enable the robot to perform wake-up-free active interaction.

8. A multimodal data processing system for robots according to claim 7, characterized in that, The dedicated storage module for face recognition uses a feature encapsulation data structure to store face feature data. The data structure includes feature extraction algorithm model identification information, custom attribute information, and verification information. The dedicated storage module for expression analysis is configured to divide expression features into a first action region and a second action region, and store them using a high-speed cache and a standard cache, respectively. The dedicated storage module for direction perception is configured to integrate a coordinate attention mechanism to store attention mapping maps generated according to features in the X and Y directions.

9. A multimodal data processing system for robots according to claim 7, characterized in that, The system also includes a storage management module, which comprises: The indexing unit is configured to establish an indexing system based on feature extraction algorithm model identifiers; The concurrency control unit is configured to use a compute node mapping strategy to allocate storage requests to sub-regions of each dedicated storage module in order to achieve concurrent access control. The cache replacement unit is configured to execute an improved cache replacement strategy based on algorithm priority and access frequency, with the face recognition algorithm having the highest priority.

10. A multimodal data processing system for robots according to claim 7, characterized in that, The system also includes a performance monitoring and fault tolerance module, configured to: monitor the access latency and hit rate of each dedicated storage module in real time; and trigger a cache preloading or storage reallocation mechanism when the access latency exceeds a threshold. It also checks data integrity by verifying information and triggers a recovery process when data is corrupted.

Images