Multi-view human posture data acquisition device and data processing method for embodied intelligence

Abstract

The present invention discloses a multi-view human posture data acquisition device for embodied intelligence and a data processing method thereof. The acquisition device includes a data acquisition unit, a control and data processing unit, a storage unit, a communication module, a human-machine interface module and an auxiliary equipment module. The data processing method uses a graph-space-time convolutional neural network ST-GCN to convert the skeleton point sequences of two cameras into a standardized action description model; based on the Graph-Cut RANSAC method, the spatial transformation matrix is ​​calculated to match and align the skeleton point sequences of the two cameras; and the audio data is segmented and labeled by the VAD technology. The present invention has the advantages of synchronously adding voice information tags, comprehensive environmental perception, high-precision skeleton point recognition, and process event serialization, which improves the performance of the system in data collection in various application scenarios.

Description

Technical Field

[0001] The present invention relates to a human body posture data acquisition device and a processing method, and in particular to a multi-view human body posture data acquisition device and a data processing method for embodied intelligence, belonging to the field of robotics technology. Background Art

[0002] Embodied intelligence is a system that integrates environmental understanding, intelligent interaction, cognitive reasoning and planning execution into the same system, integrating artificial intelligence into physical entities such as robots, giving them the ability to actively perceive, learn, dynamically interact with the environment and make decisions. Among them, active perception technology is particularly critical, which enables physical entities to actively detect and obtain key information in the environment, so as to more effectively understand and adapt to changes in the environment, and then make decisions for specific environments.

[0003] In the field of embodied intelligent application technology, people hope that physical entities can actively perceive the environment and make targeted decisions. This requires the device to fully understand the three-dimensional scene based on the first-person observation results, and convert its context into interactive language, and have the ability to recognize and think logically. Traditional single-view perception technology obtains limited information, which easily leads to reduced accuracy in environmental understanding and decision-making. In contrast, multi-view perception technology further enhances the robot's environmental understanding ability by collecting data from different angles at the same time, enabling it to build a more complete environmental model and optimize decision-making and action strategies. Voice Activity Detection (VAD) technology provides an effective method for accurately detecting the starting and ending points of voice signals. Summary of the invention

[0004] The purpose of the present invention is to provide a multi-view human posture data acquisition device and a data processing method for embodied intelligence.

[0005] To achieve the above object, the present invention provides the following technical solutions:

[0006] Technical Solution

[0007] A multi-view human posture data acquisition device for embodied intelligence, comprising a data acquisition unit, a control and data processing unit, a storage unit, a communication module, a human-machine interface module and an auxiliary equipment module;

[0008] The data acquisition unit includes an orthogonal visual perception platform built by two imaging cameras and a microphone. The optical axes of the two imaging cameras of the orthogonal visual perception platform intersect to collect time-stamped human skeleton point video stream data from two different angles, and the microphone collects corresponding time-stamped audio data.

[0009] The control and data processing unit extracts the process events of the serialized actions based on the video stream data of the human skeleton points, and realizes a general action decomposition model that can be evaluated in the process;

[0010] The storage unit is used to store the human skeleton point video stream data;

[0011] The human-computer interface is responsible for interacting with the user;

[0012] The auxiliary device module is responsible for sending and receiving voice sequence data.

[0013] Furthermore, the data acquisition device also includes a communication module, which provides a data transmission interface.

[0014] Furthermore, the data acquisition device also includes a power management module, wherein the power management module has

[0015] It has real-time battery power monitoring function, providing low-battery warning and automatic shutdown protection.

[0016] Furthermore, among the two cameras, camera A is located in front of the object, and camera B is located on the right side of the object, and the two cameras are installed perpendicular to each other.

[0017] Technical Solution

[0018] A data processing method for the data acquisition device described in technical solution 1, comprising the following steps:

[0019] Step 1: Audio data endpoint detection and interception:

[0020] Step 1-1: segment the audio data into corresponding video frames, and extract short-time energy features, frequency features and zero-crossing rate features from each audio data segment one by one;

[0021] Calculate the short-time energy of each audio data segment:

[0022] (1)

[0023] where E\left [ {M} \right ] is the short-time energy of the mth audio data segment; N is the number of samples in each audio data segment, x\left [ {n} \right ] is the amplitude value of the nth audio signal;

[0024] Calculate the spectrum centroid C and spectrum flatness P:

[0025] (2)

[0026] (3)

[0027] Where X(f k ) is the audio data x\left [ {n} \right ] The spectrum, f k is the kth frequency component;

[0028] Calculate the zero-crossing rate:

[0029] (4)

[0030] Where, sgn() is the sign function; T is the time length of the audio data segment;

[0031] Step 1-2: Calculate comprehensive features:

[0032] (5)

[0033] in, , and Represents short-time energy The threshold of , the zero-crossing rate threshold and the spectrum centroid threshold;

[0034] Step 1-3: Combine the comprehensive features CF Compare with a preset threshold, if it is greater than the preset threshold, identify it as a speech signal segment, otherwise, identify it as a non-speech signal segment;

[0035] Step 1-4: Use VAD technology to detect the starting point and ending point of the speech signal segment;

[0036] Step 1-5: After the audio data segment is segmented according to the result y of the speech endpoint detection, the identification label of the corresponding video frame is added;

[0037] Step 2: Predict the position of human key points: Use the optimized BlazePose model to detect skeleton key points frame by frame on the preprocessed video stream data;

[0038] The BlazePose model includes a coarse positioning module and a key point fine positioning module. The coarse positioning module extracts the preliminary position of the human body through a lightweight convolutional neural network, and the fine positioning module refines the position of each key point; the fine positioning module includes a feature change detection unit, a parameter adjustment unit, a parameter learning unit, and a parameter optimization unit; the feature change detection unit uses a local difference method to calculate the difference between adjacent frames in time or the gradient change in space to generate a change perception map; the parameter adjustment unit dynamically adjusts the weight of the self-adjusting convolution kernel based on the generated change perception map; the parameter learning unit adjusts the model parameters by learning parameter control under the control of the parameter adjustment unit:

[0039] (6)

[0040] Where K(x,y) is the value of the self-tuning convolution kernel at position (x,y), a and b are learnable parameters, f(x,y) represents the local description of the input features, and σ is the activation function;

[0041] The objective function of the parameter optimization unit is:

[0042] (7)

[0043] in, is the position of the real key point, is the position predicted by the model, and J is the functional of the self-adjusting convolution kernel, which is used to optimize the feature extraction ability of the convolution kernel, expressed as:

[0044] (8)

[0045] Step 3: Matching of key human nodes: Use the Graph-Cut RANSAC method to calculate the spatial transformation matrix, and match and align orthogonal key human nodes;

[0046] Step 4: Generate a time-space diagram of key human nodes: take the key human nodes of each video stream data frame as nodes, and establish edges corresponding to the key human nodes of adjacent video stream data frames; establish edges for the key human nodes in each video stream data frame according to a preset human topology diagram, and obtain a time-space diagram of key human nodes;

[0047] Step 5: Action decomposition and description: Combined with the spatiotemporal graph convolutional neural network ST-GCN, the standardized action decomposition and description are achieved;

[0048] A spatiotemporal graph convolutional neural network ST-GCN is established and trained using a preset training set. The spatiotemporal graph convolutional neural network ST-GCN includes an input layer, a processing unit consisting of a graph convolutional network GCN and a temporal convolutional network TCN, and an output layer.

[0049] The skeletonized video stream output by the multi-view acquisition system is encoded according to the human figure topology, and is time-extended according to the preset multiplication parameter value. Then, it is input into the graph spatiotemporal convolutional neural network ST-GCN for analysis. The characteristics of different actions are identified through the weak supervision method of human-computer interaction, and the action process events at each time point T are automatically extracted.

[0050] Further, the steps 1-4 use the SGHTR model for speech endpoint detection; the SGHTR model first calculates the short-time energy features of the speech signal segment through the short-time energy module, and uses the zero-crossing rate method to extract the zero-crossing features of the speech signal segment; the Gaussian mixture model G is introduced to fit the Gaussian distribution characteristics of the short-time energy features through a combination of more than one Gaussian distribution; the hidden Markov model H uses the state set S, transition probability A and observation probability B parameters to analyze the state transition characteristics of the speech data segment on this basis; the Transformer module T captures the long-distance context information of the speech signal segment through a multi-head attention mechanism, a feedforward layer and a normalization operation; the recurrent neural network RNN ​​analyzes the short-time energy features, the zero-crossing features, the Gaussian distribution characteristics of the short-time energy features, and the state transition characteristics of the speech data segment, and outputs the speech endpoint detection result; the functional form of the SGHTR model can be expressed as:

[0051] (9)

[0052] Among them, S is the short-time energy module, G is the Gaussian mixture model module, H is the hidden Markov model module, T is the Transformer module, and R is the recurrent neural network module.

[0053] Furthermore, the BlazePose model is pruned and its single-precision inference model is retrained using the instruction set-optimized self-adjusting convolution kernel to achieve the deployment of a lightweight real-time motion capture model on the edge.

[0054] Compared with the prior art, the present invention has the following beneficial effects:

[0055] 1. In the process of actively perceiving the three-dimensional environment in which the human body is located and making decisions, the present invention collects serialized human posture data for embodied intelligence and completes embodied data labeling.

[0056] 2. The present invention integrates multi-view video stream data and adopts an adaptive perception adjustment method to achieve skeleton perception, serialization processing, storage and display of skeleton point data.

[0057] 3. The present invention integrates a high-definition camera, an orthogonal visual perception platform with limited computing power, and real-time motion capture technology to decompose and annotate posture movements while completing multi-view acquisition of human posture data.

[0058] 4. The present invention automatically adds labels to video stream data, combines the human skeleton stream serialized action decomposition algorithm VAD technology, reads the voice data from the microphone in real time, accurately locates the start and end points of the voice from the noisy voice, and automatically labels a series of serialized actions. BRIEF DESCRIPTION OF THE DRAWINGS

[0059] Figure 1 It is a structural schematic diagram of the present invention;

[0060] Figure 2 The skeleton point sequence marked in Example 2 of the present invention;

[0061] Figure 3 It is a schematic diagram of the process event extraction method of the present invention;

[0062] Figure 4 This is a flow chart of Embodiment 2 of the present invention;

[0063] Figure 5 This is the SGHTR model architecture diagram in the present invention;

[0064] Figure 6 This is the model architecture of the spatiotemporal convolutional neural network of the present invention. DETAILED DESCRIPTION

[0065] Embodiment 1:

[0066] A multi-view human posture data acquisition device for embodied intelligence includes a data acquisition unit, a data processing unit, a storage unit, a communication module, a power management module, a human-machine interface module, and an auxiliary device module. These components are integrated in a compact and movable frame for easy installation and use. The overall design diagram is shown in the figure below. Figure 1 As shown. The data acquisition unit includes an orthogonal visual perception platform built by two imaging cameras. Camera A is located in front of the object, and camera B is located on the right side of the object. The distance, height, pitch angle and yaw angle of the two imaging cameras from the object are determined according to the specific parameters of the imaging cameras. The two cameras are installed perpendicular to each other to ensure that the surface data of the object can be obtained from different angles, realize the embodied intelligent dual-view data acquisition, and are responsible for collecting the video stream data of the human skeleton points. The control and data processing unit performs self-extraction of process events of serialized actions based on the video stream data of the human skeleton points, solves the problem of standardized computer description of the action process, and realizes a general action decomposition model that can be evaluated in the process. The storage unit is responsible for storing the human skeleton stream video data collected by the embodied intelligent multi-view, and sets a reasonable data buffer to ensure that the data will not be lost. The communication module is responsible for data transmission. The power management module is responsible for the endurance of the embodied intelligent multi-view acquisition device. The human-machine interface is responsible for interacting with the user. The auxiliary equipment module is responsible for sending and receiving audio data.

[0067] The data acquisition unit includes an orthogonal imaging unit and an edge computing terminal. The orthogonal imaging unit is composed of two imaging cameras in an orthogonal layout, which obtains dual-view video information of the spatial human skeleton with time stamps, effectively solving the problem of occlusion of human skeleton points during action observation, and improving the accuracy of skeleton key point detection and positioning. In the two imaging cameras of this embodiment, camera A is located directly in front of the object, and camera B is located on the right side of the object. The edge computing terminal adopts domestic edge computing chips, designs optimized video push-pull stream support software, and enriches the domestic edge computing chip software tool chain. At the same time, for the RISC-V instruction set architecture of domestic chips, a self-adjusting convolution kernel that supports single-precision models is written to improve the efficiency of human skeleton point detection, making full use of the computing power of domestic chips.

[0068] The control and data processing unit is the core of the embodied intelligent multi-view data acquisition device, equipped with a high-performance CPU or GPU and FPGA acceleration module to ensure efficient processing and analysis of the collected raw data. First, the data goes through a preprocessing stage, including denoising, filtering, and format conversion to ensure data quality. Then, based on the optimized BlazePose model, the key points of the human skeleton are detected and located in real time, and the computational efficiency is optimized through the self-adjusting convolution kernel to achieve high-precision real-time detection.

[0069] The data processing unit also performs action decomposition and recognition, automatically annotates a series of action labels for video stream data, and uses the graph spatiotemporal convolutional neural network ST-GCN to convert the skeleton point sequence into a standardized action description model. Based on the Graph-Cut RANSAC method, the data processing unit calculates the spatial transformation matrix, matches and aligns the corresponding skeleton points of the two imaging cameras' dual-view acquisition (taking orthogonality as an example), and outputs high-precision, unobstructed skeleton point positions. The microphone is used to collect the audio data output by the speaker in real time, and the audio data is segmented and annotated through the voice endpoint detection technology VAD. The processed audio data is output in real time through the communication module and stored in a large-capacity SSD to ensure data integrity and data security during high-frequency acquisition.

[0070] The storage unit is responsible for storing audio data and two-view video stream data collected by two imaging cameras. Each view of the video stream data includes human skeleton video stream data and sensor data. Equipped with a large-capacity SSD, it ensures that a large amount of high-resolution video and sensor data can be stored. The storage unit also designs a reasonable data buffer structure to ensure that data is not lost during high-frequency data collection. In addition, data storage is optimized to support fast reading and writing and efficient data retrieval, ensuring the system's performance in data storage and access.

[0071] The communication module is responsible for data transmission of the device and supports multiple communication methods. Wireless communication supports Wi-Fi and Bluetooth, which facilitates data exchange and remote control with other devices. Wired communication includes high-speed Ethernet and USB 3.0 interfaces for high-bandwidth data transmission and device programming. The communication module also has a security protocol to ensure the security and integrity of the data transmission process, support real-time monitoring and feedback, and ensure efficient connection between the system and external devices.

[0072] The power management module ensures continuous power supply to the embodied intelligent multi-view acquisition device, and is equipped with a high-capacity rechargeable battery to support long-term operation. The power management module has a real-time battery power monitoring function, provides low-battery warnings and automatic shutdown protection to prevent data loss. The module design takes into account the energy consumption optimization of the device, ensuring efficient use of power in different working modes, extending battery life and ensuring stable operation of the device.

[0073] The human-machine interface module is responsible for interacting with the user, providing intuitive device control and status display. It is equipped with a small high-definition display to display device status, data collection progress and real-time preview. The control buttons and touch interface design are easy for users to operate, support function switching and parameter setting. When there is no voice source on site, it supports manual tagging. The human-machine interface module optimizes the user experience, ensures simple operation and timely feedback, and helps users use and manage data collection devices efficiently.

[0074] The data set collection auxiliary equipment consists of a speaker and a microphone. The speaker is responsible for outputting the audio data corresponding to the human body posture and movement, while the microphone collects the audio data with time stamp and inputs it into the control and data processing unit.

[0075] Embodiment 2:

[0076] A data processing method for the acquisition device described in Example 1, comprising the following steps:

[0077] A data processing method for the acquisition device described in Example 1, comprising the following steps:

[0078] Step 1: Audio data endpoint detection and interception:

[0079] Step 1-1: segment the audio data into corresponding video frames, extract the short-time energy features, frequency features and zero-crossing rate features from each audio data segment one by one, calculate the comprehensive features, compare them with the preset threshold, if it is greater than the preset threshold, identify it as a speech signal, otherwise, identify it as a non-speech signal; use VAD technology to detect the starting point and ending point of the speech signal;

[0080] Calculate the short-time energy of each audio data segment:

[0081] (1)

[0082] where E\left [ {M} \right ] is the short-time energy of the mth audio data segment; N is the number of samples in each audio data segment, x\left [ {n} \right ] is the amplitude value of the nth audio signal;

[0083] Frequency features are usually obtained through spectrum analysis. Common frequency features include spectrum centroid and spectrum flatness. The spectrum centroid reflects the center position of the signal spectrum. Speech segment signals usually have higher spectrum centroid values. Spectral flatness describes the flatness of the spectrum. A flat spectrum usually indicates noise or silence. The spectrum of speech segment signals is more complex and has lower spectrum flatness. The calculation formulas for spectrum centroid C and spectrum flatness P are:

[0084] (2)

[0085] (3)

[0086] where f k is the frequency component, X(f k ) is x\left [ {n} \right ] spectrum.

[0087] The zero crossing rate ZCR in the time domain features indicates the number of zero crossings of an audio signal per unit time, and is usually used to distinguish between speech and noise. Speech signals usually have a higher zero crossing rate, while background noise or silence has a lower zero crossing rate. The formula for calculating the zero crossing rate is:

[0088] (4)

[0089] Among them, sgn() is the sign function; T is the time length of the audio data segment.

[0090] Step 1-2: Calculate comprehensive features:

[0091] (5)

[0092] in, Indicates short-term energy Relative to threshold The higher the short-time energy, the stronger the signal. represents the combination of zero crossing rate ZCR and spectrum centroid C, and The thresholds for the zero-crossing rate and spectral centroid, respectively. A high zero-crossing rate and a high spectral centroid usually indicate that the signal is speech. Indicates the ratio of spectrum flatness P to the threshold, Threshold indicating spectral flatness. Lower spectral flatness usually means the signal is speech, while higher spectral flatness usually indicates background noise.

[0093] Step 1-3: Combine the comprehensive features CF Compare with a preset threshold, if it is greater than the preset threshold, identify it as a speech signal segment, otherwise, identify it as a non-speech signal segment;

[0094] This embodiment performs weighted calculation on energy features, time domain features and frequency features, and determines whether a speech signal exists based on a set comprehensive threshold.

[0095] The calculated comprehensive feature is compared with the set threshold to determine whether there is a voice signal. When the comprehensive feature is greater than the set threshold, it means there is a voice signal, otherwise there is no voice signal.

[0096] Step 1-4: Use VAD technology to detect the starting point and ending point of the speech signal segment;

[0097] The VAD technology of this embodiment uses the SGHTR model. The SGHTR model is an innovative architecture designed to solve the problem of voice endpoint detection. It integrates graph neural network modeling and deep learning technology to achieve efficient detection of complex voice signals. Figure 6The architecture of the model is shown. The processing flow of the model starts with the short-time energy algorithm S. First, the energy of the audio signal is calculated by the short-time energy algorithm, and the zero-crossing rate method is used to further extract the zero-crossing features of the signal. Next, the model introduces a Gaussian mixture model G to further model the short-time energy features. Through the combination of multiple Gaussian distributions, the model can more accurately fit the distribution characteristics of the speech signal and enhance its adaptability to complex speech signals. On this basis, the hidden Markov model H is used to analyze the state transition characteristics of the signal. The hidden Markov model contains a state set S, a transition probability A, and an observation probability B, which describe the temporal changes of the signal. In order to further optimize the feature expression, the model adds a Transformer module T to capture the long-range context information of the signal through a multi-head attention mechanism, a feedforward layer, and a normalization operation. The Transformer module T enhances the model's ability to process features at different levels, enabling it to maintain efficient feature expression and classification capabilities in a complex speech environment. Finally, the output layer of the model uses a recurrent neural network RNN. The recurrent neural network (RNN) contains an input layer, a hidden layer, and an output layer, which can convert the previously extracted signal features into the final result of speech endpoint detection. Through the temporal feature modeling of the recurrent neural network (RNN), the SGHTR model can make accurate endpoint judgments on speech signals, improving the accuracy and robustness of detection. The modules work together to enable the SGHTR model to have stronger detection accuracy and adaptability in a variety of speech scenarios. This collaborative processing mechanism significantly improves the detection effect and robustness of the model in complex speech environments.

[0098] The input of the SGHTR model is the original speech signal x(n) for speech detection. Through the sequential processing of multiple modules, the signal features are gradually extracted and enhanced, and the final detection result y(n) is output. Its functional form can be expressed as:

[0099] (6)

[0100] First, the original speech signal x(n) is input into the short-time energy algorithm module S, which extracts the basic time domain features of the speech signal through short-time energy calculation and zero-crossing rate analysis. The extracted features provide key inputs for subsequent modules, forming the characterized signal data S(x(n)). Next, the signal S(x(n)) processed by the short-time energy algorithm module is passed to the Gaussian mixture model module G. In this module, the signal feature data is modeled by multiple Gaussian distribution combinations GMM to fit the probability distribution of the signal. The feature G(S(x(n))) output by the Gaussian mixture model contains the distribution information of the signal. The output G(S(x(n))) of the Gaussian mixture model is further passed to the hidden Markov model module H for processing. The HMM module can effectively capture the state changes in the speech signal, and the output feature H(G(S(x(n)))) contains the dynamic state transition information of the signal. Then, the output H(G(S(x(n)))) of the hidden Markov model enters the Transformer module T. The output T(H(G(S(x(n))))) of the Transformer module T further optimizes the feature representation of the signal. Finally, the enhanced signal feature T(H(G(S(x(n))))) is input into the recurrent neural network module R to finally generate the result y of speech endpoint detection.

[0101] Step 1-5: After the audio data segment is segmented according to the result y of speech endpoint detection, the identification label of the corresponding video frame is added.

[0102] Step 2: Predict the position of human key points: Use the optimized BlazePose model to detect skeleton key points frame by frame on the preprocessed video stream data;

[0103] The BlazePose model includes a coarse positioning module and a key point fine positioning module. The coarse positioning module extracts the preliminary position of the human body through a lightweight convolutional neural network, and the fine positioning module refines the position of each key point; the fine positioning module includes a feature change detection unit, a parameter adjustment unit, a parameter learning unit, and a parameter optimization unit; the feature change detection unit uses a local difference method to calculate the difference between adjacent frames in time or the gradient change in space to generate a change perception map; the parameter adjustment unit dynamically adjusts the weight of the self-adjusting convolution kernel based on the generated change perception map; the parameter learning unit adjusts the model parameters by learning parameter control under the control of the parameter adjustment unit:

[0104] (7)

[0105] Where K(x,y) is the value of the self-tuning convolution kernel at position (x,y), a and b are learnable parameters, f(x,y) represents the local description of the input features, and σ is the activation function;

[0106] This embodiment performs pruning operations on the BlazePose model, retrains its single-precision inference model using instruction set-optimized self-adjusting convolution kernels, and implements the deployment of a lightweight real-time motion capture model on the edge side.

[0107] This embodiment uses the optimized BlazePose architecture to detect skeleton key points frame by frame on the preprocessed video stream data, and combines the self-adjusting convolution kernel to improve the reasoning speed and efficiency. The BlazePose model is used to predict the positions of 33 human key points. Subsequently, the Graph-Cut RANSAC method is used to calculate the spatial transformation matrix, match and align the orthogonal human key nodes, and output the high-precision human skeleton point position with occlusion eliminated, thereby achieving high-precision, real-time detection and positioning of human skeleton key points. The monitoring diagram of 33 key sites of the human body is shown in the figure below. Figure 2 shown.

[0108] The design of the self-adjusting convolution kernel can be divided into several key steps, aiming to improve the key point detection ability of the BlazePose model in real-time video streaming data. First, the feature change detection module identifies the areas that need to be focused on by analyzing the changes in the adaptively extracted features. This module can use the local difference method to calculate the difference between adjacent frames in time or the gradient change in space, capture the importance of different areas in the image, and generate a change perception map.

[0109] Based on the generated change perception map, the weight of the self-regulating convolution kernel is dynamically adjusted. For areas with large changes, the weight of the convolution kernel will be moderately increased to enhance the sensitivity to these areas, thereby ensuring that the significant changes are fully computed. On the contrary, for areas with small or unimportant changes, the weight of the convolution kernel will be reduced, or even some calculations will be skipped to reduce unnecessary computational overhead.

[0110] In the convolution operation, the self-adjusting convolution kernel will be calculated according to the adjusted weights. This dynamically adjusted convolution operation enables the model to process the input feature map more accurately, especially when detecting human skeleton points, it can effectively capture the key changes of limbs.

[0111] The objective function of the parameter optimization unit is:

[0112] (8)

[0113] in, is the position of the real key point, is the position predicted by the model, and J is the functional of the self-adjusting convolution kernel, which is used to optimize the feature extraction ability of the convolution kernel, expressed as:

[0114] (9)

[0115] Step 3: Matching of key human nodes: Use the Graph-Cut RANSAC method to calculate the spatial transformation matrix, and match and align orthogonal key human nodes;

[0116] The skeleton point recognition algorithm uses lightweight convolutional neural networks and Graph-Cut RANSAC methods to detect and align key points of human skeletons in real time, and combines the spatiotemporal graph convolutional neural network ST-GCN to achieve standardized decomposition and description of complex actions. Through these technologies, high-precision synchronous data annotation is achieved, providing a solid foundation for subsequent human action recognition and analysis.

[0117] Step 4: Generate a time-space diagram of key human nodes: take the key human nodes of each video stream data frame as nodes, and establish edges corresponding to the key human nodes of adjacent video stream data frames; establish edges for the key human nodes in each video stream data frame according to a preset human topology diagram, and obtain a time-space diagram of key human nodes;

[0118] Step 5: Action decomposition and description: Combined with the spatiotemporal graph convolutional neural network ST-GCN, the standardized action decomposition and description are achieved;

[0119] A spatiotemporal graph convolutional neural network ST-GCN is established and trained using a preset training set. The spatiotemporal graph convolutional neural network ST-GCN includes an input layer, a processing unit consisting of a graph convolutional network GCN and a temporal convolutional network TCN, and an output layer.

[0120] The skeletonized video stream output by the multi-view acquisition system is encoded according to the human figure topology, and is time-extended according to the preset multiplication parameter value. Then, it is input into the graph spatiotemporal convolutional neural network ST-GCN for analysis. The characteristics of different actions are identified through the weak supervision method of human-computer interaction, and the action process events at each time point T are automatically extracted.

[0121] The present invention uses the ST-GCN architecture, a neural network model for processing graph structured data, for skeleton point sequence data in action recognition. ST-GCN effectively combines the graph convolutional network GCN and the temporal convolutional network TCN, enabling it to perform efficient convolution operations on the spatiotemporal graph, which is particularly suitable for skeleton point sequences collected from multiple cameras. In this model, the skeleton point sequence can be regarded as a spatiotemporal graph, with nodes representing the skeleton points at each time step and edges connecting the spatial and temporal relationships of these skeleton points. By simultaneously convolving space and time, ST-GCN can deeply explore the dynamic features and spatial relationships of skeleton points, thereby improving the accuracy and robustness of action recognition. The model parameters of the graph spatiotemporal convolutional neural network proposed in the present invention are shown in Table 2.

[0122] Among them, the dimension of the input layer is 2×T×N×M, which represents the skeleton point sequence data captured by two cameras, where T represents the length of the time series, N is the number of key points, and M is the number of cameras. After passing through the graph convolution layer GCN, the output feature map dimension is 64×T×N, where 64 represents the number of feature channels. In the temporal convolution layer TCN, the dimension remains 64×T×N, that is, the number of channels remains unchanged. After the combined operation of 6 layers of GCN and TCN, the number of channels of the feature map increases to 128, forming an output dimension of 128×T×N. Then, through the fully connected layer FC, the feature map is compressed into a vector of length 256. Finally, at the output layer, the dimension of the model becomes the number of action categories.

[0123] During the optimization process, we designed a comprehensive loss function L that combines the error of key point detection and the optimization goal of the convolution kernel:

[0124] (10)

[0125] in, is the position of the real key point, is the position predicted by the model, and J is the functional of the self-adjusting convolution kernel, which is used to optimize the feature extraction ability of the convolution kernel, expressed as:

[0126] (11)

[0127] By sensing the changes in inter-frame features, the self-adjusting convolution kernel can analyze the differences between video frames, thereby accurately identifying the body parts that have changed significantly in the current frame, ensuring more accurate key point detection. When these areas with large changes in body parts are detected, the self-adjusting convolution kernel will dynamically adjust the weight of the convolution calculation to enhance the computing power of these areas, while reducing the calculation of the background or static parts to optimize the overall performance.

[0128] In the scene constructed by multi-camera skeleton point sequences, the self-adjusting convolution kernel can process the input data from different perspectives separately, independently detect and adjust the weights of each convolution kernel, and ensure that the skeleton point sequence of each camera can accurately capture the posture changes of the human body. In the process of implementing the self-adjusting convolution kernel, the settings of key parameters are shown in Table 1. The change detection threshold is used to determine whether the feature change is large enough to trigger the weight adjustment; the weight adjustment factor determines the magnitude of the weight increase or decrease. The local perception window size affects the fineness of the detection range, while the weight update strategy controls the frequency and method of weight adjustment. The initial weight setting affects the default state of the convolution kernel, and the smoothing parameter ensures the stability of the weight adjustment. The response curve parameter determines the sensitivity of the weight adjustment to input changes, and the optimization target helps to find a balance between speed and accuracy. These parameters can be further optimized through hyperparameter tuning or automated search to achieve the best performance and effect.

[0129] Based on the BlazePose architecture, a lightweight convolutional neural network system designed for real-time reasoning on mobile devices, and combined with a customized self-adjusting convolution kernel, the deployment of skeletal point motion detection and registration algorithms on the edge side is realized. The core idea is to prune the BlazePose model, a mainstream human skeletal point recognition algorithm framework, and retrain the single-precision reasoning model using the instruction set-optimized self-adjusting convolution kernel, so as to realize the deployment of a lightweight real-time motion capture model on the edge side. In this way, the midpoint of a person's hips, the radius of the circle circumscribing the entire person, and the inclination angle of the straight line connecting the shoulder and the midpoint of the hips can be predicted, and a total of 33 human key points can be predicted. According to the proportion and direction information of the hands and feet, even very complex situations, such as specific yoga postures, can be tracked consistently. In addition, in combination with machine learning technology, the present invention uses the graph-cut random sampling consistency method Graph-Cut RANSAC to calculate the spatial transformation matrix, match and align orthogonal human key nodes, and output the high-precision human skeletal point position that eliminates occlusion.

[0130] Graph-Cut RANSAC improves the traditional RANSAC algorithm by optimizing the division of inliers and outliers from a global perspective, improving the efficiency and accuracy of model fitting, especially when dealing with noise and outliers. It introduces spatial consistency and uses a graph cut algorithm to optimize the judgment criteria of inliers to ensure the consistency of labels between neighboring points. This method significantly improves the fitting accuracy of the model by minimizing model residuals and penalizing inconsistent label points.

[0131] In the design of the skeleton point recognition algorithm, it is also necessary to solve the problem of standardized computer description of the action process and realize the general action decomposition model. In the algorithm processing process, the skeletonized video stream output by the multi-view acquisition system is firstly used as a variable, and the spatial position changes of each joint point within the time of T length are stored in the human figure topology. The spatial position change stream of the time length T in the human figure topology is multiplied to make it periodic, and then it is input into the graph spatiotemporal convolutional neural network ST-GCN for analysis. The characteristics of different actions are identified through the weak supervision method of human-computer interaction, and the action process events at each time point T are automatically extracted.

[0132] The extraction of process events in the human skeleton video stream serialization action decomposition method combined with the weakly supervised learning method of human-computer interaction improves the robustness of the model, especially when dealing with small probability data distribution and complex samples. The algorithm outputs data pairs with different time starting points and durations, representing the automatically extracted process events, and corresponding to the key points in the assessment criteria, thereby achieving high-precision real-time action capture and evaluation. Through these designs, the algorithm can not only describe and evaluate human actions in a standardized manner, but also effectively improve the detection accuracy and the ability to handle complex samples.

[0133] The design of the multi-view human posture data acquisition and processing system aims to systematically obtain high-quality human skeleton point data through a multi-view data acquisition device to meet the comprehensive perception needs of the embodied intelligent system for the environment and actions. The design includes multiple links such as data acquisition, preprocessing, storage, processing and transmission to ensure high accuracy and real-time performance of the data.

[0134] First, the equipment is initialized and calibrated, the camera and sensor are accurately calibrated, and time synchronization is set to ensure the consistency of multi-view data. Then, the video streams from different perspectives are collected simultaneously through orthogonal cameras, audio data is collected synchronously, and the video and voice data are stored. Then, the collected video streams and sensor data are denoised and filtered, and the raw data is converted into a structured frame sequence image data format suitable for further processing and analysis, such as JPEG, PNG or video file formats, such as MP4, AVI, to ensure that each frame of the image can be effectively extracted and processed.

[0135] In the skeleton point detection and processing module, the optimized BlazePose model is used to detect and locate the key points of the human skeleton in real time, and the graph spatiotemporal convolutional neural network ST-GCN is used to decompose and identify the action and generate a standardized action description. The processed data is stored in a large-capacity SSD, and a reasonable data buffer structure is used to ensure data integrity. At the same time, the audio data is collected and segmented by the voice endpoint detection technology VAD. After that, the voice annotation data is serialized into the video stream data for alignment and annotation, and the voice information label is added to the corresponding video segment. Finally, the data is transmitted to the remote server or local computing device in real time through wireless Wi-Fi, Bluetooth and wired Ethernet, USB 3.0 communication methods, and encryption and verification mechanisms are used to ensure the security and integrity of the data transmission process. Finally, the human-machine interface module provides intuitive device status display and control, and users can monitor the acquisition process and data quality in real time, dynamically adjust the acquisition parameters according to feedback, and optimize the acquisition effect.

[0136] This embodiment is applied to the 24-style Yang style Tai Chi training scene. Before the training begins, multi-view cameras are first installed at different locations in the Tai Chi practice field, and an orthogonal sensing platform is built to ensure that the camera can fully cover the range of motion of the practitioner. Through this multi-view arrangement, the multi-view human posture acquisition device can synchronously capture the whole body motion of the practitioner, generate skeleton points and high-definition video stream data, and provide data support for subsequent motion analysis. During the data acquisition process, the motion analysis system classifies the collected multimodal data according to posture and motion trend, such as 24 classic movements such as the left and right wild horse mane and white crane wings in Tai Chi. The system first performs preliminary motion decomposition on these movements, and stores each motion category in a preset human topology map to ensure that the data of each movement can match its specific posture model. Afterwards, these motion data are input into the convolutional neural network CNN and the graph spatiotemporal convolutional neural network ST-GCN for further analysis and feature extraction. In order to make the system more intelligent, a weakly supervised learning method of human-computer interaction is used. The system can automatically capture abnormal motion events according to the trainer's motion process and give real-time feedback. At the same time, speech decomposition technology is also applied to the audio data analysis of the 24-posture Yang Style Tai Chi. The system can automatically identify and segment the speech information in Tai Chi, align it with the video stream data, and add the corresponding speech tags to the video clips for subsequent analysis and learning. Finally, these data are stored in the corresponding data storage disk after manual review. Through these processes, the system can realize the intelligent collection, processing and storage of training data, providing comprehensive data support for the system under embodied intelligence.

[0137] The advantages of the multi-view data acquisition device applied to embodied intelligence of the present invention are:

[0138] Synchronously add voice information tags: The device decomposes the serialized video stream and audio data, adds voice tags to the video stream data in real time, and realizes automated data labeling for the entire process of human posture data set collection.

[0139] Comprehensive environmental perception: The device provides environmental data collection capabilities from multiple perspectives through an orthogonal layout of multiple cameras, solving the problems of occlusion and blind spots under a single perspective. For embodied intelligent systems, this comprehensive environmental perception capability is the basis for achieving high-precision motion capture and interaction, ensuring that the device can accurately perceive and understand various dynamic changes in the surrounding environment.

[0140] High-precision skeleton point recognition: Based on the optimized BlazePose model and graph spatiotemporal convolutional neural network ST-GCN, the device can process and analyze the collected images and sensor data in real time, achieve high-precision skeleton point detection and action recognition, and provide a high-quality human posture benchmark dataset for intelligent agents under embodied intelligence.

[0141] Process event serialization: Process event serialization can accurately capture actions and improve data processing efficiency in embodied intelligent multi-view data acquisition devices. This not only ensures the accuracy of real-time monitoring and feedback, but also improves the performance of data collection in various application scenarios.

[0142] Table 1

[0143] Parameter name describe value Change detection threshold The threshold for determining whether the feature map change is large enough to trigger the convolution kernel weight adjustment. threshold_value = 0.1 Weight Adjustment Factor Dynamically adjust the factor of the convolution kernel weight to increase or decrease the computational intensity. increase_factor=1.2decrease_factor=0.8 Local awareness window size The size of the perception window for detecting local changes in the input feature map determines the detection range. window_size=3x3 Weight update strategy Control the frequency and method of adjusting the convolution kernel weights and determine the update cycle and method. update_frequency=10update_method=stepwise Initial weight setting The initial weight configuration of the convolution kernel determines the default state when no changes are detected. initial_weights=standard Response curve parameters Defines the sensitivity curve of the convolution kernel weight to changes in input features. You can choose a linear or nonlinear relationship. response_curve = linear Optimization goals Specify the goal of adaptive convolution kernel optimization to balance speed and accuracy. optimization_goal = "balance"

[0144] Table 2

[0145] level type Parameter Description Output Dimensions 1 Input Layer Input skeleton point sequence, shape is (N, T, V, C) 2×T×N×M 2 Graph Convolutional Layer GCN Using graph convolution operation, each node calculates the feature representation of the bone point 64×T×N 3 Temporal Convolutional Layer TCN 1D temporal convolution kernel, processing the temporal dimension of the skeleton point sequence 64×T×N 4 Multi-layer GCN+TCN Multi-layer stacking increases model depth and expressiveness 128×T×N 5 Fully connected layer Mapping convolutional features to a standardized action description space 256 6 Output Layer Output standardized action description, suitable for further action recognition or classification tasks C

Claims

1. A data processing method for a multi-view human posture data acquisition device for embodied intelligence, characterized in that: Step 1: Audio data endpoint detection and interception: Step 1-1: segment the audio data into corresponding video frames, and extract short-time energy features, frequency features and zero-crossing rate features from each audio data segment one by one; Calculate the short-time energy of each audio data segment: （1） where E\left [ {m} \right ] is the short-time energy of the mth audio data segment; N is the number of samples in each audio data segment, x\left [ {n} \right ] is the amplitude value of the nth audio signal; Calculate the spectrum centroid C and spectrum flatness P: （2） （3） Where X(f k ) is the audio data x\left [ {n} \right ] The spectrum, f k is the kth frequency component; Calculate the zero-crossing rate: （4） Where, sgn() is the sign function; T is the time length of the audio data segment; Step 1-2: Calculate comprehensive features: （5） in, , and Represents short-time energy The threshold of the zero-crossing rate and the threshold of the spectrum centroid, The threshold indicating the flatness of the spectrum; Step 1-3: Combine the comprehensive features CF Compare with a preset threshold, if it is greater than the preset threshold, identify it as a speech signal segment, otherwise, identify it as a non-speech signal segment; Step 1-4: Use VAD technology to detect the starting point and ending point of the speech signal segment; Step 1-5: After the audio data segment is segmented according to the result y of the speech endpoint detection, the identification label of the corresponding video frame is added; Step 2: Predict the position of human key points: Use the optimized BlazePose model to detect skeleton key points frame by frame on the preprocessed video stream data; The BlazePose model includes a coarse positioning module and a key point fine positioning module. The coarse positioning module extracts the preliminary position of the human body through a lightweight convolutional neural network, and the fine positioning module refines the position of each key point; the fine positioning module includes a feature change detection unit, a parameter adjustment unit, a parameter learning unit, and a parameter optimization unit; the feature change detection unit uses a local difference method to calculate the difference between adjacent frames in time or the gradient change in space to generate a change perception map; the parameter adjustment unit dynamically adjusts the weight of the self-adjusting convolution kernel based on the generated change perception map; the parameter learning unit adjusts the model parameters by learning parameter control under the control of the parameter adjustment unit: （6） Where K(x,y) is the value of the self-tuning convolution kernel at position (x,y), a and b are learnable parameters, f(x,y) represents the local description of the input features, and σ is the activation function; The objective function of the parameter optimization unit is: （7） in, is the position of the real key point, is the position predicted by the model, and J is the functional of the self-adjusting convolution kernel, which is used to optimize the feature extraction ability of the convolution kernel, expressed as: （8） Step 3: Matching of key human nodes: Use the Graph-Cut RANSAC method to calculate the spatial transformation matrix, and match and align orthogonal key human nodes; Step 4: Generate a time-space diagram of key human nodes: take the key human nodes of each video stream data frame as nodes, and establish edges corresponding to the key human nodes of adjacent video stream data frames; establish edges for the key human nodes in each video stream data frame according to a preset human topology diagram, and obtain a time-space diagram of key human nodes; Step 5: Action decomposition and description: Combined with the spatiotemporal graph convolutional neural network ST-GCN, the standardized action decomposition and description are achieved; A spatiotemporal graph convolutional neural network ST-GCN is established and trained using a preset training set. The spatiotemporal graph convolutional neural network ST-GCN includes an input layer, a processing unit consisting of a graph convolutional network GCN and a temporal convolutional network TCN, and an output layer. The skeletonized video stream output by the multi-view acquisition system is encoded according to the human figure topology, and is time-extended according to the preset multiplication parameter value. Then, it is input into the graph spatiotemporal convolutional neural network ST-GCN for analysis. The characteristics of different actions are identified through the weak supervision method of human-computer interaction, and the action process events at each time point T are automatically extracted.

2. The data processing method according to claim 1, characterized in that: The steps 1-4 use the SGHTR model to perform speech endpoint detection; the SGHTR model first calculates the short-time energy features of the speech signal segment through the short-time energy module, and uses the zero-crossing rate method to extract the zero-crossing features of the speech signal segment; the Gaussian mixture model G is introduced to fit the Gaussian distribution characteristics of the short-time energy features through a combination of more than one Gaussian distribution; the hidden Markov model H uses the state set S, transition probability A and observation probability B parameters to analyze the state transition characteristics of the speech data segment on this basis; the Transformer module T captures the long-distance context information of the speech signal segment through a multi-head attention mechanism, a feedforward layer and a normalization operation; the recurrent neural network RNN ​​analyzes the short-time energy features, the zero-crossing features, the Gaussian distribution characteristics of the short-time energy features, and the state transition characteristics of the speech data segment, and outputs the speech endpoint detection result; the functional form of the SGHTR model can be expressed as: （9） Among them, S is the short-time energy module, G is the Gaussian mixture model module, H is the hidden Markov model module, T is the Transformer module, and R is the recurrent neural network module.

3. The data processing method of the multi-view human posture data acquisition device for embodied intelligence according to claim 2, characterized in that: The BlazePose model is pruned and its single-precision inference model is retrained using the instruction set-optimized self-adjusting convolution kernel to achieve the deployment of a lightweight real-time motion capture model on the edge.

4. A multi-view human posture data acquisition device for embodied intelligence using the processing method of claim 1, characterized in that: It includes a data acquisition unit, a control and data processing unit, a storage unit, a communication module, a human-machine interface module and an auxiliary equipment module; The data acquisition unit includes an orthogonal visual perception platform built by two imaging cameras and a microphone. The optical axes of the two imaging cameras of the orthogonal visual perception platform intersect to collect time-stamped human skeleton point video stream data from two different angles, and the microphone collects corresponding time-stamped audio data. The control and data processing unit extracts the process events of the serialized actions based on the video stream data of the human skeleton points, and realizes a general action decomposition model that can be evaluated in the process; The storage unit is responsible for storing the video stream data of human skeleton points; The communication module is responsible for data transmission; The human-computer interface is responsible for interacting with the user; The auxiliary device module is responsible for sending and receiving audio data.

5. The multi-view human posture data acquisition device for embodied intelligence according to claim 4, characterized in that: Of the two imaging cameras, camera A is located directly in front of the object, and camera B is located on the right side of the object. The two cameras are installed perpendicular to each other.

6. The multi-view human posture data acquisition device for embodied intelligence according to claim 4, characterized in that: It also includes a power management module, which has a real-time monitoring function for battery power and provides low-power warning and automatic shutdown protection.

7. The multi-view human posture data acquisition device for embodied intelligence according to claim 4, characterized in that: It also includes a communication module, which is used to provide a data transmission interface.

Images