Gesture operation voice playing output method based on end-cloud collaboration

Abstract

The invention relates to the technical field of man-machine interaction and voice processing, in particular to a gesture operation voice playing output method based on end-cloud collaboration, which comprises the following steps: S1, collecting gesture motion data of a user in real time; s2, extracting a bone point coordinate change rate, and generating a compression feature sequence; s3, segmenting the compressed feature sequence into a privacy related feature layer and a behavior representation feature layer; s4, dynamically matching the behavior representation feature layer with a preset semantic template; s5, selecting the target semantics with the highest weighted score and the associated voice template; and S6, encoding the voice template into streaming media data, and distributing the streaming media data to local equipment for voice broadcast. According to the method, multi-mode perception, dynamic semantic matching and edge streaming media coding are fused, high precision of gesture recognition and semantic analysis is achieved, meanwhile, user privacy protection and real-time voice interaction are ensured, and system response efficiency and application safety are remarkably improved.

Description

Technical Field

[0001] The present invention relates to the field of human-computer interaction and voice processing technology, and in particular to a gesture-operated voice playback and output method based on end-cloud collaboration. Background Art

[0002] In recent years, with the rapid development of human-computer interaction technology, contactless control methods based on gesture recognition have gradually been applied to multiple scenarios such as smart homes, in-vehicle systems, and smart wearable devices. In particular, in the field of voice playback control, users have an increasing demand for starting, stopping, switching, and adjusting the volume of audio content through gestures. However, gesture recognition systems often need to integrate multiple sensors to collect high-dimensional dynamic features, and at the same time, they need to perform semantic matching based on user intentions and drive voice output. The entire process places high demands on recognition accuracy, response speed, privacy protection, and semantic understanding capabilities. In this context, end-cloud collaborative processing architecture has gradually become a key path to improve recognition efficiency and accuracy.

[0003] However, several key issues remain unresolved in existing technologies: First, the raw gesture data is high-dimensional and noisy, lacking efficient compression and layered processing mechanisms, resulting in significant data upload delays and low recognition efficiency. Second, there is a lack of effective isolation strategies for user privacy features, which can result in data containing sensitive information such as facial and torso information being uploaded to the cloud, posing a risk of privacy leakage. Third, during semantic recognition, existing methods struggle to dynamically adjust semantic matching priorities based on user contextual operating habits, resulting in low semantic judgment accuracy. Fourth, in the voice output phase, existing solutions often rely on static file broadcasts, lacking structured encoding and distribution mechanisms for streaming media, making them difficult to adapt to the real-time playback requirements of multiple terminals. Therefore, a gesture-based voice playback and output method based on end-cloud collaboration is urgently needed to address the above issues. Summary of the Invention

[0004] Based on the above objectives, the present invention provides a gesture-operated voice playback and output method based on end-cloud collaboration.

[0005] A gesture-operated voice playback and output method based on end-cloud collaboration includes the following steps:

[0006] S1: Collect user gesture motion data in real time and generate an original gesture feature set including spatial coordinates, motion trajectory, and dwell time;

[0007] S2: Perform frame compression processing on the original gesture feature set, extract the change rate of the skeleton point coordinates, and generate a compressed feature sequence;

[0008] S3: Split the compressed feature sequence into a privacy-related feature layer and a behavior representation feature layer, and upload the behavior representation feature layer to the cloud server;

[0009] S4: The cloud server dynamically matches the behavior representation feature layer with the preset semantic template based on the dynamic semantic weight database to generate a candidate semantic set with confidence scores;

[0010] S5: Contextually weight the candidate semantic set based on the user's historical operation data, and select the target semantic and associated speech template with the highest weighted score;

[0011] S6: Encode the speech template corresponding to the target semantics into streaming data and distribute it to the local device through the edge node for voice broadcast.

[0012] Optionally, the S1 specifically includes:

[0013] S11: Initiates collaborative perception between the multispectral camera array and millimeter-wave radar, synchronously capturing infrared thermal images, depth point cloud data, and Doppler frequency shift signals of the gesture area at a 100Hz sampling rate.

[0014] S12: Perform spatiotemporal alignment processing on the original perception signal, use adaptive Kalman filtering to eliminate environmental reflection interference, and generate a denoised three-dimensional spatial coordinate data set;

[0015] S13: Capture hand motion angular velocity and acceleration through an embedded inertial measurement unit, perform sensor fusion on the measurements with visual data, and calculate the gesture transformation matrix between consecutive frames using quaternion interpolation.

[0016] S14: Track the motion trajectory of the index finger joint based on the gesture transformation matrix. Use the Euclidean distance change rate between adjacent coordinate points exceeding 10 cm / s as the segmentation threshold to divide the continuous motion into multiple trajectory segments and record the start timestamp, end timestamp and maximum motion speed of each trajectory segment.

[0017] S15: Detect the state of the hand staying in the preset sensitive area. When the same spatial coordinate point is maintained for a time of ≥300ms and the speed variance is <5cm 2 / s 2 When , it is marked as the effective stay time;

[0018] S16: Integrate the spatial coordinate data set, motion trajectory segment parameters, and dwell time information according to the time axis and encapsulate them into an original gesture feature set with a unique device identification code.

[0019] Optionally, the S2 specifically includes:

[0020] S21: Divide the original gesture feature set into frames according to a fixed time interval, setting the time interval to Δt, and grouping the three-dimensional space coordinate data within each time interval of Δt into a frame to obtain a frame sequence set;

[0021] S22: For a specified skeleton point in each frame, the displacement increment per unit time between adjacent frames is calculated based on the change in its three-dimensional coordinates, and the change rate of the skeleton point coordinates is defined as: Among them, V i (y) is the coordinate change rate of the i-th bone point at time t, P i (t) is its three-dimensional coordinate position at time t;

[0022] S23: Threshold filtering is performed on the calculated coordinate change rate to eliminate point data with a change rate less than the set threshold;

[0023] S24: normalize and encode the skeleton point number, change rate, and corresponding timestamp retained in each frame, and splice them into a time series in frame order to obtain a compressed feature sequence;

[0024] S25: In the compressed feature sequence, add the start timestamp, end timestamp and corresponding frame index of each segment to achieve time-based structured storage of trajectory segments.

[0025] Optionally, the S3 specifically includes:

[0026] S31: performing field analysis on each frame feature in the compressed feature sequence to extract data including skeleton point number, coordinate change rate, timestamp, movement speed and dwell time;

[0027] S32: Establishing a privacy-related feature determination rule, identifying the bone points in the bone point numbers that are related to private parts of the face or shoulders, and classifying the corresponding fields into a privacy-related feature layer;

[0028] S33: The remaining motion information fields not involving highly sensitive parts, including the coordinate change rate of the hand trajectory segment, velocity parameters and their corresponding time series data, are divided into a behavior representation feature layer;

[0029] S34: Locally encrypt and store the data of the privacy-related feature layer and set access control permissions; uniformly encode and normalize the data of the behavior representation feature layer.

[0030] Optionally, the S4 specifically includes:

[0031] S41: After receiving the uploaded behavior representation feature layer, the cloud server performs normalization processing on each trajectory segment according to the timestamp, skeleton point number and coordinate change rate information to generate a time sequence action vector in a unified format;

[0032] S42: Retrieving a set of semantic templates activated in the current time window from the dynamic semantic weight database, each semantic template including a standard action vector pattern and its associated semantic description, to form a semantic matching candidate library;

[0033] S43: Using a matching mechanism based on multi-dimensional feature comparison, the uploaded temporal motion vector is matched one by one with each semantic template in the semantic matching candidate library, and feature similarity is calculated based on the dimensions of trajectory shape, speed pattern, and dwell rhythm;

[0034] S44: Based on the dimension consistency and similarity distribution during the matching process, a semantic weight parameter is dynamically introduced to increase the weight of the semantic template that is consistent with the user's historical behavior preferences and reduce the weight coefficient of irrelevant templates;

[0035] S45: After fusing the semantic weight value of the semantic template itself with the action vector similarity index, a confidence score is assigned to each candidate semantic, and the scores are sorted from high to low. The top N items with the confidence scores are output to form the candidate semantic set.

[0036] Optionally, the S43 specifically includes:

[0037] S431: Segment the uploaded temporal motion vector and split the complete vector into several subsequences according to the start and end timestamps of the trajectory segments. Each subsequence contains continuous skeletal point numbers, coordinate change rates, velocity characteristics, and dwell marks.

[0038] S432: For each semantic template in the semantic matching candidate library, extract its corresponding standard action pattern sequence, perform normalization processing, and construct a template standard vector set;

[0039] S433: For each subsequence and the semantic template standard vector, perform dimension-by-dimension comparison at the same time step to calculate the feature similarity score S sim , the similarity score lies in the interval [0, 1];

[0040] S434: When the similarity score S sim When it is greater than the set matching threshold of 0.7, the corresponding semantic template is marked as a valid match.

[0041] Optionally, the S44 specifically includes:

[0042] S441: After completing the preliminary similarity score between the uploaded gesture subsequence and the semantic template standard vector, count the dimensional consistency ratio in each template match and calculate the dimensional consistency rate R c ;

[0043] S442: Perform sliding window processing on the matching score sequence of each template and calculate the similarity variance σ of the template in adjacent matching tasks 2 ;

[0044] S443: Dynamically adjust the semantic weight parameter W of the semantic template based on the above two indicatorss , the adjustment formula is: W s =W0×(1+α×R c )×(1-β×σ 2 ), where α=0.6 and β=0.4 are empirical optimization coefficients; W0 is the original weight.

[0045] Optionally, the S45 specifically includes:

[0046] S451: Calculate the confidence score of the candidate semantics based on its selection frequency in historical tasks. The formula is: C s =λ1·S sim +λ2·W s +λ3·F h , where C s is the confidence score of the candidate semantics, λ1=0.5, λ2=0.3, λ3=0.2; F h The selection frequency of candidate semantics in historical tasks;

[0047] S452: Confidence score C s Rank all candidate semantics from high to low.

[0048] Optionally, the S5 specifically includes:

[0049] S51: Get the candidate semantic set output by S45 and extract the confidence score C corresponding to each semantic in the candidate semantic set s ;

[0050] S52: Retrieve the target semantic sequence selected by the user's past 30 consecutive gesture operations from the user's historical operation database stored in the cloud, and construct the target semantic transition probability matrix M ij ;

[0051] S53: Obtain the historical target semantic index of the user's most recent selection as the current context benchmark, and define the corresponding transition probability value between each semantic in the candidate semantic set and the current context benchmark in the transition probability matrix as the context relevance weight W c ;

[0052] S54: For each candidate semantics, the context relevance weight W c Confidence score C calculated with S45 s Perform fusion calculation to obtain the final context weighted score S f ;

[0053] S55: Score S based on context weighting fSort the candidate semantic sets, select the semantic with the highest score after sorting as the target semantic of this recognition task, and call the cloud-based semantic-speech mapping database to extract the associated speech template corresponding to the target semantic.

[0054] Optionally, the S6 specifically includes:

[0055] S61: After receiving the target semantics and the corresponding voice template ID determined in S5, the cloud server retrieves the text content and audio file resources of the corresponding voice template from the semantic-voice mapping database, and reads the voice broadcast format and version information corresponding to the template;

[0056] S62: performing frame processing on the retrieved audio file, dividing the entire audio segment into a continuous frame sequence using a fixed frame length, and adding a timestamp, a frame index number, and a checksum to each frame to construct an original audio frame stream with time synchronization information;

[0057] S63: According to the audio stream protocol format supported by the terminal device, a corresponding audio coding standard is selected for transcoding processing to generate an encoded .ts slice sequence and a .m3u8 index file;

[0058] S64: Cache the encoded streaming media data in order to the edge node distribution module, and simultaneously generate a unique resource location identifier URL for content distribution;

[0059] S65: The local device pulls the corresponding streaming media content through the edge node, and decodes and plays it in order according to the timestamp.

[0060] Beneficial effects of the present invention:

[0061] The present invention realizes an efficient and low-latency behavioral feature uploading mechanism through frame compression and layered extraction of original gesture features. By setting skeleton point privacy judgment rules, data related to the head, shoulders and other parts are clearly isolated and processed locally, which significantly improves the user privacy protection capability. At the same time, a dynamic semantic weight database is constructed in the cloud to comprehensively evaluate the semantic matching results based on trajectory similarity, user historical preferences and context relevance, thereby generating a candidate semantic set that is more in line with the user's intention, improving the overall semantic recognition accuracy and the intelligent level of system response.

[0062] The present invention adopts edge node distribution and streaming media encoding mechanism in the voice output link to convert the voice template corresponding to the target semantics into a structured audio frame stream, thereby realizing real-time encoding, sequential distribution and smooth playback of voice content; this method supports concurrent access and on-demand loading of multiple terminals, improving the system's voice response fluency and scalability in complex usage environments. BRIEF DESCRIPTION OF THE DRAWINGS

[0063] In order to more clearly illustrate the technical solutions in the present invention or the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are only for the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0064] Figure 1 A schematic diagram of a gesture-operated voice playback and output method according to an embodiment of the present invention;

[0065] Figure 2 Schematic diagram of a method for generating candidate semantic sets according to an embodiment of the present invention. DETAILED DESCRIPTION

[0066] The present invention is described in detail below with reference to the accompanying drawings and specific embodiments. It is also noted that, to provide a more detailed description, the following embodiments are best and preferred embodiments, and those skilled in the art may employ alternative methods for implementing certain known technologies. Furthermore, the accompanying drawings are intended only to provide a more detailed description of the embodiments and are not intended to limit the present invention.

[0067] like Figure 1-Figure 2 As shown, a gesture-operated voice playback and output method based on end-cloud collaboration includes the following steps:

[0068] S1: Collect user gesture motion data in real time and generate an original gesture feature set including spatial coordinates, motion trajectory, and dwell time;

[0069] S2: Perform frame compression processing on the original gesture feature set, extract the change rate of the skeleton point coordinates, and generate a compressed feature sequence;

[0070] S3: Split the compressed feature sequence into a privacy-related feature layer and a behavior representation feature layer, and upload the behavior representation feature layer to the cloud server;

[0071] S4: The cloud server dynamically matches the behavior representation feature layer with the preset semantic template based on the dynamic semantic weight database to generate a candidate semantic set with confidence scores;

[0072] S5: Contextually weight the candidate semantic set based on the user's historical operation data, and select the target semantic and associated speech template with the highest weighted score;

[0073] S6: Encode the speech template corresponding to the target semantics into streaming data and distribute it to the local device through the edge node for voice broadcast.

[0074] S1 specifically includes:

[0075] S11: Initiates collaborative perception between the multispectral camera array and millimeter-wave radar, synchronously capturing infrared thermal images, depth point cloud data, and Doppler frequency shift signals of the gesture area at a 100Hz sampling rate.

[0076] S12: Performing spatiotemporal alignment processing on the original perception signal, using adaptive Kalman filtering to eliminate environmental reflection interference, and generating a denoised 3D spatial coordinate dataset. The 3D spatial coordinate dataset contains the (x, y, z) coordinate sequence of the palm key points relative to the device reference plane.

[0077] S13: Capture hand motion angular velocity and acceleration through an embedded inertial measurement unit, perform sensor fusion on the measurements with visual data, and calculate the gesture transformation matrix between consecutive frames using quaternion interpolation.

[0078] The specific steps for calculating the gesture transformation matrix between consecutive frames are as follows:

[0079] Get adjacent frame quaternions: Extract the attitude quaternion Q from the sensor fusion results of the previous frame and the next frame respectively k and Q k+1 , its expression is: Q k =(w k ,x k ,y k ,z k ), where w k is the real part, x k ,y k ,z k is the imaginary part; Q k+1 =(w k+1 ,x k+1 ,y k+1 ,z k+1 ), where w k+1 is the real part, x k+1 ,y k+1 ,z k+1 is the imaginary part;

[0080] Calculate the quaternion angle: First calculate the dot product S of the two quaternions, and then calculate the angle θ based on the dot product; the formula is as follows: S = w k ×w k+1 +x k ×x k+1 +y k ×y k+1 +z k ×z k+1 ; θ = arccos(S); Perform spherical linear interpolation: Set the interpolation factor t, interpolate the quaternion in the range of 0 ≤ t ≤ 1, and obtain the interpolated quaternion Q(t). The formula is: Where Q(t) = (w(t), x(t), y(t), z(t)), where w(t) is the real part and x(t), y(t), z(t) are the imaginary parts.

[0081] Generate the posture transformation matrix from the interpolated quaternion: convert Q(t) into the rotation matrix R(t), and then add the part with zero translation component to form the 4×4 gesture posture transformation matrix T(t). The expressions are:

[0082]

[0083] Where 0 is a 3×1 zero vector, and the matrix T(t) can realize the posture transformation from the previous frame to the next frame.

[0084] S14: Track the motion trajectory of the index finger joint based on the gesture transformation matrix. Use the Euclidean distance change rate between adjacent coordinate points exceeding 10 cm / s as the segmentation threshold to divide the continuous motion into multiple trajectory segments and record the start timestamp, end timestamp and maximum motion speed of each trajectory segment.

[0085] S15: Detect the state of the hand staying in the preset sensitive area. When the same spatial coordinate point is maintained for a time of ≥300ms and the speed variance is <5cm 2 / s 2 When , it is marked as the effective stay time;

[0086] S16: Integrate the spatial coordinate data set, motion trajectory segment parameters, and dwell time information along the time axis and encapsulate them into an original gesture feature set with a unique device identification code. The feature set data format is as follows:

[0087] Header fields: device type code, sampling time base

[0088] Data body: [coordinate triplet, trajectory segment number, stop status flag] stored by frame

[0089] Verification code: A data integrity verification code generated based on the CRC-32 algorithm.

[0090] S2 specifically includes:

[0091] S21: Divide the original gesture feature set into frames according to a fixed time interval, setting the time interval to Δt, and grouping the three-dimensional space coordinate data within each Δt time interval into a frame to obtain a frame sequence set;

[0092] S22: For a specified skeleton point in each frame, the displacement increment per unit time between adjacent frames is calculated based on the change in its three-dimensional coordinates, and the change rate of the skeleton point coordinates is defined as: Among them, V i(y) is the coordinate change rate of the i-th bone point at time t, P i (t) is its three-dimensional coordinate position at time t;

[0093] S23: Threshold filtering is performed on the calculated coordinate change rate to eliminate point data with a change rate less than the set threshold, and only skeleton points with significant motion characteristics are retained for feature representation after compression;

[0094] S24: normalize and encode the skeleton point number, change rate, and corresponding timestamp retained in each frame, and splice them into a time series in frame order to obtain a compressed feature sequence;

[0095] S25: In the compressed feature sequence, add the start timestamp, end timestamp and corresponding frame index to each segment to achieve time-based structured storage of trajectory segments; through the above steps, the original gesture feature set is compressed and encoded, effectively reducing data redundancy while retaining key dynamic change information, providing an efficient and lightweight input data structure for subsequent behavior representation feature extraction and uploading to the cloud for processing, thereby improving the response speed and recognition accuracy of the overall system.

[0096] S3 specifically includes:

[0097] S31: performing field analysis on each frame feature in the compressed feature sequence to extract data including skeleton point number, coordinate change rate, timestamp, movement speed and dwell time;

[0098] S32: Establishing a privacy-related feature determination rule, based on human body recognition relevance, spatial positioning sensitivity, and individual behavioral habit characteristics, identifying the skeletal points in the skeletal point numbers that involve private parts of the face or shoulders, and classifying their corresponding fields into a privacy-related feature layer;

[0099] The privacy-related feature determination rules are established as follows:

[0100] Identification area determination rules: If the area to which a bone point number belongs is a preset high-sensitivity area, including the head, neck, shoulder, and chest area, all coordinate change rates, velocity parameters, and dwell time fields associated with the bone point are designated as privacy-related features. This rule is based on the one-to-one correspondence between bone numbers and anatomical locations in the skeleton mapping table.

[0101] Rules for determining exposure of individual identity features: If the time series trajectory between skeletal points has obvious individual identity recognition features, such as the head-shoulder-torso region showing a fixed-proportion movement pattern in the time series, and the similarity score with the feature vector in the historical identity database exceeds 0.8, then the corresponding skeletal points and their trajectory parameters are classified as privacy-related features;

[0102] Spatial position sensitivity determination rule: If a certain skeleton point frequently appears in a preset sensitive area (such as within a 15° viewing cone of the camera center) within a set time window (such as within 1 second), and its coordinate change range is less than the set threshold (0.1m), then the skeleton point is determined to have high spatial sensitivity, and its spatiotemporal data is classified as privacy-related features.

[0103] Table 1 Skeleton mapping

[0104]

[0105]

[0106] As can be seen from Table 1 above, when executing the "Identification Area Determination Rules" in S32, the numbered items (0-5) in the table with "Is it a privacy-related part" set to "Yes" can be directly referenced as the privacy skeleton point set; for skeleton points numbered 0-5, all their time series coordinate change rate, speed, and stay information are included in the privacy-related feature layer; the corresponding parts numbered 6 and above are classified as the non-privacy behavior representation feature layer for upload to the cloud for processing.

[0107] S33: The remaining motion information fields not involving highly sensitive parts, including the coordinate change rate of the hand trajectory segment, velocity parameters and their corresponding time series data, are divided into a behavior representation feature layer;

[0108] S34: Locally encrypt and store the data of the privacy-related feature layer and set access control permissions; uniformly encode and normalize the behavior representation feature layer data; upload the behavior representation feature layer to the cloud server, and retain the privacy-related feature layer in the local device for subsequent local training or auxiliary decision-making; through the above steps, the information in the compressed feature sequence is classified and split, while ensuring the accuracy of system behavior recognition, local isolation and encryption protection of user sensitive features are achieved, effectively meeting the data privacy protection requirements in gesture recognition scenarios, and ensuring that the uploaded data is parseable and highly versatile.

[0109] S4 specifically includes:

[0110] S41: After receiving the uploaded behavior representation feature layer, the cloud server performs normalization processing on each trajectory segment according to the timestamp, skeleton point number and coordinate change rate information to generate a time sequence action vector in a unified format;

[0111] S42: Retrieving a set of semantic templates activated in the current time window from the dynamic semantic weight database, each semantic template including a standard action vector pattern and its associated semantic description, to form a semantic matching candidate library;

[0112] S43: Using a matching mechanism based on multi-dimensional feature comparison, the uploaded temporal motion vector is matched one by one with each semantic template in the semantic matching candidate library, and feature similarity is calculated based on the dimensions of trajectory shape, speed pattern, and dwell rhythm;

[0113] S44: Based on the dimension consistency and similarity distribution during the matching process, a semantic weight parameter is dynamically introduced to increase the weight of the semantic template that is consistent with the user's historical behavior preferences and reduce the weight coefficient of irrelevant templates;

[0114] S45: After fusing the semantic weight value of the semantic template itself with the action vector similarity index, a confidence score is assigned to each candidate semantic, and the scores are sorted from high to low. The top N items with the confidence scores are output to form a candidate semantic set for subsequent semantic confirmation.

[0115] S43 specifically includes:

[0116] S431: Segment the uploaded temporal motion vector and split the complete vector into several subsequences according to the start and end timestamps of the trajectory segments. Each subsequence contains continuous skeletal point numbers, coordinate change rates, velocity characteristics, and dwell marks.

[0117] S432: For each semantic template in the semantic matching candidate library, extract its corresponding standard action pattern sequence, perform normalization processing, and construct a template standard vector set;

[0118] S433: For each subsequence, compare it with the semantic template standard vector dimension by dimension at the same time step. Assume that the uploaded vector is: A = {a1, a2, ..., a n}, B={b1, b2, ..., b n}; Calculate feature similarity score S sim , the formula is: Among them, S sim is the similarity score, a i with b i denote the normalized feature values ​​of the uploaded sequence and the semantic template in the i-th dimension, w i is the semantic weight of the z-th dimension feature, D max is the normalized maximum distance constant, and the similarity score is in the interval [0, 1];

[0119] S434: When the similarity score S simWhen the value is greater than the set matching threshold of 0.7, the corresponding semantic template is marked as a valid match, and the matching score between the corresponding subsequence and the template is recorded for subsequent confidence calculation and sorting. Through the above steps, the system can perform high-dimensional structural matching of the uploaded temporal gesture features and multi-semantic templates in the cloud, and ensure the stability and discrimination of the matching results through a standardized similarity scoring mechanism, thereby providing a reliable matching basis for subsequent semantic scoring and target selection.

[0120] S44 specifically includes:

[0121] S441: After completing the preliminary similarity score between the uploaded gesture subsequence and the semantic template standard vector, count the dimensional consistency ratio in each template match and calculate the dimensional consistency rate R c , the expression is: Among them, N s To satisfy |a i -b i |≤δ, where δ is the set similarity threshold and is set to 0.15; N is the total number of dimensions;

[0122] S442: Perform sliding window processing on the matching score sequence of each template and calculate the similarity variance σ of the template in adjacent matching tasks 2 , if σ 2 <0.02, indicating high matching stability;

[0123] S443: Dynamically adjust the semantic weight parameter W of the semantic template based on the above two indicators s , the adjustment formula is: W s =W0×(1+α×R c )×(1-β×σ 2 ), where α = 0.6 and β = 0.4 are empirical optimization coefficients to ensure that templates with high consistency and good stability are given a higher weight, while unstable templates are given a lower weight; W0 is the original weight; and finally the adjusted semantic weight W s Update the corresponding template item in the candidate semantic set for subsequent confidence score calculation; by introducing dimension consistency rate and matching stability indicators, and combining with weighted adjustment strategies, the system can dynamically tune the semantic weights for each gesture recognition task, so that semantic templates with structural consistency and stable historical matching are prioritized in the candidate set, thereby improving the overall recognition accuracy and semantic expression fit.

[0124] S45 specifically includes:

[0125] S451: Calculate the confidence score of the candidate semantics based on its selection frequency in historical tasks. The formula is: C s =λ1·Ssim +λ2·W s +λ3·F h , where C s is the confidence score of the candidate semantics, λ1 = 0.5, λ2 = 0.3, λ3 = 0.2, and the sum of the three weight coefficients is 1, ensuring that the fusion score is linearly additive; F h is the selection frequency of candidate semantics in historical tasks, Among them, n h is the number of historical selections, N1 is the total number of identifications, and the minimum number of participation times is set to 10 times;

[0126] S452: Confidence score C s All candidate semantics are sorted from high to low; the confidence scoring mechanism constructed through the above steps realizes the integrated evaluation of candidate semantics from three aspects: matching accuracy, semantic adaptability and historical selection trends. This makes the final generated candidate semantic set have higher selection rationality and context relevance, enhancing the accuracy of semantic parsing and personalized expression capabilities.

[0127] S5 specifically includes:

[0128] S51: Get the candidate semantic set output by S45 and extract the confidence score C corresponding to each semantic in the candidate semantic set s ;

[0129] S52: Retrieve the target semantic sequence selected by the user's past 30 consecutive gesture operations from the user's historical operation database stored in the cloud, and construct the target semantic transition probability matrix M ij , defined as: Among them, T ij represents the number of times the semantics of the i-th word has been transferred to the semantics of the j-th word in history, T i represents the total number of times the i-th semantics is selected as the target semantics;

[0130] S53: Obtain the historical target semantic index of the user's most recent selection as the current context benchmark, and define the corresponding transition probability value between each semantic in the candidate semantic set and the current context benchmark in the transition probability matrix as the context relevance weight W c ;

[0131] S54: For each candidate semantics, the context relevance weight W c Confidence score C calculated with S45 s Perform fusion calculation to obtain the final context weighted score S f , the calculation formula is: S f =μ1·C s +μ2·W c, where the weight coefficients are μ1 = 0.7, μ2 = 0.3, and μ1 + μ2 = 1, which is used to highlight the association strength between the current gesture and the context history;

[0132] S55: Score S based on context weighting f The candidate semantic set is sorted, and the semantic with the highest score after sorting is selected as the target semantic of this recognition task. The cloud-based semantic-speech mapping database is called to extract the associated speech template corresponding to the target semantic. The scoring mechanism that integrates the contextual historical operation data and the current candidate semantics implemented through the above steps makes the selection of the target semantic not only rely on the action characteristics of the immediate gesture, but also fully integrates the user's historical operation preferences and contextual scene factors, further improving the accuracy of semantic selection.

[0133] Table 2 Cloud semantic-speech mapping database

[0134]

[0135] In Table 2 above, the semantic number is used within the system to uniquely identify a semantic intent; the target semantic text is used to describe the specific user intent in the candidate semantics; the voice template ID is used to call the speech synthesis engine or search for existing recording resources; the voice content example is used to indicate the standard broadcast voice text bound to the semantics; the audio format is used for the format of the voice file, commonly MP3, WAV, etc.; the template version is the version number used by the current voice template, which supports regular updates or personalized optimization.

[0136] S6 specifically includes:

[0137] S61: After receiving the target semantics and the corresponding voice template ID determined in S5, the cloud server retrieves the text content and audio file resources of the corresponding voice template from the semantic-voice mapping database, and reads the voice broadcast format and version information corresponding to the template;

[0138] S62: Frame the retrieved audio file, using a fixed frame length (e.g., 20ms) to segment the entire audio segment into a continuous frame sequence, and add a timestamp, frame index number, and checksum to each frame to construct an original audio frame stream with time synchronization information;

[0139] S63: According to the audio stream protocol format supported by the terminal device, the corresponding audio coding standard is selected for transcoding. If the target format is the HLS protocol, the original audio frame stream is re-encapsulated using the AAC encoding method to generate an encoded .ts slice sequence and a .m3u8 index file;

[0140] S64: Cache the encoded streaming media data in order to the edge node distribution module, and simultaneously generate a unique resource location identifier URL for content distribution to ensure that the edge device can obtain the data frame on demand;

[0141] S65: The local device pulls the corresponding streaming media content through the edge node, decodes and plays it in order according to the timestamp, and realizes the real-time broadcast output of the voice content; through the above steps, the voice template associated with the target semantics is standardized into a streaming audio format, so that the voice content has the distributability and continuous playback capabilities across network nodes, which improves the stability and compatibility of voice output in a multi-device environment and meets the timing accuracy and fluency requirements of voice playback.

[0142] The present invention encompasses any alternatives, modifications, equivalents, and solutions that fall within the spirit and scope of the present invention. To provide a thorough understanding of the present invention, specific details are described in detail below in connection with the preferred embodiments of the present invention, but those skilled in the art will be able to fully understand the present invention without these detailed descriptions. Furthermore, to avoid unnecessary confusion regarding the essence of the present invention, well-known methods, processes, procedures, components, and circuits have not been described in detail.

[0143] The above is only a preferred embodiment of the present invention. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principles of the present invention. These improvements and modifications should also be regarded as within the scope of protection of the present invention.

Claims

1. A gesture-operated voice playback and output method based on end-cloud collaboration, characterized in that: The following steps are involved: S1: Collect user gesture motion data in real time and generate an original gesture feature set including spatial coordinates, motion trajectory, and dwell time; S2: Perform frame compression processing on the original gesture feature set, extract the change rate of the skeleton point coordinates, and generate a compressed feature sequence; S3: Split the compressed feature sequence into a privacy-related feature layer and a behavior representation feature layer, and upload the behavior representation feature layer to the cloud server; S4: The cloud server dynamically matches the behavior representation feature layer with the preset semantic template based on the dynamic semantic weight database to generate a candidate semantic set with confidence scores; S5: Contextually weight the candidate semantic set based on the user's historical operation data, and select the target semantic and associated speech template with the highest weighted score; S6: Encode the speech template corresponding to the target semantics into streaming data and distribute it to the local device through the edge node for voice broadcast.

2. The method for gesture-operated voice playback and output based on end-cloud collaboration according to claim 1, characterized in that: Said S1 specifically includes: S11: Initiates collaborative perception between the multispectral camera array and millimeter-wave radar, synchronously capturing infrared thermal images, depth point cloud data, and Doppler frequency shift signals of the gesture area at a 100Hz sampling rate. S12: Perform spatiotemporal alignment processing on the original perception signal, use adaptive Kalman filtering to eliminate environmental reflection interference, and generate a denoised three-dimensional spatial coordinate data set; S13: Capture hand motion angular velocity and acceleration through an embedded inertial measurement unit, perform sensor fusion on the measurements with visual data, and calculate the gesture transformation matrix between consecutive frames using quaternion interpolation. S14: Track the motion trajectory of the index finger joint based on the gesture transformation matrix. Use the Euclidean distance change rate between adjacent coordinate points exceeding 10 cm / s as the segmentation threshold to divide the continuous motion into multiple trajectory segments and record the start timestamp, end timestamp and maximum motion speed of each trajectory segment. S15: Detect the state of the hand staying in the preset sensitive area. When the same spatial coordinate point is maintained for a time of ≥300ms and the speed variance is <5cm 2 / s 2 When , it is marked as the effective stay time; S16: Integrate the spatial coordinate data set, motion trajectory segment parameters, and dwell time information according to the time axis and encapsulate them into an original gesture feature set with a unique device identification code.

3. The method for gesture-operated voice playback and output based on end-cloud collaboration according to claim 1, characterized in that: The S2 specifically includes: S21: Divide the original gesture feature set into frames according to a fixed time interval, setting the time interval to Δt, and grouping the three-dimensional space coordinate data within each time interval of Δt into a frame to obtain a frame sequence set; S22: For a specified skeleton point in each frame, the displacement increment per unit time between adjacent frames is calculated based on the change in its three-dimensional coordinates, and the change rate of the skeleton point coordinates is defined as: Among them, V i (t) is the coordinate change rate of the i-th bone point at time t, P i (t) is its three-dimensional coordinate position at time t; S23: Threshold filtering is performed on the calculated coordinate change rate to eliminate point data with a change rate less than the set threshold; S24: normalize and encode the skeleton point number, change rate, and corresponding timestamp retained in each frame, and splice them into a time series in frame order to obtain a compressed feature sequence; S25: In the compressed feature sequence, add the start timestamp, end timestamp and corresponding frame index of each segment to achieve time-based structured storage of trajectory segments.

4. The method for gesture-operated voice playback and output based on end-cloud collaboration according to claim 1, characterized in that: The S3 specifically includes: S31: performing field analysis on each frame feature in the compressed feature sequence to extract data including skeleton point number, coordinate change rate, timestamp, movement speed and dwell time; S32: Establishing a privacy-related feature determination rule, identifying the bone points in the bone point numbers that are related to private parts of the face or shoulders, and classifying the corresponding fields into a privacy-related feature layer; S33: The remaining motion information fields not involving highly sensitive parts, including the coordinate change rate of the hand trajectory segment, velocity parameters and their corresponding time series data, are divided into a behavior representation feature layer; S34: Locally encrypt and store the data of the privacy-related feature layer and set access control permissions; uniformly encode and normalize the data of the behavior representation feature layer.

5. The method for gesture-operated voice playback and output based on end-cloud collaboration according to claim 1, characterized in that: The S4 specifically includes: S41: After receiving the uploaded behavior representation feature layer, the cloud server performs normalization processing on each trajectory segment according to the timestamp, skeleton point number and coordinate change rate information to generate a time sequence action vector in a unified format; S42: Retrieving a set of semantic templates activated in the current time window from the dynamic semantic weight database, each semantic template including a standard action vector pattern and its associated semantic description, to form a semantic matching candidate library; S43: Using a matching mechanism based on multi-dimensional feature comparison, the uploaded temporal motion vector is matched one by one with each semantic template in the semantic matching candidate library, and feature similarity is calculated based on the dimensions of trajectory shape, speed pattern, and dwell rhythm; S44: Based on the dimension consistency and similarity distribution during the matching process, a semantic weight parameter is dynamically introduced to increase the weight of the semantic template that is consistent with the user's historical behavior preferences and reduce the weight coefficient of irrelevant templates; S45: After fusing the semantic weight value of the semantic template itself with the action vector similarity index, a confidence score is assigned to each candidate semantic, and the scores are sorted from high to low. The top N items with the confidence scores are output to form the candidate semantic set.

6. The method for gesture-operated voice playback and output based on end-cloud collaboration according to claim 5, characterized in that: The S43 specifically includes: S431: Segment the uploaded temporal motion vector and split the complete vector into several subsequences according to the start and end timestamps of the trajectory segments. Each subsequence contains continuous skeletal point numbers, coordinate change rates, velocity characteristics, and dwell marks. S432: For each semantic template in the semantic matching candidate library, extract its corresponding standard action pattern sequence, perform normalization processing, and construct a template standard vector set; S433: For each subsequence and the semantic template standard vector, perform dimension-by-dimension comparison at the same time step to calculate the feature similarity score S sim , the similarity score lies in the interval [0, 1]; S434: When the similarity score S sim When it is greater than the set matching threshold of 0.7, the corresponding semantic template is marked as a valid match.

7. The method for gesture-operated voice playback and output based on end-cloud collaboration according to claim 6, characterized in that: The S44 specifically includes: S441: After completing the preliminary similarity score between the uploaded gesture subsequence and the semantic template standard vector, count the dimensional consistency ratio in each template match and calculate the dimensional consistency rate R c ; S442: Perform sliding window processing on the matching score sequence of each template and calculate the similarity variance σ of the template in adjacent matching tasks 2 ; S443: Dynamically adjust the semantic weight parameter W of the semantic template based on the above two indicators s , the adjustment formula is: W s =W0×(1+α×R c )×(1-β×σ 2 ), where α=0.6 and β=0.4 are empirical optimization coefficients; W0 is the original weight.

8. The method for gesture-operated voice playback and output based on end-cloud collaboration according to claim 7, characterized in that: The S45 specifically includes: S451: Calculate the confidence score of the candidate semantics based on its selection frequency in historical tasks. The formula is: C s =λ1·S sim +λ2·W s +λ3·F h , where C s is the confidence score of the candidate semantics, λ1=0.5, λ2=0.3, λ3=0.2; F h The selection frequency of candidate semantics in historical tasks; S452: Confidence score C s Rank all candidate semantics from high to low.

9. The method for gesture-operated voice playback and output based on end-cloud collaboration according to claim 8, characterized in that: The S5 specifically includes: S51: Get the candidate semantic set output by S45 and extract the confidence score C corresponding to each semantic in the candidate semantic set s ; S52: Retrieve the target semantic sequence selected by the user's past 30 consecutive gesture operations from the user's historical operation database stored in the cloud, and construct the target semantic transition probability matrix M ij ; S53: Obtain the historical target semantic index of the user's most recent selection as the current context benchmark, and define the corresponding transition probability value between each semantic in the candidate semantic set and the current context benchmark in the transition probability matrix as the context relevance weight W c ; S54: For each candidate semantics, the context relevance weight W c Confidence score C calculated with S45 s Perform fusion calculation to obtain the final context weighted score S f ; S55: Score S based on context weighting f Sort the candidate semantic sets, select the semantic with the highest score after sorting as the target semantic of this recognition task, and call the cloud-based semantic-speech mapping database to extract the associated speech template corresponding to the target semantic.

10. The method for gesture-operated voice playback and output based on end-cloud collaboration according to claim 1, characterized in that: The S6 specifically includes: S61: After receiving the target semantics and the corresponding voice template ID determined in S5, the cloud server retrieves the text content and audio file resources of the corresponding voice template from the semantic-voice mapping database, and reads the voice broadcast format and version information corresponding to the template; S62: performing frame processing on the retrieved audio file, dividing the entire audio segment into a continuous frame sequence using a fixed frame length, and adding a timestamp, a frame index number, and a checksum to each frame to construct an original audio frame stream with time synchronization information; S63: According to the audio stream protocol format supported by the terminal device, a corresponding audio coding standard is selected for transcoding processing to generate an encoded .ts slice sequence and a .m3u8 index file; S64: Cache the encoded streaming media data in order to the edge node distribution module, and simultaneously generate a unique resource location identifier URL for content distribution; S65: The local device pulls the corresponding streaming media content through the edge node, and decodes and plays it in order according to the timestamp.

Images