Screen-equipped intelligent terminal collaborative data processing method and system based on multi-modal interaction

Abstract

The application discloses a kind of based on multi-modal interaction's with screen intelligent terminal collaborative data processing method and system, it is related to data processing technical field, method includes: obtaining with time mark voice operation and gesture target probability sequence;According to maximum probability category and the difference of maximum, secondary maximum probability value Generation category sequence and probability interval sequence, with adjacent probability interval absolute difference value Mid-value and median absolute deviation Sum determines adaptive change boundary, candidate segment is divided in combination with category switching position;According to category consistent proportion, probability interval median and non-reverse proportion Select voice operation segment and gesture target segment, according to overlapping length Pair and generate operation target data, generate collaborative display instruction in combination with current display content mark.The application solves the problem that probability local fluctuation and two kinds of input effective interval misregistration effective segment is difficult to determine, cross-modal segment is difficult to correspond, improves segment determination and cross-modal pairing accuracy, reduces error collaborative display instruction.

Description

Technical Field

[0001] This invention relates to the field of data processing technology, and specifically to a collaborative data processing method and system for smart terminals with screens based on multimodal interaction. Background Technology

[0002] With the increasing number of smart terminals with screens in homes, offices, and public service venues, users can express display operations via voice and specify the target smart terminal with a screen to receive the currently displayed content via pointing gestures. Voice operation recognition and gesture target recognition typically output category probabilities based on voice frames and image frames, respectively. Therefore, whether collaborative display instructions can be correctly generated depends not only on the individual classification results but also on whether the voice operation segment and gesture target segment belonging to the same interaction can be identified.

[0003] In actual interaction, voice and gestures are acquired through different acquisition interfaces, and their sampling periods, processing times, and user action habits may all differ. A user might begin speaking before pointing, or maintain a pointing gesture before uttering the action; therefore, the effective range of voice and gesture usually only partially overlaps. Simultaneously, environmental noise, reverberation, and pauses in pronunciation can cause local fluctuations in the class probabilities of adjacent voice frames; hand occlusion, background object movement, and depth measurement errors can also cause local fluctuations in the target probabilities of adjacent image frames. Even if the user's actual operation and target remain unchanged, the degree of distinction between the most probable class and other classes may still change repeatedly within a short period.

[0004] Existing processing methods typically select a single probability peak or combine speech recognition and gesture recognition results using a fixed confidence threshold and a fixed time window. A single probability peak only reflects the classification output at a single acquisition moment, making it difficult to indicate whether the output belongs to a stable and continuous interaction process; a fixed confidence threshold cannot simultaneously adapt to probability distributions in quiet environments, noisy environments, and at different gesture distances; and a fixed time window cannot accurately reflect the actual effective intervals of the two inputs. For example, a user might point to the bedroom terminal shortly after saying "display current content," maintain the pointing gesture briefly after the speech ends, and then point to the study terminal. If a fixed time window is used for pairing, the previous speech action might fall into the same window as the subsequent pointing action; if only the probability peak is selected, a sudden increase in target probability caused by occlusion might be identified as a valid target. In this case, even if speech classification and gesture classification each have a large number of correct outputs, improper judgment of segment boundaries and correspondences may still lead to the operation category and target terminal identifier originating from different actual interaction processes. Summary of the Invention

[0005] This invention provides a collaborative data processing method and system for smart terminals with screens based on multimodal interaction, which solves the technical problem of difficulty in determining continuous effective segments and accurately establishing cross-modal segment correspondence when there are local fluctuations in the probability of voice operation and the probability of gesture target and the start and end times of the effective interval of the two inputs are inconsistent.

[0006] This invention provides a collaborative data processing method for a screen-equipped smart terminal based on multimodal interaction, comprising: acquiring a multimodal probability sequence group including a time-stamped voice operation probability sequence and a time-stamped gesture target probability sequence; processing each probability sequence; wherein, a category sequence is generated from the highest probability category, and a probability interval sequence is generated from the difference between the highest probability value and the second highest probability value; an adaptive change boundary is generated from the median of the absolute difference between adjacent probability intervals and the sum of the median absolute deviation, and candidate segments are divided based on the positions exceeding the adaptive change boundary and the category switching positions; a segment support ratio is generated from the category consistency ratio of the candidate segments and the median of the probability interval, a non-reversal ratio is generated from the direction of probability interval change, and a segment discrimination ratio is generated from the segment support ratio and the non-reversal ratio, and segments are selected according to the segment discrimination ratio; the selected segments are combined into multimodal segment data, the multimodal segment data including voice operation segments and gesture target segments; an overlap duration is generated based on the multimodal segment data, and segment pairing data is generated based on the multimodal segment data and the overlap duration; operation target data is generated based on the segment pairing data, and collaborative display instructions are generated based on the operation target data and the currently displayed content identifier.

[0007] Optionally, obtaining the time-stamped speech operation probability sequence includes: obtaining a speech sampling sequence; generating a pre-emphasized speech sequence based on the speech sampling sequence; generating a framed speech sequence based on the pre-emphasized speech sequence; generating a speech time-stamped sequence based on the framed speech sequence; generating a speech power spectrum sequence based on the framed speech sequence; generating a frequency band energy sequence based on the speech power spectrum sequence; generating a speech feature sequence based on the frequency band energy sequence; obtaining speech operation classification parameters; generating an unstamped speech operation probability sequence based on the speech feature sequence and the speech operation classification parameters; and generating a time-stamped speech operation probability sequence based on the unstamped speech operation probability sequence and the speech time-stamped sequence.

[0008] Optionally, obtaining the time-stamped gesture target probability sequence includes: obtaining a depth image sequence; obtaining a color image sequence; generating a registration image sequence based on the depth image sequence and the color image sequence; generating an adjacent depth difference sequence based on the registration image sequence, generating a depth change boundary based on the adjacent depth difference sequence, generating a hand region sequence based on the adjacent depth difference sequence and the depth change boundary; generating a hand displacement vector sequence based on the hand region sequence, generating a gesture feature sequence based on the hand displacement vector sequence; obtaining gesture target classification parameters, generating an unstamped gesture target probability sequence based on the gesture feature sequence and the gesture target classification parameters; generating a gesture time-stamped sequence based on the registration image sequence, and generating a time-stamped gesture target probability sequence based on the unstamped gesture target probability sequence and the gesture time-stamped sequence.

[0009] Optionally, the partitioning of candidate segments includes: generating an absolute difference sequence of intervals based on the probability interval sequence, and taking the median as the median of the interval difference; generating an interval deviation sequence based on the absolute difference between the absolute difference sequence of intervals and the median of the interval difference, and taking the median as the median absolute deviation of the interval; generating an adaptive change boundary based on the sum of the median of the interval difference and the median absolute deviation of the interval; writing the positions corresponding to the absolute differences of the intervals greater than the adaptive change boundary into the probability change position sequence, and writing the positions with different maximum probability categories into the category switching position sequence; merging, deduplicating, and sorting the probability change position sequence and the category switching position sequence to generate a segment boundary position sequence; and generating a candidate segment sequence based on the segment boundary position sequence and the probability sequence that generated the probability interval sequence.

[0010] Optionally, the generation of the segment support ratio includes: generating category quantity data based on the category sequence within the candidate segment; generating median category probability data based on the probability values ​​of each category within the candidate segment; arranging each category in descending order of occurrence count, and if the counts are the same, arranging them in descending order of the median category probability, and if they are still the same, arranging them in a fixed category order, and determining the first category as the primary category; generating the primary category quantity based on the primary category and category quantity data; generating the total number of probability vectors based on the candidate segments; generating the category consistency ratio based on the primary category quantity and the total number of probability vectors; generating the median probability interval based on the probability interval sequence within the candidate segment; and taking the smaller value between the category consistency ratio and the median probability interval as the segment support ratio.

[0011] Optionally, the generation of the segment discrimination ratio includes: comparing the adjacent probability intervals within candidate segments, writing an ascending marker when the latter is larger, a descending marker when it is smaller, and an unchanged marker when they are equal, generating a sequence of interval change directions; generating a sequence of adjacent direction pairs based on the sequence of interval change directions; generating a total number of direction pairs based on the sequence of adjacent direction pairs; generating a number of non-reversed direction pairs based on the sequence of adjacent direction pairs, wherein adjacent direction pairs formed by ascending and descending markers in any order are not included in the number of non-reversed direction pairs; when the sequence of adjacent direction pairs is not empty, generating a non-reversed ratio based on the number of non-reversed direction pairs and the total number of direction pairs, and taking the smaller value between the segment support ratio and the non-reversed ratio as the segment discrimination ratio; when the sequence of adjacent direction pairs is empty, no segment discrimination ratio is generated.

[0012] Optionally, selecting segments based on segment discrimination ratios includes: identifying candidate segments in the speech operation probability sequence that have segment discrimination ratios and whose segment discrimination ratios are positive as speech operation segments; identifying candidate segments in the gesture target probability sequence that have segment discrimination ratios and whose segment discrimination ratios are positive as gesture target segments; and stopping the generation of segment pairing data when no speech operation segment or gesture target segment is generated.

[0013] Optionally, the generation of segment pairing data includes: forming a segment pair to be judged by pairing each voice operation segment with each gesture target segment; taking the later start time of each segment pair to be judged as the overlap start time and the earlier end time as the overlap end time; generating an overlap duration based on the overlap start time and overlap end time, and determining the segment pairs to be judged with positive overlap durations as candidate segment pairs; arranging the candidate segment pairs in descending order of overlap duration; when the overlap durations are the same, taking the smaller value of the two segment discrimination ratios within the candidate segment pair as the segment pairing ratio, and arranging them in descending order of the segment pairing ratio; if they are still the same, arranging them in descending order of the overlap end time; and determining the first candidate segment pair as segment pairing data.

[0014] Optionally, the generation of operation target data and collaborative display instructions includes: generating operation field data based on voice operation segments in segment pairing data, the operation field data including display operation code or stop display operation code; generating target field data based on gesture target segments in segment pairing data, the target field data including terminal identifier of target smart terminal with screen; generating operation target data based on operation field data and target field data; reading the unique identifier of content in the current display buffer to generate current display content identifier; and generating collaborative display instructions based on operation target data and current display content identifier, the collaborative display instructions including operation field data, target field data, and current display content identifier.

[0015] Optionally, message generation after the collaborative display instruction is generated includes: identifying the smart terminal with a screen that generated the collaborative display instruction as the source smart terminal with a screen; obtaining the logical version data saved by the source smart terminal with a screen, and generating the next-order current logical version data based on the current display content identifier and the logical version data; generating a versioned collaborative display instruction based on the collaborative display instruction and the current logical version data; obtaining terminal connection mapping data, and generating a collaborative display message based on the versioned collaborative display instruction and the terminal connection mapping data, wherein the collaborative display message includes a unique message identifier, operation field data, target field data, current display content identifier, message generation time, and current logical version data.

[0016] Optionally, the execution of the collaborative display message includes: determining the target smart terminal with a screen based on the target field data; generating receiving logic version data based on the collaborative display message, and obtaining the executed logic version data saved by the target smart terminal with a screen; generating version order data based on the receiving logic version data and the executed logic version data; generating receiving discrimination data including an allowed execution code when the version order data indicates that the receiving logic version number is after the executed logic version number, otherwise generating receiving discrimination data including a rejected execution code; generating target display execution data based on the collaborative display message and the receiving discrimination data; and generating collaborative display result data based on the target display execution data.

[0017] A collaborative data processing system for a screen-equipped smart terminal based on multimodal interaction is also provided. The system implements a collaborative data processing method for screen-equipped smart terminals based on multimodal interaction. The system includes: a multimodal input module for generating a multimodal probability sequence group comprising a time-stamped voice operation probability sequence and a time-stamped gesture target probability sequence; a segment processing module for generating multimodal segment data comprising voice operation segments and gesture target segments based on the multimodal probability sequence group; a collaborative instruction module for generating segment pairing data based on the multimodal segment data, generating operation target data based on the segment pairing data, and generating collaborative display instructions based on the operation target data and the currently displayed content identifier; and a display execution module for generating collaborative display messages based on the collaborative display instructions and generating collaborative display result data based on the collaborative display messages.

[0018] The beneficial effects of this invention are reflected in: This invention addresses time-stamped speech operation probability sequences and time-stamped gesture target probability sequences. It utilizes the highest probability category to form a category sequence and the difference between the highest and second-highest probability values ​​to form a probability interval sequence. This ensures that segment processing is based not only on the classification category but also on the distinguishing power of the highest probability category relative to other categories. An adaptive change boundary is generated using the median and median absolute deviation of the absolute differences between adjacent probability intervals. Positions exceeding this boundary, along with category switching positions, are used as candidate segment boundaries. This allows segment segmentation to be directly derived from the central tendency and dispersion of the current probability sequence. Consequently, segment boundaries can be adjusted according to fluctuations in speech noise, image occlusion, and depth measurement, reducing the possibility of over-segmentation or boundary omissions caused by a fixed change threshold under different acquisition environments.

[0019] Furthermore, this invention utilizes the category consistency ratio and the median probability interval to jointly limit the segment support level, and reflects the continuity of probability changes by comparing the proportions of adjacent directions in which the probability interval change direction does not directly reverse. This allows category continuity, category discrimination, and stability of change direction to jointly participate in segment discrimination. The segment discrimination result is limited by the weaker of these factors, which can reduce the possibility of selecting candidate segments that are category-consistent but lack discrimination, have large probability intervals but frequently switch categories, or have repeatedly rising and falling probability intervals. Subsequently, only voice operation segments with positive actual overlap durations are paired with gesture target segments to form candidate segment pairs, and deterministic selection is performed sequentially based on overlap duration, segment pairing ratio, and overlap end time, without the need to set a fixed pairing time window. Finally, collaborative display instructions are generated from the main voice category, main gesture category, and current display content identifier in the same segment pairing data, thereby reducing the probability of incorrect combinations of voice operations and gesture targets during adjacent interactions, and establishing a data association with a clear temporal relationship and repeatable processing results between operation category, target terminal, and display content. Attached Figure Description

[0020] To more clearly illustrate the specific embodiments of the present invention or the technical solutions in the prior art, the accompanying drawings used in the description of the specific embodiments or the prior art will be briefly introduced below. In all the drawings, similar elements or parts are generally identified by similar reference numerals. In the drawings, the elements or parts are not necessarily drawn to scale.

[0021] Figure 1 This is a schematic diagram illustrating the steps of the collaborative data processing method for a screen-equipped smart terminal based on multimodal interaction according to the present invention. Figure 2 This is a schematic diagram of some steps in S1 of the collaborative data processing method for a screen-equipped smart terminal based on multimodal interaction according to the present invention. Figure 3This is a schematic diagram of some steps in S2 of the collaborative data processing method for a screen-equipped smart terminal based on multimodal interaction according to the present invention. Figure 4 This is a schematic diagram of some steps in S3 of the collaborative data processing method for a screen-equipped smart terminal based on multimodal interaction according to the present invention. Figure 5 This is a schematic diagram of some steps in S4 of the collaborative data processing method for a screen-equipped smart terminal based on multimodal interaction of the present invention; Figure 6 This is a schematic diagram of the module structure and data flow of the collaborative data processing system for screen-equipped smart terminals based on multimodal interaction according to the present invention. Detailed Implementation

[0022] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. It should be understood that the described embodiments are only some embodiments of the present invention, and not all embodiments. The components of the embodiments of the present invention described and shown in the accompanying drawings can generally be arranged and designed in various different configurations.

[0023] Therefore, the following detailed description of the embodiments of the invention provided in the accompanying drawings is not intended to limit the scope of the claimed invention, but merely to illustrate selected embodiments of the invention. All other embodiments obtained by those skilled in the art based on the embodiments of the invention without inventive effort are within the scope of protection of the invention.

[0024] It should be noted that similar labels and letters in the following figures indicate similar items; therefore, once an item is defined in one figure, it does not need to be further defined and explained in subsequent figures. Furthermore, the terms "first," "second," etc., are used only to distinguish descriptions and should not be construed as indicating or implying relative importance.

[0025] This invention provides a collaborative data processing method for smart terminals with screens based on multimodal interaction, such as... Figure 1 As shown, in one embodiment, the method includes: S1. Obtain the time-stamped voice operation probability sequence and the time-stamped gesture target probability sequence. The voice operation probability vector is used to represent the probability value corresponding to the display operation category and the stop display operation category. The gesture target probability vector is used to represent the probability value corresponding to the registered target smart terminal with screen. S2. Generate the maximum probability category sequence and probability interval sequence according to each probability sequence. Determine the adaptive change threshold based on the median and median absolute deviation of the absolute difference between adjacent probability intervals. Divide candidate segments into segments based on the position exceeding the adaptive change threshold and the position of the maximum probability category switch. Obtain the segment validity score by taking the smallest value in the order of the category consistency ratio of the main category of the candidate segment, the median of the probability interval, and the non-reversal ratio of the change direction of adjacent probability intervals. Determine the candidate segments with positive segment validity scores as segments to be paired. S3. Calculate the overlap duration based on the speech segment to be paired and the gesture segment to be paired. Determine the segment pairing results in descending order according to the overlap duration, segment pair score and overlap end time. The segment pair score is the smaller value of the validity scores of the two corresponding segments. S4. Generate collaborative display instructions based on the main voice category, main gesture category, and current display content identifier in the segment pairing results.

[0026] In this embodiment, it should be noted that in S1, voice operation is used to express the display operation category or the stop display operation category, and gesture target is used to express the target smart terminal with screen receiving the currently displayed content. Voice sampling data and gesture image data have different data formats and sampling frequencies, and cannot be directly time-paired. The system converts the voice sampling data and gesture image data into category probability vectors respectively, and assigns a timestamp under the same monotonic time base to each probability vector. This processing does not directly combine sound pressure data and image pixel data, but first converts the two raw data into probability vectors and timestamps with a unified structure.

[0027] Furthermore, the monotonic time base only increases forward with device uptime and is unaffected by calendar time, time zone settings, or network time synchronization correction. When the voice acquisition interface, depth image acquisition interface, and color image acquisition interface use the same processor, the system directly reads the processor's monotonic clock. When the acquisition interface has an independent hardware clock, the system determines the clock offset using the interface timestamp and processor timestamp corresponding to the same hardware trigger event, and converts the interface timestamp to the processor's monotonic clock. This process ensures that the start time, end time, and overlap duration of subsequent segments have a unified physical time base.

[0028] Specifically, the frame shift of the speech feature frames and the output interval of the speech operation probability vectors are different parameters. Speech feature frames can be generated continuously with short frame shifts, and the classification process then selects the center frame to be classified according to the probability output step size. For example, the speech feature frame shift can be 10 ms, and the probability output step size can be 18 frame shifts, so the timestamps of adjacent speech operation probability vectors are 180 ms apart. The aforementioned 180 ms is the classification output interval, not the duration of a single speech frame, nor the processing time required for the processor to complete the classification.

[0029] The reason for adopting the above processing logic is that the effective duration intervals of speech and images may partially overlap, but the output frequencies of the acquisition interfaces are usually different. A unified monotonic time reference ensures that each probability vector has a comparable acquisition time, and the probability output step size enables the classification network to utilize continuous features to form probability outputs with temporal context, thereby providing a consistent temporal basis for the continuous segmentation in S2 and the calculation of overlapping duration in S3.

[0030] In S2, to address the issue that the peak probability of a single classification is easily affected by environmental noise, occlusion, or depth measurement errors, the system does not directly select the single moment with the highest maximum probability value. Instead, it reads the difference between the most probable class and the two highest probability values ​​in each probability vector. The difference between the two highest probability values ​​is defined as the probability interval. The sequence of most probable classes reflects class continuity, while the probability interval sequence reflects the degree of distinction between the most probable class and the second most probable class.

[0031] Furthermore, the system calculates the absolute value of the difference between adjacent probability intervals in time, and determines an adaptive change threshold based on the median and median absolute deviation of all adjacent probability interval absolute differences. Locations exceeding the adaptive change threshold are used to identify locations where the classification distinction has significantly changed, and locations with the highest probability class switching are used to identify locations where the class has changed. The system merges these two types of locations to divide them into candidate segments, and then determines the segment validity score based on the class consistency ratio, the median probability interval, and the non-reversal ratio.

[0032] Specifically, the category consistency ratio indicates the degree to which the main category consistently occupies the highest probability position within the candidate fragment; the median probability interval indicates the typical distinction between the highest probability category and the second highest probability category within the fragment; and the non-reversal ratio indicates the degree to which the direction of change in the probability interval does not directly switch between increasing and decreasing. The system sequentially selects the smaller values ​​from these data, limiting the fragment validity score if any aspect is insufficient.

[0033] The reason for adopting the above processing logic is that the category consistency ratio, the median probability interval, and the non-reversal ratio correspond to category continuity, classification discrimination, and probability change continuity, respectively. Taking the smaller values ​​in sequence can avoid higher individual data compensating for lower individual data, reduce the possibility of high-rated segments being formed by sudden increases in instantaneous probability, frequent category switching, or repeated changes in probability intervals, and provide S3 with segments to be paired with clear start and end times, main categories, and segment validity scores.

[0034] In S3, to address the issue that voice input and gesture input often cannot start or end simultaneously, the system combines each voice segment to be paired with each gesture segment to be paired. For each segment combination, the system takes the later of the two start times as the overlap start time and the earlier of the two end times as the overlap end time, and determines the overlap duration based on the time length between the two. Only segment combinations with a positive overlap duration can become candidate segment pairs.

[0035] Furthermore, when multiple candidate segment pairs exist, the system first sorts them in descending order of overlap duration. If the overlap durations are the same, the system takes the smaller of the validity scores of the two segments within the same candidate segment pair as the segment pair score, and sorts them in descending order of score. If the segment pair scores are still the same, the system sorts them in descending order of overlap end time. This sorting order ensures that common duration, the validity of the weaker segment in both modalities, and temporal position participate in pairing sequentially.

[0036] The reason for adopting the above processing logic is that the actual overlap duration comes directly from the effective duration interval of the two modalities, without the need to manually set a fixed pairing time window. Using segment pair scoring after the overlap duration is the same can avoid prioritizing segments with insufficient effectiveness in either modality; finally, using the overlap end time can form a definite ranking result when the first two items are exactly the same, reducing the possibility of incorrect combination of the previous voice operation and the subsequent gesture target.

[0037] In S4, the system obtains the display operation code or stop display operation code from the main voice category in the segment pairing results, and obtains the target terminal identifier from the main gesture category. The system also reads the display status record associated with the current display buffer, obtains the current display content identifier and content retrieval method from the display status record, and establishes a content provision record between the current display content identifier and the content retrieval method.

[0038] Furthermore, the content provision record, as defined in this application, is a data record used to store the correspondence between the currently displayed content identifier, content type, and the actual content acquisition method. For local content, the content acquisition method may include the local content reading location; for web pages or dynamic interfaces, the content acquisition method may include the page resource list and interface state data; for video, the content acquisition method may include the media data stream session. The content provision record enables the target smart terminal with a screen to obtain the actual displayed content based on the currently displayed content identifier, rather than treating the abstract identifier itself as the displayed content.

[0039] Specifically, the system identifies the smart terminal with a screen that generates the collaborative display instruction as the source terminal identifier. Then, it combines the source terminal identifier, operation code, target terminal identifier, and current display content identifier into a collaborative display instruction according to a fixed field order. This combination process ensures that the executing action, the receiving object, the content source, and the display object each have defined fields.

[0040] The reason for adopting the above processing logic is that the operation code, target terminal identifier, and current display content identifier are derived from voice segments, gesture segments, and display status records, respectively. By obtaining and combining each field step by step, each field can have a traceable data source, and the target smart terminal with a screen can distinguish the operation to be performed, the message recipient, and the display content to be obtained.

[0041] In summary, the collaborative data processing method for screen-equipped smart terminals based on multimodal interaction firstly converts voice sampling data and gesture image data into voice operation probability sequences and gesture target probability sequences with unified timestamp structures, respectively, to address the problem of direct time comparison between different data formats and sampling frequencies. Simultaneously, candidate segments are divided based on the maximum probability category, probability interval, absolute difference between adjacent probability intervals, and an adaptive change threshold directly calculated from the current probability sequence, reducing the dependence of the fixed change threshold on the interaction environment. Furthermore, the segment availability is jointly determined by the category consistency ratio, the median probability interval, and the non-reversal ratio. Effectiveness scoring incorporates category continuity, classification discrimination, and probability change continuity into segment selection. This allows for the calculation of actual overlap duration for voice and gesture segments with positive scores, resulting in segment pairings based on overlap duration, segment pair scores, and overlap termination times. Finally, the main voice category, main gesture category, source terminal identifier, and current display content identifier are combined into collaborative display instructions. Content provision records and logical version numbers can be used to complete content retrieval, execution discrimination, and result feedback, creating a continuous data processing relationship between multimodal input, continuous segment extraction, cross-modal pairing, collaborative instruction generation, and target display execution.

[0042] In one implementation, such as Figure 2 As shown, S1 includes: S11. Generate speech operation classification parameters based on speech training samples with operation coding annotations.

[0043] S12. Generate gesture target classification parameters based on gesture training samples labeled with terminal identifiers.

[0044] S13. Generate a timestamped speech operation probability sequence based on the speech sampling sequence and speech operation classification parameters.

[0045] S14. Generate a time-stamped probability sequence of gesture targets based on the depth image sequence, color image sequence, and gesture target classification parameters.

[0046] In this embodiment, it should be noted that in S11, the speech training samples include speech sampling sequences and corresponding operation codes. The operation codes can be either display operation codes or stop display operation codes. Before annotation, each training sample is reviewed by at least two annotators for its speech content, start and end times of sampling, and operation codes. If the annotation results are inconsistent, another annotator listens to the sample again and determines the label. Samples whose operation content cannot be identified, whose speech is severely truncated, or whose labels still conflict are not included in the training set. The number of annotators is derived from a limited annotation consistency experiment. For example, 1, 2, and 3 initial annotators may be used to process the same batch of 200 samples. When the final consistency rate of 2 and 3 annotators is both 98%, but 2 annotators take less time, 2 initial annotators and 1 conflict reviewer can be used.

[0047] Furthermore, the speech feature parameters are determined through a finite candidate configuration and a validation set. In one optional implementation, candidate sampling frequencies include 16 kHz and 24 kHz, candidate frame lengths include 20 ms, 25 ms, and 30 ms, candidate frame shifts include 10 ms and 15 ms, and candidate pre-emphasis coefficients include 0.95, 0.97, and 0.98. The system performs the same training process on each candidate configuration and compares the validation loss and single inference time. For example, the combination of a sampling frequency of 16 kHz, a frame length of 25 ms, a frame shift of 10 ms, and a pre-emphasis coefficient of 0.97 achieves a validation loss of 0.25, and the second lowest validation loss among the candidate combinations is 0.28, thus the former combination can be selected. The aforementioned values ​​only represent the parameter selection process and do not constitute a unique limitation.

[0048] Specifically, with a sampling frequency of 16 kHz and a frame length of 25 ms, each speech frame contains 400 sample values, derived from the number of sampling points corresponding to the sampling frequency and frame length. With a frame shift of 10 ms, the starting positions of adjacent speech frames are separated by 160 sample values. Pre-emphasis processing retains the first sample value and processes the remaining sample values ​​using the current sample value, the previous original sample value, and the pre-emphasis coefficient. This parameter set falls within the commonly used narrowband speech analysis range, and the frame length can cover short-term approximately stationary speech intervals.

[0049] Furthermore, each speech frame is padded to 512 data positions after being multiplied by a Hamming window, before performing a Fast Fourier Transform (FFT) and power statistics. The padding length of 512 is greater than the smaller binary FFT length of 400 samples within a frame, sufficient to accommodate all samples and adapt to the FFT. Candidate padding lengths can include 512 and 1024; for example, their losses in the same validation set are 0.25 and 0.25 respectively, and 512 can be chosen when the average transform time is shorter.

[0050] Specifically, bandgap processing can employ 40 Mel bands, with a frequency range from 20 Hz to 7600 Hz. The upper limit is below the highest analyzable frequency corresponding to the 16 kHz sampling frequency, thus not exceeding the available spectrum range. The number of bands and frequency boundaries are determined through a validation set. For example, comparing 32, 40, and 48 bands yields validation losses of 0.29, 0.25, and 0.25, respectively. However, 40 bands have fewer features than 48 bands, therefore 40 bands can be selected. The lower frequency limit is used to reduce the influence of DC components and extremely low-frequency mechanical vibrations, while the upper frequency limit is used to preserve the main speech information and avoid edge bands approaching the sampling limit.

[0051] Furthermore, a positive lower limit is set before performing logarithmic processing on the frequency band energy; for example, 10 can be used. 10. This lower limit is used to prevent logarithmic processing from being impossible when the frequency band energy is 0. Its value is much lower than the normalized effective frequency band energy and will not change the frequency band order of normal speech frames. A candidate lower limit of 10 can be used. 8, 10 10 and 10 12. For example, in a batch of validation data containing 1000 silence frames and 1000 speech frames, none of the three lower limits changed the category ranking of valid speech frames, while 10 10 did not produce an invalid logarithm, so this value can be used.

[0052] Specifically, 13 coefficients can be retained after the discrete cosine transform. The candidate number of coefficients to retain can include 12, 13, and 20, corresponding to validation losses of 0.28, 0.25, and 0.25 respectively. Since the feature dimension of 13 coefficients is lower than that of 20 coefficients, 13 coefficients can be selected. The aforementioned pre-emphasis, framing, Hamming window, fast Fourier transform, Mel banding, logarithmic processing, and discrete cosine transform together form the speech feature generation process.

[0053] Furthermore, in one optional implementation, the speech classification network receives speech features from 101 consecutive time points, each containing 13 feature values. The candidate input window length can include 81, 101, and 121 time points, corresponding to validation losses of 0.29, 0.25, and 0.27, respectively, indicating that 101 time points can account for both speech context and boundary interference in the validation data. With a frame shift of 10 ms, the 101 time points cover approximately 1 s of speech context, which aligns with the duration range of phrase-level display operation recognition.

[0054] Specifically, the first one-dimensional convolutional layer may include 64 convolutional kernels of length 5, the second one-dimensional convolutional layer may include 128 convolutional kernels of length 3, and the third one-dimensional convolutional layer may include 128 convolutional kernels of length 3. Each convolutional layer has a stride of 1, padded with zeros at both ends to maintain the number of time steps before and after convolution, and batch normalization and linear rectification are performed sequentially after convolution. Max pooling of length 2 is performed after the first and second convolutional layers, and global average pooling is performed after the third convolutional layer. The first fully connected layer outputs 64 values, and the second fully connected layer outputs two classification values ​​corresponding to the two speech operation categories.

[0055] Furthermore, the number of network layers, kernel length, number of channels, and pooling positions are determined through comparison with a finite number of candidate networks. For example, candidate networks with 2, 3, and 4 convolutional layers can be trained, and each candidate network can be trained 5 times, yielding median validation losses of 0.31, 0.25, and 0.28, respectively. In this example, the candidate network with 3 convolutional layers achieves the lowest validation loss, and therefore the corresponding layer is selected. The number of training repetitions of 5 is derived from training fluctuations in validation. When the median validation loss ranking no longer changes after increasing the number of repetitions from 3 to 5, and increasing to 7 only increases training time, 5 repetitions can be selected.

[0056] Specifically, the probability output step size can be selected from 12, 18, and 24 frame shifts. Taking a batch of verification interactions with manually labeled start and end times as an example, the median segment boundary time error corresponding to the three candidate step sizes can be 40 ms, 50 ms, and 90 ms, respectively, with the corresponding average probability vector number decreasing sequentially. If 18 frame shifts meet the predetermined boundary time error requirements and the computational cost is significantly lower than 12 frame shifts, 18 frame shifts can be selected. When the frame shift is 10 ms, the probability output interval is 180 ms obtained from 18 frame shifts.

[0057] Furthermore, the training samples are grouped according to speaker identifiers, ensuring that samples from the same speaker only enter one of the training, validation, or test sets. For example, 2400 speech samples can be obtained, with the total number of samples derived from valid records retained after label verification. After being divided into 70%, 15%, and 15% sets, the training, validation, and test sets contain 1680, 360, and 360 samples, respectively. The division ratio is determined by comparing multiple candidate ratios, selecting the configuration with the larger number of training samples while ensuring that both the validation and test sets cover all speaker types and operation categories.

[0058] Specifically, network parameters are initialized using a fixed random seed recorded in the training configuration, and the same seed and sample order are used for comparative training within the same group. The specific value of the random seed is only used for repeated training and does not participate in inference, nor does it affect the data relationships in the claims. The training batch can contain 64 samples; candidate batches can include 32, 64, and 128, for example, corresponding to memory usage of 1.2 GB, 2.1 GB, and 3.9 GB, respectively, and validation losses of 0.27, 0.25, and 0.25, respectively. Within the allowable range of the target processor's memory, 64 samples can be selected due to its higher computational throughput and acceptable memory usage.

[0059] Furthermore, optimization can be performed using Adam, with an initial learning rate of 10. 3. The first gradient decay coefficient can be 0.9, the second gradient decay coefficient can be 0.99, and the lower limit of numerical stability can be 10. 8. Candidate initial learning rates can include 10. 4, 10 3 and 10 2. For example, the corresponding validation losses are 0.34, 0.25, and 0.37, so 10 is chosen. 3. The two gradient decay coefficients are determined by comparing two sets of configurations: 0.9 and 0.99, and 0.9 and 0.98. For example, the validation loss of the first set is 0.25, and that of the second set is 0.27, so the first set is selected.

[0060] Specifically, the maximum number of training epochs can be 100, and training stops when the validation loss fails to decrease after 10 consecutive epochs. Candidate maximum training epochs can include 60, 100, and 140, for example, corresponding to minimum validation losses of 0.29, 0.25, and 0.25 respectively. Since 140 epochs do not further reduce the loss, 100 is chosen. Candidate consecutive epochs without loss reduction can include 6, 10, and 14. For example, 6 epochs result in 8 early terminations, 10 epochs result in 2 early terminations, and 14 epochs do not improve the final loss but increase training time, so 10 is chosen.

[0061] Furthermore, the cross-entropy loss is obtained by reading the output probability corresponding to the true class of each sample, obtaining the negative natural logarithm of that probability, and averaging it over the current training batch. After each training round, the validation loss is calculated using the validation set, and the network parameters with the lowest validation loss are retained; if the validation losses are the same, the network parameters from earlier rounds are retained. After training, the speech operation classification parameters include feature dimension, input window length, probability output stride, network layers and connections, convolution kernel parameters, bias parameters, batch normalization parameters, the mean and standard deviation of each feature dimension in the training set, and the output class index table.

[0062] The reason for adopting the above training logic is that the training input, labels, feature parameters, model layers, loss calculation, parameter updates, stopping conditions, and parameter selection all have definite data sources. Those skilled in the art can construct and train a speech classification network according to the same processing order, without needing to understand "training" as a black-box action lacking input-output relationships.

[0063] In S12, the gesture training samples include depth image sequences, color image sequences, and the terminal identifiers of the corresponding target smart terminals with screens. The gesture training samples are collected within a fixed deployment space, and during training, the target terminal identifier, the target terminal's spatial location, and the classification output index are recorded. Samples with severely missing images, where the hand region cannot be continuously extracted, or where the terminal label does not match the actual pointing direction are not included in the training set.

[0064] Furthermore, the output category of the gesture classification network corresponds to the set of registered target terminals, rather than an arbitrary set of terminals discovered temporarily during each interaction. The set of registered target terminals is established before the collection of training samples, and the terminal identifier, terminal location, and output index are written into the gesture target classification parameters. When a terminal temporarily disconnects, the system retains the output location corresponding to that terminal, and only stops sending messages to that terminal during the message sending phase. When a terminal is added, deleted, or its deployment location is changed, the system re-collects the corresponding gesture training samples and updates the gesture target classification parameters.

[0065] Specifically, in one optional implementation, each gesture classification input includes nine consecutive hand displacement direction-distance histograms, each containing 32 statistical values. The candidate input window length can include 7, 9, and 11 histograms, corresponding to validation losses of 0.27, 0.23, and 0.24 respectively, thus allowing for the selection of nine histograms. Each histogram aggregates a hand displacement vector formed by multiple adjacent registered images within a probability output time interval, without using a single displacement vector as a complete statistical sample.

[0066] Furthermore, the gesture classification network can include three one-dimensional convolutional layers and two fully connected layers. The first convolutional layer can include 64 convolutional kernels of length 3, and the second and third convolutional layers can each include 128 convolutional kernels of length 3. After the convolutional layers, batch normalization, linear rectification, and pooling are performed sequentially. Then, intermediate features are generated through a fully connected layer containing 64 outputs. The final fully connected layer outputs a classification value that is the same as the number of registered target terminals.

[0067] Specifically, the number of layers and channels in the gesture network is determined through a limited pool of candidate structures. For example, networks with 2, 3, and 4 convolutional layers can be compared. Each network is trained 5 times, yielding median validation losses of 0.29, 0.23, and 0.26, respectively. Therefore, a network with 3 convolutional layers is chosen. Gesture training samples are divided into training, validation, and test sets according to user identifiers. The same loss calculation, parameter update, and early stopping rules as those used in the speech network are employed to prevent actions from the same user from simultaneously entering both the training and test sets.

[0068] Furthermore, after training, the gesture target classification parameters include image registration parameters, effective depth range, temporal matching tolerance, foreground connectivity rules, continuous hand region selection rules, feature normalization parameters, orientation interval boundaries, distance interval boundaries, probability output step size, classification network parameters, and the mapping between terminal identifiers and output indices. The reason for adopting the above training logic is that gesture target categories are related to terminal deployment locations; fixing the terminal registration set and output indices can avoid changing the dimension and category meaning of the output vector during inference.

[0069] In S13, the system acquires a speech sampling sequence and performs pre-emphasis, framing, windowing, spectral processing, frequency band energy statistics, logarithmic processing, and discrete transformation on the speech sampling sequence to generate a speech feature sequence. The system uses the feature mean and standard deviation obtained from the training set for normalization, without recalculating the normalization parameters using the current interaction sequence; when the standard deviation of a certain feature dimension is 0, 1 is used as the normalization divisor for that dimension to avoid generating invalid values.

[0070] Furthermore, the system selects a center frame to be classified at every probability output step. The center frame to be classified is a speech feature frame defined in this application, used to determine the center position of the classification input window and the timestamp of the probability vector. The system reads continuous speech features centered on the center frame to be classified; if the beginning or end of the sequence is insufficient for the length of the input window, it is padded with features of value 0. The classification network sequentially performs convolution, batch normalization, linear rectification, pooling, and fully connected processing.

[0071] Specifically, the maximum value of each classification value is subtracted first, then exponential processing is performed on each value, the results of the exponential processing are summed, and this sum is used to normalize each exponential processing result, generating a speech operation probability vector where the sum of the probabilities of each category is 1. Subtracting the maximum classification value first controls the numerical range of the exponential processing and avoids floating-point overflow when the classification value is large.

[0072] Furthermore, the system determines the center acquisition time of the frame to be classified as the timestamp of the voice operation probability vector. Using the center acquisition time instead of the classification output completion time avoids processor load fluctuations altering the time position of the probability vector. For example, the center acquisition time of a frame to be classified is 1420 ms, derived from the monotonic clock converted by the acquisition interface; the classification process yields a display operation category probability of 0.71 and a stop display operation category probability of 0.29, both derived from normalized exponential processing and added together to equal 1. The system writes both probability values ​​and 1420 ms into the same voice operation probability vector.

[0073] Specifically, when the voice acquisition interface has an independent clock, the system continuously acquires the interface timestamps and processor timestamps corresponding to the same hardware trigger event for 21 consecutive sets before the interaction begins. It calculates the time difference between the timestamps in each set, sorts the differences, removes the two values ​​at each end, and takes the median of the remaining values ​​as the clock offset for this interaction. The system only updates the offset before the next interaction begins to avoid timestamp rollback during interaction processing.

[0074] Furthermore, the number of calibration sets and the amount of data removed from both ends were determined using limited historical calibration data. For example, 30 historical interactions were processed using 9, 21, and 31 sets of calibration data, respectively, yielding median time conversion errors of 7 ms, 3 ms, and 3 ms. The median errors when removing 1, 2, and 3 values ​​from each end were then compared, yielding 4 ms, 3 ms, and 3 ms. When the errors were equal, 21 sets of data with the smaller sampling and computational complexity, along with removing 2 values ​​from each end, were selected.

[0075] The reason for adopting the above inference logic is that the probability vector must simultaneously have a defined category index, probability value, and acquisition time. Fixing the feature normalization parameters and category index table avoids changes in the input scale and category meaning during inference, and the center frame timestamp avoids processing delays in overlapping duration calculations, thus providing repeatable input for speech candidate segment segmentation.

[0076] In step S14, the system acquires depth image sequences and color image sequences. The depth and color images can be acquired at a frame rate of 30 fps. Candidate frame rates can include 15 fps, 30 fps, and 60 fps, for example, corresponding to gesture classification verification losses of 0.34, 0.23, and 0.22, and average image processing times of 24 ms, 45 ms, and 88 ms, respectively. When 60 fps only slightly reduces the verification loss but significantly increases the processing time, 30 fps can be selected.

[0077] Furthermore, during initial installation, the system uses a dual-modal calibration board capable of detecting both the depth and color channels, instead of a standard printed checkerboard pattern recognizable only by the visible light camera. The dual-modal calibration board can feature a high-contrast visible light pattern and a raised / recessed structure capable of forming boundaries in the active infrared or depth channels. The system acquires calibration images at different positions and orientations within the camera's field of view, and determines the focal length, principal point, distortion parameters, and rotation and displacement parameters from the depth camera coordinate system to the color camera coordinate system for each camera based on the pixel position of the same calibration point in both the depth and color images.

[0078] Specifically, the number of calibration images is determined through a limited number of calibration experiments. For example, calibration is performed using 20, 30, and 40 sets of calibration images, respectively, yielding average reprojection errors of 1.2 pixels, 0.8 pixels, and 0.8 pixels on independent verification images. When the errors of 30 and 40 sets are the same and do not exceed the candidate allowable error of 1.0 pixel, the 30 sets with lower acquisition costs can be selected. Calibration is re-performed if the camera mounting position, focal length configuration, or housing structure changes.

[0079] Furthermore, the system selects the color image with the closest acquisition time for each depth image based on the acquisition time. If the acquisition time difference between two images exceeds the time matching tolerance, the system discards that set of images. The time matching tolerance is determined through historical synchronization data; for example, comparing three candidate tolerances of 10 ms, 15 ms, and 25 ms, the corresponding effective pairing ratios can be 84%, 96%, and 99%, and the corresponding average misalignment number of hand boundaries can be 3.8, 1.6, and 4.2 pixels, respectively. Considering both the effective pairing ratio and the number of boundary misalignments, 15 ms can be selected.

[0080] Specifically, when the depth image acquisition time is 1380 ms and the most recent color image acquisition time is 1374 ms, the former comes from the depth acquisition interface timestamp and the latter comes from the color acquisition interface timestamp. The difference between the two is 6 ms, which is less than the 15 ms time matching tolerance, thus allowing them to form a depth-color registration image. When the difference between the most recent color image and the depth image is 22 ms, this difference exceeds the time matching tolerance, and the system does not generate the corresponding registration image.

[0081] Furthermore, the system projects the effective depth pixels onto the color image coordinates based on the image registration parameters. If any depth value is missing, exceeds the effective range of the depth acquisition interface, or cannot complete spatial projection, that pixel will not be included in subsequent depth difference calculations. The effective range is determined through equipment calibration tests; for example, if a planar target is placed at known distances of 0.3 m, 0.5 m, 1 m, 2 m, 3 m, 4 m, and 5 m, and the depth error within the range of 0.3 m to 5 m does not exceed the deployment allowable error, then this range is written into the acquisition configuration; depth values ​​exceeding this range are marked as invalid.

[0082] Specifically, the system generates an absolute difference based on two valid depth values ​​at the same registered pixel location in adjacent registered images, and aggregates all valid absolute differences within the current interactive acquisition interval. The system obtains the median of the absolute differences, then obtains the median of the absolute deviation of each absolute difference from this median, and adds the two medians to obtain the depth change threshold. The depth change threshold is directly derived from the current image sequence, not an externally preset fixed threshold, and its physical unit is consistent with the output unit of the depth acquisition interface.

[0083] Furthermore, the system identifies pixels exceeding a depth variation threshold as foreground pixels and establishes connected regions based on 8-neighborhood. The connectivity rule is determined through comparison of a limited number of foreground samples; for example, in 200 foreground images with artificial hand boundary annotations, using 4-neighborhood produces 38 incorrectly segmented hand regions, while using 8-neighborhood produces 12 incorrectly segmented hand regions, thus 8-neighborhood is suitable. This rule ensures that diagonally adjacent hand pixels remain within the same connected region.

[0084] Specifically, when initially extracting the hand region, the interface prompts the user to place the pointing hand within the image acquisition range. The system selects the connected region with the largest number of pixels as the first hand region. In subsequent extractions, the system first selects the connected region with the largest number of overlapping pixels with the previous hand region; if the overlap is the same, the system determines the region based on the number of pixels and the scanning order of the top-left pixel. If the current image has three connected regions with overlapping pixel counts of 120, 35, and 0 with the previous hand region, the number of connected regions is derived from connectivity analysis of the current foreground image, and the overlap is derived from comparing pixel location sets. The system selects the connected region with an overlap of 120 pixels.

[0085] Furthermore, the number of consecutive missing moments is determined through occlusion verification samples. For example, candidate rules for terminating acquisition after 2, 3, and 4 consecutive missing image moments can be compared. In 100 verification interactions with short-term occlusion, the number of false terminations could be 12, 4, and 3, respectively, with end confirmation delays of approximately 70 ms, 100 ms, and 130 ms. Considering both the number of false terminations and the confirmation delay, terminating the current gesture acquisition interval after 3 consecutive missing image moments can be adopted.

[0086] Specifically, the system obtains the median of the x-coordinates and y-coordinates of all pixels in each hand region, uses these two values ​​to determine the center position of the hand region, and then generates a hand displacement vector based on the changes in adjacent center positions. The x-coordinate increases to the right, and the y-coordinate increases downwards. Using the median coordinates instead of the average pixel coordinates reduces the influence of a small number of edge noise pixels on the center position.

[0087] Furthermore, each hand displacement direction-distance histogram aggregates all valid hand displacement vectors within a probability output time interval. For example, with a probability output interval of 180 ms and an image acquisition frame rate of 30 fps, each output interval typically contains approximately 5 or 6 image intervals, thus enabling the aggregation of multiple displacement vectors. When there are no valid displacement vectors in a given interval, the system sets all histogram statistics for that interval to 0 and retains the interval's time position.

[0088] Specifically, the displacement direction is divided into 8 directional intervals according to the complete circumference. The displacement distance is first normalized to the length of the image diagonal pixels, and then statistically analyzed according to four distance intervals: no more than 0.01, greater than 0.01 and no more than 0.03, greater than 0.03 and no more than 0.06, and greater than 0.06. Each directional interval corresponds to one of the four distance intervals, therefore each histogram contains 32 statistical positions. When a direction is exactly at the interval boundary, it is assigned to the interval with the smaller clockwise direction number.

[0089] Furthermore, the number of direction intervals and distance intervals are determined using a limited number of gesture verification samples. For example, 8 and 12 direction intervals, as well as 3, 4, and 5 distance intervals, can be compared. When using 8 direction intervals and 4 distance intervals, the verification loss is 0.23; when using 12 direction intervals and 4 distance intervals, the verification loss is 0.24; and when using 8 direction intervals and 5 distance intervals, the verification loss is 0.25. Therefore, 8 direction intervals and 4 distance intervals can be selected.

[0090] Specifically, the distance interval boundaries are determined using historical displacement vectors. The system can collect 1200 manually verified valid hand displacement vectors, perform image diagonal normalization on each displacement distance, and sort them in ascending order. Then, it generates boundary combinations from multiple candidate quantile positions. For example, values ​​of 0.01, 0.03, and 0.06 in the candidate combinations ensure that each distance interval contains sufficient training samples and corresponds to the lowest validation loss; therefore, these values ​​are incorporated into the gesture target classification parameters. The aforementioned boundaries are dimensionless values ​​and are not directly affected by image resolution.

[0091] Furthermore, the system generates a gesture feature sequence based on a continuous hand displacement direction-distance histogram and classifies it using gesture target classification parameters. The classification output undergoes the same normalization exponent processing as speech classification, forming a gesture target probability vector where the sum of probabilities for each category is 1. The system uses the acquisition time of the registered image near the center of the corresponding output interval as a timestamp.

[0092] Specifically, a 180 ms gesture probability output interval is determined using limited validation data. For example, three candidate intervals—120 ms, 180 ms, and 240 ms—can be compared, corresponding to median boundary time errors of 40 ms, 50 ms, and 90 ms for gesture segments, respectively, with the average classification processing workload decreasing sequentially. If 180 ms meets the boundary error requirements and significantly reduces processing workload, this interval can be adopted. The gesture output interval can be the same as or different from the speech output interval; subsequent pairing relies solely on timestamps.

[0093] The reason for adopting the above image processing logic is that depth images and color images must first complete temporal matching and spatial registration before they can be compared with corresponding depth pixels at adjacent time points. A depth change threshold is generated from the current depth difference distribution, which can adjust the foreground separation criteria according to sensor noise and scene changes; maintaining the temporal continuity of the hand region by utilizing the overlapping relationships of connected regions can reduce the switching between different motion regions at adjacent time points; and multiple displacement vectors are aggregated according to the output time interval to form a direction-distance histogram, which can provide a statistically significant fixed-dimensional input for gesture target classification.

[0094] In one implementation, such as Figure 3 As shown, S2 includes: S21. Generate the maximum probability class sequence based on the maximum probability class of each probability vector in the probability sequence.

[0095] S22. Generate a probability interval sequence based on the difference between the maximum probability value and the second largest probability value in each probability vector.

[0096] S23. Generate a candidate fragment sequence based on the highest probability category sequence and the probability interval sequence.

[0097] S24. Generate a fragment validity score based on the candidate fragment sequence, and identify candidate fragments with positive fragment validity scores as fragments to be paired.

[0098] In this embodiment, it should be noted that in S21, the input is a time-stamped voice operation probability sequence or a time-stamped gesture target probability sequence. The system reads all category probabilities from each probability vector, selects the category corresponding to the highest probability value, and retains the corresponding timestamp to form a sequence of the highest probability categories. When multiple categories have the same highest probability value, the system determines the highest probability category according to the fixed category arrangement order stored in the classification parameters, without performing random selection.

[0099] Furthermore, the fixed category order is determined by the output category index table before training. For speech classification, for example, the display operation category can be configured as output index 1, and the stop display operation category can be configured as output index 2. For gesture classification, the output index can be determined according to the byte order of the terminal identifier. The output index comes from the classification parameter configuration and is only used to determine the position of the category vector, the category with the highest probability in a tie, and the result of the category ranking; it does not change the probability value of each category.

[0100] In step S22, the system reads the maximum and second-highest probability values ​​from each probability vector and determines the difference between them as the probability interval. The probability interval, defined in this application, is a dimensionless difference used to represent the degree of distinction between the highest probability category and the second-highest probability category, and its value ranges from 0 to 1. If the maximum and second-highest probability values ​​are the same, the probability interval is 0.

[0101] For example, when the maximum probability value is 0.52 and the second-highest probability value is 0.48, the two values ​​come from the same probability vector, and the difference between them is 0.04; when the maximum probability value is 0.76 and the second-highest probability value is 0.24, the difference between them is 0.52. The latter probability vector shows a greater degree of distinction between the highest and second-highest probability categories.

[0102] In step S23, the system arranges probability intervals according to timestamps and calculates the absolute value of the difference between adjacent probability intervals in time, forming a sequence of absolute differences between adjacent probability intervals. The median across the entire data uses the same processing rules: when the number of data points is odd, the median value is taken after ascending order; when the number of data points is even, the arithmetic mean of the two median values ​​is taken. The absolute deviation of the median uses the original value without scaling.

[0103] Furthermore, the system obtains the median of the sequence of absolute differences between adjacent probability intervals, and then obtains the absolute deviation of each absolute difference between adjacent probability intervals relative to this median, forming a sequence of absolute deviations. The system obtains the median of the sequence of absolute deviations as the median absolute deviation, and adds the median of the absolute differences between adjacent probability intervals to the median absolute deviation to obtain the adaptive change threshold. The adaptive change threshold is directly calculated from the current probability sequence and does not require an externally preset value.

[0104] Specifically, the absolute difference between adjacent probability intervals corresponds to the position of the next probability vector in time. When the absolute difference between adjacent probability intervals exceeds the adaptive change threshold, the system determines the position of the next probability vector as the probability change position. When adjacent maximum probability classes are different, the system also determines the position of the next probability vector as the maximum probability class switching position. Absolute differences equal to the adaptive change threshold do not generate probability change positions.

[0105] Furthermore, the system merges the probability change positions and the maximum probability category switching positions, performs deduplication and time sorting, and then adds the position of the first probability vector of the probability sequence and the virtual termination position after the last probability vector to the sorting result. The virtual termination position is only used to close the last candidate segment and does not correspond to an actual probability vector. The probability vector corresponding to each internal boundary position becomes the first probability vector of the new candidate segment, so that each probability vector belongs to only one candidate segment.

[0106] Specifically, the system stops processing segments of the probability sequence when the input probability sequence has fewer than three probability vectors or when any probability vector has fewer than two class probabilities. The minimum of three probability vectors is determined by the formation condition of adjacent direction pairs: three probability intervals can generate two directions of change, thus generating one adjacent direction pair. This minimum number is directly determined by the data structure relationship, not by an empirically set threshold.

[0107] The reason for adopting the above boundary processing logic is that the probability change position is used to identify the position where the classification distinction degree changes significantly, and the position of the highest probability class switch is used to identify the position of class change. By uniformly taking the latter probability vector as the starting point of the new segment and explicitly adding it to the beginning and end of the sequence, the problems of unclear boundary vector assignment, omission of the first and last probability vectors, or duplicate assignment can be eliminated.

[0108] In step S24, the system counts the number of times each category becomes the most probable category for each candidate fragment, generating a category count table. For each category, the system reads the probability value corresponding to that category from all probability vectors within the candidate fragment, not just the probability value when that category becomes the most probable category, and obtains the median of these probability values ​​to generate a category probability median table.

[0109] Furthermore, the system first sorts the categories in descending order of the number of occurrences; if the number of occurrences of a category is the same, it sorts them in descending order of the median probability of the category; if the median probabilities of the categories are still the same, they are sorted according to a fixed category order. The category with the highest ranking is determined as the primary category. The primary category is the representative category of the candidate fragment as defined in this application, and is not equivalent to the highest probability category of a single probability vector.

[0110] Specifically, the proportion of the number of occurrences of the main category to the total number of probability vectors within the candidate fragments is determined as the category consistency ratio. The system also obtains the median of all probability intervals within the candidate fragments and takes the smaller value between the category consistency ratio and the median of the probability intervals as the fragment baseline score. The fragment baseline score is an intermediate score defined in this application, used to simultaneously satisfy the constraints of category continuity and classification discrimination.

[0111] Furthermore, the system compares adjacent probability intervals in chronological order. An ascending flag is generated when the later probability interval is greater than the earlier one, and a descending flag is generated when the later probability interval is less than the earlier one. An invariant flag is generated when the original floating-point numbers stored in both are exactly the same. The system pairs two temporally adjacent flags together as adjacent direction pairs.

[0112] Specifically, adjacent direction pairs where an ascending marker is followed by a descending marker, and vice versa, are identified as inverted adjacent direction pairs. All other adjacent direction pairs are identified as non-inverted adjacent direction pairs. Non-inverted adjacent direction pairs are defined in this application to indicate that the probability interval between two consecutive changes does not directly switch between ascending and descending. The system determines the non-inverted ratio based on the proportion of non-inverted adjacent direction pairs to the total number of adjacent direction pairs.

[0113] Furthermore, when a candidate segment has fewer than three probability vectors, adjacent direction pairs cannot be formed. The system retains the candidate segment and its boundary information, but does not generate a segment validity score, nor does it add the candidate segment to the set of segments to be paired. All probability comparisons, median calculations, ratio calculations, and sorting all use the saved raw floating-point numbers, without using display-oriented rounding values.

[0114] The segment validity score is obtained based on the category consistency ratio, median probability interval, number of non-reversed adjacent direction pairs, and total number of adjacent direction pairs. The segment validity score can be expressed as:

[0115] In the above formula: Indicates the sequence number is The fragment validity score corresponding to the candidate fragment, with a value range of 100%. to It does not carry physical units; This indicates the index of a candidate segment in the candidate segment sequence that has a sequence of non-empty adjacent direction pairs; This indicates that the smaller value is selected from the values ​​being compared. Indicates the sequence number is The category consistency ratio corresponding to the candidate fragment is determined by the number of occurrences of the main category of the candidate fragment and the total number of probability vectors, with a value range of [value missing]. to It does not carry physical units; Indicates the sequence number is The median probability interval corresponding to a candidate segment is determined by all probability intervals within that candidate segment, and its value ranges from [value missing]. to It does not carry physical units; Indicates the sequence number is The number of non-reversed adjacent direction pairs corresponding to the candidate segment is a non-negative integer; Indicates the sequence number is The total number of adjacent direction pairs corresponding to the candidate fragment is a positive integer; Indicates the sequence number is The non-inverted ratio corresponding to the candidate fragment, with a value range of . to It does not carry physical units.

[0116] Specifically, the calculation logic of this mathematical model is as follows: First, the smaller value between the category consistency ratio and the median probability interval is taken to obtain the basic score of the segment; then, the number of non-reversed adjacent direction pairs is used as the statistical quantity, and the total number of adjacent direction pairs is used as the statistical total to obtain the non-reversal ratio; finally, the smaller value between the basic score of the segment and the non-reversal ratio is taken to obtain the segment effectiveness score. The ratio calculation is only performed when the total number of adjacent direction pairs is a positive integer.

[0117] Furthermore, this mathematical model is adopted because the proportion of class consistency, the median probability interval, and the proportion of non-reversal are all in the dimensionless range of 0 to 1, allowing for numerical comparison. Gradually taking smaller values ​​limits the segment validity score when any evaluation factor is insufficient, avoiding compensating for a lower value with a higher one.

[0118] For example, a candidate segment contains 5 probability vectors; the number 5 comes from the actual number of probability vectors stored within the boundary of the candidate segment. The main category appears in all 5 probability vectors, therefore... We set the value to 1.00. The probability intervals within the candidate fragments are 0.30, 0.42, 0.50, 0.46, and 0.38, respectively. These values ​​are derived from the difference between the first two class probabilities in the corresponding probability vectors. The median value after sorting is 0.42. Take 0.42.

[0119] Furthermore, the aforementioned probability interval changes in the following directions: upward, upward, downward, and downward, forming three pairs of adjacent directions. The number of three pairs of adjacent directions originates from the number of combinations of adjacent markers in the four changing directions, two of which are non-reversed pairs of adjacent directions. Take 2, Taking 3, the non-inverted ratio is displayed as 0.67. Substituting the aforementioned data into the above formula, the basic score of the segment is first obtained as 0.42, and then compared with the unrounded non-inverted ratio to finally obtain... It is 0.42.

[0120] The above calculation logic solves the problem of obtaining high scores for candidate segments with continuous categories but insufficient category differentiation, obvious category differentiation but frequent category switching, and frequent changes in probability intervals, and provides evaluation data within the same numerical range for subsequent screening of segments to be paired.

[0121] Furthermore, the system identifies candidate segments with positive segment validity scores as segments to be paired. Positive values ​​are used as the criterion for entering the pairing process because when the class consistency ratio, median probability interval, or non-inversion ratio is 0, the candidate segment does not provide effective support, at least in terms of class discrimination or continuity of change. This judgment does not require setting an additional external scoring threshold.

[0122] In one implementation, such as Figure 4 As shown, S3 includes: S31. Generate candidate segment pairs based on the set of speech segments to be paired and the set of gesture segments to be paired.

[0123] S32. Generate the overlap duration and segment pair score based on the start and end times of each candidate segment pair.

[0124] S33. Determine the segment pairing results in sequence according to the overlap duration, segment pair score, and overlap end time.

[0125] In this embodiment, it should be noted that in S31, the system writes the segments to be paired from the timestamped voice operation probability sequence into the voice segment to be paired set, and writes the segments to be paired from the timestamped gesture target probability sequence into the gesture segment to be paired set. When either set is empty, the system stops the segment pairing process and does not generate a collaborative display instruction.

[0126] Furthermore, the system combines each speech segment to be paired with each gesture segment to be paired. This combination only indicates that time interval discrimination needs to be performed, not that a pairing has been determined. By using item-by-item combination between two sets, rather than pre-selecting a unique segment within a single modality, multiple potentially consecutive segments can be compared based on actual overlap, and the sorting branches of multiple candidate segment pairs can also be actually executed.

[0127] In step S32, the system reads the start time, end time, and segment validity score of the two segments within each segment combination to be judged. The system determines the later of the two start times as the overlap start time and the earlier of the two end times as the overlap end time, and then obtains the time length between the overlap end time and the overlap start time.

[0128] Furthermore, when the overlap duration is positive, the corresponding combination is written into the candidate segment pair sequence; when the overlap duration is 0 or negative, the corresponding combination is not included in the candidate segment pair sequence. Using a strictly positive value condition can exclude segment combinations that only contact at boundary times but do not share a common duration interval. The system also takes the smaller value of the validity scores of the two segments in the same candidate segment pair as the segment pair score.

[0129] In S33, the system sorts candidate segment pairs in descending order of overlap duration. When the maximum overlap duration corresponds to only one candidate segment pair, that candidate segment pair directly becomes the segment pairing result. When the maximum overlap duration corresponds to multiple candidate segment pairs, the system sorts them in descending order of segment pair score only within these candidate segment pairs. When the maximum segment pair score still corresponds to multiple candidate segment pairs, the system sorts them in descending order of overlap end time and selects the candidate segment pair with the highest score.

[0130] For example, if two candidate segment pairs both have an overlap duration of 580 ms, where 580 ms is derived from the overlap end time and overlap start time of each segment pair, and the two segment pairs have scores of 0.42 and 0.39 respectively, where the score is derived from the smaller of the validity scores of the two segments within each segment pair, the system selects the candidate segment pair with a score of 0.42. If the two segment pairs also have the same score, but their overlap end times are 1960 ms and 1780 ms respectively, the system selects the candidate segment pair with an overlap end time of 1960 ms.

[0131] The reason for adopting the above sorting logic is that the overlap duration reflects the degree to which the two modalities coexist in time, the segment pair score reflects the effectiveness of the weaker segment in the pair, and the overlap end time provides the final time position determination. The three sorting conditions are executed in a fixed order, so that the same input produces a definite segment pairing result, without the need for random selection or a fixed pairing time window.

[0132] In one implementation, such as Figure 5 As shown, S4 includes: S41. Generate operation codes based on the main speech category in the segment pairing results.

[0133] S42. Generate the target terminal identifier based on the gesture main category in the fragment pairing results.

[0134] S43. Generate the current display content identifier and content provision record based on the current display status record.

[0135] S44. Generate collaborative display instructions based on the source terminal identifier, operation code, target terminal identifier, and current display content identifier.

[0136] S45. Generate versioned collaborative display instructions based on the collaborative display instructions and logical version number.

[0137] S46. Generate a collaborative display message based on the versioned collaborative display instruction and the terminal connection mapping table.

[0138] S47. Generate an execution license determination result based on the received logic version number in the collaborative display message and the executed logic version number saved by the target smart terminal with screen.

[0139] S48. Generate display execution tasks and perform corresponding operations based on the collaborative display message and execution permission determination results.

[0140] S49. Generate collaborative display execution results based on the processing status of the displayed task.

[0141] In this embodiment, it should be noted that in S41, the system reads the main voice category from the segment pairing results. When the main voice category is a display operation category, the system reads the display operation code; when the main voice category is a stop display operation category, the system reads the stop display operation code. The operation code table is established and saved during protocol deployment, and the codes remain unique within the same protocol version.

[0142] Furthermore, the specific values ​​of the operation codes are not calculated using a probability sequence, but are determined based on field length, terminal compatibility, and code uniqueness. For example, the display operation code can be configured as 01, and the stop display operation code as 02. The values ​​01 and 02 originate from unoccupied coding positions in the protocol coding table. Before formal deployment, interoperability testing can be performed using three types of processor platforms and two types of communication interfaces, forming a total of six terminal combinations. When each combination interprets 01 as a display operation and 02 as a stop display operation, the coding table and protocol version are written into the terminal configuration. The number of platforms and interfaces is only used to illustrate the limited interoperability verification process.

[0143] In step S42, the system reads the main gesture category from the matching results of the same segment and obtains the target terminal identifier based on the output index table saved by the gesture target classification parameters. The target terminal identifier is allocated and written to persistent storage during terminal registration and does not change with the network address. When the network address changes, only the terminal connection mapping table is updated, without changing the meaning of the output category of the gesture classification network.

[0144] In step S43, the system reads the display status record associated with the current display buffer. The display buffer stores pixel or graphics resource references, while the display status record stores the unique identifier of the content related to the currently displayed object, the content type, the source terminal identifier, and the content retrieval method. The system determines the unique identifier of the content in the display status record as the identifier of the currently displayed content.

[0145] Furthermore, for local static content, the content retrieval method may include the local content reading location and content summary; for web pages or dynamic interfaces, the content retrieval method may include a page resource list, interface state data, and resource retrieval address; for video, the content retrieval method may include media data stream session and encoding type. The system writes the currently displayed content identifier and the content retrieval method into the content provision record. The content provision record is a mapping record defined in this application, used to enable the target terminal to obtain the actual content based on the currently displayed content identifier.

[0146] Specifically, the unique content identifier can be generated based on the source terminal identifier, the local monotonic content sequence number, and the content digest. The content digest algorithm is determined through terminal compatibility and collision verification. For example, the SHA-256 digest algorithm, which is supported by all deployed terminals, can be selected to repeatedly generate digests on 1000 historical display objects; when the same object generates the same digest repeatedly, and different objects do not generate the same digest, this algorithm is written into the protocol configuration. The sample size of 1000 comes from the limited set of historical display objects available before deployment and does not mean that this number must be reached.

[0147] In S44, the system identifies the smart terminal with a screen that generates the collaborative display command as the source terminal identifier. Then, it combines the source terminal identifier, operation code, target terminal identifier, and current display content identifier according to a fixed field order. This fixed field order is written into the protocol configuration, enabling the receiving end to parse the fields in the same order.

[0148] Furthermore, collaborative display instructions can be encoded using a sequence of binary fields with a length field. The protocol header records the protocol version and the total message length. Each variable-length field first records the field type and length, then the field content. Multi-byte integers are transmitted in network byte order. The field encoding method is determined through limited interoperability testing; for example, if the same test message can be parsed in six different combinations of processors and communication interfaces, the corresponding encoding method is written into the protocol configuration.

[0149] In step S45, the system identifies the smart terminal with a screen that generates the collaborative display instruction as the source smart terminal with a screen. The source smart terminal with a screen determines the version status item based on the source terminal identifier and the current display content identifier. The version status item is a persistent status record defined in this application, used to separately store the logical version number corresponding to a specific source terminal and specific display content.

[0150] Furthermore, each version status item without a history record is initialized with 0 as its logical version number. An initial value of 0 indicates that no version has been executed and does not correspond to any actual collaborative display instruction. The source smart terminal with a screen reads the current logical version number, obtains the next logical version number, and writes it back within the same atomic storage operation, avoiding concurrent collaborative display instructions obtaining the same version. The logical version number is stored as a variable-length non-negative integer to prevent wraparound after the fixed-length count reaches its limit.

[0151] In step S46, the source smart terminal with a display screen obtains the terminal connection mapping table. This table stores the target terminal identifier, communication address, supported content types, authentication status, and most recent activity time. The target terminal registers or updates its connection record through authentication, and the source smart terminal with a display screen sends versioned collaborative display instructions using the latest valid record corresponding to the same target terminal identifier.

[0152] Furthermore, the connection failure time is determined using limited historical connection data. For example, comparing three candidate failure times—10 s, 30 s, and 60 s—results in 18, 6, and 3 invalid address connections from 300 historical connection records, along with 900, 300, and 150 maintenance messages. Considering both the number of invalid connections and the maintenance communication volume, 30 s can be selected. The aforementioned 300 records are from pre-deployment limited connection tests; the failure time is not calculated from a single interaction probability sequence.

[0153] Specifically, the source smart terminal with screen generates a unique message identifier based on the source terminal identifier and the local monotonically increasing message sequence number. The local message sequence number is atomically incremented in persistent storage; when the same message is resent due to network retransmission, the original unique message identifier is used, and a new identifier is not generated. The collaborative display message may include the unique message identifier, source terminal identifier, operation code, target terminal identifier, current display content identifier, current logical version number, and message generation time.

[0154] In step S47, the target smart terminal with a screen first verifies whether the target terminal identifier matches the local terminal identifier. If the identifiers do not match, a rejection result indicating a target mismatch is generated. If the identifiers match, the target smart terminal with a screen obtains the source terminal identifier, the current display content identifier, and the received logic version number from the collaborative display message, and reads the executed logic version number from the corresponding version status item.

[0155] Furthermore, when the received logical version number is after the executed logical version number, the target smart terminal with a screen generates an execution permission determination result allowing execution; when the received logical version number is the same as or precedes the executed logical version number, an execution permission determination result rejecting execution is generated. This rule adopts a latest-state-priority approach, allowing the rejection of subsequently arriving older versions after receiving a newer version.

[0156] For example, if a version status item has an executed logical version number of 7, this value comes from the most recently successfully executed record persistently saved on the target terminal. When the target terminal receives a message with a logical version number of 9, the value 9 comes from the collaborative display message. Because it is after 7, the target terminal allows execution and updates the version status to 9 after successful execution. Subsequently, when a message with a logical version number of 8 arrives or a message with a logical version number of 9 arrives again, the target terminal refuses execution to avoid state rollback or duplicate execution.

[0157] In step S48, when the execution permission determination result is "permitted," the target smart terminal with a screen generates a display execution task based on the operation code. When the operation code is a display operation code, the target smart terminal with a screen establishes a content request connection based on the source terminal identifier, submits the current display content identifier, obtains the corresponding file data, page resource list, or media data stream, and executes the display after completing the content summary verification. When the operation code is a stop display operation code, the target smart terminal with a screen stops the display task corresponding to the current display content identifier.

[0158] Furthermore, when a content request fails, the content summary is inconsistent, the content type is unsupported, or the display interface returns a failure status, the target smart terminal with a screen does not update the executed logic version number and generates a corresponding failure reason code. After successfully displaying or stopping the display, the target smart terminal with a screen saves the received logic version number as the executed logic version number. Version comparison, display execution, and version write-back are completed within a mutually exclusive processing interval to avoid concurrent messages being judged by the same version at the same time.

[0159] In step S49, the target smart terminal with a screen generates a collaborative display execution result based on the processing status. The collaborative display execution result includes a unique message identifier, processing completion time, execution status code, failure or rejection reason code, and the processed logical version number. If execution is successful, the actual execution time is also recorded; if rejected or failed, the actual execution time is not entered. The source smart terminal with a screen matches the collaborative display execution result with the original collaborative display message based on the unique message identifier.

[0160] The reason for adopting the above version control and result feedback logic is that network messages may arrive repeatedly or in different orders, and content retrieval and display execution may fail independently. By using the source terminal identifier and the current displayed content identifier to jointly determine the version status item, the version status of different content and different source terminals can be isolated; by using atomic version updates and explicit status coding to process the execution results, a complete closed loop can be formed from instruction generation, message transmission, version discrimination, content retrieval, display execution to result feedback.

[0161] This invention also provides a collaborative data processing system for screen-equipped smart terminals based on multimodal interaction, such as... Figure 6 As shown, the system includes the following physical and logical modules: The multimodal probability acquisition module is used to acquire the probability sequence of time-stamped voice operations and the probability sequence of time-stamped gesture targets. The segment processing module is used to generate the maximum probability class sequence and the probability interval sequence based on each probability sequence, divide candidate segments according to the adaptive change threshold and the switching position of the maximum probability class, determine the segment validity score according to the class consistency ratio, the median probability interval and the non-inversion ratio, and determine the candidate segments with positive segment validity scores as segments to be paired. The segment matching module is used to generate candidate segment pairs based on the speech segment to be matched and the gesture segment to be matched, and to determine the segment matching result according to the overlap duration, segment pair score and overlap end time in sequence; The collaborative instruction module is used to generate collaborative display instructions based on the main voice category, main gesture category, and current display content identifier in the segment pairing results.

[0162] Furthermore, in one embodiment, the multimodal probability acquisition module includes a voice acquisition interface, a depth image acquisition interface, a color image acquisition interface, a monotonic clock interface, a voice classification processor, and a gesture classification processor. The voice acquisition interface outputs a voice sampling sequence, and the voice classification processor outputs a time-stamped voice operation probability sequence; the depth image acquisition interface and the color image acquisition interface output corresponding image sequences, and the gesture classification processor performs time matching, spatial registration, hand region extraction, displacement feature generation, and gesture classification, outputting a time-stamped gesture target probability sequence.

[0163] Specifically, in one implementation, the segment processing module includes speech segment processing logic and gesture segment processing logic executed by a processor. Both processing logics receive probability vectors and timestamps, and sequentially generate a maximum probability category sequence, a probability interval sequence, an adaptive change threshold, candidate segments, segment validity scores, and segments to be paired. The meanings of speech categories and gesture categories differ, but the data structure of the probability vectors is the same; therefore, both modalities can use the same probability interval and segment scoring processing logic.

[0164] Furthermore, in one embodiment, the segment pairing module receives a set of voice segments to be paired and a set of gesture segments to be paired, and generates an overlap start time, an overlap end time, an overlap duration, a segment pair score, and a segment pairing result. The collaborative instruction module receives the segment pairing result and the current display status record, and generates an operation code, a target terminal identifier, a current display content identifier, a content provision record, and a collaborative display instruction.

[0165] Specifically, in one optional implementation, the system further includes a display execution module. The display execution module includes version discrimination processing logic, content retrieval processing logic, a display control interface, and execution result generation logic. The version discrimination processing logic compares the received logic version number with the executed logic version number; the content retrieval processing logic retrieves the display content based on the current display content identifier; the display control interface performs a display operation or stops the display operation; and the execution result generation logic generates the collaborative display execution result.

[0166] Furthermore, in one embodiment, the system includes a source smart terminal with a screen and a target smart terminal with a screen. The source smart terminal with a screen includes at least a processor, memory, a display, a voice acquisition interface, a depth image acquisition interface, a color image acquisition interface, a monotonic clock interface, and a communication interface. The target smart terminal with a screen includes at least a processor, memory, a display, a monotonic clock interface, and a communication interface. Each processor is connected to its corresponding interface via a local data bus, and the communication interface of the source smart terminal with a screen is connected to the communication interface of the target smart terminal with a screen via a wired or wireless data link.

[0167] Specifically, the speech sampling sequence and image sequence enter the multimodal probability acquisition module through the speech acquisition interface and image acquisition interface, respectively, forming a timestamped probability sequence. The timestamped probability sequence enters the segment processing module, forming a set of speech segments to be paired and a set of gesture segments to be paired. These two sets enter the segment pairing module, generating a segment pairing result. The segment pairing result and the current display status record enter the collaborative instruction module, generating a collaborative display message. The collaborative display message is transmitted to the target smart terminal with a screen through the communication interface. The display execution module generates a display execution task and a collaborative display execution result, which is then returned to the source smart terminal with a screen through the communication interface.

[0168] The following uses a specific application scenario to integrate and verify the calculation logic and data flow process of each step in the above implementation method.

[0169] In this scenario, the smart terminal with a screen in the living room is currently displaying the family photo album page. The smart terminals with screens in the bedroom and study belong to the set of registered target terminals. The candidate terminal count of 2 is derived from the terminal registry read by the smart terminal with the screen in the living room before this interaction begins. The user says "Show current content" to the smart terminal with the screen in the living room and points to the smart terminal with the screen in the bedroom. The current displayed content identifier of the family photo album page is obtained from the display state record associated with the current display buffer.

[0170] Furthermore, in this application scenario, the selected speech feature frame shift of 10 ms and probability output step size of 18 frame shifts are used in the aforementioned limited verification process, resulting in a speech probability output interval of 180 ms. The gesture probability output interval is also 180 ms, selected through gesture verification data. The identical output intervals are used only for ease of displaying continuous data; the segment pairing of this invention does not require the two modalities to have the same output frequency. The following probability values ​​are precise example values ​​saved after the normalized output of the classification network, and do not contain any undisclosed decimal places.

[0171] In S1, the smart terminal with a screen in the living room generates 7 voice operation probability vectors. The number 7 comes from the actual number of probability vectors obtained within the voice acquisition interval with an output step size of 180 ms. The first timestamp of 1060 ms comes from the monotonic clock after the first frame to be classified is converted. Subsequent timestamps are all increased by 180 ms from the previous timestamp, namely 1240 ms, 1420 ms, 1600 ms, 1780 ms, 1960 ms, and 2140 ms.

[0172] Specifically, at 1060 ms, the probability of the display operation category is 0.25, and the probability of the stop display operation category is 0.75; at 1240 ms, they are 0.65 and 0.35 respectively; at 1420 ms, they are 0.71 and 0.29 respectively; at 1600 ms, they are 0.75 and 0.25 respectively; at 1780 ms, they are 0.73 and 0.27 respectively; at 1960 ms, they are 0.69 and 0.31 respectively; and at 2140 ms, they are 0.24 and 0.76 respectively. Each probability group is derived from the normalized exponent output of the speech classification network, and the sum of the two probability values ​​in each group is 1.

[0173] Furthermore, the smart terminal with a screen in the living room generates 7 probability vectors for the gesture targets. The number 7 comes from the actual number of probability vectors obtained within the gesture acquisition interval at 180 ms output intervals. The first timestamp of 1200 ms comes from the monotonic clock of the registered image near the center of the first gesture output interval. Subsequent timestamps are all increased by 180 ms from the previous timestamp, namely 1380 ms, 1560 ms, 1740 ms, 1920 ms, 2100 ms, and 2280 ms.

[0174] Specifically, at 1200 ms, the probability corresponding to the bedroom terminal is 0.24, and the probability corresponding to the study terminal is 0.76; at 1380 ms, they are 0.68 and 0.32 respectively; at 1560 ms, they are 0.72 and 0.28 respectively; at 1740 ms, they are 0.75 and 0.25 respectively; at 1920 ms, they are 0.73 and 0.27 respectively; at 2100 ms, they are 0.70 and 0.30 respectively; and at 2280 ms, they are 0.23 and 0.77 respectively. Each probability is derived from the normalized exponent output of the gesture classification network, and the sum of the two probability values ​​in each group is 1.

[0175] In step S2, the system first processes the probability sequence of voice operations. The categories with the highest probabilities are, in order: stop display, display, display, display, display, and stop display. Each category is derived from the larger of the probabilities of two voice categories at the same time.

[0176] Specifically, the speech probability intervals are 0.50, 0.30, 0.42, 0.50, 0.46, 0.38, and 0.52, respectively. The 0.50 interval originates from the difference between 0.75 and 0.25 at 1060 ms; the 0.30 interval originates from the difference between 0.65 and 0.35 at 1240 ms; the 0.42 interval originates from the difference between 0.71 and 0.29 at 1420 ms; the second 0.50 interval originates from the difference between 0.75 and 0.25 at 1600 ms; the 0.46 interval originates from the difference between 0.73 and 0.27 at 1780 ms; the 0.38 interval originates from the difference between 0.69 and 0.31 at 1960 ms; and the 0.52 interval originates from the difference between 0.76 and 0.24 at 2140 ms.

[0177] Furthermore, the absolute differences between adjacent speech probability intervals are 0.20, 0.12, 0.08, 0.04, 0.08, and 0.14, respectively. Each value is derived from the absolute value of the difference between two temporally adjacent probability intervals. Arranged in ascending order, these are 0.04, 0.08, 0.08, 0.12, 0.14, and 0.20. The number of data points is 6, derived from the number of adjacent relationships formed by 7 probability intervals; the middle two values ​​are 0.08 and 0.12, and their arithmetic mean is 0.10. Therefore, the median of the absolute differences between adjacent probability intervals is 0.10.

[0178] Specifically, the absolute deviations of the absolute differences between adjacent probability intervals relative to 0.10 are 0.10, 0.02, 0.02, 0.06, 0.02, and 0.04, respectively. Arranged in ascending order, these are 0.02, 0.02, 0.02, 0.04, 0.06, and 0.10. The middle two values ​​are 0.02 and 0.04, and their arithmetic mean is 0.03, therefore the median absolute deviation is 0.03. Adding the median absolute difference of the absolute differences between adjacent probability intervals (0.10) to the median absolute deviation (0.03), the adaptive change threshold is 0.13.

[0179] Furthermore, the absolute differences of 0.20 and 0.14 exceed the adaptive change threshold of 0.13, corresponding to positions of the next probability vector at 1240 ms and 2140 ms, respectively. The highest probability class also switches at these two positions. After merging the boundary positions, the system forms three candidate segments, with the number three originating from the continuous interval formed by the two internal boundaries and the beginning and end of the probability sequence.

[0180] Specifically, the first speech candidate segment contains only a 1060 ms probability vector, and the last speech candidate segment contains only a 2140 ms probability vector. Since both have only one probability vector, they cannot form adjacent direction pairs and therefore do not generate a segment validity score. The middle speech candidate segments start at 1240 ms and end at 1960 ms, containing five probability vectors corresponding to 1240 ms, 1420 ms, 1600 ms, 1780 ms, and 1960 ms.

[0181] Furthermore, all five most probable categories of the intermediate speech candidate segments are display operation categories, with five occurrences of the main category and a total of five probability vectors, resulting in a category consistency ratio of 1.00. The intra-segment probability intervals are 0.30, 0.42, 0.50, 0.46, and 0.38, which, when sorted in ascending order, are 0.30, 0.38, 0.42, 0.46, and 0.50, with a median value of 0.42. Therefore, the median probability interval is 0.42, and the basic segment score is 0.42.

[0182] Specifically, the probability intervals of the intermediate speech candidate segments change in the following directions: rising, rising, falling, and falling. The number 4 of these directions originates from the adjacency relationships formed by the 5 probability intervals. These directions form 3 adjacent direction pairs, where rising-rising and falling-falling are non-reversed adjacent direction pairs, and rising-falling is a reversed adjacent direction pair. There are 2 non-reversed adjacent direction pairs, bringing the total number of adjacent direction pairs to 3. The displayed value for the non-reversed proportion is 0.67. Based on the aforementioned unique formula, the segment validity score of the intermediate speech candidate segment is 0.42, thus making it a speech segment to be paired.

[0183] The system then processes the probability sequence of the gesture target. The highest probability categories are, in order, study room terminal, bedroom terminal, bedroom terminal, bedroom terminal, bedroom terminal, and study room terminal. Each category is derived from the larger of the probability of the bedroom terminal and the probability of the study room terminal at the corresponding time.

[0184] Specifically, the gesture probability intervals are 0.52, 0.36, 0.44, 0.50, 0.46, 0.40, and 0.54, respectively, derived from the differences between the probabilities of two terminals at the same time. The absolute differences between adjacent probability intervals are 0.16, 0.08, 0.06, 0.04, 0.06, and 0.14, respectively.

[0185] Furthermore, the absolute differences in the probability intervals between adjacent gestures, when sorted in ascending order, are 0.04, 0.06, 0.06, 0.08, 0.14, and 0.16, with the middle two values ​​being 0.06 and 0.08. Their arithmetic mean is 0.07, therefore the median is 0.07. The absolute deviations of each difference from 0.07 are 0.09, 0.01, 0.01, 0.03, 0.01, and 0.07, respectively. When sorted in ascending order, they are 0.01, 0.01, 0.01, 0.03, 0.07, and 0.09, with the middle two values ​​being 0.01 and 0.03. Their arithmetic mean is 0.02, therefore the median absolute deviation is 0.02. Adding the median of 0.07 to the median absolute deviation of 0.02, the adaptive change threshold is 0.09.

[0186] Specifically, the absolute differences of 0.16 and 0.14 exceed the adaptive change threshold of 0.09, corresponding to the positions of the next probability vector at 1380 ms and 2280 ms, respectively. The highest probability category also switches at these two positions. The system generates three gesture candidate segments; the first and last candidate segments each contain only one probability vector and cannot form a segment validity score.

[0187] Furthermore, the intermediate gesture candidate segments start at 1380 ms and end at 2100 ms, containing five probability vectors corresponding to 1380 ms, 1560 ms, 1740 ms, 1920 ms, and 2100 ms. All the highest probability categories are bedroom terminals, the main category appears 5 times, the total number of probability vectors is 5, and the category consistency ratio is 1.00.

[0188] Specifically, the probability intervals of the middle gesture candidate segments are 0.36, 0.44, 0.50, 0.46, and 0.40. Arranged in ascending order, these become 0.36, 0.40, 0.44, 0.46, and 0.50, with the median value being 0.44. Therefore, the basic segment score is 0.44. The probability intervals change in the following directions: increasing, increasing, decreasing, and decreasing, forming three adjacent direction pairs. Two of these are non-reversed adjacent direction pairs, with a non-reversed ratio of 0.67. According to the aforementioned unique formula, the segment validity score of the middle gesture candidate segment is 0.44, thus making it a gesture pairing segment.

[0189] In S3, the start time for the speech segment to be paired is 1240 ms, which is derived from the first probability vector of the intermediate candidate speech segment; the start time for the gesture segment to be paired is 1380 ms, which is derived from the first probability vector of the intermediate candidate gesture segment. The system uses the later 1380 ms as the overlap start time.

[0190] Furthermore, the end time of the speech segment to be paired is 1960 ms, which is derived from the last probability vector of the intermediate candidate speech segment; the end time of the gesture segment to be paired is 2100 ms, which is derived from the last probability vector of the intermediate candidate gesture segment. The system adopts the earlier 1960 ms as the overlap end time. The time length between the overlap end time and the overlap start time is 580 ms, which is derived from the time difference between 1960 ms and 1380 ms.

[0191] Specifically, the overlap duration of 580 ms is a positive value, therefore this segment combination becomes a candidate segment pair. The speech segment validity score is 0.42, the gesture segment validity score is 0.44, and the segment pair score is the smaller of the two, 0.42. Since this is the only candidate segment pair in the current application scenario, the system identifies it as the segment pairing result, and there is no need to perform parallel candidate segment pair sorting.

[0192] In S4, the system obtains display operation code 01 from the main voice category of the segment pairing result. Code 01 is derived from the operation code table used during deployment to complete interoperability verification. The system obtains the bedroom terminal identifier from the main gesture category, which is derived from the set of registered target terminals and the gesture classification output index table.

[0193] Furthermore, the system retrieves the current display content identifier of the family photo album page from the current display status record of the smart terminal with a screen in the living room. The resource list for this page contains 36 image resources and 1 page status record; the numbers 36 and 1 are derived from the item-by-item count of the current page's resource list, not from external preset parameters. The system writes the resource list, page status record, and content summary into the content provision record.

[0194] Specifically, the system combines the living room terminal identifier, display operation code 01, bedroom terminal identifier, and the current display content identifier of the family album page into a collaborative display instruction. The current logical version number of the version status item corresponding to the source terminal is 7, which comes from the most recent instruction record persistently saved on the living room terminal; the system obtains the next sequential current logical version number 8, and generates a versioned collaborative display instruction and collaborative display message.

[0195] Furthermore, after receiving the collaborative display message, the smart terminal with a screen in the bedroom verifies that the target terminal identifier matches the local terminal identifier and reads the locally executed logic version number 7. The local value 7 comes from the most recently successfully executed record saved by the bedroom terminal. Upon receiving the logic version number 8, which follows the executed logic version number 7, the smart terminal with a screen in the bedroom generates an execution permission determination result.

[0196] Specifically, the smart terminal with a screen in the bedroom requests 36 image resources and 1 page status record from the smart terminal with a screen in the living room based on the currently displayed content identifier. After completing the content summary verification, it displays the family album page and updates the executed logic version number to 8. The smart terminal with a screen in the bedroom generates a collaborative display execution result including a unique message identifier, processing completion time, execution success code, and the processed logic version number 8, and returns this result to the smart terminal with a screen in the living room.

[0197] Furthermore, if a message with the same logical version number 8 arrives again due to network retransmission, and the repeated value 8 originates from the original collaborative display message rather than a newly generated version, the smart terminal with screen in the bedroom determines that the received logical version number is not located after the executed logical version number, generates a duplicate version rejection code, and does not request or display the family album page again.

[0198] The preferred embodiments of this disclosure have been described in detail above with reference to the accompanying drawings. However, this disclosure is not limited to the specific details of the above embodiments. Within the scope of the technical concept of this disclosure, various simple modifications can be made to the technical solutions of this disclosure, and these simple modifications all fall within the protection scope of this disclosure.

[0199] It should also be noted that the various specific technical features described in the above embodiments can be combined in any suitable manner without contradiction. To avoid unnecessary repetition, this disclosure will not describe the various possible combinations separately.

[0200] Furthermore, various different embodiments of this disclosure can be combined in any way, as long as they do not violate the spirit of this disclosure, they should also be regarded as the content disclosed in this disclosure.

[0201] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some or all of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention, and they should all be covered within the scope of the claims and specification of the present invention.

Claims

1. A collaborative data processing method for a screen-equipped smart terminal based on multimodal interaction, characterized in that the method... include: Obtain a multimodal probability sequence set including time-stamped voice operation probability sequences and time-stamped gesture target probability sequences; The process involves processing various probability sequences, including generating a category sequence from the highest probability category, generating a probability interval sequence from the difference between the highest and second-highest probability values, generating an adaptive change boundary from the median of the absolute differences between adjacent probability intervals and the sum of the median absolute deviations, dividing candidate segments based on positions exceeding the adaptive change boundary and category switching positions, generating a segment support ratio from the category consistency ratio and the median of the probability intervals, generating a non-inversion ratio from the direction of probability interval change, generating a segment discrimination ratio from the segment support ratio and the non-inversion ratio, and selecting segments based on the segment discrimination ratio. The selected segments are then combined into multimodal segment data, which includes voice operation segments and gesture target segments. The overlap duration is generated based on the multimodal fragment data, and the fragment pairing data is generated based on the multimodal fragment data and the overlap duration. The operation target data is generated based on the fragment pairing data, and the collaborative display instructions are generated based on the operation target data and the currently displayed content identifier.

2. The collaborative data processing method for screen-equipped smart terminals based on multimodal interaction according to claim 1, characterized in that, The acquisition of time-stamped speech operation probability sequences includes: Acquire the speech sampling sequence and generate a pre-emphasized speech sequence based on the speech sampling sequence; Generate a framed speech sequence from the pre-emphasized speech sequence, and generate a speech time stamp sequence from the framed speech sequence; Generate a speech power spectrum sequence from the framed speech sequence, generate a frequency band energy sequence from the speech power spectrum sequence, and generate a speech feature sequence from the frequency band energy sequence. Obtain the voice operation classification parameters, and generate an unlabeled voice operation probability sequence based on the voice feature sequence and the voice operation classification parameters; Generate a time-stamped speech operation probability sequence based on the unstamped speech operation probability sequence and the speech time-stamped sequence.

3. The collaborative data processing method for screen-equipped smart terminals based on multimodal interaction according to claim 1, characterized in that, The acquisition of time-stamped gesture target probability sequences includes: Obtain the depth image sequence; obtain the color image sequence; generate the registration image sequence based on the depth image sequence and the color image sequence; Generate adjacent depth difference sequences based on the registered image sequences, generate depth change boundaries based on the adjacent depth difference sequences, and generate hand region sequences based on the adjacent depth difference sequences and depth change boundaries; A hand displacement vector sequence is generated from the hand region sequence, and a gesture feature sequence is generated from the hand displacement vector sequence. Obtain gesture target classification parameters, and generate unlabeled gesture target probability sequences based on gesture feature sequences and gesture target classification parameters; Generate a gesture time-stamped sequence based on the registered image sequence, and generate a time-stamped gesture target probability sequence based on the unstamped gesture target probability sequence and the gesture time-stamped sequence.

4. The collaborative data processing method for a screen-equipped smart terminal based on multimodal interaction according to claim 1, characterized in that, The division of candidate segments includes: Generate a sequence of absolute interval differences based on the probability interval sequence, and take the median as the median of the interval differences; An interval deviation sequence is generated based on the absolute difference between the interval absolute difference sequence and the median of the interval differences, and the median is taken as the absolute deviation of the interval median. An adaptive change boundary is generated based on the sum of the median of the interval difference and the absolute deviation of the median of the interval; Write the positions corresponding to the absolute difference of the interval greater than the adaptive change boundary into the probability change position sequence, and write the positions of adjacent maximum probability categories that are different into the category switching position sequence. The fragment boundary position sequence is generated by merging, deduplicating, and sorting the probability change position sequence and the category switching position sequence. The candidate fragment sequence is then generated based on the fragment boundary position sequence and the probability sequence that generates the probability interval sequence.

5. The collaborative data processing method for a screen-equipped smart terminal based on multimodal interaction according to claim 1, characterized in that, The fragment supports proportional generation, including: Generate category number data based on the category sequence within the candidate fragment, and generate category probability median data based on the probability values ​​of each category within the candidate fragment; Arrange each category in descending order of occurrence; if the occurrences are the same, arrange them in descending order of the median probability of each category; if they are still the same, arrange them in a fixed order of categories, and determine the first category as the main category. The number of main categories is generated based on the main category and the number of categories data, and the total number of probability vectors is generated based on the candidate fragments. The class consistency ratio is generated based on the number of main categories and the total number of probability vectors, and the median of the probability interval is generated based on the probability interval sequence within the candidate fragment. The smaller of the category consistency ratio and the median probability interval is taken as the segment support ratio.

6. The collaborative data processing method for a screen-equipped smart terminal based on multimodal interaction according to claim 1, characterized in that, The generation of the fragment discrimination ratio includes: Compare the adjacent probability intervals within the candidate segments. Write an ascending flag when the latter is larger, a descending flag when the latter is smaller, and an unchanged flag when they are equal, thus generating a sequence of interval change directions. Generate a sequence of adjacent direction pairs based on the sequence of alternating directions, and generate the total number of direction pairs based on the sequence of adjacent direction pairs. The number of non-reversed direction pairs is generated based on the adjacent direction pair sequence, wherein adjacent direction pairs formed by ascending and descending markers in any order are not included in the number of non-reversed direction pairs. When the adjacent direction pair sequence is not empty, the non-reversal ratio is generated based on the number of non-reversed direction pairs and the total number of direction pairs, and the smaller value between the segment support ratio and the non-reversal ratio is taken as the segment discrimination ratio; when the adjacent direction pair sequence is empty, no segment discrimination ratio is generated.

7. The collaborative data processing method for a screen-equipped smart terminal based on multimodal interaction according to claim 1, characterized in that, Select segments based on segment discrimination ratio, including: Candidate segments in the speech operation probability sequence that have a segment discrimination ratio and whose segment discrimination ratio is positive are identified as speech operation segments; Candidate segments in the gesture target probability sequence that have a segment discrimination ratio and whose segment discrimination ratio is positive are identified as gesture target segments; Stop generating segment pairing data when no voice operation segment or gesture target segment is generated.

8. The collaborative data processing method for a screen-equipped smart terminal based on multimodal interaction according to claim 1, characterized in that, The generation of fragment pairing data includes: Each voice operation segment is paired with each gesture target segment to form a segment pair to be judged. Take the later start time of each segment pair to be judged as the overlap start time, and take the earlier end time as the overlap end time; The overlap duration is generated based on the overlap start time and overlap end time, and the segments to be judged with positive overlap durations are identified as candidate segments. Candidate fragment pairs are sorted in descending order of overlap duration; when the overlap durations are the same, the smaller of the discrimination ratios of the two fragments in the candidate fragment pair is taken as the fragment pairing ratio, and they are sorted in descending order of the fragment pairing ratio; if they are still the same, they are sorted in descending order of the overlap end time. The first candidate fragment pair is determined as the fragment pairing data.

9. The collaborative data processing method for a screen-equipped smart terminal based on multimodal interaction according to claim 1, characterized in that, The generation of target data and collaborative display instructions includes: Operation field data is generated based on the voice operation segments in the segment pairing data. The operation field data includes the display operation code or the stop display operation code. Target field data is generated based on the gesture target fragment in the fragment pairing data. The target field data includes the terminal identifier of the target smart terminal with screen. Generate operation target data based on the operation field data and the target field data; Read the unique identifier of the content in the current display buffer and generate the identifier of the currently displayed content; A collaborative display instruction is generated based on the operation target data and the current display content identifier. The collaborative display instruction includes operation field data, target field data, and the current display content identifier.

10. A collaborative data processing system for a screen-equipped smart terminal based on multimodal interaction, characterized in that, The system is used to implement the collaborative data processing method for screen-equipped smart terminals based on multimodal interaction as described in any one of claims 1 to 9, and the system includes: A multimodal input module is used to generate a multimodal probability sequence set including a time-stamped speech operation probability sequence and a time-stamped gesture target probability sequence; The segment processing module is used to generate multimodal segment data, including voice operation segments and gesture target segments, based on multimodal probability sequence groups; The collaborative instruction module is used to generate fragment pairing data based on multimodal fragment data, generate operation target data based on fragment pairing data, and generate collaborative display instructions based on operation target data and the currently displayed content identifier. The display execution module is used to generate collaborative display messages based on collaborative display instructions, and to generate collaborative display result data based on the collaborative display messages.

Images