A multi-modal interactive control method and system for a smart blackboard

Abstract

The present invention discloses a multimodal interactive control method and system for a smart blackboard, including: obtaining the audio and video data stream of the user, generating audio and video sub-segments to match with the course data and the real-time course content slices in the smart blackboard; obtaining the gesture features, voice features and course content scene features of the user, performing adaptive multimodal fusion, and generating a multimodal data stream; constructing an interactive recognition classification model, obtaining the interactive control preference of the user for optimization, outputting the interactive intention of the user, and generating a control response using the interactive intention; and feeding back the user satisfaction to the interactive recognition classification model for optimizing the recognition and classification performance, and continuously updating the interactive control preference of the user. The present invention adopts a multimodal interaction method, improves the intention recognition rate and interactive robustness of the human-computer interaction between the user and the smart blackboard, and at the same time greatly reduces abnormal states such as low recognition efficiency caused by excessive misrecognition rate and complex interaction process in single-mode interaction.

Description

Technical Field

[0001] The present invention relates to the technical field of smart blackboards, and more specifically, to a multimodal interactive control method and system for a smart blackboard. Background Art

[0002] Traditional multimedia teaching equipment is mostly a projector combined with speakers to improve teaching efficiency. However, the improvement of teaching presentation and teaching interaction by traditional multimedia methods is still limited, and it cannot interact well with teaching calculations. With the change of educational methods and the development of information technology, smart blackboards with integrated functions have been widely used in teaching practice. Smart blackboards use stable and reliable infrared sensing positioning technology, LCD display technology and computer technology. Unlike electronic whiteboards, it integrates many office equipment functions such as projectors, electronic whiteboards, LCD TVs, computers, etc., plus special writing software, making information processing more convenient and demonstrations more vivid. It does not require complicated installation and debugging, reducing system costs.

[0003] Smart blackboards transform traditional teaching blackboards into perceptible interactive blackboards, achieving innovative breakthroughs in interactive teaching. The traditional contact interaction methods of existing smart blackboards are difficult to meet the convenience requirements, so non-contact control forms that are highly applicable, anti-interference, and easy to operate are more applicable, such as voice control, which converts input information from contact information to language information. However, contact or non-contact interactions are mostly based on single-mode device input. When there is an abnormality in any mode of interaction, the smart blackboard cannot respond correctly. In the interactive control of smart blackboards, how to accurately infer and predict user intentions based on user multimodal information, habit preference information, and interactive environment context information to achieve accurate interactive response is an urgent problem that needs to be solved. Summary of the invention

[0004] In order to solve the above technical problems, the present invention proposes a multimodal interactive control method and system for a smart blackboard.

[0005] A first aspect of the present invention provides a multi-modal interactive control method of a smart blackboard, comprising:

[0006] Using visual equipment to obtain the user's audio and video data stream, preprocessing the audio and video data stream to generate audio and video sub-segments, obtaining course data and real-time course content slices in the smart blackboard, and matching the audio and video sub-segments with the timing;

[0007] The user's gesture features, voice features and course content scene features are obtained according to the matched real-time course content slices and audio and video sub-segments, and the acquired gesture features, voice features and course content scene features are adaptively multimodally fused to generate a multimodal data stream;

[0008] Construct an interaction recognition classification model, obtain the user's interaction control preference to optimize the interaction recognition classification model, capture the relationship between the user's behavior sequence and the course content change, generate the user's interaction intention, and use the interaction intention to generate a control response;

[0009] The user's satisfaction with the control response is obtained, and the satisfaction is fed back to the interactive recognition and classification model to optimize the recognition and classification performance, and the user's interactive control preference is continuously updated.

[0010] In this solution, the user's audio and video data stream is obtained by using a visual device, and the audio and video data stream is preprocessed to generate audio and video sub-segments, specifically:

[0011] The audio and video data stream containing the user is obtained through a preset visual device, and each video image frame in the video sequence is reconstructed using a Laplacian pyramid structure, and feature extraction is performed through a dilated convolution at each level of the Laplacian pyramid;

[0012] The video image frame sequence is taken as input, and then pixel summation is performed with the residual image obtained by feature extraction after upsampling to generate a high-resolution image which is imported into the next level of the Laplacian pyramid. After multiple reconstructions of the Laplacian pyramid, a super-resolution video image is output;

[0013] De-noising and warping the output super-resolution video image to obtain the pre-processed video sequence, and construct a residual network to obtain the video features of the pre-processed video sequence;

[0014] Extracting audio sequences from audio and video data streams, filtering the audio sequences using wavelet transform, extracting audio features using residual networks, introducing an attention mechanism to align video features and audio features, and obtaining updated audio features;

[0015] The two branches of the residual network are used to obtain video features and updated audio features for matching, and the self-attention mechanism is used to obtain the weight information of different video features and audio features. The audio and video data stream is divided into audio and video sub-segments of different lengths according to the weight information.

[0016] Obtain the course data corresponding to different audio and video sub-segments, extract the real-time course content slices in the smart blackboard, and match them with the audio and video sub-segments according to the real-time timestamp.

[0017] In this solution, the user's gesture features, voice features and course content scene features are obtained based on the matched real-time course content slices and audio and video sub-segments, specifically:

[0018] Extract key video image frames based on video features in different audio and video sub-segments, segment the user's hand area in the key video image frames, and use the OpenPose algorithm to perform key point detection in the user's hand image area;

[0019] Connect the key points to obtain the skeleton mask image of the user's hand, use the convolutional neural network to extract local features in the skeleton mask image of the user's hand, and introduce spatial attention and channel attention to fuse the local features to obtain static features;

[0020] Obtain a static feature sequence of each key video image frame in the audio and video sub-segment, use a Bi-LSTM network to model the static feature sequence in time, capture the bidirectional temporal dynamic information in different static features according to the bidirectional LSTM structure, and output the hidden state containing the dynamic features as the gesture feature;

[0021] Extracting corpus information from audio sequences in different audio and video sub-segments, aligning the corpus information through a connection temporal classification algorithm, generating search constraints in a preset corpus of the smart blackboard, and generating multiple candidate paths based on the search constraints;

[0022] Calling the speech model to obtain the semantics contained in the audio and video sub-segments as speech features based on the candidate paths;

[0023] In addition, the context features of real-time course content slices are obtained based on the course data, the course data is located and intercepted according to the context features, the course content scene features are obtained, and the intercepted course data is used to expand the preset corpus of the smart blackboard.

[0024] In this solution, the acquired gesture features, voice features and course content scene features are adaptively fused to generate a multimodal data stream, specifically:

[0025] According to the historical interaction data of the smart blackboard, the intention data corresponding to each gesture feature, voice feature and course content scene feature in a single mode are extracted, and the single-modal features and the corresponding intention data are mapped to a low-dimensional space for graph representation;

[0026] Construct an undirected graph of unimodal features and intention data, use the acquired gesture features, voice features and course content scene features to locate in the undirected graph, obtain the corresponding unimodal feature nodes for marking, and count the number of nodes directly connected to the marked unimodal feature nodes;

[0027] The importance of gesture features, voice features and course content scene features is evaluated according to the ratio of the number of directly connected nodes to the maximum number of connections of the nodes.

[0028] Construct a simple intention prediction model, use gesture features, voice features and course content scene features to construct a training set, subtract the set elements of the preset single-modal category from the training set, analyze the change in model prediction accuracy and assign "positive action labels" or "negative action labels" to the set elements;

[0029] Genetic algorithms are used to establish optimization tasks for adaptive weights according to importance and feature labels, obtain the fitness of fusion of different modal features, perform cross-recombination according to the fitness, obtain the best adaptive weights for multimodal fusion and construct multimodal data streams.

[0030] In this solution, an interaction recognition classification model is constructed, and the user's interaction control preference is obtained to optimize the interaction recognition classification model, specifically:

[0031] Acquire the user's identity information according to the audio and video data stream, use the identity information to read the historical interaction data, analyze the response relationship between the multimodal features and the interaction instructions in the historical interaction data, and acquire the user's interaction control preference based on the response relationship;

[0032] An interactive recognition classification network is constructed based on the encoder network combined with the Bi-GRU network, an attention mechanism is introduced into the interactive recognition classification network, and the attention layer is trained using the user's interactive control preference;

[0033] The multimodal data stream is used as a model input, and features of the input data are reconstructed by stacking autoencoders, and the reconstructed multimodal features are sent to a decoder to obtain a decoding output, and a Bi-GRU network is introduced to capture the temporal dependency of the multimodal features;

[0034] The multimodal features containing time series features are imported into the attention layer, and the attention weights of the multimodal features are set according to the user's interactive control preferences and normalized. The hidden state of the GRU unit output is passed through the fully connected layer to obtain the probability distribution of the interactive intention. The smart blackboard generates a control response based on the interactive intention with the highest probability.

[0035] In this solution, the user's satisfaction with the control response is obtained, and the satisfaction is fed back to the interactive recognition and classification model to optimize the recognition and classification performance, specifically:

[0036] The number of user interaction instructions and the number of instructions that control responses are successfully recognized in one time are obtained within a preset time, and the recognition speed is used to evaluate the user's satisfaction with the control response. When the satisfaction is less than a preset satisfaction threshold, it is fed back to the interaction recognition classification model;

[0037] Extract the interactive instructions with recognition and classification errors and suspend them, obtain the user's feedback data on the control response after the recognition error, determine the modal category of the feedback data, generate a supplementary data set of the modal category, and optimize other modal feature data of the control response;

[0038] The user's interaction preference database is updated by supplementing the optimized multimodal feature data, and the interaction recognition classification model is regularly updated and trained according to the interaction preference database to achieve accuracy compensation, so that the interaction instruction can achieve the user's expected control at one time;

[0039] A cloud storage time threshold is preset. When the user's interaction preference database is not called for a time period exceeding the preset storage time threshold, it is deleted from the cloud.

[0040] The second aspect of the present invention further provides a multimodal interactive control system of a smart blackboard, the system comprising: a memory, a processor, the memory comprising a multimodal interactive control method program of the smart blackboard, and the multimodal interactive control method program of the smart blackboard is executed by the processor to implement the following steps:

[0041] Using visual equipment to obtain the user's audio and video data stream, preprocessing the audio and video data stream to generate audio and video sub-segments, obtaining course data and real-time course content slices in the smart blackboard, and matching the audio and video sub-segments with the timing;

[0042] The user's gesture features, voice features and course content scene features are obtained according to the matched real-time course content slices and audio and video sub-segments, and the acquired gesture features, voice features and course content scene features are adaptively multimodally fused to generate a multimodal data stream;

[0043] Construct an interaction recognition classification model, obtain the user's interaction control preference to optimize the interaction recognition classification model, capture the relationship between the user's behavior sequence and the course content change, generate the user's interaction intention, and use the interaction intention to generate a control response;

[0044] The user's satisfaction with the control response is obtained, and the satisfaction is fed back to the interactive recognition and classification model to optimize the recognition and classification performance, and the user's interactive control preference is continuously updated.

[0045] The present invention discloses a multimodal interactive control method and system for a smart blackboard, including: obtaining the audio and video data stream of the user, generating audio and video sub-segments to match with the course data and the real-time course content slices in the smart blackboard; obtaining the gesture features, voice features and course content scene features of the user, performing adaptive multimodal fusion, and generating a multimodal data stream; constructing an interactive recognition classification model, obtaining the interactive control preference of the user for optimization, outputting the interactive intention of the user, and generating a control response using the interactive intention; and feeding back the user satisfaction to the interactive recognition classification model for optimizing the recognition and classification performance, and continuously updating the interactive control preference of the user. The present invention adopts a multimodal interaction method, improves the intention recognition rate and interactive robustness of the human-computer interaction between the user and the smart blackboard, and at the same time greatly reduces abnormal states such as low recognition efficiency caused by excessive misrecognition rate and complex interaction process in single-mode interaction. BRIEF DESCRIPTION OF THE DRAWINGS

[0046] Figure 1 A flowchart of a multi-modal interactive control method of a smart blackboard according to the present invention is shown;

[0047] Figure 2 A flow chart for acquiring gesture features, voice features, and course content scene features is shown;

[0048] Figure 3 A flow chart for generating a multimodal data stream by adaptive multimodal fusion is shown;

[0049] Figure 4 A block diagram of a multi-modal interactive control system of a smart blackboard of the present invention is shown. DETAILED DESCRIPTION

[0050] In order to more clearly understand the above-mentioned purpose, features and advantages of the present invention, the present invention is further described in detail below in conjunction with the accompanying drawings and specific embodiments. It should be noted that the embodiments of the present application and the features in the embodiments can be combined with each other without conflict.

[0051] In the following description, many specific details are set forth to facilitate a full understanding of the present invention. However, the present invention may also be implemented in other ways different from those described herein. Therefore, the protection scope of the present invention is not limited to the specific embodiments disclosed below.

[0052] Figure 1 A flow chart of a multi-modal interactive control method of a smart blackboard according to the present invention is shown.

[0053] like Figure 1 As shown, the first aspect of the present invention provides a multi-modal interactive control method of a smart blackboard, comprising:

[0054] S102, using a visual device to obtain the user's audio and video data stream, preprocessing the audio and video data stream to generate audio and video sub-segments, obtaining course data and real-time course content slices in the smart blackboard, and matching the audio and video sub-segments with the timing;

[0055] S104, obtaining the user's gesture features, voice features, and course content scene features according to the matched real-time course content slices and audio and video sub-segments, and performing adaptive multimodal fusion on the obtained gesture features, voice features, and course content scene features to generate a multimodal data stream;

[0056] S106, constructing an interaction recognition classification model, obtaining the user's interaction control preference to optimize the interaction recognition classification model, capturing the relationship between the user's behavior sequence and the course content change, generating the user's interaction intention, and generating a control response using the interaction intention;

[0057] S108, obtaining the user's satisfaction with the control response, feeding back the satisfaction to the interactive recognition and classification model to optimize the recognition and classification performance, and continuously updating the user's interactive control preference.

[0058] It should be noted that the audio and video data stream containing the user is obtained through a preset visual device, and each video image frame in the video sequence is reconstructed using a Laplace pyramid structure. Feature extraction is performed through a dilated convolution at each level of the Laplace pyramid. A residual image is obtained during the feature extraction process. The residual image is added to the low-resolution image matrix during the image reconstruction process to obtain a high-resolution image. The network parameters are reduced by learning through the residual to avoid the problem of gradient disappearance or gradient explosion. The video image frame sequence is taken as input and then pixel summed with the residual image obtained by feature extraction after upsampling to generate a high-resolution image that is imported into the next level of the Laplace pyramid. After multiple reconstructions of the Laplace pyramid, a super-resolution video image is output; the output super-resolution video image is denoised and warped, and the image quality of the super-resolution video image is improved by image denoising, and the denoised image is corrected by image warping correction to facilitate the extraction of video features. The preprocessed video sequence is obtained, and a residual network is constructed to obtain the video features of the preprocessed video sequence. The residual network architecture can extract higher-level features; the audio sequence in the audio and video data stream is extracted, the audio sequence is filtered using wavelet transform, and the audio features are extracted using the residual network. The attention mechanism is introduced to align the video features and audio features, and the updated audio features are obtained. The attention alignment solves the high fusion difficulty caused by the mismatch of audio and video rates and the different information lengths. The two branches of the residual network obtain the video features and the updated audio features for matching, and the self-attention mechanism is used to obtain the weight information of different video features and audio features. The audio and video data stream is divided into audio and video sub-segments of different lengths according to the weight information; the course data corresponding to different audio and video sub-segments are obtained, and the real-time course content slices in the smart blackboard are extracted, and the matching with the audio and video sub-segments is achieved according to the real-time timestamp.

[0059] Figure 2 A flow chart for acquiring gesture features, voice features, and course content scene features is shown.

[0060] According to an embodiment of the present invention, the gesture features, voice features and course content scene features of the user are obtained based on the matched real-time course content slices and audio and video sub-segments, specifically:

[0061] S202, extracting key video image frames according to video features in different audio and video sub-segments, segmenting the user's hand region in the key video image frames, and performing key point detection in the user's hand image region using an OpenPose algorithm;

[0062] S204, connecting the key points to obtain a skeleton mask image of the user's hand, extracting local features from the skeleton mask image of the user's hand using a convolutional neural network, and introducing spatial attention and channel attention to fuse the local features to obtain static features;

[0063] S206, obtaining a static feature sequence of each key video image frame in the audio and video sub-segment, modeling the static feature sequence in time using a Bi-LSTM network, capturing bidirectional temporal dynamic information in different static features according to a bidirectional LSTM structure, and outputting a hidden state containing dynamic features as a gesture feature;

[0064] S208, extracting corpus information from audio sequences in different audio and video sub-segments, aligning the corpus information through a connection time classification algorithm, generating search constraints in a preset corpus of the smart blackboard, generating multiple candidate paths based on the search constraints, and calling a speech model to obtain the semantics contained in the audio and video sub-segments as speech features based on the candidate paths;

[0065] S210, in addition, context features of real-time course content slices are obtained based on course data, and course data is located and intercepted according to the context features to obtain course content scene features, and the intercepted course data is used to expand the preset corpus of the smart blackboard.

[0066] It should be noted that the user's hand target is recognized and the key points of the user's hand skeleton are detected according to the OpenPose algorithm. The hand skeleton is extracted based on the key points and the foreground and background are segmented using the global threshold method. The gesture image is masked to generate a corresponding mask image, which reduces redundant information unrelated to the gesture and eliminates the problem of different hand shapes of different people. Static features are obtained from the hand skeleton mask image, which reflects the shape, posture, position and other information of the gesture; and dynamic features are obtained based on the static feature sequence, which reflects the movement pattern of the gesture and its dependence on time.

[0067] Because it is difficult to align audio data with labeled corpus text, if the model is trained using unaligned input, the user's speaking speed and the distance between characters are different, which makes it difficult for the model to converge. Connected temporal classification is used to solve the problem of inconsistent input and output sequence lengths and inability to align. Search constraints are set in the preset corpus of the smart blackboard. The search constraints include boundary thresholds. When the corpus is a negative sample, there is no need to perform template matching in the preset corpus to reduce the workload. When the matching distance between corpora exceeds the boundary threshold, it is directly skipped and no further calculation is required. Find characters with similar probabilities within a reasonable time, obtain multiple candidate paths, realize speech recognition through language models such as DeepSpeech2, and obtain the semantics contained in the audio and video sub-segments.

[0068] Figure 3 A flow chart of generating multimodal data streams through adaptive multimodal fusion is shown.

[0069] According to an embodiment of the present invention, the acquired gesture features, voice features and course content scene features are adaptively multimodally fused to generate a multimodal data stream, specifically:

[0070] S302, extracting the intention data corresponding to each gesture feature, voice feature and course content scene feature in a single mode according to the historical interaction data of the smart blackboard, and mapping the single mode features and the corresponding intention data to a low-dimensional space for graph representation;

[0071] S304, constructing an undirected graph of unimodal features and intention data, using the acquired gesture features, voice features, and course content scene features to locate in the undirected graph, acquiring corresponding unimodal feature nodes for marking, and counting the number of nodes directly connected to the marked unimodal feature nodes;

[0072] S306, evaluating the importance of gesture features, voice features, and course content scene features according to the ratio of the number of directly connected nodes to the maximum number of node connections,

[0073] S308, constructing a simple intention prediction model, using gesture features, voice features and course content scene features to construct a training set, subtracting set elements of a preset single-modal category from the training set, analyzing the change in model prediction accuracy, and assigning a "positive action label" or a "negative action label" to the set elements;

[0074] S310, using a genetic algorithm to establish an optimization task of adaptive weights according to importance and feature effect labels, obtain the fitness of fusion of different modal features, perform cross-recombination according to the fitness, obtain the best adaptive weights for multi-modal fusion and construct a multi-modal data stream.

[0075] It should be noted that in network analysis, centrality is an indicator of the importance or influence of nodes in the network. According to the characteristics of graph representation, the more nodes a node is connected to, the richer the information it contains. The importance of a node in an undirected graph is positively correlated with its centrality, that is, a more central node will be more important than other nodes, and importance is proportional to the number of nodes connected to it. The calculation formula is: Representation and Node Directly connected nodes The number of Indicates the maximum number of connections. A simple intention prediction model is constructed based on deep learning methods such as neural networks, and its model prediction accuracy is represents the intention prediction model, represents the training set, Represents unimodal data and corresponding interaction intentions, represents the number of training samples in the training set, represents the eigenvector after subtracting the elements of the set, The indicator function of the intention prediction model. The objective function is constructed by taking high importance and high positive effect label data as the optimization target. The larger the objective function value corresponding to the feature combination, the larger the fitness value. The optimal adaptive weight is obtained through operations such as crossover recombination and mutation in the genetic algorithm, which lays an important foundation for the recognition and classification of user interaction intentions.

[0076] It should be noted that the user's identity information is obtained based on the audio and video data stream, the historical interaction data is read using the identity information, the response relationship between multimodal features and interaction instructions is analyzed in the historical interaction data, and the user's interaction control preference is obtained based on the response relationship; an interaction recognition and classification network is constructed based on the encoder network combined with the Bi-GRU network. The Bi-GRU network is composed of unidirectional GRUs in opposite directions, which can further improve the information memory ability and the accuracy of long time series prediction. An attention mechanism is introduced into the interaction recognition classification network, and the attention layer is trained using the user's interaction control preference; the multimodal data stream is used as the model input, and a multi-layer denoising autoencoder network is stacked on the basis of the autoencoder network to generate a deep architecture with deep nonlinear mapping capabilities; the input data is feature reconstructed by stacking autoencoders, the reconstructed multimodal features are sent to the decoder to obtain the decoding output, and the Bi-GRU network is introduced to capture the time dependency of the multimodal features; the multimodal features containing time series features are imported into the attention layer, the attention weights of the multimodal features are set using the user's interaction control preferences, and are set and normalized, and the hidden state of the GRU unit output is passed through the fully connected layer to obtain the probability distribution of the interaction intention, and the smart blackboard generates a control response according to the interaction intention with the highest probability.

[0077] It should be noted that the number of user interaction instructions and the number of instructions for which control responses are successfully recognized at one time, as well as the recognition speed within a preset time are obtained to evaluate the user's satisfaction with the control response. When the satisfaction is less than a preset satisfaction threshold, it is fed back to the interaction recognition classification model; the interaction instructions with recognition and classification errors are extracted and suspended, the user's feedback data on the control response after the recognition error is obtained, the modal category of the feedback data is determined, a supplementary data set of the modal category is generated, and other modal feature data of the control response are optimized at the same time; the user's interaction preference database is updated by supplementing and optimizing the multimodal feature data, and the interaction recognition classification model is regularly updated and trained according to the interaction preference database to achieve accuracy compensation, so that the interaction instructions can achieve the user's expected control at one time; a cloud storage time threshold is preset, and when the user's interaction preference database is not called for more than the preset storage time threshold, it is deleted in the cloud.

[0078] It should be noted that after obtaining the user's interaction intention in the current audio and video sub-segment based on the interaction recognition classification model, the interaction intention is screened according to the user interaction preference database to see if it is an easily associated interaction intention. The easily associated interaction intention is obtained based on historical interaction data. When an interaction intention is associated with an adjacent interaction intention and the frequency of occurrence is greater than a preset threshold, it is regarded as an easily associated interaction intention; if so, a monitoring task is established for subsequent unimodal data streams of different categories to determine the correlation between different unimodal data streams and the control responses corresponding to the current interaction intention. When the correlation is greater than the preset threshold, a supplementary control response is generated based on the current control response. For example, in geometry teaching, a user draws a cube image through gestures, and then uses voice to annotate parameters in the drawn cube image, and regards the instructions as a hierarchical structure, which greatly improves the user's usage efficiency; in addition, control responses of different colors are generated based on the current control response according to the voiceprint information of different users, so as to distinguish the control responses of teachers and students, and improve the course interactivity in the student group.

[0079] Figure 4 A block diagram of a multi-modal interactive control system of a smart blackboard of the present invention is shown.

[0080] The second aspect of the present invention further provides a multimodal interactive control system 4 of a smart blackboard, the system comprising: a memory 41, a processor 42, the memory comprising a multimodal interactive control method program of the smart blackboard, the multimodal interactive control method program of the smart blackboard being executed by the processor to implement the following steps:

[0081] Using visual equipment to obtain the user's audio and video data stream, preprocessing the audio and video data stream to generate audio and video sub-segments, obtaining course data and real-time course content slices in the smart blackboard, and matching the audio and video sub-segments with the timing;

[0082] The user's gesture features, voice features and course content scene features are obtained according to the matched real-time course content slices and audio and video sub-segments, and the acquired gesture features, voice features and course content scene features are adaptively multimodally fused to generate a multimodal data stream;

[0083] Construct an interaction recognition classification model, obtain the user's interaction control preference to optimize the interaction recognition classification model, capture the relationship between the user's behavior sequence and the course content change, generate the user's interaction intention, and use the interaction intention to generate a control response;

[0084] The user's satisfaction with the control response is obtained, and the satisfaction is fed back to the interactive recognition and classification model to optimize the recognition and classification performance, and the user's interactive control preference is continuously updated.

[0085] The third aspect of the present invention also provides a computer-readable storage medium, which includes a multimodal interactive control method program for a smart blackboard. When the multimodal interactive control method program for a smart blackboard is executed by a processor, the steps of the multimodal interactive control method for a smart blackboard as described in any one of the above items are implemented.

[0086] In the several embodiments provided in the present application, it should be understood that the disclosed devices and methods can be implemented in other ways. The device embodiments described above are only schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation, such as: multiple units or components can be combined, or can be integrated into another system, or some features can be ignored or not executed. In addition, the coupling, direct coupling, or communication connection between the components shown or discussed can be through some interfaces, and the indirect coupling or communication connection of the devices or units can be electrical, mechanical or other forms.

[0087] The units described above as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units; they may be located in one place or distributed on multiple network units; some or all of the units may be selected according to actual needs to achieve the purpose of the present embodiment.

[0088] In addition, all functional units in the embodiments of the present invention may be integrated into one processing unit, or each unit may be separately used as a unit, or two or more units may be integrated into one unit; the above-mentioned integrated units may be implemented in the form of hardware or in the form of hardware plus software functional units.

[0089] A person of ordinary skill in the art can understand that: all or part of the steps of implementing the above method embodiment can be completed by hardware related to program instructions, and the aforementioned program can be stored in a computer-readable storage medium. When the program is executed, it executes the steps of the above method embodiment; and the aforementioned storage medium includes: a mobile storage device, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, and other media that can store program codes.

[0090] Alternatively, if the above-mentioned integrated unit of the present invention is implemented in the form of a software function module and sold or used as an independent product, it can also be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the embodiment of the present invention can be essentially or partly reflected in the form of a software product that contributes to the prior art. The computer software product is stored in a storage medium and includes several instructions for a computer device (which can be a personal computer, server, or network device, etc.) to execute all or part of the methods described in each embodiment of the present invention. The aforementioned storage medium includes: various media that can store program codes, such as mobile storage devices, ROM, RAM, magnetic disks or optical disks.

[0091] The above is only a specific embodiment of the present invention, but the protection scope of the present invention is not limited thereto. Any person skilled in the art can easily think of changes or substitutions within the technical scope disclosed by the present invention, which should be included in the protection scope of the present invention. Therefore, the protection scope of the present invention should be based on the protection scope of the claims.

Claims

1. A multi-modal interactive control method for a smart blackboard, characterized in that: The following steps are involved: Using visual equipment to obtain the user's audio and video data stream, preprocessing the audio and video data stream to generate audio and video sub-segments, obtaining course data and real-time course content slices in the smart blackboard, and matching the audio and video sub-segments with the timing; The user's gesture features, voice features and course content scene features are obtained according to the matched real-time course content slices and audio and video sub-segments, and the acquired gesture features, voice features and course content scene features are adaptively multimodally fused to generate a multimodal data stream; Construct an interaction recognition classification model, obtain the user's interaction control preference to optimize the interaction recognition classification model, capture the relationship between the user's behavior sequence and the course content change, generate the user's interaction intention, and use the interaction intention to generate a control response; Obtaining user satisfaction with the control response, feeding back the satisfaction to the interactive recognition and classification model to optimize recognition and classification performance, and continuously updating the user's interactive control preferences; The acquired gesture features, voice features and course content scene features are adaptively fused in multimodal manner to generate a multimodal data stream, specifically: According to the historical interaction data of the smart blackboard, the intention data corresponding to each gesture feature, voice feature and course content scene feature in a single mode are extracted, and the single-modal features and the corresponding intention data are mapped to a low-dimensional space for graph representation; Construct an undirected graph of unimodal features and intention data, use the acquired gesture features, voice features and course content scene features to locate in the undirected graph, obtain the corresponding unimodal feature nodes for marking, and count the number of nodes directly connected to the marked unimodal feature nodes; The importance of gesture features, voice features and course content scene features is evaluated according to the ratio of the number of directly connected nodes to the maximum number of connections of the nodes. Construct a simple intention prediction model, use gesture features, voice features and course content scene features to construct a training set, subtract the set elements of the preset single-modal category from the training set, analyze the change in model prediction accuracy, and assign "positive action labels" or "negative action labels" to the set elements; Using genetic algorithms to establish optimization tasks for adaptive weights based on importance and feature labels, obtain the fitness of fusion of different modal features, perform cross-recombination based on the fitness, obtain the best adaptive weights for multi-modal fusion and construct multi-modal data streams; Construct an interaction recognition classification model, obtain the user's interaction control preference and optimize the interaction recognition classification model, specifically: Acquire the user's identity information according to the audio and video data stream, use the identity information to read the historical interaction data, analyze the response relationship between the multimodal features and the interaction instructions in the historical interaction data, and acquire the user's interaction control preference based on the response relationship; An interactive recognition classification network is constructed based on the encoder network combined with the Bi-GRU network, an attention mechanism is introduced into the interactive recognition classification network, and the attention layer is trained using the user's interactive control preference; The multimodal data stream is used as a model input, and features of the input data are reconstructed by stacking autoencoders, and the reconstructed multimodal features are sent to a decoder to obtain a decoding output, and a Bi-GRU network is introduced to capture the temporal dependency of the multimodal features; The multimodal features containing time series features are imported into the attention layer, and the attention weights of the multimodal features are set according to the user's interactive control preferences and normalized. The hidden state of the GRU unit output is passed through the fully connected layer to obtain the probability distribution of the interactive intention. The smart blackboard generates a control response based on the interactive intention with the highest probability.

2. According to claim 1, a multi-modal interactive control method of a smart blackboard is characterized in that: The user's audio and video data stream is obtained by using a visual device, and the audio and video data stream is preprocessed to generate audio and video sub-segments, specifically: The audio and video data stream containing the user is obtained through a preset visual device, and each video image frame in the video sequence is reconstructed using a Laplacian pyramid structure, and feature extraction is performed through a dilated convolution at each level of the Laplacian pyramid; The video image frame sequence is taken as input, and then pixel summation is performed with the residual image obtained by feature extraction after upsampling to generate a high-resolution image which is imported into the next level of the Laplacian pyramid. After multiple reconstructions of the Laplacian pyramid, a super-resolution video image is output; De-noising and warping the output super-resolution video image to obtain the pre-processed video sequence, and construct a residual network to obtain the video features of the pre-processed video sequence; Extracting audio sequences from audio and video data streams, filtering the audio sequences using wavelet transform, extracting audio features using residual networks, introducing an attention mechanism to align video features and audio features, and obtaining updated audio features; The two branches of the residual network obtain video features and updated audio features for matching, and use the self-attention mechanism to obtain the weight information of different video features and audio features. According to the weight information, the audio and video data stream is divided into audio and video sub-segments of different lengths; Obtain the course data corresponding to different audio and video sub-segments, extract the real-time course content slices in the smart blackboard, and match them with the audio and video sub-segments according to the real-time timestamp.

3. According to claim 1, the multi-modal interactive control method of a smart blackboard is characterized in that: According to the matched real-time course content slices and audio and video sub-segments, the user's gesture features, voice features and course content scene features are obtained, specifically: Extract key video image frames based on video features in different audio and video sub-segments, segment the user's hand area in the key video image frames, and use the OpenPose algorithm to perform key point detection in the user's hand image area; Connect the key points to obtain the skeleton mask image of the user's hand, use the convolutional neural network to extract local features from the skeleton mask image of the user's hand, and introduce spatial attention and channel attention to fuse the local features to obtain static features; Obtain a static feature sequence of each key video image frame in the audio and video sub-segment, model the static feature sequence in time using a Bi-LSTM network, capture bidirectional temporal dynamic information in different static features according to a bidirectional LSTM structure, and output a hidden state containing dynamic features as a gesture feature; Extracting corpus information from audio sequences in different audio and video sub-segments, aligning the corpus information through a connection temporal classification algorithm, generating search constraints in a preset corpus of the smart blackboard, and generating multiple candidate paths based on the search constraints; Calling the speech model to obtain the semantics contained in the audio and video sub-segments as speech features based on the candidate paths; In addition, the context features of real-time course content slices are obtained based on the course data, the course data is located and intercepted according to the context features, the course content scene features are obtained, and the intercepted course data is used to expand the preset corpus of the smart blackboard.

4. According to claim 1, the multi-modal interactive control method of a smart blackboard is characterized in that: The user's satisfaction with the control response is obtained, and the satisfaction is fed back to the interactive recognition and classification model to optimize the recognition and classification performance, specifically: The number of user interaction instructions and the number of instructions that control responses are successfully recognized in one time are obtained within a preset time, and the recognition speed is used to evaluate the user's satisfaction with the control response. When the satisfaction is less than a preset satisfaction threshold, it is fed back to the interaction recognition classification model; Extract the interactive instructions with recognition and classification errors and suspend them, obtain the user's feedback data on the control response after the recognition error, determine the modal category of the feedback data, generate a supplementary data set of the modal category, and optimize other modal feature data of the control response; The user's interaction preference database is updated by supplementing the optimized multimodal feature data, and the interaction recognition classification model is regularly updated and trained according to the interaction preference database to achieve accuracy compensation, so that the interaction instruction can achieve the user's expected control at one time; A cloud storage time threshold is preset. When the user's interaction preference database is not called for a time period exceeding the preset storage time threshold, it is deleted from the cloud.

5. A multi-modal interactive control system of a smart blackboard, characterized in that: The system includes: a memory and a processor, wherein the memory includes a multi-modal interactive control method program of a smart blackboard, and when the multi-modal interactive control method program of the smart blackboard is executed by the processor, the following steps are implemented: Using visual equipment to obtain the user's audio and video data stream, preprocessing the audio and video data stream to generate audio and video sub-segments, obtaining course data and real-time course content slices in the smart blackboard, and matching the audio and video sub-segments with the timing; The user's gesture features, voice features and course content scene features are obtained according to the matched real-time course content slices and audio and video sub-segments, and the acquired gesture features, voice features and course content scene features are adaptively multimodally fused to generate a multimodal data stream; Construct an interaction recognition classification model, obtain the user's interaction control preference to optimize the interaction recognition classification model, capture the relationship between the user's behavior sequence and the course content change, generate the user's interaction intention, and use the interaction intention to generate a control response; Obtaining user satisfaction with the control response, feeding back the satisfaction to the interactive recognition and classification model to optimize recognition and classification performance, and continuously updating the user's interactive control preferences; The acquired gesture features, voice features and course content scene features are adaptively fused in multimodal manner to generate a multimodal data stream, specifically: According to the historical interaction data of the smart blackboard, the intention data corresponding to each gesture feature, voice feature and course content scene feature in a single mode are extracted, and the single-modal features and the corresponding intention data are mapped to a low-dimensional space for graph representation; Construct an undirected graph of unimodal features and intention data, use the acquired gesture features, voice features and course content scene features to locate in the undirected graph, obtain the corresponding unimodal feature nodes for marking, and count the number of nodes directly connected to the marked unimodal feature nodes; The importance of gesture features, voice features and course content scene features is evaluated according to the ratio of the number of directly connected nodes to the maximum number of connections of the nodes. Construct a simple intention prediction model, use gesture features, voice features and course content scene features to construct a training set, subtract the set elements of the preset single-modal category from the training set, analyze the change in model prediction accuracy, and assign "positive action labels" or "negative action labels" to the set elements; Using genetic algorithms to establish optimization tasks for adaptive weights based on importance and feature labels, obtain the fitness of fusion of different modal features, perform cross-recombination based on the fitness, obtain the best adaptive weights for multi-modal fusion and construct multi-modal data streams; Construct an interaction recognition classification model, obtain the user's interaction control preference and optimize the interaction recognition classification model, specifically: Acquire the user's identity information according to the audio and video data stream, use the identity information to read the historical interaction data, analyze the response relationship between the multimodal features and the interaction instructions in the historical interaction data, and acquire the user's interaction control preference based on the response relationship; An interactive recognition classification network is constructed based on the encoder network combined with the Bi-GRU network, an attention mechanism is introduced into the interactive recognition classification network, and the attention layer is trained using the user's interactive control preference; The multimodal data stream is used as a model input, and features of the input data are reconstructed by stacking autoencoders, and the reconstructed multimodal features are sent to a decoder to obtain a decoding output, and a Bi-GRU network is introduced to capture the temporal dependency of the multimodal features; The multimodal features containing time series features are imported into the attention layer, and the attention weights of the multimodal features are set according to the user's interactive control preferences and normalized. The hidden state of the GRU unit output is passed through the fully connected layer to obtain the probability distribution of the interactive intention. The smart blackboard generates a control response based on the interactive intention with the highest probability.

6. The multi-modal interactive control system of a smart blackboard according to claim 5 is characterized in that: According to the matched real-time course content slices and audio and video sub-segments, the user's gesture features, voice features and course content scene features are obtained, specifically: Extract key video image frames based on video features in different audio and video sub-segments, segment the user's hand area in the key video image frames, and use the OpenPose algorithm to perform key point detection in the user's hand image area; Connect the key points to obtain the skeleton mask image of the user's hand, use the convolutional neural network to extract local features from the skeleton mask image of the user's hand, and introduce spatial attention and channel attention to fuse the local features to obtain static features; Obtain a static feature sequence of each key video image frame in the audio and video sub-segment, model the static feature sequence in time using a Bi-LSTM network, capture bidirectional temporal dynamic information in different static features according to a bidirectional LSTM structure, and output a hidden state containing dynamic features as a gesture feature; Extracting corpus information from audio sequences in different audio and video sub-segments, aligning the corpus information through a connection temporal classification algorithm, generating search constraints in a preset corpus of the smart blackboard, and generating multiple candidate paths based on the search constraints; Calling the speech model to obtain the semantics contained in the audio and video sub-segments as speech features based on the candidate paths; In addition, the context features of real-time course content slices are obtained based on the course data, the course data is located and intercepted according to the context features, the course content scene features are obtained, and the intercepted course data is used to expand the preset corpus of the smart blackboard.

Images