Wearable skill teaching system

Abstract

The invention relates to a wearable skill teaching system, which comprises a wearable camera device, a voice processing module, a front-end service module, a rear-end service module and a multi-mode interaction module based on an EgoVideo model, and is characterized in that the wearable camera device acquires a video stream for a first visual angle in real time and transmits the video stream to the rear-end service module; and monitoring a user instruction after monitoring the wake-up keyword, converting a voice instruction into text instruction information after monitoring a user voice instruction, inputting the text instruction information and a video stream into the multi-mode interaction module based on the EgoVideo model, returning text reply information and a generated prediction video, and displaying the text reply information and the generated prediction video through the front-end service module. Compared with the prior art, the method can process understanding and generating tasks at the same time, supports first-view conversation and video generation of the user, and has good user skill teaching experience.

Description

Technical Field

[0001] The present invention relates to the field of multimodal data processing, and more particularly to a wearable skill teaching system. Background Art

[0002] In daily life, especially in scenarios such as delicate operations (such as in the kitchen and assembly), users often need to obtain guidance and feedback through visual and auditory information.

[0003] However, existing teaching assistance systems often have the following problems:

[0004] 1) Perspective limitation: Most teaching systems use a third-person perspective, which is different from the first-person perspective of the user during actual operation, making it difficult for users to directly apply the teaching content to actual operations.

[0005] 2) Insufficient interactivity: Existing systems often lack real-time interactive capabilities and are unable to provide feedback or guidance based on users' immediate needs.

[0006] 3) Lack of personalized guidance: Traditional teaching methods find it difficult to provide personalized guidance and suggestions based on the specific circumstances of different users (such as current environment, historical actions).

[0007] 4) Difficulty in integrating multimodal information: Although there are a variety of teaching resources on the market, it is still a challenge to effectively integrate multiple information such as vision, language, and sound to provide a consistent and coherent user experience.

[0008] 5) Lack of real-time feedback and guidance: In situations that require quick response and decision-making, existing systems often fail to provide timely feedback and action guidance. Summary of the invention

[0009] The purpose of the present invention is to provide a wearable skill teaching system in order to overcome the defects of the above-mentioned prior art.

[0010] The purpose of the present invention can be achieved by the following technical solutions:

[0011] The present invention provides a wearable skill teaching system, the system comprising a wearable camera device, a voice processing module, a front-end service module, a back-end service module, and a multimodal interaction module based on an EgoVideo model. The real-time skill teaching interaction process of the system comprises:

[0012] The user wears the wearable camera device, which acquires a video stream for a first-person perspective in real time and transmits the video stream to a backend service module, which pushes the video stream to a frontend service module for real-time display;

[0013] When the backend service module detects the wake-up keyword from the video stream, the frontend service module displays the wake-up text message and plays it in voice. Meanwhile, the backend service module continues to monitor user commands.

[0014] When the backend service module monitors the user's voice command from the video stream, it sends the voice command to the voice processing module to convert it into text command information. The backend inputs the text command information and the video stream into the multimodal interaction module based on the EgoVideo model, and returns the text reply information and the generated predicted video;

[0015] The model-generated prediction video is transmitted to the front-end service module for video display, and the model text reply information is converted into voice through the voice processing module and sent to the front-end service module for voice broadcast.

[0016] Preferably, the multimodal interaction module based on the EgoVideo model includes a video basic model, a language module, a memory module and a generation module, and the multimodal interaction process includes:

[0017] Input the video stream into the video basic model to extract video features;

[0018] The extracted video features are input into the memory module and the language module respectively;

[0019] The language module generates corresponding text reply information based on the extracted video features and the user interaction history information stored in the memory module;

[0020] The generation module uses a long video generation model to generate a predicted video based on the video features output by the video basic model, the text reply information output by the language module, and the user interaction history information stored in the memory module.

[0021] Preferably, the system further comprises: regularly sending the video stream to a multimodal interaction module based on the EgoVideo model, and storing the input and output text information of the language module in the multimodal interaction module in the memory module.

[0022] Preferably, the video basic model adopts a dual-branch fast-slow parallel structure, specifically:

[0023] Input the low frame rate video image set into the ViT network to obtain the slow branch output E vl ;

[0024] The high frame rate video images are combined and input into the ViT network and MotionAdapter module in sequence. The MotionAdapter module learns the dynamic increment of the slow branch and obtains the fast branch output E vh ;

[0025] Output the fast branch Evh and the slow branch output E vl After the connection operation, a linear mapping layer is passed to obtain the final skill teaching prediction video E v .

[0026] Preferably, the MotionAdapter module adopts a spatiotemporal segmentation convolution method to capture temporal information and spatial information at the same time, including a GELU activation layer, a two-dimensional convolution layer, a BatchNorm layer, a ReLU activation layer and a one-dimensional convolution layer arranged in sequence.

[0027] Preferably, the output expression of the MotionAdapter module is:

[0028] MotionAdapter(Y)=Y si W u +Y

[0029] Y st =(TConv1D(Y s ))W m

[0030] Y s =ReLU(BN(Conv2D(Y′)))

[0031] Y′=σ(YW d )

[0032] Where: Y is the Transformerme of each layer in the ViT network r Output: σ represents the GELU activation function; is the network parameter, γ is the bottleneck parameter, D is the feature dimension of each layer of ViT output; Conv2D is a two-dimensional convolutional layer; TConv1D is a one-dimensional convolutional layer; ReLU is a linear rectification activation function; BN represents a BatchNorm layer.

[0033] Preferably, the EgoVideo model adopts video-language-memory joint training, wherein the video-based model and the language module are trained simultaneously by video-language alignment, and the video-based model and the language module are jointly aligned and trained by the triple of video clip-text input-text output, specifically:

[0034] The video clip is input into the video basic model to output the video features; the video features and text input are input into the language module together to obtain the output of the language module; the output of the language module is compared with the text output in the triplet, and the video basic model and the language module are optimized so that the output of the language model is close to the text output.

[0035] Preferably, the video-language alignment method is used to simultaneously train the video base model and the language module, and the corresponding loss function is:

[0036]

[0037] Where: B is a batch during training; s represents the cosine similarity function; They represent the outputs of the language module and the memory module respectively, the superscript i represents the time; τ is the temperature parameter.

[0038] Preferably, the generation module is trained separately, and the specific training process includes:

[0039] After receiving the output of the video base model and the language module, a standard diffusion model is used to predict subsequent video frames using the first frame and language.

[0040] Preferably, after the wearable camera device sets the live streaming address to point to a designated website, it transmits the video stream signal to the system.

[0041] Compared with the prior art, the present invention has the following beneficial effects:

[0042] 1) The wearable camera device can capture the user's first-person perspective, process the understanding and generation tasks at the same time, support the user's first-person perspective dialogue and video generation, have a good user skill teaching experience, and provide users with more intuitive and practical operation guidance through voice reply and generation of predicted videos.

[0043] 2) Through the multimodal interaction module based on the EgoVideo model, combined with voice wake-up technology and voice transcription technology, real-time interaction with users can be achieved, and accurate voice responses can be provided based on user questions.

[0044] 3) Through the multimodal interaction module based on the EgoVideo model, the predicted video of instructions is intelligently generated according to the user's current operating environment and instruction requirements, providing users with instant and personalized operation guidance.

[0045] 4) A memory module is set up in the multimodal interaction module based on the EgoVideo model to integrate historical information to ensure that users can obtain a consistent and coherent experience when receiving guidance in continuous step-by-step actions; in scenarios that require quick response, rapid and accurate feedback and action guidance are provided. BRIEF DESCRIPTION OF THE DRAWINGS

[0046] Figure 1 A schematic diagram of the user interaction process flow of the system of the present invention;

[0047] Figure 2Schematic diagram of the multimodal interaction module architecture based on the EgoVideo model of the present invention;

[0048] Figure 3 This is a page display diagram of the front-end service module in the embodiment. DETAILED DESCRIPTION

[0049] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present invention.

[0050] Example

[0051] like Figure 1 and Figure 2 As shown, this embodiment provides a wearable skill teaching system, including a wearable camera device, a voice processing module, a front-end service module, a back-end service module, and a multimodal interaction module based on the EgoVideo model. The real-time skill teaching interaction process of the system includes:

[0052] The user wears the wearable camera device, which acquires a video stream for a first-person perspective in real time and transmits the video stream to a back-end service module, which pushes the video stream to a front-end service module for real-time display;

[0053] When the backend service module detects the wake-up keyword from the video stream, the frontend service module displays the wake-up text message and plays it in voice. Meanwhile, the backend service module continues to monitor user commands.

[0054] When the backend service module monitors the user's voice command from the video stream, it sends the voice command to the voice processing module to convert it into text command information. The backend inputs the text command information and the video stream into the multimodal interaction module based on the EgoVideo model, and returns the text reply information and the generated predicted video;

[0055] The model-generated prediction video is transmitted to the front-end service module for video display, and the model text reply information is converted into voice through the voice processing module and sent to the front-end service module for voice broadcast.

[0056] Next, each module of the system is introduced in detail.

[0057] 1. Wearable camera

[0058] Wearable video devices include mobile phones, wearable cameras, etc., which can receive video streams in a live streaming mode. Taking the DJI Action3 camera as an example, after setting the live streaming address on the camera to point to a specified URL, the system can obtain the video stream data captured by the camera, including video signals and sound signals.

[0059] 2. Voice module

[0060] 1) Monitor the wake-up word, you can use the Mobvoi SDK;

[0061] 2) ASR (Analytics Resolution Service), using SenseTime API 3. Text-to-speech (TTS), using SenseTime API.

[0062] Each function of the speech processing module is based on open source algorithms and SDKs and can be replaced with other equivalent algorithms and models at will.

[0063] 3. Backend service module

[0064] 1) Real-time monitoring of keywords (e.g. Figure 3 "Hello Wangshu" in the dialog box is a custom setting).

[0065] After the keyword is successfully matched, the user's subsequent voice recording is sent to the voice processing model, and the voice processing model is converted into text and passed to the multimodal interaction module based on the EgoVideo model.

[0066] 2. Record user actions regularly. For example, every T = 45 seconds, pass "Describe the current video" to the multimodal interaction module based on the EgoVideo model. This function is mainly to allow the memory module in the multimodal interaction module based on the EgoVideo model to record what happened in the past.

[0067] 4. Front-end service module

[0068] The front-end service module is mainly for display and voice playback.

[0069] After the user's voice input is converted into text, it is displayed on the graphical interface.

[0070] After the text reply information output by the multimodal interaction module based on the EgoVideo model is returned, it is displayed on the graphical interface and read out through the text-to-speech TTS service. After the predicted video output by the multimodal interaction module based on the EgoVideo model is returned, it is displayed on the graphical interface.

[0071] 5. Multimodal interaction module based on EgoVideo model

[0072] like Figure 2As shown in FIG. 1 , the multimodal interaction module based on the EgoVideo model includes the video basic model EgoVideo-Enc, the language module EgoVideo-Chat, the memory module EgoVideo-mem and the generation module EgoVideo-Gen. The specific multimodal interaction implementation process includes:

[0073] Input the video stream into the video basic model EgoVideo-Enc to extract video features;

[0074] The extracted video features are input into the memory module EgoVideo-mem and the language module EgoVideo-Chat respectively;

[0075] The language module EgoVideo-Chat generates corresponding text reply information based on the extracted video features and the user interaction history information stored in the memory module EgoVideo-mem;

[0076] The generation module EgoVideo-Gen uses a long video generation model to generate a predicted video based on the video features output by the video basic model EgoVideo-Enc, the text reply information output by the language module EgoVideo-Chat, and the user interaction history information stored in the memory module EgoVideo-mem. The video output of the generation module EgoVideo-Gen will be directly displayed on an external display.

[0077] The video stream is sent to the multimodal interaction module based on the EgoVideo model at a regular interval, and the input and output text information of the language module in the multimodal interaction module is stored in the memory module.

[0078] Next, each sub-module in the multimodal interaction module based on the EgoVideo model is introduced in detail.

[0079] (1) The video basic model EgoVideo-Enc converts video input into video features for use by subsequent models. The EgoVideo-Enc model has the following unique designs:

[0080] Since first-person videos are generally long, this embodiment adopts a dual-branch fast and slow parallel model design.

[0081] In this embodiment, the original video stream is preprocessed, specifically: a 2-second video is taken as the original input, and processed into two input forms of low frame rate and high frame rate by downsampling, wherein the low frame rate is 2fps and the high frame rate is 8fps.

[0082] In order to reduce the amount of computation required for training, the EgoVideo-Enc model adopts the ViT architecture in the slow branch and takes low frame rate video as input. The output expression of the slow branch is: E vl =φ backbone (x l ), where x l Input image set for low frame rate video (2fps, 2 seconds, 4 frames in total).

[0083] The fast branch introduces the MotionAdapter architecture to efficiently learn the dynamic increment of the slow branch. The MotionAdapter module is built after each layer of Backbone (ViT). The fast branch output expression is: E vh =φ backbone (φ MotionAdapter (x h )), where x h Input image set for high frame rate video (8fps, 2 seconds, total 16 frames).

[0084] The MotionAdapter module uses a spatiotemporal convolutional method to efficiently capture both temporal and spatial information. When training MotionAdapter, Backbone parameters are locked and only MotionAdapter is trained, which can significantly save the computing resources required for training.

[0085] The MotionAdapter module uses a spatiotemporal segmentation convolution method to capture both temporal and spatial information, including a GELU activation layer, a two-dimensional convolution layer, a BatchNorm layer, a ReLU activation layer, and a one-dimensional convolution layer. The output expression of the MotionAdapter module is:

[0086] MotionAdapter(Y)=Y si W u +Y

[0087] Y st =(TConv1D(Y s ))W m

[0088] Y s =ReLU(BN(Conv2D(Y′)))

[0089] Y′=σ(YW d )

[0090] Where: Y is the output of each layer of Transformermer in the ViT network; σ represents the GELU activation function; is a network parameter, γ is a bottleneck parameter, which is set to 0.25 in this embodiment, D is the feature dimension of each layer of ViT output; Conv2D is a two-dimensional convolutional layer; TConv1D is a one-dimensional convolutional layer; ReLU is a linear rectification activation function; BN represents a BatchNorm layer.

[0091] (2) Language module

[0092] In this embodiment, the language module uses InternLM-7B. In order to align the video and language modules, this embodiment uses 1 million (video clip-text input-text output) triplets to jointly perform alignment training on the video base model and the language module. (The specific training method is: feed the video clip to the video base model to obtain a video feature. Send this video feature and the text input together to the language module to obtain the output of the language module. By comparing the output of the language module with the text output in the triplets, the video base model and the language module are guided, and finally the output of the language model is close to the given text output.)

[0093] (3) Memory module EgoVideo-mem

[0094] The memory module EgoVideo-mem uses text as the memory body. The memory module records all the inputs and outputs of each language module. Every time the language module is called (note that it is not during training), all the inputs and outputs in the memory module will be sent to the language module at the same time. The way to send it to the language module is: change the original [language module input text] into [system prompt word] + [memory module text] + [language module input text]. The specific system prompt words are as follows: "You are an intelligent assistant located on augmented reality (AR) glasses. AR glasses receive video frames from my first-person perspective. Watch the video carefully and pay attention to the movement of objects and human actions. Answer my questions based on your observations. Since you can't see the history of the video, here is the previous history of this video. This history shows what I did before, but this is not what the video is currently showing.".

[0095] (4) Generation module EgoVideo-Gen

[0096] The input of the generation module EgoVideo-Gen is the text given by the user and the last frame of the current video. The generation module uses the long video generation model SEINE (https: / / github.com / Vchitect / SEINE), which is retrained using carefully selected data based on the Ego4D dataset (https: / / ego4d-data.org / ).

[0097] In this embodiment, the EgoVideo model uses video-language-memory joint training, including:

[0098] The video-language alignment method is used to train the video base model and the language module at the same time. The video base model and the language module are jointly aligned and trained through the triples of video clip-text input-text output. Specifically:

[0099] Input the video clip into the video base model and output the video features; input the video features and text input into the language module together to obtain the output of the language module; compare the output of the language module with the text output in the triplet, optimize the video base model and the language module, so that the output of the language model is close to the text output. The video-language alignment method is used to train the video base model and the language module at the same time, and the corresponding loss function is:

[0100]

[0101] Where: B is a batch during training; s represents the cosine similarity function; They represent the outputs of the language module and the memory module respectively, the superscript i represents the time; τ is the temperature parameter.

[0102] The generation module is trained separately. The specific training process includes:

[0103] After receiving the output of the video base model and the language module, the standard diffusion model is used to train the subsequent video frames by predicting the first frame.

[0104] In summary, this embodiment improves the traditional EgoVideo model, including: introducing a memory module to increase long-term memory capability, introducing a language module to increase multimodal dialogue capability, introducing a generation module to increase video generation capability, and introducing a MotionAdapter to improve the original video base model.

[0105] like Figure 2 As shown, the data interaction process of the wearable skill teaching system of this embodiment is specifically as follows:

[0106] When the user turns on the device but does not operate it, the wearable camera device obtains the video stream and sends it to the backend service module. The backend service module sends the obtained video stream to the frontend service module for real-time display. At the same time, the backend service module continuously monitors the wake-up keywords, and sends the video image to the multi-mode output processing module based on the EgoVideo model at regular intervals, and returns text description information;

[0107] When the backend detects the wake-up keyword, it generates a wake-up command and sends it to the voice processing model and returns a response message. The backend transmits the command to the frontend and plays the voice on the frontend page. The backend continues to monitor user commands.

[0108] The user sends a command by voice, the camera obtains the video stream and sends it to the backend, the backend sends the audio signal in the video stream to the voice processing model, converts the voice into text information and returns it to the backend.

[0109] The backend conducts a multimodal dialogue with the EgoVideo model and generates a predicted video after hitting the predicted keyword;

[0110] The backend converts the text responses of the EgoVideo model into voice through a voice processing model, and plays the voice on the frontend.

[0111] Figure 3 This is a schematic diagram of the system front-end page display in this embodiment.

[0112] The above is only a specific embodiment of the present invention, but the protection scope of the present invention is not limited thereto. Any technician familiar with the technical field can easily think of various equivalent modifications or replacements within the technical scope disclosed by the present invention, and these modifications or replacements should be included in the protection scope of the present invention. Therefore, the protection scope of the present invention shall be based on the protection scope of the claims.

Claims

1. A wearable skill teaching system, characterized in that: The system includes a wearable camera device, a voice processing module, a front-end service module, a back-end service module, and a multimodal interaction module based on the EgoVideo model. The real-time skill teaching interaction process of the system includes: The user wears the wearable camera device, which acquires a video stream for a first-person perspective in real time and transmits the video stream to a backend service module, which pushes the video stream to a frontend service module for real-time display; When the backend service module detects the wake-up keyword from the video stream, the frontend service module displays the wake-up text message and plays it in voice. Meanwhile, the backend service module continues to monitor user commands. When the backend service module monitors the user's voice command from the video stream, it sends the voice command to the voice processing module to convert it into text command information. The backend inputs the text command information and the video stream into the multimodal interaction module based on the EgoVideo model, and returns the text reply information and the generated predicted video; The model-generated prediction video is transmitted to the front-end service module for video display, and the model text reply information is converted into voice through the voice processing module and sent to the front-end service module for voice broadcast.

2. A wearable skill teaching system according to claim 1, characterized in that: The multimodal interaction module based on the EgoVideo model includes a video basic model, a language module, a memory module and a generation module. The multimodal interaction process includes: Input the video stream into the video basic model to extract video features; The extracted video features are input into the memory module and the language module respectively; The language module generates corresponding text reply information based on the extracted video features and the user interaction history information stored in the memory module; The generation module uses a long video generation model to generate a predicted video based on the video features output by the video basic model, the text reply information output by the language module, and the user interaction history information stored in the memory module.

3. A wearable skill teaching system according to claim 2, characterized in that: The system also includes: regularly sending the video stream to a multimodal interaction module based on the EgoVideo model, and storing the input and output text information of the language module in the multimodal interaction module in the memory module.

4. A wearable skill teaching system according to claim 2, characterized in that: The video basic model adopts a dual-branch fast-slow parallel structure, specifically: Input the low frame rate video image set into the ViT network to obtain the slow branch output E vl ; The high frame rate video images are combined and input into the ViT network and MotionAdapter module in sequence. The MotionAdapter module learns the dynamic increment of the slow branch and obtains the fast branch output E vh ; Output the fast branch E vh and the slow branch output E vl After the connection operation, a linear mapping layer is passed to obtain the final skill teaching prediction video E v .

5. A wearable skill teaching system according to claim 4, characterized in that: The MotionAdapter module uses a spatiotemporal segmentation convolution method to capture temporal information and spatial information at the same time, including a GELU activation layer, a two-dimensional convolution layer, a BatchNorm layer, a ReLU activation layer, and a one-dimensional convolution layer arranged in sequence.

6. A wearable skill teaching system according to claim 5, characterized in that: The output expression of the MotionAdapter module is: MotionAdapter(Y)=Y st W u +Y Y st =(TConv1D(Y s ))W m AND s =ReLU(BN(Conv2D(Y′))) Y′=σ(YW d ) Where: Y is the output of each layer of Transformer in the ViT network; σ represents the GELU activation function; is the network parameter, γ is the bottleneck parameter, D is the feature dimension of each layer of ViT output; Conv2D is a two-dimensional convolutional layer; TConv1D is a one-dimensional convolutional layer; ReLU is a linear rectification activation function; BN represents a BatchNorm layer.

7. A wearable skill teaching system according to claim 2, characterized in that: The EgoVideo model adopts video-language-memory joint training, wherein the video-language alignment method is used to simultaneously train the video base model and the language module. The video base model and the language module are jointly aligned and trained through the triple of video clip-text input-text output, specifically: The video clip is input into the video basic model to output the video features; the video features and text input are input into the language module together to obtain the output of the language module; the output of the language module is compared with the text output in the triplet, and the video basic model and the language module are optimized so that the output of the language model is close to the text output.

8. A wearable skill teaching system according to claim 7, characterized in that: The video-language alignment method is used to simultaneously train the video basic model and the language module, and the corresponding loss function is: Where: B is a batch during training; s represents the cosine similarity function; They represent the outputs of the language module and the memory module respectively, the superscript i represents the time; τ is the temperature parameter.

9. A wearable skill teaching system according to claim 7, characterized in that: The generation module is trained separately, and the specific training process includes: After receiving the output of the video base model and the language module, a standard diffusion model is used to predict subsequent video frames using the first frame and language.

10. The wearable skill teaching system according to claim 1, characterized in that: After the wearable camera device sets the live streaming address to point to the designated website, it transmits the video stream signal to the system.

Images