A wearable skill teaching system

Abstract

The present application relates to a kind of wearable skill teaching system, including wearable camera, voice processing module, front-end service module, back-end service module and multi-modal interaction module based on EgoVideo model, wearable camera real-time acquisition for the video stream of first perspective and transmit to back-end service module, listen to wake-up key word after listening to user instruction, after listening to user voice instruction, convert voice instruction into text instruction information, input text instruction information and video stream into multi-modal interaction module based on EgoVideo model, return text reply information and generated prediction video, and show through front-end service module.Compared with prior art, the present application can simultaneously process understanding and generation task, support the dialogue and video generation of user first perspective, and have good user skill teaching experience.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of multi-modal data processing, and particularly relates to a wearable skill teaching system. BACKGROUND

[0002] In daily life, especially in scenarios such as kitchen, assembly, etc., users often need to obtain guidance and feedback through visual and auditory information.

[0003] However, the existing teaching auxiliary system often has the following problems:

[0004] 1) Perspective limitation: Most teaching systems adopt a third-person perspective, which is different from the first-person perspective when the user actually operates, making it difficult for the user to directly apply the teaching content to actual operation.

[0005] 2) Lack of interactivity: Existing systems often lack real-time interaction capabilities and cannot provide feedback or guidance according to the user's immediate needs.

[0006] 3) Lack of personalized guidance: Traditional teaching methods cannot provide personalized guidance and suggestions based on the specific circumstances of different users, such as the current environment and historical actions.

[0007] 4) Difficulty in integrating multi-modal information: Although there are various teaching resources on the market, effectively integrating visual, linguistic, and sound information to provide consistent and coherent user experience remains a challenge.

[0008] 5) Lack of real-time feedback and guidance: In situations requiring quick reactions and decisions, existing systems often fail to provide timely feedback and action guidance. SUMMARY

[0009] The present application relates to the field of multi-modal data processing, and particularly relates to a wearable skill teaching system.

[0010] The present application relates to the field of multi-modal data processing, and particularly relates to a wearable skill teaching system.

[0011] The present application provides a wearable skill teaching system, which includes a wearable camera, a voice processing module, a front-end service module, a back-end service module, and a multi-modal interaction module based on the EgoVideo model. The real-time skill teaching interaction process of the system includes:

[0012] The user wears the wearable camera, which acquires a video stream for a first perspective in real time and transmits the video stream to the back-end service module, which pushes the video stream to the front-end service module for real-time display.

[0013] When the backend service module listens to the wake-up keyword from the video stream, the wake-up text information is displayed through the front-end service module and voice played, and the backend service module continuously listens to the user instruction;

[0014] When the backend service module listens to the user voice instruction from the video stream, the voice instruction is sent to the voice processing module to convert into text instruction information, and the backend inputs the text instruction information and the video stream into the multi-modal interaction module based on the EgoVideo model to return the text reply information and the generated prediction video;

[0015] The model-generated prediction video is transmitted to the front-end service module for video display, and the model text reply information is converted into voice through the voice processing module and then sent to the front-end service module for voice broadcast.

[0016] Preferably, the multi-modal interaction module based on the EgoVideo model includes a video base model, a language module, a memory module, and a generation module, and the multi-modal interaction process includes:

[0017] The video stream is input into the video base model to extract video features;

[0018] The extracted video features are input into the memory module and the language module, respectively;

[0019] The language module generates corresponding text reply information according to the extracted video features and the user interaction history information stored in the memory module;

[0020] The generation module selects a long video generation model to generate a prediction video according to the video features output by the video base model, the text reply information output by the language module, and the user interaction history information stored in the memory module.

[0021] Preferably, the system further includes: sending the video stream to the multi-modal interaction module based on the EgoVideo model at a fixed time, and storing the input and output text information of the language module in the multi-modal interaction module in the memory module.

[0022] Preferably, the video base model adopts a double-branch fast-slow parallel structure, specifically:

[0023] The low-frame-rate video image set is input into the ViT network to obtain the slow branch output E vl ;

[0024] The high-frame-rate video image is sequentially input into the ViT network and the MotionAdapter module, and the dynamic increment of the slow branch is learned through the MotionAdapter module to obtain the fast branch output E vh ;

[0025] The fast branch output Evh and slow branch output E vl After the connection operation, a final skill teaching prediction video E is obtained through a linear mapping layer v .

[0026] Preferably, the MotionAdapter module adopts a spatiotemporal split convolution manner to simultaneously capture time information and spatial information, including a GELU activation layer, a two-dimensional convolution layer, a BatchNorm layer, a ReLU activation layer, and a one-dimensional convolution layer arranged in sequence.

[0027] Preferably, the output expression of the MotionAdapter module is:

[0028] MotionAdapter(Y)=Y si W u +Y

[0029] Y st =(TConv1D(Y s ))W m

[0030] Y s =ReLU(BN(Conv2D(Y′)))

[0031] Y′=σ(YW d )

[0032] In the formula, Y is the output of each layer of the ViT network; r σ represents a GELU activation function; σ W is a network parameter, γ is a bottleneck parameter, and D is the feature dimension of the output of each layer of the ViT; Conv2D is a two-dimensional convolution layer; TConv1D is a one-dimensional convolution layer; ReLU is a linear rectification activation function; and BN represents a BatchNorm layer.

[0033] Preferably, the EgoVideo model adopts video-language-memory joint training, wherein the video base model and the language module are simultaneously trained in a video-language alignment manner, and the video base model and the language module are jointly aligned and trained through a video segment-text input-text output triple, specifically as follows:

[0034] The video segment is input into the video base model to output video features; the video features and the text input are input into the language module together to obtain the output of the language module; and the output of the language module and the text output in the triple are compared to optimize the video base model and the language module, so that the output of the language model approximates to the text output.

[0035] Preferably, the video-linguistic alignment method is used to simultaneously train the video base model and the language module, and the corresponding loss function is:

[0036]

[0037] In the formula, B is a batch during training, s represents a cosine similarity function, and i represents a time point. The superscript i represents a time point, and tau is a temperature parameter.

[0038] Preferably, the generation module is trained separately, and the specific training process comprises the following steps:

[0039] After receiving the outputs of the video base model and the language module, a standard diffusion model is used to train the subsequent video frames by using the first frame and the language.

[0040] Preferably, after the wearable camera device sets the live streaming address to point to the specified website, the video stream signal is transmitted to the system.

[0041] Compared with the prior art, the present application has the following beneficial effects:

[0042] 1) The wearable camera device captures the first perspective of the user, simultaneously processes the understanding and generation tasks, supports the conversation and video generation of the first perspective of the user, has a good user skill teaching experience, and provides more intuitive and practical operation guidance for the user through voice reply and generated prediction video.

[0043] 2) Through the multi-modal interaction module based on the EgoVideo model, the voice wake-up technology and the voice transcription technology are combined to realize real-time interaction with the user, and accurate voice reply can be provided according to the user's question.

[0044] 3) Through the multi-modal interaction module based on the EgoVideo model, the prediction video of the indication is intelligently generated according to the current operation environment and instruction demand of the user, and immediate and personalized operation guidance is provided for the user.

[0045] 4) The memory module is arranged in the multi-modal interaction module based on the EgoVideo model, historical information is integrated, consistent and coherent experience is ensured when the user receives guidance in continuous step-by-step actions, and rapid and accurate feedback and action guidance is provided in scenes requiring rapid reaction. BRIEF DESCRIPTION OF DRAWINGS

[0046] Figure 1 It is a user interaction process flow diagram of the system of the present application.

[0047] Figure 2A schematic diagram of a multi-modal interaction module architecture based on an EgoVideo model of the present application;

[0048] Figure 3 A page display diagram of the front-end service module in the embodiment. DETAILED DESCRIPTION

[0049] The technical solutions in the embodiments of the present application will be described clearly and completely below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative labor should fall within the protection scope of the present application.

[0050] EMBODIMENT

[0051] As shown in Figure 1 and Figure 2 The present embodiment provides a wearable skill teaching system, which includes a wearable camera, a voice processing module, a front-end service module, a back-end service module, and a multi-modal interaction module based on an EgoVideo model. The real-time skill teaching interaction process of the system includes:

[0052] The user wears the wearable camera, which acquires a video stream for a first perspective in real time, and transmits the video stream to the back-end service module, which pushes the video stream to the front-end service module for real-time display.

[0053] When the back-end service module listens to a wake-up keyword from the video stream, it displays wake-up text information through the front-end service module and plays the information by voice, while the back-end service module continuously listens to user instructions.

[0054] When the back-end service module listens to a user voice instruction from the video stream, it sends the voice instruction to the voice processing module to convert it into text instruction information, and inputs the text instruction information and the video stream into the multi-modal interaction module based on the EgoVideo model, and returns text reply information and a generated prediction video.

[0055] The model-generated prediction video is transmitted to the front-end service module for video display, and the model text reply information is converted into voice by the voice processing module and then sent to the front-end service module for voice broadcast.

[0056] Next, the various modules of the system will be described in detail.

[0057] 1. Wearable camera

[0058] Wearable video devices, including mobile phones and wearable cameras, can receive video streams via live streaming. For example, with the DJI Action3 camera, after setting the live streaming address on the camera to a specified URL, the system can access the video stream data captured by the camera, including both video and audio signals.

[0059] 2. Voice module

[0060] 1) Monitor the wake-up word, using the Mobvoi SDK;

[0061] 2) ASR (Analytics Speech to Text), using the SenseTime API 3. Text to Speech (TTS), using the SenseTime API.

[0062] Each function of the speech processing module is based on open source algorithms and SDKs and can be replaced with other equivalent algorithms and models at will.

[0063] 3. Backend service module

[0064] 1) Real-time monitoring of keywords (e.g. Figure 3 "Hello Wangshu" in the dialog is a custom setting).

[0065] After the keyword is successfully matched, the user's subsequent voice recording is sent to the voice processing model, which converts the voice processing model into text and passes it to the multimodal interaction module based on the EgoVideo model.

[0066] 2. Periodically record user actions. For example, every T = 45 seconds, pass the "Describe the current video" command to the multimodal interaction module based on the EgoVideo model. This function is primarily intended to allow the memory module in the multimodal interaction module based on the EgoVideo model to record past events.

[0067] 4. Front-end service module

[0068] The front-end service module is mainly for display and voice playback.

[0069] After the user's voice input is converted into text, it is displayed on the graphical interface.

[0070] After the text reply information output by the multimodal interaction module based on the EgoVideo model is returned, it is displayed on the graphical interface and read aloud through the text-to-speech (TTS) service. After the predicted video output by the multimodal interaction module based on the EgoVideo model is returned, it is displayed on the graphical interface.

[0071] 5. Multimodal interaction module based on EgoVideo model

[0072] like Figure 2As shown, the multi-modal interaction module based on the EgoVideo model includes a video base model EgoVideo-Enc, a language module EgoVideo-Chat, a memory module EgoVideo-mem, and a generation module EgoVideo-Gen, and the specific multi-modal interaction implementation process includes:

[0073] The video stream is input to the video base model EgoVideo-Enc, and the video features are extracted;

[0074] The extracted video features are input to the memory module EgoVideo-mem and the language module EgoVideo-Chat, respectively;

[0075] The language module EgoVideo-Chat generates corresponding text reply information according to the extracted video features and the user interaction history information stored in the memory module EgoVideo-mem;

[0076] The generation module EgoVideo-Gen selects a long video generation model to generate a predicted video according to the video features output by the video base model EgoVideo-Enc, the text reply information output by the language module EgoVideo-Chat, and the user interaction history information stored in the memory module EgoVideo-mem. The video output of the generation module EgoVideo-Gen will be directly displayed on the external display screen.

[0077] The video stream is sent to the multi-modal interaction module based on the EgoVideo model at regular intervals, and the input and output text information of the language module in the multi-modal interaction module is stored in the memory module.

[0078] Next, each sub-module in the multi-modal interaction module based on the EgoVideo model will be introduced in detail.

[0079] (1) The video base model EgoVideo-Enc converts video input into video features for subsequent models. For the EgoVideo-Enc model, there are the following unique designs:

[0080] Since the first-person video is generally long, this embodiment adopts a model design of double-branch fast-slow parallel.

[0081] In this embodiment, the original video stream is preprocessed, specifically: taking 2 seconds of video as the original input, processing into low frame rate and high frame rate two input forms through downsampling. The low frame rate is 2fps, and the high frame rate is 8fps.

[0082] To reduce the computational cost required for training, the EgoVideo-Enc model adopts a ViT architecture in the slow branch to take low-frame-rate videos as input. The output expression of the slow branch is: vl φ backbone (x l ) , where x l is the low-frame-rate video input image set (2fps, 2 seconds, a total of 4 frames).

[0083] The fast branch introduces a motion adapter MotionAdapter architecture to efficiently learn the dynamic increment of the slow branch. The MotionAdapter module is placed after each layer of the Backbone (ViT). The output expression of the fast branch is: vh φ backbone (φ MotionAdapter (x h )) , where x h is the high-frame-rate video input image set (8fps, 2 seconds, a total of 16 frames).

[0084] The MotionAdapter module adopts a spatiotemporal split convolution method to efficiently capture temporal and spatial information. When training the MotionAdapter, the parameters of the Backbone are fixed and only the MotionAdapter is trained, thus greatly saving the computational resources required for training.

[0085] The MotionAdapter module adopts a spatiotemporal split convolution method to efficiently capture temporal and spatial information, including a GELU activation layer, a two-dimensional convolution layer, a BatchNorm layer, a ReLU activation layer, and a one-dimensional convolution layer set in turn. The output expression of the MotionAdapter module is:

[0086] MotionAdapter(Y) = Y si W u + Y

[0087] Y st = (TConv1D(Y s ))W m

[0088] Y s = ReLU(BN(Conv2D(Y')))

[0089] Y' = σ(YW d )

[0090] where Y is the output of each layer of the Transformermer in the ViT network; σ represents the GELU activation function; where is the network parameter, is the bottleneck parameter, which is set to 0.25 in this embodiment, and D is the feature dimension output by each layer of the ViT; Conv2D is a two-dimensional convolution layer; TConv1D is a one-dimensional convolution layer; ReLU is a linear rectifier activation function; and BN represents a BatchNorm layer.

[0091] (2) Language module

[0092] In this embodiment, the language module uses InternLM-7B. In order to align the video and language modules, 1 million (video clip-text input-text output) triplets are used to jointly align and train the video base model and the language module. (The specific training method is as follows: the video base model is fed with a video clip to obtain a video feature. The video feature and the text input are fed into the language module to obtain the output of the language module. By comparing the output of the language module with the text output in the triplet, the video base model and the language module are guided, and finally the output of the language model approximates the given text output.)

[0093] (3) Memory module EgoVideo-mem

[0094] The memory module EgoVideo-mem uses text as the memory subject. The memory module records all the inputs and outputs of the language module at each time. When the language module is called each time (note that it is not during training), all the inputs and outputs in the memory module are fed into the language module at the same time. The way to feed the language module is to change the original

language module input text

system prompt word

memory module text

language module input text

[0095] (4) Generation module EgoVideo-Gen

[0096] The input of the generation module EgoVideo-Gen is the text given by the user and the last frame of the current video image. The generation module selects the long video generation model SEINE (https: / / github.com / Vchitect / SEINE), which is obtained after being retrained based on the data carefully selected based on the Ego4D dataset (https: / / ego4d-data.org / ).

[0097] In this embodiment, the EgoVideo model is trained jointly in a video-language-memory manner, including:

[0098] The video base model and the language module are trained simultaneously in a video-language alignment manner, and the video base model and the language module are aligned and trained through a triple of video segment-text input-text output, specifically:

[0099] The video segment is input into the video base model to output video features; the video features and the text input are input into the language module together to obtain the output of the language module; the output of the language module and the text output in the triple are compared, and the video base model and the language module are optimized so that the output of the language model approximates to the text output. The video base model and the language module are trained simultaneously in a video-language alignment manner, and the corresponding loss function is:

[0100]

[0101] In the formula: B is a batch during training; s represents a cosine similarity function; represent the outputs of the language module and the memory module, respectively, and the superscript i represents the time; τ is a temperature parameter.

[0102] The generation module is trained separately, and the specific training process includes:

[0103] After receiving the outputs of the video base model and the language module, a standard diffusion model is used to train the subsequent video frames in a first frame prediction manner.

[0104] In summary, the traditional EgoVideo model is improved in this embodiment, including: introducing a memory module to increase long-term memory capability, introducing a language module to increase multi-modal dialogue capability, introducing a generation module to increase video generation capability, and introducing a MotionAdapter to improve the original video base model.

[0105] As shown in Figure 2 , the data interaction process of the wearable skill teaching system of this embodiment is specifically:

[0106] When the user starts the device without operation, the wearable camera device acquires a video stream and sends the video stream to the backend service module, the backend service module sends the acquired video stream to the front-end service module for real-time display, at the same time, the backend service module continuously listens to the wake-up key word, and sends the video image to the multi-modal output processing module based on the EgoVideo model at regular intervals, and returns the text description information;

[0107] When the backend listens to the wake-up keyword, the backend generates a wake-up instruction and sends it to the voice processing model and returns the response information, the backend transmits the instruction to the front end and plays the voice on the front end page, and the backend continues to listen to the user instruction;

[0108] The user sends an instruction by voice, the camera obtains a video stream and sends it to the backend, the backend sends the audio signal in the video stream to the voice processing model, converts the voice into text information and returns it to the backend.

[0109] The backend carries out multi-modal dialogue with the EgoVideo model, generates a predicted video after hitting the predicted keyword;

[0110] The backend converts the text reply of the EgoVideo model into voice through the voice processing model and plays the voice on the front end.

[0111] Figure 3 The system front end page in the embodiment is shown.

[0112] The above is only a specific embodiment of the present application, but the protection scope of the present application is not limited to this, any skilled person in the art can easily think of various equivalent modifications or replacements within the technical range disclosed by the present application, and these modifications or replacements should be covered within the protection scope of the present application. Therefore, the protection scope of the present application should be subject to the protection scope of the claims.

Claims

1. A wearable skill teaching system, characterized in that: The system includes a wearable camera device, a voice processing module, a front-end service module, a back-end service module, and a multimodal interaction module based on the EgoVideo model. The real-time skill teaching interaction process of the system includes: The user wears the wearable camera device, which acquires a first-person perspective video stream in real time and transmits the video stream to a backend service module, which pushes the video stream to a frontend service module for real-time display. When the backend service module detects the wake-up keyword from the video stream, the frontend service module displays the wake-up text message and plays it in voice. At the same time, the backend service module continues to monitor user commands. When the backend service module monitors the user's voice command from the video stream, it sends the voice command to the voice processing module to convert it into text command information. The backend inputs the text command information and the video stream into the multimodal interaction module based on the EgoVideo model, and returns the text reply information and the generated predicted video; The model-generated prediction video is transmitted to the front-end service module for video display, and the model text reply information is converted into voice through the voice processing module and sent to the front-end service module for voice broadcast.

2. A wearable skill teaching system according to claim 1, characterized in that: The multimodal interaction module based on the EgoVideo model includes a video basic model, a language module, a memory module, and a generation module. The multimodal interaction process includes: Input the video stream into the video basic model to extract video features; The extracted video features are input into the memory module and the language module respectively; The language module generates corresponding text reply information based on the extracted video features and the user interaction history information stored in the memory module; The generation module uses a long video generation model to generate a predicted video based on the video features output by the video basic model, the text response information output by the language module, and the user interaction history information stored in the memory module.

3. A wearable skill teaching system according to claim 2, characterized in that: The system further includes: regularly sending the video stream to a multimodal interaction module based on the EgoVideo model, and storing the input and output text information of the language module in the multimodal interaction module in the memory module.

4. A wearable skill teaching system according to claim 2, characterized in that: The video basic model adopts a dual-branch fast-slow parallel structure, specifically: Input the low frame rate video image set into the ViT network to obtain the slow branch output E vl ; The high frame rate video images are combined and input into the ViT network and MotionAdapter module in sequence. The dynamic increment of the slow branch is learned by the MotionAdapter module to obtain the fast branch output E vh ; Output the fast branch E vh and slow branch output E vl After the connection operation, a linear mapping layer is passed to obtain the final skill teaching prediction video E v .

5. A wearable skill teaching system according to claim 4, characterized in that: The MotionAdapter module uses a spatiotemporal segmentation convolution method to capture temporal and spatial information at the same time, including a GELU activation layer, a two-dimensional convolution layer, a BatchNorm layer, a ReLU activation layer, and a one-dimensional convolution layer arranged in sequence.

6. A wearable skill teaching system according to claim 5, characterized in that: The output expression of the MotionAdapter module is: MotionAdapter(Y)=Y st W u +Y Y st =(TConv1D(Y s ))W m AND s =ReLU(BN(Conv2D(Y′))) Y′=σ(YW d ) Where: Y is the output of each layer of Transformer in the ViT network; σ represents the GELU activation function; is the network parameter, γ is the bottleneck parameter, D is the feature dimension of each layer ViT output; Conv2D is a two-dimensional convolution layer; TConv1D is a one-dimensional convolution layer; ReLU is a linear rectification activation function; BN represents the BatchNorm layer.

7. A wearable skill teaching system according to claim 2, characterized in that: The EgoVideo model uses video-language-memory joint training, in which the video base model and the language module are trained simultaneously using a video-language alignment method. The video base model and the language module are jointly aligned and trained using triples of video clips, text inputs, and text outputs. Specifically, The video clip is input into the video base model to output video features; the video features and text input are input into the language module together to obtain the output of the language module; the output of the language module is compared with the text output in the triplet, and the video base model and language module are optimized so that the output of the language model is close to the text output.

8. A wearable skill teaching system according to claim 7, characterized in that: The video-language alignment method is used to train the video base model and the language module at the same time. The corresponding loss function is: Where: B is a batch during training; s represents the cosine similarity function; Represent the outputs of the language module and memory module respectively, the superscript i represents the time; τ is the temperature parameter.

9. The wearable skill teaching system according to claim 7, characterized in that: The generation module is trained separately, and the specific training process includes: After receiving the output of the video base model and the language module, a standard diffusion model is used to predict subsequent video frames using the first frame and language.

10. The wearable skill teaching system according to claim 1, wherein: After the wearable camera device sets the live streaming address to point to the designated website, it transmits the video stream signal to the system.

Images