All-in-one machine interaction method based on motion capture and visual interaction all-in-one machine thereof

Abstract

The invention discloses an all-in-one machine interaction method based on motion capture and a visual interaction all-in-one machine thereof, and the method comprises the steps: obtaining a teaching video of a teaching user in a teaching process and a teaching motion image corresponding to each frame in the teaching video based on a camera in a visual interaction all-in-one machine in response to an online teaching request of the teaching user; performing multi-scale feature fusion on the target features of the teaching action image by using a feature pyramid network to obtain feature maps of multiple hierarchies, the feature maps of multiple hierarchies including a first feature map of a target hierarchy and second feature maps of other hierarchies; performing sparse processing on the first feature map by adopting a multi-scale cavity convolution mask to obtain a sparse feature map; performing target identification on the second feature map and the sparse feature map to obtain a user dynamic image corresponding to the teaching action image; the user dynamic image and the teaching video are displayed on the visual interaction all-in-one machine, and the user dynamic image rhythms along with playing of the teaching video. The teaching effect is improved.

Description

Technical Field

[0001] The present invention relates to the field of computer technology, and in particular to an all-in-one machine interaction method based on motion capture and a visual interaction all-in-one machine. Background Art

[0002] With the rapid development of the Internet of Things, teaching methods have also diversified, such as online teaching and video teaching through the Internet of Things. For video teaching, it is mainly to record the teaching video of the complete lecture or course content and play it through the terminal device. Therefore, students can learn through the teaching video, resulting in a lack of interaction between teachers and students. For online teaching, it is mainly a one-way knowledge transfer from teachers to students through online teaching, resulting in a lack of face-to-face discussion and communication between teachers and students. Therefore, the existing teaching methods lack interest, which is likely to lead to a decline in students' learning enthusiasm, thereby reducing the teaching effect. Summary of the invention

[0003] The present invention provides an all-in-one machine interaction method based on motion capture and a visual interaction all-in-one machine thereof, which are used to solve the problem that existing teaching methods lack of interest and improve teaching effects.

[0004] In a first aspect, the present invention provides an all-in-one machine interaction method based on motion capture, which is applied to a visual interactive all-in-one machine, wherein a fixed camera is installed in the visual interactive all-in-one machine, and the all-in-one machine interaction method based on motion capture comprises:

[0005] In response to an online teaching request from a teaching user, obtaining a teaching video of the teaching user during the teaching process and a teaching action image corresponding to each frame in the teaching video based on a camera in the visual all-in-one machine;

[0006] Using a feature pyramid network to perform multi-scale feature fusion on the target features of the teaching action image to obtain feature maps at multiple levels, wherein the feature maps at multiple levels include a first feature map at the target level and second feature maps at other levels;

[0007] Using a multi-scale atrous convolution mask to perform sparse processing on the first feature map to obtain a sparse feature map;

[0008] Performing target recognition on the second feature map and the sparse feature map to obtain a user dynamic image corresponding to the teaching action image;

[0009] The user dynamic image and the teaching video are displayed on the visual interactive all-in-one machine, wherein the user dynamic image moves in rhythm with the playing of the teaching video.

[0010] In a second aspect, the present invention provides a visual interactive all-in-one machine, in which a fixedly arranged camera and a chassis are installed, and the visual interactive all-in-one machine includes an acquisition unit, a feature fusion unit, an image processing unit, a target recognition unit, and a display interaction unit;

[0011] The acquisition unit is used to respond to the online teaching request of the teaching user, and acquire the teaching video of the teaching user in the teaching process and the teaching action image corresponding to each frame in the teaching video based on the camera in the visual all-in-one machine;

[0012] The feature fusion unit is used to perform multi-scale feature fusion on the target features of the teaching action image using a feature pyramid network to obtain feature maps of multiple levels, wherein the feature maps of the multiple levels include a first feature map of the target level and second feature maps of other levels;

[0013] The image processing unit is used to perform sparse processing on the first feature map by using a multi-scale hole convolution mask to obtain a sparse feature map;

[0014] The target recognition unit is used to perform target recognition on the second feature map and the sparse feature map to obtain a user dynamic image corresponding to the teaching action image;

[0015] The display interaction unit is used to display the user dynamic image and the teaching video on the visual interaction all-in-one machine, wherein the user dynamic image moves in rhythm with the playing of the teaching video.

[0016] In a third aspect, the present invention further provides a visual interactive all-in-one machine, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, the all-in-one machine interaction method based on motion capture as described in the first aspect is implemented.

[0017] In a fourth aspect, the present invention further provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the all-in-one machine interaction method based on motion capture as described in the first aspect.

[0018] In a fifth aspect, the present invention further provides a computer program product, including a computer program, which, when executed by a processor, implements the all-in-one machine interaction method based on motion capture as described in the first aspect.

[0019] The motion capture-based all-in-one machine interaction method provided by the present invention can display the user's dynamic image and the teaching video together on the visual interactive all-in-one machine, and the user's dynamic image moves rhythmically with the playing of the teaching video, thereby enhancing the interactivity between the teaching user and the students, increasing the fun of teaching, and improving the students' learning enthusiasm, thereby improving the teaching effect. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] In order to more clearly illustrate the technical solutions in the present invention or related technologies, the following briefly introduces the drawings required for use in the embodiments or related technical descriptions. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0021] Figure 1 It is a flow chart of the all-in-one machine interaction method based on motion capture provided by the present invention;

[0022] Figure 2 It is a structural schematic diagram of the target recognition model provided by the present invention;

[0023] Figure 3 is a schematic diagram of the position calibration process provided by the present invention;

[0024] Figure 4 It is a structural schematic diagram of the visual interactive all-in-one machine provided by the present invention;

[0025] Figure 5 It is a structural schematic diagram of the electronic device provided by the present invention. DETAILED DESCRIPTION

[0026] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative work are within the scope of protection of the present invention.

[0027] In the description of the present invention, the terms "first" and "second" are used for descriptive purposes only and should not be understood as indicating or implying relative importance or implicitly indicating the number of the indicated technical features. Thus, the features defined as "first" and "second" may explicitly or implicitly include one or more of the features. In the description of the present invention, the meaning of "plurality" is two or more, unless otherwise clearly and specifically defined.

[0028] In the description of the present invention, the term "for example" is used to mean "used as an example, illustration or explanation". Any embodiment described as "for example" in the present invention is not necessarily to be construed as being more preferred or advantageous than other embodiments. The following description is given to enable any technician in the field to implement and use the present invention. In the following description, details are listed for the purpose of explanation. It should be understood that a person of ordinary skill in the art can recognize that the present invention can be implemented without using these specific details. In other examples, well-known structures and processes will not be elaborated in detail to avoid obscuring the description of the present invention with unnecessary details. Therefore, the present invention is not intended to be limited to the embodiments shown, but is consistent with the widest scope consistent with the principles and features disclosed in the present invention.

[0029] Figure 1 The figure is a flow chart of the all-in-one machine interaction method based on motion capture provided by the present invention. The all-in-one machine interaction method based on motion capture is applied to a visual interactive all-in-one machine. A fixedly set camera is installed in the visual interactive all-in-one machine. The all-in-one machine interaction method based on motion capture includes:

[0030] Step 10, responding to the online teaching request of the teaching user, obtaining the teaching video of the teaching user during the teaching process and the teaching action image corresponding to each frame in the teaching video based on the camera in the visual all-in-one machine.

[0031] It should be noted that when the teaching user needs to conduct online teaching, a request needs to be triggered in the visual interactive all-in-one machine. Therefore, the visual interactive all-in-one machine responds to the teaching user's online teaching request. At this time, the visual interactive all-in-one machine calls the camera in the visual all-in-one machine to record the teaching process of the teaching user, thereby obtaining the teaching video of the teaching user in the teaching process. At the same time, the visual interactive all-in-one machine obtains the teaching action image corresponding to each frame in the teaching video.

[0032] Step 20, using a feature pyramid network to perform multi-scale feature fusion on the target features of the teaching action image to obtain feature maps of multiple levels, wherein the feature maps of the multiple levels include a first feature map of the target level and second feature maps of other levels.

[0033] Among them, Feature Pyramid Network (FPN) is a deep neural network widely used in tasks such as object recognition and semantic segmentation. The Feature Pyramid Network extracts and fuses feature information of different scales by constructing a feature pyramid. The feature pyramid contains multiple levels, and each level corresponds to a feature map of a different scale.

[0034] The levels of the feature pyramid usually include: bottom-up feature extraction levels (such as C2, C3, C4, C5) and top-down feature fusion levels (such as P2, P3, P4, P5), which provide rich multi-scale feature information for target recognition tasks.

[0035] Exemplarily, the target level can be the P3 level. The feature map of the P3 level usually has a moderate resolution, which helps to capture the characteristics of the target. The targets in the teaching action images require sufficient resolution to retain their key information due to their small pixel area, and the P3 level is just at the balance point between high resolution and strong semantic information. In addition, the P3 level fuses the strong semantic information from high levels (such as P4 and P5) and the high-resolution information from low levels (such as C3) through lateral connections. This fusion enables the feature map of the P3 level to contain rich semantic information while retaining sufficient image details, which helps to improve the recognition performance of the target.

[0036] After extracting the target features of the teaching action images, the visual interactive all-in-one machine can use the feature pyramid network to perform multi-scale feature fusion on the target features to obtain feature maps of multiple levels. It can be understood that each level corresponds to a feature map of a different scale, and the feature map corresponding to the target level (such as the P3 level) among the multiple levels is the first feature map, and the feature maps corresponding to the other levels except the target level among the multiple levels are the second feature maps. There are as many second feature maps as there are other levels.

[0037] In one embodiment, the all-in-one machine interaction method based on motion capture may also include: the visual interactive all-in-one machine uses a backbone network (Backbone) to extract features from the teaching action image to obtain target features; wherein the backbone network may include: multiple second convolutional layers and cross-stage partial networks (Cross Stage Partial networks, CSP). The visual interactive all-in-one machine may use a backbone network (including multiple second convolutional layers and cross-stage partial networks CSP) to extract features from the teaching action image. Specifically, the visual interactive all-in-one machine may use a series of convolution operations and residual connections in the backbone network to extract key feature information in the teaching action image, namely, target features.

[0038] Step 30, using a multi-scale dilated convolution mask to perform sparse processing on the first feature map to obtain a sparse feature map.

[0039] Among them, a multi-scale dilated convolution mask (MDCM) is a module proposed by the present invention for performing sparse processing on a feature map to obtain a sparse feature map.

[0040] After determining the feature maps of multiple levels, the visual interactive all-in-one machine can use a multi-scale hole convolution mask to perform sparse processing on the first feature map of the target level in the feature maps of multiple levels to obtain a sparse feature map. The whole process can not only retain the key feature information in the first feature map, but also effectively filter out redundant information. The obtained sparse feature map is more accurate and has lower redundancy, making the subsequent target recognition process more accurate and efficient.

[0041] In one embodiment, the multi-scale atrous convolution mask may include: a first convolution layer and a first output layer; the visual interactive all-in-one machine uses the multi-scale atrous convolution mask to perform sparse processing on the first feature map to obtain a sparse feature map, which may include: the visual interactive all-in-one machine uses the first convolution layer to process the first feature map to obtain a sparse mask vector; the visual interactive all-in-one machine uses the first output layer to perform sparse processing on the first feature map based on the sparse mask vector to obtain a sparse feature map.

[0042] It should be noted that the sparse mask vector can accurately locate the foreground area.

[0043] In the process of determining the sparse feature map, the visual interactive all-in-one machine can first use the first convolution layer to process the first feature map to obtain a sparse mask vector. In this way, the visual interactive all-in-one machine can use the first output layer to perform sparse processing on the first feature map based on the sparse mask vector to obtain a sparse feature map. While retaining the key feature information in the first feature map, the redundant information in the first feature map is effectively filtered to reduce unnecessary computational burden and effectively improve the efficiency of subsequent target recognition.

[0044] The first convolution layer may include: multiple dilated convolution kernels with different dilation rates, at least one channel compression convolution layer and a second output layer; the visual interactive all-in-one machine uses the first convolution layer to process the first feature map to obtain a sparse mask vector, which may include: the visual interactive all-in-one machine uses multiple dilated convolution kernels with different dilation rates to extract initial information of different scales from the first feature map; the visual interactive all-in-one machine uses at least one channel compression convolution layer to perform channel compression processing on the initial information of different scales to obtain target information; the visual interactive all-in-one machine uses the second output layer to determine the sparse mask vector based on the target information.

[0045] Among them, dilated convolution (abbreviated as Dilated Conv), also known as dilated convolution, is a method to expand the receptive field of the convolution kernel without increasing the computational cost, and is increasingly used in target recognition. In order to enhance the recognition head's ability to recognize different large targets, different dilation rates are set and multiple dilated convolution kernels with different dilation rates are used to extract features at different scales.

[0046] In the process of determining the sparse mask vector, the visual interactive all-in-one machine can use multiple dilated convolution kernels with different dilation rates in the first convolution layer to extract initial information of different scales from the first feature map. Then, the visual interactive all-in-one machine can use at least one channel compression convolution layer in the first convolution layer to perform channel compression processing on the initial information of different scales (the purpose is to effectively filter the redundant information in the initial information of different scales to reduce the amount of calculation) to obtain the target information; then, the visual interactive all-in-one machine can use the second output layer to determine the sparse mask vector based on the target information. In the whole process, through multiple dilated convolution kernels with different dilation rates, a diverse receptive field can be obtained to capture the initial information of different scales, and through at least one channel compression convolution layer, the amount of calculation can be effectively reduced.

[0047] Step 40, performing target recognition on the second feature map and the sparse feature map to obtain a user dynamic image corresponding to the teaching action image.

[0048] Among them, the recognition head is mainly responsible for predicting bounding boxes, confidence levels, and category probabilities.

[0049] After determining the sparse feature map, the visual interactive all-in-one machine can use the second feature map of other levels and the sparse feature map Figure 1 The teaching action image is sent to the recognition head for target recognition, and the user dynamic image corresponding to the teaching action image output by the recognition head is obtained. The user dynamic image is relatively accurate. In the whole process, while ensuring the recognition efficiency, the accuracy of target recognition is effectively improved.

[0050] For example, Figure 2 It is a schematic diagram of the structure of the target recognition model provided by the present invention. Figure 2 It can be seen that the target recognition model can use the YOLOv5 framework. YOLOv5 can include: input layer (Input), backbone network (Backbone), neck network (Neck) and recognition head (Head). The backbone network can include multiple second convolutional layers (Conv) and cross-stage local networks (CSP), and the cross-stage local network includes multiple CSP modules; the neck network includes the feature pyramid network (FPN) mentioned above. Figure 2In the multi-scale atrous convolution mask (MDCM), Conv represents a channel compression convolution layer; Dilated Conv represents a dilated convolution kernel, and the dilation rates of the two Dilated Convs are different. In this way, the visual interactive all-in-one machine can use a feature pyramid network to perform multi-scale feature fusion on the target features of the teaching action image to obtain feature maps of multiple levels. The feature maps of multiple levels include the first feature map of the target level (such as the P3 level) and the second feature maps of other levels. For the P3 level, the visual interactive all-in-one machine can use a multi-scale atrous convolution mask to perform sparse processing on the first feature map of the P3 level to obtain a sparse feature map, and then use the sparse feature map and the second feature map of other levels. Figure 1 The teaching action image is sent to the recognition head for target recognition, and the user dynamic image corresponding to the teaching action image output by the recognition head is obtained.

[0051] Step 50, displaying the user's dynamic image and the teaching video on the visual interactive all-in-one machine, wherein the user's dynamic image moves in rhythm with the playing of the teaching video.

[0052] Furthermore, the visual interactive all-in-one machine displays the user's dynamic image and the teaching video on the visual interactive all-in-one machine, wherein the user's dynamic image moves in rhythm with the playing of the teaching video.

[0053] The embodiment of the present invention can display the user's dynamic image and the teaching video together on the visual interactive all-in-one machine, and the user's dynamic image follows the playing of the teaching video, which enhances the interactivity between the teaching user and the students, increases the interest of teaching, and improves the students' learning enthusiasm, thereby improving the teaching effect.

[0054] In one embodiment, in order to improve the accuracy of motion capture, before performing motion capture, it is also necessary to calibrate the position of the camera in the visual interactive all-in-one machine. Therefore, the all-in-one machine interaction method based on motion capture provided by the embodiment of the present invention also includes:

[0055] The visual interactive all-in-one machine is controlled to shoot the calibration object through a camera to obtain a first image set; the first image set includes images of the calibration object at different shooting distances and / or different shooting angles; the pitch angle of the camera is calibrated based on the first image set, and the pitch angle calibration result of the camera is determined; the visual interactive all-in-one machine is controlled to move to the front of the calibration object; the chassis is controlled to rotate, and the calibration object is shot through the camera to obtain a second image set; the second image set includes images of the calibration object shot at different rotation angles; based on the second image set and the rotation angle information of the chassis, the relative position of the camera and the chassis is calibrated to determine the position calibration result of the camera.

[0056] The first image set includes images of the calibration object at different shooting distances and / or different shooting angles.

[0057] Specifically, the calibration object is first fixed at a certain position (such as on a wall) so that the center of the calibration object is basically consistent with the height of the camera fixed on the visual interactive all-in-one machine, and ensure that there is no debris in the area around the calibration object to avoid interference with the movement of the visual interactive all-in-one machine.

[0058] Optionally, the calibration object is a calibration plate.

[0059] Further, the visual interactive all-in-one machine is controlled to shoot the calibration object through a camera to obtain a first image set. In one embodiment, the visual interactive all-in-one machine is controlled to be about 2 meters away from the calibration plate, and the pitch angle calibration program is started. The visual interactive all-in-one machine will move autonomously according to the set pitch angle calibration program, shoot the calibration plate through the camera, and collect a series of calibration object images at different shooting distances and / or different shooting angles to form a first image set.

[0060] After obtaining the first image set, the pitch angle calibration program can process and analyze the collected first image set, and use the PNP algorithm to solve the external parameter data of the camera, combine the internal parameter data of the camera and the calibration object data (that is, the known parameters of the calibration object) to calibrate the pitch angle of the camera, and calculate the attitude angle of the camera relative to the calibration object. The attitude angle is recorded as (Roll, Pitch, Yaw), where Roll represents the roll angle, Yaw represents the yaw angle, and Pitch represents the pitch angle. The calculated Pitch angle is the pitch angle of the 3D camera relative to the moving plane of the visual interactive all-in-one machine. By calculating multiple sets of Pitch angle data and performing weighted aggregation calculations, the final pitch angle calibration result of the camera can be determined.

[0061] Furthermore, the visual interactive all-in-one machine is controlled to move to a position about 0.5 meters in front of the calibration object so that the visual interactive all-in-one machine faces the calibration object.

[0062] Furthermore, the chassis is controlled to rotate, and the calibration object is photographed by a camera to obtain a second image set, where the second image set includes images of the calibration object photographed at different rotation angles.

[0063] When the visual interactive all-in-one machine is facing the calibration object, the position calibration program is started. The visual interactive all-in-one machine will automatically control the chassis rotation according to the set position calibration program, and shoot the calibration object through the camera to obtain the second image set.

[0064] The visual interactive all-in-one machine can first collect the reference image I according to the set position calibration program, collect the current calibration object image by controlling the chassis to rotate at a small angle θ, and record the current rotation angle θ; the calibration object is photographed by rotating at different angles, and the calibration object images photographed at different rotation angles are collected to form a second image set, and multiple groups of data are generated, each group of data includes a calibration object image and the rotation angle corresponding to the calibration object image (that is, the rotation angle of the chassis).

[0065] Furthermore, based on the second image set and the rotation angle information of the chassis, the relative position of the camera and the chassis is calibrated, and the horizontal position offset of the 3D camera relative to the center of the chassis of the visual interactive all-in-one machine is calculated, that is, the position calibration result of the camera.

[0066] The embodiment of the present invention does not require technicians to use additional collection tooling to perform data collection and camera calibration. The camera calibration can be completed by using images collected by the camera set up by the visual interactive all-in-one machine itself, thereby improving the accuracy of camera calibration and thus improving the accuracy of motion capture.

[0067] In one embodiment, the rotation angle information includes the rotation angle corresponding to each calibration object image in the second image set; based on the rotation angle information of the second image set and the chassis, the relative position of the camera and the chassis is calibrated to determine the position calibration result of the camera, including: obtaining the camera internal parameter data and calibration object data of the camera; based on the camera internal parameter data, the calibration object data, each calibration object image in the second image set and the rotation angle corresponding to each calibration object image in the second image set, the relative position of the camera and the chassis is calibrated to determine multiple position deviation values; based on the multiple position deviation values, weighted aggregation calculation is performed to obtain the position calibration result of the camera.

[0068] Specifically, the camera internal parameter data and calibration object data of the camera are obtained.

[0069] The calibration object data refers to the known parameters of the calibration object.

[0070] In computer vision, especially in the field of camera calibration and stereo vision, camera intrinsic parameters and camera extrinsic parameters are very important concepts, both of which are related to the geometric properties and posture of the camera.

[0071] The camera internal parameter data is the parameters describing the internal properties of the camera, including focal length, principal point (optical center) coordinates, distortion coefficient, etc. The camera internal parameter data is usually determined when the camera is calibrated. The camera internal parameter data of a specific camera model is usually fixed and does not change over time. Once the camera internal parameter data is determined, the camera internal parameter data usually remains unchanged during the use of the camera.

[0072] Camera extrinsic data are parameters that describe the position and posture of the camera in the world coordinate system, usually including rotation matrix and translation vector. Camera extrinsic data may change at different camera positions or shooting times. For example, in stereo vision, assuming there are two cameras, the relative position and direction between the two cameras will change every time the camera is moved, resulting in changes in the camera extrinsic data.

[0073] Furthermore, based on the camera internal parameter data, the calibration object data, each calibration object image in the second image set and the rotation angle corresponding to each calibration object image in the second image set, the relative position of the camera and the chassis is calibrated to determine multiple position deviation values.

[0074] Furthermore, a weighted aggregation calculation is performed based on the multiple position deviation values ​​to obtain a position calibration result of the camera.

[0075] In one embodiment, based on the camera internal parameter data, the calibration object data, each calibration object image in the second image set, and the rotation angle corresponding to each calibration object image in the second image set, the relative position of the camera and the chassis is calibrated to determine a plurality of position deviation values, including: selecting any two calibration object images in the second image set as the first calibration object image and the second calibration object image, respectively; based on the camera internal parameter data and the calibration object data, determining a first position vector and a second position vector, respectively; the first position vector is a position vector between the calibration object and the camera in the first calibration object image, and the second position vector is a position vector between the calibration object and the camera in the second calibration object image a position vector between the first position and the second position; based on the first position vector and the second position vector, determining the distance between the first position and the second position; the first position is the position of the camera when the camera shoots the first calibration object image, and the second position is the position of the camera when the camera shoots the second calibration object image; based on the plane geometric relationship between the first position, the second position and the center point of the chassis, the rotation angle corresponding to the first calibration object image and the rotation angle corresponding to the second calibration object image, the relative position of the camera and the chassis is calibrated, and a position deviation value is calculated; returning to the step of selecting any two calibration object images in the second image set until multiple position deviation values ​​are calculated.

[0076] After obtaining the camera internal parameter data and calibration object data of the camera, the camera position may be calibrated.

[0077] See also Figure 3 , Figure 3 is a schematic diagram of the position calibration process provided by an embodiment of the present invention. Since in this embodiment, the camera is fixedly mounted on the visual interactive all-in-one machine, when the visual interactive all-in-one machine performs a self-rotation action (i.e., controls the chassis to rotate), the camera will perform a circular motion with the center point o of the chassis of the visual interactive all-in-one machine as the center of the circle and the position deviation length length as the radius.

[0078] Specifically, every time the visual interactive all-in-one machine rotates an angle, the position of the camera before and after the rotation will form an isosceles triangle with the center o of the circle.

[0079] Specifically, after obtaining the camera's internal parameter data and calibration object data, the three-axis translation vector of the camera relative to the calibration plate can be calculated through the OpenCV hand-eye calibration function.

[0080] Further, any two calibration object images are selected in the second image set as the first calibration object image i and the second calibration object image j, respectively, and the first position vector is determined based on the camera intrinsic parameter data and the calibration object data. and the second position vector

[0081] First position vector is the position vector between the calibration object and the camera in the first calibration object image, and the second position vector is the position vector between the calibration object and the camera in the second calibration object image.

[0082] Furthermore, it can be seen from the spatial vector relationship that based on the first position vector and the second position vector The distance l between the first position ci and the second position cj can be determined, and the calculation formula of the distance l is as follows:

[0083]

[0084] The first position is the position of the camera when the camera captures the first calibration object image, and the second position is the position of the camera when the camera captures the second calibration object image.

[0085] Furthermore, based on the planar geometric relationship between the first position ci, the second position cj and the center point o of the chassis, the rotation angle corresponding to the first calibration object image and the rotation angle corresponding to the second calibration object image, the relative position of the camera and the chassis is calibrated to calculate a position deviation value.

[0086] like Figure 3 As shown, any two calibration object images are selected in the second image set, which are respectively recorded as the first calibration object image 1 and the second calibration object image 2, and the first position vector is determined based on the camera intrinsic parameter data and the calibration object data. and the second position vector

[0087] From the spatial vector relationship, we can see that based on the first position vector and the second position vector The distance between the first position c1 and the second position c2 can be determined

[0088] Assuming that the visual interactive all-in-one machine performs a rotation action (i.e., controls the rotation of the chassis), the camera makes a circular motion with the center point o of the chassis of the visual interactive all-in-one machine as the center and the position deviation length as the radius. Then the first position c1, the second position c2 and the center point o of the chassis can form an isosceles triangle with a vertex angle of θ. The vertex angle θ is the rotation angle of the chassis, which can be determined based on the rotation angle corresponding to the first calibration object image and the rotation angle corresponding to the second calibration object image.

[0089] like Figure 3 As shown, at this time, based on the plane geometric relationship between the first position ci, the second position cj and the center point o of the chassis, the base angle α of the isosceles triangle can be obtained, and the calculation formula of the base angle α is as follows:

[0090] α=(π-θ) / 2.

[0091] According to the plane geometric relationship, the relative position of the camera and the chassis can be calibrated to calculate a position deviation value length. The calculation formula of the position deviation value length is as follows:

[0092]

[0093] Furthermore, the process returns to the step of selecting any two calibration object images in the second image set until a plurality of position deviation values ​​are calculated.

[0094] Specifically, based on the collected multiple calibration object images in the second image set, multiple position deviation values ​​are calculated and respectively recorded as length1, length2, length3, length4, etc.

[0095] Furthermore, based on multiple position deviation values, a weighted aggregation calculation is performed to obtain the position calibration result of the camera.

[0096] The embodiment of the present invention collects calibration object image data of different viewing angles through the self-rotation of the visual interactive all-in-one machine, combines the angle data of the chassis rotation, and utilizes the geometric relationship between space and plane to realize the calibration of the relative position of the camera and the chassis through simple calculation, thereby realizing the accurate calculation of the position deviation of the camera relative to the chassis of the visual interactive all-in-one machine.

[0097] A weighted aggregation calculation is performed based on multiple position deviation values ​​to obtain a position calibration result of a video camera, including: selecting a position deviation value as reference data and using the remaining position deviation values ​​as comparison data; respectively determining the distance difference between each comparison data and the reference data; respectively comparing each distance difference with a first preset threshold to obtain multiple comparison results; based on each comparison result, determining the error count of the reference data; returning to the step of selecting a position deviation value as reference data and using the remaining position deviation values ​​as comparison data until the error counts corresponding to all position deviation values ​​are obtained; based on the error count corresponding to each position deviation value, abnormal data is eliminated to obtain multiple normal position deviation values; based on the multiple normal position deviation values, a weighted aggregation calculation is performed to obtain the position calibration result of the video camera.

[0098] It is understandable that when collecting the calibration object image, when the visual interactive all-in-one machine has slight movement or the shooting angle is not good, the collected calibration object image may contain blurred or unclear images, and such image data will affect the calculation of the position deviation value.

[0099] Therefore, it is necessary to eliminate abnormal position deviation values ​​and then perform weighted aggregation calculation to effectively improve the accuracy of the position calibration results.

[0100] In this embodiment, a statistical voting method is used to identify and eliminate abnormal position deviation values. The specific steps are as follows:

[0101] (1) Select one position deviation value as the reference data and the remaining position deviation values ​​as the comparison data.

[0102] Specifically, among the multiple position deviation values, the first position deviation value is selected as reference data, and the remaining position deviation values ​​are used as comparison data.

[0103] (2) Determine the distance difference between each comparison data and the reference data respectively.

[0104] Specifically, each remaining position deviation value is sequentially compared with the reference data (ie, the first position deviation value) for analysis: for each remaining position deviation value, the distance difference between the remaining position deviation value and the reference data is calculated.

[0105] For example, calculate the second position deviation value length 2 and reference data length 1 The distance difference Δdis between them is calculated as follows:

[0106] Δdis=|length 2 -length 1 |.

[0107] (3) Compare each distance difference Δdis with the first preset threshold value to obtain multiple comparison results.

[0108] Optionally, the first preset threshold is 0.1m (meter).

[0109] For each distance difference Δdis, if the distance difference Δdis is greater than the first preset threshold, it can be considered that there is a large error between the remaining position deviation values ​​and the reference data; if the distance difference Δdis is less than or equal to the first preset threshold, it can be considered that there is a small error between the remaining position deviation values ​​and the reference data.

[0110] (4) Based on each comparison result, determine the error count of the reference data.

[0111] Specifically, if the distance difference Δdis is greater than a first preset threshold, the error count of the reference data is increased by 1. The error count of the reference data may be determined based on each comparison result.

[0112] It is understandable that the error count of a reference data should start increasing from 0.

[0113] (5) Return to the step of selecting a position deviation value as reference data and using the remaining position deviation values ​​as comparison data until the error counts corresponding to all position deviation values ​​are obtained.

[0114] Specifically, for each reference data, loop through all other position deviation values, complete a round of comparison, and then replace the new reference data, and use length2, length3, length4, etc. as reference data in turn, and repeat the above comparison process until the error counts corresponding to all position deviation values ​​are obtained.

[0115] (6) Based on the error count corresponding to each position deviation value, abnormal data is eliminated to obtain multiple normal position deviation values; based on the multiple normal position deviation values, weighted aggregation calculation is performed to obtain the position calibration result of the camera.

[0116] Specifically, based on each position deviation value, the error count values ​​are sorted from large to small to obtain a sorting result; the top 20% of the position deviation values ​​in the sorting result are taken as abnormal data, the top 20% of the position deviation values ​​in the sorting result are eliminated, and the remaining position deviation values ​​are normal position deviation values.

[0117] Furthermore, based on multiple normal position deviation values, a weighted aggregation calculation is performed to obtain a position calibration result of the camera.

[0118] Considering that the calculation of abnormal position deviation values ​​is based on images, when abnormal position deviation values ​​appear, there may be problems with the image. Therefore, the image corresponding to the abnormal position deviation value should be eliminated, and then the position calibration result of the camera should be recalculated based on the retained normal image.

[0119] Preferably, based on each position deviation value, the error count values ​​are sorted from large to small. After the sorting results are obtained, the calibration object images corresponding to the top 20% of the position deviation values ​​in the sorting results are taken as abnormal images, all abnormal images are eliminated, and multiple normal images are retained; based on the normal images, the relative position of the camera and the chassis is calibrated to determine the position calibration result of the camera.

[0120] It should be noted that, in the prior art, there are few studies on the accuracy verification of data sets. Most studies assume that the collected data sets are normal and can be used directly, so the collected image data are not verified and screened. The present invention can further improve the accuracy of the calibration results by performing a secondary screening on the collected images, traversing the position deviation values, comparing the differences between them, eliminating image data based on statistical voting results, and calculating the final position calibration results based on the remaining images.

[0121] The embodiment of the present invention adopts a data elimination strategy to eliminate abnormal data, identifies and eliminates abnormal data by means of statistical voting, ensures the reliability and accuracy of the final result, and enhances the robustness of the algorithm.

[0122] In one embodiment, based on a first image set, the pitch angle of a camera is calibrated to determine the pitch angle calibration result of the camera, including: obtaining camera extrinsic parameter data of the camera, camera intrinsic parameter data of the camera, and calibration object data of the camera; based on the camera extrinsic parameter data, the camera intrinsic parameter data, the calibration object data, and the first image set, the pitch angle of the camera is calibrated to determine multiple pitch angles; based on the multiple pitch angles, a weighted aggregation calculation is performed to obtain the pitch angle calibration result of the camera.

[0123] Specifically, after obtaining the first image set, the pitch angle calibration program can process and analyze the collected first image set, solve the camera extrinsic parameter data of the camera by using the PNP algorithm, and obtain the camera intrinsic parameter data and calibration object data (i.e., the known parameters of the calibration object) of the camera.

[0124] Furthermore, based on the camera extrinsic parameter data, the camera intrinsic parameter data, the calibration object data and the first image set, the pitch angle of the camera is calibrated, and multiple attitude angles of the camera relative to the calibration object are calculated. The attitude angles are recorded as (Roll, Pitch, Yaw), where Roll represents the roll angle, Yaw represents the yaw angle, and Pitch represents the pitch angle.

[0125] Furthermore, based on the multiple pitch angles, a weighted aggregation calculation is performed to obtain a pitch angle calibration result of the camera.

[0126] The embodiment of the present invention collects multiple groups of calibration object images and combines an external parameter calibration algorithm and a weighted average method to achieve camera pitch angle calibration, which has simple operation and low calculation complexity.

[0127] In one embodiment, a weighted aggregation calculation is performed based on multiple pitch angles to obtain a pitch angle calibration result of the camera, including: selecting a pitch angle as reference data and using the remaining pitch angles as comparison data; respectively determining the angle difference between each comparison data and the reference data; respectively comparing each angle difference with a second preset threshold to obtain multiple comparison results; based on each comparison result, determining an error count of the reference data; returning to the step of selecting a pitch angle as reference data and using the remaining pitch angles as comparison data until the error counts corresponding to all pitch angles are obtained; based on the error count corresponding to each pitch angle, abnormal data is eliminated to obtain multiple normal pitch angles; based on the multiple normal pitch angles, weighted aggregation calculation is performed to obtain the pitch angle calibration result of the camera.

[0128] It is understandable that when collecting the calibration object image, when the visual interactive all-in-one machine has slight movement or the shooting angle is not good, the collected calibration object image may contain blurred or unclear images, and such image data will affect the calculation of the pitch angle.

[0129] Therefore, it is necessary to eliminate abnormal pitch angles and then perform weighted aggregation calculation to effectively improve the accuracy of the pitch angle calibration results.

[0130] In this embodiment, a statistical voting method is used to identify and eliminate abnormal pitch angles. The specific steps are as follows:

[0131] (1) Select one pitch angle as the reference data and the remaining pitch angles as the comparison data.

[0132] Specifically, among the multiple pitch angles, the first pitch angle is selected as reference data, and the remaining pitch angles are used as comparison data.

[0133] (2) Determine the angle difference between each comparison data and the reference data respectively.

[0134] Specifically, each remaining pitch angle is sequentially compared and analyzed with the reference data (ie, the first pitch angle): for each remaining pitch angle, the angle difference between the remaining pitch angle and the reference data is calculated.

[0135] (3) Compare each angle difference with the second preset threshold value to obtain multiple comparison results.

[0136] For each angle difference, if the angle difference is greater than the second preset threshold, it can be considered that there is a large error between the remaining pitch angles and the reference data; if the angle difference is less than or equal to the second preset threshold, it can be considered that there is a small error between the remaining pitch angles and the reference data.

[0137] (4) Based on each comparison result, determine the error count of the reference data.

[0138] Specifically, if the angle difference is greater than the second preset threshold, the error count of the reference data is increased by 1, and the error count of the reference data can be determined based on each comparison result.

[0139] It is understandable that the error count of a reference data should start increasing from 0.

[0140] (5) Return to the step of selecting one pitch angle as reference data and using the remaining pitch angles as comparison data until the error counts corresponding to all pitch angles are obtained.

[0141] (6) Based on the error count corresponding to each pitch angle, abnormal data is eliminated to obtain multiple normal pitch angles; based on the multiple normal pitch angles, weighted aggregation calculation is performed to obtain the pitch angle calibration result of the camera.

[0142] Specifically, based on each pitch angle, the error count values ​​are sorted from large to small to obtain a sorting result; the first 20% of the pitch angles in the sorting result are taken as abnormal data, the first 20% of the pitch angles in the sorting result are eliminated, and the remaining pitch angles are normal pitch angles.

[0143] Furthermore, based on multiple normal pitch angles, a weighted aggregation calculation is performed to obtain a pitch angle calibration result of the camera.

[0144] Considering that the calculation of abnormal pitch angles is based on images, when a pitch angle appears, there may be a problem with the image. Therefore, the image corresponding to the abnormal pitch angle should be eliminated, and then the pitch angle calibration result of the camera should be recalculated based on the retained normal image.

[0145] Preferably, based on each pitch angle, the error count values ​​are sorted from large to small. After the sorting results are obtained, the calibration object images corresponding to the first 20% of the pitch angles in the sorting results are taken as abnormal images, all abnormal images are eliminated, and multiple normal images are retained; based on the normal images, the pitch angle of the camera is calibrated to determine the pitch angle calibration result of the camera.

[0146] The visual interactive all-in-one machine provided by the present invention is described below. The visual interactive all-in-one machine described below and the all-in-one machine interaction method based on motion capture described above can be referred to each other. Figure 4 , Figure 4 Schematic diagram of an embodiment of a visual interactive all-in-one machine provided by an embodiment of the present invention. Figure 4 As shown, a fixed camera and chassis are installed in the visual interactive all-in-one machine, and the visual interactive all-in-one machine includes an acquisition unit, a feature fusion unit, an image processing unit, a target recognition unit and a display interaction unit;

[0147] The acquisition unit 410 is used to respond to the online teaching request of the teaching user, and acquire the teaching video of the teaching user in the teaching process and the teaching action image corresponding to each frame in the teaching video based on the camera in the visual all-in-one machine;

[0148] A feature fusion unit 420 is used to perform multi-scale feature fusion on the target features of the teaching action image using a feature pyramid network to obtain feature maps at multiple levels, wherein the feature maps at multiple levels include a first feature map at the target level and second feature maps at other levels;

[0149] An image processing unit 430 is used to perform a sparse processing on the first feature map by using a multi-scale hole convolution mask to obtain a sparse feature map;

[0150] The target recognition unit 440 is used to perform target recognition on the second feature map and the sparse feature map to obtain a user dynamic image corresponding to the teaching action image;

[0151] The display interaction unit 450 is used to display the user's dynamic image and the teaching video on the visual interaction all-in-one machine, wherein the user's dynamic image moves in rhythm with the playing of the teaching video.

[0152] The visual interactive all-in-one machine provided by the present invention can display the user's dynamic image and the teaching video together on the visual interactive all-in-one machine, and the user's dynamic image moves rhythmically with the playing of the teaching video, thereby enhancing the interactivity between the teaching user and the students, increasing the fun of teaching, and improving the students' learning enthusiasm, thereby improving the teaching effect.

[0153] See also Figure 5 , Figure 5 is a structural diagram of an electronic device provided by the present invention. An embodiment of the present invention provides an electronic device 500, including a memory 510, a processor 520, and a computer program 511 stored in the memory 510 and executable on the processor 520. When the processor 520 executes the computer program 511, the following steps are implemented:

[0154] In response to an online teaching request from a teaching user, obtaining a teaching video of the teaching user during the teaching process and a teaching action image corresponding to each frame in the teaching video based on a camera in the visual all-in-one machine;

[0155] Using a feature pyramid network to perform multi-scale feature fusion on the target features of the teaching action image to obtain feature maps at multiple levels, wherein the feature maps at multiple levels include a first feature map at the target level and second feature maps at other levels;

[0156] Using a multi-scale atrous convolution mask to perform sparse processing on the first feature map to obtain a sparse feature map;

[0157] Performing target recognition on the second feature map and the sparse feature map to obtain a user dynamic image corresponding to the teaching action image;

[0158] The user dynamic image and the teaching video are displayed on the visual interactive all-in-one machine, wherein the user dynamic image moves in rhythm with the playing of the teaching video.

[0159] Through the description of the above implementation methods, those skilled in the art can clearly understand that each implementation method can be implemented by means of software plus a necessary general hardware platform, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solution is essentially or the part that contributes to the prior art can be embodied in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a disk, an optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods of each embodiment or some parts of the embodiment.

[0160] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. An all-in-one machine interaction method based on motion capture, characterized in that: Applied to a visual interactive all-in-one machine, the visual interactive all-in-one machine is equipped with a fixed camera, and the all-in-one machine interaction method based on motion capture includes: In response to an online teaching request from a teaching user, obtaining a teaching video of the teaching user during the teaching process and a teaching action image corresponding to each frame in the teaching video based on a camera in the visual all-in-one machine; Using a feature pyramid network to perform multi-scale feature fusion on the target features of the teaching action image to obtain feature maps at multiple levels, wherein the feature maps at multiple levels include a first feature map at the target level and second feature maps at other levels; Using a multi-scale atrous convolution mask to perform sparse processing on the first feature map to obtain a sparse feature map; Performing target recognition on the second feature map and the sparse feature map to obtain a user dynamic image corresponding to the teaching action image; The user dynamic image and the teaching video are displayed on the visual interactive all-in-one machine, wherein the user dynamic image moves in rhythm with the playing of the teaching video.

2. The all-in-one machine interaction method based on motion capture according to claim 1, characterized in that: The multi-scale hole convolution mask includes: a first convolution layer and a first output layer; the multi-scale hole convolution mask is used to perform sparse processing on the first feature map to obtain a sparse feature map, including: Processing the first feature map using the first convolutional layer to obtain a sparse mask vector; The first output layer is used to perform sparse processing on the first feature map based on the sparse mask vector to obtain the sparse feature map.

3. The all-in-one machine interaction method based on motion capture according to claim 2, characterized in that: The first convolution layer includes: a plurality of dilated convolution kernels with different dilation rates, at least one channel compression convolution layer, and a second output layer; the first feature map is processed by the first convolution layer to obtain a sparse mask vector, including: Extracting initial information of different scales from the first feature map using the plurality of dilated convolution kernels with different dilation rates; Using the at least one channel compression convolution layer to perform channel compression processing on the initial information of different scales to obtain target information; The second output layer is used to determine the sparse mask vector based on the target information.

4. The all-in-one machine interaction method based on motion capture according to claim 1, characterized in that: The visual interactive all-in-one machine also includes a base, and the all-in-one machine interaction method based on motion capture also includes: Controlling the visual interactive all-in-one machine to shoot the calibration object through the camera to obtain a first image set; the first image set includes images of the calibration object at different shooting distances and / or different shooting angles; Calibrate the pitch angle of the camera based on the first image set, and determine a calibration result of the pitch angle of the camera; Controlling the visual interactive integrated machine to move to the front of the calibration object; Controlling the chassis to rotate, and photographing the calibration object through the camera to obtain a second image set; the second image set includes images of the calibration object photographed at different rotation angles; Based on the second image set and the rotation angle information of the chassis, the relative position of the camera and the chassis is calibrated to determine the position calibration result of the camera.

5. The all-in-one machine interaction method based on motion capture according to claim 4, characterized in that: The rotation angle information includes the rotation angle corresponding to each of the calibration object images in the second image set; The step of calibrating the relative position of the camera and the chassis based on the second image set and the rotation angle information of the chassis, and determining the position calibration result of the camera, comprises: Acquire camera internal reference data and calibration object data of the camera; Based on the camera internal parameter data, the calibration object data, each calibration object image in the second image set, and the rotation angle corresponding to each calibration object image in the second image set, calibrate the relative position of the camera and the chassis to determine a plurality of position deviation values; Based on the multiple position deviation values, a weighted aggregation calculation is performed to obtain the position calibration result of the camera.

6. The all-in-one machine interaction method based on motion capture according to claim 5, characterized in that: The method of calibrating the relative position of the camera and the chassis based on the camera internal parameter data, the calibration object data, each calibration object image in the second image set, and the rotation angle corresponding to each calibration object image in the second image set to determine a plurality of position deviation values ​​includes: Select any two calibration object images from the second image set as the first calibration object image and the second calibration object image respectively; Based on the camera intrinsic parameter data and the calibration object data, respectively determine a first position vector and a second position vector; the first position vector is a position vector between the calibration object and the camera in the first calibration object image, and the second position vector is a position vector between the calibration object and the camera in the second calibration object image; Determine a distance between a first position and a second position based on the first position vector and the second position vector; the first position is a position of the camera when the camera takes the first calibration object image, and the second position is a position of the camera when the camera takes the second calibration object image; Based on the planar geometric relationship between the first position, the second position and the center point of the chassis, the rotation angle corresponding to the first calibration object image and the rotation angle corresponding to the second calibration object image, the relative position of the camera and the chassis is calibrated to calculate the position deviation value; Return to the step of selecting any two of the calibration object images in the second image set until a plurality of the position deviation values ​​are calculated.

7. The all-in-one machine interaction method based on motion capture according to claim 5, characterized in that: The step of performing weighted aggregation calculation based on the plurality of position deviation values ​​to obtain the position calibration result of the camera includes: Selecting one of the position deviation values ​​as reference data, and using the remaining position deviation values ​​as comparison data; respectively determining the distance difference between each of the comparison data and the reference data; Comparing each of the distance differences with a first preset threshold value to obtain a plurality of comparison results; determining an error count of the reference data based on each of the comparison results; Returning to the step of selecting one of the position deviation values ​​as reference data and using the remaining position deviation values ​​as comparison data, until error counts corresponding to all the position deviation values ​​are obtained; Based on the error count corresponding to each of the position deviation values, abnormal data is eliminated to obtain a plurality of normal position deviation values; Based on the multiple normal position deviation values, a weighted aggregation calculation is performed to obtain the position calibration result of the camera.

8. The all-in-one machine interaction method based on motion capture according to claim 4, characterized in that: The step of calibrating the pitch angle of the camera based on the first image set and determining the pitch angle calibration result of the camera includes: Acquire camera external parameter data of the camera, camera internal parameter data of the camera, and calibration object data; Calibrate the pitch angle of the camera based on the camera external parameter data, the camera internal parameter data, the calibration object data and the first image set to determine a plurality of pitch angles; A weighted aggregation calculation is performed based on the multiple pitch angles to obtain a pitch angle calibration result of the camera.

9. The all-in-one machine interaction method based on motion capture according to claim 8, characterized in that: The step of performing weighted aggregation calculation based on the plurality of pitch angles to obtain a pitch angle calibration result of the camera includes: Selecting one of the pitch angles as reference data, and using the remaining pitch angles as comparison data; respectively determining the angle difference between each of the comparison data and the reference data; Comparing each of the angle differences with a second preset threshold value to obtain a plurality of comparison results; determining an error count of the reference data based on each of the comparison results; Returning to the step of selecting one of the pitch angles as reference data and using the remaining pitch angles as comparison data, until error counts corresponding to all the pitch angles are obtained; Eliminate abnormal data based on the error counts corresponding to the pitch angles to obtain a plurality of normal pitch angles; Based on the multiple normal pitch angles, a weighted aggregation calculation is performed to obtain a pitch angle calibration result of the camera.

10. A visual interactive all-in-one machine, characterized in that: The visual interactive all-in-one machine is equipped with a fixed camera and a chassis, and the visual interactive all-in-one machine includes an acquisition unit, a feature fusion unit, an image processing unit, a target recognition unit and a display interaction unit; The acquisition unit is used to respond to the online teaching request of the teaching user, and acquire the teaching video of the teaching user in the teaching process and the teaching action image corresponding to each frame in the teaching video based on the camera in the visual all-in-one machine; The feature fusion unit is used to perform multi-scale feature fusion on the target features of the teaching action image using a feature pyramid network to obtain feature maps of multiple levels, wherein the feature maps of the multiple levels include a first feature map of the target level and second feature maps of other levels; The image processing unit is used to perform sparse processing on the first feature map by using a multi-scale hole convolution mask to obtain a sparse feature map; The target recognition unit is used to perform target recognition on the second feature map and the sparse feature map to obtain a user dynamic image corresponding to the teaching action image; The display interaction unit is used to display the user dynamic image and the teaching video on the visual interaction all-in-one machine, wherein the user dynamic image moves in rhythm with the playing of the teaching video.

Images