Holographic naked eye teaching experiment module somatosensory interaction method and system

Abstract

The invention relates to the technical field of somatosensory interaction, and discloses a holographic naked eye teaching experiment module somatosensory interaction method and system, and the system comprises a virtual environment scene construction module, a holographic naked eye scene display module, a somatosensory behavior feature recognition module, a somatosensory interaction behavior recognition module, an interaction command conversion module, and a multi-mode feedback module. A depth coordinate system is established through a somatosensory camera to perform joint point recognition on a user and obtain a user behavior depth image, user behavior feature extraction is performed based on the user behavior depth image, a tracking algorithm is utilized to perform real-time tracking on the user behavior features, and a somatosensory interaction behavior recognition result is obtained through analysis. The somatosensory interaction behavior is converted into the corresponding operation command based on the somatosensory interaction behavior recognition result, accurate recognition and quick response to the user behavior are achieved, and the accuracy and efficiency of somatosensory interaction are effectively improved.

Description

Technical Field

[0001] The present invention relates to the field of somatosensory interaction technology, and more specifically to a somatosensory interaction method and system for a holographic naked-eye teaching experiment cabin. Background Art

[0002] Holographic naked-eye technology is a technology that can achieve three-dimensional visual effects without the aid of glasses, helmets or other auxiliary equipment. Holographic technology generates a hologram by recording the amplitude and phase information of light waves reflected or transmitted by an object. When the hologram is illuminated by a specific light source, the three-dimensional image of the object can be reproduced. Naked-eye display technology uses the parallax characteristics of the human eye to project images of different perspectives to the left and right eyes respectively through optical elements such as microlens arrays, gratings or spatial light modulators, thereby forming stereoscopic vision.

[0003] Traditional experimental teaching methods are often limited by physical space and experimental equipment, and it is difficult to meet large-scale and diversified teaching needs. Holographic naked-eye technology has brought revolutionary changes to the teaching experimental cabin. Holographic naked-eye technology allows students to intuitively observe the three-dimensional virtual environment scene without wearing any auxiliary equipment, greatly enhancing the immersion and realism of the experiment; at the same time, the application of somatosensory interaction technology allows students to interact with the virtual environment through natural body movements, further enhancing the interactivity and fun of the experiment.

[0004] However, the combination of holographic naked-eye technology and somatosensory interaction technology and application in teaching experimental cabins still faces some technical challenges, such as insufficient realism and interactivity of virtual environment scenes, and inaccurate somatosensory behavior recognition and real-time interactive command conversion. These problems limit the application effect and user experience of the holographic naked-eye teaching experimental cabin. Summary of the invention

[0005] In order to overcome the above-mentioned defects of the prior art, the present invention provides a holographic naked-eye teaching experiment cabin somatosensory interaction system to solve the problems existing in the above-mentioned background technology.

[0006] The present invention provides the following technical solutions: a holographic naked-eye teaching experiment cabin somatosensory interaction system, comprising: a virtual environment scene construction module, a holographic naked-eye scene display module, a somatosensory behavior feature recognition module, a somatosensory interaction behavior recognition module, an interactive command conversion module and a multimodal feedback module;

[0007] The virtual scene equipment deployment module uses a somatosensory camera and a high-definition camera to build a three-dimensional virtual environment scene according to the teaching content;

[0008] The holographic naked-eye scene display module uses light field display technology to present a stereoscopic visual effect of a three-dimensional virtual environment scene;

[0009] The somatosensory behavior feature recognition module includes an image acquisition unit and a feature extraction unit, which acquires the user behavior depth image through a somatosensory camera and extracts the user behavior features based on the user behavior depth image;

[0010] The somatosensory interaction behavior recognition module uses a tracking algorithm to track the user behavior features in real time and analyzes to obtain the somatosensory interaction behavior recognition result;

[0011] The interaction command conversion module converts the somatosensory interaction behavior into a corresponding operation command based on the somatosensory interaction behavior recognition result to achieve the scene control of the three-dimensional virtual environment;

[0012] The multimodal feedback module provides real-time feedback to the terminal according to the somatosensory interaction behavior recognition result and the interaction command operation result of the user.

[0013] Preferably, the virtual scene device deployment module deploys 6 high-definition cameras and 4 somatosensory cameras within a 50-square-meter space, supports 30 students to interact simultaneously. The deployed somatosensory cameras are diagonally distributed, and the deployed high-definition cameras are arranged in a triangular layout and aligned with the teaching area for environmental background collection;

[0014] The somatosensory camera includes three lenses. The middle lens is an RGB color camera, and the left and right lenses are an infrared laser emitter and a 3D depth sensor composed of an infrared camera respectively.

[0015] Preferably, the holographic naked-eye scene display module controls the light emission direction by installing a holographic functional screen based on the nano-photonic crystal structure using a speckle hologram, and precisely encodes and projects the light distribution of the three-dimensional scene through a spatial light modulator. The light distribution includes the light direction, light intensity, and light color.

[0016] Preferably, the somatosensory behavior feature recognition module extracts behavior features by real-time capturing human postures and action trajectories combined with an AI algorithm, including an image acquisition unit and a feature extraction unit. The specific content is as follows:

[0017] Image acquisition unit: Establish a depth coordinate system through a somatosensory camera to identify joint points of the user and acquire the user behavior depth image;

[0018] Feature extraction unit: Extract the user behavior features based on the user behavior depth image using a preset pixel value and calculate the user behavior feature value.

[0019] Preferably, the specific content of the image acquisition unit acquiring the user behavior depth image through a somatosensory camera is as follows:

[0020] Step S1: Identify the joint points of the user through a somatosensory camera, obtaining N joint points of the user, where i = 1, 2, 3, ..., N, and i represents the number of each joint point, and N represents the total number of the user's joint points obtained. Taking the perspective center of the infrared camera as the origin of the depth coordinate system, establish a depth coordinate system, where the X-axis is the connection direction between the infrared camera and the infrared laser emitter;

[0021] Step S2: When the user is at the initial position, the infrared camera captures and records the light spot reflected by the user. When the user moves, the infrared camera captures the light spot reflected by the user again. Calculate the distance between the user and the origin of the depth coordinate system in the spatial target based on the offset distance of the light spot reflected by the target captured by the infrared camera when the user moves. The calculation formula is: where L represents the distance between the user and the origin of the depth coordinate system in the spatial target, l0 represents the distance between the user and the somatosensory camera in the plane, f represents the focal length of the infrared camera, l represents the length between the infrared camera and the infrared laser emitter, and d represents the offset distance of the light spot reflected by the target captured by the infrared camera when the user moves;

[0022] Step S3: Obtain the coordinates of each joint point in the user image through the somatosensory camera as (x i , y i ). The origin coordinates of the depth coordinate system are (x0, y0). Combine the distance between the user and the origin of the depth coordinate system in the spatial target to calculate the plane coordinates of each joint point in the user image in the depth coordinate system. The calculation formula is:

[0023]

[0024] where (X i , Y i ) are the plane coordinates of the joint points in the user image in the depth coordinate system, η x represents the lens lateral distortion coefficient, and η y represents the lens longitudinal distortion coefficient;

[0025] Step S4: Connect the plane coordinates of each joint point in the user image in the depth coordinate system in sequence to obtain the user behavior depth image.

[0026] Preferably, the calculation formula for extracting the user behavior feature value based on the user behavior depth image to calculate the user behavior feature value is: where f(r) represents the user behavior feature value, r represents the preset pixel value, D(r) represents the depth value of the user behavior depth image at the preset pixel value, and α and β represent the feature parameters of the user behavior depth image.

[0027] Preferably, the somatosensory interaction behavior recognition module uses a tracking algorithm to track the user's behavior characteristics in real time, and the specific content of the somatosensory interaction behavior recognition result obtained by analysis is as follows:

[0028] Obtain the frame-by-frame images of the user's somatosensory interaction behavior through a somatosensory camera, and the user behavior characteristic value of each frame-by-frame image is f m (r), m = 1, 2, 3,..., M, where M represents the total number of frames of the frame-by-frame images of the user's somatosensory interaction behavior, and m represents the number of each frame-by-frame image;

[0029] Use the tracking algorithm to calculate the change value of the user behavior characteristic value of each frame-by-frame image to track the user behavior characteristics in real time. The calculation formula is: U m = f m (r) - f m-1 (r), m = 2, 3,..., M, where U m represents the change value of the user behavior characteristic value of each frame-by-frame image;

[0030] Based on a preset range of multiple somatosensory interaction behavior recognition thresholds, perform somatosensory interaction behavior recognition based on the change value of the user behavior characteristic value of each frame-by-frame image: match the threshold range of the change value of the user behavior characteristic value of the frame-by-frame image, and transmit the matching threshold range result to the interaction command conversion module.

[0031] Preferably, the interaction command conversion module receives the matching threshold range result transmitted by the somatosensory interaction behavior recognition module, and converts the somatosensory interaction behavior into a corresponding operation command according to the somatosensory interaction behavior preset for each threshold range.

[0032] Preferably, the multi-modal feedback module monitors in real time whether the somatosensory interaction behavior recognition result and the interaction command operation result of the user match through a high-definition camera, and provides real-time feedback to the terminal.

[0033] A somatosensory interaction method for a holographic naked-eye teaching experiment cabin includes the following steps:

[0034] Step S01: Construct a three-dimensional virtual environment scene according to the teaching content by using a somatosensory camera and a high-definition camera;

[0035] Step S02: Use the light field display technology to present the stereoscopic visual effect of the three-dimensional virtual environment scene;

[0036] Step S03: Obtain the user behavior depth image through the somatosensory camera, and extract the user behavior characteristics based on the user behavior depth image;

[0037] Step S04: Use the tracking algorithm to track the user behavior characteristics in real time, and analyze and obtain the somatosensory interaction behavior recognition result;

[0038] Step S05: Convert the somatosensory interaction behavior into a corresponding operation command based on the somatosensory interaction behavior recognition result, and realize the scene control of the three-dimensional virtual environment;

[0039] Step S06: Provide real-time feedback to the terminal according to the somatosensory interaction behavior recognition result of the user and the operation result of the interaction command.

[0040] Technical effects and advantages of the present invention:

[0041] The present invention is provided with a virtual environment scene construction module, a holographic naked-eye scene display module, a somatosensory behavior feature recognition module, a somatosensory interaction behavior recognition module, an interaction command conversion module, and a multimodal feedback module. By establishing a depth coordinate system through a somatosensory camera to identify the joint points of the user and obtain the user behavior depth image, the two-dimensional coordinates in the depth image are directly used for three-dimensional pose estimation to identify complex actions, improving the accuracy of action recognition while simplifying the calculation amount;

[0042] Extract user behavior features based on the user behavior depth image, use a tracking algorithm to track the user behavior features in real time, analyze and obtain the somatosensory interaction behavior recognition result, capture the dynamic behavior pattern through the joint point movement trajectory of consecutive frames, ensure feature continuity, and thus improve the stability and reliability of behavior recognition. Description of the drawings

[0043] Figure 1 It is a schematic structural diagram of a somatosensory interaction system for a holographic naked-eye teaching experimental cabin.

[0044] Figure 2 It is a schematic flow diagram of a somatosensory interaction method for a holographic naked-eye teaching experimental cabin. Detailed implementation manners

[0045] Next, the technical solutions in the present invention will be clearly and completely described in conjunction with the drawings in the present invention. In addition, the forms of each structure described in the following embodiments are only examples, and a somatosensory interaction method and system for a holographic naked-eye teaching experimental cabin involved in the present invention are not limited to the structures described in the following embodiments. All other embodiments obtained by those of ordinary skill in the art without creative efforts fall within the scope of protection of the present invention.

[0046] As Figure 1 shown, the present invention provides a somatosensory interaction system for a holographic naked-eye teaching experimental cabin, including: a virtual environment scene construction module, a holographic naked-eye scene display module, a somatosensory behavior feature recognition module, a somatosensory interaction behavior recognition module, an interaction command conversion module, and a multimodal feedback module;

[0047] The virtual scene device deployment module constructs a three-dimensional virtual environment scene according to the teaching content by using a somatosensory camera and a high-definition camera;

[0048] The holographic naked-eye scene display module presents a stereoscopic visual effect of the three-dimensional virtual environment scene by using light field display technology;

[0049] The somatosensory behavior feature recognition module includes an image acquisition unit and a feature extraction unit, which acquires the user behavior depth image through the somatosensory camera and extracts the user behavior features based on the user behavior depth image;

[0050] The somatosensory interaction behavior recognition module uses a tracking algorithm to track the user behavior features in real time and analyzes to obtain the somatosensory interaction behavior recognition result;

[0051] The interaction command conversion module converts the somatosensory interaction behavior into a corresponding operation command based on the somatosensory interaction behavior recognition result, realizing the scene control of the three-dimensional virtual environment;

[0052] The multimodal feedback module provides real-time feedback to the terminal according to the somatosensory interaction behavior recognition result of the user and the operation result of the interaction command.

[0053] In this embodiment, it should be specifically noted that the virtual scene device deployment module deploys 6 high-definition cameras and 4 somatosensory cameras within a 50㎡ space, supporting 30 students to interact simultaneously. The deployed somatosensory cameras are diagonally distributed, and the deployed high-definition cameras are aligned with the teaching area in a triangular layout for environmental background collection. The number of the high-definition cameras and somatosensory cameras changes with the size of the space. The larger the space, the more the number of the deployed high-definition cameras and somatosensory cameras; the smaller the space, the fewer the number of the deployed high-definition cameras and somatosensory cameras;

[0054] The somatosensory camera includes three lenses. The middle lens is an RGB color camera, and the lenses on the left and right are an infrared laser emitter and a 3D depth sensor composed of an infrared camera respectively. It can obtain the depth map of the current image and calculate the depth information of the feature points in the image, and can generate a high-density point cloud, which can greatly improve the running time of the system. The somatosensory camera has functions such as human body dynamic capture, human body recognition, face recognition, and voice recognition. Therefore, the somatosensory camera can achieve natural and efficient human-computer interaction.

[0055] In this embodiment, it should be specifically noted that the holographic naked-eye scene display module controls the light emission direction by installing a holographic functional screen based on the nano-photonic crystal structure and uses a spatial light modulator to accurately encode and project the light distribution of the three-dimensional scene. The light distribution includes light direction, light intensity, and light color;

[0056] The single display can provide dozens to hundreds of viewpoints. When the user views from different angles, the two eyes receive light from different perspectives, forming a continuous stereoscopic perception.

[0057] In this embodiment, it should be specifically noted that the somatosensory behavior feature recognition module extracts behavior features by capturing the human body posture and movement trajectory in real time and combining with AI algorithms, including an image acquisition unit and a feature extraction unit. The specific content is as follows:

[0058] Image acquisition unit: Establish a depth coordinate system through a somatosensory camera to identify joint points of the user and obtain the user's behavior depth image;

[0059] Feature extraction unit: Based on the user's behavior depth image, use preset pixel values to extract user behavior features and calculate the user behavior feature values.

[0060] In this embodiment, it should be specifically noted that the specific content of the image acquisition unit obtaining the user's behavior depth image through a somatosensory camera is as follows:

[0061] Step S1: Identify the joint points of the user through a somatosensory camera, obtain N joint points of the user, where i = 1, 2, 3,..., N, and i represents the number of each joint point, and N represents the total number of user joint points obtained. Take the perspective center of the infrared camera as the origin of the depth coordinate system and establish a depth coordinate system, where the X-axis is the connection direction between the infrared camera and the infrared laser emitter;

[0062] Step S2: When the user is in the initial position, the infrared camera captures and records the light spot reflected by the user. When the user moves, the infrared camera captures the light spot reflected by the user again. Calculate the distance between the user and the origin of the depth coordinate system in the space target based on the offset distance of the light spot reflected by the target captured by the infrared camera when the user moves. The calculation formula is: where L represents the distance between the user and the origin of the depth coordinate system in the space target, l0 represents the distance between the user and the somatosensory camera in the plane, f represents the focal length of the infrared camera, l represents the length between the infrared camera and the infrared laser emitter, and d represents the offset distance of the light spot reflected by the target captured by the infrared camera when the user moves. When the user is in the initial position, the infrared camera captures and records the light spot reflected by the user, and the coordinates of the center point of the light spot are (1, 2). When the user moves, the coordinates of the center point of the light spot reflected by the user captured by the infrared camera again are (4, 6). Then the offset distance of the light spot reflected by the target captured by the infrared camera when the user moves

[0063] Step S3: Obtain the coordinates of each joint point in the user image through a somatosensory camera as (x i , yi ) The origin coordinates of the depth coordinate system are (x0, y0). The planar coordinates of each joint point in the user image in the depth coordinate system are calculated in combination with the distance between the user and the origin of the depth coordinate system in the spatial target. The calculation formula is as follows:

[0064]

[0065] Where (X i , Y i ) are the planar coordinates of the joint point in the user image in the depth coordinate system, and η x represents the lateral distortion coefficient of the lens, and η y represents the longitudinal distortion coefficient of the lens;

[0066] Step S4: Connect the planar coordinates of each joint point in the user image in the depth coordinate system in sequence to obtain the user behavior depth image.

[0067] In this embodiment, it should be specifically noted that the calculation formula for extracting the user behavior characteristics based on the user behavior depth image and calculating the user behavior characteristic value is: Where f(r) represents the user behavior characteristic value, r represents the preset pixel value, D(r) represents the depth value of the user behavior depth image at the preset pixel value, and α and β represent the characteristic parameters of the user behavior depth image;

[0068] Characteristic parameters: α = 2, β = 1, depth value function: D(r) = 0.5r + 1. When r = 10, D(r) = 0.5r + 1 = 6;

[0069] Then

[0070] In this embodiment, it should be specifically noted that the somatosensory interaction behavior recognition module uses a tracking algorithm to track the user behavior characteristics in real time, and the specific content of the analysis and obtained somatosensory interaction behavior recognition result is as follows:

[0071] The somatosensory interaction behavior frame images of the user are obtained through the somatosensory camera, and the user behavior characteristic value of each frame image is f m (r), m = 1, 2, 3,..., M, where M represents the total number of frames of the somatosensory interaction behavior frame images of the user obtained, and m represents the number of each frame image;

[0072] The change value of the user behavior characteristic value of each frame image is calculated using the tracking algorithm to track the user behavior characteristics in real time. The calculation formula is: U m = f m (r) - f m-1 (r), m = 2, 3,..., M, where U mRepresents the change value of the user behavior feature value of each sub-frame image;

[0073] Based on a plurality of preset somatosensory interaction behavior recognition threshold ranges, perform somatosensory interaction behavior recognition based on the change value of the user behavior feature value of each sub-frame image: match the change value of the user behavior feature value of the sub-frame image with the threshold range, and transmit the matching threshold range result to the interaction command conversion module.

[0074] In this embodiment, it should be specifically noted that the interaction command conversion module receives the matching threshold range result transmitted by the somatosensory interaction behavior recognition module, and converts the somatosensory interaction behavior into a corresponding operation command according to the somatosensory interaction behavior preset for each threshold range.

[0075] In this embodiment, it should be specifically noted that the multi-modal feedback module monitors in real time whether the somatosensory interaction behavior recognition result of the user matches the interaction command operation result through a high-definition camera, and provides real-time feedback to the terminal. When it is detected that the somatosensory interaction behavior recognition result of the user does not match the interaction command operation result, the error operation area is displayed in the three-dimensional virtual environment scene through the holographic naked-eye scene display module. At the same time, an error prompt sound is emitted through the sound prompt device to remind the user to perform a correction operation. In addition, the multi-modal feedback module gives corresponding scores and feedback suggestions according to the user's operation performance to help the user improve the operation skills and experimental efficiency.

[0076] As Figure 2 shown, in this embodiment, it should be specifically noted that a somatosensory interaction method for a holographic naked-eye teaching experiment cabin includes the following steps:

[0077] Step S01: Construct a three-dimensional virtual environment scene according to the teaching content by using a somatosensory camera and a high-definition camera;

[0078] Step S02: Present the stereoscopic visual effect of the three-dimensional virtual environment scene by using the light field display technology;

[0079] Step S03: Obtain the user behavior depth image through the somatosensory camera, and perform user behavior feature extraction based on the user behavior depth image;

[0080] Step S04: Use the tracking algorithm to perform real-time tracking on the user behavior features, and analyze and obtain the somatosensory interaction behavior recognition result;

[0081] Step S05: Convert the somatosensory interaction behavior into a corresponding operation command based on the somatosensory interaction behavior recognition result, and realize the scene control of the three-dimensional virtual environment;

[0082] Step S06: Provide real-time feedback to the terminal according to the somatosensory interaction behavior recognition result and the interaction command operation result of the user.

[0083] In this embodiment, it should be specifically noted that the main difference between this embodiment and the prior art is that this embodiment is provided with a virtual environment scene construction module, a holographic naked-eye scene display module, a somatosensory behavior feature recognition module, a somatosensory interaction behavior recognition module, an interaction command conversion module, and a multimodal feedback module. A depth coordinate system is established through a somatosensory camera to identify the joint points of the user and obtain the user behavior depth image. The two-dimensional coordinates in the depth image are directly used for three-dimensional pose estimation to recognize complex actions, improving the accuracy of action recognition while simplifying the calculation amount.

[0084] Based on the user behavior depth image, user behavior features are extracted, and a tracking algorithm is used to track the user behavior features in real time. The somatosensory interaction behavior recognition result is obtained through analysis. The dynamic behavior pattern is captured through the joint point movement trajectory of consecutive frames to ensure feature continuity, thereby improving the stability and reliability of behavior recognition.

[0085] Finally: The above are only the preferred embodiments of the present invention and are not used to limit the present invention. Any modifications, equivalent replacements, improvements, etc. made within the spirit and principle of the present invention shall be included within the protection scope of the present invention.

[0086] The above is only the specific implementation manner of the present application, but the protection scope of the present application is not limited thereto. Any person skilled in the art within the technical scope disclosed in the present application can easily think of changes or replacements, which should all be covered within the protection scope of the present application. Therefore, the protection scope of the present application shall be subject to the protection scope of the claims.

Claims

1. A holographic naked-eye teaching experimental cabin somatosensory interaction system, characterized in that: Including: A virtual environment scene construction module, a holographic naked-eye scene display module, a somatosensory behavior feature recognition module, a somatosensory interaction behavior recognition module, an interaction command conversion module, and a multimodal feedback module; The virtual scene device deployment module constructs a three-dimensional virtual environment scene using a somatosensory camera and a high-definition camera according to the teaching content; The holographic naked-eye scene display module presents the stereoscopic visual effect of the three-dimensional virtual environment scene using light field display technology; The somatosensory behavior feature recognition module includes an image acquisition unit and a feature extraction unit, which acquires the user behavior depth image through the somatosensory camera and extracts the user behavior features based on the user behavior depth image; The somatosensory interaction behavior recognition module uses a tracking algorithm to track the user behavior features in real time and analyzes to obtain the somatosensory interaction behavior recognition result; The interaction command conversion module converts the somatosensory interaction behavior into a corresponding operation command based on the somatosensory interaction behavior recognition result to achieve the scene control of the three-dimensional virtual environment; The multimodal feedback module provides real-time feedback to the terminal according to the somatosensory interaction behavior recognition result and the interaction command operation result of the user.

2. The somatosensory interaction system for a holographic naked-eye teaching experiment cabin according to claim 1, wherein: The virtual scene device deployment module deploys 6 high-definition cameras and 4 somatosensory cameras within a 50㎡ space, supporting 30 students to interact simultaneously. The deployed somatosensory cameras are diagonally distributed, and the deployed high-definition cameras are aligned with the teaching area in a triangular layout for environmental background collection; The somatosensory camera includes three lenses. The middle lens is an RGB color camera, and the lenses on the left and right are an infrared laser emitter and a 3D depth sensor composed of an infrared camera respectively.

3. The somatosensory interaction system of the holographic naked-eye teaching experiment cabin according to claim 1, characterized in that: The holographic naked-eye scene display module controls the light emission direction by installing a holographic functional screen based on the nano-photonic crystal structure using a speckle hologram, and accurately encodes and projects the light distribution of the three-dimensional scene through a spatial light modulator. The light distribution includes the light direction, light intensity, and light color.

4. The somatosensory interaction system of the holographic naked-eye teaching experiment cabin according to claim 1, wherein: The somatosensory behavior feature recognition module extracts behavior features by real-time capturing the human body posture and movement trajectory in combination with an AI algorithm, including an image acquisition unit and a feature extraction unit. The specific content is as follows: Image acquisition unit: Establish a depth coordinate system through the somatosensory camera to identify the joint points of the user and acquire the user behavior depth image; Feature extraction unit: Extract the user behavior features based on the user behavior depth image using the preset pixel values and calculate the user behavior feature values.

5. The somatosensory interaction system of a holographic naked-eye teaching experimental cabin according to claim 4, wherein: The specific content of the image acquisition unit acquiring the user behavior depth image through the somatosensory camera is as follows: Step S1: Identify the joint points of the user through the somatosensory camera, and acquire N joint points of the user, where i = 1, 2, 3,..., N, and i represents the number of each joint point, and N represents the total number of user joint points acquired. Take the perspective center of the infrared camera as the origin of the depth coordinate system, and establish a depth coordinate system, where the X-axis is the connection direction between the infrared camera and the infrared laser emitter; Step S2: When the user is at the initial position, the infrared camera captures and records the light spot reflected by the user. When the user moves, the infrared camera captures the light spot reflected by the user again. The distance between the user and the origin of the depth coordinate system in the space target is calculated based on the offset distance of the light spot reflected by the target captured by the infrared camera when the user moves. The calculation formula is as follows: Where L represents the distance between the user and the origin of the depth coordinate system in the space target, l0 represents the distance between the user and the somatosensory camera in the plane, f represents the focal length of the infrared camera, l represents the length between the infrared camera and the infrared laser emitter, and d represents the offset distance of the light spot reflected by the target captured by the infrared camera when the user moves; Step S3: Obtain the coordinates of each joint point in the user image as (x i , y i ) through the body sensor camera. The origin coordinates of the depth coordinate system are (x0, y0). Combine the distance between the user and the origin of the depth coordinate system in the spatial target to calculate the plane coordinates of each joint point in the user image. The calculation formula is as follows: where (X i , Y i ) are the planar coordinates of the joint points in the user image in the depth coordinate system, η x represents the lateral lens distortion coefficient, and η y represents the longitudinal lens distortion coefficient; Step S4: Connect the plane coordinates of each joint point in the user image in the depth coordinate system in sequence to obtain the user behavior depth image.

6. The somatosensory interaction system of a holographic naked-eye teaching experiment cabin according to claim 4, wherein: The calculation formula for extracting user behavior features based on the user behavior depth image and calculating the user behavior feature value is as follows: Where f(r) represents the user behavior feature value, r represents the preset pixel value, D(r) represents the depth value of the user behavior depth image at the preset pixel value, and α and β represent the feature parameters of the user behavior depth image.

7. The somatosensory interaction system of the holographic naked-eye teaching experiment cabin according to claim 1, characterized in that: The somatosensory interaction behavior recognition module uses a tracking algorithm to track the user behavior characteristics in real time, and the specific content of the recognized result of the somatosensory interaction behavior is as follows: The somatosensory camera is used to obtain the user's somatosensory interaction behavior frame images, and the user behavior feature value of each frame image is f m (r), m=1, 2, 3, ..., M, where M represents the total number of frames of the framed images of the user's somatosensory interaction behavior, and m represents the number of each framed image; The user behavior characteristics are tracked in real time by using the tracking algorithm to calculate the change value of the user behavior characteristic value of each sub-frame image. The calculation formula is: Um = fm(r) - fm-1(r), where m = 2, 3,..., M. Here, Um represents the change value of the user behavior characteristic value of each sub-frame image; Based on multiple preset somatosensory interaction behavior recognition threshold ranges, the somatosensory interaction behavior is recognized based on the change value of the user behavior characteristic value of each sub-frame image: the change value of the user behavior characteristic value of the sub-frame image is matched with the threshold range, and the matching threshold range result is transmitted to the interaction command conversion module.

8. The somatosensory interaction system of a holographic naked-eye teaching experiment cabin according to claim 1, characterized in that: The interaction command conversion module receives the matching threshold range result transmitted by the somatosensory interaction behavior recognition module, and converts the somatosensory interaction behavior into a corresponding operation command according to the somatosensory interaction behavior preset for each threshold range.

9. The somatosensory interaction system of a holographic naked-eye teaching experiment cabin according to claim 1, characterized in that: The multi-modal feedback module monitors in real time whether the recognized result of the user's somatosensory interaction behavior matches the operation result of the interaction command through a high-definition camera, and provides real-time feedback to the terminal.

10. A somatosensory interaction method for a holographic naked-eye teaching experiment cabin, which is used for a holographic naked-eye teaching experiment cabin somatosensory interaction system described in any one of the above claims 1-9, and is characterized in that: It includes the following steps: Step S01: Construct a three-dimensional virtual environment scene according to the teaching content by using a somatosensory camera and a high-definition camera; Step S02: Adopt a light field display technology to present the stereoscopic visual effect of the three-dimensional virtual environment scene; Step S03: Obtain the user behavior depth image through the somatosensory camera, and extract the user behavior characteristics based on the user behavior depth image; Step S04: Use the tracking algorithm to track the user behavior characteristics in real time, and analyze and obtain the recognized result of the somatosensory interaction behavior; Step S05: Convert the somatosensory interaction behavior into a corresponding operation command based on the recognized result of the somatosensory interaction behavior to realize the scene control of the three-dimensional virtual environment; Step S06: Provide real-time feedback to the terminal according to the recognized result of the user's somatosensory interaction behavior and the operation result of the interaction command.

Images