An adaptive 3D space interaction system and method based on monocular camera

Abstract

The present invention discloses an adaptive 3D space interaction system and method based on a monocular camera. The adaptive 3D space interaction system based on a monocular camera includes: an acquisition module for acquiring an RGB image captured by the monocular camera of a user's movements; an extraction module for extracting 3D human skeleton joints in the RGB image using a 3D human skeleton extraction algorithm; a generation module for dynamically generating an octree interaction cube (AOIC) based on the extracted 3D human skeleton joints, and dividing the AOIC into an interaction space consisting of multiple voxels; a construction module for constructing a virtual interaction space (VIZ), and dividing the VIZ into virtual interaction spaces with an equal number of voxels as the AOIC; a mapping module for constructing a mapping relationship between each voxel in the AOIC and a corresponding voxel in the VIZ; and an interaction control module for controlling the interaction operation of the virtual space in the VIZ through the mapping relationship according to the position of the user's hand joints in the AOIC and gesture control instructions.

Description

Technical Field

[0001] The present invention relates to the technical field of computer vision and human-computer interaction, and in particular to an adaptive 3D space interaction system and method based on a monocular camera. Background Art

[0002] As computer vision and human-computer interaction technologies continue to advance, 3D spatial interaction technology is becoming a key enabler in many fields. However, existing 3D spatial interaction technologies have numerous limitations, making it difficult to meet the growing and diverse interaction needs. Currently, there are three main types of typical interaction technology solutions:

[0003] Solutions of this type, such as the Kinect Physical Interaction Zone, leverage depth information to achieve stable mapping of planar interaction zones. Their advantage lies in their ability to precisely determine the interaction plane using depth data, ensuring interaction stability within a certain range. However, these solutions also have significant limitations. First, they rely heavily on depth sensors, significantly increasing hardware costs and making them difficult to deploy in resource-constrained scenarios such as mobile devices. Second, because their interaction accuracy is globally fixed, they typically only support coarse-grained interactions, failing to meet the demands of fine-grained operations. Furthermore, their interactive virtual space is primarily limited to two-dimensional objects, with limited support for 3D interaction spaces. Furthermore, their 3D human skeleton relies on depth sensors, requiring additional algorithms to ensure that the interaction plane remains aligned with motion occlusion. Even so, significant lag in the interaction plane occurs when the human body moves, severely impacting the interactive experience.

[0004] Interaction solutions based on ray-based human skeleton joints define ray directions using joints like the shoulder and wrist, enabling 3D pointing operations. Their advantage lies in leveraging joint information to achieve a certain degree of 3D pointing interaction. However, they also face several challenges. First, they rely on binoculars or depth cameras to obtain the physical, realistic 3D pose of the joints, which undoubtedly increases hardware complexity and cost. Second, to align physical and virtual space, they rely on external calibration (such as calibration plates), making deployment complex and cumbersome, limiting their widespread adoption. Furthermore, while ray-based solutions support point-and-click operations, complex interactions like grabbing and dragging require a certain amount of waiting time, making the interaction less natural and smooth, and making them difficult to meet the needs of interactive scenarios with high real-time requirements.

[0005] Virtual digital human solutions based on direct mapping of the human skeleton utilize 3D human skeleton models (such as those from Mediapipe, SMPler-X, VIBE, and Frankmocap) to drive the movements of virtual characters. While this approach offers advantages in mimicking the poses of virtual characters through skeletal information, providing users with a certain level of visual feedback, its drawbacks are not negligible. For one thing, this approach lacks a dynamic mapping between physical and virtual space, remaining limited to pose imitation. It cannot achieve precise manipulation of virtual objects, limiting the depth and breadth of interaction. Furthermore, it also requires extrinsic calibration to align the coordinate system, significantly limiting the flexibility and diversity of its application scenarios.

[0006] In general, existing 3D spatial interaction technologies generally rely on depth sensors or multi-target positioning, which leads to high hardware costs, complex deployment, and difficulty in achieving a low-cost, natural and smooth interactive experience. This greatly limits the application and promotion of 3D spatial interaction technology in a wider range of scenarios. There is an urgent need for a new 3D spatial interaction system and method that can overcome these limitations. Summary of the Invention

[0007] The purpose of the present invention is to provide an adaptive 3D space interaction system and method based on a monocular camera to address the shortcomings of the existing technology.

[0008] In order to achieve the above objectives, the present invention adopts the following technical solutions:

[0009] An adaptive 3D spatial interaction system based on a monocular camera, comprising:

[0010] The acquisition module is used to obtain the RGB image of the user's action captured by the monocular camera;

[0011] An extraction module is used to extract 3D human skeleton joints from RGB images using a 3D human skeleton extraction algorithm;

[0012] A generation module is used to dynamically generate an octree interaction cube (AOIC) based on the extracted 3D human skeleton joints and divide the AOIC into an interaction space consisting of multiple voxels.

[0013] A construction module is used to construct a virtual interaction space VIZ and divide the VIZ into virtual interaction spaces with the same number of voxels as the AOIC;

[0014] The mapping module is used to construct the mapping relationship between each voxel in AOIC and the corresponding voxel in VIZ;

[0015] The interactive control module is used to control the interactive operations of the virtual space in VIZ through mapping relationships based on the positions of the user's hand joints in AOIC and gesture control instructions.

[0016] Furthermore, the dynamic generation of the octree interactive cube AOIC in the generation module is to calculate the side length of the AOIC, the position of the AOIC center point and the voxel side length of the AOIC through the joint points of the human skeleton.

[0017] Furthermore, the side length of the AOIC is expressed as:

[0018] AOIC 边长 =min(W scale ×W s ,H scale ×T c ,D scale ×L a );

[0019] Among them, AOIC 边长 Indicates the side length of AOIC; W s Represents the shoulder width in the 3D human skeleton joint; L a Represents the arm length in the 3D human skeleton joint; T c Indicates the height from neck to waist in the 3D human skeleton joint; W scale 、H scale 、D scale Indicates experience value;

[0020] AOIC center point position (X p +J shoulder_x , Y p +J shoulder_y , Z p +0.5×L a ), where (X p ,Y p ,Z p ) represents the position coordinates of the human hip joint in the current camera coordinate system, which is the origin of the human body; J shoulder_x Indicates the offset from the hip joint position to the shoulder joint position in the x-axis direction; J shoulder_y Indicates the offset in the y-axis direction from the hip joint position to the shoulder joint position;

[0021] The voxel side length of AOIC is expressed as:

[0022] voxel 边长 =AOIC 边长 / step;

[0023] Among them, voxel 边长 Indicates the voxel side length of AOIC; step indicates the number of equal-proportion divisions of the shortest side of AOIC.

[0024] Furthermore, it also includes an AOIC update module, which is used to reconstruct and update the octree structure in the AOIC when it is detected that the movement range of the AOIC center point position is greater than a preset threshold or an instruction to change the VIZ center point by a gesture is received; when it is detected that the movement range of the AOIC center point position is less than a preset threshold, it updates part of the octree structure in the AOIC.

[0025] Furthermore, an adjustment module is included for obtaining the movement speed of the hand joints in the 3D human skeleton joints, and dynamically adjusting the number of octree recursive segmentation times according to the movement speed to change the voxel side length of the AOIC.

[0026] Furthermore, the number of recursive segmentation is expressed as:

[0027] ;

[0028] Among them, n(v hand ) represents the number of recursive segmentation times after dynamic adjustment according to the movement speed of the hand joints; clamp() represents the function that limits the number of recursive segmentation times; v hand Indicates the movement speed of the hand joints; v min and v max Respectively represent the minimum and maximum values ​​of the dynamic range of motion speed; n max Indicates the maximum number of recursive splits.

[0029] Furthermore, the side length of the voxel of the AOIC is expressed as:

[0030] l current (n)=l root / 2 n ;

[0031] Among them, l current (n) represents the AOIC voxel side length corresponding to the current recursive segmentation number n; l root Indicates the initial voxel edge length of the root node.

[0032] Accordingly, an adaptive 3D space interaction method based on a monocular camera is also provided, including:

[0033] S1. Obtain an RGB image of the user's action captured by a monocular camera;

[0034] S2. Use 3D human skeleton extraction algorithm to extract 3D human skeleton joint points from RGB images;

[0035] S3. Dynamically generate an octree interaction cube (AOIC) based on the extracted 3D human skeleton joints, and divide the AOIC into an interaction space consisting of multiple voxels;

[0036] S4. Construct a virtual interaction space VIZ and divide the VIZ into virtual interaction spaces with the same number of voxels as the AOIC;

[0037] S5. Construct a mapping relationship between each voxel in AOIC and the corresponding voxel in VIZ;

[0038] S6. Based on the positions of the user's hand joints in AOIC and the gesture control instructions, the interactive operations in the virtual space in VIZ are controlled through the mapping relationship.

[0039] Furthermore, after step S6, the following steps are further included:

[0040] S7. Obtain the movement speed of the hand joints in the 3D human skeleton joints, and dynamically adjust the number of octree recursive segmentation times according to the movement speed to change the voxel side length of AOIC.

[0041] Furthermore, after step S7, the following steps are further included:

[0042] S8. When it is detected that the movement range of the AOIC center point position is greater than a preset threshold or an instruction to change the VIZ center point is received due to a gesture, the octree structure in the AOIC is reconstructed and updated. When it is detected that the movement range of the AOIC center point position is less than a preset threshold, part of the octree structure in the AOIC is updated.

[0043] Compared with the prior art, the present invention has the following beneficial effects:

[0044] 1. Calibration-free deployment: No external calibration is required, and alignment of physical and virtual spaces is achieved through a dynamic projection matrix, reducing the complexity and cost of system deployment.

[0045] 2. Adaptive Interaction Precision: Dynamically adjusts the number of AOIC recursive segmentations based on the movement speed of the user's hand joints, thereby meeting interaction requirements of varying precision and enabling high-precision micro-operations.

[0046] 3. Expandable virtual interaction space: The AOIC mapping scale automatically adjusts with the virtual camera's viewing angle, supporting infinitely expandable virtual space and overcoming the limitations of traditional methods with a fixed operating range.

[0047] 4. Low-cost compatibility: Requires only a monocular RGB camera and is compatible with consumer devices such as mobile phones and tablets, reducing hardware requirements and costs.

[0048] 5. Support natural 3D space interaction: Users can interact with objects in the virtual space through intuitive hand movements in the AOIC space, supporting more interactive actions and improving the user experience. BRIEF DESCRIPTION OF THE DRAWINGS

[0049] Figure 1 This is a structural diagram of an adaptive 3D spatial interaction system based on a monocular camera provided in Example 1;

[0050] Figure 2 Schematic diagram of an adaptive 3D spatial interaction system based on a monocular camera provided in Example 1;

[0051] Figure 3 This is a schematic diagram of an octree provided in Example 1;

[0052] Figure 4 This is the node adding and deleting update strategy diagram provided in the first embodiment. DETAILED DESCRIPTION

[0053] The following describes the embodiments of the present invention through specific examples. Those skilled in the art can easily understand other advantages and effects of the present invention from the content disclosed in this specification. The present invention can also be implemented or applied through other different specific embodiments. The details in this specification can also be modified or changed based on different viewpoints and applications without departing from the spirit of the present invention. It should be noted that the following embodiments and features in the embodiments can be combined with each other unless they conflict.

[0054] The purpose of the present invention is to provide an adaptive 3D space interaction system and method based on a monocular camera to address the shortcomings of the existing technology.

[0055] Example 1

[0056] This embodiment provides an adaptive 3D space interaction system based on a monocular camera, such as Figure 1 Shown, including:

[0057] An acquisition module 11 is used to acquire an RGB image of the user's action captured by a monocular camera;

[0058] Extraction module 12, for extracting 3D human skeleton joint points from RGB images using a 3D human skeleton extraction algorithm;

[0059] A generation module 13 is used to dynamically generate an octree interaction cube AOIC based on the extracted 3D human skeleton joint points, and divide the AOIC into an interaction space composed of multiple voxels;

[0060] A construction module 14 is used to construct a virtual interaction space VIZ and divide the VIZ into virtual interaction spaces with the same number of voxels as the AOIC;

[0061] A mapping module 15 is used to construct a mapping relationship between each voxel in the AOIC and the corresponding voxel in the VIZ;

[0062] The interactive control module 16 is used to control the interactive operations in the virtual space of VIZ through mapping relationships according to the positions of the user's hand joints in AOIC and gesture control instructions.

[0063] In this embodiment, a computer equipped with an ordinary RGB camera is equipped with a corresponding 3D human skeleton extraction algorithm library (such as SMPL-X, Frankmocap, etc.), and a gesture recognition algorithm and octree-related data structures and operation functions are configured.

[0064] In the acquisition module 11, an RGB image of the user's action captured by the monocular camera is acquired.

[0065] A camera is used to capture the user's RGB image in real time and input it into the system.

[0066] In the extraction module 12, a 3D human skeleton extraction algorithm is used to extract the joint points of the 3D human skeleton in the RGB image.

[0067] Call the 3D human skeleton extraction algorithm (such as SMPL-X, Frankmocap, etc.), process the input RGB image, and calculate the 3D human skeleton joint information, where the joint information includes shoulder width (W s )、Arm length(L a ), height from neck to waist (T c ) and the position of the human hip joint (X p , Y p , Z p ), taking the hip joint point as the origin of the human body, providing basic data support for the subsequent generation of AOIC, such as Figure 2 shown.

[0068] In the generation module 13, an octree interaction cube AOIC is dynamically generated according to the extracted 3D human skeleton joint points, and the AOIC is divided into an interaction space composed of multiple voxels.

[0069] According to the shoulder width of the skeleton (W s )、Arm length(L a ), height from neck to waist (T c ) etc. to generate the corresponding proportion of the adaptive octree interaction cube AOIC (Adaptive-Octree-Interaction-Cube), such as Figure 2 As shown in Figure 2, AOIC is divided into an interaction space consisting of N*N*N coarse-grained voxels according to empirical values.

[0070] Among them, the dynamic generation of octree interactive cube AOIC is to calculate the side length of AOIC, the position of AOIC center point and the voxel side length of AOIC through the joint points of the human skeleton, generate AOIC in real time based on the human body origin, and use octree to store the segmented voxels.

[0071] In this embodiment, the side length of AOIC is calculated as:

[0072] AOIC 边长 =min(W scale ×W s ,H scale ×T c ,D scale ×L a );

[0073] Among them, AOIC 边长 Indicates the side length of AOIC; W s Represents the shoulder width in the 3D human skeleton joint; L a Represents the arm length in the 3D human skeleton joint; T c Indicates the height from neck to waist in the 3D human skeleton joint; W scale 、H scale 、D scale Indicates the experience value that can be adjusted according to actual conditions.

[0074] For example: Assume W scale is 0.8, H scale is 1.2, D scale is 1.0, the shoulder width W s The height from neck to waist is 0.5m c 0.8m, arm length L a is 0.7m, then the side length of AOIC is: 边长 = min(0.8×0.5, 1.2×0.8, 1.0×0.7) = min(0.4, 0.96, 0.7) = 0.4m.

[0075] In this embodiment, the AOIC center point position is specifically: the AOIC center point position (X p +J shoulder_x , Y p +J shoulder_y , Z p +0.5×L a ), updated in real time as the user moves; among them, (X p ,Y p ,Z p) represents the position coordinates of the human hip joint in the current camera coordinate system, which is the origin of the human body; J shoulder_x Indicates the offset from the hip joint position to the shoulder joint position in the x-axis direction; J shoulder_y Indicates the offset in the y-axis direction from the hip joint position to the shoulder joint position.

[0076] For example, assuming the hip joint position (i.e., the origin of the human body) is (1.0, 1.2, 0.5) and the shoulder joint position is (1.1, 1.3, 0.5), then the AOIC center point position is (1.0 + 0.1, 1.2 + 0.1, 0.5 +0.5×0.7) = (1.1, 1.3, 0.85).

[0077] In this embodiment, the shortest side of the AOIC is divided into steps in equal proportion. The step value can be set according to the actual accuracy requirement. Then, the side length of the voxel of the AOIC is calculated, which is expressed as:

[0078] voxel 边长 =AOIC 边长 / step;

[0079] Among them, voxel 边长 Indicates the voxel side length of AOIC; step indicates the number of equal-proportion divisions of the shortest side of AOIC.

[0080] For example, assuming step=10, the voxel side length is: voxel 边长 =0.4 / 10=0.04m. Then, based on the position and size of the AOIC, an octree structure is constructed to divide the AOIC space into 10×10×10 voxels, and the coordinates and related information of each voxel are stored in the octree node.

[0081] In the construction module 14, a virtual interaction space VIZ is constructed and the VIZ is divided into virtual interaction spaces with the same number of voxels as the AOIC.

[0082] The corresponding virtual interaction space VIZ (Virutal-Interaction-Zone) is divided into the same number of virtual interaction space voxels as AOIC according to empirical values. Assume that the size of VIZ is the same as AOIC, that is, a cube with a side length of 0.4m, and divide it into 10×10×10 voxels to ensure that each voxel in AOIC corresponds to a voxel in VIC.

[0083] In the mapping module 15 , a mapping relationship between each voxel in the AOIC and the corresponding voxel in the VIZ is constructed.

[0084] A one-to-one mapping relationship is established between each voxel in the AOIC and the corresponding voxel in the VIZ. This can be achieved by creating an index or coordinate table for each voxel. When an interaction operation is required on a voxel in the AOIC, the corresponding voxel in the VIZ is quickly found using the index or coordinate table, and the corresponding interaction operation is then performed in the VIZ. For example, if the user's hand moves to a voxel in the AOIC (such as the voxel with index 358), the system will use the index or coordinate table to determine the corresponding voxel in the VIZ (also with index 358). Then, based on the user's gesture command (such as a click), the system will perform an operation on the virtual object corresponding to that voxel in the VIZ (such as selecting the virtual object).

[0085] In the interactive control module 16, the interactive operation of the virtual space in VIZ is controlled through a mapping relationship according to the position of the user's hand joints in AOIC and the gesture control instructions.

[0086] In this embodiment, the gesture control instruction is obtained as follows:

[0087] The human skeleton joint information is input into a pre-configured gesture recognition algorithm module. Based on the angles and positions of the hand joints, the module identifies the user's gesture control commands, such as click, grab, and drag. For example, if the five fingers of a hand are detected to be open and the palm is facing the virtual object, it is recognized as a click command; if the four fingers of a hand are detected to be close together and the thumb is bent close to the index finger, it is recognized as a grab command.

[0088] This embodiment further includes an adjustment module for obtaining the movement speed of the hand joints in the 3D human skeleton joints and dynamically adjusting the number of octree recursive segmentation times according to the movement speed to change the voxel side length of the AOIC.

[0089] The number of recursive splits is expressed as:

[0090] ;

[0091] Among them, n(v hand ) represents the number of recursive segmentation times after dynamic adjustment according to the movement speed of the hand joints; clamp() represents the function that limits the number of recursive segmentation times; v hand Indicates the movement speed of the hand joints; v min and v max Respectively represent the minimum and maximum values ​​of the dynamic range of motion speed; n max Indicates the maximum number of recursive splits.

[0092] The voxel side length of AOIC is expressed as:

[0093] l current (n)=l root / 2 n ;

[0094] Constraints:

[0095] l current (n)≥l min , that is, n≤n max ;

[0096] Among them, l current (n) represents the AOIC voxel side length corresponding to the current recursive segmentation number n; l root Indicates the initial voxel edge length of the root node; l min represents the minimum allowed voxel side length; n max =log2(l root / l min ) represents the maximum number of recursive splits, and l root and l min The constraint condition can ensure that the voxel edge length will not be less than the minimum allowed value, thereby ensuring the stability and feasibility of the system while achieving high-precision interaction.

[0097] In this embodiment, a 3D human skeleton extraction algorithm (such as SMPL-X or Frankmocap) is used to extract the position information of the hand joints from a sequence of continuously captured RGB images. This results in a sequence of hand joint positions at consecutive moments. The motion speed of the hand joints is then calculated using methods such as the difference method of consecutive frames or the difference method of sliding averages. This calculated motion speed information is used to dynamically adjust the number of recursive segmentations in the octree, thereby changing the voxel edge length and achieving dynamic adjustment of interaction accuracy. For example, when a slow hand movement is detected, the number of recursive segmentations is increased to improve interaction accuracy; when a fast hand movement is detected, the number of recursive segmentations is reduced or only the root node is retained to reduce computational complexity.

[0098] Assume that the minimum hand speed v is set min is 0.1m / s, the maximum value v max is 0.5m / s, and the initial voxel edge length of the root node is l root is 0.04m, the minimum allowed voxel side length l min is 0.005m. When the hand speed v hand In v min and v max When the hand speed is between , the recursive segmentation times n is calculated according to the formula. For example, if the current hand speed v hand is 0.3m / s, then n(v hand )=clamp(n max ·⌊(0.3-0.1) / (0.5-0.1)⌋,0,n max ).

[0099] Assume n max =log2(l root / l min )=log2(0.04 / 0.005)=log2(8)=3, then:

[0100] , so n(v hand )=clamp(3×0,0,3)=0, that is, no recursive segmentation is performed and the voxel side length remains at the initial 0.04m.

[0101] If the hand speed v hand Reduced to 0.05m / s, then:

[0102] n(v hand )= =clamp(3×(-1),0,3)=clamp(-3,0,3)=0, that is, recursive splitting is still not performed.

[0103] But if higher precision is required at this time, you can adjust v appropriately min and v max The value of , or redefine the calculation formula of the number of recursive segmentation according to the actual application scenario. When the hand speed is slow, the voxels around the arm joints are recursively subdivided to meet the high-precision interaction of the fixed position. For example, when the hand speed is lower than 0.1m / s, the number of recursive segmentation is increased to 2, and the voxel side length becomes l current =0.04 / (2 2 )=0.01m, thus achieving finer interaction accuracy. When the hand movement speed exceeds a certain threshold (such as 0.5m / s), only the root node of the octree is retained, and recursive subdivision is not considered. The voxel side length remains at 0.04m and no further subdivision is performed.

[0104] This embodiment also includes an AOIC update module for reconstructing and updating the octree structure in the AOIC when it is detected that the movement range of the AOIC center point position is greater than a preset threshold or an instruction to change the VIZ center point by a gesture is received; and when it is detected that the movement range of the AOIC center point position is less than a preset threshold, partially updating the octree structure in the AOIC.

[0105] like Figure 3-Figure 4As shown, this embodiment introduces an octree to store voxels, enabling rapid positioning and access. Specifically, when user movement is detected, causing a change in the AOIC center (e.g., a change in the position of a hip joint due to human body movement), or when a gesture changes the center of the VIZ, the octree is determined to determine whether reconstruction is necessary based on the degree of change in the AOIC center and the direction and distance of the VIZ movement. If the range of movement is small, only some nodes in the octree are updated, such as adding new nodes or deleting old nodes, or changing the root node, to accommodate the positional changes of the AOIC and VIZ. This reduces computational complexity and resource consumption. For example, if the AOIC center moves 0.05m in the x-direction, and the range of movement does not exceed twice the voxel side length, the affected nodes are found in the octree, translated 0.05m, and the coordinates and boundaries of their child nodes are recalculated. Simultaneously, based on the new mapping between the AOIC and VIZ, the corresponding voxel information in the virtual interaction space is updated to ensure the accuracy and consistency of the interaction.

[0106] Through the above content, the interactive system and method based on monocular camera and adaptive octree space can achieve multiple advantages such as calibration-free deployment, adaptive interaction accuracy, scalable virtual interaction space and low-cost compatibility, providing users with a more natural and efficient 3D space interaction experience.

[0107] Example 2

[0108] This embodiment provides an adaptive 3D space interaction method based on a monocular camera, including:

[0109] S1. Obtain an RGB image of the user's action captured by a monocular camera;

[0110] S2. Use 3D human skeleton extraction algorithm to extract 3D human skeleton joint points from RGB images;

[0111] S3. Dynamically generate an octree interaction cube (AOIC) based on the extracted 3D human skeleton joints, and divide the AOIC into an interaction space consisting of multiple voxels;

[0112] S4. Construct a virtual interaction space VIZ and divide the VIZ into virtual interaction spaces with the same number of voxels as the AOIC;

[0113] S5. Construct a mapping relationship between each voxel in AOIC and the corresponding voxel in VIZ;

[0114] S6. Based on the positions of the user's hand joints in AOIC and the gesture control instructions, the interactive operations in the virtual space in VIZ are controlled through the mapping relationship.

[0115] Furthermore, after step S6, the following steps are further included:

[0116] S7. Obtain the movement speed of the hand joints in the 3D human skeleton joints, and dynamically adjust the number of octree recursive segmentation times according to the movement speed to change the voxel side length of AOIC.

[0117] Furthermore, after step S7, the following steps are further included:

[0118] S8. When it is detected that the movement range of the AOIC center point position is greater than a preset threshold or an instruction to change the VIZ center point is received due to a gesture, the octree structure in the AOIC is reconstructed and updated. When it is detected that the movement range of the AOIC center point position is less than a preset threshold, part of the octree structure in the AOIC is updated.

[0119] It should be noted that the adaptive 3D space interaction method based on a monocular camera provided in this embodiment is similar to that in the first embodiment, and will not be described in detail here.

[0120] Note that the above are only preferred embodiments of the present invention and the technical principles employed. Those skilled in the art will understand that the present invention is not limited to the specific embodiments described herein, and that various obvious changes, readjustments, and substitutions can be made by those skilled in the art without departing from the scope of protection of the present invention. Therefore, although the present invention has been described in detail through the above embodiments, the present invention is not limited to the above embodiments and may include many other equivalent embodiments without departing from the concept of the present invention. The scope of the present invention is determined by the scope of the appended claims.

Claims

1. An adaptive 3D spatial interaction system based on a monocular camera, characterized in that: include: The acquisition module is used to obtain the RGB image of the user's action captured by the monocular camera; An extraction module is used to extract 3D human skeleton joints from RGB images using a 3D human skeleton extraction algorithm; A generation module is used to dynamically generate an octree interaction cube (AOIC) based on the extracted 3D human skeleton joints and divide the AOIC into an interaction space consisting of multiple voxels. A construction module is used to construct a virtual interaction space VIZ and divide the VIZ into virtual interaction spaces with the same number of voxels as the AOIC; The mapping module is used to construct the mapping relationship between each voxel in AOIC and the corresponding voxel in VIZ; The interactive control module is used to control the interactive operations in the virtual space of VIZ through mapping relationships based on the positions of the user's hand joints in AOIC and gesture control instructions; The dynamic generation of the octree interactive cube AOIC in the generation module is to calculate the side length of the AOIC, the position of the AOIC center point and the voxel side length of the AOIC through the joint points of the human skeleton; The side length of the AOIC is expressed as: AOIC 边长 =min(W scale ×W s ,H scale ×T c ,D scale ×L a ); Among them, AOIC 边长 Indicates the side length of AOIC; W s Represents the shoulder width in the 3D human skeleton joint; L a Represents the arm length in the 3D human skeleton joint; T c Indicates the height from neck to waist in the 3D human skeleton joint; W scale 、H scale 、D scale Indicates experience value; AOIC center point position (X p +J shoulder_x , Y p +J shoulder_y , Z p +0.5×L a ), where (X p ,Y p ,Z p ) represents the position coordinates of the human hip joint in the current camera coordinate system, which is the origin of the human body; J shoulder_x Indicates the offset from the hip joint position to the shoulder joint position in the x-axis direction; J shoulder_y Indicates the offset in the y-axis direction from the hip joint position to the shoulder joint position; The voxel side length of AOIC is expressed as: voxel 边长 =AOIC 边长 / step; Among them, voxel 边长 Indicates the voxel side length of AOIC; step indicates the number of equal-proportion divisions of the shortest side of AOIC.

2. The adaptive 3D spatial interaction system based on a monocular camera according to claim 1, characterized in that: The system also includes an AOIC update module for reconstructing and updating the octree structure in the AOIC when it detects that the movement range of the AOIC center point position is greater than a preset threshold or receives an instruction to change the VIZ center point by a gesture; and for updating part of the octree structure in the AOIC when it detects that the movement range of the AOIC center point position is less than a preset threshold.

3. The adaptive 3D spatial interaction system based on a monocular camera according to claim 1, characterized in that: The invention also includes an adjustment module for obtaining the movement speed of the hand joints in the 3D human skeleton joints, and dynamically adjusting the number of octree recursive segmentation times according to the movement speed to change the voxel side length of the AOIC.

4. The adaptive 3D spatial interaction system based on a monocular camera according to claim 3, characterized in that: The number of recursive segmentation times is expressed as: ； Among them, n(v hand ) represents the number of recursive segmentation times after dynamic adjustment according to the movement speed of the hand joints; clamp() represents the function that limits the number of recursive segmentation times; v hand Indicates the movement speed of the hand joints; v min and v max Respectively represent the minimum and maximum values ​​of the dynamic range of motion speed; n max Indicates the maximum number of recursive splits.

5. The adaptive 3D space interaction system based on a monocular camera according to claim 3, characterized in that: The voxel side length of the AOIC is expressed as: l current (n)=l root / 2 n ； Among them, l current (n) represents the AOIC voxel side length corresponding to the current recursive segmentation number n; l root Indicates the initial voxel edge length of the root node.

6. An adaptive 3D space interaction method based on a monocular camera, characterized in that: include: S1. Obtain an RGB image of the user's action captured by a monocular camera; S2. Use 3D human skeleton extraction algorithm to extract 3D human skeleton joint points from RGB images; S3. Dynamically generate an octree interaction cube (AOIC) based on the extracted 3D human skeleton joints, and divide the AOIC into an interaction space consisting of multiple voxels; S4. Construct a virtual interaction space VIZ and divide the VIZ into virtual interaction spaces with the same number of voxels as the AOIC; S5. Construct a mapping relationship between each voxel in AOIC and the corresponding voxel in VIZ; S6. Control interactive operations in the virtual space of VIZ through mapping relationships based on the positions of the user's hand joints in AOIC and gesture control commands; In step S3, the octree interaction cube AOIC is dynamically generated by calculating the side length of the AOIC, the position of the AOIC center point, and the voxel side length of the AOIC through the joint points of the human skeleton; The side length of the AOIC is expressed as: AOIC 边长 =min(W scale ×W s ,H scale ×T c ,D scale ×L a ); Among them, AOIC 边长 Indicates the side length of AOIC; W s Represents the shoulder width in the 3D human skeleton joint; L a Represents the arm length in the 3D human skeleton joint; T c Indicates the height from neck to waist in the 3D human skeleton joint; W scale 、H scale 、D scale Indicates experience value; AOIC center point position (X p +J shoulder_x , Y p +J shoulder_y , Z p +0.5×L a ), where (X p ,Y p ,Z p ) represents the position coordinates of the human hip joint in the current camera coordinate system, which is the origin of the human body; J shoulder_x Indicates the offset from the hip joint position to the shoulder joint position in the x-axis direction; J shoulder_y Indicates the offset in the y-axis direction from the hip joint position to the shoulder joint position; The voxel side length of AOIC is expressed as: voxel 边长 =AOIC 边长 / step; Among them, voxel 边长 Indicates the voxel side length of AOIC; step indicates the number of equal-proportion divisions of the shortest side of AOIC.

7. The method of adaptive 3D space interaction based on a monocular camera according to claim 6, characterized in that: After step S6, the following steps are also included: S7. Obtain the movement speed of the hand joints in the 3D human skeleton joints, and dynamically adjust the number of octree recursive segmentation times according to the movement speed to change the voxel side length of AOIC.

8. The method of adaptive 3D space interaction based on a monocular camera according to claim 7, characterized in that: After step S7, the following steps are further included: S8. When the movement range of the AOIC center point is detected to be greater than a preset threshold or a gesture instruction to change the VIZ center point is received, the octree structure in the AOIC is reconstructed and updated. When the movement range of the AOIC center point is detected to be less than a preset threshold, part of the octree structure in the AOIC is updated.

Images