Chest wearing type double-sided four-eye camera array and target reconstruction method thereof

Abstract

The invention relates to the field of three-dimensional reconstruction, and discloses a chest-wearing double-sided four-eye camera array and a target reconstruction method thereof, and the camera array is of a triangular prism structure fitting a human anatomical curve; the bottom surface is attached to the chest of a human body, two monocular cameras are respectively arranged on the top surface and the bottom surface, and the two monocular cameras are respectively arranged at an elevation angle and a depression angle; the reconstruction method comprises the following steps: acquiring a human body image in real time through a chest-wearing double-sided four-eye camera array, and performing target reconstruction positioning by using a deep learning target detection algorithm and a monocular vision inertial state estimator; compared with a traditional scheme, the method breaks through the limitation of hardware and scenes, is low in cost, brings high-quality interaction experience to the VR and AR fields with high stability, precision and reliability, and promotes the development of an immersive interaction technology.

Description

Technical Field

[0001] This invention relates to the field of three-dimensional reconstruction, and more particularly to a chest-mounted dual-face quad-camera array and its target reconstruction method. Background Technology

[0002] In the fields of virtual reality (VR) and augmented reality (AR), the realization of natural interactive experiences highly relies on precise positioning technology. Not only is accurate tracking of the human body's position and posture in space necessary, but 3D hand positioning technology has become a crucial core support for achieving natural interaction. With the deepening of the metaverse concept, users' demands for immersive interactive experiences are constantly increasing. Whether it's object manipulation in virtual scenes, gesture command input, or the transmission of fine movements in AR remote collaboration, high-precision human body and hand motion capture technology is essential as a foundational guarantee. The core challenge of motion capture and 3D environment reconstruction technology lies in the stability of hardware deployment and scene adaptability.

[0003] Head-mounted camera solutions integrate multi-camera arrays into VR / AR headsets, utilizing head-mounted inside-out tracking technology to achieve environmental perception and human motion capture. At its core, it relies on SLAM algorithms, using cameras to collect environmental data in real time, combining this with an inertial measurement unit (IMU) for pose estimation, and supporting gesture recognition or partial limb tracking.

[0004] Head-mounted camera solutions require no external equipment and can directly capture actions within the user's field of view, offering advantages such as low latency. However, the camera is fixed to the head, making it unable to capture lower body movements such as the torso and legs, and hand movements are easily obstructed by the head. During user movement, high-frequency head movements cause significant shifts in the sensor's field of view, requiring frequent data corrections by algorithms, increasing computational burden and affecting interaction accuracy. Furthermore, prolonged wear can lead to visual fatigue and neck strain.

[0005] The multi-angle camera solution uses an external sensor network to capture the user's full-body movements by deploying fixed multi-angle optical cameras around the user. The system relies on marker points or depth cameras to calculate the user's limb positions using triangulation principles. This solution requires a professionally calibrated camera array and dedicated space, resulting in high maintenance costs and significantly limiting its application and widespread adoption. Summary of the Invention

[0006] The purpose of this invention is to propose a chest-worn dual-sided quad-camera array and its target reconstruction method, which solves the technical problems of high cost, complex reconstruction calculation, and uncomfortable wearing of existing 3D reconstruction cameras.

[0007] Specifically, the present invention provides a chest-mounted dual-sided quad-camera array, comprising: a triangular prism structure conforming to the anatomical curve of the human body; its bottom surface conforming to the human chest, and two monocular cameras on each of the top and bottom surfaces, the two monocular cameras being arranged at an elevation angle and a depression angle respectively.

[0008] A wearable device for a chest-worn dual-sided quad-camera array includes: an elastic chest strap assembly and an adjustable shoulder strap assembly; the elastic chest strap assembly is used to fit the wearer's chest, and the adjustable shoulder strap assembly is used to fit the wearer's shoulders; the adjustable shoulder strap assembly and the elastic chest strap assembly cooperate with each other so that the chest-worn dual-sided quad-camera array can be tightly fixed to the wearer's chest.

[0009] A target reconstruction method using a chest-mounted dual-face quad-camera array, comprising the following steps:

[0010] S1. Real-time acquisition of human body images via a chest-mounted dual-sided quad-camera array;

[0011] S2. Segment the human body and hands in a human body image;

[0012] S3, Predicting two-dimensional heat maps;

[0013] S4. Decode the 2D heatmap into 2D key point coordinates and depth;

[0014] S5. Combine image features to project two-dimensional key points into three-dimensional space;

[0015] S6. Use a monocular vision inertial state estimator to locate the device in the surrounding environment.

[0016] The beneficial effects provided by this invention are: compared with traditional solutions, this invention breaks through hardware and scene limitations, has low cost, and brings a high-quality interactive experience to the VR and AR fields with high stability, accuracy and reliability, thus promoting the development of immersive interactive technology. Attached Figure Description

[0017] Figure 1 This is a schematic diagram of the chest-mounted dual-sided quad-eye camera array structure of the present invention;

[0018] Figure 2 This is a side view of the chest-mounted dual-sided quad-camera array of the present invention;

[0019] Figure 3 This is a front view of the chest-mounted dual-sided quad-camera array of the present invention;

[0020] Figure 4 This is a schematic diagram of the chest-mounted dual-sided quad-lens camera array of the present invention after being worn on the human body by the wearing device;

[0021] Figure 5 This is a front view of the wearable device;

[0022] Figure 6 It's the back of the wearable device. Detailed Implementation

[0023] To make the objectives, technical solutions, and advantages of the present invention clearer, the embodiments of the present invention will be further described below with reference to the accompanying drawings.

[0024] Before formally describing the present invention, a general description of the solution of the present invention will be given first to facilitate understanding.

[0025] Please refer to Figures 1-3 The present invention provides a chest-mounted dual-sided quad-lens camera array, comprising:

[0026] It has a triangular prism structure that conforms to the anatomical curve of the human body; its bottom surface conforms to the human chest, and there are two monocular cameras on the top and bottom surfaces, with the two monocular cameras arranged at an elevation angle and a depression angle, respectively.

[0027] The vertex angle of each lateral triangle of the triangular prism structure ranges from [90°, 110°].

[0028] The field of view of a monocular camera is [140°, 160°];

[0029] Two monocular cameras on the top are used to capture human head movement information.

[0030] Two monocular cameras on the bottom are used to capture information about the movements of the human body in front of it and its hands;

[0031] Specifically, the camera array of this invention has an overall triangular prism structure, designed to conform to the anatomical curves of the human body, with dual-sided quad-lens array cameras arranged at both elevation and depression angles. Wearing it on the chest, a naturally stable reference coordinate system, facilitates the capture of hand and head movements and enables SLAM 3D reconstruction of the surrounding scene.

[0032] Figure 2 , Figure 3 The figures shown are a side view and a front view of the invention. A1, A2, B1, and B2 represent four monocular cameras. The value of α ranges from 90° to 110°, and the value of β ranges from 140° to 160°. The field of view of a single-sided binocular camera is 140° to 160°, meaning the camera can cover a wide area in this direction. This effectively captures the movements of the human hand, head, etc., within this field of view, facilitating comprehensive perception of human actions and SLAM 3D reconstruction of the surrounding scene, providing a solid visual foundation for the efficient operation of the device.

[0033] A wearable device for a chest-worn dual-sided quad-camera array includes: an elastic chest strap assembly and an adjustable shoulder strap assembly; the elastic chest strap assembly is used to fit the wearer's chest, and the adjustable shoulder strap assembly is used to fit the wearer's shoulders; the adjustable shoulder strap assembly and the elastic chest strap assembly cooperate with each other so that the chest-worn dual-sided quad-camera array can be tightly fixed to the wearer's chest.

[0034] Please refer to Figure 4 In the dual-sided quad-eye array layout of the present invention, the two cameras on the upper front are set at an upward angle, which can effectively cover a certain range of field of view above and in front of the human head, and can clearly capture head movements such as tilting and turning.

[0035] The two front-facing cameras, positioned at a downward angle, focus on the human hand and the area in front of the body, capturing both fine hand movements and wide-ranging gestures in front of the body. This distributed perspective design closely follows the common range of human hand and head movements, ensuring all-around, blind-spot-free capture, thus providing accurate and comprehensive data support for subsequent motion analysis and scene SLAM 3D reconstruction.

[0036] Regarding the ease of wearing and stability of the device, this invention proposes a chest-worn dual-sided quad-camera array wearing device, mainly composed of an elastic chest strap assembly and an adjustable shoulder strap assembly. A front view of the wearing device is shown below. Figure 5 As shown, the rear view is as follows Figure 6 As shown.

[0037] The elastic chest band 5 is made of wide composite fabric with an array of anti-slip silicone bumps spaced longitudinally on the inside. With the Velcro 6 overlapping structure, users can easily achieve stepless adjustment of the chest circumference to fit different body shapes and ensure that every user can get a comfortable and fit wearing experience.

[0038] The adjustable shoulder strap 1 is hinged to both sides of the camera body 3 via a rotatable connector 4. Combined with an easy-to-use adjustment tongue 2, users can quickly adjust the tightness of the shoulder strap according to their comfort and the actual usage scenario, ensuring unrestricted natural shoulder movement. This design, through its elastic restraint and three-point support mechanical layout, allows the camera to fit snugly and stably against the torso during user movement.

[0039] A target reconstruction method using a chest-mounted dual-face quad-camera array, comprising the following steps:

[0040] The method includes the following steps:

[0041] S1. Real-time acquisition of human body images via a chest-mounted dual-sided quad-camera array;

[0042] S2. Segment the human body and hands in a human body image;

[0043] S3, Predicting two-dimensional heat maps;

[0044] S4. Decode the 2D heatmap into 2D key point coordinates and depth;

[0045] S5. Combine image features to project two-dimensional key points into three-dimensional space;

[0046] S6. Use a monocular vision inertial state estimator to locate the device in the surrounding environment.

[0047] More specifically, the method includes the following steps:

[0048] S1. Real-time acquisition of human body images via a chest-mounted dual-sided quad-camera array;

[0049] S2. Use deep semantic segmentation to segment the human body and hands in a human image;

[0050] S3. Extract feature maps using a convolutional neural network model, pass the extracted feature maps to an upsampler and an MLP network, and finally output a two-dimensional heatmap.

[0051] S4. By normalizing the 2D heatmap into a probability distribution, the probability distribution is first marginalized, and then the expected values ​​in the x and y directions are calculated separately. These expected values ​​are then used as the coordinates of the 2D keypoints. Simultaneously, an MLP network is used to obtain the depth values ​​of the keypoints.

[0052] S5. The key point depth value is concatenated with the 2D key point coordinates to obtain the 2.5D coordinates. Based on the pinhole camera model, the 2.5D coordinates are transformed into the camera space using the camera intrinsic parameters to finally obtain the 3D key point coordinates.

[0053] S6. Based on the VINS-Mono method, IMU and monocular vision are fused to perform localization in the surrounding environment;

[0054] The key point of this invention is:

[0055] 1. The chest is used as the main support for the device, utilizing the biomechanical characteristic that the range of motion of the torso is significantly lower than that of the head to construct a naturally stable reference coordinate system. A three-point fixation system consisting of an elastic chest strap and bionic shoulder straps securely attaches the device to the midline region of the sternum, effectively reducing the interference of upper limb movements on visual data acquisition.

[0056] 2. This invention features a triangular prism structure designed according to the anatomical curves of the human body, perfectly conforming to the chest. This enhances wearing comfort and ensures device stability during use, reducing capture errors caused by shaking. The structure allows the device to tilt naturally forward, achieving natural field of view coverage, enabling the camera to capture hand and head movements at the optimal angle and range, reducing blind spots and improving capture accuracy and completeness.

[0057] 3. The two front-upper cameras are positioned at an upward angle, covering a certain range of the field of view above and in front of the human head, clearly capturing head movements such as tilting and turning. The two front-lower cameras are positioned at a downward angle, focusing on the human hands and the area in front of the body, fully capturing fine hand movements and wide-range waving in the space in front of the body. Through this unique dual-sided four-camera array layout, this invention achieves all-round, blind-spot-free coverage of common ranges of human hand and head movements.

[0058] 4. Cameras simultaneously capture human motion data from multiple angles, and by overlaying multi-view images, the 3D motion trajectory of the human body can be reconstructed. This method makes full use of image information captured by different cameras, improving the accuracy and reliability of motion capture.

[0059] In practical applications, the chest-mounted human motion capture and SLAM reconstruction system utilizes a dual-faced quad-camera system that operates in an orderly manner. Once activated, the dual-faced quad-camera array captures real-time, omnidirectional video of subtle hand movements and overall posture changes. The chest anchoring design and triangular prism structure ensure high-quality and stable data acquisition. Subsequently, algorithms segment the hand and body, predicting a 2D heatmap, which is then decoded into 2D keypoint coordinates and depth. Combined with image features, the 2D keypoint coordinates are projected into a 3D model. Multi-view image overlay further enhances positioning accuracy, reconstructing a highly realistic 3D human motion trajectory. Simultaneously, deep learning object detection algorithms identify environmental objects and provide their 3D shapes. This, combined with a monocular visual inertial state estimator for localization, achieves precise SLAM 3D reconstruction.

[0060] In summary, the beneficial effects of this invention are: compared with traditional solutions, this invention breaks through hardware and scene limitations, bringing a high-quality interactive experience to the VR and AR fields with high stability, accuracy and reliability, and promoting the development of immersive interactive technology.

[0061] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A chest-mounted dual-sided quad-lens camera array, characterized in that: It has a triangular prism structure that conforms to the anatomical curve of the human body; its bottom surface conforms to the human chest, and there are two monocular cameras on the top and bottom surfaces, with the two monocular cameras arranged at an elevation angle and a depression angle, respectively.

2. The chest-mounted dual-sided quad-lens camera array as described in claim 1, characterized in that: The vertex angle of each lateral triangle of the triangular prism structure ranges from [90°, 110°].

3. The chest-mounted dual-sided quad-lens camera array as described in claim 1, characterized in that: The field of view of a monocular camera is [140°, 160°].

4. The chest-mounted dual-sided quad-lens camera array as described in claim 1, characterized in that: Two monocular cameras on the top are used to capture human head movement information.

5. A chest-mounted dual-sided quad-lens camera array as described in claim 1, characterized in that: Two monocular cameras on the bottom are used to capture information about the movements of the human body in front of it and its hands.

6. A chest-worn dual-sided quad-lens camera array wearing device, characterized in that: include: Elastic chest strap assembly and adjustable shoulder strap assembly; the elastic chest strap assembly is designed to fit the wearer's chest, and the adjustable shoulder strap assembly is designed to fit the wearer's shoulders; The adjustable shoulder strap assembly and the elastic chest strap assembly work together to ensure that the chest-mounted dual-sided quad-camera array can be securely fixed to the wearer's chest.

7. The wearing device for a chest-mounted dual-sided quad-lens camera array as described in claim 6, characterized in that: The elastic chest band assembly is made of wide composite fabric with an array of anti-slip silicone bumps spaced longitudinally on the inner side, and uses a Velcro closure structure to achieve stepless adjustment of the chest circumference.

8. The wearing device for a chest-mounted dual-sided quad-lens camera array as described in claim 6, characterized in that: The adjustable shoulder strap assembly is hinged to both sides of the chest-mounted dual-sided quad-camera array body via rotatable connectors.

9. A target reconstruction method for a chest-mounted dual-face quad-camera array, characterized in that: The method includes the following steps: S1. Real-time acquisition of human body images via a chest-mounted dual-sided quad-camera array; S2. Segment the human body and hands in a human body image; S3, Predicting two-dimensional heat maps; S4. Decode the 2D heatmap into 2D key point coordinates and depth; S5. Combine image features to project two-dimensional key points into three-dimensional space; S6. Use a monocular vision inertial state estimator to locate the device in the surrounding environment.

Images