Using human joints to calibrate cameras on unmanned aerial vehicles

Abstract

A system and method for calibrating a camera on an unmanned aerial vehicle (UAV) is provided. The system receives a set of anchor images of a human subject from a set of anchor cameras and receives a set of images of the human subject from multiple viewpoints in a three-dimensional (3D) space from a camera set on the UAV set. The system determines a first set of two-dimensional (2D) positions of human joints from the set of anchor images and determines a second set of 2D positions of the human joints from the image set. The system calculates the 3D positions of the human joints as 3D key points based on triangulation using the first set of 2D positions and determines a 2D reprojection error between the 3D key points and the second set of 2D positions. Thereafter, the system calibrates each camera in the camera set by minimizing the 2D reprojection error.

Description

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS / INCORPORATED BY REFERENCE

[0002] none Technical Field

[0003] Various embodiments of the present disclosure relate to camera calibration. More specifically, various embodiments of the present disclosure relate to systems and methods for calibrating a camera mounted on or integrated with an unmanned aerial vehicle (UAV) using human joints. Background Art

[0004] Various two-dimensional patterns, such as checkerboard patterns, are used for camera calibration. While camera calibration using such patterns can produce accurate calibration results, these techniques can be time consuming and can additionally require a significant amount of manual work. For example, the application of such camera calibration techniques can be prone to point matching problems.

[0005] Further limitations and disadvantages of conventional and traditional approaches will become apparent to those skilled in the art by comparing the described systems with certain aspects of the present disclosure, as set forth in the remainder of this application and with reference to the accompanying figures. Summary of the invention

[0006] As more fully set forth in the claims, substantially as shown in and / or described in conjunction with at least one of the figures, a system and method for calibrating a camera mounted on or integrated with an unmanned aerial vehicle (UAV) using human joints is provided.

[0007] These and other features and advantages of the present disclosure may be understood by reading the following detailed description of the disclosure and the accompanying drawings, in which like reference numerals refer to like parts throughout. BRIEF DESCRIPTION OF THE DRAWINGS

[0008] Figure 1 is a block diagram illustrating an exemplary network environment for calibrating a camera group mounted on or integrated with a UAV group by using human joints according to an embodiment of the present disclosure.

[0009] Figure 2 is a block diagram illustrating an exemplary system for calibrating a camera group mounted on or integrated with a UAV group according to an embodiment of the present disclosure.

[0010] Figure 3 is a block diagram illustrating an exemplary UAV according to an embodiment of the present disclosure.

[0011] Figure 4is a diagram illustrating an exemplary implementation of a camera mounted on or integrated with a UAV and an anchor camera device to acquire images of a human subject according to an embodiment of the present disclosure.

[0012] Figure 5 is a diagram illustrating an exemplary implementation of a camera mounted on or integrated with a UAV and an anchor camera device to acquire images of a human subject according to an embodiment of the present disclosure.

[0013] Figure 6 is a diagram illustrating an exemplary operation for calibrating a camera on a UAV based on human body joints according to an embodiment of the present disclosure.

[0014] Figure 7 is a flow chart illustrating exemplary operations for calibrating a camera group mounted on or integrated with a UAV group according to an embodiment of the present disclosure. DETAILED DESCRIPTION

[0015] The embodiments described below may be found in the disclosed system and method for automatically calibrating a camera set mounted on or integrated with an unmanned aerial vehicle (UAV) using human joints. Exemplary aspects of the present disclosure provide a set of anchor cameras that may be pre-calibrated and fixed at certain known positions and orientations in 3D space. In addition, the camera set may be mounted on each UAV in the UAV group or integrated with them. The camera group may not be pre-calibrated for external camera parameters (such as translation and rotation parameters). The set of anchor cameras and the UAV group may surround a human subject in 3D space. A first set of two-dimensional (2D) positions of human joints may be determined from a set of anchor images of a human subject captured by the set of anchor cameras. In addition, a second set of 2D positions of human joints of a human subject may be determined from a set of images of a human subject captured by the camera group from multiple three-dimensional (3D) viewpoints. The system can compute 3D key points as 3D positions of human body joints based on triangulation of the determined first set of 2D positions, and can calibrate each camera in the camera group relative to a set or gauge of anchor cameras (i.e., extrinsic and / or intrinsic parameters) by minimizing the 2D reprojection error between the 3D key points and the second set of 2D positions.

[0016] Using human joints for camera calibration may be less prone to point matching problems, which often occur in conventional calibration methods using 2D patterns (such as a chessboard). Moreover, using human joints for calibration may remove the need to manually place a calibration pattern (such as placing a chessboard in front of an uncalibrated camera).

[0017] Figure 1is a block diagram illustrating an exemplary network environment for calibrating a camera group mounted on or integrated with a UAV group by using human joints according to an embodiment of the present disclosure. Figure 1 , a network environment 100 is shown. The network environment 100 may include a system 102 and a machine learning (ML) model 104, which may be included in the system 102. The network environment 100 may also include an anchor camera device set 106, a UAV group 108, and a communication network 110. As shown, for example, the anchor camera device set 106 may include a first anchor camera device 106a and a second anchor camera device 106b. Similarly, the UAV group 108 may include a first UAV 108a and a second UAV 108b, both of which may be equipped with cameras, such as a first camera 112a for the first UAV 108a and a second camera 112b for the second UAV 108b. The first camera 112a and the second camera 112b may be collectively referred to as a camera group 112. The system 102 may be communicatively coupled to the anchor camera device set 106 and the UAV group 108 via the communication network 110.

[0018] Also shown is a three-dimensional (3D) space 114 as part of a 3D environment in which the anchor camera device set 106 and the UAV group 108 may be deployed. In the 3D space 114, a human subject 116 is also shown surrounded by the anchor camera device set 106 and the UAV group 108. The anchor camera device set 106 and the UAV group 108 may be (e.g., sparsely) located at multiple viewpoints in the 3D space 114 and may together form a multi-scanner network with a combined field of view (FoV) that may cover at least the front or back side of the body of the human subject 116. The human subject 116 may be part of the foreground of the scene, which may also include other objects, details, and descriptions that are omitted in the present disclosure for brevity.

[0019] It should be noted that Figure 1 The positions, orientations, arrangements, and numbers of cameras associated with the anchor camera device set 106 and the UAV group 108 in the embodiment are presented only as examples and should not be construed as limiting the present disclosure. The present disclosure may also be applicable to other positions, orientations, arrangements, and greater or lesser numbers of anchor camera devices and UAVs without departing from the scope of the present disclosure.

[0020] The system 102 may include suitable logic, circuitry, interfaces, and / or code that may be configured to calibrate each camera mounted on or integrated with a corresponding UAV in the group of UAVs 108 by using information associated with human joints of the human subject 116. Examples of implementations of the system 102 may include, but are not limited to, a video broadcast system, an augmented reality-based device, a gaming device, a video processing device, a video editing system, a media production system, a computer workstation, a mainframe computer, a handheld computer, a mobile phone, a smart appliance, a video player, a digital camera, a video camera, a server, a consumer electronics (CE) device, and / or any other computing device with image processing capabilities.

[0021] The ML model 104 may correspond to a human joint detection framework, such as a neural network that can be trained for a 2D human joint detection task. For example, the ML model 104 may include a trained neural network that can receive an image of a human subject 116 to detect human joints from the image, with a certain confidence score for each human joint. The ML model 104 may include electronic data that may, for example, be implemented as a software component and may rely on a code database, library, external script, or other logic or instructions for a computing system (such as by the system 102) to perform an ML algorithm. The ML model 104 may be implemented as code and routines that are configured to enable a computing system such as the system 102 to perform human joint detection from an image of a human subject 116. Additionally or alternatively, the ML model 104 may be implemented using hardware including a processor, a microprocessor (e.g., a performance for performing or controlling one or more operations), a field programmable gate array (FPGA), or an application specific integrated circuit (ASIC). In some other cases, the ML model 104 may be implemented using a combination of both hardware and software.

[0022] According to an embodiment, the ML model 104 may be a neural network model. Examples of neural network models may include, but are not limited to, a deep neural network (DNN), a convolutional neural network (CNN), a recurrent neural network (RNN), a CNN-recurrent neural network (CNN-RNN), R-CNN, a fast R-CNN, a faster R-CNN, an artificial neural network (ANN), a (You Only Look Once) YOLO network, a RNN-based long short-term memory (LSTM) network, a CNN+ANN, a LSTM+ANN, a RNN based on a gated recurrent unit (GRU), a fully connected neural network, a RNN based on connectionist temporal classification (CTC), a deep Bayesian neural network, a generative adversarial network (GAN), and / or a combination of such networks. In some embodiments, the neural network model may be based on a hybrid architecture of multiple deep neural networks (DNNs).

[0023] Each of the set of anchor camera devices 106 may include suitable logic, circuitry, interfaces, and / or code that may be configured to capture one or more images that may include a human subject 116. Each anchor camera device may be fixed at a location in the 3D space 114. For example, the anchor camera device may be fixed at a predetermined location on a wall or a roof. Examples of implementations of the anchor camera device may include, but are not limited to, a pre-calibrated fixed camera in the 3D space 114, a pre-calibrated camera mounted on or integrated with a UAV that may be configured to maintain a fixed pose in the 3D space 114, or a pre-calibrated camera movably coupled to a remotely controlled camera movement assembly.

[0024] Each UAV in the UAV group 108 may include a UAV that may be equipped with a camera (initially uncalibrated). In addition, each UAV may be controlled by a remote system or may be capable of autonomous flight based on a pre-programmed route or path. In at least one embodiment, each UAV may implement a human tracking method to align itself in the 3D space 114 and follow the human subject 116 so that the human subject 116 is always in the FOV area of ​​the camera equipped on the corresponding UAV. In at least one embodiment, each UAV may be controlled by a human pilot from a central control station, which may be part of the system 102.

[0025] According to an embodiment, each UAV in the UAV group 108 (e.g., the first UAV 108a and the second UAV 108b) may receive control instructions from the system 102 via the communication network 110. The control instructions may include a 3D position (X-axis, Y-axis, or Z-axis) or a 3D path based on which one or more of the UAV group 108 may move in the 3D space 114. In some embodiments, such instructions may include tilt or orientation information that may be associated with the 3D posture of each UAV in the 3D space 114. The UAV group 108 may control their tilt angle or orientation and vertical or horizontal alignment based on the control instructions received from the system 102. Examples of UAVs may include, but are not limited to, camera drones, smart aircraft equipped with cameras, or other aircraft and / or automated systems (such as the system 102) that may be controlled based on a preprogrammed flight plan. Based on the frame type and the number of motors, examples of UAVs may include, but are not limited to, a tricopter with three arms and one motor, a quadcopter with four arms and one motor, a hexacopter with six arms and one motor, a Y6 helicopter with six arms and six motors, an octocopter with eight arms and one motor, and / or an X8 helicopter with eight arms and eight motors.

[0026] Each of the first camera 112a and the second camera 112b may include suitable logic, circuitry, interfaces, and / or code that may be configured to capture one or more images that may include a human subject 116. Each of the first camera 112a and the second camera 112b may be mounted on or integrated with the UAV. Examples of the first camera 112a and the second camera 112b may include, but are not limited to, a SLR camera, a digital SLR camera, a studio or broadcast camera, a high-speed camera, a wide-angle camera, an action camera, a closed-circuit television (CCTV) camera, a camcorder, a digital camera, a camera phone, a time-of-flight camera (ToF camera), a night vision camera, and / or other image capture devices.

[0027] The communication network 110 may include a communication medium through which the system 102, the anchor camera device set 106, and the UAV group 108 can communicate with each other. Examples of the communication network 110 may include, but are not limited to, the Internet, a cloud network, a wireless fidelity (Wi-Fi) network, a personal area network (PAN), a local area network (LAN), or a metropolitan area network (MAN). Various devices in the network environment 100 may be configured to connect to the communication network 110 according to various wired and wireless communication protocols. Examples of such wired and wireless communication protocols may include, but are not limited to, transmission control protocol and Internet protocol (TCP / IP), user datagram protocol (UDP), hypertext transfer protocol (HTTP), file transfer protocol (FTP), Zig Bee, EDGE, IEEE 802.11, light fidelity (Li-Fi), 802.16, IEEE 802.11s, IEEE802.11g, multi-hop communication, wireless access point (AP), device-to-device communication, cellular communication protocol, and Bluetooth (BT) communication protocol.

[0028] In the depicted embodiment, 3D space 114 may be a built environment (eg, an indoor, outdoor, or studio space) or a natural environment in which images of human subject 116 may be captured from multiple viewpoints.

[0029] In operation, the system 102 may receive user input from a user (such as the human subject 116 or another human operator) through a user interface of the system 102. The user input may instruct the system 102 to calibrate each camera (such as the first camera 112a and the second camera 112b) mounted on or integrated with the UAV group 108.

[0030] To initialize the calibration process for such cameras on the UAV group 108, images may be acquired from the anchor camera device set 106 and from cameras (first camera 112a and second camera 112b) mounted on or integrated with the UAV group 108. Thereafter, the system 102 may receive a set of anchor images of a human subject 116 from the anchor camera device set 106. In addition, the system 102 may receive a set of images of a human subject 116 from cameras mounted on or integrated with the UAV group 108. The received set of images may be acquired from multiple viewpoints in the 3D space 114. For example, a first number of images from the set of images may be captured by the first camera 112a, and a remaining number of images may be captured by the second camera 112b.

[0031] The system 102 may determine a first set of 2D positions of human joints of a human subject 116 in each anchor image of the received set of anchor images. The set of anchor images captured by the set of anchor camera devices 106 may include accurate positions of key points of an object (e.g., 2D positions of human joints) because each anchor camera device may be pre-calibrated and fixed to a known position in the 3D space 114. For the present disclosure, the positions of human joints may be regarded as rigid and hinge points in the 3D space 114. Moreover, the distances between human joints may be set based on experimental results. The system 102 may also determine a second set of 2D positions of human joints of a human subject 116 in each image of the received set of images. In an embodiment, the system 102 may determine a first set of 2D positions from the set of anchor images and a second set of 2D positions from the set of images using the ML model 104. Examples of implementations of the ML model 104 may include, but are not limited to, Open Pose, AlphaPose, and Mask Region-Convolution Neural Network (R-CNN).

[0032] The system 102 can also calculate the 3D position of the human joint in the 3D space 114 as a 3D key point based on the triangulation of the first set of 2D positions of the determined human joints (e.g., using a geometric triangulation method). In addition, the system 102 can determine the 2D reprojection error between the 3D key point and the second set of the determined 2D position. The system 102 can calibrate each camera on the corresponding UAV of the UAV group 108 by minimizing the determined 2D reprojection error. For example, in order to calibrate the first camera 112a, the 3D key point can be reprojected onto the image plane of the first camera 112a mounted on the first UAV 108a or integrated with the first UAV 108a. Thereafter, the reprojection error between each reprojected 3D key point and the 2D position of the corresponding human joint can be determined. The objective function can be formulated and solved to minimize the reprojection error. The above-mentioned method can also be repeated for the second camera 112b. Since the camera on each UAV in the UAV group 108 may initially be uncalibrated, 3D keypoint-based camera calibration of the cameras can result in estimates of values ​​of extrinsic calibration parameters (i.e., 3D poses) for each camera mounted on the corresponding UAV in the UAV group 108.

[0033] The detection of 2D positions of body joints of the human subject 116 based on the ML model 104 may be less prone to error and may not require manual input. In addition, the system 102 may compute a greater number of 3D key points than may be computed based on conventional camera calibration techniques using conventional 2D patterns, such as a checkerboard pattern. The disclosed camera calibration techniques may be less prone to point mismatch issues because point mismatch errors, if any, may be suppressed by using one or more statistical techniques for a large number of 3D key points. Moreover, calibration using body joints may remove the need to manually place calibration patterns, such as a checkerboard.

[0034] Figure 2 is a block diagram illustrating an exemplary system for calibrating a camera group mounted on or integrated with a UAV group according to an embodiment of the present disclosure. Figure 2 Combination Figure 1 The elements in are explained. Figure 2 , a block diagram 200 of a system 102 is shown. The system 102 may include a circuit system 202, a memory 204, an input / output (I / O) device 206, and a network interface 208. The circuit system 202 may be communicatively coupled to the memory 204, the I / O device 206, and the network interface 208. In addition, the circuit system 202 may be communicatively coupled to the set of anchor camera devices 106 and the camera group 112 mounted on or integrated with the UAV group 108 via a communication network 110. Figure 2 As shown in , the UAV group 108 may include a first UAV 108a that may carry a first camera 112a and a second UAV 108b that may carry a second camera 112b. Furthermore, the anchor camera device set 106 is shown to include a first anchor camera device 106a and a second anchor camera device 106b.

[0035] The circuit system 202 may include suitable logic, circuit systems, interfaces, and / or codes that may be configured to execute instructions stored in the memory 204. The executed instructions may correspond to a set of control operations for calibrating each camera (such as the first camera 112a and the second camera 112b) in the camera group 112 installed on or integrated with the UAV group 108. The circuit system 202 may be implemented based on processor technology known in the art. Examples of the circuit system 202 may include, but are not limited to, a graphics processing unit (GPU), a coprocessor, a central processing unit (CPU), an x86-based processor, a reduced instruction set computing (RISC) processor, an application specific integrated circuit (ASIC) processor, a complex instruction set computing (CISC) processor, and combinations thereof.

[0036] The memory 204 may include suitable logic, circuitry, interfaces, and / or code that may be configured to store instructions executable by the circuitry 202. Furthermore, the memory 204 may be configured to store a set of anchor images and a group of images of human subjects 116. The memory 204 may also be configured to store the ML model 104, which may be used to determine a first set of 2D positions of human joints from the set of anchor images and a second set of 2D positions of human joints from the group of images. Furthermore, the memory 204 may store the calculated X, Y, and Z coordinates of the 3D key points. Examples of implementations of the memory 204 may include, but are not limited to, a random access memory (RAM), a read-only memory (ROM), an electrically erasable programmable read-only memory (EEPROM), a hard disk drive (HDD), a solid-state drive (SSD), a CPU cache, and / or a secure digital (SD) card.

[0037] I / O device 206 may include suitable logic, circuit system, interface and / or code, which may be configured to receive input from a user and provide output to the user based on the received input. I / O device 206 may include various input and output devices, which may be configured to communicate with circuit system 202. Examples of input devices may include, but are not limited to, a touch screen, a keyboard and / or a mouse. Examples of output devices may include, but are not limited to, a display and an audio device.

[0038] The network interface 208 may include suitable logic, circuitry, interfaces, and / or code that may be configured to establish communication between the system 102, the set of anchor camera devices 106, the camera group 112, and the UAV group 108 via the communication network 110. The network interface 208 may be implemented using various known techniques to support wired or wireless communication of the system 102 via the communication network 110. The network interface 208 may include, but is not limited to, an antenna, a radio frequency (RF) transceiver, one or more amplifiers, a tuner, one or more oscillators, a digital signal processor, a coder-decoder (CODEC) chipset, a subscriber identity module (SIM) card, and / or a local buffer.

[0039] The network interface 208 may communicate with a network such as the Internet, an intranet, and / or a wireless network such as a cellular telephone network, a wireless local area network (LAN), and / or a metropolitan area network (MAN) via wireless communications. The wireless communications may use any of a variety of communication standards, protocols, and technologies, such as Global System for Mobile Communications (GSM), Enhanced Data GSM Environment (EDGE), Wideband Code Division Multiple Access (W-CDMA), Long Term Evolution (LTE), Code Division Multiple Access (CDMA), Time Division Multiple Access (TDMA), Bluetooth, Wireless Fidelity (Wi-Fi) (such as IEEE 802.11a, IEEE 802.11b, IEEE 802.11g, and / or IEEE 802.11n), Voice over Internet Protocol (VoIP), Light Fidelity (Li-Fi), Wi-MAX, email protocols, instant messaging, and / or short message service (SMS).

[0040] Functions or operations performed by system 102, such as Figure 1 As described in, it can be performed by the circuit system 202. For example, Figure 4 , 5 The operations performed by circuit system 202 are described in detail in FIGS.

[0041] Figure 3 is a block diagram of an exemplary UAV according to an embodiment of the present disclosure. Figure 3 Combination Figure 1 and Figure 2 The elements in are explained. Figure 3 , a block diagram 300 of a first UAV 108a is shown, which may be one of the UAV group 108. The first UAV 108a may include a circuit system 302, a memory 304, an I / O device 306, a position sensor 308, a propulsion system 310, and a network interface 312. The propulsion system 310 may include a motor 314, a propeller 316, an electronic speed controller (ESC) 318, and a battery 320. It will be understood by those of ordinary skill in the art that in addition to the components or systems shown herein for describing and explaining the functions and operations of the present disclosure, the first UAV 108a may also include other suitable components or systems. For the sake of brevity, the present disclosure omits a detailed description of other components or systems of the first UAV 108a.

[0042] The first UAV 108a and the system 102 may be communicatively coupled to each other via a communication network 110. The first camera 112a may be configured to be mounted on or integrated with the first UAV 308a such that the first camera 112a may have various degrees of freedom (DOF), such as 6DOF in the 3D space 114. As shown in the block diagram 300 of the first UAV 108a, the functionality of the circuit system 302, the memory 304, the I / O device 306, and the network interface 312 may be associated with, for example, Figure 2 The functions of the circuit system 202, memory 204, I / O device 206, and network interface 208 described in the foregoing description are the same. Therefore, for the sake of brevity, the present disclosure omits the description of the circuit system 302, memory 304, I / O device 306, and network interface 312.

[0043] The position sensor 308 may include suitable logic, circuit systems, interfaces, and / or codes that may be configured to determine the current position of the first UAV 108a in the 3D space 114. In at least one embodiment, the position sensor 308 may also determine the orientation (along the pitch axis, roll axis, or yaw axis) in the 3D space 114. The position sensor 308 may transmit the current position and / or orientation of the first UAV 108a to the circuit system 302 and / or the system 102. The circuit system 302 and / or the system 102 may control the position and / or orientation of the first UAV 108a based on the transmitted current position and / or orientation of the first UAV 108a. Examples of the position sensor 308 may include, but are not limited to, a global navigation satellite system (GNSS) receiver, a motion sensor, a tilt sensor, an accelerometer, a gyroscope sensor, an inertial measurement unit (IMU), or a sensing camera.

[0044] In an embodiment, the position sensor 308 may be implemented as an infrared (IR) tag sensor or a pattern code scanner. In the case of an IR tag sensor, an IR light source may be fixed at a predetermined position in the 3D space 114. The light source may emit IR light in a periodic flash or in a continuous manner, and a first camera 112a mounted on or integrated with the first UAV 108a may track and capture the emitted IR light as one or more IR images. Thereafter, the circuit system 302 may apply a 3D pose estimation technique (such as a pose and iterative scaling (POSIT) method from orthogonality) on one or more IR images to determine the position and / or orientation of the light source of the IR light of the first UAV 108a. In some scenarios, the predetermined position of the light source of the IR light may be known and stored in the memory 304 of the first UAV 108a. In such a scenario, the circuit system 302 may determine the absolute position and / or orientation of the first UAV 108a based on the predetermined position of the light source of the IR light and the determined position and / or orientation of the first UAV 108a.

[0045] In the case of a pattern code scanner, a pattern code marker (e.g., a QR code marker or a barcode marker) can be placed at a predetermined position in the 3D space 114. The first camera 112a can track the pattern code marker and capture an image of the pattern code marker. The memory 304 can store a reference image of the pattern code marker and a predetermined position of the pattern code marker. The circuit system 302 can compare the captured image with the reference image to determine the position and / or orientation of the first UAV 108a relative to the pattern code marker. Moreover, the circuit system 302 can determine the absolute position and / or orientation of the first UAV 108a based on the predetermined position of the pattern code marker and the determined position and / or orientation of the first UAV 108a.

[0046] The propulsion system 310 may include a collection of mechanical, electrical, or fuel-based components that can generate lift and thrust to move the first UAV 108a between two locations in the 3D space 114. The propulsion system 310 may control the movement of the first UAV 108a based on one or more control instructions received from the circuit system 302 or the system 102. The propulsion system 310 may also include a motor 314, a propeller 316, an ESC 318, and a battery 320.

[0047] Circuit systems (such as circuit system 302), memories (such as memory 304), I / O devices (such as I / O devices 306), position sensors (such as position sensor 308), propulsion systems (such as propulsion system 310), and network interfaces (such as network interface 312) may also be included in other UAVs in the UAV group 108. In an embodiment, the first UAV 108a may act as a master UAV and the other UAVs may act as slave UAVs that may receive one or more control instructions from the master UAV.

[0048] Figure 4 is a diagram illustrating an exemplary implementation of a camera mounted on or integrated with a UAV and an anchor camera device to acquire images of a human subject according to an embodiment of the present disclosure. Figure 4 Combination Figure 1 , 2 or 3. Figure 4 , a diagram 400 is shown depicting an arrangement of a first anchor camera 402a and a second anchor camera 402b (collectively referred to as a set of anchor cameras 402) that may face the rear side of a human subject 116. Diagram 400 also depicts an arrangement of a first camera 404a and a second camera 404b (collectively referred to as a camera set 404) mounted on or integrated with a first UAV 406a and a second UAV 406b, respectively. The first UAV 406a and the second UAV 406b may be collectively referred to as a UAV set 406 and may face the front side of a human subject 116.

[0049] Each of the set of anchor cameras 402 may correspond to a pre-calibrated camera that may be fixed to a location (e.g., a surface such as a ceiling, floor, or wall or a fixed structure). The first anchor camera 402a and the second anchor camera 402b may be configured to capture a first anchor image and a second anchor image (collectively, a set of anchor images) of a human subject 116, respectively. To acquire the set of anchor images, the set of anchor cameras 402 may be controlled by the system 102 based on control instructions shared via the communication network 110.

[0050] In addition, the system 102 can control the UAV group 406 to move at multiple viewpoints in the 3D space 114, and can control the camera group 404 mounted on or integrated with the UAV group 406 to obtain image groups of the human subject 116 from such viewpoints. In at least one embodiment, the system 102 can select such viewpoints to ensure that the FOV area of ​​each of the camera groups 404 captures a portion of the body of the human subject 116 from which a minimum required number of human joints can be subsequently detected. Once captured, the set of anchor cameras 402 and the camera group 404 mounted on or integrated with the UAV group 406 can transmit the set of anchor images and the image group of the human subject 116, respectively, to the system 102. For example, in Figure 6 Calibration of the first camera 404a and the second camera 404b based on the anchor image set and the image group is explained in detail in FIG.

[0051] Figure 5 is a diagram illustrating an exemplary implementation of a camera mounted on or integrated with a UAV and an anchor camera device to acquire images of a human subject according to an embodiment of the present disclosure. Figure 5 Combination Figure 1 , 2 and 3. Figure 5 , a diagram 500 is shown depicting a first camera 502a, a second camera 502b, and a third camera 502c in a 3D space 114. The first camera 502a, the second camera 502b, and the third camera 502c may be mounted on or integrated with a first UAV 504a, a second UAV 504b, and a third UAV 504c, respectively.

[0052] The first UAV 504a can maintain a fixed pose in the 3D space 114. In this article, the fixed pose can correspond to the static position and static orientation (roll, yaw, pitch) of the first UAV 504a in the 3D space 114. In this case, the first camera 502a and the components of the first UAV 504a can be designated as an anchor camera device, which can use the first camera 502a to capture a set of anchor images of the human subject 116 while maintaining a fixed pose. Moreover, the first camera 502a can be pre-calibrated relative to the fixed pose of the first UAV 504a. The remaining UAVs (i.e., the second UAV 504b and the third UAV 504c, collectively referred to as the UAV group) can be controlled to move relative to the fixed pose to obtain a set of images of the human subject 116 from multiple viewpoints.

[0053] The system 102 may control each of the second UAV 504b and the third UAV 504c to move to such viewpoints and capture sets of images from such viewpoints in the 3D space 114. Once captured, the first UAV 504a or the first camera 502a on the first UAV 504a may transmit the set of anchor images of the human subject 116 to the system 102. Also, the camera set (i.e., the second camera 502b and the third camera 502c) may transmit the set of images of the human subject 116 to the system 102. The calibration of the second camera 502b and the third camera 502c (collectively, the camera set) may be based on the anchor image set and the set of images, such as, for example. Figure 6 Explained in detail in .

[0054] Figure 6 is a diagram illustrating an exemplary operation for calibrating a camera on a UAV based on human body joints according to an embodiment of the present disclosure. Figure 6 Combination Figure 1 , 2 , 3, 4, or 5 elements. Figure 6 , a diagram 600 is shown describing exemplary operations from 602 to 610 for calibrating a first camera 404a mounted on or integrated with a first UAV 406a and a second camera 404b mounted on or integrated with a second UAV 406b. The exemplary operations shown in diagram 600 may begin at 602 and may be performed by any computing system, apparatus, or device, such as by Figure 1 System 102 or Figure 3 The circuit system 302 is executed. Although illustrated with discrete blocks, operations associated with one or more of the blocks of diagram 600 may be divided into additional blocks, combined into fewer blocks, or eliminated, depending on the particular implementation.

[0055] At 602, data acquisition may be performed. For data acquisition, the circuit system 202 may control a set 402 of anchor cameras (which includes a first anchor camera 402a and a second anchor camera 402b) to capture a set of anchor images of a human subject 116. The set 402 of anchor cameras may be fixed at certain positions in the 3D space 114, and the set 402 of anchor cameras may capture a set of anchor images from such positions. In addition, the circuit system 202 may control the UAV group 406 to move the UAV group 406 at multiple viewpoints in the 3D space 114. Thereafter, the camera group 404 mounted on or integrated with the UAV group 406 may be controlled to acquire a set of images of the human subject 116 from multiple viewpoints. At 602, the circuit system 202 may control a set of anchor cameras 402 (which includes a first anchor camera 402a and a second anchor camera 402b) to capture a set of anchor images of a human subject 116. Figure 6, an example of an anchor image 602a and an image 602b acquired from one of the camera set 404 is shown. Both the anchor image 602a and the image 602b include the human subject 116. The circuit system 202 may receive the set of anchor images from the set 402 of anchor cameras and the set of images from the camera set 404 mounted on or integrated with the UAV set 406 via the communication network 110.

[0056] At 604, a first set of 2D positions of human joints may be determined from the set of anchor images. The circuit system 202 may be configured to determine a first set of 2D positions of human joints of the human subject 116 from a set of anchor images (e.g., anchor image 602a). For such determination, the circuit system 202 may apply the ML model 104 to each anchor image (such as anchor image 602a) in the set of anchor images. The ML model 104 may correspond to a human joint detection framework, which may include a neural network trained for a human joint detection task. The 2D positions of the human joints (i.e., the first set of 2D positions) may be represented by equation (1), given as follows:

[0057]

[0058] in,

[0059] The 2D position of the i-th detected human joint;

[0060] The confidence value of the i-th detected human joint;

[0061] the type of the i-th detected human joint (e.g., left wrist or right elbow); and

[0062] M: The total number of detected human joints.

[0063] As shown, anchor image 602a may be captured from a rear view of human subject 116. Human joint locations 612a and 612b may correspond to the head and neck of human subject 116 in anchor image 602a, respectively. Additionally, human joint locations 612c, 612d, and 612e may correspond to the left shoulder joint, left elbow joint, and left wrist joint of human subject 116 in anchor image 602a. Human joint locations 612f, 612g, and 612h may correspond to the right shoulder joint, right elbow joint, and right wrist joint of human subject 116 in anchor image 602a. In addition, human joint positions 612i, 612j and 612k can correspond to the left waist joint, left knee joint and left ankle joint of the human subject 116 in the anchor image 602a, and human joint positions 612l, 612m and 612n can correspond to the right waist joint, right knee joint and right ankle joint of the human subject 116 in the anchor image 602a.

[0064] At 606, a second set of 2D positions of human joints may be detected from the set of images. The circuit system 202 may be configured to detect a second set of 2D positions of human joints of the human subject 116 from the set of images (e.g., image 602b). To detect the second set of 2D positions, the circuit system 202 may apply the ML model 104 to each image from the set of images (e.g., image 602b). The second set of 2D positions may also be given by equation (1).

[0065] As shown, circuit system 202 may determine a second set of 2D positions of body joints of human subject 116 from image 602b as body joint positions 614a to 614n. Image 602b may be captured from the front of human subject 116. Body joint positions 614a and 614b may correspond to the head and neck of human subject 116 in image 602b. Additionally, body joint positions 614c, 614d, and 614e may correspond to the right shoulder joint, right elbow joint, and right wrist joint of human subject 116 in image 602b. Body joint positions 614f, 614g, and 614h may correspond to the left shoulder joint, left elbow joint, and left wrist joint of human subject 116 in image 602b. In addition, human joint positions 614i, 616j and 614k can correspond to the right waist joint, right knee joint and right ankle joint of the human subject 116 in image 602b, and human joint positions 614l, 614m and 614n can correspond to the left waist joint, left knee joint and left ankle joint of the human subject 116 in image 602b.

[0066] At 608, 3D key points 618 can be calculated. The circuit system 202 can use the first set of determined 2D positions of the human joints to calculate the 3D positions of the human joints of the human subject 116 based on triangulation. The circuit system 202 can specify the 3D key points 618 as the calculated 3D positions of the human joints of the human subject 116 relative to the 3D coordinate system 620. The calculation of the 3D key points 618 is explained herein. A simple perspective camera model can be used to convert the human joint position values ​​from the 2D image plane to the 3D real space. In this scenario, the perspective projection ray (with weights and labels) of each detected human joint point can be obtained by using predetermined intrinsic camera parameters and a three-dimensional special Euclidean group (i.e., se(3) parameters). The perspective camera model according to the above scenario can be expressed as follows:

[0067]

[0068] in,

[0069] the 3D normalized direction vector of the i-th body joint from the n-th camera; and

[0070] The nth camera position in 3D.

[0071] The circuit system 202 can perform triangulation using a first set of 2D positions of human joints along 3D lines (i.e., rays) using a known or calibrated optical center of a calibrated camera (e.g., the first anchor camera 402a). Consider "e" as a normalized oriented 3D vector associated with the 2D position (from the first set of 2D positions) of a certain human joint along the 3D line passing through the calibrated optical center. "e" can represent the normalized oriented 3D vector of a specific human joint along the 3D line from the calibrated camera. In addition, consider "a" as a 3D point on the 3D line passing through the calibrated optical center. In this article, the 3D point "a" on the 3D line can represent the camera position of a calibrated camera (e.g., the first anchor camera 402a) intrinsic to a known pose. The distance between the 3D line and the 3D point can be given by equations (3.1) and (3.2), as shown below:

[0072] d 2 = l 2 sin 2 θ=|e X(xa)| 2 (3.1)

[0073] d 2 = l 2 (1-cos 2 θ)=|xa| 2-{e.(xa)} 2 (3.2)

[0074] in,

[0075] θ may represent the angle between the normalized 3D vector "e" and the 3D line, x may represent the point on the 3D line from which a perpendicular line may be dropped on the 3D vector "e", and

[0076] l can represent the distance between 3D point "a" and point "x".

[0077] In one embodiment, the circuit system 202 may determine a weighted least squares of a plurality of 3D lines and a corresponding plurality of normalized 3D vectors “e” of human joints for different positions of an anchor camera (such as the first anchor camera 402 a). n In this paper, each of the multiple 3D lines passes through multiple 3D points “a n "The corresponding 3D points in these 3D points "a n ” represents the different positions of the anchor camera. The weighted least squares is expressed in equations (4.1) and (4.2) as follows:

[0078]

[0079]

[0080] in,

[0081] w n can represent the weight of the nth 3D line, and

[0082] d n Can represent a 3D point "a n ” and the distance between the nth 3D line.

[0083] Equation (4.2) can be expanded for the x-axis coordinates of the 3D points along the 3D line. The expansion of equation (4.2) is given by equation (5), as follows:

[0084]

[0085] where i may represent the index of the x-axis coordinate of the 3D point along the 3D line.

[0086] Circuitry 202 may apply a partial derivative operation to equation (5) to obtain equation (6), as shown below:

[0087]

[0088] Where j can represent the index of the y-axis coordinate of the 3D point along the 3D line. Assuming the stationary condition Equations (7.1) and (7.2) can be obtained as follows:

[0089]

[0090]

[0091] Based on equation (7.2), the linear system Ax = B can be obtained. The linear system of equations (8.1) and (8.2) for the linear system Ax = B is provided as follows:

[0092]

[0093]

[0094] The circuit system 202 can be based on the solution of the linear system of equations (8.1) and (8.2) {x i} Calculate a first 3D key point of 3D key points 618. The first 3D key point can be associated with one of the human body joints. Similarly, the circuit system 202 can calculate the remaining 3D key points associated with the remaining human body joints using the corresponding 2D positions from the first set of determined 2D positions by following the above-mentioned method.

[0095] At 610, the camera group 404 mounted on or integrated with the UAV group 406 can be calibrated. The circuit system 202 can calibrate the first camera 404a and the second camera 404b by estimating the 3D pose (position and orientation) of each of the first camera 404a and the second camera 404b. In order to estimate the target parameters for camera calibration, including the camera position and orientation of multiple cameras (e.g., the first camera 404a and the second camera 404b), 6 degrees of freedom (DoF), according to Equation 9, a special Euclidean se (3) Lie algebra parameterization can be used:

[0096]

[0097] in,

[0098] N: Total number of cameras.

[0099] The special Euclidean Lie algebra se(3) can have 6 elements. The first 3 elements It can represent a special orthogonal Lie algebra (i.e., so(3)), where the last three rotation elements can represent the translation element T(3)∈R in 3D space respectively 3 For practical purposes, we can use the exponential mapping se(3) → SE(3) to obtain a matrix representation.

[0100] In an embodiment, the value of the intrinsic calibration parameter of each of the plurality of cameras (including the first anchor camera 402a, the second anchor camera 402b, the first camera 404a, and the second camera 404b) may be predetermined or given.

[0101] In an embodiment, the circuit system 202 may be configured to set a global gauge for estimating the 3D pose of each camera (e.g., the first camera 404a and the second camera 404b). To set the global gauge, the circuit system 202 may set the position and orientation of the 0th camera (e.g., the first anchor camera 402a) to a predetermined 3D point and direction and expressed as:

[0102]

[0103] The circuit system 202 may set the relative x distance between the 0th camera (e.g., the first anchor camera 402a) and the first camera (e.g., the second anchor camera 402b) to:

[0104]

[0105] The circuit system 202 can set a global gauge for calibrating the camera group 404 (eg, the first camera 404a and the second camera 404b) using "common gauge fixing" in a bundle adjustment based technique by using equations (10) and (11).

[0106] The circuit system 202 can be configured to calibrate the camera group 404 (e.g., the first camera 404a and the second camera 404b) by estimating the 3D pose (i.e., external camera parameters, such as position (i.e., translation parameters) and orientation (i.e., rotation parameters)) of each camera in the camera group 404. For such camera calibration, an optimization problem regarding 2D reprojection errors can be formulated. The circuit system 202 can determine the 2D reprojection errors between the 3D key points 618 and the second set of determined 2D positions of the human body joints. Thereafter, calibration can be performed by minimizing the determined 2D reprojection errors using the formulated optimization problem.

[0107] For example, to calibrate the first camera 404a, the 3D key points 618 may be reprojected onto the image plane of the first camera 404a. Thereafter, the reprojection error between each reprojected 3D key point and the 2D position of the corresponding human joint may be determined. The objective function of the optimization problem may be formulated and solved to minimize the reprojection error. The above-mentioned method may also be repeated for the second camera 404b. Since the camera on each UAV in the UAV group 406 may not be calibrated initially, the camera calibration based on the 3D key points of the camera may result in an estimate of the value of the extrinsic calibration parameters (i.e., 3D pose) of each camera installed on the corresponding UAV in the UAV group 108. The 2D reprojection error may correspond to the geometric error between the point projected based on the perspective camera model view and the measured position of the point in the 3D coordinate system 620. The 2D reprojection error may quantify the estimated proximity of the 3D key point relative to the action projection of the point (e.g., the second set of 2D positions of the determined human joints).

[0108] For example, consider that a 3D keypoint (target point) of a human joint is represented by coordinates (X, Y, Z) in 3D coordinate system 620. The 3D keypoint can be reprojected as a 2D point on an image plane associated with the optical center of a camera (such as one of camera set 404). The 2D point can be represented by (x p / z p ,y p / z p ). Here, x p ,y p 、z p The x-axis, y-axis, and z-axis coordinates of the perspective projection of the reprojected 3D point on the image plane can be respectively obtained from the second set of 2D positions of the human joints in the image set acquired by the camera set 404. The 2D positions of the human joints on the image plane of the camera can be obtained from the 2D point (x d ,y d ) represents a 2D point (x p / z p ,y p / z p ) and the 2D point (x d ,y d The 2D reprojection error δ between ) can be expressed by equation (12) as follows:

[0109] δ 2 =(x p / z p -x d ) 2 +(y p / z p -y d ) 2 (12)

[0110] Here, xp, yp and zp are given by equation (13), as follows:

[0111]

[0112] Where M is the 4×4 camera projection matrix, including the intrinsic parameters of the camera.

[0113] The aforementioned technique for calibrating the camera group 404 may be referred to as a frame-by-frame calibration technique. Such a process may be parallelized to be processed on several processors of several electronic devices or computing devices. The present disclosure may also be applicable to other techniques for calibrating the camera group 404, such as a camera calibration technique based on time accumulation.

[0114] In an embodiment, at least one UAV in the UAV group 406 (e.g., the first UAV 406a) may include a position sensor. The position sensor may be one of a global navigation satellite system (GNSS) receiver, an inertial measurement unit (IMU), a sensing camera, an infrared marker sensor, or a pattern code scanner. In an embodiment, the circuit system 202 of the system 102 or the circuit system 302 (of the first UAV 406a) may be configured to calibrate each camera mounted on or integrated with the remaining UAVs (e.g., the second UAV 406b) from the UAV group 406 based on the absolute position information obtained from the position sensor of the corresponding UAV (e.g., the first UAV 406a).

[0115] In another embodiment, each camera in the camera group 404 (e.g., the first camera 404a and the second camera 404b) may be further calibrated based on values ​​of intrinsic calibration parameters of the camera group 404 mounted on or integrated with the UAV group 406. Examples of such intrinsic calibration parameters may include, but are not limited to, field of view (i.e., focal length and screen size associated with a camera), aspect ratio, and distortion parameters.

[0116] Figure 7 is a flow chart illustrating exemplary operations for calibrating a camera array mounted on or integrated with a UAV array according to an embodiment of the present disclosure. Figure 7 , a flowchart 700 is shown. The flowchart 700 is combined with Figure 1 , 2 , 3, 4, 5 and 6. The method from 702 to 716 can be performed by any computing system (such as Figure 1 System 102 or Figure 3 The operations in flowchart 700 may start at 702 and proceed to 704.

[0117] At 704, a set of anchor images of human subjects 116 may be received from the set of anchor camera devices 106. In at least one embodiment, the circuitry 202 may be configured to receive the set of anchor images from the set 106 of anchor camera devices.

[0118] At 706, a set of images of a human subject 116 from multiple viewpoints in a 3D space 114 may be received from a camera set 112 mounted on or integrated with the UAV set 108. In at least one embodiment, the circuit system 202 may be configured to receive the set of images from the camera set.

[0119] At 708, for the human subject 116 in each anchor image in the received set of anchor images, a first set of 2D positions of body joints may be determined. In at least one embodiment, the circuit system 202 may determine a first set of 2D positions of body joints of the human subject 116 in each anchor image in the set of anchor images.

[0120] At 710, for human subject 116 in each image in the received set of images, a second set of 2D positions of human joints may be determined. In at least one embodiment, circuit system 202 may determine a second set of 2D positions of human joints for human subject 116 in each image in the set of images.

[0121] At 712, the 3D position of the human joint in a 3D space (e.g., 3D space 114) may be calculated as a 3D key point based on triangulation using the first set of determined 2D positions of the human joint. In at least one embodiment, the circuit system 202 may calculate the 3D position of the human joint in the 3D space 114 as a 3D key point based on triangulation using the first set of determined 2D positions of the human joint.

[0122] At 714, a 2D reprojection error between the 3D keypoint and the second set of determined 2D positions may be determined. In at least one embodiment, circuitry 202 may determine a 2D reprojection error between the 3D keypoint and the second set of determined 2D positions.

[0123] At 716, each camera in camera group 112 may be calibrated by minimizing the determined 2D reprojection error. In at least one embodiment, circuit system 202 may calibrate each camera in camera group 112 by minimizing the determined 2D reprojection error. Control may pass to end.

[0124] Although flowchart 700 is illustrated as discrete operations, such as 704, 706, 708, 710, 712, 714, and 716, in certain embodiments, such discrete operations may be further divided into additional operations, combined into fewer operations, or eliminated, depending on the particular implementation, without departing from the essence of the disclosed embodiments.

[0125] Various embodiments of the present disclosure may provide a non-transitory computer-readable medium and / or storage medium, and / or a non-transitory machine-readable medium and / or storage medium, on which a set of instructions executable by a machine and / or computer including one or more circuits is stored. The set of instructions may be executed by a machine and / or computer to perform operations including receiving a set of anchor images of a human subject from a set of anchor camera devices. The operation also includes receiving a set of images of a human subject from multiple viewpoints in a three-dimensional (3D) space from a camera group mounted on or integrated with a UAV group. In addition, the operation includes determining a first set of two-dimensional (2D) positions of human joints for a human subject in each anchor image in the received set of anchor images. The operation also includes determining a second set of 2D positions of human joints for a human subject in each image in the received set of images. The operation also includes calculating the 3D positions of human joints as 3D key points in a 3D space based on triangulation using the first set of determined 2D positions of human joints. The operations also include determining a 2D reprojection error between the 3D keypoints and the second set of determined 2D positions, and calibrating each camera in the camera set by minimizing the determined 2D reprojection error.

[0126] Certain embodiments of the present disclosure may be found in systems and methods for calibrating a camera mounted on or integrated with a UAV. Various embodiments of the present disclosure may provide a system (such as system 102 ( Figure 1 )), which may include circuitry (such as circuitry 202 ( Figure 2 )). The circuit system 202 may be configured to receive a video from a set of anchor camera devices (such as the set of anchor camera devices 106 ( Figure 1 )) receives a human subject (eg, human subject 116 ( Figure 1 )). In addition, the circuit system 202 can be configured to receive an anchor image set from a UAV group (such as UAV group 108 ( Figure 1 )) on a camera group (e.g., a first camera 112a and a second camera 112b ( Figure 1 )) from a 3D space (e.g., 3D space 114 ( Figure 1)) receives a group of images of a human subject 116 from multiple viewpoints in the received anchor image set. For the human subject 116 in each anchor image in the received anchor image set, the circuit system 202 can determine a first set of 2D positions of human joints. In addition, for the human subject 116 in each image in the received image group, the circuit system 202 can determine a second set of 2D positions of human joints. In addition, the circuit system 202 can use the determined first set of 2D positions of human joints to calculate the 3D positions of the human joints in the 3D space as 3D key points based on triangulation. The circuit system 202 can determine a 2D reprojection error between the 3D key points and the determined second set of 2D positions. In addition, the circuit system 202 can calibrate each camera in the camera group (e.g., the first camera 112a and the second camera 112b) by minimizing the determined 2D reprojection error.

[0127] In an embodiment, the 3D space 114 may be associated with one of an outdoor space, an indoor space, or a studio setting for volumetric capture. The circuit system 202 may also be configured to control the UAV group 108 to move the UAV group at multiple viewpoints in the 3D space 114. The circuit system 202 may also control the camera group (e.g., the first camera 112a and the second camera 112b) to acquire a set of images of the human subject 116 from multiple viewpoints. In an embodiment, the set of anchor camera devices may include at least one camera (e.g., the first camera 502a ( Figure 5 )), which may be mounted on a UAV (such as the first UAV 504a ( Figure 5 )) or integrated with a UAV, and can be configured to maintain a fixed posture while a UAV group (such as can include a second UAV 504b and a third UAV 504c ( Figure 5 )'s UAV group) is controlled to move relative to a fixed posture to acquire a set of images.

[0128] In an embodiment, the set of anchor camera devices may include at least one pre-calibrated camera (such as the first anchor camera 402a ( Figure 4 )), which can be fixed to a position in the 3D space 114 while the UAV group (including, for example, the first UAV 406a and the second UAV 406b) is controlled to acquire the image set. The set of anchor camera devices may include at least one pre-calibrated camera that is movably coupled to the remote camera movement assembly. The circuit system 202 may also be configured to control the set of anchor camera devices in the 3D space 114 to acquire the set of anchor images.

[0129] The circuit system 202 may be configured to generate a machine learning (ML) model (e.g., the ML model 104 ( Figure 1)) is applied to each anchor image in the received set of anchor images to determine a first set of 2D positions of human joints. As a human joint detection framework, the ML model 104 may include a neural network trained for a 2D human joint detection task. The circuit system 202 may also be configured to determine a second set of 2D positions of human joints by applying the ML model 104 on each image in the received set of images.

[0130] In an embodiment, the calibration of each UAV (e.g., the first UAV 406a and the second UAV 406b) can correspond to an estimate of the 3D pose of the corresponding camera (e.g., the first camera 404a and the second camera 404b). The 3D pose can include the 3D position and orientation of the corresponding camera in the 3D space 114.

[0131] In an embodiment, at least one UAV in the UAV group 406 (e.g., the first UAV 406a ( Figure 4 )) may include a position sensor. Examples of the position sensor may include, but are not limited to, a global navigation satellite system (GNSS) receiver, an inertial measurement unit (IMU), a sensing camera, an infrared marker sensor, or a pattern code scanner. The circuit system 202 may be configured to further calibrate each camera in the camera group (e.g., the first UAV 406a) based on the absolute position information obtained from the position sensor of the corresponding UAV (e.g., the first UAV 406a). Figure 4 In an embodiment, the circuit system 202 may be configured to calibrate each camera in the camera group (eg, the first camera 404a and the second camera 404b) further based on the value of the intrinsic calibration parameter of the corresponding camera.

[0132] The present disclosure may be implemented in hardware or in a combination of hardware and software. The present disclosure may be implemented in a centralized manner, in at least one computer system, or in a distributed manner, wherein different elements may be distributed over a plurality of interconnected computer systems. A computer system or other device suitable for performing the methods described herein may be suitable. The combination of hardware and software may be a general-purpose computer system with a computer program that, when loaded and executed, can control the computer system so that it performs the methods described herein. The present disclosure may be implemented in hardware including a portion of an integrated circuit that also performs other functions.

[0133] The present disclosure may also be embedded in a computer program product comprising all the features enabling the implementation of the methods described herein and which, when loaded into a computer system, is capable of carrying out these methods. In this document, a computer program refers to any expression of a set of instructions in any language, code or symbolic form, which is intended to cause a system with information processing capabilities to perform specific functions directly, or after one or both of the following: a) conversion into another language, code or symbol; b) reproduction in a different material form.

[0134] Although the present disclosure is described with reference to certain embodiments, it will be appreciated by those skilled in the art that various changes may be made and equivalents may be substituted without departing from the scope of the present disclosure. In addition, many modifications may be made to adapt specific circumstances or materials to the teachings of the present disclosure without departing from the scope of the present disclosure. Therefore, it is intended that the present disclosure is not limited to the specific embodiments disclosed, but that the present disclosure will include all embodiments falling within the scope of the appended claims.

Claims

1. A system comprising: A circuit system communicatively coupled to a set of anchor camera devices and a camera group mounted on or integrated with an unmanned aerial vehicle (UAV) group, wherein the circuit system is configured to: receiving a set of anchor images of human subjects from a set of anchor camera devices; receiving a set of images of a human subject from a plurality of viewpoints in three-dimensional (3D) space from a set of cameras implementing a human tracking method to follow the human subject in 3D space; determining a first set of two-dimensional (2D) positions of human body joints for a human subject in each anchor image in the received set of anchor images; determining a second set of 2D positions of human body joints for a human subject in each image in the received set of images; Using the first set of determined 2D positions of human joints, calculating 3D positions of the human joints in 3D space as 3D key points based on triangulation; determining a 2D reprojection error between the 3D keypoints and the second set of determined 2D positions; and Calibrate each camera in the camera set by minimizing the determined 2D reprojection error, in The set of anchor camera devices includes at least one pre-calibrated camera, At least one camera in the camera group mounted on or integrated with the UAV group has the function of an infrared (IR) tag sensor, The infrared marker sensor tracks and captures infrared light emitted from an infrared light source fixed at a predetermined position in the 3D space as one or more infrared images, The circuit system applies 3D pose estimation techniques on the one or more infrared images to determine target parameters for camera calibration including 6 degrees of freedom for a UAV having a camera with infrared marker sensor functionality.

2. The system of claim 1, wherein the 3D space is associated with one of: an outdoor space, an indoor space, or a studio setting for volumetric capture.

3. The system of claim 1 , wherein the circuit system is further configured to: controlling the group of UAVs to move the group of UAVs at multiple viewpoints in 3D space; and A camera group mounted on or integrated with the UAV group is controlled to acquire a group of images of the human subject from the plurality of viewpoints.

4. A system according to claim 3, wherein the anchor camera device set includes at least one camera, which is mounted on the UAV or integrated with the UAV and is configured to maintain a fixed posture, while the camera group is controlled to move relative to the fixed posture to obtain an image group.

5. The system of claim 3, wherein the at least one pre-calibrated camera is fixed to a position in 3D space while the camera set is controlled to acquire the set of images.

6. The system of claim 3, wherein the at least one pre-calibrated camera is movably coupled to a remote camera movement assembly.

7. The system of claim 1, wherein the circuit system is further configured to control a set of anchor camera devices in a 3D space to acquire a set of anchor images.

8. A system according to claim 1, wherein the circuit system is configured to determine a first set of 2D positions of human joints by applying a machine learning (ML) model on each anchor image in a received set of anchor images, and wherein the ML model includes a neural network trained for a 2D human joint detection task as a human joint detection framework.

9. The system of claim 1 , wherein the circuit system is configured to determine a second set of 2D locations of human joints by applying an ML model on each image in the received set of images, and wherein the ML model comprises a neural network trained for a 2D human joint detection task as a human joint detection framework.

10. The system of claim 1, wherein The calibration of each camera in the camera set corresponds to an estimate of the 3D pose of the corresponding camera, and The 3D pose includes the 3D position and orientation of the corresponding camera in 3D space.

11. The system of claim 1, wherein At least one UAV in the group of UAVs includes a position sensor, and The position sensor is one of: a global navigation satellite system (GNSS) receiver, an inertial measurement unit (IMU), a sensing camera, an infrared marker sensor, or a pattern code scanner.

12. The system of claim 11, wherein the circuit system is configured to calibrate each camera in the camera group further based on absolute position information obtained from a position sensor of the corresponding UAV.

13. The system of claim 1, wherein the circuit system is configured to calibrate each camera in the camera group further based on a value of an intrinsic calibration parameter of the corresponding camera.

14. A method comprising: receiving a set of anchor images of human subjects from a set of anchor camera devices; receiving a set of images of a human subject from a plurality of viewpoints in three-dimensional (3D) space from a camera set mounted on or integrated with the UAV set implementing a human tracking method to follow the human subject in 3D space; determining a first set of two-dimensional (2D) positions of human body joints for a human subject in each anchor image in the received set of anchor images; determining a second set of 2D positions of human body joints for a human subject in each image in the received set of images; Using the first set of determined 2D positions of human joints, calculating 3D positions of the human joints in 3D space as 3D key points based on triangulation; determining a 2D reprojection error between the 3D keypoints and the second set of determined 2D positions; and Calibrate each camera in the camera set by minimizing the determined 2D reprojection error, in The set of anchor camera devices includes at least one pre-calibrated camera, At least one camera in the camera group mounted on or integrated with the UAV group has the function of an infrared (IR) tag sensor, The infrared marker sensor tracks and captures infrared light emitted from an infrared light source fixed at a predetermined position in the 3D space as one or more infrared images, A 3D pose estimation technique is applied on the one or more infrared images to determine target parameters for camera calibration including 6 degrees of freedom for a UAV having a camera with infrared marker sensor functionality.

15. The method according to claim 14, further comprising: controlling the UAV group to move the UAV group at multiple viewpoints in 3D space; as well as A camera group mounted on or integrated with the UAV group is controlled to acquire a group of images of the human subject from the plurality of viewpoints.

16. The method according to claim 14, further comprising: determining a first set of 2D locations of human body joints by applying a machine learning (ML) model on each anchor image in the received set of anchor images; as well as A second set of 2D positions of human joints is determined by applying the ML model on each image in the received image set, wherein the ML model as a human joint detection framework includes a neural network trained for a 2D human joint detection task.

17. The method according to claim 14, wherein The calibration of each camera corresponds to an estimate of the 3D pose of the corresponding UAV, and The 3D pose includes the 3D position and orientation of the corresponding UAV in 3D space.

18. The method according to claim 14, wherein At least one UAV in the group of UAVs includes a position sensor, and The position sensor is one of: a global navigation satellite system (GNSS) receiver, an inertial measurement unit (IMU), a sensing camera, an infrared marker sensor, or a pattern code scanner.

19. The method of claim 18, further comprising calibrating each UAV in the UAV group based on absolute position information obtained from a position sensor of a corresponding camera-equipped UAV.

20. A non-transitory computer-readable medium having computer-implemented instructions stored thereon, the instructions, when executed by a computer in a system, causing the system to perform operations comprising: receiving a set of anchor images of human subjects from a set of anchor camera devices; receiving a set of images of a human subject from a plurality of viewpoints in three-dimensional (3D) space from a camera set mounted on or integrated with the UAV set implementing a human tracking method to follow the human subject in 3D space; determining a first set of two-dimensional (2D) positions of human body joints for a human subject in each anchor image in the received set of anchor images; determining a second set of 2D positions of human body joints for a human subject in each image in the received set of images; Using the first set of determined 2D positions of human joints, calculating 3D positions of the human joints in 3D space as 3D key points based on triangulation; determining a 2D reprojection error between the 3D keypoints and the second set of determined 2D positions; and Calibrate each camera in the camera set by minimizing the determined 2D reprojection error, in The set of anchor camera devices includes at least one pre-calibrated camera, At least one camera in the camera group mounted on or integrated with the UAV group has the function of an infrared (IR) tag sensor, The infrared marker sensor tracks and captures infrared light emitted from an infrared light source fixed at a predetermined position in the 3D space as one or more infrared images, A 3D pose estimation technique is applied on the one or more infrared images to determine target parameters for camera calibration including 6 degrees of freedom for a UAV having a camera with infrared marker sensor functionality.

Images