Camera calibration methods, motion capture image generation methods, motion capture equipment and media

Abstract

This disclosure discloses a camera calibration method, a motion capture image generation method, a motion capture device, and a medium. One specific implementation of the method includes: performing internal calibration processing on the camera; determining an initial frame number and initial camera pose information; and, based on the initial frame number and initial camera pose information, performing the following calibration steps: controlling the camera to acquire a checkerboard image; generating a sensor image position information group based on the checkerboard image; and, for each preset transmitter identifier, performing the following steps: determining a sensor spatial position information group; determining a target image position information set and a target spatial position information set based on the sensor image position information group and the sensor spatial position information group; and, in response to determining that the initial frame number meets a preset frame number condition, generating coordinate system transformation information based on each determined target image position information set and each determined target spatial position information set. This implementation can improve the accuracy and real-time performance of capturing user movements.

Description

Technical Field

[0001] The embodiments disclosed herein relate to the field of computer technology, specifically to camera calibration methods, motion capture image generation methods, motion capture devices, and media. Background Technology

[0002] In the field of motion tracking and recognition technology, such as in AR / VR gesture interaction, motion capture devices are needed to accurately capture user movements. Then, cameras are used to collect images of the user's movements, and finally, the motion postures captured by the motion capture device are projected onto the user's motion images for motion tracking and recognition. Currently, the methods commonly used to capture user movements are: mechanical motion capture devices that track and measure motion trajectories physically, or acoustic motion capture; optical motion capture devices that use optical principles to capture and locate objects; and inertial navigation motion capture devices that use inertial navigation sensors (AHRS, Attitude and Bearing Reference System) and IMUs (Inertial Measurement Unit) to measure the acceleration, orientation, tilt angle, and other characteristics of the captured person or object.

[0003] However, in practice, the following technical problems often arise when using the above methods to capture user movements: Mechanical motion capture devices require users to wear a complete set of motion capture mechanical devices, which is inconvenient for users; Acoustic motion capture devices have poor real-time performance and low accuracy, and are easily affected by ambient noise; Optical motion capture devices, due to the use of multiple sophisticated and complex high-speed optical cameras, are easily confused and obstructed by each other; Inertial navigation motion capture devices are generally large in size, cumbersome to operate, and due to the characteristics of the sensors, inertial motion capture is prone to error accumulation, resulting in low accuracy, thus causing low accuracy and poor real-time performance in capturing user movements.

[0004] The information disclosed in this background section is only intended to enhance the understanding of the background of the present disclosure concept, and therefore may contain information that does not constitute prior art known to those skilled in the art. Summary of the Invention

[0005] The summary portion of this disclosure is intended to provide a brief overview of the concepts, which will be described in detail in the detailed description portion. This summary portion is not intended to identify key or essential features of the claimed technical solutions, nor is it intended to limit the scope of the claimed technical solutions.

[0006] Some embodiments of this disclosure provide a camera calibration method, a user motion capture image generation method, an electromagnetic motion capture device, and a computer-readable medium to address the technical problems mentioned in the background section above.

[0007] In a first aspect, some embodiments of this disclosure provide a camera calibration method applied to an electromagnetic motion capture device. The method includes: performing internal calibration processing on a camera included in the electromagnetic motion capture device to obtain camera parameter information; determining an initial frame number and initial camera pose information; and, based on the initial frame number and initial camera pose information, performing the following calibration steps: controlling the camera to acquire a checkerboard image based on the camera parameter information and initial camera pose information; generating a sensor image position information group corresponding to each electromagnetic sensor included in the electromagnetic motion capture device based on the checkerboard image; and, for each preset transmitter identifier, performing the following steps: determining a sensor spatial position information group corresponding to the preset transmitter identifier, wherein each preset transmitter identifier corresponds to an electromagnetic signal transmitter included in the electromagnetic motion capture device; determining a target image position information set and a target spatial position information set based on the sensor image position information group and the sensor spatial position information group; and, in response to determining that the initial frame number meets a preset frame number condition, generating coordinate system transformation information based on the determined target image position information sets and the determined target spatial position information sets.

[0008] Optionally, the above calibration steps further include: in response to determining that the initial frame number does not meet the above preset frame number condition, updating the initial frame number and the initial camera pose information, and using the updated initial frame number as the initial frame number and the updated initial camera pose information as the initial camera pose information, and continuing to execute the above calibration steps.

[0009] Optionally, determining the target image location information set and the target spatial location information set based on the sensor image location information set and the sensor spatial location information set includes: dividing the sensor image location information set to obtain a training image location information set and a verification image location information set; dividing the sensor spatial location information set to obtain a training spatial location information set and a verification spatial location information set; generating initial coordinate system transformation information based on the training image location information set and the training spatial location information set; generating a reprojection location information set based on the camera parameter information, the initial coordinate system transformation information, and the verification spatial location information set; generating a projection average error based on the reprojection location information set and the verification image location information set; and, in response to determining that the projection average error satisfies a preset error condition, determining the training image location information set as the target image location information set and determining the training spatial location information set as the target spatial location information set.

[0010] Optionally, the above method further includes: in response to determining that the average projection error does not meet the preset error condition, updating the initial camera pose information, and using the updated initial camera pose information as the initial camera pose information to continue performing the above calibration steps.

[0011] Optionally, determining the sensor spatial position information group corresponding to the preset transmitter identifier includes: controlling each electromagnetic sensor corresponding to the preset transmitter identifier to collect electromagnetic signals to obtain a sensor pose information group; and generating a sensor spatial position information group based on the sensor pose information group.

[0012] Optionally, the above-mentioned generation of coordinate system transformation information based on the determined target image position information set and the determined target spatial position information set includes: for each target spatial position information included in the determined target spatial position information set, performing the following steps: generating transmitter distance information based on the target spatial position information; in response to determining that the transmitter distance information satisfies a preset distance condition, determining the target spatial position information as target sensor spatial position information, and determining the target image position information corresponding to the target spatial position information in the determined target image position information set as target sensor image position information; and generating coordinate system transformation information based on the determined target sensor spatial position information and the determined target sensor image position information.

[0013] Optionally, the above method further includes: generating transmitter coordinate transformation information based on the generated coordinate system transformation information.

[0014] Secondly, some embodiments of this disclosure provide a user motion capture image generation method, applied to an electromagnetic motion capture device. The method includes: controlling a camera included in the electromagnetic motion capture device to acquire user motion images according to preset camera parameter information; and controlling each electromagnetic signal transmitter included in the electromagnetic motion capture device to perform an activation operation, wherein the preset camera parameter information is pre-generated according to the method described in any implementation of the first aspect; controlling each electromagnetic sensor included in the electromagnetic motion capture device to acquire electromagnetic signals to obtain a sensor pose information group; and generating each transmitter identifier corresponding to each preset transmitter identifier based on the sensor pose information group. Sensor spatial location information groups; for each preset transmitter identifier, perform the following steps: determine the sensor spatial location information group corresponding to the preset transmitter identifier in each of the above sensor spatial location information groups as the target spatial location information group; generate sensor image location information groups according to the preset camera parameter information, the target spatial location information group and the preset coordinate system transformation information corresponding to the preset transmitter identifier, wherein the preset coordinate system transformation information is pre-generated by the method described in any implementation of the first aspect; perform image reconstruction processing on the user action image according to each generated sensor image location information group to obtain a user action capture image.

[0015] Thirdly, some embodiments of this disclosure provide an electromagnetic motion capture device, comprising: at least one electromagnetic signal transmitter for transmitting electromagnetic signals; at least one electromagnetic sensor for sensing electromagnetic signals; a camera for acquiring images; one or more processors; and a storage device storing one or more programs, wherein when the one or more programs are executed by the one or more processors, the one or more processors implement the method described in any implementation of the first or second aspect above.

[0016] Fourthly, some embodiments of this disclosure provide a computer-readable medium having a computer program stored thereon, wherein the computer program, when executed by a processor, implements the method described in any implementation of the first or second aspect.

[0017] The various embodiments of this disclosure have the following beneficial effects: the camera calibration method of some embodiments of this disclosure can improve the accuracy and real-time performance of capturing user movements. Specifically, the reasons for low accuracy and poor real-time performance in capturing user movements are as follows: mechanical motion capture devices require users to wear a complete set of motion capture mechanical devices, which is inconvenient for users; acoustic motion capture devices have poor real-time performance and low accuracy in object capture, and are easily affected by ambient noise; optical motion capture devices use multiple precise and complex high-speed optical cameras, which are easily confused and obstructed; inertial navigation motion capture devices are generally large in size, cumbersome to operate, and due to the characteristics of the sensors, inertial motion capture is prone to error accumulation, resulting in low accuracy, thus causing low accuracy and poor real-time performance in capturing user movements. Based on this, the camera calibration method of some embodiments of this disclosure is applied to electromagnetic motion capture devices. First, the camera included in the electromagnetic motion capture device is internally calibrated to obtain camera parameter information. Thus, the intrinsic parameter information of the camera can be obtained, which can be used to improve the image quality acquired by the camera. Secondly, the initial frame rate and initial camera pose information are determined, and based on these information, the following calibration steps are performed: First, the camera is controlled to acquire a checkerboard image based on the camera parameter information and the initial camera pose information. This yields a checkerboard image for calibration, which can then be used to identify the positions of each electromagnetic sensor in the camera coordinate system within the checkerboard. Second, based on the checkerboard image, a sensor image position information group corresponding to each electromagnetic sensor in the electromagnetic motion capture device is generated. This yields the pixel coordinates of each electromagnetic sensor in the checkerboard, which can then be used to calculate the coordinate transformation information between the transmitter coordinate system and the camera coordinate system. Third, for each preset transmitter identifier, the following steps are performed: Step one, the sensor spatial position information group corresponding to the preset transmitter identifier is determined. Each preset transmitter identifier corresponds to an electromagnetic signal transmitter in the electromagnetic motion capture device. This yields the position of each electromagnetic sensor in the corresponding transmitter coordinate system. Step two: Based on the aforementioned sensor image position information set and sensor spatial position information set, determine the target image position information set and the target spatial position information set. This allows us to obtain the positions of each electromagnetic sensor used to calculate the coordinate transformation information between the transmitter coordinate system and the camera coordinate system. Step three: In response to the initial frame count meeting a preset frame count condition, generate coordinate system transformation information based on the determined target image position information sets and the determined target spatial position information sets. Therefore, when enough image frames have been acquired, the coordinate transformation information between the transmitter coordinate system and the camera coordinate system can be obtained based on the positions of each corresponding electromagnetic sensor in the transmitter coordinate system and the camera coordinate system, thus enabling the external parameter calibration of the camera relative to the corresponding electromagnetic signal transmitter.Because electromagnetic motion capture equipment can be used for user motion capture, errors caused by external interference can be reduced, and there is no need to wait for signal transmission and reception, thus improving the real-time performance and accuracy of motion capture. Furthermore, when using electromagnetic motion capture equipment to capture user movements, the camera and various electromagnetic signal transmitters can be jointly calibrated beforehand, thereby improving the accuracy of user motion capture and recognition. Therefore, the accuracy and real-time performance of capturing user movements can be improved. Attached Figure Description

[0018] The above and other features, advantages, and aspects of the embodiments of this disclosure will become more apparent from the accompanying drawings and the following detailed description. Throughout the drawings, the same or similar reference numerals denote the same or similar elements. It should be understood that the drawings are schematic, and elements are not necessarily drawn to scale.

[0019] Figure 1 This is a schematic diagram illustrating an application scenario of a camera calibration method according to some embodiments of the present disclosure;

[0020] Figure 2 This is a flowchart of some embodiments of the camera calibration method according to the present disclosure;

[0021] Figure 3 This is a schematic diagram of the relative pose relationship between the electromagnetic sensor coordinate system and the electromagnetic signal transmitter coordinate system in a camera calibration method according to some embodiments of this disclosure;

[0022] Figure 4 These are flowcharts of other embodiments of the camera calibration method according to this disclosure;

[0023] Figure 5 This is a flowchart of some embodiments of the user motion capture image generation method of this disclosure;

[0024] Figure 6 This is a schematic diagram showing the transformation relationship between the electromagnetic sensor coordinate system, the coordinate system of each electromagnetic signal transmitter, and the camera coordinate system according to some embodiments of the user motion capture image generation method of this disclosure;

[0025] Figure 7 This is a schematic diagram of the hardware structure of an electromagnetic motion capture device suitable for implementing some embodiments of the present disclosure. Detailed Implementation

[0026] Embodiments of this disclosure will now be described in more detail with reference to the accompanying drawings. While some embodiments of this disclosure are shown in the drawings, it should be understood that this disclosure can be implemented in various forms and should not be construed as limited to the embodiments set forth herein. Rather, these embodiments are provided to provide a more thorough and complete understanding of this disclosure. It should be understood that the accompanying drawings and embodiments of this disclosure are for illustrative purposes only and are not intended to limit the scope of protection of this disclosure.

[0027] It should also be noted that, for ease of description, only the parts relevant to the invention are shown in the accompanying drawings. Unless otherwise specified, the embodiments and features described in this disclosure can be combined with each other.

[0028] It should be noted that the concepts of "first" and "second" mentioned in this disclosure are used only to distinguish different devices, modules or units, and are not used to limit the order of functions performed by these devices, modules or units or their interdependencies.

[0029] It should be noted that the terms "a" and "a plurality of" used in this disclosure are illustrative rather than restrictive, and those skilled in the art should understand that, unless otherwise expressly indicated in the context, they should be understood as "one or more".

[0030] The names of messages or information exchanged between multiple devices in the embodiments of this disclosure are for illustrative purposes only and are not intended to limit the scope of such messages or information.

[0031] This disclosure will now be described in detail with reference to the accompanying drawings and embodiments.

[0032] Figure 1 This is a schematic diagram illustrating an application scenario of the camera calibration method according to some embodiments of this disclosure.

[0033] exist Figure 1In the application scenario, firstly, the electromagnetic motion capture device 101 can perform internal calibration processing on the camera 1011 included in the electromagnetic motion capture device 101 to obtain camera parameter information 102. Then, the electromagnetic motion capture device 101 determines the initial frame number 103 and the initial camera pose information 104, and performs the following calibration steps based on the initial frame number and the initial camera pose information: First, control the camera to acquire a checkerboard image 105 based on the camera parameter information 102 and the initial camera pose information 104; Second, generate sensor image position information groups 106 corresponding to each electromagnetic sensor included in the electromagnetic motion capture device based on the checkerboard image 105; For each preset transmitter identifier 107, perform the following steps: determine the corresponding preset transmitter identifier 107. Sensor spatial position information group 108 of transmitter identifier 107, wherein each preset transmitter identifier 107 corresponds to the electromagnetic signal transmitter included in the electromagnetic motion capture device; target image position information set 109 and target spatial position information set 110 are determined according to the sensor image position information group 106 and the sensor spatial position information group 108; in response to determining that the initial frame number meets the preset frame number condition 111, coordinate system transformation information 112 is generated according to each determined target image position information set 109 and each determined target spatial position information set 110.

[0034] It should be understood that Figure 1 The number of electromagnetic motion capture devices shown is merely illustrative. Any number of electromagnetic motion capture devices can be used depending on the implementation requirements.

[0035] Continue to refer to Figure 2 The diagram illustrates a flow 200 of some embodiments of a camera calibration method according to the present disclosure. This camera calibration method, applied to an electromagnetic motion capture device, includes the following steps:

[0036] Step 201: Perform internal calibration processing on the camera included in the electromagnetic motion capture device to obtain camera parameter information.

[0037] In some embodiments, the execution subject of the camera calibration method (e.g. Figure 1The electromagnetic motion capture device shown can perform internal calibration processing on the camera included in the electromagnetic motion capture device to obtain camera parameter information. The electromagnetic motion capture device can be a motion capture device that uses the strength of a magnetic field for position and orientation tracking. The electromagnetic motion capture device can include, but is not limited to, at least one electromagnetic signal transmitter, at least one electromagnetic sensor, and a camera. The electromagnetic signal transmitter can be used to emit electromagnetic signals to form a magnetic field. For example, the electromagnetic signal transmitter can be a transmitter in an electromagnetic motion tracking system. The electromagnetic sensor can be used to sense electromagnetic signals in the magnetic field generated by the emitted electromagnetic signals and convert the sensed electromagnetic signals into electrical signals. It should be noted that when using the electromagnetic motion capture device to capture a user's posture, each electromagnetic sensor needs to be placed at the limb of the user whose posture needs to be captured. As an example, when using the electromagnetic motion capture device to capture a user's hand posture, each electromagnetic sensor needs to be placed at one of the 21 joints of the hand. The camera can be a monocular fisheye camera. The camera parameter information can characterize the camera's intrinsic parameters. The camera parameter information can include, but is not limited to, the camera's intrinsic parameter matrix and distortion parameters. The intrinsic parameter matrix can be expressed by the following formula:

[0038]

[0039] Where K represents the intrinsic parameter matrix. p represents the pixel coordinate system. x represents the x-axis in the 3D coordinate system. y represents the y-axis in the 3D coordinate system. p It can represent the x-axis in a pixel coordinate system. y p It can represent the y-axis in the pixel coordinate system. f can represent the focal length. It can represent the camera's focal length on the x-axis. 'c' can represent the camera's focal length on the y-axis. 'c' can represent the coordinates of the camera's principal point. This can represent the camera's position on the x-axis.

[0040] The coordinates of the principal point on the [the graph]. It can represent the principal point coordinates of the camera on the y-axis.

[0041] The above distortion parameters can be expressed by the following formula:

[0042] d = [k1, k2, k3, k4].

[0043] Where d represents the distortion parameter, k represents the radial distortion coefficient, k1 represents the first-order radial distortion coefficient, k2 represents the second-order radial distortion coefficient, k3 represents the third-order radial distortion coefficient, and k4 represents higher-order radial distortion coefficients.

[0044] In practice, the aforementioned executing entity can perform internal calibration processing on the cameras included in the electromagnetic motion capture device using a preset checkerboard calibration method to obtain camera parameter information. This preset checkerboard calibration method can be a method for calibrating camera intrinsic parameters based on a checkerboard pattern. For example, the preset checkerboard calibration method can be the Zhang Zhengyou checkerboard calibration method.

[0045] Optionally, the aforementioned electromagnetic motion capture device may further include a clock synchronization device. This clock synchronization device can be used to synchronize the time when the camera acquires images with the time when the electromagnetic sensor senses electromagnetic signals. This improves the accuracy of camera calibration.

[0046] Step 202: Determine the initial frame number and initial camera pose information, and based on the initial frame number and initial camera pose information, perform the following calibration steps:

[0047] Step 2021: Control the camera to acquire a chessboard image based on camera parameter information and initial camera pose information.

[0048] In some embodiments, the aforementioned executing entity can control the aforementioned camera to acquire a checkerboard image based on the aforementioned camera parameter information and initial camera pose information. The aforementioned initial frame count can be the initial number of acquired images. The aforementioned initial camera pose information can characterize the pose of the camera when initially capturing the image. The aforementioned initial camera pose information can include, but is not limited to, the camera shooting angle. The aforementioned camera shooting angle can be the camera's shooting angle. The shooting angle can include shooting height, shooting direction, and shooting distance. The shooting height can be the height of the camera relative to the camera calibration checkerboard. Shooting height can be divided into three types: level shot, overhead shot, and low shot. The shooting direction can be the direction of the camera relative to the camera calibration checkerboard. The shooting direction can be, but is not limited to, one of the following: front angle, side angle, oblique side angle, and back angle. The shooting distance can be the relative distance between the camera and the camera calibration checkerboard. The checkerboard image can be an image of the camera calibration checkerboard. The aforementioned camera calibration checkerboard can be a checkerboard used for camera calibration. For example, the aforementioned camera calibration checkerboard can be the AprilTag checkerboard. It should be noted that the electromagnetic sensors included in the aforementioned electromagnetic motion capture device are evenly distributed and fixed at the corner points of the camera calibration checkerboard. The center point of each electromagnetic sensor is aligned with the corresponding corner point. In practice, the aforementioned execution entity can determine the preset initial frame number and the preset initial camera pose information as the initial frame number. The preset initial frame number can be a pre-set initial frame number. For example, the preset initial frame number can be 1. The preset initial camera pose information can be pre-set initial camera pose information. For example, the preset initial camera pose information can be "flat shot, frontal angle 0, distance 50 cm".

[0049] Step 2022: Based on the checkerboard image, generate sensor image position information groups for each electromagnetic sensor included in the corresponding electromagnetic motion capture device.

[0050] In some embodiments, the execution entity can generate a sensor image position information group corresponding to each electromagnetic sensor included in the electromagnetic motion capture device based on the chessboard image. The sensor image position information in the sensor image position information group corresponds one-to-one with the electromagnetic sensors in each electromagnetic sensor. The sensor image position information can characterize the position of the corresponding electromagnetic sensor in the chessboard image. In practice, firstly, the execution entity can perform corner detection and recognition processing on the chessboard using a preset corner recognition algorithm to obtain a corner detection information set corresponding to each chessboard corner. The preset corner recognition algorithm can be a pre-set algorithm for detecting and recognizing the position of chessboard corners. For example, the preset corner recognition algorithm can be the AprilTag positioning algorithm. The chessboard corners in each chessboard corner can correspond one-to-one with the corner detection information in the corner detection information set. The chessboard corners can be corners in the camera-calibrated chessboard. The corner detection information can characterize the position of the corresponding chessboard corner in the chessboard image. The corner detection information can include, but is not limited to, the pixel coordinates of the chessboard corners. The aforementioned corner pixel coordinates can be the pixel coordinates of the corner points in the chessboard grid image under the camera coordinate system. Then, for each preset electromagnetic sensor identifier, the corner detection information corresponding to the preset electromagnetic sensor identifier in the aforementioned corner detection information set is determined as the sensor image position information. Here, the preset electromagnetic sensor identifier can be a pre-defined unique identifier for the corresponding electromagnetic sensor. For example, the preset electromagnetic sensor identifier can be the corresponding electromagnetic sensor number. The corner detection information corresponding to the preset electromagnetic sensor identifier can be: the corner detection information corresponding to the target chessboard grid corner point. The target chessboard grid corner point can be: the same chessboard grid corner point as the corner point where the electromagnetic sensor corresponding to the preset electromagnetic sensor identifier is placed on the camera calibration chessboard grid. Finally, the determined sensor image position information is determined as a sensor image position information group.

[0051] Step 2023: For each preset transmitter identifier, perform the following steps:

[0052] Step 20231: Determine the sensor spatial location information group corresponding to the preset transmitter identifier.

[0053] In some embodiments, the execution entity may determine a sensor spatial position information group corresponding to the preset transmitter identifier. Each preset transmitter identifier corresponds to an electromagnetic signal transmitter included in the electromagnetic motion capture device. The preset transmitter identifier can be a unique identifier for the corresponding electromagnetic signal transmitter. The sensor spatial position information in the sensor spatial position information group can correspond one-to-one with the electromagnetic sensors in each electromagnetic sensor group. The sensor spatial position information in the sensor spatial position information group represents the coordinate point of the corresponding electromagnetic sensor in the coordinate system of the electromagnetic signal transmitter corresponding to the preset transmitter identifier. In practice, in response to detecting the checkerboard origin offset information corresponding to the preset transmitter identifier, the execution entity may perform the following steps for each sensor image position information included in the sensor image position information group:

[0054] The first step is to determine the x-axis spatial coordinate by summing the x-axis offset of the checkerboard origin offset information and the corresponding x-axis pixel coordinates of the sensor image position information. The checkerboard origin offset information characterizes the positional offset between the center point of the checkerboard and the center point of the electromagnetic signal transmitter corresponding to the preset transmitter identifier. This checkerboard origin offset information may include, but is not limited to, x-axis offset, y-axis offset, and z-axis offset. The x-axis offset can be the horizontal distance between the center point of the checkerboard and the center point of the electromagnetic signal transmitter. The y-axis offset can be the front-back distance between the center point of the checkerboard and the center point of the electromagnetic signal transmitter. The z-axis offset can be the vertical distance between the center point of the checkerboard and the center point of the electromagnetic signal transmitter. It should be noted that the checkerboard origin offset information can be obtained by the calibration user through physical measurement methods. The checkerboard origin offset information can be sent by the calibration user to the execution entity via a terminal, or it can be input by the calibration user through an interface on the execution entity. The aforementioned calibration users can be users who calibrate cameras and electromagnetic motion capture equipment.

[0055] The second step is to determine the y-axis spatial coordinate by summing the y-axis offset included in the chessboard origin offset information and the corresponding y-axis pixel coordinates included in the sensor image position information.

[0056] The third step is to determine the z-axis offset, which is included in the above chessboard origin offset information, as the z-axis spatial coordinate.

[0057] The fourth step is to determine the above-mentioned x-axis spatial coordinates, y-axis spatial coordinates, and z-axis spatial coordinates as the sensor's spatial position information.

[0058] Then, the determined spatial location information of each sensor is defined as a sensor spatial location information group.

[0059] In some optional implementations of certain embodiments, the aforementioned execution entity may determine the sensor spatial location information group corresponding to the aforementioned preset transmitter identifier through the following steps:

[0060] The first step involves controlling each electromagnetic sensor corresponding to the preset transmitter identifier to acquire electromagnetic signals, thereby obtaining a sensor pose information set. The sensor pose information in this set represents the position and orientation of the corresponding electromagnetic sensor within the corresponding electromagnetic signal transmitter coordinate system. This electromagnetic signal transmitter coordinate system can be a Cartesian right-hand coordinate system with the center point of the electromagnetic signal transmitter corresponding to the preset transmitter identifier as its origin. The sensor pose information in the set may include, but is not limited to, sensor translation vectors and sensor rotation matrices. The sensor translation vector represents the offset of the center point of the corresponding electromagnetic sensor relative to the origin of the electromagnetic signal transmitter coordinate system. The sensor rotation matrix represents the rotation angle of the electromagnetic sensor coordinate system relative to the electromagnetic signal transmitter coordinate system. The electromagnetic sensor coordinate system can be a Cartesian right-hand coordinate system with the center point of the electromagnetic sensor as its origin. The relative pose relationship between the electromagnetic sensor coordinate system and the electromagnetic signal transmitter coordinate system can be as follows: Figure 3 As shown. The sensor translation vector described above can be expressed by the following formula:

[0061] T s =[x s y s , z s ].

[0062] Where T can represent a translation vector, and S can represent an electromagnetic sensor. s It can represent the sensor translation vector. x s This can represent the offset of the center point of the electromagnetic sensor relative to the origin of the coordinate system of the electromagnetic signal transmitter in the X-axis direction. s This can represent the offset of the center point of the electromagnetic sensor relative to the origin of the coordinate system of the electromagnetic signal transmitter in the Y-axis direction. s This can represent the offset of the center point of the electromagnetic sensor relative to the origin of the coordinate system of the electromagnetic signal transmitter along the Z-axis. The sensor rotation matrix can be expressed by the following formula:

[0063]

[0064] Here, R can represent the rotation matrix. sThe rotation matrix of the sensor can be represented by ψ. ψ represents the yaw angle. θ represents the pitch angle. φ represents the roll angle. The yaw angle represents the angle of deflection of the electromagnetic sensor coordinate system relative to the electromagnetic signal transmitter reference coordinate system along the Z-axis. The pitch angle represents the angle of deflection of the electromagnetic sensor coordinate system relative to the electromagnetic signal transmitter reference coordinate system along the Y-axis. The roll angle represents the angle of deflection of the electromagnetic sensor coordinate system relative to the electromagnetic signal transmitter reference coordinate system along the Z-axis.

[0065] The second step is to generate a sensor spatial position information group based on the aforementioned sensor pose information group. In practice, for each sensor pose information group included in the aforementioned sensor pose information group, the executing entity can input the sensor translation vector and sensor rotation matrix included in the aforementioned sensor pose information into the following formula to obtain the sensor spatial coordinates as the sensor spatial position information.

[0066] (x source ,y source , z source ) = R s ×(0, 0, 0) T +T s .

[0067] Among them, (x source y souce , z source This can represent the sensor's spatial coordinates. The aforementioned sensor spatial coordinates can be the coordinates (DT) of the center point of the corresponding electromagnetic sensor in the source coordinate system. s This can represent the sensor rotation matrix. T s It can represent the sensor translation vector.

[0068] Therefore, the position of the electromagnetic sensor in the coordinate system of the electromagnetic signal transmitter can be determined by sensing the electromagnetic signal through the electromagnetic sensor.

[0069] Step 20232: Determine the target image location information set and the target spatial location information set based on the sensor image location information set and the sensor spatial location information set.

[0070] In some embodiments, the executing entity can determine a target image location information set and a target spatial location information set based on the sensor image location information set and the sensor spatial location information set. The target image location information in the target image location information set can be sensor image location information used as a sample calibration camera and an electromagnetic motion capture device. The target spatial location information in the target spatial location information set can be sensor spatial location information used as a sample calibration camera and an electromagnetic motion capture device. In practice, the executing entity can determine the sensor image location information set as the target image location information set and the sensor spatial location information set as the target spatial location information set.

[0071] In some optional implementations of certain embodiments, the execution entity may determine the target image location information set and the target spatial location information set based on the sensor image location information set and the sensor spatial location information set through the following steps:

[0072] The first step involves dividing the aforementioned sensor image location information group into a training image location information group and a verification image location information group. In practice, the executing entity can determine the location information of each sensor image corresponding to a preset training sensor identifier set within the aforementioned sensor image location information group as the training image location information group, and determine the location information of each sensor image corresponding to a preset verification sensor identifier set within the aforementioned sensor image location information group as the verification image location information group. The preset training sensor identifier in the aforementioned preset training sensor identifier set can be a pre-defined identifier for an electromagnetic sensor used for training. The preset verification sensor identifier in the aforementioned preset verification sensor identifier set can be a pre-defined identifier for an electromagnetic sensor used for verification. The sensor image location information corresponding to the preset training sensor identifier can be: sensor image location information where the corresponding preset electromagnetic sensor identifier is the same as the preset training sensor identifier. The sensor image location information corresponding to the preset verification sensor identifier can be: sensor image location information where the corresponding preset electromagnetic sensor identifier is the same as the preset verification sensor identifier.

[0073] The second step involves dividing the aforementioned sensor spatial location information group into a training spatial location information group and a verification spatial location information group. In practice, the executing entity can determine each sensor spatial location information corresponding to a preset training sensor identifier set within the aforementioned sensor spatial location information group as the training spatial location information group, and determine each sensor spatial location information corresponding to a preset verification sensor identifier set within the aforementioned sensor spatial location information group as the verification spatial location information group. Specifically, the sensor spatial location information corresponding to the preset training sensor identifier can be: sensor spatial location information whose corresponding preset electromagnetic sensor identifier is the same as the preset training sensor identifier. Similarly, the sensor spatial location information corresponding to the preset verification sensor identifier can be: sensor spatial location information whose corresponding preset electromagnetic sensor identifier is the same as the preset verification sensor identifier.

[0074] The third step involves generating initial coordinate system transformation information based on the aforementioned training image location information set and training spatial location information set. This initial coordinate system transformation information characterizes the transformation relationship between the source coordinate system and the camera coordinate system. In practice, the executing entity can use the Perspective-n-Point (PnP) algorithm to solve for the aforementioned training image location information set and training spatial location information set, obtaining the initial rotation matrix and initial translation vector. Then, the aforementioned initial rotation matrix and initial translation vector are used as the initial coordinate system transformation information.

[0075] Fourth, based on the aforementioned camera parameter information, the aforementioned initial coordinate system transformation information, and the aforementioned verification spatial location information group, a reprojection location information group is generated. In practice, for each verification spatial location information in the aforementioned verification spatial location information group, the executing entity can input the intrinsic parameter matrix included in the aforementioned camera parameter information, the initial rotation matrix and the initial translation vector included in the aforementioned initial coordinate system transformation information, and the aforementioned verification spatial location information into the following formula to obtain the reprojection pixel coordinates, and determine the aforementioned reprojection pixel coordinates as the reprojection location information.

[0076]

[0077] Among them, (x′ t y′ t , z′ t R can represent the three-dimensional coordinates of the center point of the corresponding electromagnetic sensor in the camera coordinate system after reprojection. source This can represent the initial rotation matrix. (x′) source y′ source , z′ source This can represent the verification of spatial location information. source It can represent the initial translation vector. t can represent the reprojection. (u′)t , v′ t (u, v) can represent the pixel coordinates of the center point of the corresponding electromagnetic sensor after reprojection in the pixel coordinate system, i.e., the reprojected pixel coordinates (u, v).

[0078] Then, the determined reprojection position information is defined as a reprojection position information group.

[0079] Fifth, based on the aforementioned reprojection location information group and the aforementioned verification image location information group, generate the average projection error. In practice, firstly, for each reprojection location information included in the aforementioned reprojection location information group, the executing entity can perform the following sub-steps:

[0080] The first sub-step is to determine the verification image position information corresponding to the reprojection position information in the above verification image position information group as the target verification image position information.

[0081] The second sub-step involves inputting the target verification image location information and the reprojection location information into the following formula to obtain the projection error.

[0082]

[0083] Here, err can represent the projection error. j This can represent the x-coordinate of the point corresponding to the above target verification image location information in the pixel coordinate system. j The vertical coordinate (y) of the point corresponding to the target verification image location information in the pixel coordinate system can be represented by 'u'. The horizontal coordinate (x) of the pixel coordinate system can be represented by 'v'.

[0084] Then, the average value of the obtained projection errors is determined as the projection average error.

[0085] Step 6: In response to determining that the above-mentioned average projection error satisfies a preset error condition, the above-mentioned training image location information set is determined as the target image location information set, and the above-mentioned training spatial location information set is determined as the target spatial location information set. The preset error condition can be that the above-mentioned average projection error is less than a preset error value. The preset error value can be a pre-set average projection error.

[0086] Optionally, the execution entity may also update the initial camera pose information in response to determining that the average projection error does not meet the preset error condition, and use the updated initial camera pose information as the initial camera pose information to continue executing the calibration steps. In practice, the execution entity may update the initial camera pose information according to a preset adjustment angle in response to determining that the average projection error does not meet the preset error condition, and use the updated initial camera pose information as the initial camera pose information to continue executing the calibration steps. The preset adjustment angle can be a pre-set angle for adjusting the shooting angle. For example, the preset adjustment angle can be "proof angle 30 degrees".

[0087] Therefore, the initial coordinate system transformation information can be verified. If the coordinate system transformation information does not meet the requirements, the checkerboard image can be re-acquired for camera calibration, thereby improving the accuracy of the obtained coordinate system transformation information.

[0088] Step 20233: In response to determining that the initial frame number meets the preset frame number condition, coordinate system transformation information is generated based on the determined set of position information of each target image and the determined set of spatial position information of each target.

[0089] In some embodiments, the execution entity may, in response to determining that the initial frame number satisfies a preset frame number condition, generate coordinate system transformation information based on the determined sets of target image position information and the determined sets of target spatial position information. The preset frame number condition may be that the initial frame number equals a preset frame number threshold. The preset frame number threshold may be a pre-set maximum value for the number of frames acquired in the chessboard image acquisition process. In practice, the execution entity may use the Perspective-n-Point (PnP) algorithm to solve for the determined sets of target image position information and the determined sets of target spatial position information to obtain a rotation matrix and a translation vector. Then, the rotation matrix and the translation vector are used to determine the coordinate system transformation information.

[0090] In some optional implementations of certain embodiments, the execution entity can generate coordinate system transformation information based on the determined set of location information for each target image and the determined set of spatial location information for each target through the following steps:

[0091] The first step is to perform the following sub-steps for each target spatial location information included in the determined target spatial location information sets:

[0092] The first sub-step involves generating transmitter distance information based on the aforementioned target spatial location information. In practice, the executing entity can determine the transmitter distance information as the Euclidean distance between the aforementioned target spatial location information and the origin of the corresponding source coordinate system.

[0093] The second sub-step involves, in response to determining that the transmitter distance information satisfies a preset distance condition, defining the target spatial location information as the target sensor spatial location information, and defining the target image location information corresponding to the target spatial location information in the determined target image location information set as the target sensor image location information. The preset distance condition can be: the transmitter distance information is less than a preset transmitter distance. The preset transmitter distance can be: a pre-set maximum distance between the electromagnetic sensor and the electromagnetic signal transmitter, representing stable signal reception.

[0094] The second step involves generating coordinate system transformation information based on the determined spatial and image position information of each target sensor. In practice, the executing entity can use the Perspective-n-Point (PnP) algorithm to solve for the determined spatial and image position information of each target sensor, obtaining a rotation matrix and a translation vector. Then, the rotation matrix and translation vector are used to determine the coordinate system transformation information.

[0095] Therefore, sample points corresponding to electromagnetic sensors that are actually far away from the corresponding electromagnetic signal transmitter can be eliminated, thereby reducing the error caused by the inaccuracy of the electromagnetic signals detected by the electromagnetic sensors and thus improving the accuracy of the obtained coordinate system transformation information.

[0096] Optionally, the execution entity may also, in response to determining that the initial frame number does not meet the preset frame number condition, update the initial frame number and initial camera pose information, and use the updated initial frame number and the updated initial camera pose information as the initial camera pose information, and continue to execute the above calibration steps. In practice, the execution entity may, in response to determining that the initial frame number does not meet the preset frame number condition, update the initial camera pose information according to a preset adjustment angle, and determine the sum of the initial frame number and the preset increment step as the updated initial frame number, and use the updated initial frame number and the updated initial camera pose information as the initial camera pose information, and continue to execute the above calibration steps. The preset increment step can be a pre-set value for increasing the number of image frames acquired each time. For example, the preset increment step can be 1. Therefore, by acquiring checkerboard images from different angles multiple times for camera calibration, the generalization performance of the obtained coordinate system transformation information can be improved, thereby improving the accuracy of motion capture by the electromagnetic motion capture device.

[0097] The various embodiments of this disclosure have the following beneficial effects: the camera calibration method of some embodiments of this disclosure can improve the accuracy and real-time performance of capturing user movements. Specifically, the reasons for low accuracy and poor real-time performance in capturing user movements are as follows: mechanical motion capture devices require users to wear a complete set of motion capture mechanical devices, which is inconvenient for users; acoustic motion capture devices have poor real-time performance and low accuracy in object capture, and are easily affected by ambient noise; optical motion capture devices use multiple precise and complex high-speed optical cameras, which are easily confused and obstructed; inertial navigation motion capture devices are generally large in size, cumbersome to operate, and due to the characteristics of the sensors, inertial motion capture is prone to error accumulation, resulting in low accuracy, thus causing low accuracy and poor real-time performance in capturing user movements. Based on this, the camera calibration method of some embodiments of this disclosure is applied to electromagnetic motion capture devices. First, the camera included in the electromagnetic motion capture device is internally calibrated to obtain camera parameter information. Thus, the intrinsic parameter information of the camera can be obtained, which can be used to improve the image quality acquired by the camera. Secondly, the initial frame rate and initial camera pose information are determined, and based on these information, the following calibration steps are performed: First, the camera is controlled to acquire a checkerboard image based on the camera parameter information and the initial camera pose information. This yields a checkerboard image for calibration, which can then be used to identify the positions of each electromagnetic sensor in the camera coordinate system within the checkerboard. Second, based on the checkerboard image, a sensor image position information group corresponding to each electromagnetic sensor in the electromagnetic motion capture device is generated. This yields the pixel coordinates of each electromagnetic sensor in the checkerboard, which can then be used to calculate the coordinate transformation information between the transmitter coordinate system and the camera coordinate system. Third, for each preset transmitter identifier, the following steps are performed: Step one, the sensor spatial position information group corresponding to the preset transmitter identifier is determined. Each preset transmitter identifier corresponds to an electromagnetic signal transmitter in the electromagnetic motion capture device. This yields the position of each electromagnetic sensor in the corresponding transmitter coordinate system. Step two: Based on the aforementioned sensor image position information set and sensor spatial position information set, determine the target image position information set and the target spatial position information set. This allows us to obtain the positions of each electromagnetic sensor used to calculate the coordinate transformation information between the transmitter coordinate system and the camera coordinate system. Step three: In response to the initial frame count meeting a preset frame count condition, generate coordinate system transformation information based on the determined target image position information sets and the determined target spatial position information sets. Therefore, when enough image frames have been acquired, the coordinate transformation information between the transmitter coordinate system and the camera coordinate system can be obtained based on the positions of each corresponding electromagnetic sensor in the transmitter coordinate system and the camera coordinate system, thus enabling the external parameter calibration of the camera relative to the corresponding electromagnetic signal transmitter.Because electromagnetic motion capture equipment can be used for user motion capture, errors caused by external interference can be reduced, and there is no need to wait for signal transmission and reception, thus improving the real-time performance and accuracy of motion capture. Furthermore, when using electromagnetic motion capture equipment to capture user movements, the camera and various electromagnetic signal transmitters can be jointly calibrated beforehand, thereby improving the accuracy of user motion capture and recognition. Therefore, the accuracy and real-time performance of capturing user movements can be improved.

[0098] Further reference Figure 4 The diagram illustrates a flow 400 of another embodiment of the camera calibration method. This flow 400, applied to an electromagnetic motion capture device, includes the following steps:

[0099] Step 401: Perform internal calibration processing on the camera included in the electromagnetic motion capture device to obtain camera parameter information.

[0100] Step 402: Determine the initial frame number and initial camera pose information, and based on the initial frame number and initial camera pose information, perform the following calibration steps:

[0101] Step 4021: Control the camera to acquire a chessboard image based on the camera parameter information and the initial camera pose information.

[0102] Step 4022: Based on the above checkerboard image, generate sensor image position information groups corresponding to each electromagnetic sensor included in the above electromagnetic motion capture device.

[0103] Step 4023: For each preset transmitter identifier, perform the following steps:

[0104] Step 40231: Determine the sensor spatial location information group corresponding to the preset transmitter identifier.

[0105] Step 40232: Determine the target image location information set and the target spatial location information set based on the above sensor image location information set and the above sensor spatial location information set.

[0106] Step 40233: In response to determining that the initial frame number meets the preset frame number condition, coordinate system transformation information is generated based on the determined set of position information of each target image and the determined set of spatial position information of each target.

[0107] In some embodiments, the specific implementation of steps 401-40233 and the resulting technical effects can be found in [reference needed]. Figure 2 Steps 201-20233 in the corresponding embodiments will not be repeated here.

[0108] Step 403: Generate transmitter coordinate transformation information based on the generated coordinate system transformation information.

[0109] In some embodiments, the aforementioned execution entity (e.g. Figure 1 The electromagnetic motion capture device shown can generate transmitter coordinate transformation information based on the generated coordinate system transformation information. This transmitter coordinate transformation information characterizes the coordinate transformation between each electromagnetic signal transmitter coordinate system and a pre-defined electromagnetic signal transmitter coordinate system used for direct coordinate transformation with the camera coordinate system in practical applications. In practice, the executing entity can determine the coordinate system transformation information corresponding to a preset transmitter identifier from the generated coordinate system transformation information as the target coordinate system transformation information. This preset transmitter identifier can be a pre-defined identifier of an electromagnetic signal transmitter that directly transforms with the camera coordinate system. Then, for each preset transmitter identifier, the coordinate system transformation information corresponding to the preset transmitter identifier and the target coordinate system transformation information are input into the following formula to obtain the coordinate transformation information corresponding to the preset transmitter identifier. This coordinate transformation information may include a coordinate transformation rotation matrix and a coordinate transformation translation vector.

[0110] R i0 =R wi T ·R w0

[0111] T i0 =T wi T ·(T w0 -T wi ).

[0112] Wherein, "above" can represent the source coordinate system corresponding to the preset transmitter identifier. "0" can represent the source coordinate system corresponding to the preset transformed transmitter identifier. "w" can represent the camera coordinate system. R i0 This can represent the rotation matrix between the source coordinate system corresponding to the preset transmitter identifier and the aforementioned source coordinate system corresponding to the preset transmitter identifier, i.e., the coordinate transformation rotation matrix. i0 This can represent the translation vector between the source coordinate system corresponding to the preset transmitter identifier and the aforementioned source coordinate system corresponding to the preset transmitter identifier, i.e., the coordinate transformation translation vector. R wi This can represent the rotation matrix included in the coordinate system transformation information corresponding to the aforementioned preset transmitter identifier. w0 This can represent the rotation matrix included in the above target coordinate system transformation information. wi This can represent the translation vector included in the coordinate system transformation information corresponding to the aforementioned preset transmitter identifier. w0This can represent the translation vector included in the target coordinate system transformation information mentioned above. It should be noted that the preset transmitter identifier here does not include transmitter identifiers that are identical to the preset transformation transmitter identifier.

[0113] Finally, the obtained coordinate transformation information is determined as the transmitter coordinate transformation information.

[0114] Optionally, the aforementioned executing entity may also generate transmitter coordinate transformation information based on the determined set of spatial location information for each target through the following steps:

[0115] The first step is to determine the spatial location information of each target corresponding to the preset conversion transmitter identifier as the first spatial location information group.

[0116] The second step is to identify the preset transmitter identifiers that are not preset conversion transmitter identifiers among the preset transmitter identifiers as the target transmitter identifier group.

[0117] Third, for each target transmitter identifier included in the above target transmitter identifier group, perform the following steps:

[0118] The first sub-step involves determining the spatial location information of each target corresponding to the spatial location information of the target transmitter as the second spatial location information group.

[0119] The second sub-step involves solving the first and second spatial position information groups using a preset point cloud registration algorithm to obtain coordinate transformation information. The preset point cloud registration algorithm can be a pre-defined method for finding the optimal rotation matrix and translation vector between two coordinate systems. For example, the preset point cloud registration algorithm could be the Kabsch point cloud registration algorithm.

[0120] The fourth step is to determine the obtained coordinate transformation information as the transmitter coordinate transformation information.

[0121] from Figure 4 It can be seen from this that, with Figure 2 Compared to the description of some corresponding embodiments, Figure 4 The flow 400 of the camera calibration method in some corresponding embodiments illustrates the steps of generating coordinate transformation information between the coordinate systems of various electromagnetic signal transmitters. Therefore, the schemes described in these embodiments can obtain the transformation relationships between the coordinate systems of various electromagnetic signal transmitters. Thus, when using multiple electromagnetic signal transmitters, the spatial coordinates can be uniformly transformed to the same electromagnetic signal transmitter coordinate system first, and then transformed to the camera coordinate system. This improves the consistency of the attitude transformations of each electromagnetic sensor to the camera coordinate system, thereby improving the accuracy of motion capture.

[0122] Continue to refer to Figure 5 The diagram illustrates a flow 500 of some embodiments of a user motion capture image generation method according to the present disclosure. This user motion capture image generation method, applied to an electromagnetic motion capture device, includes the following steps:

[0123] Step 501: Control the camera included in the electromagnetic motion capture device to acquire user motion images according to preset camera parameter information, and control each electromagnetic signal transmitter included in the electromagnetic motion capture device to perform an activation operation.

[0124] In some embodiments, the execution entity can control the camera included in the electromagnetic motion capture device to acquire user motion images according to preset camera parameter information, and control each electromagnetic signal transmitter included in the electromagnetic motion capture device to perform an activation operation. The preset camera parameter information is camera parameter information pre-generated according to the camera calibration method described in steps 201-20233. It should be noted that each electromagnetic sensor included in the electromagnetic motion capture device needs to be installed on the user's limb to be detected. The user motion image can represent the user's actions and posture. The activation operation can be the operation of activating the electromagnetic signal transmitter. As an example, if the electromagnetic motion capture device needs to detect the user's hand posture, then each electromagnetic sensor is installed at 21 key points representing the user's hand posture. The user motion image can represent the user's hand posture.

[0125] Step 502: Control the electromagnetic sensors included in the electromagnetic motion capture device to collect electromagnetic signals and obtain sensor pose information group.

[0126] Step 503: Generate a spatial position information group for each sensor corresponding to each preset transmitter identifier based on the sensor pose information group.

[0127] In some embodiments, the execution entity described above may perform the following steps for each preset transmitter identifier:

[0128] The first step is to determine the sensor pose information of each sensor corresponding to the preset transmitter identifier in the above sensor pose information group as the target sensor pose information group.

[0129] The second step is to generate a sensor spatial position information group based on the aforementioned target sensor pose information group. The specific implementation method and the resulting technical effects are the same as some optional implementation methods in step 20231, and will not be repeated here.

[0130] Step 504: For each preset transmitter identifier, perform the following steps:

[0131] Step 5041: Determine the sensor spatial location information group corresponding to the preset transmitter identifier in each sensor spatial location information group as the target spatial location information group.

[0132] In some embodiments, the execution entity may determine the sensor spatial location information group corresponding to the preset transmitter identifier in each of the sensor spatial location information groups as the target spatial location information group.

[0133] Step 5042: Generate sensor image position information group based on preset camera parameter information, target spatial position information group and preset coordinate system transformation information corresponding to preset transmitter identifier.

[0134] In some embodiments, the executing entity can generate a sensor image position information group based on the preset camera parameter information, the target spatial position information group, and the preset coordinate system transformation information corresponding to the preset transmitter identifier. The preset coordinate system transformation information can be coordinate system transformation information pre-generated according to the camera calibration method implemented in steps 201-20233. In practice, firstly, for each target spatial position information included in the target spatial position information group, the executing entity can input the preset camera parameter information, the target spatial position information, and the preset coordinate system transformation information corresponding to the preset transmitter identifier into the following formula to obtain the sensor image position information.

[0135]

[0136] Where n can represent the electromagnetic signal transmitter coordinate system corresponding to the aforementioned preset transmitter identifier. (x n y n , z n (x) can represent the three-dimensional coordinates of the corresponding electromagnetic sensor in the electromagnetic signal transmitter coordinate system corresponding to the aforementioned preset transmitter identifier, i.e., the aforementioned target spatial location information. t y t , z t (u) can represent the three-dimensional coordinates of the corresponding electromagnetic sensor in the camera coordinate system. t v t ) can represent the two-dimensional coordinates of the corresponding electromagnetic sensor in the pixel coordinate system, i.e., the sensor image position information. R wn This can represent the rotation matrix included in the preset coordinate system transformation information corresponding to the aforementioned preset transmitter identifier. wn It can represent the translation vector included in the preset coordinate system transformation information corresponding to the preset transmitter identifier mentioned above.

[0137] Then, the obtained sensor image position information is determined as a sensor image position information group.

[0138] Optionally, the aforementioned execution entity can generate a sensor image position information group based on the aforementioned preset camera parameter information, the aforementioned target spatial position information group, and the preset transmitter coordinate transformation information through the following steps:

[0139] In the first step, in response to determining that the above-mentioned preset transmitter identifier is a preset conversion transmitter identifier, for each target spatial location information included in the target spatial location information group, the above-mentioned preset camera parameter information, the above-mentioned target spatial location information, and the preset coordinate system conversion information corresponding to the above-mentioned preset transmitter identifier are input into the following formula to obtain the sensor image location information.

[0140]

[0141] The aforementioned preset transmitter coordinate transformation information can be transmitter coordinate transformation information pre-generated according to the camera calibration method implemented in steps 401-403. (x0, y0, z0) can represent the three-dimensional coordinate points of the corresponding electromagnetic sensor in the electromagnetic signal transmitter coordinate system corresponding to the aforementioned preset transmitter identifier (or preset transformed transmitter identifier), i.e., the aforementioned target spatial position information. (x t y t , z t (u) can represent the three-dimensional coordinates of the corresponding electromagnetic sensor in the camera coordinate system. t v t ) can represent the two-dimensional coordinate point of the corresponding electromagnetic sensor in the pixel coordinate system, i.e., the sensor image position information. Then, in response to determining that the above-mentioned preset transmitter identifier is not a preset converted transmitter identifier, for each target spatial position information included in the target spatial position information group, the above-mentioned target spatial position information is input into the following formula to obtain the converted spatial position information.

[0142] (x0, y0, z0) = R i0 ×(x i y i , z i ) T +T i0

[0143] Among them, (x i y i , z i(x0, y0, z0) can represent the three-dimensional coordinates of the corresponding electromagnetic sensor in the electromagnetic signal transmitter coordinate system corresponding to the preset transmitter identifier, i.e., the target spatial position information. (x0, y0, z0) can represent the three-dimensional coordinates of the corresponding electromagnetic sensor in the electromagnetic signal transmitter coordinate system corresponding to the preset transmitter identifier, i.e., the converted spatial position information. Then, the preset camera parameter information, the converted spatial position information, and the preset coordinate system conversion information corresponding to the preset transmitter identifier are input into the following formula to obtain the sensor image position information.

[0144]

[0145] Finally, the obtained sensor image location information is determined as a sensor image location information group.

[0146] As an example, when an electromagnetic motion capture device needs to capture a user's hand posture, and the device includes two electromagnetic signal transmitters, the source1 coordinate system is the coordinate system of the electromagnetic signal transmitter corresponding to the preset transmitter identifier, and the source2 coordinate system is the coordinate system of the electromagnetic signal transmitter corresponding to the preset transmitter identifier "electromagnetic signal transmitter 2". The transformation relationships between the electromagnetic sensor coordinate system, each electromagnetic signal transmitter coordinate system, and the camera coordinate system are as follows: Figure 6 As shown.

[0147] Step 505: Based on the generated sensor image location information groups, perform image reconstruction processing on the user motion image to obtain the user motion capture image.

[0148] In some embodiments, the execution entity can perform image reconstruction processing on the user action image based on the generated sensor image position information sets to obtain a user motion capture image. In practice, the execution entity can draw the user's limb posture on the user action image based on the generated sensor image position information sets and a preset user limb model to obtain a user motion capture image. The preset user limb model can be a pre-defined model describing the user's limbs to be captured. As an example, when capturing user hand posture movements, the preset user limb model can be a pre-constructed hand model used to draw the hand contour.

[0149] The above-described embodiments of this disclosure have the following beneficial effects: the user motion capture image generation method of some embodiments of this disclosure can improve the accuracy and real-time performance of capturing user movements. Specifically, the reasons for low accuracy and poor real-time performance in capturing user movements are as follows: mechanical motion capture devices require users to wear a complete set of motion capture mechanical devices, which is inconvenient for users; acoustic motion capture devices have poor real-time performance and low accuracy in object capture, and are easily affected by ambient noise; optical motion capture devices, due to the use of multiple precise and complex high-speed optical cameras, are easily confused and obstructed; inertial navigation motion capture devices are generally large in size, cumbersome to operate, and due to the characteristics of the sensors, inertial motion capture is prone to error accumulation, resulting in low accuracy, thus causing low accuracy and poor real-time performance in capturing user movements. Based on this, the user motion capture image generation method of some embodiments of this disclosure uses electromagnetic motion capture devices when capturing user movements, which can reduce errors caused by external interference and eliminate the need to wait for signal transmission and reception, thereby improving the real-time performance and accuracy of motion capture. Furthermore, because the camera and each electromagnetic signal transmitter are jointly calibrated before using electromagnetic motion capture equipment to capture user movements, the accuracy of user movement capture and recognition is improved. This, in turn, enhances the accuracy and real-time performance of user movement capture.

[0150] The following is for reference. Figure 7 It illustrates electromagnetic motion capture devices suitable for implementing some embodiments of the present disclosure (e.g., Figure 1 The diagram shows the hardware structure of the electromagnetic motion capture device 700. Figure 7 The electromagnetic motion capture device shown is merely an example and should not be construed as limiting the functionality and scope of the embodiments of this disclosure.

[0151] like Figure 7As shown, the electromagnetic motion capture device 700 may include a processing device (e.g., a central processing unit, graphics processing unit, etc.) 701, a memory 702, an input unit 703, and an output unit 704. The processing device 701, memory 702, input unit 703, and output unit 704 are interconnected via a bus 705. Here, the method according to embodiments of this disclosure can be implemented as a computer program and stored in the memory 702. For example, some embodiments of this disclosure include a computer program product comprising a computer program carried on a computer-readable medium, the computer program containing program code for performing the methods shown in the flowchart. The processing device 701 in the electromagnetic motion capture device implements the camera calibration method or user motion capture image generation method of this disclosure by calling the aforementioned computer program stored in the memory 702. In some implementations, the output unit 704 may include at least one electromagnetic signal transmitter for transmitting electromagnetic signals. The input unit 703 may include at least one electromagnetic sensor and a camera. The electromagnetic sensor can be used to sense electromagnetic signals. The camera can be used to acquire images.

[0152] It should be noted that, in some embodiments of this disclosure, the computer-readable medium may be a computer-readable signal medium or a computer-readable storage medium, or any combination thereof. A computer-readable storage medium may be, for example,—but not limited to—an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of a computer-readable storage medium may include, but are not limited to: an electrical connection having one or more wires, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination thereof. In some embodiments of this disclosure, a computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, apparatus, or device. In some embodiments of this disclosure, a computer-readable signal medium may include a data signal propagated in baseband or as part of a carrier wave, carrying computer-readable program code. Such propagated data signals may take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. A computer-readable signal medium can be any computer-readable medium other than a computer-readable storage medium, which can send, propagate, or transmit a program for use by or in connection with an instruction execution system, apparatus, or device. The program code contained on the computer-readable medium can be transmitted using any suitable medium, including but not limited to: wires, optical fibers, RF (radio frequency), etc., or any suitable combination thereof.

[0153] In some implementations, clients and servers can communicate using any currently known or future-developed network protocol such as HTTP (Hypertext Transfer Protocol) and can interconnect with digital data communication (e.g., communication networks) of any form or medium. Examples of communication networks include local area networks (“LANs”), wide area networks (“WANs”), the Internet (e.g., the Internet of Things), and peer-to-peer networks (e.g., ad hoc peer-to-peer networks), as well as any currently known or future-developed networks.

[0154] The aforementioned computer-readable medium may be included in the aforementioned electromagnetic motion capture device; or it may exist independently and not assembled into the electromagnetic motion capture device. The aforementioned computer-readable medium carries one or more programs, which, when executed by the electromagnetic motion capture device, cause the electromagnetic motion capture device to: perform internal calibration processing on the camera included in the electromagnetic motion capture device to obtain camera parameter information; determine the initial frame number and initial camera pose information; and, based on the initial frame number and initial camera pose information, perform the following calibration steps: control the camera to acquire a checkerboard image based on the aforementioned camera parameter information and initial camera pose information; and generate a transmission signal corresponding to each electromagnetic sensor included in the electromagnetic motion capture device based on the checkerboard image. Sensor image position information group; for each preset transmitter identifier, perform the following steps: determine the sensor spatial position information group corresponding to the preset transmitter identifier, wherein each preset transmitter identifier corresponds to the electromagnetic signal transmitter included in the electromagnetic motion capture device; determine the target image position information set and the target spatial position information set based on the sensor image position information group and the sensor spatial position information group; in response to determining that the initial frame number meets the preset frame number condition, generate coordinate system transformation information based on each determined target image position information set and each determined target spatial position information set; or control The electromagnetic motion capture device includes a camera that acquires user motion images based on preset camera parameter information, and controls each electromagnetic signal transmitter included in the electromagnetic motion capture device to perform an activation operation, wherein the preset camera parameter information is pre-generated according to the method described in any implementation of the first aspect; controls each electromagnetic sensor included in the electromagnetic motion capture device to acquire electromagnetic signals to obtain a sensor pose information group; generates each sensor spatial position information group corresponding to each preset transmitter identifier based on the sensor pose information group; for each preset transmitter identifier, performs the following steps: determines the sensor spatial position information group corresponding to the preset transmitter identifier in the sensor spatial position information group as the target spatial position information group; generates a sensor image position information group based on the preset camera parameter information, the target spatial position information group, and the preset coordinate system transformation information corresponding to the preset transmitter identifier, wherein the preset coordinate system transformation information is pre-generated according to the method described in any implementation of the first aspect; and performs image reconstruction processing on the user motion images based on the generated sensor image position information groups to obtain user motion capture images.

[0155] Computer program code for performing operations of some embodiments of this disclosure can be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, and C++, and conventional procedural programming languages ​​such as the "C" language or similar programming languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving remote computers, the remote computer can be connected to the user's computer via any type of network—including a local area network (LAN) or a wide area network (WAN)—or can be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0156] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this disclosure. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0157] The functions described above in this document can be performed, at least in part, by one or more hardware logic components. For example, exemplary types of hardware logic components that can be used, without limitation, include: Field Programmable Gate Arrays (FPGAs), Application-Specific Integrated Circuits (ASICs), Application Standard Products (ASSPs), System-on-Chip (SoCs), Complex Programmable Logic Devices (CPLDs), and so on.

[0158] The above description is merely a selection of preferred embodiments of this disclosure and an explanation of the technical principles employed. Those skilled in the art should understand that the scope of the invention involved in the embodiments of this disclosure is not limited to technical solutions formed by specific combinations of the above-described technical features, but should also cover other technical solutions formed by arbitrary combinations of the above-described technical features or their equivalents without departing from the above-described inventive concept. For example, technical solutions formed by substituting the above-described features with (but not limited to) technical features with similar functions disclosed in the embodiments of this disclosure.

Claims

1. A camera calibration method applied to an electromagnetic motion capture device, the method comprising: The camera included in the electromagnetic motion capture device is internally calibrated to obtain camera parameter information; Determine the initial frame number and initial camera pose information, and based on the initial frame number and initial camera pose information, perform the following calibration steps: The camera is controlled to acquire a chessboard image based on the camera parameter information and the initial camera pose information; Based on the chessboard image, a sensor image position information group corresponding to each electromagnetic sensor included in the electromagnetic motion capture device is generated. For each preset transmitter identifier, perform the following steps: Determine the sensor spatial location information group corresponding to the preset transmitter identifier, wherein each preset transmitter identifier corresponds to the electromagnetic signal transmitter included in the electromagnetic motion capture device; Based on the sensor image location information group and the sensor spatial location information group, determine the target image location information set and the target spatial location information set; In response to the determination that the initial frame number meets the preset frame number condition, coordinate system transformation information is generated based on the determined set of position information of each target image and the determined set of spatial position information of each target.

2. The method according to claim 1, wherein, The calibration steps also include: In response to the determination that the initial frame number does not meet the preset frame number condition, the initial frame number and the initial camera pose information are updated, and the updated initial frame number and the updated initial camera pose information are used as the initial camera pose information, and the calibration steps are continued.

3. The method according to claim 1, wherein, The step of determining the target image location information set and the target spatial location information set based on the sensor image location information set and the sensor spatial location information set includes: The sensor image location information group is divided into training image location information group and verification image location information group; The sensor spatial location information group is divided into a training spatial location information group and a verification spatial location information group. Based on the training image location information group and the training spatial location information group, generate initial coordinate system transformation information; Based on the camera parameter information, the initial coordinate system transformation information, and the verification spatial position information group, a reprojection position information group is generated; Based on the reprojection position information group and the verification image position information group, an average projection error is generated; In response to determining that the average projection error meets a preset error condition, the training image location information set is determined as the target image location information set, and the training spatial location information set is determined as the target spatial location information set.

4. The method according to claim 3, wherein, The method further includes: In response to determining that the average projection error does not meet the preset error condition, the initial camera pose information is updated, and the updated initial camera pose information is used as the initial camera pose information to continue the calibration steps.

5. The method according to claim 1, wherein, The step of determining the sensor spatial location information group corresponding to the preset transmitter identifier includes: Control each electromagnetic sensor corresponding to the preset transmitter identifier to collect electromagnetic signals and obtain sensor pose information group; Based on the sensor pose information group, a sensor spatial position information group is generated.

6. The method according to claim 1, wherein, The step of generating coordinate system transformation information based on the determined set of target image location information and the determined set of target spatial location information includes: For each target spatial location information included in the determined target spatial location information set, perform the following steps: Based on the target spatial location information, transmitter distance information is generated; In response to determining that the transmitter distance information meets a preset distance condition, the target spatial position information is determined as the target sensor spatial position information, and the target image position information corresponding to the target spatial position information in the determined target image position information set is determined as the target sensor image position information; Based on the determined spatial location information of each target sensor and the determined image location information of each target sensor, coordinate system transformation information is generated.

7. The method according to claim 1, wherein, The method further includes: Based on the generated coordinate system transformation information, the transmitter coordinate transformation information is generated.

8. A method for generating user motion capture images, applied to an electromagnetic motion capture device, comprising: The electromagnetic motion capture device includes a camera that acquires user motion images according to preset camera parameter information, and controls each electromagnetic signal transmitter included in the electromagnetic motion capture device to perform an activation operation, wherein the preset camera parameter information is pre-generated by the camera calibration method according to any one of claims 1-7. The electromagnetic sensors included in the electromagnetic motion capture device are controlled to collect electromagnetic signals to obtain a sensor pose information group. Based on the sensor pose information group, generate each sensor spatial position information group corresponding to each preset transmitter identifier; For each preset transmitter identifier, perform the following steps: The sensor spatial location information group corresponding to the preset transmitter identifier in each sensor spatial location information group is determined as the target spatial location information group; Based on the preset camera parameter information, the target spatial location information group, and the preset coordinate system transformation information corresponding to the preset transmitter identifier, a sensor image location information group is generated, wherein the preset coordinate system transformation information is pre-generated by the camera calibration method according to any one of claims 1-7; Based on the generated sensor image location information groups, the user motion image is reconstructed to obtain the user motion capture image.

9. An electromagnetic motion capture device, comprising: At least one electromagnetic signal transmitter for transmitting electromagnetic signals; At least one electromagnetic sensor for sensing electromagnetic signals; A camera is used to capture images; One or more processors; Storage device, on which one or more programs are stored, When the one or more programs are executed by the one or more processors, the one or more processors implement the method as described in any one of claims 1-7 or 8.

10. A computer-readable medium having a computer program stored thereon, wherein, When the computer program is executed by a processor, it implements the method as described in any one of claims 1-7 or 8.

Images