Camera calibration method, motion capture image generation method, motion capture device and medium

Abstract

The embodiment of the invention discloses a camera calibration method, a motion capture image generation method, motion capture equipment and a medium. A specific embodiment of the method comprises the following steps: performing internal calibration processing on a camera; determining an initial frame number and initial camera pose information, and executing the following calibration steps according to the initial frame number and the initial camera pose information: controlling a camera to collect a checkerboard image; generating a sensor image position information group according to the checkerboard image; for each preset transmitter identifier, the following steps are executed: determining a sensor spatial position information group; determining a target image position information set and a target space position information set according to the sensor image position information set and the sensor space position information set; and in response to determining that the initial frame number meets a preset frame number condition, generating coordinate system conversion information according to the determined target image position information sets and the determined target space position information sets. According to the embodiment, the accuracy and the real-time performance of capturing the user actions can be improved.

Description

Technical Field

[0001] Embodiments of the present disclosure relate to the field of computer technology, and in particular to a camera calibration method, a motion capture image generation method, a motion capture device, and a medium. Background Art

[0002] In the field of motion tracking and recognition technology, for example, in the field of gesture interaction of AR / VR, motion capture equipment is required to accurately capture user movements, and then collect user movement images through cameras, and finally project the movement postures captured by the motion capture equipment onto the user movement images for motion tracking and recognition. At present, when capturing user movements through motion capture equipment, the methods usually adopted are: mechanical motion capture equipment that tracks and measures the motion trajectory by physical means, or acoustic motion capture, or optical motion capture equipment that completes the capture and positioning of objects through optical principles, or inertial navigation motion capture equipment that uses inertial navigation sensors AHRS (attitude reference system) and IMU (inertial measurement unit) to measure the motion acceleration, orientation, tilt angle and other characteristics of the captured person or object.

[0003] However, it has been found in practice that when the above method is used to capture user movements, the following technical problems often exist: mechanical motion capture equipment requires users to wear a complete set of motion capture mechanical devices, which is inconvenient for users to use; when using acoustic motion capture equipment, the real-time performance of object capture is poor, the accuracy is low, and it is easily disturbed by surrounding noise; optical motion capture equipment uses multiple precise and complex high-speed optical cameras, which are easily confused and blocked by each other; inertial navigation motion capture equipment is generally large in size and cumbersome to operate, and due to the characteristics of the sensor, inertial motion capture is prone to error accumulation, resulting in low accuracy, thereby resulting in low accuracy and poor real-time performance in capturing user movements.

[0004] The above information disclosed in this Background section is only for enhancement of understanding of the background of the present disclosure concept and therefore it may contain information that does not form the prior art that is already known in this country to a person of ordinary skill in the art. Summary of the invention

[0005] The content of this disclosure is used to introduce concepts in a brief form, which will be described in detail in the detailed implementation section below. The content of this disclosure is not intended to identify the key features or essential features of the technical solution claimed for protection, nor is it intended to limit the scope of the technical solution claimed for protection.

[0006] Some embodiments of the present disclosure propose a camera calibration method, a user motion capture image generation method, an electromagnetic motion capture device and a computer-readable medium to solve the technical problems mentioned in the above background technology section.

[0007] In a first aspect, some embodiments of the present disclosure provide a camera calibration method, which is applied to an electromagnetic motion capture device, the method comprising: performing internal calibration processing on a camera included in the electromagnetic motion capture device to obtain camera parameter information; determining an initial frame number and initial camera pose information, and performing the following calibration steps according to the initial frame number and the initial camera pose information: controlling the camera to collect a checkerboard image according to the camera parameter information and the initial camera pose information; generating a sensor image position information group corresponding to each electromagnetic sensor included in the electromagnetic motion capture device according to the checkerboard image; performing the following steps for each preset transmitter identifier: determining a sensor space position information group corresponding to the preset transmitter identifier, wherein each preset transmitter identifier corresponds to an electromagnetic signal transmitter included in the electromagnetic motion capture device; determining a target image position information set and a target space position information set according to the sensor image position information group and the sensor space position information group; in response to determining that the initial frame number meets the preset frame number condition, generating coordinate system conversion information according to each determined target image position information set and each determined target space position information set.

[0008] Optionally, the above calibration step also includes: in response to determining that the initial frame number does not meet the above preset frame number condition, updating the initial frame number and initial camera pose information, and using the updated initial frame number as the initial frame number, and the updated initial camera pose information as the initial camera pose information, and continuing to execute the above calibration step.

[0009] Optionally, the above-mentioned determination of the target image position information set and the target space position information set based on the above-mentioned sensor image position information group and the above-mentioned sensor space position information group includes: dividing the above-mentioned sensor image position information group to obtain a training image position information group and a verification image position information group; dividing the above-mentioned sensor space position information group to obtain a training space position information group and a verification space position information group; generating initial coordinate system conversion information based on the above-mentioned training image position information group and the above-mentioned training space position information group; generating a reprojection position information group based on the above-mentioned camera parameter information, the above-mentioned initial coordinate system conversion information and the above-mentioned verification space position information group; generating a projection average error based on the above-mentioned reprojection position information group and the above-mentioned verification image position information group; in response to determining that the above-mentioned projection average error satisfies a preset error condition, determining the above-mentioned training image position information group as the target image position information set, and determining the above-mentioned training space position information group as the target space position information set.

[0010] Optionally, the method further includes: in response to determining that the projection average error does not satisfy the preset error condition, updating the initial camera pose information, and using the updated initial camera pose information as the initial camera pose information, and continuing to perform the calibration step.

[0011] Optionally, the above-mentioned determination of the sensor spatial position information group corresponding to the above-mentioned preset transmitter identifier includes: controlling each electromagnetic sensor corresponding to the above-mentioned preset transmitter identifier to collect electromagnetic signals to obtain a sensor posture information group; and generating a sensor spatial position information group according to the above-mentioned sensor posture information group.

[0012] Optionally, the above-mentioned generation of coordinate system conversion information based on the determined target image position information sets and the determined target space position information sets includes: for each target space position information included in the determined target space position information sets, performing the following steps: generating transmitter distance information based on the above-mentioned target space position information; in response to determining that the above-mentioned transmitter distance information meets a preset distance condition, determining the above-mentioned target space position information as target sensor space position information, and determining the target image position information corresponding to the above-mentioned target space position information in the determined target image position information sets as target sensor image position information; generating coordinate system conversion information based on the determined target sensor space position information and the determined target sensor image position information.

[0013] Optionally, the above method further includes: generating transmitter coordinate transformation information according to each generated coordinate system transformation information.

[0014] In a second aspect, some embodiments of the present disclosure provide a method for generating a user motion capture image, which is applied to an electromagnetic motion capture device, the method comprising: controlling a camera included in the electromagnetic motion capture device to collect user motion images according to preset camera parameter information, and controlling each electromagnetic signal transmitter included in the electromagnetic motion capture device to perform a start operation, wherein the preset camera parameter information is pre-generated according to the method described in any implementation method of the first aspect; controlling each electromagnetic sensor included in the electromagnetic motion capture device to collect electromagnetic signals to obtain a sensor posture information group; and generating each sensor corresponding to each preset transmitter identifier according to the sensor posture information group. sensor space position information group; for each preset transmitter identifier, perform the following steps: determine the sensor space position information group corresponding to the above-mentioned preset transmitter identifier in the above-mentioned each sensor space position information group as the target space position information group; generate a sensor image position information group according to the above-mentioned preset camera parameter information, the above-mentioned target space position information group and the preset coordinate system conversion information corresponding to the above-mentioned preset transmitter identifier, wherein the above-mentioned preset coordinate system conversion information is pre-generated by the method described in any implementation method of the above-mentioned first aspect; according to the generated each sensor image position information group, perform image reconstruction processing on the above-mentioned user action image to obtain a user action capture image.

[0015] In a third aspect, some embodiments of the present disclosure provide an electromagnetic motion capture device, comprising: at least one electromagnetic signal transmitter for transmitting electromagnetic signals; at least one electromagnetic sensor for sensing electromagnetic signals; a camera for capturing images; one or more processors; and a storage device on which one or more programs are stored, and when the one or more programs are executed by one or more processors, the one or more processors implement the method described in any implementation of the first or second aspect above.

[0016] In a fourth aspect, some embodiments of the present disclosure provide a computer-readable medium having a computer program stored thereon, wherein when the computer program is executed by a processor, the method described in any implementation of the first aspect or the second aspect is implemented.

[0017] The above-mentioned embodiments of the present disclosure have the following beneficial effects: the camera calibration method of some embodiments of the present disclosure can improve the accuracy and real-time performance of capturing user actions. Specifically, the reasons for the low accuracy and poor real-time performance of capturing user actions are: the mechanical motion capture device requires the user to wear a complete set of motion capture mechanical devices, which is inconvenient for the user to use; the acoustic motion capture device has poor real-time performance and low accuracy when the object is captured, and is easily disturbed by surrounding noise; the optical motion capture device uses multiple precise and complex high-speed optical cameras, which are easy to confuse and block each other; the inertial navigation motion capture device is generally large in size and cumbersome to operate, and due to the characteristics of the sensor, inertial motion capture is prone to error accumulation, resulting in low accuracy, thereby resulting in low accuracy and poor real-time performance of capturing user actions. Based on this, the camera calibration method of some embodiments of the present disclosure is applied to the electromagnetic motion capture device. First, the camera included in the electromagnetic motion capture device is internally calibrated to obtain camera parameter information. Thus, the internal parameter information of the camera can be obtained, which can be used to improve the image quality captured by the camera. Secondly, determine the initial frame number and the initial camera pose information, and perform the following calibration steps according to the initial frame number and the initial camera pose information: the first step is to control the camera to collect a checkerboard image according to the camera parameter information and the initial camera pose information. Thus, a checkerboard image for calibration can be obtained, which can be used to identify the positions of each electromagnetic sensor included in the electromagnetic motion capture device in the checkerboard in the camera coordinate system. The second step is to generate a sensor image position information group corresponding to each electromagnetic sensor included in the electromagnetic motion capture device according to the checkerboard image. Thus, each pixel coordinate of each electromagnetic sensor in the checkerboard can be obtained, which can be used to calculate the coordinate conversion information between the transmitter coordinate system and the camera coordinate system. The third step is to perform the following steps for each preset transmitter identifier: step 1, determine the sensor space position information group corresponding to the preset transmitter identifier. Among them, each preset transmitter identifier corresponds to the electromagnetic signal transmitter included in the electromagnetic motion capture device. Thus, the position of each electromagnetic sensor in the corresponding transmitter coordinate system can be obtained. Step 2: Determine the target image position information set and the target space position information set according to the above-mentioned sensor image position information group and the above-mentioned sensor space position information group. Thus, the positions of the various electromagnetic sensors used to calculate the coordinate conversion information between the transmitter coordinate system and the camera coordinate system can be obtained. Step 3: In response to determining that the initial number of frames meets the preset frame number condition, generate coordinate system conversion information according to the determined target image position information sets and the determined target space position information sets. Thus, when enough image frames are collected, the coordinate conversion information between the transmitter coordinate system and the camera coordinate system can be obtained according to the corresponding positions of each electromagnetic sensor in the transmitter coordinate system and the camera coordinate system, so that the external parameter calibration of the camera relative to the corresponding electromagnetic signal transmitter can be completed.Also, electromagnetic motion capture equipment can be used when capturing user motion, thereby reducing errors caused by external interference, and there is no need to wait for signal transmission and reception, thereby improving the real-time performance and capture accuracy of motion capture. Also, when electromagnetic motion capture equipment is used to capture user motion, the camera and each electromagnetic signal transmitter can be jointly calibrated first, thereby improving the user motion capture accuracy and user motion recognition accuracy. Thus, the accuracy and real-time performance of capturing user motion can be improved. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] The above and other features, advantages and aspects of the embodiments of the present disclosure will become more apparent with reference to the following detailed description in conjunction with the accompanying drawings. Throughout the accompanying drawings, the same or similar reference numerals represent the same or similar elements. It should be understood that the drawings are schematic and that components and elements are not necessarily drawn to scale.

[0019] Figure 1 is a schematic diagram of an application scenario of a camera calibration method according to some embodiments of the present disclosure;

[0020] Figure 2 is a flow chart of some embodiments of the camera calibration method according to the present disclosure;

[0021] Figure 3 is a schematic diagram of the relative position and posture relationship between the electromagnetic sensor coordinate system and the electromagnetic signal transmitter coordinate system according to the camera calibration method of some embodiments of the present disclosure;

[0022] Figure 4 is a flowchart of other embodiments of the camera calibration method according to the present disclosure;

[0023] Figure 5 is a flow chart of some embodiments of the user motion capture image generation method of the present disclosure according to the present disclosure;

[0024] Figure 6 is a schematic diagram of the conversion relationship between the electromagnetic sensor coordinate system, each electromagnetic signal transmitter coordinate system and the camera coordinate system according to some embodiments of the user motion capture image generation method of the present disclosure;

[0025] Figure 7 It is a schematic diagram of the hardware structure of an electromagnetic motion capture device suitable for implementing some embodiments of the present disclosure. DETAILED DESCRIPTION

[0026] Embodiments of the present disclosure will be described in more detail below with reference to the accompanying drawings. Although certain embodiments of the present disclosure are shown in the accompanying drawings, it should be understood that the present disclosure can be implemented in various forms and should not be construed as being limited to the embodiments set forth herein. On the contrary, these embodiments are provided to provide a more thorough and complete understanding of the present disclosure. It should be understood that the drawings and embodiments of the present disclosure are only for exemplary purposes and are not intended to limit the scope of protection of the present disclosure.

[0027] It should also be noted that, for ease of description, only the parts related to the invention are shown in the drawings. In the absence of conflict, the embodiments and features in the embodiments of the present disclosure can be combined with each other.

[0028] It should be noted that the concepts such as "first" and "second" mentioned in the present disclosure are only used to distinguish different devices, modules or units, and are not used to limit the order or interdependence of the functions performed by these devices, modules or units.

[0029] It should be noted that the modifications of "one" and "plurality" mentioned in the present disclosure are illustrative rather than restrictive, and those skilled in the art should understand that unless otherwise clearly indicated in the context, it should be understood as "one or more".

[0030] The names of the messages or information exchanged between multiple devices in the embodiments of the present disclosure are only used for illustrative purposes and are not used to limit the scope of these messages or information.

[0031] The present disclosure will be described in detail below with reference to the accompanying drawings and in conjunction with embodiments.

[0032] Figure 1 It is a schematic diagram of an application scenario of the camera calibration method of some embodiments of the present disclosure.

[0033] exist Figure 1In the application scenario, first, the electromagnetic motion capture device 101 can perform internal calibration processing on the camera 1011 included in the electromagnetic motion capture device 101 to obtain camera parameter information 102. Then, the electromagnetic motion capture device 101 determines the initial frame number 103 and the initial camera pose information 104, and performs the following calibration steps according to the initial frame number and the initial camera pose information: the first step is to control the camera to collect the checkerboard image 105 according to the camera parameter information 102 and the initial camera pose information 104; the second step is to generate a sensor image position information group 106 corresponding to each electromagnetic sensor included in the electromagnetic motion capture device according to the checkerboard image 105; for each preset transmitter identifier 107, perform the following steps: determine the corresponding preset transmitter identifier 107; A sensor space position information group 108 of a transmitter identifier 107, wherein each preset transmitter identifier 107 corresponds to an electromagnetic signal transmitter included in the above-mentioned electromagnetic motion capture device; according to the above-mentioned sensor image position information group 106 and the above-mentioned sensor space position information group 108, a target image position information set 109 and a target space position information set 110 are determined; in response to determining that the initial frame number satisfies a preset frame number condition 111, a coordinate system conversion information 112 is generated according to each determined target image position information set 109 and each determined target space position information set 110.

[0034] It should be understood that Figure 1 The number of electromagnetic motion capture devices in the embodiment is only for illustration. Any number of electromagnetic motion capture devices may be provided according to implementation requirements.

[0035] Continue to refer Figure 2 , shows a process 200 of some embodiments of the camera calibration method according to the present disclosure. The camera calibration method, applied to an electromagnetic motion capture device, comprises the following steps:

[0036] Step 201 , performing internal calibration processing on a camera included in an electromagnetic motion capture device to obtain camera parameter information.

[0037] In some embodiments, the execution subject of the camera calibration method (eg Figure 1The electromagnetic motion capture device shown in the figure can perform internal calibration processing on the camera included in the electromagnetic motion capture device to obtain camera parameter information. Among them, the electromagnetic motion capture device can be a motion capture device that uses the strength of the magnetic field to track the position and orientation. The electromagnetic motion capture device can include but is not limited to at least one electromagnetic signal transmitter, at least one electromagnetic sensor and a camera. The electromagnetic signal transmitter can be used to transmit electromagnetic signals to form a magnetic field. For example, the electromagnetic signal transmitter can be a transmitter in an electromagnetic motion tracking system. The electromagnetic sensor can be used to sense electromagnetic signals in the magnetic field generated by the electromagnetic signal transmission, and convert the sensed electromagnetic signals into electrical signals. It should be noted that when the electromagnetic motion capture device is used to capture the user's motion posture, each electromagnetic sensor needs to be set at the user's motion limb that needs to be captured. As an example, when the electromagnetic motion capture device is used to capture the user's hand posture, each electromagnetic sensor needs to be set at the 21 joints of the hand. The camera can be a monocular fisheye camera. The above camera parameter information can characterize the camera's internal parameters. The above camera parameter information can include but is not limited to the camera's internal parameter matrix and distortion parameters. The above internal parameter matrix can be expressed by the following formula:

[0038]

[0039] Wherein, K may represent an intrinsic parameter matrix. p may represent a pixel coordinate system. x may represent an x-axis in a three-dimensional coordinate system. y may represent a y-axis in a three-dimensional coordinate system. p Can represent the x-axis in the pixel coordinate system. p It can represent the y-axis in the pixel coordinate system. f can represent the focal length. It can represent the focal length of the camera on the x-axis. It can represent the focal length of the camera on the y-axis. c can represent the coordinates of the principal point of the camera. It can be said that the camera is on the x-axis

[0040] The coordinates of the principal point on . It can represent the coordinates of the camera's principal point on the y-axis.

[0041] The above distortion parameters can be expressed by the following formula:

[0042] d=[k1,k2,k3,k4].

[0043] Wherein, d may represent a distortion parameter. k may represent a radial distortion coefficient. k1 may represent a first-order radial distortion coefficient. k2 may represent a second-order radial distortion coefficient. k3 may represent a third-order radial distortion coefficient. k4 may represent a higher-order radial distortion coefficient.

[0044] In practice, the execution subject may perform internal calibration processing on the camera included in the electromagnetic motion capture device through a preset checkerboard calibration method to obtain camera parameter information. The preset checkerboard calibration method may be a method for calibrating camera internal parameters according to a checkerboard. For example, the preset checkerboard calibration method may be a Zhang Zhengyou checkerboard calibration method.

[0045] Optionally, the electromagnetic motion capture device may further include a clock synchronization device. The clock synchronization device may be used to synchronize the time when the camera captures images with the time when the electromagnetic sensor senses electromagnetic signals, thereby improving the accuracy of camera calibration.

[0046] Step 202, determining the initial frame number and the initial camera pose information, and performing the following calibration steps according to the initial frame number and the initial camera pose information:

[0047] Step 2021, controlling the camera to capture a checkerboard image according to the camera parameter information and the initial camera pose information.

[0048] In some embodiments, the execution subject may control the camera to collect a checkerboard image according to the camera parameter information and the initial camera pose information. The initial frame number may be the number of initial captured images. The initial camera pose information may characterize the pose of the initial camera when capturing an image. The initial camera pose information may include but is not limited to a camera shooting angle. The camera shooting angle may be a camera shooting angle. The shooting angle may include a shooting height, a shooting direction, and a shooting distance. The shooting height may be the height of the camera relative to the camera calibration checkerboard. The shooting height may be divided into three types: horizontal shooting, downward shooting, and upward shooting. The shooting direction may be the direction of the camera relative to the camera calibration checkerboard. The shooting direction may be but is not limited to one of the following: a front angle, a side angle, an oblique side angle, and a back angle. The shooting distance may be the relative distance between the camera and the camera calibration checkerboard. The checkerboard image may be an image of the camera calibration checkerboard. The camera calibration checkerboard may be a checkerboard used for camera calibration. For example, the camera calibration checkerboard may be an AprilTag checkerboard. It should be noted that the electromagnetic sensors included in the electromagnetic motion capture device are evenly distributed and fixed on the corner points of the camera calibration chessboard. The center point of each electromagnetic sensor is aligned with the corresponding corner point. In practice, the execution entity may determine the preset initial frame number as the initial frame number, and determine the preset initial camera pose information as the initial camera pose information. Among them, the preset initial frame number may be a preset initial frame number. For example, the preset initial frame number may be 1. The preset initial camera pose information may be a preset initial camera pose information. For example, the preset initial camera pose information may be "flat shot, front angle 0, distance 50 cm".

[0049] Step 2022: Generate a sensor image position information group corresponding to each electromagnetic sensor included in the electromagnetic motion capture device according to the checkerboard image.

[0050] In some embodiments, the execution subject may generate a sensor image position information group corresponding to each electromagnetic sensor included in the electromagnetic motion capture device according to the checkerboard image. The sensor image position information in the sensor image position information group may correspond to the electromagnetic sensor in each electromagnetic sensor. The sensor image position information may represent the position of the corresponding electromagnetic sensor in the checkerboard image. In practice, first, the execution subject may perform corner point detection and recognition processing on the checkerboard by a preset corner point recognition algorithm to obtain a corner point detection information set corresponding to each checkerboard corner point. The preset corner point recognition algorithm may be a pre-set algorithm for detecting and identifying the position of the checkerboard corner point. For example, the preset corner point recognition algorithm may be an AprilTag positioning algorithm. The checkerboard corner points in each checkerboard corner point may correspond to the corner point detection information in the corner point detection information set. The checkerboard corner points may be the corner points in the camera calibration checkerboard. The corner point detection information may represent the position of the corresponding checkerboard corner point in the checkerboard image. The corner point detection information may include but is not limited to the pixel coordinates of the checkerboard corner points. The pixel coordinates of the chessboard corner points may be the pixel coordinates of the chessboard corner points in the chessboard image in the camera coordinate system. Then, for each preset electromagnetic sensor identifier, the corner point detection information corresponding to the preset electromagnetic sensor identifier in the corner point detection information set is determined as the sensor image position information. Among them, the preset electromagnetic sensor identifier may be a unique identifier of the corresponding electromagnetic sensor that is preset. For example, the preset electromagnetic sensor identifier may be the serial number of the corresponding electromagnetic sensor. The corner point detection information corresponding to the preset electromagnetic sensor identifier may be: the corner point detection information corresponding to the target chessboard corner point. The target chessboard corner point may be: the chessboard corner point that is the same as the corner point placed by the electromagnetic sensor corresponding to the preset electromagnetic sensor identifier on the camera calibration chessboard. Finally, the determined sensor image position information is determined as a sensor image position information group.

[0051] Step 2023, for each preset transmitter identifier, perform the following steps:

[0052] Step 20231, determine the sensor spatial position information group corresponding to the preset transmitter identifier.

[0053] In some embodiments, the above-mentioned execution entity may determine a sensor space position information group corresponding to the above-mentioned preset transmitter identifier. Each preset transmitter identifier corresponds to an electromagnetic signal transmitter included in the above-mentioned electromagnetic motion capture device. The preset transmitter identifier may be a unique identifier of the corresponding electromagnetic signal transmitter. The sensor space position information in the sensor space position information group may correspond one-to-one to the electromagnetic sensors in each electromagnetic sensor. The sensor space position information in the sensor space position information group represents the coordinate point of the corresponding electromagnetic sensor in the electromagnetic signal transmitter coordinate system corresponding to the preset transmitter identifier. In practice, the above-mentioned execution entity may, in response to detecting the checkerboard origin offset information corresponding to the above-mentioned preset transmitter identifier, perform the following steps for each sensor image position information included in the sensor image position information group:

[0054] In the first step, the sum of the x-axis offset included in the chessboard origin offset information and the pixel coordinates of the corresponding x-axis included in the sensor image position information is determined as the x-axis spatial coordinate. Among them, the chessboard origin offset information can characterize the position offset between the center point of the chessboard and the center point of the electromagnetic signal transmitter corresponding to the preset transmitter identification. The chessboard origin offset information may include but is not limited to the x-axis offset, the y-axis offset and the z-axis offset. The x-axis offset may be the distance between the center point of the chessboard and the center point of the electromagnetic signal transmitter in the horizontal direction. The y-axis offset may be the distance between the center point of the chessboard and the center point of the electromagnetic signal transmitter in the front-back direction. The z-axis offset may be the distance between the center point of the chessboard and the center point of the electromagnetic signal transmitter in the vertical direction. It should be noted that the chessboard origin offset information may be measured by the calibration user through a physical measurement method. The chessboard origin offset information may be sent to the execution subject by the calibration user through a terminal, or may be input by the calibration user through an interface on the execution subject. The above-mentioned calibration user may be a user who calibrates the camera and the electromagnetic motion capture device.

[0055] In the second step, the sum of the y-axis offset included in the chessboard origin offset information and the corresponding y-axis pixel coordinate included in the sensor image position information is determined as the y-axis spatial coordinate.

[0056] The third step is to determine the z-axis offset included in the above chessboard origin offset information as the z-axis space coordinate.

[0057] In the fourth step, the x-axis spatial coordinate, the y-axis spatial coordinate and the z-axis spatial coordinate are determined as the sensor spatial position information.

[0058] Then, the determined individual sensor space position information are determined as a sensor space position information group.

[0059] In some optional implementations of some embodiments, the execution subject may determine the sensor spatial position information group corresponding to the preset transmitter identifier through the following steps:

[0060] The first step is to control each electromagnetic sensor corresponding to the above-mentioned preset transmitter identifier to collect electromagnetic signals and obtain a sensor posture information group. Among them, the sensor posture information in the above-mentioned sensor posture information group can represent the position and posture of the corresponding electromagnetic sensor in the corresponding electromagnetic signal transmitter coordinate system. The above-mentioned electromagnetic signal transmitter coordinate system can be: a Cartesian right-handed coordinate system with the center point of the electromagnetic signal transmitter corresponding to the preset transmitter identifier as the origin. The sensor posture information in the above-mentioned sensor posture information group may include but is not limited to a sensor translation vector and a sensor rotation matrix. The above-mentioned sensor translation vector can represent the offset of the center point of the corresponding electromagnetic sensor relative to the origin of the above-mentioned electromagnetic signal transmitter coordinate system. The above-mentioned sensor rotation matrix can represent the rotation angle of the electromagnetic sensor coordinate system relative to the above-mentioned electromagnetic signal transmitter coordinate system. The above-mentioned electromagnetic sensor coordinate system can be a Cartesian right-handed coordinate system with the center point of the electromagnetic sensor as the origin. The relative posture relationship between the electromagnetic sensor coordinate system (sensor coordinate system) and the electromagnetic signal transmitter coordinate system (source coordinate system) can be as follows Figure 3 As shown. The above sensor translation vector can be expressed by the following formula:

[0061] T s =[x s ,y s , z s ].

[0062] Wherein, T may represent a translation vector. S may represent an electromagnetic sensor. T s Can represent the sensor translation vector. x s It can represent the offset of the center point of the electromagnetic sensor relative to the origin of the coordinate system of the electromagnetic signal transmitter in the X-axis direction. s It can represent the offset of the center point of the electromagnetic sensor relative to the origin of the coordinate system of the electromagnetic signal transmitter in the Y-axis direction. s It can represent the offset of the center point of the electromagnetic sensor relative to the origin of the coordinate system of the electromagnetic signal transmitter in the Z-axis direction. The sensor rotation matrix can be expressed by the following formula:

[0063]

[0064] Where R can represent the rotation matrix. scan represent the sensor rotation matrix. ψ can represent the yaw angle. θ can represent the pitch angle. φ can represent the roll angle. The above yaw angle can represent the deflection angle of the electromagnetic sensor coordinate system relative to the electromagnetic signal transmitter reference coordinate system in the Z-axis direction. The above pitch angle can represent the deflection angle of the electromagnetic sensor coordinate system relative to the electromagnetic signal transmitter reference coordinate system in the Y-axis direction. The above roll angle can represent the deflection angle of the electromagnetic sensor coordinate system relative to the electromagnetic signal transmitter reference coordinate system in the Z-axis direction.

[0065] The second step is to generate a sensor spatial position information group based on the sensor pose information group. In practice, for each sensor pose information included in the sensor pose information group, the execution subject can input the sensor translation vector and sensor rotation matrix included in the sensor pose information into the following formula to obtain the sensor spatial coordinates as the sensor spatial position information.

[0066] (x source ,y source , z source )=R s ×(0, 0, 0) T +T s .

[0067] Among them, (x source ,y souce , z source ) may represent the sensor space coordinates. The sensor space coordinates may be the coordinates of the center point of the corresponding electromagnetic sensor in the source coordinate system. s It can represent the sensor rotation matrix. T s Can represent the sensor translation vector.

[0068] Therefore, the position of the electromagnetic sensor in the coordinate system of the electromagnetic signal transmitter can be determined by sensing the electromagnetic signal through the electromagnetic sensor.

[0069] Step 20232: Determine a target image position information set and a target space position information set according to the sensor image position information set and the sensor space position information set.

[0070] In some embodiments, the execution subject may determine the target image position information set and the target space position information set based on the sensor image position information group and the sensor space position information group. The target image position information in the target image position information set may be sensor image position information used as a sample to calibrate a camera and an electromagnetic motion capture device. The target space position information in the target space position information set may be sensor space position information used as a sample to calibrate a camera and an electromagnetic motion capture device. In practice, the execution subject may determine the sensor image position information group as the target image position information set, and determine the sensor space position information group as the target space position information set.

[0071] In some optional implementations of some embodiments, the execution subject may determine the target image position information set and the target space position information set according to the sensor image position information group and the sensor space position information group through the following steps:

[0072] The first step is to divide the above-mentioned sensor image position information group into a training image position information group and a verification image position information group. In practice, the above-mentioned execution subject can determine each sensor image position information corresponding to the preset training sensor identification set in the above-mentioned sensor image position information group as a training image position information group, and determine each sensor image position information corresponding to the preset verification sensor identification set in the above-mentioned sensor image position information group as a verification image position information group. Among them, the preset training sensor identification in the above-mentioned preset training sensor identification set can be an identification of a preset electromagnetic sensor for training. The preset verification sensor identification in the above-mentioned preset verification sensor identification set can be an identification of a preset electromagnetic sensor for verification. The sensor image position information corresponding to the preset training sensor identification can be: the sensor image position information of the corresponding preset electromagnetic sensor identification being the same as the preset training sensor identification. The sensor image position information corresponding to the preset verification sensor identification can be: the sensor image position information of the corresponding preset electromagnetic sensor identification being the same as the preset verification sensor identification.

[0073] The second step is to divide the above-mentioned sensor space position information group into a training space position information group and a verification space position information group. In practice, the above-mentioned execution subject can determine each sensor space position information corresponding to the preset training sensor identification set in the above-mentioned sensor space position information group as a training space position information group, and determine each sensor space position information corresponding to the preset verification sensor identification set in the above-mentioned sensor space position information group as a verification space position information group. Among them, the sensor space position information corresponding to the preset training sensor identification can be: the sensor space position information whose corresponding preset electromagnetic sensor identification is the same as the preset training sensor identification. The sensor space position information corresponding to the preset verification sensor identification can be: the sensor space position information whose corresponding preset electromagnetic sensor identification is the same as the preset verification sensor identification.

[0074] The third step is to generate initial coordinate system conversion information based on the above-mentioned training image position information group and the above-mentioned training space position information group. The above-mentioned initial coordinate system conversion information can characterize the conversion relationship between the source coordinate system and the camera coordinate system. In practice, the above-mentioned execution subject can solve the above-mentioned training image position information group and the above-mentioned training space position information group through the PnP (Perspective-n-Point) algorithm to obtain the initial rotation matrix and the initial translation vector. Then, the above-mentioned initial rotation matrix and the above-mentioned initial translation vector are determined as the initial coordinate system conversion information.

[0075] The fourth step is to generate a reprojection position information group according to the camera parameter information, the initial coordinate system conversion information and the verification space position information group. In practice, for each verification space position information in the verification space position information group, the execution subject may input the intrinsic parameter matrix included in the camera parameter information, the initial rotation matrix and the initial translation vector included in the initial coordinate system conversion information, and the verification space position information into the following formula to obtain the reprojection pixel coordinates, and determine the reprojection pixel coordinates as the reprojection position information.

[0076]

[0077] Among them, (x′ t , y′ t , z′ t ) can represent the three-dimensional coordinates of the center point of the corresponding electromagnetic sensor in the camera coordinate system after reprojection. source can represent the initial rotation matrix. (x′ source , y′ source , z′ source ) can represent the verification space location information. source can represent the initial translation vector. t can represent the reprojection. (u′t , v′ t ) can represent the pixel coordinates of the center point of the corresponding electromagnetic sensor in the pixel coordinate system after reprojection, that is, the reprojected pixel coordinates (u, v). (u, v) can represent the pixel coordinates.

[0078] Then, the determined individual reprojection position information are determined as a reprojection position information group.

[0079] Step 5: Generate a projection average error based on the above reprojection position information group and the above verification image position information group. In practice, first, for each reprojection position information included in the above reprojection position information group, the execution subject may perform the following sub-steps:

[0080] In a first sub-step, the verification image position information corresponding to the re-projection position information in the verification image position information group is determined as the target verification image position information.

[0081] In the second sub-step, the target verification image position information and the re-projection position information are input into the following formula to obtain the projection error.

[0082]

[0083] Among them, err can represent the projection error. j It can represent the horizontal coordinate of the point corresponding to the position information of the target verification image in the pixel coordinate system. j The vertical coordinate of the point corresponding to the position information of the target verification image in the pixel coordinate system may be represented by u, the horizontal coordinate in the pixel coordinate system, and the vertical coordinate in the pixel coordinate system may be represented by V.

[0084] Then, the average value of the obtained projection errors is determined as the projection average error.

[0085] Step 6: In response to determining that the projection average error satisfies a preset error condition, the training image position information group is determined as a target image position information set, and the training space position information group is determined as a target space position information set. The preset error condition may be that the projection average error is less than a preset error value. The preset error value may be a preset projection average error.

[0086] Optionally, the execution subject may also update the initial camera pose information in response to determining that the projection average error does not meet the preset error condition, and use the updated initial camera pose information as the initial camera pose information to continue to perform the calibration step. In practice, the execution subject may update the initial camera pose information according to a preset adjustment angle in response to determining that the projection average error does not meet the preset error condition, and use the updated initial camera pose information as the initial camera pose information to continue to perform the calibration step. The preset adjustment angle may be a pre-set angle for adjusting the shooting angle. For example, the preset adjustment angle may be a "proof angle of 30 degrees".

[0087] Thus, the initially obtained coordinate system conversion information can be verified. When the coordinate system conversion information does not meet the requirements, the checkerboard image can be recaptured for camera calibration, thereby improving the accuracy of the obtained coordinate system conversion information.

[0088] Step 20233, in response to determining that the initial frame number meets the preset frame number condition, generating coordinate system conversion information according to the determined target image position information sets and the determined target space position information sets.

[0089] In some embodiments, the execution subject may generate coordinate system conversion information in response to determining that the initial frame number satisfies the preset frame number condition, based on the determined target image position information sets and the determined target space position information sets. The preset frame number condition may be that the initial frame number is equal to the preset frame number threshold. The preset frame number threshold may be a preset maximum value of the number of frames for capturing a checkerboard image. In practice, the execution subject may solve the determined target image position information sets and the determined target space position information sets through a PnP (Perspective-n-Point) algorithm to obtain a rotation matrix and a translation vector. Then, the rotation matrix and the translation vector are determined as coordinate system conversion information.

[0090] In some optional implementations of some embodiments, the execution subject may generate coordinate system conversion information according to the determined target image position information sets and the determined target space position information sets through the following steps:

[0091] The first step is to perform the following sub-steps for each target space position information included in each determined target space position information set:

[0092] The first sub-step is to generate the transmitter distance information according to the target space position information. In practice, the execution subject may determine the Euclidean distance between the target space position information and the origin of the corresponding source coordinate system as the transmitter distance information.

[0093] In a second sub-step, in response to determining that the transmitter distance information satisfies a preset distance condition, the target space position information is determined as the target sensor space position information, and the target image position information corresponding to the target space position information in the determined target image position information set is determined as the target sensor image position information. The preset distance condition may be: the transmitter distance information is less than a preset transmitter distance. The preset transmitter distance may be: a preset maximum value of the distance between the corresponding electromagnetic sensor and the electromagnetic signal transmitter that indicates stable signal reception.

[0094] The second step is to generate coordinate system conversion information based on the determined spatial position information of each target sensor and the determined image position information of each target sensor. In practice, the above-mentioned execution subject can solve the determined spatial position information of each target sensor and the determined image position information of each target sensor through the PnP (Perspective-n-Point) algorithm to obtain a rotation matrix and a translation vector. Then, the above-mentioned rotation matrix and the above-mentioned translation vector are determined as the coordinate system conversion information.

[0095] Thus, sample points corresponding to electromagnetic sensors that are actually far away from corresponding electromagnetic signal transmitters can be eliminated, thereby reducing errors caused by inaccurate electromagnetic signals detected by the electromagnetic sensors, thereby improving the accuracy of the obtained coordinate system conversion information.

[0096] Optionally, the execution subject may also update the initial frame number and the initial camera pose information in response to determining that the initial frame number does not meet the preset frame number condition, and use the updated initial frame number as the initial frame number, and use the updated initial camera pose information as the initial camera pose information, and continue to perform the calibration step. In practice, the execution subject may update the initial camera pose information according to the preset adjustment angle in response to determining that the initial frame number does not meet the preset frame number condition, and determine the sum of the initial frame number and the preset increase step length as the updated initial frame number, and use the updated initial frame number as the initial frame number, and use the updated initial camera pose information as the initial camera pose information, and continue to perform the calibration step. Among them, the preset increase step length may be a preset value for increasing the number of frames per image acquisition. For example, the preset increase step length may be 1. Thus, by acquiring checkerboard images at different angles for multiple times for camera calibration, the generalization performance of the obtained coordinate system conversion information may be improved, and the accuracy of motion capture of the electromagnetic motion capture device may be improved.

[0097] The above-mentioned embodiments of the present disclosure have the following beneficial effects: the camera calibration method of some embodiments of the present disclosure can improve the accuracy and real-time performance of capturing user actions. Specifically, the reasons for the low accuracy and poor real-time performance of capturing user actions are: the mechanical motion capture device requires the user to wear a complete set of motion capture mechanical devices, which is inconvenient for the user to use; the acoustic motion capture device has poor real-time performance and low accuracy when the object is captured, and is easily disturbed by surrounding noise; the optical motion capture device uses multiple precise and complex high-speed optical cameras, which are easy to confuse and block each other; the inertial navigation motion capture device is generally large in size and cumbersome to operate, and due to the characteristics of the sensor, inertial motion capture is prone to error accumulation, resulting in low accuracy, thereby resulting in low accuracy and poor real-time performance of capturing user actions. Based on this, the camera calibration method of some embodiments of the present disclosure is applied to the electromagnetic motion capture device. First, the camera included in the electromagnetic motion capture device is internally calibrated to obtain camera parameter information. Thus, the internal parameter information of the camera can be obtained, which can be used to improve the image quality captured by the camera. Secondly, determine the initial frame number and the initial camera pose information, and perform the following calibration steps according to the initial frame number and the initial camera pose information: the first step is to control the camera to collect a checkerboard image according to the camera parameter information and the initial camera pose information. Thus, a checkerboard image for calibration can be obtained, which can be used to identify the positions of each electromagnetic sensor included in the electromagnetic motion capture device in the checkerboard in the camera coordinate system. The second step is to generate a sensor image position information group corresponding to each electromagnetic sensor included in the electromagnetic motion capture device according to the checkerboard image. Thus, each pixel coordinate of each electromagnetic sensor in the checkerboard can be obtained, which can be used to calculate the coordinate conversion information between the transmitter coordinate system and the camera coordinate system. The third step is to perform the following steps for each preset transmitter identifier: step 1, determine the sensor space position information group corresponding to the preset transmitter identifier. Among them, each preset transmitter identifier corresponds to the electromagnetic signal transmitter included in the electromagnetic motion capture device. Thus, the position of each electromagnetic sensor in the corresponding transmitter coordinate system can be obtained. Step 2: Determine the target image position information set and the target space position information set according to the above-mentioned sensor image position information group and the above-mentioned sensor space position information group. Thus, the positions of the various electromagnetic sensors used to calculate the coordinate conversion information between the transmitter coordinate system and the camera coordinate system can be obtained. Step 3: In response to determining that the initial number of frames meets the preset frame number condition, generate coordinate system conversion information according to the determined target image position information sets and the determined target space position information sets. Thus, when enough image frames are collected, the coordinate conversion information between the transmitter coordinate system and the camera coordinate system can be obtained according to the corresponding positions of each electromagnetic sensor in the transmitter coordinate system and the camera coordinate system, so that the external parameter calibration of the camera relative to the corresponding electromagnetic signal transmitter can be completed.Also, electromagnetic motion capture equipment can be used when capturing user motion, thereby reducing errors caused by external interference, and there is no need to wait for signal transmission and reception, thereby improving the real-time performance and capture accuracy of motion capture. Also, when electromagnetic motion capture equipment is used to capture user motion, the camera and each electromagnetic signal transmitter can be jointly calibrated first, thereby improving the user motion capture accuracy and user motion recognition accuracy. Thus, the accuracy and real-time performance of capturing user motion can be improved.

[0098] Further references Figure 4 , which shows a process 400 of another embodiment of a camera calibration method. The process 400 of the camera calibration method is applied to an electromagnetic motion capture device and includes the following steps:

[0099] Step 401 , performing internal calibration processing on the camera included in the electromagnetic motion capture device to obtain camera parameter information.

[0100] Step 402, determining the initial frame number and the initial camera pose information, and performing the following calibration steps according to the initial frame number and the initial camera pose information:

[0101] Step 4021, controlling the camera to collect a checkerboard image according to the camera parameter information and the initial camera pose information.

[0102] Step 4022: Generate a sensor image position information group corresponding to each electromagnetic sensor included in the electromagnetic motion capture device according to the checkerboard image.

[0103] Step 4023, for each preset transmitter identifier, perform the following steps:

[0104] Step 40231, determine the sensor spatial position information group corresponding to the above-mentioned preset transmitter identifier.

[0105] Step 40232: Determine a target image position information set and a target space position information set based on the sensor image position information set and the sensor space position information set.

[0106] Step 40233, in response to determining that the initial frame number meets the preset frame number condition, generating coordinate system conversion information according to the determined target image position information sets and the determined target space position information sets.

[0107] In some embodiments, the specific implementation of steps 401-40233 and the technical effects brought about can be referred to Figure 2 The corresponding steps 201-20233 in those embodiments are not repeated here.

[0108] Step 403: Generate transmitter coordinate transformation information according to each generated coordinate system transformation information.

[0109] In some embodiments, the above execution entities (e.g. Figure 1 The electromagnetic motion capture device shown in the figure) can generate transmitter coordinate conversion information according to the generated coordinate system conversion information. Wherein, the above-mentioned transmitter coordinate conversion information can represent: the coordinate conversion between each electromagnetic signal transmitter coordinate system and a preset electromagnetic signal transmitter coordinate system used to directly perform coordinate conversion with the camera coordinate system in actual applications. In practice, the above-mentioned execution subject can determine the coordinate system conversion information corresponding to the preset conversion transmitter identifier in the generated coordinate system conversion information as the target coordinate system conversion information. Wherein, the above-mentioned preset conversion transmitter identifier can be the identifier of the electromagnetic signal transmitter that is preset to directly perform coordinate conversion with the camera coordinate system. Then, for each preset transmitter identifier, the coordinate system conversion information corresponding to the above-mentioned preset transmitter identifier and the above-mentioned target coordinate system conversion information are input into the following formula to obtain the coordinate conversion information corresponding to the above-mentioned preset transmitter identifier. Wherein, the above-mentioned coordinate conversion information can include a coordinate conversion rotation matrix and a coordinate conversion translation vector.

[0110] R i0 =R wi T ·R w0

[0111] T i0 =T wi T ·(T w0 -T wi ).

[0112] Wherein, the above may represent the source coordinate system corresponding to the above preset transmitter identifier. 0 may represent the source coordinate system corresponding to the preset conversion transmitter identifier. w may represent the camera coordinate system. R i0 It can represent the rotation matrix between the source coordinate system corresponding to the preset conversion emitter identifier and the source coordinate system corresponding to the above preset emitter identifier, that is, the coordinate conversion rotation matrix. i0 It can represent the translation vector between the source coordinate system corresponding to the preset conversion emitter identifier and the source coordinate system corresponding to the above preset emitter identifier, that is, the coordinate conversion translation vector. wi It can represent the rotation matrix included in the coordinate system conversion information corresponding to the above preset transmitter identification. w0 It can represent the rotation matrix included in the above target coordinate system transformation information. wi It can represent the translation vector included in the coordinate system conversion information corresponding to the above preset transmitter identification. w0It may represent the translation vector included in the above target coordinate system conversion information. It should be noted that the preset transmitter identifier here does not include the transmitter identifier that is the same as the preset conversion transmitter identifier.

[0113] Finally, the obtained coordinate transformation information is determined as the transmitter coordinate transformation information.

[0114] Optionally, the execution subject may also generate the transmitter coordinate conversion information according to each determined target space position information set through the following steps:

[0115] In the first step, each target spatial position information corresponding to a preset conversion transmitter identifier in each determined target spatial position information set is determined as a first spatial position information group.

[0116] In the second step, each preset transmitter identification that is not a preset conversion transmitter identification among the preset transmitter identifications is determined as a target transmitter identification group.

[0117] In the third step, for each target transmitter identifier included in the target transmitter identifier group, the following steps are performed:

[0118] In the first sub-step, each target spatial position information corresponding to the target transmitter identification in the determined target spatial position information set is determined as a second spatial position information group.

[0119] In the second sub-step, according to a preset point cloud registration algorithm, the first spatial position information group and the second spatial position information group are solved to obtain coordinate transformation information. The preset point cloud registration algorithm may be a pre-set method for solving the optimal rotation matrix and translation vector between two coordinate systems. For example, the preset point cloud registration algorithm may be a Kabsch point cloud registration algorithm.

[0120] The fourth step is to determine the obtained coordinate transformation information as the transmitter coordinate transformation information.

[0121] from Figure 4 It can be seen that Figure 2 Compared with the description of some corresponding embodiments, Figure 4 The process 400 of the camera calibration method in some corresponding embodiments embodies the steps of how to generate coordinate conversion information between the coordinate systems of each electromagnetic signal transmitter. Therefore, the schemes described in these embodiments can obtain the conversion relationship between the coordinate systems of each electromagnetic signal transmitter, so that when multiple electromagnetic signal transmitters are used, the spatial coordinates can be uniformly converted to the same electromagnetic signal transmitter coordinate system first, and then converted to the camera coordinate system, so as to improve the consistency of the posture conversion corresponding to each electromagnetic sensor to the camera coordinate system, and then improve the accuracy of motion capture.

[0122] Continue to refer Figure 5 , which shows a process 500 of some embodiments of the method for generating user motion capture images according to the present disclosure. The method for generating user motion capture images, applied to an electromagnetic motion capture device, comprises the following steps:

[0123] Step 501 , controlling a camera included in an electromagnetic motion capture device to collect user motion images according to preset camera parameter information, and controlling each electromagnetic signal transmitter included in the electromagnetic motion capture device to perform a start operation.

[0124] In some embodiments, the execution subject may control the camera included in the electromagnetic motion capture device to collect user motion images according to preset camera parameter information, and control the various electromagnetic signal transmitters included in the electromagnetic motion capture device to perform a turn-on operation. The preset camera parameter information is camera parameter information pre-generated according to the camera calibration method described in steps 201-20233. It should be noted that the various electromagnetic sensors included in the electromagnetic motion capture device need to be set on the user's limbs to be detected. The user motion image may characterize the user's motion and posture. The turn-on operation may be an operation to turn on the electromagnetic signal transmitter. As an example, if the electromagnetic motion capture device needs to detect the user's hand posture, then the various electromagnetic sensors are set at 21 key points that characterize the user's hand posture. The user motion image may characterize the user's hand posture.

[0125] Step 502: Control each electromagnetic sensor included in the electromagnetic motion capture device to collect electromagnetic signals to obtain a sensor posture information group.

[0126] Step 503: Generate, based on the sensor posture information group, each sensor spatial position information group corresponding to each preset transmitter identifier.

[0127] In some embodiments, the above execution subject may perform the following steps for each preset transmitter identifier:

[0128] The first step is to determine each sensor posture information corresponding to the preset transmitter identifier in the sensor posture information group as a target sensor posture information group.

[0129] The second step is to generate a sensor space position information group according to the above target sensor posture information group. The specific implementation method and the technical effect brought are the same as some optional implementation methods of step 20231, which will not be repeated here.

[0130] Step 504: for each preset transmitter identifier, perform the following steps:

[0131] Step 5041: Determine the sensor space position information group corresponding to the preset transmitter identifier in each sensor space position information group as the target space position information group.

[0132] In some embodiments, the execution entity may determine the sensor space location information group corresponding to the preset transmitter identifier among the sensor space location information groups as the target space location information group.

[0133] Step 5042, generating a sensor image position information group according to the preset camera parameter information, the target space position information group and the preset coordinate system conversion information corresponding to the preset transmitter identifier.

[0134] In some embodiments, the execution subject may generate a sensor image position information group according to the preset camera parameter information, the target space position information group and the preset coordinate system conversion information corresponding to the preset transmitter identifier. The preset coordinate system conversion information may be the coordinate system conversion information pre-generated according to the camera calibration method implemented in the above steps 201-20233. In practice, first, for each target space position information included in the target space position information group, the execution subject may input the preset camera parameter information, the target space position information and the preset coordinate system conversion information corresponding to the preset transmitter identifier into the following formula to obtain the sensor image position information.

[0135]

[0136] Wherein, n may represent the coordinate system of the electromagnetic signal transmitter corresponding to the above-mentioned preset transmitter identification. (x n ,y n , z n ) can represent the three-dimensional coordinate point of the corresponding electromagnetic sensor in the coordinate system of the electromagnetic signal transmitter corresponding to the above-mentioned preset transmitter identifier, that is, the above-mentioned target space position information. (x t ,y t , z t ) can represent the three-dimensional coordinate point of the corresponding electromagnetic sensor in the camera coordinate system. (u t , v t ) can represent the two-dimensional coordinate point of the corresponding electromagnetic sensor in the pixel coordinate system, that is, the sensor image position information. wn It can represent the rotation matrix included in the preset coordinate system conversion information corresponding to the preset transmitter identifier. wn It can represent the translation vector included in the preset coordinate system conversion information corresponding to the above preset transmitter identifier.

[0137] Then, the obtained individual sensor image position information are determined as a sensor image position information group.

[0138] Optionally, the execution subject may generate a sensor image position information group according to the preset camera parameter information, the target space position information group and the preset transmitter coordinate conversion information through the following steps:

[0139] The first step, in response to determining that the above-mentioned preset transmitter identifier is a preset conversion transmitter identifier, for each target space position information included in the target space position information group, the above-mentioned preset camera parameter information, the above-mentioned target space position information and the preset coordinate system conversion information corresponding to the above-mentioned preset transmitter identifier are input into the following formula to obtain the sensor image position information.

[0140]

[0141] The preset transmitter coordinate conversion information may be transmitter coordinate conversion information pre-generated according to the camera calibration method implemented in steps 401-403. (x0, y0, z0) may represent the three-dimensional coordinate point of the corresponding electromagnetic sensor in the electromagnetic signal transmitter coordinate system corresponding to the preset transmitter identifier (or preset conversion transmitter identifier), that is, the target space position information. (x t ,y t , z t ) can represent the three-dimensional coordinate point of the corresponding electromagnetic sensor in the camera coordinate system. (u t , v t ) can represent the two-dimensional coordinate point of the corresponding electromagnetic sensor in the pixel coordinate system, that is, the sensor image position information. Then, in response to determining that the above-mentioned preset transmitter identifier is not the preset conversion transmitter identifier, for each target space position information included in the target space position information group, the above-mentioned target space position information is input into the following formula to obtain the conversion space position information.

[0142] (x0, y0, z0) = R i0 ×(x i ,y i , z i ) T +T i0

[0143] Among them, (x i ,y i , z i) can represent the three-dimensional coordinate point of the corresponding electromagnetic sensor in the electromagnetic signal transmitter coordinate system corresponding to the above-mentioned preset transmitter identifier, that is, the above-mentioned target space position information. (x0, y0, z0) can represent the three-dimensional coordinate point of the corresponding electromagnetic sensor in the electromagnetic signal transmitter coordinate system corresponding to the above-mentioned preset conversion transmitter identifier, that is, the conversion space position information. Then, the above-mentioned preset camera parameter information, the above-mentioned conversion space position information and the preset coordinate system conversion information corresponding to the above-mentioned preset transmitter identifier are input into the following formula to obtain the sensor image position information.

[0144]

[0145] Finally, the obtained individual sensor image position information is determined as a sensor image position information group.

[0146] As an example, the electromagnetic motion capture device needs to capture the user's hand posture. When the electromagnetic motion capture device includes two electromagnetic signal transmitters, the source1 coordinate system is the electromagnetic signal transmitter coordinate system corresponding to the preset conversion transmitter identifier, and the source2 coordinate system is the electromagnetic signal transmitter coordinate system corresponding to the preset transmitter identifier as electromagnetic signal transmitter 2. The conversion relationship between the electromagnetic sensor coordinate system, each electromagnetic signal transmitter coordinate system and the camera coordinate system is as follows: Figure 6 shown.

[0147] Step 505 : performing image reconstruction processing on the user motion image according to each generated sensor image position information group to obtain a user motion capture image.

[0148] In some embodiments, the execution subject may perform image reconstruction processing on the user motion image according to the generated sensor image position information groups to obtain a user motion capture image. In practice, the execution subject may draw the user's limb posture on the user motion image according to the generated sensor image position information groups and a preset user limb model to obtain a user motion capture image. The preset user limb model may be a preset model that describes the user's limbs that need to be captured. As an example, when it is necessary to capture the user's hand gesture, the preset user limb model may be a pre-built hand model for drawing the hand outline.

[0149] The above-mentioned embodiments of the present disclosure have the following beneficial effects: the user motion capture image generation method of some embodiments of the present disclosure can improve the accuracy and real-time performance of capturing user motion. Specifically, the reasons for the low accuracy and poor real-time performance of capturing user motion are: the mechanical motion capture device requires the user to wear a complete set of motion capture mechanical devices, which is inconvenient for the user to use; the acoustic motion capture device has poor real-time performance and low accuracy when the object is captured, and is easily interfered by surrounding noise; the optical motion capture device uses multiple precise and complex high-speed optical cameras, which are easy to confuse and block each other; the inertial navigation motion capture device is generally large in size and cumbersome to operate, and due to the characteristics of the sensor, inertial motion capture is prone to error accumulation, resulting in low accuracy, resulting in low accuracy of capturing user motion, and poor real-time performance. Based on this, the user motion capture image generation method of some embodiments of the present disclosure uses an electromagnetic motion capture device when capturing user motion, thereby reducing the error caused by external interference, and there is no need to wait for signal sending and receiving, thereby improving the real-time performance and capture accuracy of motion capture. In addition, before the electromagnetic motion capture device is used to capture the user's movements, the camera and each electromagnetic signal transmitter are first calibrated together, thereby improving the accuracy of user movement capture and user movement recognition. As a result, the accuracy and real-time performance of capturing user movements can be improved.

[0150] Reference below Figure 7 , which shows an electromagnetic motion capture device (e.g. Figure 1 Schematic diagram of the hardware structure of the electromagnetic motion capture device 700 shown in FIG. Figure 7 The electromagnetic motion capture device shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present disclosure.

[0151] like Figure 7As shown, the electromagnetic motion capture device 700 may include a processing device (such as a central processing unit, a graphics processing unit, etc.) 701, a memory 702, an input unit 703, and an output unit 704. Among them, the processing device 701, the memory 702, the input unit 703 and the output unit 704 are connected to each other through a bus 705. Here, the method according to the embodiment of the present disclosure can be implemented as a computer program and stored in the memory 702. For example, some embodiments of the present disclosure include a computer program product, which includes a computer program carried on a computer-readable medium, and the computer program contains program code for executing the method shown in the flowchart. The processing device 701 in the electromagnetic motion capture device specifically implements the camera calibration method or user motion capture image generation method of the present disclosure by calling the above-mentioned computer program stored in the memory 702. In some implementations, the output unit 704 may include at least one electromagnetic signal transmitter, which can be used to transmit electromagnetic signals. The input unit 703 may include at least one electromagnetic sensor and a camera. The electromagnetic sensor can be used to sense electromagnetic signals. The camera can be used to collect images.

[0152] It should be noted that the computer-readable medium recorded in some embodiments of the present disclosure may be a computer-readable signal medium or a computer-readable storage medium or any combination of the above two. The computer-readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In some embodiments of the present disclosure, a computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in combination with an instruction execution system, device or device. In some embodiments of the present disclosure, a computer-readable signal medium may include a data signal propagated in a baseband or as part of a carrier wave, which carries a computer-readable program code. This propagated data signal may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. The computer readable signal medium may also be any computer readable medium other than a computer readable storage medium, which may send, propagate or transmit a program for use by or in conjunction with an instruction execution system, apparatus or device. The program code contained on the computer readable medium may be transmitted using any suitable medium, including but not limited to: wires, optical cables, RF (radio frequency), etc., or any suitable combination of the above.

[0153] In some embodiments, the client and the server may communicate using any currently known or future developed network protocol such as HTTP (HyperText Transfer Protocol), and may be interconnected with any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include a local area network ("LAN"), a wide area network ("WAN"), an internet (e.g., the Internet), and a peer-to-peer network (e.g., an ad hoc peer-to-peer network), as well as any currently known or future developed network.

[0154] The computer-readable medium may be included in the electromagnetic motion capture device; or it may exist independently without being assembled into the electromagnetic motion capture device. The computer-readable medium carries one or more programs. When the one or more programs are executed by the electromagnetic motion capture device, the electromagnetic motion capture device: performs internal calibration processing on the camera included in the electromagnetic motion capture device to obtain camera parameter information; determines the initial frame number and initial camera pose information, and performs the following calibration steps according to the initial frame number and initial camera pose information: controls the camera to collect a checkerboard image according to the camera parameter information and initial camera pose information; generates sensor data corresponding to each electromagnetic sensor included in the electromagnetic motion capture device according to the checkerboard image; The method comprises the steps of: determining a sensor image position information group corresponding to the preset transmitter identifier, wherein each preset transmitter identifier corresponds to an electromagnetic signal transmitter included in the electromagnetic motion capture device; determining a target image position information set and a target space position information set according to the sensor image position information group and the sensor space position information group; generating coordinate system conversion information according to each determined target image position information set and each determined target space position information set in response to determining that the initial frame number meets the preset frame number condition; or controlling The camera included in the electromagnetic motion capture device is controlled to collect user motion images according to preset camera parameter information, and each electromagnetic signal transmitter included in the electromagnetic motion capture device is controlled to perform a start operation, wherein the preset camera parameter information is pre-generated according to the method described in any implementation of the first aspect; the electromagnetic sensors included in the electromagnetic motion capture device are controlled to collect electromagnetic signals to obtain a sensor posture information group; according to the sensor posture information group, each sensor space position information group corresponding to each preset transmitter identifier is generated; for each preset transmitter identifier, the following steps are performed: the sensor space position information group corresponding to the preset transmitter identifier in the above-mentioned each sensor space position information group is determined as a target space position information group; according to the preset camera parameter information, the target space position information group and the preset coordinate system conversion information corresponding to the preset transmitter identifier, a sensor image position information group is generated, wherein the preset coordinate system conversion information is pre-generated according to the method described in any implementation of the first aspect; according to the generated each sensor image position information group, the user motion image is subjected to image reconstruction processing to obtain a user motion capture image.

[0155] Computer program code for performing the operations of some embodiments of the present disclosure may be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, C++, and conventional procedural programming languages ​​such as "C" or similar programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, as a separate software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving a remote computer, the remote computer may be connected to the user's computer via any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0156] The flow chart and block diagram in the accompanying drawings illustrate the possible architecture, function and operation of the system, method and computer program product according to various embodiments of the present disclosure. In this regard, each square box in the flow chart or block diagram can represent a module, a program segment or a part of a code, and the module, the program segment or a part of the code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some implementations as replacements, the functions marked in the square box can also occur in a sequence different from that marked in the accompanying drawings. For example, two square boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each square box in the block diagram and / or flow chart, and the combination of the square boxes in the block diagram and / or flow chart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0157] The functions described above herein may be performed at least in part by one or more hardware logic components. For example, without limitation, exemplary types of hardware logic components that may be used include: field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), complex programmable logic devices (CPLDs), and the like.

[0158] The above descriptions are only some preferred embodiments of the present disclosure and an explanation of the technical principles used. Those skilled in the art should understand that the scope of the invention involved in the embodiments of the present disclosure is not limited to the technical solutions formed by a specific combination of the above-mentioned technical features, but should also cover other technical solutions formed by any combination of the above-mentioned technical features or their equivalent features without departing from the above-mentioned inventive concept. For example, the above-mentioned features are replaced with the technical features with similar functions disclosed in the embodiments of the present disclosure (but not limited to) and the technical solutions formed.

Claims

1. A camera calibration method, applied to an electromagnetic motion capture device, comprising: Performing internal calibration processing on the camera included in the electromagnetic motion capture device to obtain camera parameter information; Determine the initial frame number and initial camera pose information, and perform the following calibration steps based on the initial frame number and initial camera pose information: Controlling the camera to collect a checkerboard image according to the camera parameter information and the initial camera pose information; Generating a sensor image position information group corresponding to each electromagnetic sensor included in the electromagnetic motion capture device according to the chessboard image; For each preset transmitter ID, perform the following steps: Determine a sensor spatial position information group corresponding to the preset transmitter identifier, wherein each preset transmitter identifier corresponds to an electromagnetic signal transmitter included in the electromagnetic motion capture device; Determine a target image position information set and a target space position information set according to the sensor image position information set and the sensor space position information set; In response to determining that the initial frame number satisfies a preset frame number condition, coordinate system conversion information is generated according to the determined sets of target image position information and the determined sets of target space position information.

2. The method according to claim 1, wherein: The calibration step also includes: In response to determining that the initial frame number does not meet the preset frame number condition, the initial frame number and the initial camera pose information are updated, and the updated initial frame number is used as the initial frame number, and the updated initial camera pose information is used as the initial camera pose information, and the calibration step is continued.

3. The method according to claim 1, wherein: The step of determining a target image position information set and a target space position information set according to the sensor image position information set and the sensor space position information set comprises: Dividing the sensor image position information group to obtain a training image position information group and a verification image position information group; Dividing the sensor spatial position information group to obtain a training spatial position information group and a verification spatial position information group; Generate initial coordinate system conversion information according to the training image position information group and the training space position information group; Generate a reprojection position information group according to the camera parameter information, the initial coordinate system conversion information and the verification space position information group; Generate a projection average error according to the reprojection position information group and the verification image position information group; In response to determining that the projection average error satisfies a preset error condition, the training image position information set is determined as a target image position information set, and the training space position information set is determined as a target space position information set.

4. The method according to claim 3, wherein: The method further comprises: In response to determining that the projection average error does not meet the preset error condition, the initial camera pose information is updated, and the updated initial camera pose information is used as the initial camera pose information to continue the calibration step.

5. The method according to claim 1, wherein: The determining of the sensor spatial position information group corresponding to the preset transmitter identifier includes: Controlling each electromagnetic sensor corresponding to the preset transmitter identifier to collect electromagnetic signals to obtain a sensor posture information group; A sensor spatial position information group is generated according to the sensor posture information group.

6. The method according to claim 1, wherein: The generating of coordinate system conversion information according to the determined target image position information sets and the determined target space position information sets comprises: For each target space position information included in each determined target space position information set, the following steps are performed: generating transmitter distance information according to the target spatial position information; In response to determining that the transmitter distance information satisfies a preset distance condition, determining the target spatial position information as target sensor spatial position information, and determining the target image position information corresponding to the target spatial position information in the determined respective target image position information sets as target sensor image position information; Coordinate system conversion information is generated according to the determined spatial position information of each target sensor and the determined image position information of each target sensor.

7. The method according to claim 1, wherein: The method further comprises: The emitter coordinate transformation information is generated based on the generated coordinate system transformation information.

8. A method for generating a user motion capture image, applied to an electromagnetic motion capture device, comprising: Controlling the camera included in the electromagnetic motion capture device to collect user motion images according to preset camera parameter information, and controlling each electromagnetic signal transmitter included in the electromagnetic motion capture device to perform a start operation, wherein the preset camera parameter information is pre-generated according to the camera calibration method according to any one of claims 1 to 7; Controlling each electromagnetic sensor included in the electromagnetic motion capture device to collect electromagnetic signals and obtain a sensor posture information group; According to the sensor posture information group, generating each sensor spatial position information group corresponding to each preset transmitter identifier; For each preset transmitter ID, perform the following steps: Determine the sensor space position information group corresponding to the preset transmitter identifier among the sensor space position information groups as the target space position information group; Generate a sensor image position information group according to the preset camera parameter information, the target space position information group and the preset coordinate system conversion information corresponding to the preset transmitter identifier, wherein the preset coordinate system conversion information is pre-generated by the camera calibration method according to any one of claims 1 to 7; According to the generated sensor image position information groups, image reconstruction processing is performed on the user action image to obtain a user action capture image.

9. An electromagnetic motion capture device, comprising: at least one electromagnetic signal transmitter, configured to transmit an electromagnetic signal; at least one electromagnetic sensor for sensing electromagnetic signals; A camera, used to capture images; one or more processors; a storage device having one or more programs stored thereon, When the one or more programs are executed by the one or more processors, the one or more processors implement the method according to any one of claims 1 to 7 or 8.

10. A computer readable medium having a computer program stored thereon, wherein: When the computer program is executed by a processor, the method according to any one of claims 1 to 7 or 8 is implemented.

Images