Camera calibration and attitude reconstruction method, related device, equipment and storage medium

Abstract

The invention discloses a camera calibration and attitude reconstruction method, a related device, equipment and a storage medium, and the method comprises the steps: obtaining a plurality of first target images, each first target image being obtained through the collection of a same target object at the same moment by a plurality of cameras located at different positions; determining a plurality of image block groups from the plurality of first target images; different image blocks in the image block group belong to different first target images and are image areas containing at least one same key point in the target object; determining a plurality of groups of first matching points among different image blocks in each image block group; and calibrating each camera based on the plurality of groups of first matching points in each image block group to obtain a target calibration parameter of each camera. In this way, the position of the camera can be accurately calibrated.

Description

Technical Field

[0001] The present application relates to the field of three-dimensional reconstruction technology, and in particular to a camera calibration and posture reconstruction method and related devices, equipment and storage media. Background Art

[0002] Reconstructing the 3D posture of a target in a natural scene is a basic task of visual perception. Taking the human body as an example, the traditional method of estimating the human posture based on a single camera is prone to occlusion and tracking failure, and the posture measurement accuracy does not meet the requirements. Although the multi-camera measurement method can solve the occlusion problem well, the final 3D posture reconstruction accuracy is easily affected by the camera calibration accuracy. Therefore, how to accurately calibrate each camera is of great significance for posture reconstruction. Summary of the invention

[0003] The main technical problem solved by the present application is to provide a camera calibration and posture reconstruction method and related devices, equipment and storage media, which can accurately calibrate the position of the camera.

[0004] To solve the above technical problems, another technical solution adopted in the present application is: providing a camera calibration method, the method comprising: acquiring a plurality of first target images, each first target image being acquired by a plurality of cameras located at different positions at the same time and respectively capturing the same target object; determining a plurality of image block groups from the plurality of first target images; different image blocks in the image block groups belonging to different first target images, and being image regions containing at least one identical key point in the target object; determining a plurality of groups of first matching points between different image blocks in each image block group; calibrating each camera based on the plurality of groups of first matching points in each image block group, and obtaining target calibration parameters of each camera.

[0005] In order to solve the above technical problems, a technical solution adopted in the present application is: to provide a posture reconstruction method, the method comprising: obtaining a plurality of second target images, each second target image being respectively acquired by a plurality of cameras located at different positions on the same target object; performing key point recognition on the target object in each second target image to obtain the pixel position of each key point in the corresponding second target image; determining the target position of each key point in the world coordinate system based on the target calibration parameters of each camera and the pixel position of each key point in the corresponding second target image; the target calibration parameters of each camera are the conversion parameters between the camera coordinate system of each camera and the world coordinate system; wherein, the target calibration parameters of each camera are determined using the above camera calibration method.

[0006] In order to solve the above technical problems, another technical solution adopted in the present application is: to provide a camera calibration device, including: a first acquisition module, a first determination module, a second determination module and a calibration module; the first acquisition module is used to acquire a plurality of first target images, each of which is acquired by a plurality of cameras located at different positions at the same time for the same target object; the first determination module is used to determine a plurality of image block groups from the plurality of first target images; different image blocks in the image block groups belong to different first target images and are image areas containing at least one identical key point in the target object; the second determination module is used to determine a plurality of groups of first matching points between different image blocks in each image block group; the calibration module is used to calibrate each camera based on the plurality of groups of first matching points in each image block group to obtain target calibration parameters of each camera.

[0007] In order to solve the above technical problems, another technical solution adopted in the present application is: to provide a posture reconstruction device, including: a second acquisition module, an identification module and a position determination module; the second acquisition module is used to acquire a plurality of second target images, each second target image is respectively acquired by a plurality of cameras located at different positions on the same target object; the identification module is used to identify key points of the target object in each second target image, and obtain the pixel position of each key point in the corresponding second target image; the position determination module is used to determine the target position of each key point in the world coordinate system based on the target calibration parameters of each camera and the pixel position of each key point in the corresponding second target image; the target calibration parameters of each camera are the conversion parameters between the camera coordinate system of each camera and the world coordinate system; wherein, the target calibration parameters of each camera are determined using the above-mentioned camera calibration method.

[0008] To solve the above technical problems, another technical solution adopted in the present application is: to provide an electronic device, comprising a memory and a processor coupled to each other, the memory storing program instructions; the processor is used to execute the program instructions stored in the memory to implement the above method.

[0009] In order to solve the above technical problem, another technical solution adopted by the present application is: providing a computer-readable storage medium for storing program instructions, which can be executed to implement the above method.

[0010] The above scheme, based on the key points, determines the image block group containing the same key points of the target object, and then performs camera calibration based on multiple groups of first matching points between different image blocks in each image block group. It can be seen that the first matching points of the present application include matching points that are located in the image block and are not key points in addition to the key points. In other words, the present application expands the distribution range of matching points in the first target image. Compared with the method of using only a small number of matching key points for camera calibration, the method of introducing more matching points in the image block of the present application can reduce the impact of false detection and false matching of key points, which is conducive to accurately calibrating the position of the camera. BRIEF DESCRIPTION OF THE DRAWINGS

[0011] Figure 1 It is a flowchart of an embodiment of a camera calibration method provided by the present application;

[0012] Figure 2 yes Figure 1 The flowchart of step S12 is shown as an embodiment;

[0013] Figure 3 yes Figure 1 The flowchart of step S14 is shown as an embodiment;

[0014] Figure 4 It is a flowchart of an embodiment of a posture reconstruction method provided by the present application;

[0015] Figure 5 It is a schematic diagram of the framework of an embodiment of a camera calibration device provided by the present application;

[0016] Figure 6 It is a schematic diagram of the framework of an embodiment of a posture reconstruction device provided by the present application;

[0017] Figure 7 It is a schematic diagram of a framework of an embodiment of an electronic device provided by the present application;

[0018] Figure 8 It is a schematic diagram of the framework of the computer-readable storage medium provided by this application. DETAILED DESCRIPTION

[0019] In order to make the purpose, technical solution and effect of the present application clearer and more specific, the present application is further described in detail below with reference to the accompanying drawings and examples.

[0020] In addition, if there are descriptions involving "first", "second", etc. in the embodiments of the present application, the descriptions of "first", "second", etc. are only used for descriptive purposes and cannot be understood as indicating or suggesting their relative importance or implicitly indicating the number of technical features indicated. Therefore, the features defined as "first" and "second" may explicitly or implicitly include at least one of the features. In addition, the technical solutions between the various embodiments can be combined with each other, but they must be based on the ability of ordinary technicians in the field to implement them. When the combination of technical solutions is contradictory or cannot be implemented, it should be deemed that such combination of technical solutions does not exist and is not within the scope of protection required by this application.

[0021] See also Figure 1 , Figure 1 is a flow chart of an embodiment of the camera calibration method provided by the present application. It should be noted that if there are substantially the same results, this embodiment does not use Figure 1 The process sequence shown is limited. Figure 1 As shown, this embodiment includes:

[0022] S11: Acquire a plurality of first target images, each of which is acquired by capturing the same target object at the same time by a plurality of cameras located at different positions.

[0023] This embodiment is used to determine the image block groups containing the same key points by using the key points in each first target image, and then perform camera calibration based on the multiple groups of first matching points corresponding to each image block group.

[0024] It should be noted that, in this embodiment, when calibrating each camera, there is no need to set up additional calibration objects (such as a chessboard, etc.), but the first target images obtained by respectively capturing the same target object in the image capture scene at the same image capture time by cameras located at different positions are used to calibrate each camera. Specifically, multiple groups of matching points between different first target images are used to calibrate each camera. Each group of matching points in this embodiment represents the mapping point (projection point) of the same position point on the target object on different first target images, wherein, in order to ensure that the same position point on the same target object is used and the same position point captured by each camera is in the same position in the world coordinate system, the images captured by each camera at the same time for the same target object are used as each first target image.

[0025] The target object may be a stationary or moving target object in the acquisition scene, and may be, but not limited to, a living body, such as a human body, etc. The specific object may be determined according to the object to be reconstructed, and is not specifically limited here.

[0026] Taking into account that even if the target object is in motion, the position movement of the target object can be ignored if the deviation at the acquisition time is small, the same moment in this embodiment is a moment with a time deviation within a certain range, that is, images acquired by different cameras within a certain time deviation can be considered as images acquired at the same time.

[0027] The first target images may be images captured in real time of the same target object, or may be images captured at the same time from videos captured by multiple cameras. The first target images captured at the same time from videos captured by multiple cameras may be extracted in the following manner.

[0028] In one embodiment, at least one time coder may be connected to each camera, so as to use the time coder to add a time stamp to each video frame in the video captured by the camera, so as to obtain each first target image captured at the same time.

[0029] In another embodiment, considering that connecting one time coder to multiple cameras is not convenient for camera deployment, time coders equal to the number of cameras are provided, wherein each camera is connected to a corresponding time coder. Each time coder is used to synchronize video frames of videos captured by cameras at different positions, so as to obtain the first target image captured by cameras at different positions at the same time.

[0030] Specifically, after each camera captures the video, the videos captured by all cameras are post-processed. The time coder records the time information of each frame through the audio signal, such as the 24th frame at 10:10:10 Beijing time. Therefore, the start frame time of each captured video can be obtained, and all videos are extracted to obtain video frames. Combining the time code and the acquisition frame rate, the specific timestamp of each video frame can be obtained. The first target image captured by each camera at the same time can be obtained by intercepting the common time segment of all cameras. The specific acquisition frame rate can be determined according to the actual scene.

[0031] The above method of connecting one camera to one time coder allows the cameras to work independently without being restricted by the site, and is applicable to both indoor and outdoor scenes. It should be noted that the positions of the cameras can be placed according to actual needs to ensure that the shooting images of the cameras overlap.

[0032] S12: Determine a plurality of image block groups from a plurality of first target images; different image blocks in the image block groups belong to different first target images and are image regions containing at least one same key point in the target object.

[0033] Key points are some special position points on the target object that are easy to identify or used to characterize the target object. In this embodiment, the category of the key points needs to be determined first, and then the key points of the same category are selected from a plurality of first images, wherein the key points of the same category are essentially a group of matching points, and the image area of ​​the preset range of each key point of the same category is taken as an image block, and the image blocks where the same key points are located are combined to obtain each image block group. The same key points in this embodiment represent key points of the same category.

[0034] Specifically, see Figure 2 , Figure 2 yes Figure 1 The flowchart of an embodiment of step S12 is shown. In this embodiment, step S12 further includes:

[0035] S21: using the key point recognition model to recognize each first target image, and obtaining a key point recognition result of the target object in each first target image; the key point recognition result includes the position and category of each key point in the target object.

[0036] The key point recognition model is used to recognize each key point on the target object in each first target image, and obtain the recognition result of each key point. The key point recognition result includes the position of each key point in the target object, the category to which it belongs, and at least part of the confidence of the corresponding category. In this embodiment, the key point recognition result includes at least the position of each key point and the category to which it belongs.

[0037] Taking the target object as a human body as an example, the key points are the joints of the human body, and the key point recognition result is the joint point recognition result. The joint point recognition result includes the position of each joint point in the human body and the category of each joint point.

[0038] Among them, joint points include, for example, the head, neck, left shoulder, left elbow, left wrist, left palm, left thumb, left middle finger tip, right shoulder, right elbow, right wrist, right palm, right thumb, right middle finger tip, upper spine, middle spine, lower spine, left hip, left knee, left ankle, left foot, right hip, right knee, right ankle and right foot, and the category of each joint point is, for example, the neck or right thumb, etc.

[0039] S22: Based on the key point recognition results corresponding to each first target image, key points of the same category are selected from a plurality of first target images, and the key points of the same category are combined into a key point group.

[0040] In one embodiment, key points of the same category may be directly selected from a plurality of first target images according to the category to which each key point belongs, and the key points of the same category may be combined into a key point group corresponding to the category.

[0041] For example, the number of first target images is 5, and key points of the left palm can be selected from the 5 first target images, and the key points of the left palm can be combined into a key point group, that is, combined into a key point group of the left palm.

[0042] In another embodiment, considering the sparsity of key points and the influence of factors such as the camera shooting distance, key points of the same category in different first target images may have errors. In order to ensure the accuracy of subsequent calibration, some key points with low confidence can be filtered out according to the confidence recognition results of the corresponding categories of each key point, so that the confidence of each key point in the key point group finally selected is greater than the confidence threshold. The confidence threshold can be set according to actual needs, for example, the confidence threshold is set to any confidence value between 0.5 and 0.9.

[0043] Since the first target image is an image of the same target object captured by different cameras at the same time, the key points of the same category selected represent the projection points of the same key point on the target object on different first target images. That is, the key points of the same category are a set of matching points between different target images.

[0044] S23: For the first target image where each key point in each key point group is located, an image area within a preset range of the key point in the first target image is used as an image block containing the key point.

[0045] Different from the above-mentioned embodiment, this step performs region expansion based on key points of the same category in different first target images, and obtains image blocks containing key points of the same category in each first target image. However, if the different first target images do not contain key points belonging to the same category, the region expansion operation is not performed on the key points.

[0046] For example, if the confidence that a key point in the two first target images is the left palm is greater than the confidence threshold, or both images contain the left palm key point, the left palm key point is used as the center point to perform region expansion (for example, expand an image region of 50*50 pixels) to obtain an image block containing the left palm key point in each image. However, if only one target image recognizes the key point belonging to the left palm, the region expansion operation with the left palm key point as the center point is not performed.

[0047] S24: For each key point group, combine the image blocks corresponding to each key point in the key point group to obtain an image block group corresponding to the key point group.

[0048] For example, for a group of key points belonging to the left palm, the image blocks corresponding to the left palm key points in the key point group are combined to obtain an image block group related to the left palm.

[0049] In summary, this embodiment first identifies the position of each key point through a key point recognition model. Although there may be errors in the identified position, each key point is roughly near the identified key point position. This embodiment uses the key point expansion area method to facilitate the subsequent feature point matching between image blocks. Improve the accuracy of key point matching, and at the same time increase new matching points in addition to key points, and increase the distribution of matching points in the image.

[0050] Furthermore, the present application utilizes key points of the same category to determine image blocks, and determines multiple groups of first matching points between subsequent different image blocks. Compared with the method of matching between entire images, the method of the present application greatly reduces the matching area, reduces the difficulty of matching, and reduces the risk of high errors in matching the entire image, which is conducive to finding accurate groups of first matching points, and is conducive to using high-quality groups of first matching points to accurately calibrate the position of each camera.

[0051] S13: Determine multiple groups of first matching points between different image blocks in each image block group.

[0052] In one embodiment, an existing deep learning feature matching algorithm may be used to perform pixel-by-pixel feature point matching between image blocks to obtain multiple groups of first matching points between different image blocks.

[0053] Exemplarily, feature points can be first extracted from each image block of each image block group, and then a feature matching algorithm of deep learning can be used to perform pixel-by-pixel feature point matching between each image block to obtain feature point matching results. Then, based on the feature point matching results between different image blocks in each image block group, multiple groups of first matching points between different image blocks in each image block group are determined.

[0054] In a specific embodiment, the feature point matching result includes the matching degree between the feature points in the image block group. In order to find multiple groups of matching points with higher accuracy, for each image block group, feature points in the image block group with a matching degree greater than a preset threshold are obtained to obtain multiple groups of high-quality first matching points of the image block group, that is, the matching degree between different matching points in each group of first matching points is greater than a preset threshold (for example, 0.9).

[0055] In some embodiments, there may be a common (overlapping) area between the image block groups corresponding to the key point groups of different categories, and there may be overlapping matching points in the corresponding matching point groups. In this embodiment, all matching points between the image blocks can be deduplicated, and the matching point groups obtained after deduplication are used as the final multiple groups of first matching points. Each group of first matching points is a mapping point (projection point) of a certain three-dimensional space point in the acquisition environment at the same time in different first target images, that is, a group of multi-view matching points.

[0056] S14: Calibrate each camera based on the multiple groups of first matching points in each image block group to obtain target calibration parameters of each camera.

[0057] In one embodiment, a camera may be selected from each camera as the main camera (i.e., the first camera hereinafter), and the camera coordinate system of the main camera may be used as the reference coordinate system. Then, based on the multiple sets of first matching points determined above, the other cameras (i.e., the second camera hereinafter) may be calibrated to obtain the first conversion parameters between the other cameras and the main camera. Then, the pre-established world coordinate system may be used to determine the second conversion parameters between each camera and the world coordinate system, wherein the second conversion parameters between each camera and the world coordinate system are the target calibration parameters of each camera. For details, please refer to the following Figure 3 Description of the illustrated embodiment.

[0058] In another embodiment, the camera coordinate system of the main camera (the above-mentioned reference coordinate system) can be directly used as the world coordinate system, and then the other cameras are calibrated based on the multiple groups of first matching points determined above to obtain the first conversion parameters between the other cameras and the main camera, and the first conversion parameters between the other cameras and the main camera are directly used as the target calibration parameters of the other cameras.

[0059] The above scheme determines the image block group containing the same key points on the basis of the key points, and then performs camera calibration based on multiple groups of first matching points between different image blocks in each image block group. It can be seen that the first matching points of the present application include points that are located in the image block and are not key points in addition to the key points. In other words, the present application expands the distribution range of the matching points in the first target image. Compared with the method of only using a small number of matched key points for camera calibration, the method of introducing more matching points in the image block of the present application can reduce the impact of false detection and false matching of key points, which is conducive to accurately calibrating the position of the camera.

[0060] See also Figure 3 , Figure 3 yes Figure 1 The flowchart of an embodiment of step S14 is shown. In this embodiment, step S14 further includes:

[0061] S31: selecting a camera from each camera as a first camera, and using the camera coordinate system of the first camera as a reference coordinate system.

[0062] S32: Calibrate each second camera based on multiple groups of first matching points in each image block group to obtain a first calibration parameter of each second camera; the second camera is a camera other than the first camera, and the first calibration parameter represents a first conversion parameter between a camera coordinate system corresponding to the second camera and a reference coordinate system.

[0063] S33: Based on the reference coordinate system, each first calibration parameter, a world coordinate system established with a preset shape marker, and the pixel position of the preset shape marker in each first target image, determine the second calibration parameter of each camera, the second calibration parameter representing the second conversion parameter between the corresponding camera and the world coordinate system.

[0064] S34: Using the second calibration parameters of each camera as target calibration parameters of the corresponding camera.

[0065] Steps S31 and S32 are used to calibrate each second camera using the first camera to obtain the first conversion parameters between the camera coordinate system of each second camera and the reference coordinate system of the first camera. Steps S33 and S34 are used to establish a world coordinate system using a preset shape marker to determine the second conversion parameters between each camera and the world coordinate system. Figure 3 The illustrated embodiment uses the conversion parameters between each camera and the world coordinate system as the target calibration parameters of the camera.

[0066] It should be noted that the role of the above-mentioned preset shape marker is to establish a world coordinate system, so that after obtaining the first calibration parameters between the cameras, the first calibration parameters between the cameras and the world coordinate system are used to calibrate the second calibration parameters (external parameters) of each camera relative to the world coordinate system.

[0067] In some embodiments, the preset shape marker is set within the image acquisition range of each camera and is formed by the intersection of two preset length markers, wherein the intersection point of the two preset length markers is the origin of the world coordinate system. Optionally, the shape of the preset shape marker is L-shaped or T-shaped.

[0068] It can be understood that the preset shape markers in this embodiment are simple and easy to make. In order to facilitate camera acquisition and recognition of preset shape markers in images, the preset shape markers can be made of materials that are easy to perceive (such as white tape) and placed within the image acquisition range of each camera. The two preset lengths are used to provide corresponding size references for subsequent posture reconstruction.

[0069] In a specific embodiment, markers of a preset shape (two markers of preset length and at right angles) may be pasted on the ground to establish a world coordinate system and provide a corresponding dimension reference.

[0070] In one embodiment, step S32 calibrates each second camera based on multiple groups of first matching points in each image block group to obtain first calibration parameters of each second camera, further comprising the following steps:

[0071] Step 1: Use the first target image captured by the first camera as the first image, and use the first target images captured by each second camera as the second image.

[0072] Step 2: Find each group of second matching points between the first image and each second image from the multiple groups of first matching points.

[0073] Step 3: Estimation is performed based on each group of second matching points to obtain an essential matrix between the first image and each second image.

[0074] Step 4: Perform singular value decomposition on each essential matrix to obtain the first calibration parameters of each second camera.

[0075] In one embodiment, in order to further improve the accuracy of the calibration parameters, before the above steps 1 to 4, the RANSAC algorithm can be used to filter abnormal matching points of multiple groups of matching points between image block groups, so as to select high-quality matching point groups from multiple groups of matching points, and then more accurately calculate the intrinsic matrix and camera calibration parameters through steps 1 to 4. The details are as follows:

[0076] The above multiple groups of first matching points are used as multiple groups of candidate matching points. The multiple groups of candidate matching points are used to estimate the essential matrix between different images. The calculated essential matrix is ​​used for projection calculation, and the matching points are projected into three-dimensional space. Then, the points projected into three-dimensional space are reprojected back into the images of other cameras. The distance between the reprojected points and the candidate matching points is calculated. If the distance error is greater than the preset value, it means that the error of the original candidate matching points is relatively large, and the corresponding group of candidate matching points can be deleted. Then repeat the above steps until convergence or the number of iterations reaches the preset number, and finally select a high-precision matching point group as the first matching point of each group for camera calibration.

[0077] See also Figure 4 , Figure 4 is a flow chart of an embodiment of the posture reconstruction method provided by the present application. It should be noted that if there are substantially the same results, this embodiment does not use Figure 4 The process sequence shown is limited. Figure 4 As shown, this embodiment includes:

[0078] S41: Acquire a plurality of second target images, each of which is acquired by capturing the same target object by a plurality of cameras located at different positions.

[0079] S42: performing key point recognition on the target object in each second target image to obtain the pixel position of each key point in the corresponding second target image.

[0080] S43: Determine the target position of each key point in the world coordinate system based on the target calibration parameters of each camera and the pixel position of each key point in the corresponding second target image.

[0081] The target calibration parameters of each camera are conversion parameters between the camera coordinate system of each camera and the world coordinate system; wherein the target calibration parameters of each camera are determined by using any of the above-mentioned camera calibration methods.

[0082] In this embodiment, after obtaining the pixel position of each key point in the corresponding second target image and the target calibration parameters of each camera, the position of each key point in the world coordinate system can be reconstructed through triangulation reconstruction (multi-view geometry algorithm).

[0083] Among them, the pixel position of each key point in the corresponding second target image can be obtained by performing key point recognition on the target object in each second target image using the above-mentioned key point recognition model. Optionally, the pixel position of each key point in the corresponding second target image obtained by key point recognition can be directly used as the pixel position of each key point in the corresponding second target image. Preferably, after identifying the pixel position of each key point in the corresponding second target image, the confidence of each key point can be used for filtering, and the key points with confidence less than the confidence threshold are filtered out, and only the pixel positions of the key points with confidence greater than the confidence threshold in the corresponding second target image are retained, and step S43 is executed.

[0084] Among them, the determination of the world coordinate system can refer to the relevant description of the calibration method embodiment, and will not be elaborated here.

[0085] To facilitate understanding of the above-mentioned solution of the present application, the above-mentioned camera calibration method and posture reconstruction method provided by the present application are explained below using a human posture reconstruction scene as an example.

[0086] The camera calibration method includes the following steps:

[0087] Step 1: Obtain multi-view images (several first target images) of the same human body captured by a plurality of cameras located at different positions at the same time.

[0088] Step 2: Use the human body posture estimation model to process each image in the multi-view image separately to obtain the estimation result of the human body joint points in each image.

[0089] Among them, the human body joint estimation results include the type, coordinates and confidence of each joint point.

[0090] Furthermore, considering the sparsity of human pose estimation tasks and the influence of camera shooting distance, the same joint point in multi-view images may have errors and cannot be directly used as a matching point for camera calibration. Therefore, according to the confidence threshold, the joint points with confidence greater than the confidence threshold can be screened out from each joint point (for easy distinction, called key joint points).

[0091] Step 3: Take the same type of joint point (the same joint point or the same key joint point) in the multi-view image as the center point, and the pixel area within the preset range of the same type of joint point as the image block including the joint point, and combine the image blocks of the same type of joint point in the multi-view image to obtain the corresponding image block group.

[0092] Step 4: Perform pixel-by-pixel feature point matching on each image block in the same image block group to obtain a plurality of matching groups of first matching points.

[0093] Among them, the image blocks determined based on the position prior of the human joint points (the positions estimated by the human posture estimation model) greatly narrow the matching range, thereby greatly reducing the matching difficulty and error of the feature points. At the same time, the image block method also increases the position distribution of the matching points in the image space, laying the foundation for the subsequent precise calibration of the camera position.

[0094] Step 5: Select a camera as the main camera, and the corresponding captured image as the main image. Calculate the essential matrix with the matching points of other images in turn, and obtain the calibration parameters of other cameras relative to the main camera through singular value decomposition.

[0095] Step 6: Use the world coordinate system constructed by the preset shape marker to convert the calibration parameters of other cameras relative to the main camera into the calibration parameters of each camera relative to the world coordinate system.

[0096] The calibration parameters obtained in any one of step five and step six may be used as the final calibration parameters of each camera.

[0097] Furthermore, the posture reconstruction method is as follows:

[0098] According to the human joint point estimation results (coordinates) determined in the above step 2 and the camera calibration parameters obtained in step 5 or step 6, 3D joint point synthesis is performed through triangulation reconstruction (multi-view geometry algorithm) to reconstruct the three-dimensional posture of the human body.

[0099] See also Figure 5 , Figure 5It is a schematic diagram of the framework of an embodiment of a camera calibration device provided by the present application. In this embodiment, the camera calibration device 50 includes: a first acquisition module 51, a first determination module 52, a second determination module 53 and a calibration module 54. The first acquisition module 51 is used to acquire a plurality of first target images, each of which is acquired by a plurality of cameras located at different positions at the same time for the same target object; the first determination module 52 is used to determine a plurality of image block groups from the plurality of first target images; different image blocks in the image block groups belong to different first target images, and are image regions containing at least one identical key point in the target object; the second determination module 53 is used to determine a plurality of groups of first matching points between different image blocks in each image block group; the calibration module 54 is used to calibrate each camera based on the plurality of groups of first matching points in each image block group to obtain target calibration parameters of each camera.

[0100] In some embodiments, the first determination module 52 is used to determine multiple image block groups from a number of first target images, including: using a key point recognition model to identify each first target image to obtain a key point recognition result of the target object in each first target image; the key point recognition result includes the position of each key point in the target object and the category to which it belongs; based on the key point recognition results corresponding to each first target image, each key point of the same category is selected from a number of first target images, and the key points of the same category are combined into a key point group; for the first target image where each key point in each key point group is located, the image area within a preset range of the key point in the first target image is used as an image block containing the key point; for each key point group, the image blocks corresponding to each key point in the key point group are combined to obtain an image block group corresponding to the key point group.

[0101] In some embodiments, the key point recognition result also includes the confidence that each key point in the first target image corresponds to the category to which it belongs; the confidence of each key point in the selected key point group is greater than the confidence threshold.

[0102] In some embodiments, the second determination module 53 determines multiple groups of first matching points between different image blocks in each image block group, including: extracting feature points from each image block in each image block group; and determining multiple groups of first matching points between different image blocks in each image block group based on the feature point matching results between different image blocks in each image block group.

[0103] In some embodiments, the feature point matching results include the degree of matching between feature points in the image block group; based on the feature point matching results between different image blocks in each image block group, multiple groups of first matching points between different image blocks in each image block group are determined, including: for each image block group, obtaining feature points in the image block group whose matching degree is greater than a preset threshold to obtain multiple groups of first matching points of the image block group.

[0104] In some embodiments, the calibration module 54 calibrates each camera based on multiple groups of first matching points in each image block group to obtain target calibration parameters of each camera, including: selecting a camera from each camera as the first camera, and using the camera coordinate system of the first camera as the reference coordinate system; calibrating each second camera based on the multiple groups of first matching points in each image block group to obtain the first calibration parameters of each second camera; the second camera is a camera other than the first camera, and the first calibration parameter represents the first conversion parameter between the camera coordinate system of the corresponding second camera and the reference coordinate system; based on the reference coordinate system, each first calibration parameter, a world coordinate system established with a preset shape marker, and the pixel position of the preset shape marker in each first target image, determining the second calibration parameter of each camera, the second calibration parameter represents the second conversion parameter between the corresponding camera and the world coordinate system; using the second calibration parameter of each camera as the target calibration parameter of the corresponding camera.

[0105] In some embodiments, a preset shape marker is located within the image acquisition range of each camera and is formed by the intersection of two preset length markers, wherein the intersection point of the two preset length markers is the origin of the world coordinate system; and / or, the preset shape marker is L-shaped or T-shaped.

[0106] In some embodiments, each second camera is calibrated based on multiple groups of first matching points in each image block group to obtain first calibration parameters of each second camera, including: taking the first target image captured by the first camera as the first image, and taking the first target images captured by each second camera as the second image; finding each group of second matching points between the first image and each second image from the multiple groups of first matching points; estimating based on each group of second matching points to obtain an essential matrix between the first image and each second image; and performing singular value decomposition on each essential matrix to obtain the first calibration parameters of each second camera.

[0107] In some embodiments, the target object is a living body, and the key points in the target object are the joints of the living body; and / or, each camera is connected to a corresponding time coder, and each time coder is used to synchronize the video frames of videos captured by cameras at different positions to obtain the first target image captured by cameras at different positions at the same time.

[0108] See also Figure 6 , Figure 6It is a schematic diagram of the framework of an embodiment of a posture reconstruction device provided by the present application. In this embodiment, the posture reconstruction device 60 includes: a second acquisition module 61, an identification module 62 and a position determination module 63. The second acquisition module 61 is used to acquire a plurality of second target images, each of which is acquired by a plurality of cameras located at different positions for the same target object; the identification module 62 is used to identify key points of the target object in each second target image, and obtain the pixel position of each key point in the corresponding second target image; the position determination module 63 is used to determine the target position of each key point in the world coordinate system based on the target calibration parameters of each camera and the pixel position of each key point in the corresponding second target image; the target calibration parameters of each camera are the conversion parameters between the camera coordinate system of each camera and the world coordinate system; wherein, the target calibration parameters of each camera are determined using any of the above-mentioned camera calibration methods.

[0109] See also Figure 7 , Figure 7 Schematic diagram of the framework of an electronic device provided by the present application. In this embodiment, the electronic device 70 includes a memory 71 and a processor 72 coupled to each other.

[0110] The memory 71 stores program instructions, and the processor 72 is used to execute the program instructions stored in the memory 71 to implement the steps of any of the above method implementations. In a specific implementation scenario, the electronic device 70 may include, but is not limited to: a microcomputer, a server, and in addition, the electronic device 70 may also include a mobile device such as a laptop computer and a tablet computer, which is not limited here.

[0111] Specifically, the processor 72 is used to control itself and the memory 71 to implement the steps of any of the above-mentioned embodiments. The processor 72 can also be called a CPU (Central Processing Unit). The processor 72 may be an integrated circuit chip with signal processing capabilities. The processor 72 can also be a general-purpose processor, a digital signal processor (Digital Signal Processor, DSP), an application-specific integrated circuit (Application Specific Integrated Circuit, ASIC), a field-programmable gate array (Field-Programmable Gate Array, FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components. The general-purpose processor can be a microprocessor or the processor can also be any conventional processor, etc. In addition, the processor 72 can be implemented by an integrated circuit chip.

[0112] See also Figure 8 , Figure 8It is a schematic diagram of the framework of the computer-readable storage medium provided by the present application. The computer-readable storage medium 80 of the embodiment of the present application stores a program instruction 81, and when the program instruction 81 is executed, the method provided by any embodiment of the above method and any non-conflicting combination is implemented. Among them, the program instruction 81 can form a program file and be stored in the above-mentioned computer-readable storage medium 80 in the form of a software product, so that a computer device (which can be a personal computer, a server, or a network device, etc.) executes all or part of the steps of each implementation method of the present application. The aforementioned computer-readable storage medium 80 includes: various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM, Read-Only Memory), a random access memory (RAM, Random Access Memory), a disk or an optical disk, or a terminal device such as a computer, a server, a mobile phone, and a tablet.

[0113] The above scheme determines the image block group containing the same key points on the basis of the key points, and then performs camera calibration based on multiple groups of first matching points between different image blocks in each image block group. It can be seen that the first matching points of the present application include points that are located in the image block and are not key points in addition to the key points. In other words, the present application expands the distribution range of the matching points in the first target image. Compared with the method of only using a small number of matched key points for camera calibration, the method of introducing more matching points in the image block of the present application can reduce the impact of false detection and false matching of key points, which is conducive to accurately calibrating the position of the camera.

[0114] In some embodiments, the functions or modules included in the device provided by the embodiments of the present disclosure can be used to execute the method described in the above method embodiments. The specific implementation can refer to the description of the above method embodiments, and for the sake of brevity, it will not be repeated here.

[0115] The above description of various embodiments tends to emphasize the differences between the various embodiments. The same or similar aspects can be referenced to each other, and for the sake of brevity, they will not be repeated herein.

[0116] In the several embodiments provided in the present application, it should be understood that the disclosed methods and devices can be implemented in other ways. For example, the device implementation described above is only schematic. For example, the division of modules or units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, and the indirect coupling or communication connection of devices or units can be electrical, mechanical or other forms.

[0117] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the present embodiment.

[0118] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0119] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application is essentially or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions to enable a computer device (which can be a personal computer, server, or network device, etc.) or a processor (processor) to perform all or part of the steps of each implementation method of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk and other media that can store program code.

[0120] The above description is only an implementation method of the present application, and does not limit the patent scope of the present application. Any equivalent structure or equivalent process transformation made using the contents of the present application specification and drawings, or directly or indirectly used in other related technical fields, are also included in the patent protection scope of the present application.

[0121] In addition, if the technical solution of this application involves personal information, the product using the technical solution of this application has clearly informed the personal information processing rules and obtained the individual's voluntary consent before processing the personal information. If the technical solution of this application involves sensitive personal information, the product using the technical solution of this application has obtained the individual's separate consent before processing the sensitive personal information, and at the same time meets the "explicit consent" requirement. For example, at personal information collection devices such as cameras, clear and prominent signs are set to inform that the personal information collection scope has been entered and personal information will be collected. If the individual voluntarily enters the collection scope, it is deemed that he or she agrees to collect his or her personal information; or on the device for processing personal information, when the personal information processing rules are notified by obvious signs / information, the individual's authorization is obtained through pop-up information or by asking the individual to upload his or her personal information; among them, the personal information processing rules may include information such as the personal information processor, the purpose of personal information processing, the processing method, and the type of personal information processed.

Claims

1. A camera calibration method, characterized in that: The method comprises: Acquire a plurality of first target images, each of which is acquired by a plurality of cameras located at different positions and capturing the same target object at the same time; Determine a plurality of image block groups from the plurality of first target images; different image blocks in the image block groups belong to different first target images and are image regions containing at least one same key point in the target object; Determining a plurality of groups of first matching points between different image blocks in each of the image block groups; Each of the cameras is calibrated based on the multiple groups of first matching points in each of the image block groups to obtain target calibration parameters of each of the cameras.

2. The method according to claim 1, characterized in that: The step of determining a plurality of image block groups from the plurality of the first target images comprises: Using a key point recognition model to recognize each of the first target images, and obtaining a key point recognition result of the target object in each of the first target images; the key point recognition result includes a position and a category of each key point in the target object; Based on the key point recognition results corresponding to each of the first target images, key points of the same category are selected from a plurality of first target images, and the key points of the same category are combined into a key point group; For the first target image where each key point in each key point group is located, an image area within a preset range of the key point in the first target image is used as an image block containing the key point; For each of the key point groups, the image blocks corresponding to each key point in the key point group are combined to obtain the image block group corresponding to the key point group.

3. The method according to claim 2, characterized in that The key point recognition result also includes the confidence level of each key point in the first target image corresponding to the category to which it belongs; The confidence of each key point in the key point group is selected to be greater than the confidence threshold.

4. The method according to claim 1, characterized in that The determining of a plurality of groups of first matching points between different image blocks in each of the image block groups comprises: Extracting feature points from each image block of each image block group; Based on the feature point matching results between different image blocks in each of the image block groups, a plurality of groups of first matching points between different image blocks in each of the image block groups are determined.

5. The method according to claim 4, characterized in that The feature point matching result includes the matching degree between the feature points in the image block group; The determining, based on the feature point matching results between different image blocks in each of the image block groups, a plurality of groups of first matching points between different image blocks in each of the image block groups comprises: For each of the image block groups, the feature points in the image block group whose matching degree is greater than a preset threshold are acquired to obtain the multiple groups of first matching points of the image block group.

6. The method according to claim 1, characterized in that The step of calibrating each camera based on the multiple groups of first matching points in each of the image block groups to obtain target calibration parameters of each camera includes: Selecting a camera from each of the cameras as a first camera, and using a camera coordinate system of the first camera as a reference coordinate system; Calibrate each second camera based on multiple groups of first matching points in each of the image block groups to obtain first calibration parameters of each second camera; the second camera is the camera other than the first camera, and the first calibration parameter represents a first conversion parameter between a camera coordinate system corresponding to the second camera and the reference coordinate system; Determine a second calibration parameter of each camera based on the reference coordinate system, each of the first calibration parameters, a world coordinate system established with a preset shape marker, and a pixel position of the preset shape marker in each of the first target images, wherein the second calibration parameter represents a second conversion parameter between the corresponding camera and the world coordinate system; The second calibration parameters of each camera are used as target calibration parameters of the corresponding camera.

7. The method according to claim 6, characterized in that The preset shape marker is set within the image acquisition range of each camera and is formed by the intersection of two preset length markers, wherein the intersection point of the two preset length markers is the origin of the world coordinate system; And / or, the preset shape marker is in an L-shape or a T-shape.

8. The method according to claim 6, characterized in that The step of calibrating each second camera based on the multiple groups of first matching points in each of the image block groups to obtain a first calibration parameter of each second camera includes: Using the first target image captured by the first camera as the first image, and using the first target images captured by each of the second cameras as the second images respectively; Finding groups of second matching points between the first image and each second image from the plurality of groups of first matching points; Estimating based on each group of the second matching points to obtain an essential matrix between the first image and each second image; Performing singular value decomposition on each of the essential matrices to obtain first calibration parameters of each of the second cameras.

9. The method according to claim 1, characterized in that: The target object is a living body, and the key points in the target object are joint points of the living body; And / or, each of the cameras is connected to a corresponding time coder, and each of the time coders is used to synchronize video frames of videos captured by cameras at different positions to obtain the first target image captured by cameras at different positions at the same time.

10. A posture reconstruction method, characterized in that: The method comprises: Acquire a plurality of second target images, each of which is acquired by capturing the same target object by a plurality of cameras located at different positions; Performing key point recognition on the target object in each of the second target images to obtain a pixel position of each of the key points in the corresponding second target image; Based on the target calibration parameters of each camera and the pixel position of each key point in the corresponding second target image, the target position of each key point in the world coordinate system is determined; the target calibration parameters of each camera are the conversion parameters between the camera coordinate system of each camera and the world coordinate system; wherein the target calibration parameters of each camera are determined using any calibration method described in claims 1-9.

11. A camera calibration device, characterized in that: The device comprises: A first acquisition module is used to acquire a plurality of first target images, each of which is acquired by a plurality of cameras located at different positions and capturing the same target object at the same time; A first determination module is used to determine a plurality of image block groups from a plurality of the first target images; different image blocks in the image block groups belong to different first target images and are image regions containing at least one same key point in the target object; A second determination module, used to determine a plurality of groups of first matching points between different image blocks in each of the image block groups; The calibration module is used to calibrate each of the cameras based on the multiple groups of first matching points in each of the image block groups to obtain target calibration parameters of each of the cameras.

12. A posture reconstruction device, characterized in that: The device comprises: A second acquisition module is used to acquire a plurality of second target images, each of which is acquired by capturing the same target object by a plurality of cameras located at different positions; A recognition module, used for performing key point recognition on the target object in each of the second target images to obtain a pixel position of each of the key points in the corresponding second target image; A position determination module, used to determine the target position of each key point in the world coordinate system based on the target calibration parameters of each camera and the pixel position of each key point in the corresponding second target image; the target calibration parameters of each camera are the conversion parameters between the camera coordinate system of each camera and the world coordinate system; wherein the target calibration parameters of each camera are determined using any calibration method described in claims 1-9.

13. An electronic device, characterized in that: comprising a memory and a processor coupled to each other, The memory stores program instructions; The processor is used to execute the program instructions stored in the memory to implement the method according to any one of claims 1 to 10.

14. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores program instructions that can be run by a processor, and the program instructions can be executed by the processor to implement the method according to any one of claims 1 to 10.

Images