Multi-camera selection method and apparatus for human motion pose analysis

Abstract

The application provides a multi-camera selection method and device for human motion posture analysis. The method comprises the following steps: calibrating the multi-camera, obtaining the camera internal parameter and external parameter; judging whether there is a human eye, marking and recording the position if there is a human eye; calculating the human eye spatial position with the highest confidence through any two cameras with marked human eye positions; calculating the human face front direction, using the direction to divide the human posture space into two parts; calculating the spatial position relationship between all cameras and the human face front; performing one-time three-dimensional reconstruction of all body joint nodes in all viewpoints; calculating the limb spatial volume; calculating whether any joint node is blocked by the body part; obtaining the camera list without any blockage in the camera sorting; and deleting the cameras with spatial positions in the rear part through the spatial position relationship between the camera and the human face front. The application improves the accuracy of camera screening.

Description

Technical Field

[0001] This invention belongs to the field of motion analysis technology, specifically relating to a method and apparatus for selecting multi-view cameras for human motion posture analysis. Background Technology

[0002] Current technologies for analyzing human motion posture based on multi-camera video mainly fall into two categories: one uses marker-assisted algorithms to analyze human joints, and the other estimates human joints based on prior knowledge without markers. The first method, relying on markers, is unusable in many scenarios due to the need for pre-marking the subject, resulting in poor versatility. The second method, relying solely on prior knowledge to estimate human joints, suffers from low accuracy. To address this, this invention employs a marker-free approach, enhancing the versatility of human motion analysis. Simultaneously, by utilizing multi-camera selection technology, it improves the accuracy of human joint recognition.

[0003] Existing human motion capture and pose analysis systems employ algorithms combining background subtraction and inter-frame subtraction to extract human targets from videos, improving the robustness of separating human targets from background noise. Current multi-camera optimization methods, such as the optimal viewpoint selection method and 3D human skeleton detection method for multi-camera systems (Chinese patent application CN201910524334), can solve the multi-camera optimization problem in some simple scenarios, but due to a lack of constraints, they suffer from incorrect camera selection.

[0004] In summary, the drawbacks of existing technologies are that the marker-based method requires pre-marking the object being collected, which makes it unusable in many scenarios and results in poor versatility; while current methods that do not use markers rely solely on prior knowledge to estimate human joints, which is prone to misjudging joints and has a low accuracy rate. Summary of the Invention

[0005] In view of this, and to address the aforementioned shortcomings, this invention provides a multi-camera selection method that effectively improves the accuracy of human joint point recognition without requiring the pasting of marker points. The solution of this invention adopts a marker-free approach, enhancing the versatility of human motion analysis. Simultaneously, through multi-camera selection technology, the spatial positional relationship between the human body and the cameras is analyzed using the human eye, and this is used as a constraint condition for multi-camera selection for each joint, thereby improving the accuracy of human joint point recognition. This invention's multi-view vision-based human joint point recognition method requires 3D reconstruction based on monocular human joint point recognition to generate 3D human joint point data. Since human joint points, especially the limbs, are distributed on both sides of the torso, their spatial positions change during movement. There are instances of occlusion of joint points by the torso and limbs. In the multi-view human joint point recognition process, no single perspective can completely observe all the joint points of the human body. Image-based human joint point recognition algorithms struggle to obtain accurate joint point estimation results when occluded. Because the recognition probabilities of images from different viewpoints cannot be effectively unified in multi-view mode, it is not suitable to directly use the recognition probability to remove erroneous points or obtain valid points.

[0006] In a first aspect, the present invention provides a multi-camera selection method for human motion posture analysis, comprising:

[0007] Step 1: Calibrate the multi-view camera and obtain its intrinsic and extrinsic parameters;

[0008] Step 2: Process the images captured by each camera to determine if a human eye is present. If so, mark it and record its location.

[0009] Step 3: Using any two cameras that have been calibrated to determine the position of the human eye, calculate the spatial position of the human eye with the highest confidence, calculate the direction of the face facing forward, and use this direction to divide the human posture space into two parts, P-Front and P-Back.

[0010] Step 4: Calculate the spatial relationship between all cameras and the front of the face;

[0011] Step 5: Perform a unified 3D reconstruction of all relevant nodes of the body from all viewpoints;

[0012] Step 6: Obtain the basic spatial positions of the torso, head and neck, and limbs, and use basic human body parameters to calculate the spatial volume of the limbs;

[0013] Step 7: Based on the camera's intrinsic and extrinsic parameters, calculate whether any joint point in each camera viewpoint is obscured by the body.

[0014] Step 8: For each key point, sort the cameras according to the occlusion status and obtain a list of cameras that are completely unobstructed in the camera sort; then, constrain the spatial position relationship between the camera and the front of the face, and delete the cameras whose spatial position is in the P-Back position.

[0015] Preferably, the COCO-WholeBody method is used to analyze and determine whether a human eye is present.

[0016] Preferably, step 4 specifically includes identifying whether the camera belongs to P-Front or P-Back, as a supplementary condition for subsequent keypoint camera screening.

[0017] Preferably, in step 6, a cylinder is used to approximate the limb space.

[0018] Preferably, step 8 further includes:

[0019] If the number of cameras after screening is less than 2, add cameras with smaller obstruction angles.

[0020] Preferably, step 8 further includes:

[0021] Step 9: Based on the camera selection for each joint, reconstruct all 3D joints again.

[0022] Furthermore, step 9 also includes:

[0023] Step 10: Iterate through steps 2 through 6 multiple times to reduce the initial reconstruction error.

[0024] In a second aspect, the present invention provides a multi-camera selection device for human motion posture analysis, comprising:

[0025] The calibration module is used to calibrate multi-view cameras and obtain the camera's intrinsic and extrinsic parameters.

[0026] The human eye detection module processes the images captured by each camera, determines whether a human eye is present, and if so, marks and records its location.

[0027] The spatial segmentation module is used to calculate the spatial position of the human eye with the highest confidence using any two cameras whose positions have been calibrated, calculate the direction of the human face, and use this direction to segment the human posture space into two parts, P-Front and P-Back.

[0028] The spatial position calculation module is used to calculate the spatial positional relationship between all cameras and the front of the face;

[0029] The 3D reconstruction module is used to perform a unified 3D reconstruction of all relevant nodes of the body from all viewpoints.

[0030] The limb space calculation module is used to obtain the basic spatial positions of the trunk, head and neck, and limbs, and to calculate the limb space volume using basic human body parameters;

[0031] The occlusion calculation module is used to calculate whether any joint point in each camera viewpoint is occluded by body parts based on the camera's intrinsic and extrinsic parameters.

[0032] The camera selection module is used to sort the cameras according to the occlusion status for each key point, obtain a list of cameras that are completely unobstructed in the camera sorting; and then, by constraining the spatial position relationship between the camera and the front of the face, delete the cameras whose spatial position is in the P-Back position.

[0033] Thirdly, the present invention provides an electronic device including a processor and a memory, wherein the memory stores a computer program that, when executed by the processor, causes the processor to perform the steps of any of the methods described in the first aspect.

[0034] Fourthly, the present invention provides a computer-readable storage medium comprising a computer program that, when executed on an electronic device, causes the electronic device to perform the steps of any of the methods described in the first aspect.

[0035] By adopting the above technical solutions, the beneficial technical effect achieved by this invention is that, through the above camera selection method, highly accurate three-dimensional information of human motion posture joints can be obtained. This invention adds the ability to identify low-precision human eye positions in images, analyze the low-precision spatial relationship between the camera capturing the human eye and the front of the person, and then obtain the low-precision spatial relationship between all cameras and the front of the face. Using the low-precision spatial relationship between each camera and the front of the face as a supplementary constraint condition, the accuracy of camera selection is improved. Attached Figure Description

[0036] Figure 1 This is a schematic diagram of the human body's spatial position according to an embodiment of the present invention;

[0037] Figure 2 This is a schematic diagram of the positions of multiple cameras according to an embodiment of the present invention. Detailed Implementation

[0038] The present invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative of the invention and are not intended to limit the invention.

[0039] This embodiment provides a multi-camera selection method that effectively improves the accuracy of human joint point recognition without requiring the pasting of marker points. Multi-view vision-based human joint point recognition methods require 3D reconstruction based on monocular human joint point recognition to generate 3D human joint point data. Since human joint points, especially the limbs, are distributed on both sides of the torso, their spatial positions change during movement. Occlusion of joint points by the torso and limbs is possible. In multi-view human joint point recognition, no single viewpoint can completely observe all the human joint points. Image-based human joint point recognition algorithms struggle to obtain accurate joint point estimation results when occluded. Furthermore, because the recognition probabilities between images from different viewpoints cannot be effectively unified in multi-view mode, it is not suitable to directly use recognition probabilities to remove erroneous points or obtain valid points.

[0040] The detailed steps of the multi-camera selection method provided in this embodiment are as follows:

[0041] Step 1: Calibrate the multi-camera system and obtain its intrinsic and extrinsic parameters.

[0042] Step 2: Process the images captured by each camera, analyze them using the COCO-WholeBody method to determine if a human eye is present, and if so, mark and record its location.

[0043] Step 3: Using any two cameras whose eye positions have been calibrated, calculate the spatial position of the eye with the highest confidence level. Calculate the direction the face is facing. Use this direction to divide the human pose space into two parts: P-Front and P-Back. The human spatial position is as follows: Figure 1 As shown.

[0044] Step 4: Calculate the spatial positional relationship between all cameras and the front of the face, that is, identify whether the camera belongs to P-Front or P-Back, as a supplementary condition for subsequent key point screening of cameras.

[0045] Step 5: Perform a unified 3D reconstruction of all relevant nodes of the body from all viewpoints.

[0046] Step 6: Obtain the basic spatial positions of the torso, head and neck, and limbs. Calculate the spatial volume of the limbs using basic human body parameters and approximate them with cylinders.

[0047] Step 7: Based on the camera's intrinsic and extrinsic parameters, calculate whether any joint point in each camera viewpoint is obscured by the body.

[0048] Step 8: For each keypoint, sort the cameras according to their occlusion level, and obtain a list of cameras that are completely unoccluded. Then, constrain the spatial position of the camera relative to the front of the face, and delete cameras whose spatial position is at the P-Back. If the number of cameras after filtering is less than two, add cameras with smaller occlusion angles. Multiple camera positions are as follows... Figure 2 As shown.

[0049] Step 9: Based on the camera selection for each joint, reconstruct all 3D joints again.

[0050] Step 10, the process from step 2 to step 6, can be iterated multiple times to reduce the initial reconstruction error.

[0051] The above camera selection method can obtain highly accurate 3D information of human motion posture joints. This embodiment provides a high-accuracy multi-camera selection method, and adds the ability to identify low-precision human eye positions in the image, analyze the low-precision spatial position relationship between the camera capturing the human eye and the front of the human face, and then obtain the low-precision spatial position relationship between all cameras and the front of the human face. The low-precision spatial position relationship between each camera and the front of the human face is used as a supplementary constraint, thereby improving the accuracy of camera selection.

[0052] This embodiment also provides a multi-view camera selection device for human motion posture analysis, including:

[0053] The calibration module is used to calibrate multi-view cameras and obtain the camera's intrinsic and extrinsic parameters.

[0054] The human eye detection module processes the images captured by each camera, determines whether a human eye is present, and if so, marks and records its location.

[0055] The spatial segmentation module is used to calculate the spatial position of the human eye with the highest confidence using any two cameras whose positions have been calibrated, calculate the direction of the human face, and use this direction to segment the human posture space into two parts, P-Front and P-Back.

[0056] The spatial position calculation module is used to calculate the spatial positional relationship between all cameras and the front of the face;

[0057] The 3D reconstruction module is used to perform a unified 3D reconstruction of all relevant nodes of the body from all viewpoints.

[0058] The limb space calculation module is used to obtain the basic spatial positions of the trunk, head and neck, and limbs, and to calculate the limb space volume using basic human body parameters;

[0059] The occlusion calculation module is used to calculate whether any joint point in each camera viewpoint is occluded by body parts based on the camera's intrinsic and extrinsic parameters.

[0060] The camera selection module is used to sort the cameras according to the occlusion status for each key point, obtain a list of cameras that are completely unobstructed in the camera sorting; and then, by constraining the spatial position relationship between the camera and the front of the face, delete the cameras whose spatial position is in the P-Back position.

[0061] Specific limitations regarding the multi-camera selection device for human motion posture analysis can be found in the limitations of the multi-camera selection method for human motion posture analysis described above, and will not be repeated here. Each module in the aforementioned multi-camera selection device for human motion posture analysis can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in or independent of the processor in a computer device in hardware form, or stored in the memory of a computer device in software form, so that the processor can call and execute the corresponding operations of each module.

[0062] Specifically, according to this embodiment, the present invention discloses an electronic device, which includes one or more processors and a memory, wherein the memory is used to store one or more programs; when the one or more programs are executed by the processor, the processor enables the processor to implement the multi-camera selection method for human motion posture analysis provided in the embodiments of the present invention.

[0063] Specifically, according to this embodiment, the multi-camera selection method for human motion posture analysis described in any of the above embodiments can be implemented as a computer software program. For example, this embodiment includes a computer program product comprising a computer program tangibly contained on a machine-readable medium, the computer program containing program code for a multi-camera selection method for human motion posture analysis. In such an embodiment, the computer program can be downloaded and installed from a network via a communication component, and / or installed from a removable medium.

[0064] The one or more programs stored in the read-only memory (ROM) or the random access memory (RAM) perform various appropriate actions and processes. The RAM includes software programs for the server to complete corresponding business operations, as well as various programs and data required for vehicle driving operations. The server, its controlled hardware devices, the ROM, and the RAM are interconnected via a bus, and various input / output interfaces are also connected to the bus.

[0065] The following components are connected to the input / output interface: input sections including keyboards, mice, etc.; output sections including cathode ray tube (CRT) displays, liquid crystal displays (LCDs), and speakers; and communication sections including network interface cards such as LAN cards and modems. The communication section performs communication processing via a network such as the Internet. Drives are also connected to the input / output interface as needed. Removable media, such as disks, optical discs, magneto-optical discs, semiconductor memory, etc., are installed on the drive as needed so that computer programs read from them can be installed into memory as required.

[0066] Specifically, according to this embodiment, the multi-camera selection method for human motion posture analysis described in any of the above embodiments can be implemented as a computer software program. For example, this embodiment includes a computer program product comprising a computer program tangibly contained on a machine-readable medium, the computer program containing program code for a multi-camera selection method for human motion posture analysis. In such an embodiment, the computer program can be downloaded and installed from a network via a communication component, and / or installed from a removable medium.

[0067] The units or modules described in the embodiments of this application can be implemented in software or hardware. The described units or modules can also be located in a processor. The names of these units or modules do not necessarily constitute a limitation on the unit or module itself.

[0068] The above description is merely an embodiment of this application and is not intended to limit the scope of this application. Various modifications and variations can be made to this application by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the scope of the claims of this application.

Claims

1. A method for selecting multi-view cameras for human motion posture analysis, characterized in that, include: Step 1: Calibrate the multi-view camera and obtain its intrinsic and extrinsic parameters; Step 2: Process the images captured by each camera to determine if a human eye is present. If so, mark it and record its location. Step 3: Using any two cameras that have been calibrated to indicate the position of the human eye, calculate the spatial position of the human eye with the highest confidence, calculate the direction of the face facing forward, and use this direction to divide the human posture space into two parts, front and back, with the front part denoted as P-Front and the back part denoted as P-Back. Step 4: Calculate the spatial relationship between all cameras and the front of the face; Step 5: Perform a unified 3D reconstruction of all relevant nodes of the body from all viewpoints; Step 6: Obtain the spatial positions of the torso, head and neck, and limbs, and use human body parameters to calculate the spatial volume of the limbs; Step 7: Based on the camera's intrinsic and extrinsic parameters, calculate whether any joint point in each camera viewpoint is obscured by the body. Step 8: For each key point, sort the cameras according to the occlusion status and obtain a list of cameras that are completely unoccluded in the camera sort; then, constrain the spatial position relationship between the camera and the front of the face and delete the camera whose spatial position is in P-Back. The COCO-WholeBody method was used to analyze and determine whether a human eye was present. Step 4 specifically includes identifying whether the camera belongs to P-Front or P-Back, as a supplementary condition for subsequent keypoint camera screening; In step 6, a cylinder is used to approximate the limb space; Step 8 is followed by: Step 9, based on the camera selection for each joint, reconstruct all 3D joints again; Step 9 is followed by: Step 10, and multiple iterations of Steps 2 to 6 to reduce the initial reconstruction error.

2. A multi-camera selection device for human motion posture analysis, characterized in that, The method for selecting a multi-view camera for human motion posture analysis as described in claim 1 includes: The calibration module is used to calibrate multi-view cameras and obtain the camera's intrinsic and extrinsic parameters. The human eye detection module processes images captured by each camera and determines whether a human eye is present. If present, mark and record the location; The spatial segmentation module is used to calculate the spatial position of the human eye with the highest confidence using any two cameras whose positions have been calibrated, calculate the direction of the human face facing forward, and use this direction to divide the human posture space into two parts, front and back, with the front part denoted as P-Front and the back part denoted as P-Back. The spatial position calculation module is used to calculate the spatial positional relationship between all cameras and the front of the face; The 3D reconstruction module is used to perform a unified 3D reconstruction of all relevant nodes of the body from all viewpoints. The limb space calculation module is used to obtain the spatial position of the torso, head and neck, and limbs, and to calculate the limb space volume using human body parameters; The occlusion calculation module is used to calculate whether any joint point in each camera viewpoint is occluded by body parts based on the camera's intrinsic and extrinsic parameters. The camera selection module is used to sort the cameras according to the occlusion status for each key point, obtain a list of cameras that are completely unobstructed in the camera sorting; and then, by constraining the spatial position relationship between the camera and the front of the face, delete the cameras whose spatial position is in the P-Back position.

3. An electronic device, characterized in that, It includes a processor and a memory, wherein the memory stores a computer program that, when executed by the processor, causes the processor to perform the steps of the method of claim 1.

4. A computer-readable storage medium, characterized in that, It includes a computer program that, when run on an electronic device, causes the electronic device to perform the steps of the method of claim 1.

Images