Motion capture method and device based on multi-view camera and storage medium

Abstract

The invention relates to a motion capture method and device based on a multi-view camera and a storage medium. Comprising the following steps: simultaneously imaging an observation object by using a multi-view camera to obtain a plurality of sample photos, calibrating the multi-view camera, and calculating a homography matrix of each camera in the multi-view camera; selecting articulation points needing to be captured from the plurality of sample photos for labeling, performing training by applying an action recognition model, and generating two-dimensional coordinates of the articulation points on each of the plurality of sample photos and corresponding confidence coefficients of the two-dimensional coordinates; screening all the two-dimensional coordinates according to a preset threshold value of confidence, and combining every two screened two-dimensional coordinates to obtain a plurality of two-dimensional coordinate pairs; performing three-dimensional reconstruction on the plurality of two-dimensional coordinate pairs based on a homography matrix, and taking a calculation result as a three-dimensional coordinate set of the joint point; and determining a final coordinate of the joint point from the three-dimensional coordinate set according to a preset rule.

Description

Technical Field

[0001] The present disclosure relates to a method, device and storage medium for capturing motion of an observed object, and in particular, to a method, device and storage medium for capturing motion of a non-human primate using a multi-camera. Background Art

[0002] Generally speaking, 3D motion capture is a technology that records and processes the real-time movements of people or other objects. Motion capture is often used in the film and television industries. By installing sensors on actors and recording position information, calculating the action skeleton based on the individual actors, and then using computer technology to generate virtual characters, we can get the changing trends of the actors' body movements or facial expressions. The key to this type of motion capture technology lies in the many sensors fixed on the limbs. With the development of technology, there are also many 3D motion capture technologies based on computer vision, including marked (obvious color marks are covered on the surface of the limbs' points of interest) and unmarked (no color marks on the limbs' surface) methods.

[0003] In the prior art, there is a markerless three-dimensional motion capture method implemented by binocular cameras and convolutional neural networks. However, its solution for capturing two-dimensional joints only supports 18 joints of the human body and cannot be applied to any animal or any joint. At the same time, the binocular camera used in this solution has very high requirements on the accuracy of the joints captured by the two-dimensional convolutional neural network and the accuracy of the camera calibration, which may affect the final calculation results.

[0004] In particular, in the field of brain science research, both in brain science research aimed at exploring the secrets of the brain and conquering brain diseases and in brain-like research aimed at establishing and developing artificial intelligence technology, it includes studying the correlation between brain activity and limb movements, and aligning the information obtained related to brain activity with the motion capture results. Therefore, motion capture methods and devices specific to the field of brain science are also needed. Summary of the invention

[0005] The present application proposes a method, device and storage medium for motion capture based on multi-camera.

[0006] According to a first aspect of an embodiment of the present disclosure, a motion capture method based on a multi-camera is provided, comprising: using a multi-camera to simultaneously image an observed object to obtain multiple sample photos, wherein the multi-camera is calibrated and the homography matrix of each camera in the multi-camera is calculated; selecting joint points to be captured from the multiple sample photos for annotation, and applying a motion recognition model for training to generate two-dimensional coordinates of the joint points on each of the multiple sample photos and their corresponding confidence levels; screening all two-dimensional coordinates according to a preset threshold of confidence levels, and combining the screened two-dimensional coordinates in pairs to obtain multiple two-dimensional coordinate pairs; performing three-dimensional reconstruction on the multiple two-dimensional coordinate pairs based on the homography matrix, and using the calculation results as a three-dimensional coordinate set of the joint points; and determining the final coordinates of the joint points from the three-dimensional coordinate set according to preset rules.

[0007] According to a second aspect of an embodiment of the present disclosure, a motion capture device based on a multi-camera is provided, comprising: a memory on which instructions are stored; and a processor, configured to execute the instructions stored in the memory to perform the following processing: using a multi-camera to simultaneously image an observed object to obtain a plurality of sample photos, wherein the multi-camera is calibrated and the homography matrix of each camera in the multi-camera is calculated; selecting joint points to be captured from the plurality of sample photos for annotation, and applying a motion recognition model for training to generate two-dimensional coordinates of the joint points on each of the plurality of sample photos and their corresponding confidences; screening all two-dimensional coordinates according to a preset threshold of confidence, and combining the screened two-dimensional coordinates in pairs to obtain a plurality of two-dimensional coordinate pairs; performing three-dimensional reconstruction on the plurality of two-dimensional coordinate pairs based on the homography matrix, and using the calculation results as a three-dimensional coordinate set of the joint points; and determining the final coordinates of the joint points from the three-dimensional coordinate set according to preset rules.

[0008] According to a third aspect of an embodiment of the present disclosure, there is provided a computer-readable storage medium having program instructions stored thereon, which, when executed, enables a computer to implement the motion capture method according to the first aspect of an embodiment of the present disclosure.

[0009] According to a fourth aspect of an embodiment of the present disclosure, a computer program product is provided, comprising computer program instructions, which, when executed by a processor, implement the motion capture method according to the first aspect of an embodiment of the present disclosure.

[0010] The advantage of the embodiments of the present disclosure is that in scenarios where it is not suitable to use marked three-dimensional motion capture methods (i.e., requiring humans or animals to wear sensors or perform color markings on joints), especially in the field of brain science research where sensors or color markings may affect the experimental results, the technical solution of the present disclosure can achieve markerless motion capture.

[0011] Another advantage of the embodiments of the present disclosure is that it solves the occlusion problem in the motion capture scene, such as occlusion from objects other than the target, or self-occlusion of the target, etc. The technical solution of the present disclosure helps to reduce the impact of occlusion in motion capture by using cameras with any number of meshes to surround the target.

[0012] Another advantage of the embodiments of the present disclosure is that, in view of the fact that the limb movements of humans or animals are usually at a high speed, the high-frame-rate behavioral data and the higher-frequency EEG signal data can be time-aligned while maintaining a high frame rate for motion capture, which is very beneficial for analytical research in the field of brain science research.

[0013] It should be appreciated that the above advantages do not need to be all realized in one or some specific embodiments, but may be partially dispersed in different embodiments according to the present disclosure. The embodiments according to the present disclosure may have one or some of the above advantages, or may have other advantages alternatively or additionally.

[0014] Other features and advantages of the present invention will become more apparent from the following detailed description of exemplary embodiments of the present invention with reference to the accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] Figure 1 is a schematic diagram showing a motion capture device based on a multi-camera and a processing method thereof according to an embodiment of the present disclosure.

[0016] Figure 2 An exemplary configuration block diagram of a motion capture device based on multi-cameras according to an embodiment of the present disclosure is shown.

[0017] Figure 3 An exemplary flowchart of a motion capture method based on multi-cameras according to an embodiment of the present disclosure is shown.

[0018] Figure 4 An exemplary configuration diagram of a multi-camera based motion capture system according to another embodiment of the present disclosure is shown.

[0019] Figure 5A and Figure 5B An exemplary perspective view of a multi-eye camera for a motion capture device according to an embodiment of the present disclosure is shown.

[0020] Figure 6 An exemplary configuration of a computing device that can implement an embodiment according to the present disclosure is shown. DETAILED DESCRIPTION

[0021] Various exemplary embodiments of the present disclosure will be described in detail below with reference to the accompanying drawings. It should be noted that unless otherwise specifically stated, the relative arrangement of components and steps, numerical expressions and numerical values ​​set forth in these embodiments do not limit the scope of the present disclosure.

[0022] The following description of at least one exemplary embodiment is in fact merely illustrative and is in no way intended to limit the present disclosure and its application or use. That is, the structures and methods herein are shown in an exemplary manner to illustrate different embodiments of the structures and methods in the present disclosure. However, those skilled in the art will appreciate that they merely illustrate exemplary ways of the present disclosure that can be implemented, rather than exhaustive ways. In addition, the drawings need not be drawn to scale, and some features may be enlarged to illustrate the details of specific components.

[0023] Technologies, methods, and apparatus known to ordinary technicians in the relevant field may not be discussed in detail, but where appropriate, such technologies, methods, and apparatus should be considered part of the authorization specification.

[0024] Three-dimensional motion capture technology is often used in film and television production, games, human-computer interaction and other fields. It is generally based on sensors installed on the human body to record the position information of limbs (especially joints), and combines time information (such as sampling frequency, imaging frame number, etc.) for data processing to establish a human body motion model. Furthermore, there are also motion capture methods based on computer vision, such as covering the surface of points of interest with obvious color marks, and using computer models to calculate the motion trajectory of these marks. However, in some application scenarios, it is not suitable for humans or other primates to wear sensors or color mark joints. For example, in the field of brain science, since behavioral data needs to be analyzed together with EEG data, sensors or color marks will affect the original EEG signals, so it becomes necessary to use a markerless motion capture method, but it is more difficult in actual applications.

[0025] On the other hand, in some motion capture scenarios, since two two-dimensional coordinates in different directions / angles can generally complete the reconstruction of the three-dimensional coordinates of an object, two cameras are often used to perform imaging, which will lead to occlusion of the target object, such as the target object being occluded by irrelevant objects, or the target object being occluded by itself when observing a certain joint point of the target object, etc., which in turn affects the accuracy of the motion capture results. Especially in the field of brain science, motion capture of non-human primates may place the observed objects such as monkeys in a specific experimental device, and at least partially restrict the monkey's limbs, body and head. In this case, the body part of the monkey to be observed may be blocked by the joint points to be observed due to the shooting angle or the monkey's own special movements, so that the joint points to be observed cannot be analyzed and processed.

[0026] In addition, especially in the field of brain science, the subsequent purpose of motion capture is to provide support for the research direction of brain science and brain-like science topics, such as assisting in the analysis of brain nerve signal control. Therefore, it is also necessary to time-align high-frame rate behavioral data with higher-frequency EEG signal data, which also puts higher requirements on motion capture methods and devices.

[0027] In summary, the technical solution of the present disclosure mainly proposes a motion capture method and device for one or more of the above-mentioned problems, which is particularly used for the observation of non-human primates. The markerless three-dimensional motion capture technology based on computer vision can be applied to solve the problem of the target being blocked during the motion capture process, and can achieve alignment with the EEG signals collected at the same time. It should be understood that the motion capture method involved in the present disclosure can be applied to various different fields. In addition to the field of brain science research described as an embodiment, the motion capture method and device of the present disclosure can be used in other fields such as film and television, animation, games, behavior recognition, human-computer interaction, etc., in order to grasp the movement laws and movement trajectories of the human or animal body.

[0028] Figure 1 1 is a schematic diagram showing a motion capture device based on a multi-eye camera and a processing method thereof according to an embodiment of the present disclosure. The figure shows a multi-eye camera 100 and an observation object 200. In a non-limiting embodiment, the observation object 200 is a monkey as a representative of non-human primates. The multi-eye camera 100 includes three or more cameras, such as Figure 1 The embodiment shown has four cameras, each of which is generally a high-speed camera, which images the observed object 200 at a specified frequency to obtain sample photos 1100 of the observed object 200 performing actions. It should be understood that Figure 1 The sample photos 1100 are only four randomly selected examples and do not represent photos acquired from four cameras of the multi-eye camera 100 at the same time, nor do they represent four photos acquired by the same camera in the multi-eye camera 100 at different times. The number of sample photos 1100 may also be three or more.

[0029] It should be understood that for a specified observation object and / or a specified joint point of the observation object, in theory, only two two-dimensional coordinates are needed to reconstruct a three-dimensional coordinate, so the imaging device used in the existing three-dimensional motion capture method is a binocular camera (binocular camera), that is, the observation object is photographed from two angles and directions. However, since the observation object may have dynamic behaviors such as limb translation or rotation, the imaging may be blocked, the imaging may be incomplete, the imaging may be unclear, and the imaging quality may be high or low. Taking the field of brain science research as an example, it is necessary to perform motion capture on observation objects including non-human primates, such as capturing the movement trajectory of a monkey's hand within a specific time period. The imaging quality of the sample photo with a complete, unobstructed, clear and ghost-free monkey hand part is judged to be high, and the more complete and clearer the photo, the higher the imaging quality; on the contrary, if the monkey's hand part is incomplete, blocked or unclear, the imaging quality of the sample photo is low. For a binocular camera, if one of the two sample photos captured at the same time has the above-mentioned problem affecting the imaging quality, the accuracy of the calculated two-dimensional coordinates will be reduced, and the reconstructed three-dimensional coordinates will have a large deviation. In contrast, the multi-eye camera 100 in the device or system used in the present disclosure includes three or more cameras, and three or more sample photos of the observed object are obtained at the same time. Based on the imaging quality, photos with significant defects can be excluded, or some photos with high imaging quality can be preferred, thereby improving the reliability of the sample photos entering the subsequent processing.

[0030] Specifically, firstly, joint points are marked 1102 in the sample photo 1100 according to the preset requirements of motion capture, such as Figure 1 The sample photos with the annotated joint points are input into the action recognition model to calculate the two-dimensional coordinates and confidence level 1200 of the joint points in each sample photo, including the two-dimensional coordinates (x i , y i ) and confidence α i (where i is the serial number of the sample photo, which can be 1, 2 or 3 in this embodiment, or other numbers in different embodiments.) Since the observed object 200 is an animal body that may change its movements at any time, the sample photo 1100 may not always obtain a sample photo with high imaging quality. To solve this problem, in some embodiments, a preprocessing step is performed before the action recognition model, including but not limited to defect elimination based on imaging quality 1104 and confidence α calculated according to the action recognition model. i Select the confidence interval to perform two-dimensional coordinate screening, etc. For example, the confidence interval greater than 0.5 can be used as the threshold interval to screen the sample photo 1100 to obtain valid two-dimensional coordinates to be analyzed. These two-dimensional coordinates are combined into two-dimensional coordinate pairs, and three-dimensional reconstruction is performed based on them to obtain the three-dimensional coordinates of the joint point (Xj , Y j , Z j ), where j is the sequence number of the three-dimensional coordinates, which is determined by the number of two-dimensional coordinate pairs that can be combined. For the same joint point, due to factors including but not limited to imaging accuracy and error accumulation, the three-dimensional coordinates obtained by the above calculations may not coincide with each other. The final coordinates 2000 of the joint point need to be determined in the final three-dimensional coordinate set 1300, expressed as (X f , Y f ,Z f ).

[0031] Figure 2 An exemplary configuration block diagram of a motion capture device 2000 based on a multi-eye camera according to an embodiment of the present disclosure is shown. In a non-limiting embodiment, the motion capture device 2000 based on a multi-eye camera may include a processor 2100. The processor 2100 of the motion capture device 2000 provides various functions for implementing the motion capture method of the present disclosure.

[0032] Processor 2100 may refer to various implementations of digital circuitry, analog circuitry, or mixed-signal (a combination of analog and digital) circuitry that performs functions in a computing system. Processing circuitry may include, for example, circuits such as integrated circuits (ICs), application specific integrated circuits (ASICs), portions or circuits of separate processor cores, entire processor cores, separate processors, programmable hardware devices such as field programmable gate arrays (FPGAs), and / or systems including multiple processors.

[0033] In some embodiments, the motion capture device 2000 may further include a memory (not shown). The memory of the motion capture device 2000 may store information generated by the processor 2100 and programs and data for the operation of the processor 2100. The memory may be a volatile memory and / or a non-volatile memory. For example, the memory may include, but is not limited to, a random access memory (RAM), a dynamic random access memory (DRAM), a static random access memory (SRAM), a read-only memory (ROM), and a flash memory.

[0034] In addition, the motion capture device 2000 may also be implemented at a chip level, or may be implemented as a device by including other external components.

[0035] The motion capture device 2000 disclosed in the present invention is configured to capture and analyze the motion of the moving observation object based on a multi-eye camera set up around the observation object. In some embodiments, the observation object is a non-human primate such as a monkey, which is generally placed in an experimental device such as a monkey chair when used for research in the field of brain science. In order to ensure that the monkey's movements can be followed in real time and the multi-eye camera is relatively still with the fixed body part of the monkey, the multi-eye camera is usually installed on the monkey chair and used in combination with it. The motion capture device 2000 receives sample photos imaged by the multi-eye camera, so it is necessary to communicate with the multi-eye camera, that is, the motion capture device 2000 should be physically or electrically connected to the multi-eye camera. In some embodiments, the motion capture device 2000 can also be installed on the monkey chair, or be separated from the monkey chair.

[0036] like Figure 2 As shown, in some embodiments, the processor 2100 of the multi-camera based motion capture device 2000 may include an imaging control unit 2010, a camera calibration unit 2020, a motion recognition unit 2030, a two-dimensional coordinate processing unit 2040, a three-dimensional reconstruction unit 2050 and a result generation unit 2060.

[0037] It should be understood that Figure 2 The various units of the multi-camera-based motion capture device 2000 shown are logical modules divided according to the specific functions they implement, rather than being used to limit the specific implementation method. In actual implementation, the above modules can be implemented as independent physical entities, or can also be implemented by a single entity (for example, a processor (CPU or DSP, etc.), an integrated circuit, etc.).

[0038] Next, we will refer to Figure 2 The specific operations of each unit of the motion capture device 2000 are described in detail.

[0039] The imaging control unit 2010 can be configured to control the multi-eye camera set around the observation object such as a monkey to perform imaging, so as to obtain sample photos of the monkey at multiple angles, wherein the multi-eye camera includes three or more imaging devices. The imaging control unit 2010 can be configured to control the multi-eye camera to perform changes in mechanical structure or hardware arrangement, and can also be configured to control the internal parameters of the multi-eye camera via wired or wireless communication and the like. In a non-limiting embodiment, these imaging devices are installed on a monkey chair for limiting some limbs of the monkey, generally higher than the position where the monkey is fixed, and further higher than the height of the monkey's head, so that the multi-eye camera can shoot the monkey with a bird's-eye view. The control of the multi-eye camera by the imaging control unit 2010 includes but is not limited to changing the shooting angle of each camera in the multi-eye camera, opening or closing some cameras, setting imaging frequency and other camera parameters, etc. In addition, the imaging control unit 2010 can also be configured to perform control of the multi-eye camera according to preset imaging requirements, such as in the field of brain science research, the experimenter using the monkey chair can perform personalized settings on the multi-eye camera based on the characteristics of the observation object or the experimental objectives, etc.

[0040] The camera calibration unit 2020 can be configured to calibrate the multi-eye camera and calculate the homography matrix of each camera in the multi-eye camera. Since multiple cameras are used to image the monkey, it is necessary to align the sample photos taken at different angles at the same time in time, and use the posture estimation of each camera to calibrate the multi-eye camera. In a non-limiting embodiment, the Zhang Zhengyou calibration method is used to solve the intrinsic parameters of each camera in the multi-eye camera, and the rotation vector and translation vector of each camera based on the world coordinate system are calculated; the rotation matrix is ​​calculated using the Rodriguez equation, and then the homography matrix of each camera is calculated using the aforementioned variables. In general, the homography matrix describes the transformation relationship of coplanar points from one perspective to another perspective, which contains the intrinsic and extrinsic parameters of each camera. When using the Zhang Zhengyou calibration method, the homography matrix is ​​used to calculate the transformation relationship between the calibration plate plane and the pixel plane used for imaging.

[0041] The action recognition unit 2030 can be configured to perform action recognition of the observed object based on multiple sample photos obtained by the imaging control unit 2010. Generally, it is necessary to take a sufficient number of sample photos of the observed object, and apply a preset action recognition model to train these sample photos. In a non-limiting embodiment, the action recognition model includes a deep learning model based on deepcut, mainly from the deeplabcut open source library. The specific idea is: propose candidate areas of the observed object in the sample photos, and take each candidate area as a joint point; all related nodes form a dense connection graph, and the correlation between the joint points is used as the weight between the joint points in the sample photos, which is solved as an optimization problem. The conventional deepcut model can be used to identify sample photos of an unknown number of objects, in which the joint points belonging to the same observed object are classified into one class, and each object is a separate class.

[0042] In the field of brain science, when observing non-human primates such as monkeys, there is usually only one object in the sample photo, so the deepcut model mainly identifies the specific limb parts of the observed object based on the marked joint points. In a non-limiting embodiment, in the motion capture research in the field of brain science, refer to Figure 1 The joint point annotation shown in (operation 1102) can set the specific positions of the monkey's hands, tail or head as the joint points that need to be identified or tracked, and input the sample photos with the joint points of motion capture annotated into the deepcut model for training. For example, based on a large number of collected monkey hand samples, a two-dimensional monkey hand recognition model is trained using deeplabcut. Next, the data obtained by the data training is applied to generate the two-dimensional coordinates of these joints on each sample photo and their corresponding confidence levels. The confidence level is used to describe the range of credibility of the corresponding two-dimensional coordinates, such as being set in the interval [0, 1].

[0043] Additionally, this application improves the open source deepercut model. The traditional deepercut model is a human posture estimation model written based on tensorflow, but the original model in the open source library removes the parallel reasoning supported by tensorflow, and only supports the input of one picture and returns the joint point coordinate information of the picture. In order to increase the running speed of the model, this application modifies the underlying logic to add parallel reasoning back to the deepercut model, so that the modified model supports the input of multiple pictures and returns the joint point coordinate information of the multiple pictures at the same time. The modified model can realize real-time calculation of multiple sample photos, that is, it can output the recognition results while collecting the original picture.

[0044] Additionally or alternatively, before the sample photos are input into the deepcut model for calculation, the sample photos are preprocessed, including excluding defective photos, as mentioned above, that is, sample photos with high imaging quality are selected to enter the action recognition model. Figure 1 , the sample photo 1100 is schematically shown as four photos, and the joint points of motion capture are marked on the monkey's hand when performing joint point marking (operation 1102). Among the four photos shown in the sample photo 1100, the latter three all capture the complete and clear monkey's hand, while the first photo does not capture the image of the monkey's hand within the imaging range of the camera due to the random movement of the monkey's limbs. Therefore, the photo is excluded in the defect elimination (operation 1104) and will not enter the subsequent processing steps. In addition, defect elimination 1104 can also include situations such as blurred images due to the monkey's excessive movement speed, and the appearance of occlusions in the picture due to the imaging angle.

[0045] The two-dimensional coordinate processing unit 2040 can be configured to perform further data processing based on the two-dimensional coordinates of the joint points generated in the motion recognition unit 2030 and their confidence. Generally speaking, the confidence reflects the credibility of a certain parameter of the sample. In the embodiment of the present disclosure, it is the credibility of the joint points identified in the sample photos, which to a certain extent reflects the accuracy of the two-dimensional coordinate position that the sample photos can provide. Therefore, the required two-dimensional coordinates can be screened by setting a confidence threshold, and more accurate motion capture results can be obtained by inputting only the two-dimensional coordinates with higher credibility into subsequent processing. Among them, the selected confidence range can be one (or more) intervals, or one (or more) values, which can be pre-set by the user based on a specific purpose of use. Accordingly, the screening of the two-dimensional coordinates can directly eliminate the two-dimensional coordinates whose confidence is lower than the preset threshold, or only retain the two-dimensional coordinates whose confidence falls within the preset interval or value set. In a non-limiting embodiment, the preset threshold of the confidence is set to 0.5, and the two-dimensional coordinates that enter the subsequent analysis process (i.e., participate in the three-dimensional reconstruction of the joint point) are obtained by eliminating the two-dimensional coordinates with a confidence lower than 0.5. Further, since two two-dimensional coordinates can be used to reconstruct the three-dimensional coordinates, the two-dimensional coordinate processing unit 2040 is also configured to combine the two-dimensional coordinates screened in the above steps in pairs to obtain a plurality of two-dimensional coordinate pairs.

[0046] Next, the 3D reconstruction unit 2050 performs 3D reconstruction based on the received multiple 2D coordinate pairs. The 2D coordinate processing unit 2040 selects the coordinate values ​​that meet the confidence requirements, which is beneficial to improving the accuracy of 3D reconstruction. Based on these 2D coordinate pairs and the homography matrix calculated by the camera calibration unit 2020, the 3D coordinates of the marked joint points in the world coordinate system are calculated. The multi-camera includes three or more cameras, such as Figure 1 The multi-eye camera 100 includes four cameras. When the imaging quality of all sample photos meets the requirements and the two-dimensional coordinates of the joint points marked in each sample photo have a confidence level that meets a preset threshold, at most four valid two-dimensional coordinates can be obtained. Three-dimensional reconstruction is performed based on all six two-dimensional coordinate pairs obtained by pairwise combination to form a three-dimensional coordinate set about the joint point.

[0047] Further, the result generation unit 2060 can be configured to determine the final coordinates of the joint point from the aforementioned three-dimensional coordinate set according to a preset rule. In a non-limiting embodiment, the multi-eye camera is configured as a high-speed camera that obtains sample photos at a certain frequency, and the three-dimensional coordinates of the joint points arranged by imaging time can be obtained in the three-dimensional reconstruction unit 2050, and the calculation results are determined in the result generation unit 2060. For each joint point, the preset rule generally includes taking the median or average value of its three-dimensional coordinate set. Preferably, among all the calculated three-dimensional coordinates, taking the median value can well exclude the coordinate points whose confidence meets the preset threshold but still identifies errors, avoiding the negative impact on the final result. Alternatively, for the polygon or polyhedron formed by the three-dimensional coordinate set of each joint point in the world coordinate system, the preset rule can also include taking the center of gravity of the polygon or the center of mass of the polyhedron. Based on this, the motion trajectory of each joint point changing with time in the world coordinate system can be established, thereby realizing the motion capture of the observed object.

[0048] Additionally or alternatively, the imaging control unit 2010 is further configured to receive information from the camera calibration unit 2020 or the result generation unit 2060, and control the multi-eye camera based on the received information. In a non-limiting embodiment, the multi-eye camera can be personalized according to preset imaging requirements (such as design arrangements based on experimental purposes), and the camera calibration unit 2020 is further configured to check the posture of each camera in the multi-eye camera when performing calibration to determine that the posture of the camera meets the preset requirements. Wherein, when the inspection result of the camera calibration unit 2020 on the camera posture does not meet the preset requirements, the camera calibration unit 2020 sends information associated with the inspection result to the imaging control unit 2010, and the imaging control unit 2010 makes necessary adjustments to the multi-eye camera based on the information. Similarly, based on the final coordinates determined by the result generation unit 2060, the motion trajectory of the joint point in the world coordinate system can be established. When performing motion capture on monkey limbs in the field of brain science research, it is hoped that the motion trajectory of the joint point can be located in the best position for observation (generally, in the center of the picture) and the imaging angle can be adjusted, or it is hoped that the part of the picture that captures the trajectory can be resized and adjusted, etc. The result generation unit 2060 sends the information associated with the desired adjustment to the imaging control unit 2010, and the imaging control unit 2010 performs these desired adjustments to the multi-eye camera based on the information.

[0049] Refer to the following Figure 3 The exemplary flow chart of the motion capture method 3000 based on multi-camera according to an embodiment of the present disclosure is described as follows. The method 3000 can be used, for example, Figure 2 The multi-camera based motion capture device 2000 is shown.

[0050] like Figure 3 As shown, in step S301, a multi-camera is used to simultaneously image the observed object to obtain multiple sample photos, wherein the multi-camera is calibrated and the homography matrix of each camera in the multi-camera is calculated. In step S302, the joint points to be captured are selected from multiple sample photos for annotation, and the action recognition model is applied for training to generate the two-dimensional coordinates of the joint points on each photo in the multiple sample photos and their corresponding confidence. In step S303, all two-dimensional coordinates are screened according to a preset threshold of confidence, and the screened two-dimensional coordinates are combined in pairs to obtain multiple two-dimensional coordinate pairs. In step S304, multiple two-dimensional coordinate pairs are respectively reconstructed in three dimensions based on the homography matrix, and the calculation results are used as the three-dimensional coordinate set of the joint point. In step S305, the final coordinates of the joint point are determined from the three-dimensional coordinate set according to the preset rules.

[0051] The above steps S301 to S305 can be implemented separately or together by the imaging control unit 2010, the camera calibration unit 2020, the action recognition unit 2030, the two-dimensional coordinate processing unit 2040, the three-dimensional reconstruction unit 2050 and the result generation unit 2060. Figure 2 The description is similar and will not be repeated here.

[0052] Next, we will refer to Figure 4 An exemplary configuration diagram of a motion capture system based on a multi-eye camera according to another embodiment of the present disclosure is described. As shown in the figure, in a non-limiting embodiment, a single-chip microcomputer 410 is used to control a multi-eye camera 420, and the multi-eye camera 420 sends sample photos obtained by imaging to a computer 430 for subsequent processing. The multi-eye camera 420 includes three or more separate high-speed cameras, which shoot the observed object at a specific imaging frequency. Figure 4 In the figure, cameras 421, 422, 423, and 424 are shown. In one embodiment, the type of the single-chip microcomputer 410 may include the Arduino open source electronic platform. Generally, a control signal is sent at a specific first frequency on a pin of the Arduino board, and the control signal is simultaneously connected to cameras 421, 422, 423, and 424 in parallel through a signal line. Each camera performs imaging once when it receives one of the control signals, thereby acquiring one frame of imaging. Among them, the control signal can be a TTL signal. At this time, since the source of the control signal is the same pin on the board, all the images acquired by performing one imaging are from the same moment, thereby realizing synchronous triggering of multiple cameras.

[0053] In particular, in the field of brain science research, motion capture of the observed object can provide a research basis for exploring the correlation between brain activity and limb movement. Therefore, the motion capture system in the embodiment of the present disclosure can also be used to match the captured image with the brain activity occurring at the same time. Generally, brain activity can be represented by electroencephalogram signals collected by flexible electrodes suitable for brain biological tissue. Figure 4 In the example, the processed EEG signal 40 is also connected to the pin of the microcontroller 410. For example, the EEG signal of the observed object is collected at a specific frequency such as 20kHz. The EEG signal is generally an irregular analog signal. For the convenience of analysis, it is processed into a 6-bit code form and transmitted to the microcontroller 410 (such as an Arduino board) as the EEG signal 40. The EEG signal 40 is transmitted via 6 signal lines and read by 6 pins on the microcontroller 410. Its code range is 0 to 2. 6(i.e. 0 to 64). The codes can be defined differently according to the user's experimental purpose, such as 1 represents the start of the task, 8 represents successful grasping, 23 represents the appearance of the target, etc.

[0054] In a non-limiting embodiment, the EEG signal 40 in the form of a code is read at a second frequency in the single-chip microcomputer 410. Since the acquisition frequency of the aforementioned brain signal is as high as 20kHz, the higher the monitoring frequency of the code is required, the better, such as the second frequency can be 10kHz. And in particular, the first frequency (i.e., the fps of the camera) that triggers imaging is higher than the multiple N between the second frequencies of reading the EEG signal 40, i.e., N>1. Each time the EEG signal 40 is read N times, an imaging of the multi-eye camera 420 is triggered, and the control signal is counted in the cycle. Further, the first signal in the control signal on the single-chip microcomputer 410 is aligned with the first frame of the sample photo obtained by imaging the multi-eye camera 420 on the time axis. Thus, the code in the EEG signal 40 read each time and the trigger count to which it belongs, i.e., the frame number in the sample photo, can be obtained. After analyzing multiple sample photos at the same time as described above to obtain the three-dimensional coordinates of the joint point, the code in the EEG signal 40 can be aligned to the three-dimensional coordinates. In other words, the correspondence between the EEG signal 40 representing brain activity and the captured motion trajectory can be obtained, providing a useful reference for studying the triggering or control of animal limbs by brain activity.

[0055] Additionally, the code and frame count value in the EEG signal 40 need to be transmitted to the computer via the serial port in real time. Since the Arduino-type microcontroller 410 cannot perform parallel calculations, and the transmission of serial port data will take time and thus affect the second frequency used for reading, in the Arduino board, all the read codes (characters: single bytes 0 to 2 8 ) and frame count (integer: four bytes 0 to 2 32 ) are put into a first-in-first-out queue byte by byte as data to be processed. In the main loop with a frequency of 10kHz, one byte is taken out of the queue and sent to the computer in each cycle. When the baud rate is set to 115200, it takes about 80μs to send one byte, and the time taken for one cycle needs to be fixed to 100μs (corresponding to the second frequency of 10kHz), which is achieved through the clock in the Arduino. This is because if two or more bytes are sent in one cycle, it will not be possible because the sending time exceeds 100μs, so the data needs to be split byte by byte into the queue and then sent byte by byte.

[0056] Furthermore, the aforementioned serial port data stream is received on the computer side and needs to be decoded at the same time. Some of the received data are frame counts composed of four bytes, and some are single-byte codes. There is no pattern between them, so the data to be transmitted needs to be encoded at the single-chip microcomputer 410. Since the single-byte code (8 bits) only uses 0 to 64, that is, the last 6 bits, the first bit is always 0; in contrast, the maximum integer that can be represented by the four-byte integer frame count value (32 bits) is 4294967296 (frames), and the camera's acquisition frame rate is 200fps, which can be converted to record 5965 hours (before overflow), which is much longer than the actual time of an experiment. If the first bit is set to the mark bit ("1") relative to the first bit of the single-byte code, the remaining 31 bits can still represent a recording time of 2982 hours, which meets the required duration of an experiment. Based on this, if the mark bit of the first bit of the data is 1, the data represented by the subsequent four bytes including the byte itself is the frame count value, otherwise (the mark bit is 0) the byte is the code value.

[0057] Furthermore, when the computer decodes the received serial data stream, if the first bit of the received byte is 1, the subsequent four bytes (including the byte itself) are decoded together as a frame count value, otherwise the byte is decoded alone as a code value. Ultimately, the computer can obtain a series of frame count values, as well as the signal values ​​read in these frames (multiple codes can be read simultaneously in each frame), and then align these codes with the actual captured real EEG signals. Ultimately, the three-dimensional coordinates of all joint points are calculated and saved frame by frame based on the imaging frequency, and the corresponding code values ​​are recorded at the frames with codes, which facilitates the joint analysis of brain signals and motion capture data.

[0058] Figure 5A and Figure 5B An exemplary perspective view of a multi-eye camera for a motion capture device according to an embodiment of the present disclosure is shown. In a non-limiting embodiment, the motion capture method of the present disclosure can be used to observe a monkey (not shown) whose body is fixed in a monkey chair, and can be particularly used to capture the motion of the monkey's hands. Figure 5A and 5B The main figure shows a motion capture device 510 which can be a functional accessory of the monkey chair; and in addition, the motion capture device 510 can also be used in combination with a head fixing device 520 which is also a functional accessory of the monkey chair, such as to avoid interference of head movement with motion capture. Generally speaking, the motion capture device 510 is arranged above the monkey chair to image the monkey fixed in the monkey chair, and it can be directly arranged on the monkey chair or can be connected to the monkey chair via the head fixing device 520.

[0059] Reference Figure 5A and Figure 5BIn the non-limiting embodiment shown, the motion capture device 510 can be configured to capture the movements of the limbs, especially the hands, of a monkey in a monkey chair. The motion capture device 510 can include a camera support frame 5101 and at least three cameras 5102 arranged on the camera support frame 5101. Figure 5A The embodiment shown in the figure has four cameras 5102 (one of which is blocked in the figure). As mentioned above, two sets of two-dimensional coordinates can be used for three-dimensional reconstruction, that is, the images captured by two cameras 5102 are sufficient for motion capture. However, in practice, it is found that when only two cameras are set, the viewing angle that each camera 5102 can provide is relatively limited, so the effect of three-dimensional motion capture, especially the accuracy of three-dimensional motion capture, may be damaged due to the occlusion of the captured object. By setting more than two cameras 5102 (such as Figure 5A The four cameras 5102 shown in FIG. 5 can better eliminate the influence of occlusion on motion capture. In general, it is expected that the four cameras 5102 shoot the captured object from different and relatively different viewing angles as much as possible.

[0060] In this embodiment, the camera support frame 5101 can be directly connected to the monkey chair (not shown) located below the motion capture device 510; in some other embodiments, the camera support frame 5101 can also be fixed to the connecting side ears of the head fixing device 520 via corresponding bolts 5103.

[0061] like Figure 5A and Figure 5B As shown, the camera support frame 5101 may include a vertical support rod 5104 ( Figure 5A The embodiment shown in FIG. 5 is two vertical support rods 5104) and a horizontal support frame 5105 extending substantially horizontally from the vertical support rods 5104. The horizontal support frame 5105 includes a crossbeam 5106 extending between the two vertical support rods 5104 and an arc frame 5107 extending horizontally forward from the vertical support rods 5104, and a plurality of cameras 5102 are arranged on the arc frame 5107. In order to ensure the stability of the structure and avoid disruptive shaking, a reinforcement portion 5108 is constructed between the vertical support rods 5104, the crossbeam 5106 and the arc frame 5107.

[0062] Further, two cameras 5102 may be provided at the front of the arc frame 5107, and one camera 5102 may be provided at the side of the arc frame 5107. Each camera 5102 may be connected to the arc frame 5107 via a universal ball 5109, in particular, to a corresponding connection portion 5110 of the arc frame 5107. Figure 5AIn the figure, the two cameras 5102 located at the front are connected to the corresponding universal balls 5109 via an L-shaped adapter plate 5111 respectively, and the two cameras 5102 located at the side are connected to the corresponding universal balls 5109 via a flat adapter plate 5112 respectively.

[0063] like Figure 5B As shown, the motion capture device 510 may further include an anti-collision cover 5113 partially surrounding the camera 5102. In a non-limiting embodiment, the two cameras 5102 disposed on the side are respectively equipped with an anti-collision cover 5113, and the two cameras 5102 disposed in the front are jointly equipped with an anti-collision cover 5113, so that the anti-collision cover 5113 has a larger size. The anti-collision cover 5113 can be fixed to the corresponding connection part 5110 via bolts 5114.

[0064] In some embodiments, when the motion capture device 510 is mounted on a monkey chair, the projection of the motion capture device 510 on the ground can be completely surrounded by the projection of the monkey chair on the ground, thereby preventing the motion capture device 510 from directly colliding with other objects when the monkey chair moves.

[0065] Figure 6 An exemplary configuration of a computing device that can implement an embodiment of the present disclosure is shown. The computing device includes one or more processors 601, an input / output interface 605 connected to the processor 601 via a bus 604, and memories 602 and 603 connected to the bus 604. In some embodiments, the memory 602 may be a read-only memory (ROM), and the memory 603 may be a random access memory (RAM).

[0066] Processor 601 may be any kind of processor, and may include, but is not limited to, one or more general-purpose processors or special-purpose processors (such as special-purpose processing chips). Memories 602 and 603 may be any non-transitory storage device that can implement data storage, and may include, but is not limited to, disk drives, optical storage devices, solid-state memories, floppy disks, flexible disks, hard disks, magnetic tapes or any other magnetic media, compressed disks or any other optical media, cache memories and / or any other storage chips or modules, and / or any other media from which a computer can read data, instructions and / or codes.

[0067] The bus 604 may include, but is not limited to, an Industry Standard Architecture (ISA) bus, a Micro Channel Architecture (MCA) bus, an Enhanced ISA (EISA) bus, a Video Electronics Standards Association (VESA) local bus, and a Peripheral Component Interconnect (PCI) bus.

[0068] In some embodiments, the input / output interface 605 is connected to an input unit 606 configured by an input device such as a keyboard and a mouse for a user to input an operation command, an output unit 607 that outputs a processing operation screen and an image of a processing result to a display device, a storage unit 608 including a hard disk drive and the like for storing programs and various data, and a communication unit 609 including a local area network (LAN) adapter and the like and performing communication processing via a network represented by the Internet. In addition, a drive 610 that reads and writes data from and on a removable storage medium 611 is connected.

[0069] Various aspects, embodiments, implementations or features of the foregoing embodiments may be used alone or in any combination. Various aspects of the foregoing embodiments may be implemented by software, hardware or a combination of hardware and software.

[0070] For example, the foregoing embodiments may be embodied as computer readable code on a computer readable medium. A computer readable medium is any data storage device that can store data that can thereafter be read by a computer system. Examples of computer readable media include read-only memory, random access memory, CD-ROM, DVD, magnetic tape, hard drive, solid state drive, and optical data storage device. The computer readable medium may also be distributed in a network coupled computer system so that the computer readable code is stored and executed in a distributed manner.

[0071] For example, the aforementioned embodiments may be in the form of hardware circuits. The hardware circuits may include any combination of combinational logic circuits, clock storage devices (such as floppy disks, flip-flops, latches, etc.), finite state machines, memories such as static random access memories or embedded dynamic random access memories, custom designed circuits, programmable logic arrays, etc.

[0072] In one embodiment, hardware circuits according to the present disclosure may be implemented by coding and designing one or more integrated circuits using a hardware description language (HDL) such as Verilog or VHDL, or in combination using discrete circuits.

[0073] It should be noted that the terms used herein are for the purpose of illustrating specific aspects only and are not intended to limit the disclosure. As used herein, the singular forms "a" and "the one" shall include the plural forms unless the context clearly states otherwise. It is understood that the terms "include" and "comprises" and other similar terms, when used in the application documents, specify the existence of the stated operations, elements and / or parts, without excluding the existence or addition of one or more other operations, elements, parts and / or combinations thereof. As used herein, the term "and / or" includes all arbitrary combinations of one or more related enumerated items. In the description of the drawings, similar reference numerals always represent similar elements.

[0074] The thickness of the elements in the accompanying drawings can be exaggerated for the sake of clarity. It is also understood that if an element is referred to as being on another element, coupled to another element or connected to another element, then the element can be directly formed on the other element, coupled to it or connected to it, or one or more intervening elements can be arranged between them. On the contrary, if the expressions "directly on...", "directly coupled to..." and "directly connected to..." are used herein, then there is no intervening element. Other words used to illustrate the relationship between elements should be similarly interpreted, such as "between..." and "directly between...", "attachment" and "direct attachment", "adjacent" and "directly adjacent", etc.

[0075] Terms such as "top," "bottom," "above," "below," "above," "below," etc. are used herein to describe the relationship of one element, layer or region to another element, layer or region as illustrated in the drawings. It will be understood that these terms are intended to encompass other orientations of the device in addition to the orientation depicted in the drawings.

[0076] It is to be understood that although the terms "first", "second", etc. may be used herein to describe different elements, these elements should not be limited by these terms. These terms are only used to distinguish one element from another element. Therefore, a first element may be referred to as a second element without departing from the teachings of the present application.

[0077] It is also contemplated that all exemplary embodiments disclosed herein may be arbitrarily combined with one another. Finally, it should be noted that the above embodiments are only used to understand the present application and do not limit the scope of protection of the present application. For those skilled in the art, modifications may be made based on the above embodiments, and these modifications do not depart from the scope of protection of the present application.

Claims

1. A motion capture method based on multi-cameras, comprising: Using a multi-camera to simultaneously image the observed object to obtain a plurality of sample photos, wherein the multi-camera is calibrated and a homography matrix of each camera in the multi-camera is calculated; Selecting joint points to be captured from the multiple sample photos for annotation, and applying an action recognition model for training to generate two-dimensional coordinates of the joint points on each of the multiple sample photos and their corresponding confidences; All two-dimensional coordinates are screened according to a preset threshold of confidence, and the screened two-dimensional coordinates are combined in pairs to obtain a plurality of two-dimensional coordinate pairs; Reconstructing the plurality of two-dimensional coordinate pairs in three dimensions based on the homography matrix, and using the calculation results as the three-dimensional coordinate sets of the joint points; The final coordinates of the joint point are determined from the three-dimensional coordinate set according to a preset rule.

2. The motion capture method according to claim 1, wherein: The action recognition model includes a deep learning model based on deepcut.

3. The motion capture method according to claim 2, wherein: The action recognition model supports parallel reasoning on the multiple sample photos.

4. The motion capture method according to claim 1, wherein: Screening the two-dimensional coordinates includes eliminating two-dimensional coordinates whose confidence is lower than the preset threshold.

5. The motion capture method according to claim 1, wherein: Screening the two-dimensional coordinates includes excluding abnormal values.

6. The motion capture method according to claim 1, wherein: The preset rule includes taking a median value or an average value for the three-dimensional coordinate set.

7. The motion capture method according to claim 1, further comprising: The plurality of sample photos are pre-processed, wherein defective photos are excluded.

8. The motion capture method according to claim 1, wherein: A control signal is sent to each camera in the multi-eye camera at the same time at a first frequency, and each camera acquires a frame of imaging when receiving one of the control signals.

9. The motion capture method according to claim 8, further comprising: The electroencephalogram (EEG) signal of the observed object is collected and processed into a combination of preset codes, the codes are read at a second frequency, and the corresponding relationship between the codes and the final coordinates is analyzed.

10. The motion capture method according to claim 9, wherein: The second frequency is an integer N times the first frequency, where N>1.

11. The motion capture method according to claim 9, wherein: A first signal of the control signals is aligned with a first frame of the plurality of sample photos on a time axis.

12. The motion capture method according to claim 9, wherein: The read codes and frame counts of the plurality of sample photos are put into a first-in-first-out queue byte by byte as data to be processed.

13. A motion capture device based on a multi-camera, comprising: a memory having instructions stored therein; as well as A processor configured to execute instructions stored in the memory to perform the following processing: Using a multi-camera to simultaneously image the observed object to obtain a plurality of sample photos, wherein the multi-camera is calibrated and a homography matrix of each camera in the multi-camera is calculated; Selecting joint points to be captured from the multiple sample photos for annotation, and applying an action recognition model for training to generate two-dimensional coordinates of the joint points on each of the multiple sample photos and their corresponding confidences; All two-dimensional coordinates are screened according to a preset threshold of confidence, and the screened two-dimensional coordinates are combined in pairs to obtain a plurality of two-dimensional coordinate pairs; Reconstructing the plurality of two-dimensional coordinate pairs in three dimensions based on the homography matrix, and using the calculation results as the three-dimensional coordinate sets of the joint points; The final coordinates of the joint point are determined from the three-dimensional coordinate set according to a preset rule.

14. The motion capture device according to claim 13, wherein The processor is further configured to execute instructions on the memory to perform the following processes: Screening the two-dimensional coordinates includes eliminating two-dimensional coordinates with confidence levels lower than the preset threshold or excluding abnormal values.

15. The motion capture device according to claim 13, wherein The processor is further configured to execute instructions on the memory to perform the following processes: The preset rule includes taking a median value or an average value for the three-dimensional coordinate set.

16. The motion capture device according to claim 13, wherein The processor is further configured to execute instructions on the memory to perform the following processes: A control signal is sent to each camera in the multi-eye camera at the same time at a first frequency, and each camera acquires a frame of imaging when receiving one of the control signals.

17. The motion capture device according to claim 16, wherein The processor is further configured to execute instructions on the memory to perform the following processes: The electroencephalogram (EEG) signal of the observed object is collected and processed into a combination of preset codes, the codes are read at a second frequency, and the corresponding relationship between the codes and the final coordinates is analyzed.

18. The motion capture device according to claim 17, wherein The processor is further configured to execute instructions on the memory to perform the following processes: The read codes and frame counts of the plurality of sample photos are put into a first-in-first-out queue byte by byte as data to be processed.

19. A computer-readable storage medium having a computer program stored thereon, wherein when the computer program is executed by a processor, the processor is caused to execute the motion capture method according to any one of claims 1 to 12.

20. A computer program product, comprising a computer program, wherein when the computer program is executed by a processor, the processor is caused to perform the motion capture method according to any one of claims 1 to 12.

Images