Motion Capture System

Abstract

A motion capture system and a method for performing motion capture that can accommodate camera mounting misalignments and environmental changes that adversely affect the motion capture process. In a system for motion capture, a plurality of cameras (4) having computer processing hardware (20) are provided. The processing unit performs processing using image data from each of the plurality of cameras. Each camera includes an accelerometer (22). The processing unit performs processing dependent on output from the accelerometer. Each camera also includes a temperature sensor (18) mounted on the computer processing hardware (20). The processing unit performs processing dependent on output from the temperature sensor.

Description

[Technical field]

[0001] The present invention relates to a motion capture system and a method for performing motion capture, in which a plurality of cameras each include an accelerometer and / or a temperature sensor. [Background technology]

[0002] A motion capture system typically includes multiple cameras configured to monitor a target area where the movement of a target, e.g., an actor, is captured. The captured movement may be used, for example, to animate a digital character in a movie or computer game based on the movement of the actor. Typically, in such a system, each of the cameras must be mounted in a separate location and pointed towards the target area. The cameras also need to be calibrated and connected to a processing device, such as a desktop or laptop computer or tablet. Setting up such a motion capture system can be time-consuming, for example, requiring multiple trips by an operator between the processing device and each camera and / or the multiple people involved. For example, one person may directly operate the processing device while one or more other people move around the room to individually adjust and check each camera while interacting with the processing person. The processing person can provide feedback to the mobile operator about the images each of the cameras captures as the mobile operator moves from one camera to the next.

[0003] Motion capture systems can be sensitive to environmental conditions and disturbances. For example, misalignment of the camera mounting (e.g., due to tapping, thermal expansion / contraction, bending, etc.) can cause the camera position or orientation to change over time. This can have a detrimental effect on the motion capture process and / or require frequent and time-consuming recalibration.

[0004] A further challenge with existing systems is that temperature changes in the camera itself, for example by creating differences in the optical path from the lens to the camera's imager, can alter the camera's performance. These effects can in some cases be mitigated by calibrating the camera after the temperature has reached a steady state. However, determining when such a calibration should occur can be difficult. Users are sometimes instructed to wait a period of time after booting up the system before calibrating the camera or beginning to use the system. However, this can result in inconvenient delays. Furthermore, such approaches are not effective when the steady state temperature is not reached reproducibly or when the temperature changes during use of the system, for example due to changes in the activity of the individual cameras or the temperature of the surrounding environment. Summary of the Invention [Problem to be solved by the invention]

[0005] It is an object of the present invention to provide a motion capture system and method that at least partially addresses one or more of the above problems. [Means for solving the problem]

[0006] According to one aspect of the invention, a motion capture system comprises a plurality of cameras and a processing unit configured to perform processing using image data from each of the plurality of cameras, each of the cameras comprising an accelerometer, the processing unit configured to perform processing dependent on output from the accelerometer.

[0007] Thus, a system is provided in which each individual camera can respond to the output from an accelerometer provided on each camera. This capability provides increased flexibility in how motion capture systems can be set up and maintained, and facilitates certain procedures involving the proximity of a user to an individual camera. For example, the system can be configured such that the accelerometer output is used to allow a user to select an individual camera by tapping or double tapping the camera, thereby causing a variation in the accelerometer output. The accelerometer may alternatively or additionally be used to detect undesirable deviations in the orientation of the camera, thereby prompting investigation actions and / or recalibration procedures.

[0008] According to one aspect of the invention, a motion capture system comprises a plurality of cameras and a processing unit configured to perform processing using image data from each of the plurality of cameras, each camera comprising a temperature sensor, the processing unit configured to perform processing dependent on an output from the temperature sensor.

[0009] Thus, a system is provided that allows individual cameras to react to temperature changes for each camera. This feature makes it easier to detect when a steady state temperature has been reached since the camera was powered on, and allows the calibration procedure to begin more reliably at steady state, and possibly sooner than using other techniques. This feature also allows the system to react to temperature changes that may occur other than immediately after powering on, for example due to changes in environmental temperature.

[0010] According to another aspect of the present invention, there is provided a method of performing motion capture using a plurality of cameras, where each camera includes an accelerometer, the method including performing, on a processing device, processing dependent on output from the accelerometers.

[0011] According to another aspect of the present invention, there is provided a method of performing motion capture using a plurality of cameras, each camera comprising a temperature sensor, the method including performing, in a processing device, processing dependent on output from the temperature sensors.

[0012] Embodiments of the present invention will now be described, by way of example only, with reference to the accompanying drawings in which corresponding reference symbols indicate corresponding parts and in which: [Brief description of the drawings]

[0013] [Figure 1] The motion capture system is shown. [Diagram 2] A top view of multiple cameras of a motion capture system aimed at a target area. [Diagram 3] FIG. 3 is a schematic side view of a single camera of the multiple cameras of FIG. 2. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0014] In one embodiment, an example of which is shown in Figures 1-3, a motion capture system 2 is provided that includes a plurality of cameras 4. A processing device 6 is configured to perform processing using image data received from each of the plurality of cameras 4. The processing may include, for example, processing related to performing motion capture. In this context, motion capture is understood to encompass at least any process or technique of digitally recording patterns of movement, for example to record the movement of an actor for the purpose of animating a digital character in a movie or computer game. Optionally, one or more of the plurality of cameras 4 may be configured to perform processing involved in the motion capture procedure. For example, the cameras may pre-process image data captured by the cameras before sending the data to the processing device. The processing device 6 may take various forms. For example, the processing device 6 may include a computer (e.g., a desktop or laptop PC), a tablet computer, a smartphone, or any combination thereof in communication with each other. The processing device 6 may include a display 8. The processing device 6 may further include processing hardware 10 (e.g., a CPU, memory, etc.), and an interface 12 that allows a user to interact with the processing device 6. The interface may include a touch screen, a keyboard, a mouse, or other input devices.

[0015] In one embodiment, each of the cameras 4 is equipped with an accelerometer 22. An example of such a camera is shown in Figure 3. In this type of embodiment, the processing unit 6 is configured to perform processing that depends on the output from the accelerometer 22.

[0016] In one embodiment, the processing unit 6 is configured to select one of the cameras 4 based on the output from the accelerometer 22. The processing unit 6 is further configured to perform a selected camera action responsive to which camera 4 is selected. The selected camera action may include a variety of different functions associated with the selected camera. In one embodiment, the selected camera action includes displaying data from the selected camera 4, for example, on the display 8. This functionality thus allows a user to view data associated with a particular camera by simply changing the output from the accelerometer in a particular way (e.g., by tapping or otherwise manipulating the camera 4).

[0017] In one embodiment, the selected camera processing includes displaying data from multiple cameras and providing visual information indicating which of the displayed data corresponds to the selected camera. For example, the processing device 6 may be configured to display image data captured by multiple cameras 4 in a corresponding plurality of separate windows on the display 8. The selected camera may be indicated, for example, by highlighting the window corresponding to the selected camera or by bringing the window to the front relative to other windows.

[0018] Alternatively or additionally, the selected camera processing may include displaying data from a selected camera and not displaying data from other cameras. Thus, camera manipulation that results in a change in the characteristics of the output from the camera's accelerometer 22 can be used to switch between displaying data from different cameras on the display 8.

[0019] The above functionality may be particularly useful where the processing device 6 includes a portable part such as a laptop or tablet. In this case, a user may carry the portable part from one camera 4 to another in the system and conveniently select and view data relating to the camera currently next to them by simply changing the output from the accelerometer (e.g., by tapping on the camera 4). This functionality frees the user from having to manually interact with the portable part itself to select the camera. This approach can save considerable time and effort, especially where many different cameras 4 are provided.

[0020] In one embodiment, the data displayed in the selected camera process includes image data captured by the selected camera 4. Alternatively or additionally, the displayed data includes configuration data of the selected camera. The configuration data may include any operational or other parameters of the camera 4. In one embodiment, the selected camera process includes providing a selected camera interface, e.g., as part of the display 8, e.g., an active portion of a touch screen of the display 12, that allows a user to control the selected camera 4. Thus, a user can conveniently invoke an interface that allows the user to control the selected camera 4 via the processing device 6 (e.g., a laptop or tablet) by tapping or manipulating the camera 4 to change the output from the accelerometer 22.

[0021] The selected camera interface may allow for control of multiple cameras 4 and provide a visual indication of which of the cameras 4 is the selected camera 4. This approach allows a user to view configuration data or other related data from other cameras while controlling the selected camera 4. This may be useful when data from other cameras is relevant for comparison or reference.

[0022] In one embodiment, the selected camera interface is configured to be capable of controlling the selected camera 4 and not other cameras 4. Various other functions related to the selected camera 4 may be provided by the selected camera interface.

[0023] In one embodiment, the processing unit 6 is configured to select one of the cameras 4 in response to a variation (e.g., the amplitude of the variation, which is proportional to the magnitude of acceleration, or the duration of the variation) in the output of the accelerometer 22 from that camera 4 that is greater than a predetermined threshold. The variation may be caused, for example, by a user of the system lightly tapping on the camera 4.

[0024] Alternatively or additionally, the processing unit 6 may be configured to select one of the cameras 4 if the output of the accelerometer 22 from that camera 4 matches a predefined signature output. The predefined signature output may correspond to an output from a user tapping the camera with a finger, or any other manipulation of the camera by the user that does not interfere with the mounting of the camera 4. Thus, in general, the processing unit 6 is configured to detect a relatively light user interaction with the camera 4, such as a tap (single or multiple taps, such as double taps) or other similar light interaction. The predefined signature output may be determined by recording the response of the camera accelerometer 22 to one or more types of expected interactions. For example, the response to various single finger taps can be recorded, and the average response can be used as the predefined signature output. When a user taps a camera 4 in use, the response of the accelerometer 22 can be compared to the recorded average response, and if the two responses are sufficiently similar, a match between the output from the accelerometer and the predefined signature output can be inferred. Various aspects of the accelerometer output can be used for comparison, including, for example, the duration of the fluctuations in the output, the amplitude of the fluctuations in the output, or the change in the amplitude of the fluctuations over time. Using the change in the amplitude of the fluctuations over time may be particularly appropriate, for example, when the user interaction takes a more complex form, such as a double tap. In this case, it may be appropriate, for example, to detect pairs of local maxima and / or time separations of maxima in the fluctuations in the accelerometer output 22 to determine whether the user interaction matches a predetermined signature output of a double tap.

[0025] In one embodiment, the processing unit 6 is configured to react differently depending on the direction of the variation in the output from the accelerometer 22. For example, the processing unit 6 is configured to perform a first direction selection process when the camera 4 is tapped on one side (thus in a first direction, e.g., in an inward direction substantially perpendicular to the first side) and to perform a second direction selection process different from the first direction selection process when the camera 4 is tapped on a second side (thus in a second direction different from the first direction, e.g., in an inward direction substantially perpendicular to the second side). For example, the first direction selection process may include displaying image data captured by the camera and the second direction selection process may include displaying configuration data of the camera.

[0026] FIG. 2 illustrates how four cameras 4 of a motion capture system 2 may be configured to point toward a target area 16. The placement of four cameras 4 is exemplary. Fewer or more than four cameras 4 may be placed, as desired. Typically, each of the cameras 4 is a physically separate device. Typically, each of the cameras 4 is mountable independently of the other cameras 4. In the illustrated example, the cameras 4 are mounted on separate mounting supports 14.

[0027] The cameras 4 can be configured to interact with the processing unit 6 in a variety of ways. For example, wires may be provided between each of the cameras 4 and the processing unit 6. Alternatively or additionally, wireless data transfer between the cameras 4 and the processing unit 6 is possible.

[0028] In the above embodiment, the accelerometer 22 in the camera 4 is used to provide a convenient way for a user to select which camera is in close proximity and / or to select functions associated with that camera, however, the accelerometer 22 may be used for other purposes in addition to or as an alternative to the above functions.

[0029] For example, in one embodiment, the processing unit 6 is configured to detect a change in the orientation of one or more of the cameras 4 based on the output from the accelerometer 22. This may be achieved by detecting a change in the direction of gravity relative to the camera 4. This is shown diagrammatically in a side view in FIG. 3. An arrow 26 indicates the direction of gravity relative to the accelerometer 22. Gravity acts as an acceleration and is detected by the accelerometer 22. The accelerometer 22 can detect the direction of gravity 26 relative to a reference axis 28 of the camera 4. In the example shown in FIG. 3, this allows the accelerometer 22 to determine the tilt angle 24 of the camera 4 relative to the vertical. Typically, when setting up the motion capture system 2, one or more of the cameras 4 are fixedly mounted and calibrated before the motion capture system is used to capture movement of the target area 16. Any deviation in the position or orientation of the camera 4 after the calibration process is completed will result in a degradation of the image capture process. Such misalignment may occur, for example, by tapping the camera 4 or mounting 14, by temperature changes and the associated thermal expansion / contraction of the camera 4 or mounting 14, or by flexing or bending of the mounting 14. The inventors have recognized that such a wide range of misalignments includes changes in the orientation of the camera 4 relative to the direction of gravity, which may be detected by the accelerometer 22. Thus, the accelerometer 22 provides a convenient and accurate way for the processor 6 to detect when a misalignment has occurred in the camera mounting. Even for changes that occur slowly over time, such as when the mounting gradually tilts or shifts due to temperature changes over time, the accelerometer 22 can detect the change in orientation by comparing an absolute measurement of the orientation of the camera 4 with a previous absolute record of the orientation of the same camera 4. For example, if the system is left overnight, the output from the accelerometer 22 in the morning can be compared to the output recorded from the previous day. A significant variation (e.g., a variation greater than a predetermined threshold) indicates that the camera position has changed significantly and therefore that the mounting needs to be checked and / or recalibrated.

[0030] In the above embodiment, each of the cameras 4 is provided with an accelerometer 22. In these and other embodiments, each of the cameras 4 may alternatively or additionally be provided with one or more other sensors that provide useful information regarding the state of the camera.

[0031] In one embodiment, each of the cameras 4 includes a temperature sensor 18. The temperature sensor 18 may be provided instead of or in addition to the accelerometer 22. The processing device 6 is configured to perform processing dependent on the output from the temperature sensor 18 in such an embodiment. In one embodiment, the processing device 6 is configured to provide a visual indication of the temperature of one or more of the cameras 4. The visual indication may be provided, for example, on the display 12 of the processing device 6. Alternatively or additionally, a visual indication of the temperature may be provided for each individual camera 4. In one embodiment, the temperature sensor 18 of each camera 4 is mounted on a computer processing hardware 20, such as a PCB, within the camera 4. Thus, the temperature sensor 18 in such an embodiment is configured to measure the temperature of the computer processing hardware 20. In general, when the camera 4 is powered on, power is provided to the computer processing hardware 20 within the camera 4, which causes the hardware 20 to heat up. This heating may result in changes in the operating characteristics of the computer processing hardware 20 and / or changes in the characteristics of other components of the camera. For example, heat may be conducted from the computer processing hardware 20 to optical or mechanical components of the camera, such as between the lens of the camera 4 and an image capture device configured to convert light received by the pixels of the lens into electrical signals representative of the captured image. By measuring the temperature of the computer processing hardware 20, for example, it is possible to detect when the temperature of the hardware 20 has reached a steady state. At this point, further changes in the characteristics of the camera 4 due to heat will be very small. Thus, the user can confidently start the calibration procedure at this point, since subsequent temperature changes are unlikely to cause a significant deviation from the calibrated state. Compared to alternative approaches where the user simply waits for a predetermined type selected to be longer than any temperature stabilization period, the approach of the present embodiment allows the user to start the calibration procedure earlier and / or at a more reliable point in the steady state temperature region.

Claims

1. a plurality of cameras fixedly mounted such that each camera is directed toward a target scene; a processing unit configured to perform processing using image data from each of the plurality of cameras; the plurality of cameras are adjusted to capture movement of the target scene; each of the cameras includes an accelerometer, the processor being configured to perform processing dependent on an output from the accelerometer; The process comprises: detecting a change in any orientation of the camera based on output from the accelerometer by detecting a change in the direction of gravity relative to the camera; if the detected change in camera orientation is greater than a predetermined threshold, indicating that the camera position has changed significantly to prompt a recalibration procedure; the recalibration procedure includes monitoring the image data captured by each of the plurality of cameras and individually verifying and adjusting each of the plurality of cameras based on the image data. Motion capture system.

2. The processing device includes: selecting one of the cameras based on an output from the accelerometer; and configured to perform selected camera actions depending on which of the cameras is selected; The system of claim 1 .

3. and processing the selected camera includes displaying data from the selected camera. The system of claim 2.

4. the selected camera processing includes displaying data from a plurality of the cameras and providing visual information indicating which of the displayed data corresponds to the selected camera. The system of claim 3.

5. the selected camera processing includes displaying data from the selected camera and not displaying data from other cameras. The system of claim 3.

6. the displayed data includes image data captured by the selected camera; A system according to any one of claims 2 to 5.

7. the displayed data includes configuration data for the selected camera; A system according to any one of claims 2 to 6.

8. the selected camera processing includes providing a user with a selected camera interface that enables the user to control the selected camera. A system according to any one of claims 2 to 7.

9. the selected camera interface permits control of a plurality of the cameras and provides a visual indication of which of the cameras is the selected camera. The system of claim 8.

10. the selected camera interface allows control of the selected camera and not other cameras; The system of claim 8.

11. the processing unit is configured to select one of the cameras in response to a variation in the accelerometer output from that camera greater than a predetermined threshold. A system according to any one of claims 2 to 10.

12. the processing unit is configured to select one of the cameras if the accelerometer output from that camera matches a predetermined signature output. A system according to any one of claims 2 to 11.

13. the predetermined signature output corresponds to an output from a user tapping a finger on the camera; The system of claim 12.

14. the predetermined signature output corresponds to an output from a user tapping the camera a predetermined number of times; The system according to claim 12 or claim 13.

15. each of said cameras being a physically separate device and mountable independently of said other cameras; A system according to any one of claims 1 to 14.

16. the processing unit is provided in a portable computing device configured to wirelessly communicate with each of the cameras; A system according to any one of claims 1 to 15.

17. the processing device is configured to perform a first direction selection process associated with the camera when a change in acceleration in a first direction is detected at the camera, and to perform a second direction selection process associated with the camera when a change in acceleration in a second direction is detected at the camera, the first and second directions being different from one another, and the first and second direction selection processes being different from one another; A system according to any preceding claim.

18. 1. A method of performing motion capture using multiple cameras, each of which is equipped with an accelerometer and fixedly mounted so as to be pointed at a target scene; the plurality of cameras are adjusted to capture movement of the target scene; The method includes performing, on a processor, processing dependent on an output from the accelerometer, The process comprises: detecting a change in any orientation of the camera based on output from the accelerometer by detecting a change in the direction of gravity relative to the camera; if the detected change in camera orientation is greater than a predetermined threshold, indicating that the camera position has changed significantly to prompt a recalibration procedure; the recalibration procedure includes monitoring the image data captured by each of the plurality of cameras and individually verifying and adjusting each of the plurality of cameras based on the image data. method.

19. The process comprises: selecting one of the cameras based on an output from the accelerometer; and performing selected camera actions in response to which of the cameras is selected.

20. The method of claim 18.

20. and processing the selected camera includes displaying data from the selected camera.

20. The method of claim 19.

21. the selected camera processing includes displaying data from a plurality of the cameras and providing visual information indicating which of the displayed data corresponds to the selected camera.

21. The method of claim 20.

22. the selected camera processing includes displaying data from the selected camera and not displaying data from other cameras.

21. The method of claim 20.

23. the displayed data includes image data captured by the selected camera; 23. A method according to any one of claims 20 to 22.

24. the displayed data includes configuration data for the selected camera; 24. A method according to any one of claims 20 to 23.

25. the selected camera processing includes providing a user with a selected camera interface that enables the user to control the selected camera.

25. A method according to any one of claims 19 to 24.

26. the selected camera interface permits control of a plurality of the cameras and provides a visual indication of which of the cameras is the selected camera.

26. The method of claim 25.

27. the selected camera interface allows control of the selected camera and not other cameras; 26. The method of claim 25.

28. the processing includes selecting one of the cameras in response to a variation in the accelerometer output from the camera greater than a predetermined threshold.

28. A method according to any one of claims 19 to 27.

29. the process includes selecting one of the cameras if the accelerometer output from that camera matches a predetermined signature output; 29. A method according to any one of claims 19 to 28.

30. the predetermined signature output corresponds to an output from a user tapping a finger on the camera; 30. The method of claim 29.

31. the predetermined signature output corresponds to an output from a user tapping the camera a predetermined number of times; A method according to claim 29 or claim 30.

32. and using the detected change in camera orientation to detect a misalignment of the mounting of the camera.

32. A method according to any one of claims 18 to 31.

33. the processing includes performing a first direction selection processing associated with the camera when a change in acceleration in a first direction is detected in the camera, and performing a second direction selection processing associated with the camera when a change in acceleration in a second direction is detected in the camera, the first and second directions being different from one another, and the first and second direction selection processing being different from one another.

33. A method according to any one of claims 18 to 32.

Images