An actuator control method and system

Abstract

Generating a control signal for an actuator comprises: receiving 100 motion capture data corresponding to movement performed by a user; identifying 110, from the motion capture data, a control movemen

Description

FIELD OF DISCLOSURE The present invention relates to an actuator control method and system for controlling an actuator by motion capture. More specifically, the actuator control method and system is such that control of an actuator is based on a control movement performed by at least one user. BACKGROUND OF DISCLOSURE Motion capture involves recording the movement of people. It is known for use in virtual reality (VR) and augmented reality (AR) systems. VR involves generating a virtual environment with which a user can seemingly interact. AR involves enhancing the real world through the superimposition of computer generated images, for example, on a user’s real-world view. Examples of some known motion capture systems and related technologies are discussed below. OptiTrack (registered trade mark) provides motion capture systems primarily used in research, animation, robotics, and movement sciences. Their systems typically include a set of infrared cameras, which detect markers placed on the user's body, and software that processes and interprets the data from these cameras to generate a 3D model of the user's movements. Several other technologies also exist for motion capture, such as Vicon's (registered trade mark) system which operates similarly to OptiTrack's system using infrared markers. Xsens (registered trade mark), on the other hand, offers an inertial system where sensors are attached to the body to measure acceleration, and rotation to capture motion. These systems also use software to interpret the captured data. UltraLeap's (registered trade mark) Leap Motion system provides detailed hand and finger tracking by using a combination of infrared LEDs and cameras to detect and map hand and finger positions in three dimensions. The device is often used in virtual reality (VR) / augmented reality (AR) applications to provide natural interaction without the need for handheld controllers. A further form of motion capture uses stereo cameras and advanced machine learning techniques to reconstruct three-dimensional motion from two-dimensional image data. One example is the ZED Stereo Camera by Stereolabs (registered trade mark). The ZED stereo camera is a passive-stereo camera system that uses two sensors to capture image data, like a human pair of eyes. The captured information is processed using an artificial intelligence (Al) algorithm to create a depth map of an imaged scene, thus enabling 3D motion capture. The system includes a software development kit that uses Al to enhance the depth and motion sensing capabilities, providing 6-DoF (degree of freedom) tracking. Similar systems include the Intel RealSense Stereo Cameras (registered trade mark). Intel's RealSense technology uses stereo vision to calculate depth and enable functions like gesture tracking, object recognition, and motion tracking. The system consists of two IR (Infrared) cameras, an IR projector, and an RGB camera. The depth is calculated by comparing the offset of pixels between the two IR cameras. Microsoft Kinect Azure (registered trade mark), on the other hand, is a time-of-flight camera system using a single IR projector and a monochrome CMOS (complementary metal-oxide-semiconductor) sensor. While not technically a stereo vision system, the principles of depth perception are similar. The device captures reflected IR light and calculates depth by measuring the time it takes for the light to travel to the object and back. The depth image and RGB image are then processed by Al algorithms to perform skeletal tracking, gesture recognition, and other forms of motion capture. Furthermore, DeepMotion (registered trade mark) and Wrnch (registered trade mark) use Al-powered algorithms to capture and analyse human motion using single or multiple cameras, without the need for suits or markers. Their machine learning models are trained on vast amounts of data to recognize and interpret human movement in real-time. SUMMARY OF INVENTION Known motion capture systems, such as those discussed above, are successful in defining the position of users within a space for VR and AR systems. However, their use in real-world entertainment environments is limited. In particular, their use in entertainment productions is limited. For example, in concerts, museums, amusement parks, exhibitions, music festivals and theatre productions. This is due to the limited interaction in existing systems between captured motion and real-world objects. In particular, real world objects such as stage lighting and motors which are of importance in the entertainment production industry. In other words, existing motion capture systems, such as those described above, are designed primarily for capturing and recording motion, not for real-time control of real-world objects based on the captured motion data. This limits the extent of audience / user interaction possible in real-world entertainment applications which rely on the control of real-world objects. The inventor has appreciated that existing motion capture systems are unsuitable for enabling interaction with real-world devices in an entertainment industry setting. Aspects of the present invention are defined by the independent claims below to which reference should now be made. Optional features are defined by the dependent claims. According to one aspect of the present invention, there is provided an actuator control method for controlling an actuator by motion capture, the method comprising: receiving motion capture data corresponding to movement performed by at least one user; identifying, from the motion capture data, a control movement performed by the at least one user; generating a control signal corresponding to the identified control movement performed by the at least one user; and outputting the control signal to an actuator, wherein the control signal is configured to control the actuator such that control of the actuator is based on the control movement performed by the at least one user. In one example, the method further comprises receiving the control signal at the actuator. In one example, the control signal is based on a standardised protocol. In one example, wherein the standardised protocol is DMX512. In one example, the method further comprises generating the motion capture data. In one example, generating the motion capture data comprises imaging a region of interest, the region of interest corresponding to an expected location of the at least one user so as to capture a sequence of images of the user. In one example, generating the motion capture data further comprises generating a time dependent representation in a three dimensional coordinate space of the at least one user’s body from the sequence of images so as to capture movements performed by the user. In one example, movements performed by the at least one user comprise changes in the time dependent representation of the at least one user’s body. In one example, generating the motion capture data further comprises selecting one or more nodes of interest in the time dependent representation of the at least one user’s body. In one example, identifying the control movement performed by the at least one user comprises comparing changes in position of the one or more nodes of interest to one or more possible control movements. In one example, the method further comprises linking each of the one or more possible control movements to a corresponding control signal. In one example, generating the motion capture data is in real time. According to another aspect of the present invention, there is provided an actuator control system for controlling an actuator by motion capture, the system comprising: a controller configured to receive motion capture data corresponding to movement performed by at least one user, identify from the motion capture data a control movement performed by the at least one user, generate a control signal corresponding to the control movement performed by the at least one user; and output the control signal to an actuator, the control signal being configured to control the actuator such that control of the actuator is based on the control movement performed by the at least one user. In one example, the system further comprises an actuator configured to receive the control signal. In one example, the motion capture data comprises a sequence of images of the at least one user. In one example, the system further comprises an imager configured to image a region of interest, the region of interest corresponding to an expected location of the at least one user so as to capture the sequence of images of the at least one user. In one example, the imager comprises a stereo camera. In one example, the system further comprises a processor configured to receive the sequence of images and generate the motion capture data from the received sequence of images. In one example, the actuator is a digital actuator. In one example, the actuator is a physical actuator. In one example, the control signal comprises an on-off command. In one example, the control signal is configured to control a plurality of actuators. In one example, the system further comprises a store configured to store one or more possible control movements and their corresponding control signals. According to one aspect of the present disclosure, there is provided a computer program for executing the actuator control method. According to one aspect of the present disclosure, there is provided a non-transitory computer-readable medium on which are encoded instructions for carrying out the actuator control method. According to one aspect of the present disclosure, there is provided a computer-readable medium on which are encoded instructions for carrying out the actuator control method. BRIEF DESCRIPTION OF FIGURES The invention will be described in more detail by way of example with reference to the accompanying drawings, in which: Figure 1 is a schematic diagram of an actuator control system according to aspects of the present disclosure; Figure 2 is a schematic diagram demonstrating generating motion capture data as part of the actuator control system according to aspects the present disclosure; and Figure 3 is a flowchart describing an actuator control system according to aspects of the present disclosure. DETAILED DESCRIPTION An actuator control system 10 according to aspects of the present disclosure is shown in Figure 1. The actuator control system 10 comprises an imager 20. The imager 20 is one or more cameras. More specifically, the imager 20 is one or more stereo cameras. In this example, the stereo cameras 20 are ZED Stereo Cameras by Stereo Labs (registered trade mark). The stereo cameras 20 are configured to image a region of interest 25. The region of interest 25 corresponds to an expected location of at least one user 30. Where there are multiple cameras 20, they are strategically positioned to ensure full coverage of the region of interest 25 without occlusion. In this way, the stereo cameras 20 are configured to capture image data. More specifically, the stereo cameras are configured to capture a sequence of images of the user 30 (in other words, a video of the user). The image data captures the one or more user’s 30 body position in three dimensions. The stereo cameras 20 are connected to an image processor 40. The stereo cameras 20 are configured to continuously capture the region of interest 25. The continuously captured image data is then output to the image processor 40. In other words, image data is output to the image processor 40 in real time. In this example, the image processor 40 comprises a computer. The image processor 40 is configured to receive the image data captured by the stereo cameras 20 and process it to generate motion capture data. In this way, motion capture data is generated in real time. The motion capture data captures movement performed by the at least one user 30. In other words, the motion capture data comprises a time dependent representation of the one or more users’ 30 body positioning. This captures positions, movement, poses, gestures and / or actions performed by the one or more users 30. More specifically, the motion capture data captures how the positioning of the one or more users 30 changes over time. The image processor is configured to use Al and machine learning techniques to process the image capture data, as is known and discussed above. In this example, the image processor uses software configured for use with the ZED Stereo Camera. The process of generating the motion capture data is illustrated in Figure 2. On receiving the image data from the stereo cameras 20, the image processor 40 is configured to generate a representation 50 of the at least one user’s 30 body. In this example, the representation 50 is time dependent. This time dependent representation 50 exists in a three dimensional coordinate space. In this example, the time dependent representation 50 comprises a skeleton 60 overlaid on each user in the field of view of the cameras. Each skeleton 60 comprises a series of intersecting lines following the profile of the corresponding user’s body. The skeleton further comprises nodes 70 at points of interest. In this example, points of interest 70 represent joints and points of the human body that are key indicators of movement. Each node 70 is assigned cartesian coordinates relative to a zero point in the three dimensional coordinate space. In this example, the system 10 further comprises a user interface (not shown in Figures). The skeletons 60 are viewable on said user interface. In this example, each skeleton 60 is colour coordinated. More specifically, when there is more than one user in the region of interest and, therefore, more than one skeleton 60 generated, each skeleton 60 is displayed in a different colour. This enables an operator / user to distinguish between skeletons 60. Referring back to the system 10 of Figure 1, the image processor 40 is connected to a controller 80. The controller 80 is configured to receive the generated motion capture data. From the generated motion capture data, the image processor 40 is configured to identify a control movement performed by the at least one user 30. A control movement is a movement, gesture, action or pose, for example, performed by the user 30. Possible control movements include simple movements, sequences of movements, and / or the actions of multiple people. Examples of control movements include pointing or putting hands over the head. Each control movement should trigger a corresponding real-world event. The real-world event is, for example, turning on a light or starting a motor. The real-world event is associated with an actuator 90 configured to trigger the event in question. The controller 80 is connected to the actuator 90 and is configured to control the actuator 90. As a result, the controller 80 is able to use the generated motion capture data to identify control movements performed by the at least one user 30 and trigger the appropriate real world event by way of the actuator 90. The way in which this is done is significant and is described in more detail below. The way in which the controller 80 uses the generated motion capture data to trigger real-world events is described by the flowchart of Figure 3. At step 100, the controller 80 receives the generated motion capture data. In this example, the motion capture data is that generated by the image processor 40 from the image data captured by the stereo cameras 20. In other words, the received motion capture data comprises the skeleton 60 representation of the one of more users 30. The motion capture data further comprises the cartesian coordinates associated with each node 70. At step 110, the controller is then configured to identify whether a control movement has been performed by the user 30. To do this, the controller 80 uses pattern recognition to identify a control movement. In this example, said pattern recognition uses the positioning and movement of the nodes 70 to identify control movements. The speed and / or direction of the movement of the nodes is also taken into consideration. In other words, the controller 80 analyses the position and movement of each node 70 to detect control movements. The control movements are predefined. Each control movement is further associated with a corresponding real-world event. Referring back to the system of Figure 1, the controller is further coupled to a store 120. The store 120 is configured to store one or more possible control movements. In this example, the one or more possible control movements include pre-defined control movements. A user is able to define further control movements. The store 120 is further configured to store the associated real-world event. In this example, the associated real-world event is selectable by the user. In this way, a database of control movements and associated real-world events is stored. Each real-world event has an associated control signal. Said control signal is configured to control an actuator in such a way that the real-world event is triggered. The data is retrievable by the connected controller 80. In other words, the system is configurable such that a real-world event can be initiated from any detectable movement defined as a control movement. At step 130, on identifying a control movement, the controller 80 generates a control signal. The control signal is configured to be output to an actuator 90 (see Figure 1). More specifically, the control signal is configured to be output to the actuator 90 that will trigger the real world event with which the identified control signal is associated. Furthermore, the control signal is configured to control the actuator 90. In this example, possible control signals include a simple on-off command. Other control signals comprise a variable output. For example, between 0 and 10 VDC. Other control signals are a complex sequence controlling a plurality of actuators. Some control signals are digital and, therefore, configured to be output to a digital actuator. Other control signals are physical signals and are configured to be output to a physical, or real-world, actuator. At step 140, the generated control signal is sent to the actuator 90. The configuration of the control signal is significant. The control signal is based on a standardised protocol. More specifically, the control signal is based on a standardised protocol used in the entertainment industry for the control of real-world devices. In this example, the control signal takes the form of a digital multiplex (DMX) signal. More specifically, DMX512. DMX512 is a known standard for digital communication networks commonly used to control stage lighting and effects. It enables the individual control of actuators / devices. It operates over a serial connection with 5-pin XLR connectors, controlling up to 512 channels per universe. Each channel controls a specific attribute of a light or effect. For example, brightness or colour. Each channel can be set to a value between 0 and 255. Data is transmitted in binary format. The order of packets determines the receiving devices' address. It will be appreciated that in other examples, the control signals will rely on other standards for digital communications for the entertainment industry. For example, MIDI, TIMECODE, ANALOGUE, MODBUS or ETHERCAT. Once output by the controller 80 , the control signal is received by the one or more actuators 90. This triggers the real-world event. As a result, the real world event is triggered by the controller 80 identifying the corresponding control movement performed by the at least one user 30. In other words, control of the actuator is based on the motion of the user 30. Actuators 90 are any device that can be controlled via digital or physical signals. More specifically, the actuators 90 correspond to devices that are commonplace in the entertainment industry. For example, fans, lights, heaters, video content display systems and sound systems. In summary, when a user moves within the camera's field of view, their movement is translated into a digital model, interpreted by the controller, and used to trigger predefined actions on a connected device. This allows users to directly interact with and control their surroundings simply by moving their bodies. By way of example, a user points at a light bulb (the control movement). On recognition of this gesture, a control signal is generated and output to the lightbulb. The control signal is configured to increase the brightness of the light until it achieves full illumination (the real world event). By way of further example, a group of users put their hands in the air (the control movement). On recognition of this gesture, a control signal is generated and output to an actuator. The signal is received at the actuator and a real-world event triggered. For example, confetti is released, a ride is started, video content is initiated, or a light is turned on. This creates an immersive and interactive environment for entertainment and amusement. The above described actuator control method provides has several advantages over existing technology. As explained above, existing motion capture systems are of limited use in the entertainment industry due to a lack of interaction between the captured user motion and real-world effects. The above described system and method addresses this deficiency by providing a mechanism to initiate real-world actions (like turning on a light or starting a motor) in response to specific movements or positions detected by the motion capture system. A direct link between human movement and the control of such devices in the environment is enabled. This greatly expands the range of potential applications and user engagement, particularly in the entertainment industries. As discussed above, the system is configurable to allow any control output required based on any user movement. This is the result of relying on standardised communication protocols that are used for the control of real-world objects in the entertainment industry. This makes the system highly versatile. Furthermore, the selection of an Al-enabled stereo camera system for collecting the motion capture data is not arbitrary. Many of the existing motion capture systems discussed above are complex and, therefore, high-end, limiting their widespread usage in the entertainment and amusement industry where budgets may be more constrained. As a result of their complexity, they typically require specialized knowledge and training to set up and operate, which can be a barrier for smaller companies or individual users. They also often require a dedicated space and specific conditions (like special markers or lighting) to function properly, making them unsuitable for impromptu or spontaneous audience interactions in various settings. Using Al and machine learning to enhance the capabilities of stereo camera systems for motion capture is a more efficient, less complex and user-friendly system that is of increased suitability for use in the entertainment industry. The system simplifies the motion capture process by leveraging stereo cameras that can operate in a wide range of environments without the need for special markers or lighting conditions, making the technology more accessible and user-friendly. Embodiments of the present invention have been described with particular reference to the examples illustrated. However, it will be appreciated that variations and modifications may be made to the examples described within the scope of the present invention. As will be appreciated by one of skill in the art, the invention described herein may be embodied in whole or in part as a method, a data processing system, or a computer program product including computer readable instructions. Accordingly, the invention may take the form of an entirely hardware embodiment or an embodiment combining software, hardware and any other suitable approach or apparatus. The computer readable program instructions may be stored on a non-transitory, tangible computer readable medium. The computer readable storage medium may include one or more of an electronic storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or Flash memory), a static random access memory (SRAM), a portable compact disc read-only memory (CD-ROM), a digital versatile disk (DVD), a memory stick, a floppy disk. Exemplary embodiments of the invention may be implemented as a circuit board which may include a CPU, a bus, RAM, flash memory, one or more ports for operation of connected I / O apparatus such as printers, display, keypads, sensors and cameras, ROM, a 5 communications sub-system such as a modem, and communications media. As will be appreciated by one of skill in the art, the term “processing means” may correspond to any suitable processing device. For example, the processing means may be any of a computer processor, graphics processor, programmable logic device, 10 microprocessor, or any other suitable device. It will be appreciated that the processing means may comprise a plurality of processing devices, and may be a combination of different processing devices such as those described above.

Claims

1. An actuator control method for controlling an actuator by motion capture, the method comprising:receiving motion capture data corresponding to movement performed by at least one user;identifying, from the motion capture data, a control movement performed by the at least one user;generating a control signal corresponding to the identified control movement performed by the at least one user; andoutputting the control signal to an actuator, wherein the control signal is configured to control the actuator such that control of the actuator is based on the control movement performed by the at least one user.

2. The method of claim 1 further comprising receiving the control signal at the actuator.

3. The method of any preceding claim wherein the control signal is based on a standardised protocol.

4. The method of claim 3 wherein the standardised protocol is DMX512.

5. The method of any preceding claim further comprising generating the motion capture data.

6. The method of claim 5 wherein generating the motion capture data comprises imaging a region of interest, the region of interest corresponding to an expected location of the at least one user so as to capture a sequence of images of the user.

7. The method of claim 6 wherein generating the motion capture data further comprises generating a time dependent representation in a three dimensional coordinate space of the at least one user’s body from the sequence of images so as to capture movements performed by the user.

8. The method of claim 7 wherein movements performed by the at least one user comprise changes in the time dependent representation of the at least one user’s body.

9. The method of any of claims 7 to 8 wherein generating the motion capture data further comprises selecting one or more nodes of interest in the time dependent representation of the at least one user’s body.

10. The method of claim 9 wherein identifying the control movement performed by the at least one user comprises comparing changes in position of the one or more nodes of interest to one or more possible control movements.

11. The method of claim 10 further comprising linking each of the one or more possible control movements to a corresponding control signal.

12. The method of any of claims 5 to 11 wherein generating the motion capture data is in real time.

13. An actuator control system for controlling an actuator by motion capture, the system comprising:a controller configured to receive motion capture data corresponding to movement performed by at least one user, identify from the motion capture data a control movement performed by the at least one user, generate a control signal corresponding to the control movement performed by the at least one user; and output the control signal to an actuator, the control signal being configured to control the actuator such that control of the actuator is based on the control movement performed by the at least one user.

14. The system of claim 13 further comprising an actuator configured to receive the control signal.

15. The system of claims 13 or 14 wherein the motion capture data comprises a sequence of images of the at least one user.

16. The system of claim 15 further comprising an imager configured to image a region of interest, the region of interest corresponding to an expected location of the at least one user so as to capture the sequence of images of the at least one user.

17. The system of claim 16 wherein the imager comprises a stereo camera.

18. The system of any of claims 15 to 17 further comprising a processor configured to receive the sequence of images and generate the motion capture data from the received sequence of images.

19. The system of any of claims 13 to 18 wherein the actuator is a digital actuator.5 20. The system of any of claims 13 to 18 wherein the actuator is a physical actuator.

21. The system of any of claims 13 to 20 wherein the control signal comprises an on-offcommand.

22. The system of any of claims 13 to 20 wherein the control signal is configured to control a plurality of actuators.10 23. The system of any of claims 13 to 22 further comprising a store configured to store oneor more possible control movements and their corresponding control signals.

24. A computer program for executing the method of claims 1 to 12.

25. A non-transitory computer-readable medium on which are encoded instructions for carrying out the method of claims 1 to 12.15 26. A computer-readable medium on which are encoded instructions for carrying out themethod of claims 1 to 12.

Images