Real-time motion monitoring control system, device and method for moving objects

Abstract

A real-time motion monitoring control system (100), device and method for moving objects. The control system (100) comprises: a multi-dimensional treadmill (110); a first camera array (120); a second camera array (140); a processing apparatus, used for receiving first images captured by the first camera array (120) and a plurality of second images captured by the second camera array (140), reconstructing a three-dimensional model of a target moving object, calculating a real-time position of the target moving object in the field of view of cameras, controlling the movement of the multi-dimensional treadmill (110) to keep the target moving object at a fixed position relative to the cameras, and detecting key points of the target moving object in the first images; a scene generation apparatus (130); and a laser stimulation assembly (150).

Description

Real-time motion monitoring control system, device and method for moving objects Technical Field

[0001] The present application relates to the field of animal behavior and machine vision technology, and more specifically to a real-time motion monitoring control system, device, and method for a moving object. Background Art

[0002] In order to more precisely measure the behavior of moving objects (such as small animals or small machines) in free-moving situations, it is necessary to perform precise three-dimensional spatial position detection of key points on their body parts, and therefore it is necessary to use multiple cameras to capture high-resolution images from different angles. When the moving object is in motion, in order to capture the image, the camera needs a larger field of view, which will result in lower resolution, but if the mobile camera tracks the target, it will become too complicated when the number of cameras increases. In the prior art, the method of shooting the animal by fixing it on a suspended ball greatly restricts the animal's behavior and makes it impossible for the animal to interact with its peers in free-moving situations. Although some other methods can obtain high-resolution two-dimensional images of freely moving animals, they only use a single camera and cannot obtain three-dimensional spatial information.

[0003] In some studies, it is necessary to investigate and control the function of neural circuits by combining optogenetics with spatiotemporal optical stimulation of specific neurons. Currently, in order to achieve precise laser stimulation of unilateral brain regions, commonly used methods include immobilizing animals or using transgenic technology to randomly express unilateral neurons (such as the FLP technology in fruit flies). These methods restrict animal behavior or are time-consuming and labor-intensive.

[0004] In order to provide a virtual reality visual environment for the moving object, it is necessary to provide a ring screen around it that can precisely control the display content. The current method mainly uses a projector to project onto the ring screen or assembles flat display units in a ring. These methods have problems such as uneven brightness and obvious splicing edges. In addition, the system is large in size and complicated to install and debug.

[0005] Therefore, there is a need for a real-time motion monitoring and control system that can provide complete and real-time multi-perspective high-resolution behavioral records of multiple moving objects (such as small animals) in a free social environment, and can also provide precise virtual reality stimulation to the moving objects according to different scenarios.

[0006] Summary of the Invention

[0007] To address the above-mentioned issues, the present disclosure provides a real-time motion monitoring control system, device, and method for moving objects, which can track the movement of moving objects in real-time and closed-loop, and record high-resolution images from multiple angles of moving objects (such as animals, movable facilities, or robots) in free activities including complex social behaviors, and measure their behaviors. At the same time, it can combine technologies such as optogenetic stimulation and virtual reality environments to intervene and control them according to specific social scenarios.

[0008] In a first aspect, the present application discloses a real-time motion monitoring and control system for moving objects, comprising: a multi-dimensional treadmill configured to allow multiple moving objects to freely move thereon. The real-time motion monitoring and control system further comprises: a first camera array configured to capture first images of the multiple moving objects. The real-time motion monitoring and control system further comprises: a second camera array, wherein a plurality of cameras in the second camera array are arranged circumferentially around the multi-dimensional treadmill and capture multiple second images of the multiple moving objects synchronously in real time from different lateral directions. The real-time motion monitoring and control system further comprises: a processing device configured to: receive the first image from the first camera array and the plurality of second images from the second camera array; reconstruct a three-dimensional model of a target moving object among the multiple moving objects based on the first image and the plurality of second images; calculate the real-time position of the target moving object in the camera field of view based on the first image; control the movement of the multi-dimensional treadmill based on information regarding the real-time position of the target moving object so that the target moving object on the treadmill remains in a fixed position relative to the first and second camera arrays; and detect key points of the target moving object in the first image, wherein the key points correspond to parts of the moving object. The real-time motion monitoring and control system also includes: a scene generation device, which is arranged around the activity area of ​​the multiple moving objects on the multi-dimensional treadmill and is configured to generate a scene for the free movement under the control of the processing device; and a laser stimulation component, which is configured to perform laser stimulation on the key points of the target moving object in the scene under the control of the processing device.

[0009] In a second aspect, the present application discloses a real-time motion monitoring and control device for a moving object, comprising: a multi-dimensional treadmill configured to allow multiple moving objects to freely move thereon. The real-time motion monitoring and control device further comprises: a first camera array configured to capture first images of the multiple moving objects. The real-time motion monitoring and control device further comprises: a second camera array, wherein a plurality of cameras in the second camera array are arranged circumferentially around the multi-dimensional treadmill and capture multiple second images of the multiple moving objects synchronously in real time from different lateral directions. The real-time motion monitoring and control device further comprises: a processing device configured to: receive the first image from the first camera array and the plurality of second images from the second camera array; reconstruct a three-dimensional model of a target moving object among the multiple moving objects based on the first image and the plurality of second images; calculate the real-time position of the target moving object within the camera field of view based on the first image; control the movement of the multi-dimensional treadmill based on information regarding the real-time position of the target moving object so that the target moving object on the treadmill remains in a fixed position relative to the first and second camera arrays; and detect key points of the target moving object in the first image, wherein the key points correspond to parts of the moving object. The real-time motion monitoring and control device also includes: a scene generation device, which is arranged around the activity area of ​​the multiple moving objects on the multi-dimensional treadmill and is configured to generate a scene for the free movement under the control of the processing device; and a laser stimulation component, which is configured to perform laser stimulation on the key points of the target moving object in the scene under the control of the processing device.

[0010] In a third aspect, the real-time motion monitoring and control method for moving objects disclosed in the present application includes the following steps: using a first camera array to capture a first image of the multiple moving objects, and calculating the real-time position of the target moving object in the camera field of view based on the first image; based on information about the real-time position, controlling the movement of the multi-dimensional treadmill so that the target moving object on it remains in a fixed position relative to the first camera array; detecting key points of the target moving object in the first image, wherein the key points correspond to parts of the target moving object; generating a scene for the free movement of the multiple moving objects; using a second camera array to capture multiple second images of the target moving object synchronously in real time from different lateral orientations; reconstructing a three-dimensional model of the target moving object based on the first image and the multiple second images; and performing laser stimulation on the key points in the scene.

[0011] The real-time motion monitoring control system, device and method mainly use multiple cameras with different angles and resolutions to monitor and record multiple animals in social scenes in real time, and estimate and analyze the position of the center and specific body parts of the animals in real time and perform identity recognition, so as to drive the multi-dimensional motor closed loop to track specific target animals, thereby keeping them in a (dynamic) fixed position relative to the camera and the ground, and obtaining multi-perspective and high-resolution images of the animals, which can be used for subsequent three-dimensional key point detection. In addition, the laser position can be adjusted in real time by the laser stimulation component, and combined with optogenetics, specific neural activities can be manipulated in a spatiotemporal stimulation manner. In addition, controllable virtual stimulation or virtual peers can be displayed in real time through a flexible screen that surrounds nearly 360° to simulate specific life or social scenes. Using these components, the technical solution of the present disclosure is realized by integrating infrared light imaging, image processing, motion control, laser control, artificial intelligence, virtual reality and other technologies. BRIEF DESCRIPTION OF THE DRAWINGS

[0012] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the following will briefly introduce the drawings required for the embodiments or the description of the prior art. Obviously, the drawings described below are only some embodiments described in this application. When appropriate, the same reference numerals are used in all drawings to refer to the same or similar parts. Such embodiments are illustrative and are not intended to be exhaustive or exclusive embodiments of the present device or method. In the drawings:

[0013] FIG1 is a schematic structural diagram of a real-time motion monitoring and control system according to an embodiment of the present disclosure;

[0014] FIG2 is an enlarged view of area A in FIG1 ;

[0015] FIG3 is a schematic functional structure diagram of a real-time motion monitoring and control system according to an embodiment of the present disclosure;

[0016] FIG4 shows images captured by a first camera array and a second camera array according to an embodiment of the present disclosure;

[0017] FIG5 is an image captured by a second positioning camera according to an embodiment of the present disclosure, showing the center position and several key point positions of a moving object; and

[0018] FIG6 is a schematic flowchart of a real-time motion monitoring control method according to an embodiment of the present disclosure. DETAILED DESCRIPTION

[0019] In order to make the purpose, technical solutions and advantages of the embodiments of the present application clearer, the technical solutions of the embodiments of the present application will be clearly and completely described below in conjunction with the drawings of the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, not all of the embodiments. Based on the described embodiments of the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0020] Unless otherwise defined, the technical or scientific terms used in this application should have the usual meanings understood by people with ordinary skills in the field to which this application belongs. The "first", "second" and similar words used in this application do not indicate any order, quantity or importance, but are only used to distinguish different components. "Include" or "comprise" and similar words mean that the elements or objects appearing before the word include the elements or objects listed after the word and their equivalents, without excluding other elements or objects. "Connect" or "connected" and similar words are not limited to physical or mechanical connections, but may include electrical connections, whether direct or indirect. "Up", "down", "left", "right" and the like are only used to indicate relative position relationships. When the absolute position of the object being described changes, the relative position relationship may also change accordingly.

[0021] In order to keep the following description of the embodiments of the present application clear and concise, detailed descriptions of known functions and known components are omitted in this application.

[0022] The present disclosure is dedicated to being able to carry out more comprehensive behavioral analysis of moving objects, where the moving objects can be any one of animals (such as insects), movable facilities and robots. The animals here can include but are not limited to insects, small fish, small amphibians, small reptiles, etc., and robots can also include logistics robots, etc. For ease of description, in the embodiments provided below, the real-time motion monitoring control system and method of the present disclosure are described with the classic model organism Drosophila as the object. However, it should be understood that any other animals, movable facilities or robots that can move freely and interact with each other are also equally applicable to the system and method provided by the present disclosure.

[0023] As shown in Figure 3, the control software for this system was developed using LabVIEW, and image processing was performed using a C++ program. This allows for convenient system control and observation via a graphical user interface. The control software primarily includes synchronization signal generation and acquisition, data communication, 2D treadmill control, light stimulation control, and virtual reality control. Synchronization signals primarily include camera trigger signals and laser control signals; data communication primarily involves network communication with the image processing program.

[0024] Figure 1 shows a schematic block diagram of a real-time motion monitoring and control system 100 for a moving object (fruit fly) according to an embodiment of the present disclosure. As shown in Figure 1 , the real-time motion monitoring and control system 100 includes at least: a multi-dimensional treadmill 110, a first camera array 120, a scene generation device 130, a second camera array 140, a laser stimulation assembly 150, and a processing device (not shown).

[0025] In this embodiment, the multi-dimensional treadmill 110 can be configured to operate in multiple dimensions. As an example, the multi-dimensional treadmill 110 is configured as a two-dimensional treadmill 110, which includes two high-precision servo motors perpendicularly coupled to each other to minimize the displacement of the fruit fly relative to the first camera array 120 and the second camera array 140 (described in detail below) to obtain high-resolution images. A PID (Proportional-Integral-Derivative) control algorithm is used to close-loop control the movement of the servo motors to track and compensate for the movement of the fruit fly.

[0026] Here, as shown in FIG. 2 , a plurality of fruit flies may be placed in a container 170 to define the activity area, and the container 170 may be placed on a two-dimensional treadmill 110 .

[0027] The first camera array 120 is used to capture real-time first images of multiple fruit flies freely moving on the two-dimensional treadmill 110. Specifically, in Figure 1, the first camera array 120 is preferably arranged at the top of the multi-dimensional treadmill 110 and includes a first positioning camera 121, a second positioning camera 122, and a spectroscope 123. Correspondingly, the resolution of the images captured by the first positioning camera 121 is lower than the resolution of the images captured by the second positioning camera 122. By using a 25mm wide-band spectroscope 123, the first positioning camera 121 and the second positioning camera 122 can simultaneously capture images from above the treadmill. The first positioning camera 121 has a resolution of 640×480 pixels (first resolution), an exposure time of 1ms, and a recording frame rate of 200fps. In order to obtain the displacement of the two-dimensional treadmill without affecting system performance in order to calculate the absolute position of the fruit flies, the first positioning camera 121 uses a wide-angle lens with an 8mm focal length to ensure that the camera field of view can cover the entire activity area of ​​the fruit flies and the four landmarks surrounding the activity area. The second positioning camera 122 has a resolution of 1280×1024 pixels (second resolution), an exposure time of 1 ms, and a recording frame rate of 200 fps. To estimate the position and posture of the fruit fly with higher resolution, the second positioning camera 122 uses a 35 mm focal length lens and a 2x lens expander.

[0028] A first camera array 110 is used to capture a first image of multiple fruit flies in real time from the top of the two-dimensional treadmill 110. The first image here should include an image captured by the first positioning camera 121 and an image captured by the second positioning camera 122 (the former having a lower resolution than the latter). The first image is sent to a processing device, which calculates the real-time position of the multiple fruit flies within the camera's field of view and their absolute position in space in real time. Image recognition technology is used to identify a target fruit fly from the multiple fruit flies as the target for monitoring and control. The deviation between the target fruit fly's real-time position and a preset target position (e.g., the center of the camera's field of view) is then used as input to a PID controller. The position deviation is converted into a servo motor speed, which is then used to control the motor's movement, maintaining the target fruit fly in a (dynamically) fixed position relative to the first camera array 120, the second camera array 140, and the ground, thereby producing an effect similar to that of a fruit fly with its body fixed on a suspended ball.

[0029] In controlling the servo motor's motion, because the motor's maximum acceleration is small, the converted motor speed is limited by a square root controller (Formula (1)). Furthermore, when the position deviation is greater than 4 mm or less than 0.1 mm, the motor speed control is reduced to 1 / 5 to prevent interference with the fly's behavior due to rapid motor adjustments and to prevent jitter caused by the fly's own movement while maintaining the same position.

[0030] Where v is the output motor speed, K p is the frame rate of image processing, E Pos Is the absolute value of the position deviation output by the PID controller, A max is the maximum acceleration of the motor.

[0031] In terms of calculating the real-time position of fruit flies, the present disclosure provides a self-developed C++ program loaded on a processing device to realize low-latency position calculation (estimation). The target position calculation main program contains three threads running in parallel, including a video acquisition thread, a prediction thread, and a recording thread. The video acquisition thread controls the first camera array 120 to capture the original image and transmits the image to the other two threads. The recording thread writes the image to a video file and writes the timestamp to a log file. The video acquisition thread and the recording thread together constitute the MIAS program, which is also applicable to the second camera array 140. The prediction thread detects target body parts of, for example, multiple fruit flies, and communicates with the key point detection program to pass the position information to the two-dimensional treadmill control program. Here, as shown in Figure 5, the target body part here can be the center of the fruit fly's body (shown as "+" in the figure); however, other body parts of the fruit fly, such as the head and tail, can also be tracked as needed.

[0032] In the prediction thread, a threshold-based method is used to detect the body center of the fruit fly. Connected regions are detected within a binary image obtained through threshold segmentation. To eliminate interference such as shadows and noisy background, regions approximately the size of the fruit fly are selected as candidate regions. For multiple fruit flies, tracking targets can be selected based on body size, such as the smallest male fruit fly. Fly overlap can be further detected by determining whether the number of fruit fly regions suddenly decreases. If overlap is detected, the position of the last frame is used as the target position, minimizing rapid treadmill jumps. Four landmarks of varying sizes and shapes, invisible to the fruit flies, surround the fruit fly's activity area. These landmarks limit the detection area, and the absolute position of the fruit fly within the activity area can be calculated by comparing the landmarks' offset from the initial position.

[0033] In addition to detecting the center of the fruit fly, the high-resolution, small-field-of-view image captured by the second positioning camera 120 within the first image captured by the first camera array 120 also segments a 128×128 pixel region at the center of the fruit fly. This region serves as input to a trained neural network model (e.g., a U-Net model) for real-time detection of two key points of the fruit fly, such as the left and right compound eyes, as marked with "O" in FIG5 . It should be understood that in this technical field, various other methods besides neural network models can also be used to detect key point locations, such as image recognition technology.

[0034] Using the absolute positions and key point information of multiple fruit flies obtained, the orientation of each fruit fly and the relative positions between fruit flies can be calculated in real time, and then different social moments can be selected to perform laser stimulation and virtual reality control on the fruit flies as described below.

[0035] To more accurately measure the behavior of fruit flies in both free-range and complex social environments, the present disclosure also includes a scene generation device 130, as shown in Figures 1 and 2. This device is arranged around the fruit fly's activity area on a two-dimensional treadmill 110 to generate a scene for free-range movement. Because visual-based virtual reality technology can provide rich, varied, and controllable visual scenes, in an exemplary embodiment, scene generation device 130 is a screen, preferably an OLED flexible screen (resolution 2160×1080, refresh rate 60Hz), to generate a panoramic virtual reality scene. This screen is surrounded by a cylinder with a diameter of 44mm and a nearly 360-degree angle, within which the fruit flies can move freely within a container 170.

[0036] Preferably, the system 100 can call a function library for controlling the content displayed on the screen, and calculate a specific social scene in real time by obtaining the absolute positions and key point information of multiple fruit flies, and can choose to generate low-latency and controllable visual stimulation at a specific moment. At the same time, based on the three-dimensional reconstructed virtual fruit flies, the fruit flies can be given a more realistic virtual social scene. For example, assuming that there is a fruit fly in front of the left of the target fruit fly at the detection point, and it is judged based on the key point information that the two are facing each other, the processing device can generate another virtual fruit fly in front of it on the right, and measure the social behavior of the target fruit fly in this scene; or, the absolute position can indicate how far the target fruit fly has traveled and the movement trajectory, etc., and the processing device can generate different scenes according to the absolute position.

[0037] It should be understood that the scene generation device 130 can also generate scenes based on needs (such as operator experience or preferences), and does not necessarily need to be based on the absolute positions and key point information of multiple fruit flies.

[0038] In addition, the scene generating device 130 provided in the present disclosure may be other devices besides the OLED screen for generating a panoramic virtual reality scene described in the example, such as a device for providing a panoramic augmented reality (Augmented Reality) scene.

[0039] As described above, closed-loop control of the two-dimensional treadmill 110 maintains the target fruit fly at a fixed position or within a relatively small range relative to the first camera array 120, the second camera array 140, and the ground, creating an effect similar to that of a fixed fruit fly on a suspended ball. Therefore, this system allows for unilateral light stimulation of the fruit fly using the laser stimulation assembly 150 while the fruit fly is freely moving. Furthermore, by estimating the absolute positions of multiple fruit flies and detecting key points, light stimulation experiments can be performed on different parts of the fruit fly in different social situations to study specific neural activity.

[0040] As shown in FIG1 , the laser stimulation assembly 150 may include: a laser source 151 ; and a two-dimensional galvanometer 152 , which is configured to deflect and scan the laser from the laser source 151 to achieve controllable directional stimulation of the identified key points of the target fruit fly.

[0041] In order to conduct optogenetic activation experiments with minimal interference to the behavior of the target fruit fly, the present disclosure uses a red light semiconductor laser with a central wavelength of 638nm. In order to adjust the beam diameter of the laser, the present disclosure uses a 0.5-fold beam reduction mirror 153 and two adjustable aperture diaphragms 154 and 155 with a minimum aperture of 0.8mm to adjust the laser. The adjusted beam half-width (FWHM) diameter is approximately 0.6mm. At the same time, real-time laser control can be achieved through a controller, including switching, intensity and frequency modulation. Then, the adjusted laser is reflected by the reflector 156 into the two-dimensional galvanometer 152 for deflection scanning. The scanning laser is filtered and reflected by the dichroic mirror 157, and finally irradiated at the corresponding key point position of the target fruit fly to achieve laser stimulation. Since the fruit fly is relatively fixed in absolute space, the above-mentioned laser stimulation can be conveniently applied without the need to track the movement of the fruit fly significantly when applying the stimulation.

[0042] The present invention tracks and compensates for the motion of a target fruit fly on a horizontal plane by linearly adjusting a two-dimensional galvanometer 152. Control of the two-dimensional galvanometer 152 is an open-loop control method. After determining the coordinate transformation relationship between the two-dimensional galvanometer 152 and the two-dimensional treadmill 110, the target key point position for stimulation is selected based on a specific scenario calculated in real time. The error from the tracking center is calculated to linearly control the two-dimensional galvanometer 152. The two-dimensional galvanometer 152 can be controlled to move at a frame rate of 200Hz, guiding the laser to track the specific target key point position of the fruit fly in real time.

[0043] In order to calibrate the laser of the light stimulation system, this embodiment also uses a photodiode power meter placed at the position of the fruit fly container 170, and covers the container 170 with a Teflon-coated coverslip 171 to simulate the actual laser intensity reaching the fruit fly. By changing the driving voltage of the laser, a series of laser output intensities are generated. In addition, the optical power meter is fixed to a precision translation stage, and the detection target surface is partially blocked with opaque black tape. The laser beam reaching the optical power meter can be gradually blocked from unobstructed to completely blocked by a fixed micro-movement translation stage. This process measures a series of integrated laser intensities, and the intensity distribution of the laser spot can be obtained by calculating its derivative.

[0044] In addition, in this embodiment, other stimulation means besides laser stimulation can also be provided, such as physical and chemical stimulation of the target fruit fly through a physical and chemical stimulation component (not shown in the figure), such as sound, light, heat and smell stimulation, so as to further study the behavior of the fruit fly under different stimulation means.

[0045] Returning to Figure 1 , in addition to the top first camera array 120, this embodiment also utilizes a second camera array 140 comprised of multiple cameras (eight are shown in Figure 1 , for example). This second camera array 140 is positioned lateral to the two-dimensional treadmill 110 and can simultaneously capture multiple, high-resolution, second images of the target fruit fly from different lateral angles. These multiple second images, along with the first image provided by the first camera array 120, are processed to reconstruct a three-dimensional model of the target fruit fly, facilitating the measurement of the target fruit fly's animal behavior. This can also be used for training and detecting neural network models of the fruit fly's three-dimensional key points. These eight side-view cameras are evenly spaced approximately 45° apart on the side of the motion control platform of the two-dimensional treadmill 110, with an inclination angle of approximately 22 (±1)°. They have a resolution of 1280×1024 pixels and utilize a telecentric lens with a magnification of 0.7 and a working distance of 65 mm. FIG4 shows the first image and the second image captured simultaneously by the first camera array 120 and the second camera array 140 provided in this embodiment. The upper left corner shows the first image captured by the first positioning camera 121, the middle shows the first image captured by the second positioning camera 122, and the remaining images are second images captured by eight cameras at different lateral positions. The second images are slightly cropped to match the composite image, and the target position being tracked is the center position of the male fruit fly.

[0046] In addition, during the implementation of the above embodiment, a light source is also provided to provide illumination for multiple fruit flies on the two-dimensional treadmill, thereby improving the brightness and contrast of the recorded image. Preferably, the present embodiment evenly illuminates the activity area of ​​the fruit flies from the top and bottom. A customized infrared ring light source 161 (FIG. 1) with a wavelength of 850 nm and a square light source 162 (FIG. 1) are used at the top and bottom, respectively. At the same time, the fruit flies maintain a sufficient distance from the square light source 162 at the bottom to reduce behavioral changes caused by the heating of the light source. Because infrared light is invisible to fruit flies, in order to provide the necessary visual cues to the fruit flies, the present embodiment adds a ring light source 161 at the top, which can provide white light or blue light. At the same time, an infrared filter is added between all cameras and lenses to allow only infrared light to pass through, eliminating the influence of the fruit fly's visible light and excitation light on the image.

[0047] In a preferred embodiment, in addition to recording black and white images, the real-time motion monitoring system 100 disclosed herein can also be flexibly constructed and configured as needed, such as using a color camera and a white light source to record color images; the recording frame rate and resolution can also be flexibly selected according to the computer hardware configuration and requirements.

[0048] In addition, to ensure the time synchronization of all images captured by the top first camera array 120 and the side second camera array 140, this embodiment uses a multi-function I / O device card to generate and record synchronization trigger signals, and simultaneously records images and corresponding timestamps through the aforementioned MIAS program for offline image synchronization processing.

[0049] In addition, the present disclosure also relates to a real-time motion monitoring and control device for a moving object, the specific components of which have been described above in conjunction with the real-time motion monitoring and control system and will not be repeated here.

[0050] As shown in FIG6 , the present disclosure further provides a real-time motion monitoring and control method 200 for a moving object. The method 200 is implemented using the real-time motion monitoring and control system 100 described above, and includes the following steps:

[0051] S210, using the first camera array 120 to capture first images of a plurality of moving objects, and calculating the real-time positions of the target moving objects in the camera field of view based on the first images;

[0052] S220, based on the information about the real-time position, controlling the movement of the multi-dimensional treadmill 110 so that the target moving object on the multi-dimensional treadmill 110 remains in a fixed position relative to the first camera array 120 and the second camera array 140;

[0053] S230, detecting key points of the target moving object in the first image using a neural network model, wherein the key points correspond to parts of the target moving object;

[0054] S240, generating a scene for free movement of the plurality of moving objects;

[0055] S250, using a second camera array to synchronously capture multiple second images of the target moving object from different lateral directions in real time;

[0056] S260, reconstructing a three-dimensional model of the target moving object based on the first image and the plurality of second images; and

[0057] S270: In this scenario, laser stimulation is performed on the key point.

[0058] The above steps S210-S270 have been described in detail above in conjunction with the real-time motion monitoring and control system 100 and will not be repeated here. It should be understood that the above steps S210-S270 do not indicate the execution order of these steps, but rather their execution order can be flexibly changed as needed to implement the technical solution of the present invention.

[0059] It should be understood that in various embodiments of the present application, the processing device may be a processor. The processor may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field programmable gate arrays (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. It should be noted that the processor may also be integrated with components for storage such as a memory unit and / or a cache unit.

[0060] During implementation, each step of the above method can be completed by an integrated logic circuit of the hardware in the processor or by instructions in the form of software. The steps of the method disclosed in conjunction with the embodiments of the present application can be directly embodied as being executed by a hardware processor, or can be executed by a combination of hardware and software modules in the processor. The software module can be located in a storage medium mature in the art such as a random access memory, a flash memory, a read-only memory, a programmable read-only memory or an electrically erasable programmable memory, a register, etc. The storage medium is located in the memory, and the processor reads the information in the memory and completes the steps of the above method in conjunction with its hardware. To avoid repetition, it will not be described in detail here.

[0061] The above description is merely a preferred embodiment of the present application and an illustration of the technical principles employed. Those skilled in the art should understand that the scope of disclosure in this application is not limited to the technical solutions formed by a specific combination of the above-mentioned technical features, but also encompasses other technical solutions formed by any combination of the above-mentioned technical features or their equivalents without departing from the above-mentioned disclosed concepts. For example, a technical solution formed by replacing the above-mentioned features with (but not limited to) technical features with similar functions disclosed in this application.

[0062] In addition, although adopting specific order to describe each operation, this should not be interpreted as requiring these operations to be executed in the specific order shown or in sequential order.Under certain environment, multitasking and parallel processing may be advantageous.Similarly, although comprising some specific implementation details in the above discussion, these should not be interpreted as limiting the scope of the present invention.Some features described in the context of independent embodiment can also be implemented in single embodiment in combination.On the contrary, the various features described in the context of independent embodiment also can be implemented in multiple embodiments individually or in the mode of any suitable subcombination.

[0063] Although the subject matter has been described in language specific to structural features and / or methodological logical acts, it should be understood that the subject matter defined in the appended claims is not necessarily limited to the specific features or acts described above. Rather, the specific features and acts described above are merely example forms of implementing the claims.

[0064] The above describes in detail several embodiments of the present application, but the present application is not limited to these specific embodiments. Those skilled in the art can make various variations and modifications to the embodiments based on the concept of the present application, and these variations and modifications should all fall within the scope of protection claimed by the present application.

Claims

1. A real-time motion monitoring and control system for a moving object, characterized in that: include: a multi-dimensional treadmill configured to allow a plurality of exercise subjects to freely move thereon; a first camera array configured to capture first images of the plurality of moving objects; a second camera array, wherein a plurality of cameras in the second camera array are arranged circumferentially around the multi-dimensional treadmill and synchronously capture a plurality of second images of the plurality of moving objects in real time from different lateral directions; A processing device configured to: receiving the first image from the first camera array and the plurality of second images from the second camera array; reconstructing a three-dimensional model of a target moving object among the multiple moving objects based on the first image and the multiple second images; Calculating a real-time position of the target moving object in the camera field of view based on the first image; Based on the information about the real-time position of the target moving object, controlling the movement of the multi-dimensional treadmill so that the target moving object on the multi-dimensional treadmill remains in a fixed position relative to the first camera array and the second camera array; and Detecting key points of the target moving object in the first image, wherein the key points correspond to parts of the moving object; a scene generating device, which is arranged around an activity area of the plurality of moving objects on the multi-dimensional treadmill and is configured to generate a scene for the free activity under the control of the processing device; and A laser stimulation component is configured to perform laser stimulation on the key points of the target moving object in the scene under the control of the processing device.

2. The real-time motion monitoring and control system according to claim 1, characterized in that: The scene generating device is a screen, which is configured to generate a panoramic virtual reality scene.

3. The real-time motion monitoring and control system according to claim 1 or 2, characterized in that: Also includes: a light source configured to provide illumination to the plurality of moving objects on the multi-dimensional treadmill; Wherein, the multi-dimensional treadmill includes multiple servo motors or multiple piezoelectric motors.

4. The real-time motion monitoring and control system according to claim 1 or 2, characterized in that: The laser stimulation assembly includes: laser source; and A two-dimensional galvanometer is configured to deflect and scan the laser light from the laser source to achieve controllable directional stimulation of the key point.

5. The real-time motion monitoring and control system according to claim 4, characterized in that: The laser stimulation assembly further includes a first aperture stop, a beam reducer, a second aperture stop, and a reflecting mirror sequentially arranged between the laser source and the two-dimensional galvanometer, and a dichroic mirror arranged between the two-dimensional galvanometer and the multi-dimensional treadmill.

6. The real-time motion monitoring and control system according to claim 1 or 2, characterized in that: The first camera array includes a first positioning camera having a first resolution, a second positioning camera having a second resolution, and a spectroscope, wherein the first resolution is smaller than the second resolution; The first image includes an image taken by the first positioning camera and an image taken by the second positioning camera, and the processing device detects the key points based on the image taken by the second positioning camera.

7. The real-time motion monitoring and control system according to claim 1 or 2, characterized in that: The multi-dimensional treadmill further comprises a container for accommodating the plurality of sports objects to define the activity area, wherein the container is placed on the multi-dimensional treadmill.

8. The real-time motion monitoring and control system according to claim 1 or 2, characterized in that: The system also includes a physical and chemical stimulation component configured to perform physical and chemical stimulation on the target motion object based on the scene.

9. The real-time motion monitoring and control system according to claim 1 or 2, characterized in that: The moving object is any one of an animal, a movable facility and a robot.

10. A real-time motion monitoring and control device for a moving object, characterized in that: include: a multi-dimensional treadmill configured to allow a plurality of exercise subjects to freely move thereon; a first camera array configured to capture first images of the plurality of moving objects; a second camera array, wherein a plurality of cameras in the second camera array are arranged circumferentially around the multi-dimensional treadmill and synchronously capture a plurality of second images of the plurality of moving objects in real time from different lateral directions; A processing device configured to: receiving the first image from the first camera array and the plurality of second images from the second camera array; Reconstruct one of the plurality of moving objects based on the first image and the plurality of second images 3D model of the moving object; Calculating a real-time position of the target moving object in the camera field of view based on the first image; Based on the information about the real-time position of the target moving object, controlling the movement of the multi-dimensional treadmill so that the target moving object on the multi-dimensional treadmill remains in a fixed position relative to the first camera array and the second camera array; and Detecting key points of the target moving object in the first image, wherein the key points correspond to parts of the moving object; a scene generating device, which is arranged around an activity area of the plurality of moving objects on the multi-dimensional treadmill and is configured to generate a scene for the free activity under the control of the processing device; and A laser stimulation component is configured to perform laser stimulation on the key points of the target moving object in the scene under the control of the processing device.

11. A real-time motion monitoring and control method using the real-time motion monitoring and control system according to any one of claims 1 to 9, comprising: Using the first camera array to capture first images of the plurality of moving objects, and calculating the real-time positions of the target moving objects in the camera field of view based on the first images; Based on the information about the real-time position, controlling the movement of the multi-dimensional treadmill so that the target moving object on the multi-dimensional treadmill remains in a fixed position relative to the first camera array; Detecting key points of the target moving object in the first image, wherein the key points correspond to parts of the target moving object; generating a scene for the free movement of the plurality of moving objects; Using a second camera array to synchronously capture multiple second images of the target moving object from different lateral directions in real time; reconstructing a three-dimensional model of the target moving object based on the first image and the plurality of second images; and In the scene, laser stimulation is performed on the key points.

Images