Real-time motion monitoring control system, device and method for moving objects
Through the combination of a multi-dimensional treadmill and a multi-camera array, the multi-view high-resolution behavior recording and three-dimensional spatial position detection of moving objects are achieved, solving the problems of recording and virtual reality environments in the existing technology, and providing detailed behavioral analysis and virtual reality environment.
Patent Information
- Application Number
- CN202410176439.3
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2024-02-08
- Publication Date
- 2024-11-15
- Estimated Expiration
- 2044-02-08
AI Technical Summary
The prior art is difficult to perform multi-view high-resolution behavior recording and fine three-dimensional spatial position detection on moving objects under free activities, and the brightness of the virtual reality environment is uneven and the system is large, so the installation and debugging are complicated.
A multi-dimensional treadmill is used to combine multi-camera arrays for real-time motion monitoring and recording. The three-dimensional model is reconstructed through the processing device, the treadmill movement is controlled to maintain the fixed position of the object, and the laser stimulation and virtual reality technology are used for intervention and control.
It realizes multi-view high-resolution behavior recording and three-dimensional key point detection of moving objects, provides an accurate virtual reality environment, reduces system size and installation complexity, and improves the precision of behavior analysis.
Smart Images

Figure CN118383288B_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the field of animal behavior and machine vision technology, and more specifically to a real-time motion monitoring control system, device and method for a moving object. Background Art
[0002] In order to more accurately measure the behavior of moving objects (such as small animals or small machines) in free movement, it is necessary to perform precise three-dimensional spatial position detection of key points of their body parts, so it is necessary to use multiple cameras to capture high-resolution images from different angles. When the moving object is in motion, in order to capture the image, the camera needs a larger field of view, which will result in lower resolution, but if the mobile camera tracks the target, it will be too complicated when the number of cameras increases. In the prior art, the method of shooting the animal by fixing it on a suspended ball greatly restricts the behavior of the animal and makes it impossible for the animal to interact with its peers in free movement. Although some other methods can obtain high-resolution two-dimensional images of freely moving animals, they are only single cameras and cannot obtain three-dimensional spatial information.
[0003] In some studies, it is necessary to examine and control the function of neural circuits by combining optogenetics with spatiotemporal light stimulation of specific neurons. Currently, in order to achieve precise laser stimulation of unilateral brain regions, the commonly used methods are to fix the animals or use transgenic technology to randomly express unilateral neurons (such as FLP technology in fruit flies), which restricts the behavior of the animals or is time-consuming and laborious.
[0004] In order to provide a virtual reality visual environment for the moving object, it is necessary to provide a ring screen around it that can accurately control the display content. The current method mainly uses a projector to project on the ring screen or assembles flat display units in a ring. These methods have problems such as uneven brightness and obvious splicing edges. In addition, the system is large in size and complicated to install and debug.
[0005] Therefore, there is a need for a real-time motion monitoring and control system that can provide complete and real-time multi-perspective high-resolution behavior records of multiple moving objects (such as small animals) in a free social environment, and can provide precise virtual reality stimulation to the moving objects according to different scenarios. Summary of the invention
[0006] To solve the above problems, the present disclosure provides a real-time motion monitoring control system, device and method for moving objects, which can track the movement of moving objects in real-time closed-loop, and record high-resolution images from multiple angles of moving objects (such as animals, movable facilities or robots) in free activities including complex social behaviors, measure their behaviors, and can also combine technologies such as optogenetic stimulation and virtual reality environment to intervene and control them according to specific social scenarios.
[0007] In the first aspect, the real-time motion monitoring and control system for a moving object disclosed in the present application includes: a multi-dimensional treadmill, which is configured to allow a plurality of moving objects to move freely thereon. The real-time motion monitoring and control system also includes: a first camera array, which is configured to capture a first image of the plurality of moving objects. The real-time motion monitoring and control system also includes: a second camera array, in which a plurality of cameras are arranged around the multi-dimensional treadmill in a circumferential direction, and a plurality of second images of the plurality of moving objects are captured synchronously in real time from different lateral directions. The real-time motion monitoring and control system also includes: a processing device, which is configured to: receive the first image from the first camera array and the plurality of second images from the second camera array; reconstruct a three-dimensional model of a target moving object among the plurality of moving objects based on the first image and the plurality of second images; calculate the real-time position of the target moving object in the camera field of view based on the first image; control the movement of the multi-dimensional treadmill based on the information about the real-time position of the target moving object so that the target moving object on it remains in a fixed position relative to the first camera array and the second camera array; and detect key points of the target moving object in the first image, wherein the key points correspond to the parts of the moving object. The real-time motion monitoring and control system also includes: a scene generating device, which is arranged around the activity area of the multiple moving objects on the multi-dimensional treadmill and is configured to generate a scene for the free activity under the control of the processing device; and a laser stimulation component, which is configured to perform laser stimulation on the key points of the target moving object in the scene under the control of the processing device.
[0008] In a second aspect, the real-time motion monitoring and control device for a moving object disclosed in the present application includes: a multi-dimensional treadmill, which is configured to allow a plurality of moving objects to move freely thereon. The real-time motion monitoring and control device also includes: a first camera array, which is configured to capture a first image of the plurality of moving objects. The real-time motion monitoring and control device also includes: a second camera array, wherein a plurality of cameras in the second camera array are arranged around the multi-dimensional treadmill in a circumferential direction, and a plurality of second images of the plurality of moving objects are captured synchronously in real time from different lateral directions. The real-time motion monitoring and control device also includes: a processing device, which is configured to: receive the first image from the first camera array and the plurality of second images from the second camera array; reconstruct a three-dimensional model of a target moving object among the plurality of moving objects based on the first image and the plurality of second images; calculate the real-time position of the target moving object in the camera field of view based on the first image; control the movement of the multi-dimensional treadmill based on the information about the real-time position of the target moving object so that the target moving object on it remains in a fixed position relative to the first camera array and the second camera array; and detect key points of the target moving object in the first image, wherein the key points correspond to the parts of the moving object. The real-time motion monitoring and control device also includes: a scene generating device, which is arranged around the activity area of the multiple moving objects on the multi-dimensional treadmill and is configured to generate a scene for the free activity under the control of the processing device; and a laser stimulation component, which is configured to perform laser stimulation on the key points of the target moving object in the scene under the control of the processing device.
[0009] In a third aspect, the real-time motion monitoring and control method for moving objects disclosed in the present application includes the following steps: using a first camera array to capture a first image of the multiple moving objects, and calculating the real-time position of the target moving object in the camera field of view based on the first image; based on information about the real-time position, controlling the movement of the multi-dimensional treadmill so that the target moving object on it remains in a fixed position relative to the first camera array; detecting key points of the target moving object in the first image, wherein the key points correspond to parts of the target moving object; generating a scene for the free movement of the multiple moving objects; using a second camera array to synchronously capture multiple second images of the target moving object in real time from different lateral directions; reconstructing a three-dimensional model of the target moving object based on the first image and the multiple second images; and performing laser stimulation on the key points in the scene.
[0010] The real-time motion monitoring control system, device and method mainly use multiple cameras with different angles and resolutions to monitor and record multiple animals in social scenes in real time, and estimate and analyze the center and specific body parts of the animals in real time and perform identity recognition, so as to drive the multi-dimensional motor closed loop to track specific target animals, thereby keeping them in a (dynamic) fixed position relative to the camera and the ground, and obtaining multi-perspective and high-resolution images of the animals, which can be used for subsequent three-dimensional key point detection. In addition, the laser position can be adjusted in real time by the laser stimulation component, and combined with optogenetics, specific neural activities can be manipulated in a spatiotemporal stimulation manner. In addition, controllable virtual stimulation or virtual peers can be displayed in real time through a flexible screen that surrounds nearly 360° to simulate specific life or social scenes. Using these components, the technical solution of the present disclosure is realized by integrating infrared light imaging, image processing, motion control, laser control, artificial intelligence, virtual reality and other technologies. BRIEF DESCRIPTION OF THE DRAWINGS
[0011] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the drawings required for use in the embodiments or the prior art descriptions are briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in this application. When appropriate, the same reference numerals are used in all drawings to refer to the same or similar parts. Such embodiments are illustrative and are not intended to be exhaustive or exclusive embodiments of the present device or method. In the drawings:
[0012] Figure 1 is a schematic structural diagram of a real-time motion monitoring control system according to an embodiment of the present disclosure;
[0013] Figure 2 for Figure 1 Magnified view of area A in the middle;
[0014] Figure 3 is a schematic functional structure diagram of a real-time motion monitoring and control system according to an embodiment of the present disclosure;
[0015] Figure 4 images captured by the first camera array and the second camera array according to an embodiment of the present disclosure;
[0016] Figure 5 is an image taken by a second positioning camera according to an embodiment of the present disclosure, wherein the center position and several key point positions of a moving object are shown; and
[0017] Figure 6 It is a schematic flow chart of a real-time motion monitoring control method according to an embodiment of the present disclosure. DETAILED DESCRIPTION
[0018] In order to make the purpose, technical solution and advantages of the embodiment of the present application clearer, the technical solution of the embodiment of the present application will be clearly and completely described in conjunction with the drawings of the embodiment of the present application. Obviously, the described embodiment is a part of the embodiment of the present application, not all of the embodiments. Based on the described embodiment of the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.
[0019] Unless otherwise defined, the technical terms or scientific terms used in this application should be understood by people with ordinary skills in the field to which this application belongs. "First", "second" and similar words used in this application do not indicate any order, quantity or importance, but are only used to distinguish different components. "Include" or "comprise" and similar words mean that the elements or objects appearing before the word cover the elements or objects listed after the word and their equivalents, without excluding other elements or objects. "Connect" or "connected" and similar words are not limited to physical or mechanical connections, but can include electrical connections, whether direct or indirect. "Up", "down", "left", "right" and the like are only used to indicate relative positional relationships. When the absolute position of the described object changes, the relative positional relationship may also change accordingly.
[0020] In order to keep the following description of the embodiments of the present application clear and concise, the present application omits detailed descriptions of known functions and known components.
[0021] The present disclosure is dedicated to being able to perform a more comprehensive behavioral analysis of moving objects, where the moving objects can be any of animals (such as insects), movable facilities and robots. The animals here can include but are not limited to insects, small fish, small amphibians, small reptiles, etc., and robots can also include logistics robots, etc. For ease of description, in the embodiments provided below, the real-time motion monitoring control system and method of the present disclosure are described with the classic model organism fruit fly as the object. However, it should be understood that any other animals, movable facilities or robots that can move freely and interact with each other are also equally applicable to the system and method provided by the present disclosure.
[0022] This system Figure 3 As shown in the figure, the control software was developed through Labview program and the image processing was performed through C++ program, so the system can be easily controlled and observed through the graphical user interface. The control software mainly includes functional parts such as synchronization signal generation and acquisition, data communication, two-dimensional treadmill control, light stimulation control and virtual reality control. Among them, the synchronization signal mainly includes camera trigger signal and laser control signal; data communication is mainly network communication with the image processing program.
[0023] Figure 1 FIG. 1 shows a schematic structural diagram of a real-time motion monitoring and control system 100 for a moving object (fruit fly) according to an embodiment of the present disclosure. Figure 1 As shown, the real-time motion monitoring and control system 100 comprises at least: a multi-dimensional treadmill 110, a first camera array 120, a scene generating device 130, a second camera array 140, a laser stimulation component 150 and a processing device (not shown).
[0024] In this embodiment, the multi-dimensional treadmill 110 can be configured to be operable in multiple dimensions. As an example, the multi-dimensional treadmill 110 is configured as a two-dimensional treadmill 110, which includes two high-precision servo motors vertically combined with each other to minimize the displacement of the fruit fly relative to the first camera array 120 and the second camera array 140 (described in detail below) to obtain high-resolution images, and a PID (Proportional-Integral-Derivative) control algorithm is used to close-loop control the movement of the servo motors to track and compensate for the movement of the fruit fly.
[0025] Here, if Figure 2 As shown, a plurality of fruit flies may be placed in a container 170 to define the activity area, and the container 170 may be placed on the two-dimensional treadmill 110 .
[0026] The first camera array 120 is used to capture a real-time first image of a plurality of fruit flies moving freely on the two-dimensional treadmill 110. Figure 1 In the embodiment, the first camera array 120 is preferably arranged at the top of the multi-dimensional treadmill 110, and includes a first positioning camera 121, a second positioning camera 122 and a spectroscope 123. Correspondingly, the resolution of the image captured by the first positioning camera 121 is lower than the resolution of the image captured by the second positioning camera 122. By using a 25mm wideband spectroscope 123, the first positioning camera 121 and the second positioning camera 122 can simultaneously capture images from above the treadmill. For the first positioning camera 121, its resolution is 640×480 pixels (first resolution), the exposure time is 1ms, and the recording frame rate is 200fps. In order to obtain the displacement of the two-dimensional treadmill without affecting the system performance so as to calculate the absolute position of the fruit fly, the first positioning camera 121 uses a wide-angle lens with a focal length of 8mm to ensure that the camera field of view can cover the entire activity area of the fruit fly and the four landmarks around the activity area. For the second positioning camera 122, its resolution is 1280×1024 pixels (second resolution), the exposure time is 1ms, and the recording frame rate is 200fps. In order to estimate the position and posture of the fruit fly with higher resolution, the second positioning camera 122 uses a 35mm focal length lens and a 2x lens expander.
[0027] The first camera array 110 is used to take a first image of a plurality of fruit flies from the top of the two-dimensional treadmill 110 in real time. The first image here should include an image taken by the first positioning camera 121 and an image taken by the second positioning camera 122 (the resolution of the former is lower than the latter). The first image is sent to the processing device, so as to calculate the real-time position of the plurality of fruit flies in the camera field of view and the absolute position in space in real time. The target fruit fly as the monitoring and control object is identified from the plurality of fruit flies using image recognition technology. Then, the deviation between the real-time position of the target fruit fly and the preset target position (such as the center position of the camera field of view) is used as the input of the PID controller, and the position deviation is converted into the servo motor speed, and then the movement of the motor is controlled, so that the target fruit fly is kept in a (dynamic) fixed position relative to the first camera array 120, the second camera array 140 and the ground, thereby producing an effect similar to that of a fruit fly with a fixed body on a suspended ball.
[0028] In the process of controlling the servo motor movement, because the maximum acceleration of the motor is small, the converted motor speed is limited to a certain extent by the square root controller (Formula (1)). In addition, when the position deviation is greater than 4 mm or less than 0.1 mm, the motor speed control amount is reduced to 1 / 5 to prevent the interference of the fruit fly behavior due to the rapid adjustment of the motor and the jitter caused by the fruit fly staying in the same position but moving by itself.
[0029]
[0030] Where v is the output motor speed, K p is the frame rate of image processing, E Pos is the absolute value of the position deviation output by the PID controller, A max is the maximum acceleration of the motor.
[0031] In terms of calculating the real-time position of fruit flies, the present disclosure provides a self-developed C++ program loaded on a processing device to realize low-latency position calculation (estimation). The target position calculation main program contains three threads running in parallel, including a video acquisition thread, a prediction thread, and a recording thread. The video acquisition thread controls the first camera array 120 to capture the original image and transmits the image to the other two threads. The recording thread writes the image to a video file and writes the timestamp to a log file. The video acquisition thread and the recording thread together constitute the MIAS program, which is also applicable to the second camera array 140. The prediction thread detects target body parts of multiple fruit flies, for example, and communicates with the key point detection program to pass the position information to the two-dimensional treadmill control program. Here, as shown in FIG. Figure 5As shown, the target body part here can be the center of the fruit fly's body (indicated by "+" in the figure); however, other body parts of the fruit fly, such as the head and tail, can also be tracked as needed.
[0032] In the prediction thread, a threshold-based method is used to detect the body center of the fruit fly. A binary image is obtained by threshold segmentation, and connected regions are detected on the binary image. In order to eliminate interference such as shadows and noise background, regions with the size of fruit flies are selected as candidate regions. In the case of multiple fruit flies, the tracking target can be selected according to the size, such as selecting the smallest male fruit fly as the tracking target. At the same time, whether the fruit flies overlap is further detected by judging whether the number of fruit fly regions suddenly decreases. If overlap is detected, the position of the last frame is used as the target position to minimize the rapid jump of the treadmill. At the same time, there are four landmarks of different sizes and shapes around the activity area of the fruit fly that are invisible to the fruit fly. The landmarks can limit the detection area of the fruit fly and the absolute position of the fruit fly in the activity area can be calculated by comparing the offset between the landmarks and the initial position.
[0033] For the small field of view image with higher resolution captured by the second positioning camera 120 in the first image captured by the first camera array 120, in addition to being used to detect the center of the fruit fly, a 128×128 pixel area is segmented at the center of the fruit fly, and the area is used as the input of the trained neural network model (such as the U-Net model) to detect the positions of two key points of the fruit fly, such as the left and right compound eyes, in real time. Figure 5 It should be understood that in this technical field, various other means other than the neural network model can also be used to detect the key point position, such as image recognition technology.
[0034] By using the absolute positions and key point information of multiple fruit flies, the orientation of each fruit fly and the relative positions between fruit flies can be calculated in real time, and then different social moments can be selected to perform laser stimulation and virtual reality control on the fruit flies as described below.
[0035] In order to more accurately measure the behavior of fruit flies in free activities and complex social environments, the present disclosure also provides a scene generation device 130, such as Figure 1 and 2As shown, it is arranged around the activity area of the fruit fly on the two-dimensional treadmill 110 to generate a scene for free movement. Since the vision-based virtual reality technology can provide rich and controllable visual scenes, in an exemplary embodiment, the scene generation device 130 is a screen, preferably an OLED flexible screen (resolution 2160×1080, refresh rate 60Hz), to generate a panoramic virtual reality (Virtual Reality) scene, which is surrounded into a cylinder with a diameter of 44mm and close to 360°, and the fruit fly can move freely in the container 170 in the middle.
[0036] Preferably, the system 100 can call a function library for controlling the content displayed on the screen, and calculate a specific social scene in real time by obtaining the absolute positions and key point information of multiple fruit flies, and can choose to generate low-latency and controllable visual stimulation at a specific moment. At the same time, based on the three-dimensionally reconstructed virtual fruit flies, a more realistic virtual social scene can be given to the fruit flies. For example, assuming that there is a fruit fly in front of the left side of the target fruit fly at the detection location, and it is judged based on the key point information that the two are facing each other, the processing device can generate another virtual fruit fly in front of it on the right side, and measure the social behavior of the target fruit fly in this scene; or, the absolute position can indicate how far the target fruit fly has traveled and the trajectory of movement, etc., and the processing device can generate different scenes according to the absolute position.
[0037] It should be understood that the scene generation device 130 can also generate scenes according to needs (such as the operator's experience or preference), and does not have to be based on the absolute positions and key point information of multiple fruit flies.
[0038] In addition, the scene generating device 130 provided in the present disclosure may be other devices besides the OLED screen used to generate a panoramic virtual reality scene described in the example, such as a device that provides a panoramic augmented reality scene.
[0039] As described above, through the closed-loop control of the two-dimensional treadmill 110, the target fruit fly can be kept in a fixed position or a relatively small range relative to the first camera array 120, the second camera array 140 and the ground, producing an effect similar to that of a fruit fly with a fixed body on a suspended ball. Therefore, through this system, the laser stimulation component 150 can be provided to perform, for example, unilateral light stimulation of the fruit fly when the fruit fly moves freely. At the same time, through the absolute position estimation and key point detection of multiple fruit flies, light stimulation experiments can be performed on different parts of the fruit fly in different social scenes to study specific neural activities.
[0040] like Figure 1As shown, the laser stimulation component 150 here may include: a laser source 151; and a two-dimensional galvanometer 152, which is configured to deflect and scan the laser from the laser source 151 to achieve controllable directional stimulation of the identified key points of the target fruit fly.
[0041] In order to conduct optogenetic activation experiments with less interference to the behavior of the target fruit fly, the present disclosure uses a red light semiconductor laser with a central wavelength of 638nm. In order to adjust the beam diameter of the laser, the present disclosure uses a 0.5-fold beam reduction mirror 153 and two adjustable aperture diaphragms 154 and 155 with a minimum aperture of 0.8mm to adjust the laser, and the adjusted beam half-width (FWHM) diameter is about 0.6mm. At the same time, real-time laser control can be achieved through a controller, including switching, intensity and frequency modulation. Then, the adjusted laser is reflected into the two-dimensional galvanometer 152 by the reflector 156 for deflection scanning, and the scanning laser is filtered and reflected by the dichroic mirror 157, and finally irradiated at the corresponding key point position of the target fruit fly to achieve laser stimulation. Since the fruit fly is relatively fixed in absolute space, the above-mentioned laser stimulation can be conveniently applied without the need to track the movement of the fruit fly significantly when applying the stimulation.
[0042] The present disclosure tracks and compensates the movement of the target fruit fly on the horizontal plane by linearly adjusting the two-dimensional galvanometer 152. The control of the two-dimensional galvanometer 152 is an open-loop control. After determining the coordinate conversion relationship between the two-dimensional galvanometer 152 and the two-dimensional treadmill 110, the target key point position of the stimulus is selected according to the specific scene calculated in real time, and the error deviating from the tracking center can be calculated to linearly control the two-dimensional galvanometer 152. The two-dimensional galvanometer 152 can be controlled to move at a frame rate of 200Hz to guide the laser to track the specific target key point position of the fruit fly in real time.
[0043] In order to calibrate the laser of the light stimulation system, this embodiment also uses a photodiode power meter placed at the position of the fruit fly container 170, and covers the container 170 with a Teflon-coated cover glass 171 to simulate the actual laser intensity reaching the fruit fly. By changing the driving voltage of the laser, a series of laser output intensities are generated. In addition, the optical power meter is fixed on a precision displacement stage, and the detection target surface is half-shielded with opaque black tape. The laser beam reaching the optical power meter can be gradually blocked from unblocked to completely blocked by a fixed micro-movable displacement stage. This process measures a series of integrated laser intensities, and the intensity distribution of the laser spot can be obtained by calculating its derivative.
[0044] In addition, in this embodiment, other stimulation means besides laser stimulation can also be provided, for example, physical and chemical stimulation of the target fruit fly can be performed through a physical and chemical stimulation component (not shown in the figure), such as sound, light, heat and smell stimulation, so as to further study the behavior of the fruit fly under different stimulation means.
[0045] return Figure 1 In addition to the first camera array 120 at the top, this embodiment also adopts a plurality of ( Figure 1 140 is exemplarily shown in the figure. The second camera array 140 is arranged at the lateral position of the two-dimensional treadmill 110, and can simultaneously take multiple second images of the target fruit fly with high resolution from different lateral angles. The multiple second images together with the first image provided by the first camera array 120 are used to reconstruct the three-dimensional model of the target fruit fly through the processing device, so as to measure the animal behavior of the target fruit fly, and can also be used for the neural network model training and detection of the three-dimensional key points of the fruit fly. The eight side view cameras are evenly placed at intervals of about 45° on the side of the motion control platform of the two-dimensional treadmill 110, with an inclination angle of about 22 (±1)°, a resolution of 1280×1024 pixels, and a telecentric lens with a magnification of 0.7 and a working distance of 65 mm is used. Figure 4 The diagram shows the first image and the second image taken at the same time by the first camera array 120 and the second camera array 140 provided in this embodiment, wherein the upper left corner is the first image taken by the first positioning camera 121, the middle is the first image taken by the second positioning camera 122, and the rest are the second images taken by 8 cameras located at different lateral directions. The second image is slightly cropped to match the composite image, and the target position to be tracked is the center position of the male fruit fly.
[0046] In addition, during the implementation of the above embodiment, a light source is also provided to provide illumination for multiple fruit flies on the two-dimensional treadmill to improve the brightness and contrast of the recorded image. Preferably, the present embodiment evenly illuminates the activity area of the fruit flies from the top and bottom. The top and bottom use a customized infrared ring light source 161 ( Figure 1 ) and square light source 162 ( Figure 1 ), and the fruit flies are kept at a sufficient distance from the square light source 162 at the bottom to reduce behavioral changes caused by the heat of the light source. Because infrared light is invisible to fruit flies, in order to provide the necessary visual cues to fruit flies, this embodiment adds a ring light source 161 on the top, which can provide white light or blue light. At the same time, an infrared filter is added between all cameras and lenses, allowing only infrared light to pass, eliminating the influence of fruit fly visible light and excitation light on the image.
[0047] In a preferred embodiment, in addition to recording black and white images, the real-time motion monitoring system 100 of the present invention can also be flexibly constructed and configured as needed, such as using a color camera and a white light source to record color images; the recording frame rate and resolution can also be flexibly selected according to the computer hardware configuration and requirements.
[0048] In addition, in order to ensure the time synchronization of images captured by all the top first camera arrays 120 and the side second camera arrays 140, this embodiment uses a multi-function I / O device card to generate and record synchronization trigger signals, and simultaneously records images and corresponding timestamps through the aforementioned MIAS program for offline image synchronization processing.
[0049] In addition, the present disclosure also relates to a real-time motion monitoring and control device for a moving object, the specific components of which have been described above in conjunction with the real-time motion monitoring and control system, and will not be repeated here.
[0050] like Figure 6 As shown, the present disclosure also provides a real-time motion monitoring control method 200 for a moving object. The method 200 is implemented using the real-time motion monitoring control system 100 described above, and includes the following steps:
[0051] S210, using the first camera array 120 to capture first images of a plurality of moving objects, and calculating the real-time position of the target moving objects in the camera field of view according to the first images;
[0052] S220, based on the information about the real-time position, controlling the movement of the multi-dimensional treadmill 110 so that the target moving object thereon remains in a fixed position relative to the first camera array 120 and the second camera array 140;
[0053] S230, detecting key points of the target moving object in the first image using a neural network model, wherein the key points correspond to parts of the target moving object;
[0054] S240, generating a scene for free movement of the plurality of moving objects;
[0055] S250, using a second camera array to synchronously capture multiple second images of the target moving object in real time from different lateral directions;
[0056] S260, reconstructing a three-dimensional model of the target moving object based on the first image and the plurality of second images; and
[0057] S270: In this scenario, laser stimulation is performed on the key point.
[0058] The above steps S210-S270 have been described in detail in the previous text in conjunction with the real-time motion monitoring control system 100, and will not be repeated here. It should be understood that the above steps S210-S270 do not indicate the execution order of these steps, but the execution order can be flexibly changed as needed to implement the technical solution of the present invention.
[0059] It should be understood that in various embodiments of the present application, the processing device may be a processor. The processor may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application specific integrated circuits (ASIC), field programmable gate arrays (FPGA), or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components, etc. It should be noted that the processor may also be integrated with components for storage such as memory units and / or cache units.
[0060] In the implementation process, each step of the above method can be completed by an integrated logic circuit of hardware in a processor or an instruction in the form of software. The steps of the method disclosed in conjunction with the embodiment of the present application can be directly embodied as a hardware processor for execution, or a combination of hardware and software modules in a processor for execution. The software module can be located in a storage medium mature in the art such as a random access memory, a flash memory, a read-only memory, a programmable read-only memory or an electrically erasable programmable memory, a register, etc. The storage medium is located in a memory, and the processor reads the information in the memory and completes the steps of the above method in conjunction with its hardware. To avoid repetition, it is not described in detail here.
[0061] The above description is only a preferred embodiment of the present application and an explanation of the technical principles used. Those skilled in the art should understand that the scope of disclosure involved in the present application is not limited to the technical solution formed by a specific combination of the above technical features, but should also cover other technical solutions formed by any combination of the above technical features or their equivalent features without departing from the above disclosed concept. For example, the above features are replaced with the technical features with similar functions disclosed in this application (but not limited to) by each other to form a technical solution.
[0062] In addition, although each operation is described in a specific order, this should not be construed as requiring these operations to be performed in the specific order shown or in a sequential order. Under certain circumstances, multitasking and parallel processing may be advantageous. Similarly, although some specific implementation details are included in the above discussion, these should not be construed as limiting the scope of the present invention. Some features described in the context of a separate embodiment can also be implemented in a single embodiment in combination. On the contrary, the various features described in the context of a single embodiment can also be implemented in multiple embodiments individually or in any suitable sub-combination mode.
[0063] Although the subject matter has been described in language specific to structural features and / or methodological logical actions, it should be understood that the subject matter defined in the appended claims is not necessarily limited to the specific features or actions described above. On the contrary, the specific features and actions described above are merely example forms of implementing the claims.
[0064] Several embodiments of the present application have been described in detail above, but the present application is not limited to these specific embodiments. Those skilled in the art can make various variations and modifications to the embodiments based on the concept of the present application, and these variations and modifications should all fall within the scope of protection claimed by the present application.
Claims
1. A real-time motion monitoring and control system for a moving object, It is characterized in that include: A multi-dimensional treadmill configured to allow a plurality of exercise subjects to freely move thereon; a first camera array configured to capture first images of the plurality of moving objects; A second camera array, wherein a plurality of cameras in the second camera array are arranged around the multi-dimensional treadmill in a circumferential direction, and synchronously capture a plurality of second images of the plurality of moving objects in real time from different lateral directions; A processing device configured to: receiving the first image from the first camera array and the plurality of second images from the second camera array; reconstructing a three-dimensional model of a target moving object among the multiple moving objects based on the first image and the multiple second images; Calculating the real-time position of the target moving object in the camera field of view based on the first image; Based on the information about the real-time position of the target moving object, controlling the movement of the multi-dimensional treadmill so that the target moving object on the treadmill remains in a fixed position relative to the first camera array and the second camera array; and Detecting key points of the target moving object in the first image, wherein the key points correspond to parts of the moving object; a scene generating device, which is arranged around the activity area of the plurality of moving objects on the multi-dimensional treadmill and is configured to generate a scene for the free activity under the control of the processing device; and a laser stimulation component configured to perform laser stimulation on the key points of the target moving object in the scene under the control of the processing device; The real-time motion monitoring and control system further comprises a container for accommodating the plurality of motion objects to define the activity area, wherein the container is placed on the multi-dimensional treadmill.
2. The real-time motion monitoring and control system according to claim 1, It is characterized in that The scene generating device is a screen, which is configured to generate a panoramic virtual reality scene.
3. The real-time motion monitoring and control system according to claim 1 or 2, It is characterized in that Also includes: a light source configured to provide illumination to the plurality of moving objects on the multi-dimensional treadmill; Wherein, the multi-dimensional treadmill includes multiple servo motors or multiple piezoelectric motors.
4. The real-time motion monitoring and control system according to claim 1 or 2, It is characterized in that The laser stimulation assembly comprises: a laser source; and A two-dimensional galvanometer is configured to deflect and scan the laser light from the laser source to achieve controllable directional stimulation of the key point.
5. The real-time motion monitoring and control system according to claim 4, It is characterized in that The laser stimulation assembly further includes a first aperture stop, a beam reducer, a second aperture stop and a reflector which are sequentially arranged between the laser source and the two-dimensional galvanometer, and a dichroic mirror which is arranged between the two-dimensional galvanometer and the multi-dimensional treadmill.
6. The real-time motion monitoring and control system according to claim 1 or 2, It is characterized in that The first camera array includes a first positioning camera having a first resolution, a second positioning camera having a second resolution, and a spectroscope, wherein the first resolution is smaller than the second resolution; The first image includes an image taken by the first positioning camera and an image taken by the second positioning camera, and the processing device detects the key points based on the image taken by the second positioning camera.
7. The real-time motion monitoring and control system according to claim 1 or 2, It is characterized in that Also included is a physical and chemical stimulation component, which is configured to perform physical and chemical stimulation on the target motion object based on the scene.
8. The real-time motion monitoring and control system according to claim 1 or 2, It is characterized in that The moving object is any one of an animal, a movable facility and a robot.
9. A real-time motion monitoring and control device for a moving object, It is characterized in that include: A multi-dimensional treadmill configured to allow a plurality of exercise subjects to freely move thereon; a first camera array configured to capture first images of the plurality of moving objects; A second camera array, wherein a plurality of cameras in the second camera array are arranged around the multi-dimensional treadmill in a circumferential direction, and synchronously capture a plurality of second images of the plurality of moving objects in real time from different lateral directions; A processing device configured to: receiving the first image from the first camera array and the plurality of second images from the second camera array; reconstructing a three-dimensional model of a target moving object among the multiple moving objects based on the first image and the multiple second images; Calculating the real-time position of the target moving object in the camera field of view based on the first image; Based on the information about the real-time position of the target moving object, controlling the movement of the multi-dimensional treadmill so that the target moving object on the treadmill remains in a fixed position relative to the first camera array and the second camera array; and Detecting key points of the target moving object in the first image, wherein the key points correspond to parts of the moving object; a scene generating device, which is arranged around the activity area of the plurality of moving objects on the multi-dimensional treadmill and is configured to generate a scene for the free activity under the control of the processing device; and a laser stimulation component configured to perform laser stimulation on the key points of the target moving object in the scene under the control of the processing device; The real-time motion monitoring and control device further comprises a container for accommodating the plurality of motion objects to define the activity area, wherein the container is placed on the multi-dimensional treadmill.
10. A real-time motion monitoring control method using the real-time motion monitoring control system according to any one of claims 1 to 8, include: Using the first camera array to capture first images of the plurality of moving objects, and calculating the real-time positions of the target moving objects in the camera field of view according to the first images; Based on the information about the real-time position, controlling the movement of the multi-dimensional treadmill so that the target moving object thereon remains in a fixed position relative to the first camera array; Detecting key points of the target moving object in the first image, wherein the key points correspond to parts of the target moving object; generating a scene for the free movement of the plurality of moving objects; Using a second camera array to synchronously capture multiple second images of the target moving object in real time from different lateral directions; reconstructing a three-dimensional model of the target moving object based on the first image and the plurality of second images; and In the scene, laser stimulation is performed on the key points.
