Self-adaptive following intelligent treadmill based on real-time computer vision and control method

Abstract

The invention provides a self-adaptive following intelligent treadmill based on real-time computer vision and a control method, and provides a solution for promoting intelligence of experimental equipment and development of new-generation fitness and rehabilitation auxiliary training equipment. Comprising a positioning tracking method for label detection of a basic color block or April tag and a positioning tracking method for label-free detection based on a moving object body feature training model, the position of an object in a treadmill area is judged by collecting and detecting the image position of the color block or April tag in real time, or moving object body feature image data is directly imported into the model; the position of a moving object is identified and detected through a model, a conforming target object is judged through cyclic data flow, target object features are drawn, the deviation between the center of the target object and the center of an image is calculated, and motion parameters are dynamically controlled to adjust the position of the moving object. According to the invention, the position of the human body / experimental animal can be automatically tracked, completely free movement in a limited movement space is realized, and the gait behavior of the moving object can be stably observed for a long time in a free state.

Description

Technical Field

[0001] The present application relates to the fields of intelligent experimental instruments and rehabilitation and health medical equipment technology, and more specifically to an adaptive following intelligent treadmill based on real-time computer vision, a control method and an electronic device. Background Art

[0002] The treadmill is one of the most commonly used devices in animal gait behavior tracking research and is widely used in fields such as neuroscience, rehabilitation medicine, and sports biology. With the rapid development of computer vision technology and automatic control technology, as well as the continuous emergence of new functional microelectronic products, these technologies are changing traditional research methods and promoting deep changes in the automation and intelligent design of animal treadmills. The introduction of these new technologies not only improves experimental accuracy, but also expands the dimensions of animal behavior research.

[0003] At present, the design of most animal treadmills is relatively straightforward, and researchers are usually required to preset a fixed speed and direction before the experiment. In this case, the animal's movement behavior usually relies on the passive regulation of the treadmill motor speed, which makes the animal's gait behavior show significant passive characteristics, limiting the study of its autonomous movement intention and coordination ability; especially in injury-induced models, animals with impaired motor ability may find it difficult to maintain stable and continuous movement output. In order to achieve stable gait tracking during exercise, it is usually necessary to restrict the animal to a specific support. This support method induces completely involuntary movement behavior and limits the exploration of functions such as movement intention, movement planning and regulation ability.

[0004] Gait behavior tracking technology based on computer vision is becoming an important part of animal treadmill design. However, in practical applications, the gait changes and dynamic adjustments of animal posture during movement will cause significant deformation, especially during fast or irregular movements, and tracking errors caused by deformation are difficult to avoid. Summary of the invention

[0005] The present application provides an adaptive following intelligent treadmill and control method based on real-time computer vision, which can adapt to changes in the movement speed and direction of a moving object, dynamically adjust the treadmill motion parameters, actively synchronize with the movement rhythm of the human body / experimental animals and automatically follow the movement position of the moving object, so as to achieve completely free movement in a limited movement space, and further achieve long-term and stable observation of the gait behavior of the moving object in a free state.

[0006] In the first aspect, the present application provides an adaptive following intelligent treadmill based on real-time computer vision, including: a real-time closed-loop tracking control method for a treadmill based on a color block or April tag; and a real-time closed-loop tracking control method for a treadmill based on a pre-trained model of object surface features; through the real-time closed-loop tracking control method for a treadmill based on a color block or April tag, real-time acquisition and detection of color blocks or April tag image position, judge the position of the moving object in the preset area of ​​the treadmill; through the label-free detection method based on the surface features of the object, import the image data of the body features of the moving object into the trained visual tracking pre-trained model based on the surface features of the object, and identify and detect the position information of the moving animal through the visual tracking model; through the cyclic data flow, judge in real time whether the image data contains a target object that meets the preset threshold according to the moving object information detected by the visual tracking model; if so, draw the preset features of the target object according to the target object in the image data, if not, return to re-collect the image data of the preset area of ​​the treadmill; according to the preset features of the target object, calculate the deviation value between the center of mass of the target object and the center of the image data; according to the output of the deviation value, dynamically control the motion parameters of the treadmill to adjust the moving object to the preset position of the treadmill. In an optional scheme of the first aspect, before collecting the image data of the preset area of ​​the treadmill, the method includes: initializing the camera unit; setting the color format of the collected image; setting the resolution of the collected image; setting the window area of ​​the collected image; skipping the image frame of the preset time, and turning off the automatic gain and automatic white balance; and adjusting the light unit to the preset frequency and brightness value.

[0007] In an optional scheme of the first aspect, the training process of the visual tracking model based on color blocks or April tags or surface features of biological bodies includes: generating a training set based on the preset processed moving object training image data; using FOMO MobileNetV2 as the model architecture and using the Edge Impulse platform to continuously train the training set, and the training process includes forward propagation and back propagation and updating weights; performing performance evaluation on the trained model, and then generating a visual tracking model based on color blocks or April tags or surface features of biological bodies that meets preset requirements; and integrating the visual tracking model and deploying it to a preset object.

[0008] In an optional scheme of the first aspect, when performing preset processing on the training image data of the moving object, the method includes: standardizing all environmental factors, the standardization processing including one or more of lighting, camera unit angle and field of view; setting a preset background color for the environment where the moving object is located; collecting the image data of the moving object and marking the border of the target moving object in the image data; and using the image data as the training image data of the moving object, thereby comprising a data set of a training set and a test set.

[0009] In an optional scheme of the first aspect, when performing performance evaluation on a trained model, the method includes: inputting a test set into the trained model; calculating the F1 score of the model; calculating the accuracy of the model; analyzing the error classification results of the model to understand the impact of the errors on actual task performance; and looping the training and performance evaluation process until the model meets preset requirements and sets the model as a visual tracking model based on color blocks or April tags and organisms.

[0010] In an optional solution of the first aspect, when performing model training, at least one independent model is set for each moving object in different environments and the independent model is trained separately.

[0011] In an optional scheme of the first aspect, the method for setting the preset threshold includes: defining a color threshold range for target detection; defining an area threshold range for target detection; and setting a preset function to select a target of a preset size from multiple detected targets.

[0012] In an optional scheme of the first aspect, when drawing the preset features of the target object, the method includes: drawing the edge of the target object; drawing the main axis of the target object; drawing the secondary axis of the target object; drawing a rectangular bounding box of the target object; and drawing the center point of the target object.

[0013] In an optional scheme of the first aspect, when drawing the preset features of the target object, the method also includes: drawing a first preset graphic at the center point of the target object to visualize the center of the target object; drawing a second preset graphic at the center point of the target object to display the exact position of the target object; and displaying label information on the target object.

[0014] In an optional solution of the first aspect, when calculating the deviation value between the center of mass of the target object and the center of the image data, the method includes: calculating the center coordinates of the image data: Wherein, W and H are the width and height of the image data, and (x0, y0) are the center coordinates of the image data; according to the preset features of the target object, the center of mass coordinates of the target object are calculated: Among them, (x min,y min ) is the upper left corner of the target bounding box, (x max ,y max ) is the lower right corner of the target bounding box, (x c ,y c ) is the coordinate of the target mass center; and calculates the deviation between the mass center of the target object and the image data center:

[0015] Δx=x c -x0, Δy=y c -y0, where Δx and Δy are the deviation values ​​in the horizontal and vertical directions respectively.

[0016] In an optional solution of the first aspect, when dynamically controlling the motion parameters of the treadmill, the method includes:

[0017] Use the PID control method to dynamically generate adjustment values ​​for the treadmill speed and direction based on the deviation value:

[0018] Among them, u adjust (t) is the adjustment amount of the treadmill, K p , K i , K d are proportional gain, integral gain and differential gain respectively; judging the movement direction of the treadmill according to the adjustment amount of the treadmill; and converting the movement direction and the adjustment amount into a treadmill control signal to dynamically adjust the moving object to the preset position of the treadmill: v adjust = k·|u adjust (t)|, where v adjust is the adjusted speed of the treadmill motor, and k is the speed proportional coefficient.

[0019] In a second aspect, the present application provides an adaptive following intelligent treadmill based on real-time computer vision using the control method, comprising: a camera unit, used to collect image data of a moving object in a preset area of ​​the treadmill; a lighting unit, used to irradiate light of a preset frequency and brightness value to the preset area of ​​the treadmill; a control unit, which is respectively connected to the camera unit and the motion unit, and the control unit comprises: an image processing module, used to collect and detect the position of the color block or April tag and the image of the organism in real time through a label-free treadmill real-time closed-loop tracking control method based on color blocks or April tags and surface features of the organism, and judge the position of the moving object in the preset area of ​​the treadmill; through a label-free detection method based on the surface features of the object, the image data of the body features of the moving object is imported into the trained color block or April tag based real-time closed-loop tracking control method. A visual tracking model for detecting a tag or surface features of an organism; a target recognition module for determining whether the image data contains a target object that meets a preset threshold value based on the information of the moving object detected by the visual tracking model; if so, drawing the preset features of the target object based on the target object in the image data; if not, returning to re-collect image data of a preset area of ​​a treadmill; a deviation calculation module for calculating the deviation value between the center of mass of the target object and the center of the image data based on the preset features of the target object; and a dynamic control module for dynamically controlling the motion parameters of the treadmill based on the output of the deviation value to adjust the moving object to the preset position of the treadmill.

[0020] In a third aspect, the present application provides a terminal device, which includes a memory and a processor, wherein the memory stores a computer program that can be run on the processor, and when the processor executes the computer program, the above-mentioned adaptive following intelligent treadmill control method based on real-time computer vision is implemented.

[0021] It should be understood that the foregoing general description and the following detailed description are exemplary only and are not restrictive of the present application. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] The accompanying drawings, which are incorporated herein and form a part of the specification, illustrate one or more embodiments of the present application and, together with the description, serve to explain the principles of the present application and to enable one of ordinary skill in the relevant art to make and use the present application.

[0023] Figure 1 This is a schematic diagram of the hardware structure of an exemplary smart treadmill according to some embodiments of the present application.

[0024] Figure 2 This is a schematic diagram of module connections of an exemplary smart treadmill according to some embodiments of the present application.

[0025] Figure 3is a circuit connection diagram of an exemplary driver module according to some embodiments of the present application.

[0026] Figure 4 This is a tracking simulation diagram of an exemplary smart treadmill according to some embodiments of the present application.

[0027] Figure 5 This is a flowchart of an exemplary intelligent treadmill control method according to some embodiments of the present application.

[0028] Figure 6 It is a flowchart of an exemplary model training method according to some embodiments of the present application.

[0029] Figure 7 This is an experimental schematic diagram of an exemplary model training according to some embodiments of the present application.

[0030] Figure 8 It is a flowchart of an exemplary method for drawing target features according to some embodiments of the present application.

[0031] Fig. 9 This is a connection diagram of an exemplary electronic device according to some embodiments of the present application. DETAILED DESCRIPTION

[0032] Example embodiments will now be described more fully with reference to the accompanying drawings. However, example embodiments can be implemented in a variety of forms and should not be construed as limited to the examples set forth herein; rather, these embodiments are provided so that the present application will be more comprehensive and complete and fully convey the concepts of the example embodiments to those skilled in the art. The described features, structures, or characteristics may be combined in any suitable manner in one or more embodiments. In the following description, many specific details are provided to provide a full understanding of the embodiments of the present application.

[0033] Based on the existing treadmill design, only the mechanical structure is modified, so its functionality does not reflect a high practical value and advantage. With the development of intelligent technology, the intelligent technology is applied to rehabilitation-assisted training and experimental animal treadmills. The intelligent transformation can not only improve the functionality and automation level of the equipment, but also monitor the movement status of experimental animals or patients in real time through sensors and computer vision technology, such as speed, gait, posture, etc. These precise motion data are of great significance for research analysis and the formulation of personalized training plans. In scientific research, the intelligent treadmill can realize automatic monitoring and data recording, reduce manual intervention, reduce errors, and greatly improve experimental efficiency and data quality; in rehabilitation training, through built-in algorithms and intelligent technologies, the speed is automatically adjusted according to the movement status or fatigue level, or other training parameters are automatically reduced to prevent accidents or injuries, which can better meet the needs of different individuals and realize personalized training design, and improve training effects.

[0034] Therefore, the present application first briefly introduces the existing treadmill.

[0035] A treadmill is a type of sports equipment widely used in fitness, medical treatment and rehabilitation. When used in the fields of fitness and rehabilitation, the treadmill helps users perform aerobic exercise or specialized training by simulating a walking or running environment. When used in the gait behavior tracking study of moving animals, it is used to monitor and analyze the animal's movement behavior, biomechanical characteristics and neuromuscular function to complete research in neuroscience, kinematics, drug development and rehabilitation medicine.

[0036] The smart treadmill 1 of the present application can be applied to the fields of fitness and rehabilitation, and can also be applied to animal gait behavior tracking research. The following description will be based on the field of animal gait behavior tracking research. If it is necessary to apply it to other moving objects such as human bodies, the training object can be replaced with the corresponding moving object image data.

[0037] Therefore, reference Figure 1 and Figure 2 As shown, Figure 1 is a schematic diagram of the hardware structure of an exemplary smart treadmill according to some embodiments of the present application, Figure 2This is a schematic diagram of module connections of an exemplary smart treadmill according to some embodiments of the present application. The structure of the intelligent treadmill 1 of the present application includes a camera unit 10, a lighting unit 11, a driver 12, a motor 13, a conveyor belt 14, a control unit 15 and a power supply 16. The camera unit 10 adopts a high-precision camera device, usually including an infrared camera or a high-speed camera, to record the image data of the moving object in the preset area of ​​the treadmill 1, capture the landing position and gait characteristics of each foot, and then analyze the gait parameters in combination with the tracking technology of the present application; the lighting unit 11 is used to irradiate the preset area of ​​the treadmill 1 with light of a preset frequency and brightness value; the driver 12 is used to drive the rotation direction and speed of the motor 13 of the treadmill 1; the motor 13 is used to provide driving force for the conveyor belt 14, and usually the motor 13 structure of a treadmill of a common model style is adopted by researchers according to needs; the conveyor belt 14 is used to provide movement of the moving object; the control unit 15 is respectively connected to the camera unit 10, the lighting unit 11, the driver 12, the motor 13 and the electronic device 2, and is used to dynamically control the movement parameters of the treadmill 1 to adjust the moving object to the preset position of the treadmill 1.

[0038] Specifically, the present application exemplarily adopts OpenMV as an embedded vision platform to significantly reduce the complexity of system design and ensure the stability of the system; the preset area of ​​the specific treadmill 1 is arbitrarily adjusted by the researchers according to actual needs; the preset frequency and brightness value of the specific light unit 11 are arbitrarily adjusted by the researchers according to actual needs. In the example of the present application, a PWM signal frequency of 50kHz and 50% brightness can be exemplarily referenced; specifically, since OpenMV is used as an embedded vision platform in the present application, its output current is limited and it cannot directly drive the treadmill motor 13. For example, the present application refers to an H-bridge DC motor driver module with high power output capacity, such as Figure 3 As shown, Figure 3 A schematic diagram of an exemplary H-bridge DC motor driver module of some embodiments of the present application is shown, in which: A1.A2=0.0 is used for brake, A1.A2=1.0 is used for forward rotation, A1.A2=0.1 is used for reverse rotation, PWM is used as the input signal, and the motor 13 is driven to adjust the speed of the motor 13 under the control of a 15K square wave, and a 5V voltage is used to power the module.

[0039] The sports object of the present application uses SD rats as the object for model training; if it needs to be applied to other animals or humans, the type of sports object is modified accordingly for model training.

[0040] refer to Figure 4 and Figure 5 As shown, Figure 4 is a tracking simulation schematic diagram of an exemplary smart treadmill according to some embodiments of the present application, Figure 5is a flow chart of an exemplary intelligent treadmill control method according to some embodiments of the present application. The present application also relates to a control method of an adaptive following intelligent treadmill 1 based on real-time computer vision, the method comprising:

[0041] S1: Importing the image data of the preset area of ​​the treadmill 1 collected in real time into the trained visual tracking model based on color blocks or April tags or biological surface features.

[0042] Specifically, in S1, before collecting image data of a preset area of ​​the treadmill 1, the method further includes:

[0043] The camera unit 10 is reset to initialize the setting parameters of the camera, ensuring that the camera unit 10 is started in a clean state.

[0044] Setting the color format of the image captured by the camera unit 10;

[0045] Among them, the specific color format is set by the researchers according to actual needs, including at least black and white format and color format. For example, in this application, the image format of the camera unit 10 is set to RGB565, which occupies 16 bits per pixel (5 bits of red, 6 bits of green, and 5 bits of blue) to provide higher color fidelity while maintaining lower data storage requirements.

[0046] Setting the resolution of the image captured by the camera unit 10;

[0047] Among them, the specific resolution is set by the researchers according to actual needs. For example, in this application, the acquisition resolution of the camera unit 10 is set to QVGA (320x240), which provides a good balance between performance and data volume and is suitable for real-time processing requirements of embedded devices.

[0048] Setting a window area for the camera unit 10 to capture images;

[0049] Among them, the specific window area is set by the researchers according to actual needs. For example, in this application, the effective field of view of the camera unit 10 is cropped to the 240x240 area in the center of the image. This window area is used to reduce unnecessary background interference in the periphery, so that the subsequent visual tracking model pays more attention to the target area, thereby improving the detection efficiency.

[0050] Skip the image frames for a preset time and turn off the automatic gain and automatic white balance;

[0051] Among them, the specific preset time is set by the researchers according to actual needs. For example, in this application, the preset time is set to 2000 milliseconds, that is, the camera unit 10 skips the image frames of the first 2000 milliseconds after initialization to ensure that after adjusting the automatic settings such as brightness and contrast, the camera unit 10 can provide stable and reliable subsequent image input; at the same time, the self-gain of the camera unit 10 is turned off to prevent changes in illumination from affecting the recognition effect; at the same time, the automatic white balance of the camera unit 10 is turned off to prevent the automatic adjustment of white balance from affecting color recognition.

[0052] Adjust the light unit 11 to a preset frequency and brightness value;

[0053] Among them, the specific preset frequency and brightness value are arbitrarily adjusted by researchers according to actual needs. In the example of this application, the LED lamp can be set as an exemplary reference, and the PWM signal frequency of the LED lamp is 50kHz and the brightness is 50%.

[0054] For example, the following pseudo code can be used for illustration:

[0055] sensor.reset()

[0056] sensor.set_pixformat(sensor.RGB565)

[0057] sensor.set_framesize(sensor.QVGA)

[0058] sensor.skip_frames(time=2000)

[0059] sensor.set_auto_gain(False)

[0060] sensor.set_auto_whitebal(False)

[0061] light=Timer(2,freq=50000).channel(1,Timer.PWM,pin=Pin("P6"))

[0062] light.pulse_width_percent(50)

[0063] Specifically, refer to Figure 6 As shown, Figure 6 is a flowchart of an exemplary model training method according to some embodiments of the present application. In S1, the training process of the visual tracking model based on color blocks or April tags or biological surface features includes:

[0064] S10: generating a training set according to the preset processed moving object training image data;

[0065] Specifically, in S10, all environmental factors are standardized, and the standardized processing includes one or more of illumination, camera unit 10 angle and field of view to ensure the consistency between the training environment and the actual conditions. A preset background color is set for the environment where the moving object is located. The preset background color is set by the researchers according to actual needs, and the background color is avoided to be the same as the color of the color block or April tag, including but not limited to color, black, white, etc. For example, in this application, the background color is set to black. The image data of the moving object is collected and the bounding boxes are marked for the target moving object in the image data to calibrate the target object. The image data is used as the training image data of the moving object, and then the data set includes a training set and a test set. The number of training image data of the moving object is set by the researchers according to actual needs. For example, in this application, the reference to the collection of moving object image data is 1000 images, of which 800 are used for training and 200 are used for testing.

[0066] S11: FOMO MobileNetV2 is used as the model architecture and the Edge Impulse platform is used to continuously train the training set. The training process includes forward propagation and backpropagation and updates the weights.

[0067] Specifically, in step S11, FOMO MobileNetV2 is a MobileNetV2 variant optimized for mobile devices and resource-constrained environments. Compared with the standard MobileNetV2, it is more compact, requires significantly less computing and memory resources, and is suitable for real-time single target recognition. Although there is some compromise in accuracy, the model is not only used for object classification, but also provides detailed information about the location, size and number of target objects, which is crucial for object detection tasks. Exemplarily, FOMO MobileNetV2 0.35 is selected as the model architecture in this application. FOMOMobileNetV2 0.35 has a relatively large width multiplier compared to FOMO MobileNetV2 0.1, which helps to improve the accuracy of the model while maintaining computational efficiency. Edge Impulse is a platform designed for optimizing machine learning models for embedded systems and edge devices.

[0068] Thus, the training process includes:

[0069] Set parameters such as learning rate, batch size, loss function, and optimizer. Use a learning rate scheduler to dynamically adjust according to the training process; select a larger batch size to stabilize gradient updates if computing resources allow; use the default FOMO loss function, which combines classification loss and position loss; and use the Adam optimizer;

[0070] Forward propagation: Use the convolution layer of MobileNetV2 to extract features and complete the convolution operation. Use the loss function to calculate, emphasize the target points that are difficult to classify, and reduce the interference of easy-to-classify samples on model optimization;

[0071] Back propagation: Derivative the loss function with respect to the network weights, calculate the gradient, and according to the chain rule, calculate the gradient layer by layer and propagate it back from the output layer to the input layer;

[0072] Weight update: Use the Adam optimizer to update weights, combined with adaptive learning rate and momentum;

[0073] The model loss is gradually converged by cyclic forward propagation, backpropagation, and weight update until the number of iterations set by the researchers is reached; among them, the model checkpoint is saved every 10 epochs to prevent training interruption or later fine-tuning.

[0074] S12: Evaluate the performance of the trained model.

[0075] Specifically, in step S12, a test set of 200 annotated images is used to evaluate the model performance, the F1 score and accuracy of the model are calculated, and the misclassification results of the model are analyzed; specifically, the F1 score is used to evaluate the performance of the model, which is a common indicator for balancing precision and recall. Exemplarily, the threshold of the F1 score is set to 95% in the present application, that is, only when the F1 score of the model exceeds the threshold, it is considered qualified for deployment; exemplarily, in the present application, the accuracy of the model is set to exceed 95% to prove that it can work effectively in actual tasks; after training, the researchers analyze the misclassification of the model to understand the impact of these errors on the actual task performance; and then adjust the model hyperparameters (such as learning rate, regularization strength) and architecture (such as depth or width multipliers) according to the performance evaluation.

[0076] S13: Generate a visual tracking model based on color blocks or April tags or biological surface features that meets preset requirements.

[0077] The training and evaluation process is repeated until the number of iterations set by the researchers is reached or the performance evaluation meets the requirements set by the researchers. The model that meets the requirements is set as a visual tracking model based on color blocks or April tags.

[0078] S14: Integrate the visual tracking model and deploy it to a preset object.

[0079] Specifically, in step S13, the visual tracking model based on color blocks or April tags or biological surface features that has been trained to meet performance requirements is exported to a format that is compatible with embedded hardware (such as TensorFlow Lite, ONNX), and then deployed to the target object device specified by the researcher.

[0080] Furthermore, during the model training process, at least one independent model may be set for each moving object in different environments and the independent model may be trained separately.

[0081] Therefore, reference Figure 7 As shown, Figure 7 It is an experimental schematic diagram of an exemplary model training according to some embodiments of the present application. The exemplary model training is:

[0082] Experimental subjects: SD rats were used in the experiment;

[0083] Training models for different environments: Each rat has an independently trained model for its specific environment, which is important to account for the effects of different environmental factors on model performance;

[0084] Environmental standardization: To ensure consistency between the training environment and actual conditions, the researchers standardized all environmental factors, including: lighting, camera angle, and field of view;

[0085] Optimal background: For models used in a polycarbonate box environment, testing has shown that a white or black background provides the best performance, so black was chosen as the background color for the model under this condition.

[0086] S2: Based on the moving object information detected by the visual tracking model, determine whether the image data contains a target object that meets a preset threshold.

[0087] Specifically, in S2, the method for setting the preset threshold includes:

[0088] Define a color threshold range for target object detection; wherein the color threshold range is set by researchers according to actual needs. For example, in this application, reference is made to red and the color threshold range is set to a value suitable for the red area.

[0089] For example, the following pseudo code can be used to illustrate:

[0090] threshold=[(0,100,107,26,-48,127)]

[0091] size_threshold=2000

[0092] Define the area threshold range for target detection; the area threshold range is set by researchers according to actual needs to filter out smaller color blocks and prevent noise interference. For example, 2000 square millimeters can be used as a reference in this application.

[0093] A preset function is set to select a target of preset size from among the multiple detected targets; the function is set by researchers according to actual needs, and is used to select the target with the largest area from among the multiple detected targets. When judging, the width of the target is multiplied by the height to obtain the area size; by selecting the target with the largest area, incorrect decisions can be avoided due to interference from secondary objects (smaller color blocks). The target can be a color block, April tag, barcode, etc., which is set by researchers according to actual needs.

[0094] For example, the following pseudo code can be used to illustrate:

[0095]

[0096] S3: If yes, draw the preset features of the target object according to the target object in the image data; if no, return to re-collect the image data of the preset area of ​​the treadmill 1.

[0097] Specifically, refer to Figure 8 As shown, Figure 8 is a flow chart of an exemplary method for drawing target features according to some embodiments of the present application. In S3, when drawing the preset features of the target, the method includes:

[0098] S30: Draw the edge of the target object; wherein the minimum circumscribed polygon of the target object is drawn and marked with a set color, which may be red in this application.

[0099] S31: Draw the main axis of the target object; wherein the main axis of the target object is drawn and marked with a set color, which may be green in this application.

[0100] S32: Draw a secondary axis line of the target object; wherein the secondary axis line of the target object is drawn and marked with a set color, which can be blue in this application.

[0101] S33: Draw a rectangular bounding box of the target object.

[0102] S34: Draw the center point of the target object; wherein the center point of the target object is drawn and marked with a set shape, which may refer to a cross in this application.

[0103] For example, the following pseudo code can be used to illustrate:

[0104]

[0105] In some examples of the present application, when drawing the preset features of the target object, the method further includes:

[0106] Draw a first preset graphic at the center point of the target object to visualize the center of the target object;

[0107] Exemplarily, a circle is drawn at the center of the detected target to visualize the center of the target.

[0108] Draw a second preset graphic at the center point of the target object to show the exact position of the target object;

[0109] Exemplarily, a circle is drawn at the center of the detected target to visualize the exact location of the target.

[0110] Display label information on the target object; wherein the label information may be the name of the target object.

[0111] In some embodiments, one or more display units may be provided for the treadmill 1 to display the processed image data in real time.

[0112] S4: Calculate the deviation between the center of mass of the target object and the center of the image data according to the preset features of the target object.

[0113] Specifically, in S4, when calculating the deviation value between the center of mass of the target object and the center of the image data, the method includes:

[0114] Calculate the center coordinates of the image data: Where W and H are the width and height of the image data, and (x0, y0) are the center coordinates of the image data;

[0115] According to the preset features of the target object, the center of mass coordinates of the target object are calculated:

[0116] Among them, (x min ,y min ) is the upper left corner of the target bounding box, (x max ,y max ) is the lower right corner of the target bounding box, (x c ,y c ) are the coordinates of the target mass center; and

[0117] Calculate the deviation between the center of mass of the target object and the center of the image data:

[0118] Δx=x c-x0, Δy=y c -y0, where Δx and Δy are the deviation values ​​in the horizontal and vertical directions respectively.

[0119] S5: According to the output of the deviation value, the motion parameters of the treadmill 1 are dynamically controlled to adjust the moving object to a preset position of the treadmill 1 .

[0120] Among them, the preset position is set by researchers according to actual needs, and is at least the effective movement area of ​​the conveyor belt 14 of the treadmill 1. For example, in this application, reference is made to the center position of the conveyor belt 14 of the treadmill 1.

[0121] Specifically, in S5, when dynamically controlling the motion parameters of the treadmill 1, the method includes:

[0122] Use the PID control method to dynamically generate adjustment values ​​for the speed and direction of treadmill 1 based on the deviation value:

[0123] Among them, u adjus (t) is the adjustment amount of treadmill 1, K p , K i , K d They are proportional gain, integral gain and differential gain respectively. The proportional gain determines the direct amplification effect of the error, the integral gain is used to eliminate the cumulative error, and the differential gain is used to suppress the rapid fluctuation of the error change. Therefore, the proportional gain, integral gain and differential gain are determined by the researchers based on actual experimental debugging to obtain the best control effect.

[0124] The movement direction of the treadmill 1 is determined according to the adjustment amount of the treadmill 1; wherein, when u adjust (t)>0, the motor 13 of the treadmill 1 rotates clockwise. adjust When (t)<0, the motor 13 of the treadmill 1 rotates counterclockwise.

[0125] The movement direction and adjustment amount are converted into a treadmill 1 control signal, and the moving object is dynamically adjusted to a preset position of the treadmill 1: adjust = k·|u adjust (t)|, where v adjust is the adjusted speed of the motor 13 of the treadmill 1, k is the speed proportional coefficient, and the specific k is adjusted by the researchers according to the equipment performance.

[0126] In some embodiments, the output of the adjustment amount can be reduced by a preset value to slowly adjust the speed of the treadmill 1 motor 13 to avoid excessive single adjustment amount that affects the moving object; the specific preset value is set by the researchers according to actual needs. In this embodiment, the adjustment amount can be reduced by one third each time.

[0127] Since the conveyor belts 14 of different treadmills 1 have different resistances, during actual use, researchers need to adjust the PID adjustment amount and test the smoothness of the operation of the motor 13.

[0128] In some embodiments, the entire detection process and treadmill 1 control process of the present application can be continued in a finite or infinite loop, and new image data is acquired and processed in each loop until the loop ends; the specific number of loops is set by the researcher according to actual needs.

[0129] In some embodiments, reference Figure 2 The present application relates to an adaptive following intelligent treadmill 1 based on real-time computer vision using the above control method, and its control unit 15 includes:

[0130] An image processing module 150 is used to import the image data of a preset area of ​​the treadmill 1 collected in real time into a trained visual tracking model based on color blocks or April tags;

[0131] The target recognition module 151 is used to determine whether the image data contains a target object that meets a preset threshold value according to the moving object information detected by the visual tracking model; if so, draw the preset features of the target object according to the target object in the image data; if not, return to re-collect the image data of the preset area of ​​the treadmill 1;

[0132] A deviation calculation module 152, used for calculating the deviation value between the center of mass of the target object and the center of the image data according to the preset features of the target object;

[0133] The dynamic control module 153 is used to dynamically control the motion parameters of the treadmill 1 according to the output of the deviation value so as to adjust the moving object to a preset position of the treadmill 1 .

[0134] In some embodiments, reference Fig. 9 As shown, Fig. 9 The connection diagram of an electronic device 2 used to implement an embodiment of the present application is shown. The electronic device 2 includes: a memory 201 and a processor 202. The memory 201 stores a computer program that can be run on the processor 202. When the processor 202 executes the computer program, the method in the above embodiment is implemented. The number of the memory 201 and the processor 202 can be one or more.

[0135] The electronic device 2 also includes:

[0136] The communication interface 203 is used to communicate with external devices and perform data exchange transmission.

[0137] If the memory 201, the processor 202 and the communication interface 203 are implemented independently, the memory 201, the processor 202 and the communication interface 203 can be connected to each other through a bus and communicate with each other. The bus can be an Industry Standard Architecture (ISA) bus, a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus. The bus can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Fig. 9 Only one thick line is used in the diagram, but this does not mean that there is only one bus or only one type of bus.

[0138] Optionally, in a specific implementation, if the memory 201, the processor 202 and the communication interface 203 are integrated on a chip, the memory 201, the processor 202 and the communication interface 203 can communicate with each other through an internal interface.

[0139] An embodiment of the present application provides a computer-readable storage medium storing a computer program, which implements the method provided in the embodiment of the present application when the program is executed by the processor 202.

[0140] The embodiment of the present application also provides a chip, which includes a processor 202, which is used to call and execute instructions stored in the memory 201 from the memory 201, so that a communication device equipped with the chip executes the method provided by the embodiment of the present application.

[0141] The embodiment of the present application also provides a chip, including: an input interface, an output interface, a processor 202 and a memory 201. The input interface, the output interface, the processor 202 and the memory 201 are connected through an internal connection path. The processor 202 is used to execute the code in the memory 201. When the code is executed, the processor 202 is used to execute the method provided in the embodiment of the application.

[0142] It should be understood that the processor 202 may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or any conventional processor, etc. It is worth noting that the processor 202 may be a processor supporting an Advanced RISC Machines (ARM) architecture.

[0143] Further, the above-mentioned memory 201 may include a read-only memory and a random access memory, and may also include a non-volatile random access memory. The memory may be a volatile memory or a non-volatile memory, or may include both volatile and non-volatile memories. Among them, the non-volatile memory may include a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), or a flash memory. The volatile memory may include a random access memory (RAM), which is used as an external cache. By way of exemplary but not limiting description, many forms of RAM are available. For example, static random access memory (SRAM), dynamic random access memory (DRAM), synchronous dynamic random access memory (SDRAM), double data rate synchronous dynamic random access memory (DDR SDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synchronous link dynamic random access memory (SLDRAM) and direct memory bus random access memory (DR RAM).

[0144] In the above embodiments, it can be implemented in whole or in part by software, hardware, firmware or any combination thereof. When implemented using software, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the process or function according to the present application is generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable devices. The computer instructions can be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium.

[0145] The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any person skilled in the art who is familiar with the present technical field can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

Claims

1. A method for controlling an adaptive following intelligent treadmill based on real-time computer vision, characterized in that: The method includes: a real-time closed-loop tracking control method of a treadmill based on a color block or an April tag and a real-time closed-loop tracking control method of a treadmill based on a pre-trained model of object surface features; Through the real-time closed-loop tracking control method of the treadmill based on color blocks or Apriltag labels, the color block or April tag image position is collected and detected in real time to determine the position of the moving object in the preset area of ​​the treadmill; Through a label-free detection method based on the surface features of objects, the image data of the moving object’s body features are imported into the trained visual tracking model; Identify and detect the position information of the moving object through the visual tracking model; Determining in real time through a cyclic data stream whether the image data contains a target object that meets a preset threshold; If yes, then draw the preset features of the target object according to the target object in the image data; if no, return to re-collect the image data of the preset area of ​​the treadmill; Calculating a deviation between the center of mass of the target object and the center of the image data according to preset features of the target object; and According to the output of the deviation value, the motion parameters of the treadmill are dynamically controlled to adjust the moving object to a preset position of the treadmill.

2. The control method according to claim 1, characterized in that: Before collecting image data of a preset area of ​​the treadmill, the method includes: Initialize the camera unit; Set the color format of the acquired image; Set the resolution of the acquired image; Set the window area for collecting images; Skip image frames for a preset time and turn off automatic gain and automatic white balance; and Adjust the light unit to the preset frequency and brightness value.

3. The control method according to claim 1 or 2, characterized in that: The training process of the label-free detection visual tracking model based on color blocks or April tags and the pre-trained model based on the surface features of objects includes: Generate a training set based on the preset processed moving object training image data; Using FOMO MobileNetV2 as the model architecture and using the Edge Impulse platform to continuously train the training set; Evaluate the performance of the trained model to generate a visual tracking model based on color blocks or April tags and surface features of biological organisms that meets the preset requirements; and The visual tracking model is integrated and deployed to a preset object.

4. The control method according to claim 3, characterized in that: When performing preset processing on the moving object training image data, the method includes: Standardizing all environmental factors, including one or more of lighting, camera unit angle, and field of view; Set a preset background color for the environment where the moving object is located; Collecting image data of a moving object and marking a frame of the target moving object in the image data; and The image data is used as moving object training image data, thereby forming a data set including a training set and a test set.

5. The control method according to claim 4, characterized in that: When evaluating the performance of the trained model, the method includes: Input the test set into the trained model; Calculate the F1 score of the model; Calculating the accuracy of the model; Analyze the model's misclassification results to understand the impact of the errors on actual task performance; and The training and performance evaluation process is repeated until the model meets the preset requirements and the model is set as a visual tracking model based on color blocks or Apriltag or biological surface features.

6. The control method according to claim 1 or 5, characterized in that: When performing model training, at least one independent model is set for each moving object in different environments and the independent model is trained separately.

7. The control method according to claim 1, characterized in that: The method for setting the preset threshold includes: Define the color threshold range for target detection; Define the area threshold range for object detection; and A preset function is set to select an object of a preset size from among the multiple detected objects.

8. The control method according to claim 1, characterized in that: When drawing the preset features of the target object, the method includes: Draw the edge of the target object; Draw the main axis of the target object; Draw the secondary axis of the target object; Draw a rectangular bounding box of the object; and Draw the center point of the target object.

9. The control method according to claim 7, characterized in that: When drawing the preset features of the target object, the method further includes: Draw a first preset graphic at the center point of the target object to visualize the center of the target object; Draw a second preset graphic at the center point of the target object to display the exact location of the target object; and Display label information on the target object.

10. The control method according to claim 1, characterized in that: When calculating the deviation value between the center of mass of the target object and the center of the image data, the method includes: Calculate the center coordinates of the image data; Calculating the center of mass coordinates of the target object according to the preset features of the target object; Calculating a deviation between the center of mass of the target object and the center of the image data; Use PID control method to dynamically generate adjustment values ​​of treadmill speed and direction according to the deviation value; determining the movement direction of the treadmill according to the adjustment amount of the treadmill; and The movement direction and adjustment amount are converted into a treadmill control signal to dynamically adjust the moving object to a preset position of the treadmill.

11. An adaptive following intelligent treadmill based on real-time computer vision using the control method according to any one of claims 1 to 10, characterized in that: include: A camera unit, used for collecting image data of moving objects in a preset area of ​​the treadmill; A lighting unit, used for irradiating a light of a preset frequency and brightness value to a preset area of ​​the treadmill; A control unit is connected to the camera unit and the motion unit respectively, and the control unit includes: The image processing module is used to collect and detect the image position of the color block or Apriltag or the surface feature of the organism in real time through a real-time closed-loop tracking control method of a treadmill based on a color block or Apriltag label or a pre-trained model of the surface feature of an organism, and judge the position of the moving object in the preset area of ​​the treadmill; through a label-free detection method based on the surface feature of the object, the image data of the body feature of the moving object is imported into the training to complete the visual tracking model based on the color block or Apriltag or the organism; A target recognition module is used to determine whether the image data contains a target object that meets a preset threshold value based on the moving object information detected by the visual tracking model; if so, draw the preset features of the target object based on the target object in the image data; if not, return to re-collect image data of the preset area of ​​the treadmill; A deviation calculation module, used for calculating the deviation value between the center of mass of the target object and the center of the image data according to the preset features of the target object; The dynamic control module is used to dynamically control the motion parameters of the treadmill according to the output of the deviation value so as to adjust the moving object to a preset position of the treadmill.

Images