A somatosensory-based multi-motor array host computer control system

Abstract

The application discloses a kind of multi-motor array host computer control systems based on somatic sensation, including angle control module, somatic sensation detection module, serial communication module, bluetooth communication module and logic control module, initialize each module and enter power-on self-test state;When system self-checking is completed, bluetooth communication module starts receiving external control information, somatic sensation control module starts detecting whether human body appears, if human body is detected, track its posture, and determine human action in combination with prior knowledge;At the same time, logic control module updates system state control word according to the output result of each module, so that the multi-motor array has multiple functions defined by the user;Finally, angle control module transmits angle data to the slave computer for control through serial communication module and data encoding.The application introduces modular processing and system state control word strategy, realizes multiple modes that can be defined by the user, and combines multi-motor array control with somatic sensation control.

Description

Technical Field

[0001] This invention belongs to the field of multi-motor array control and relates to a host computer control system for multi-motor arrays based on motion sensing. Background Technology

[0002] Multi-motor control technology has become an important development direction in my country's control field. Real-time, accurate, and efficient control of a multi-motor array is crucial for high-precision industrial manufacturing and a vital technology for realizing concepts from interactive installation art to practical tools. Achieving real-time, accurate, efficient, and intelligent control is a relatively new field. Currently, my country's multi-motor control technology is still in its developmental stage, and existing technologies have room for improvement. A mature, systematic, and sustainably maintainable system has yet to emerge in this area. Meanwhile, with the advancement of artificial intelligence technology, motion sensing technology has the potential to be integrated into production-level multi-motor array control systems. Further integration would revolutionize control methods and solutions.

[0003] The design and implementation of a polar bear interactive game based on Unity3D and Kinect 2.0: Human posture is acquired using the Kinect sensor, and a motion-sensing game for a polar bear is developed by combining it with Unity3D. This technology integrates motion sensors with games, but the following issues still exist:

[0004] 1. There is no real-time communication with the lower-level machine to achieve control of the hardware via motion sensing;

[0005] 2. Failed to achieve systematic logical switching between multiple motion-sensing control functions;

[0006] 3. Failed to achieve advanced encapsulation of the entire motion control system. Summary of the Invention

[0007] This invention aims to enrich the control methods for multi-motor arrays and improve their control efficiency and real-time performance by combining motion sensing control, Bluetooth control, and logic control methods into a three-layer host computer control protocol established by five modules. Therefore, it provides a motion sensing-based multi-motor array host computer control system. Based on motion sensing detection, the system is improved by introducing modular processing and system state control word strategies. This invention utilizes modular processing and multi-motor angle control clusters to achieve multiple user-customizable modes, combining multi-motor array control with motion sensing control.

[0008] The present invention is achieved by at least one of the following technical solutions.

[0009] A motion-sensing-based multi-motor array host computer control system includes a three-layer control architecture: a user layer, an intermediate layer, and a machine layer. The user layer includes a logic control module; the intermediate layer includes an angle control module and a motion sensing detection module; and the machine layer includes a serial communication module, a Bluetooth communication module, and the motion sensing detection module.

[0010] Users can modify and control the system's logic in the user-level logic control module;

[0011] The middle layer is used to process angle data and human bone and joint data, and transmits the processed data to the user layer for user operation. It also receives user layer instructions and query requests, and transparently transmits the data sent down to the machine layer after querying.

[0012] The machine layer receives data from the middle layer and packages the received data into frames according to the protocol. It then transmits the frames to the lower-level machine via the serial communication module. Simultaneously, it receives data transmitted by sensors or camera frames, unpacks the sensor data or performs morphological processing on the frame data, and transmits it to the middle layer for parsing and further processing.

[0013] Furthermore, the logic control module combines data from the motion detection module and the Bluetooth communication module to update the system status control word, and different system status control words call different functions to achieve the desired results.

[0014] The logic control module is the core scheduling module, using the motion detection module and Bluetooth communication module as inputs to directly control the angle control module. The system status control word consists of 10 bits (0000000000), with each bit starting from the most significant bit and having the following meanings: the first two bits are the Bluetooth control protocol control number; the third bit is the human body tracking cancellation flag; the fourth bit is the power-off confirmation flag; the fifth bit is the manual selection mode flag for the motor angle set; the sixth bit is the flag for the motor following the relative position of the human body; the seventh bit is the flag for the motor following the angle of the human arm; the eighth bit is the flag for a specific human body action triggering a specific motor angle set; the ninth bit is the flag for whether a human body has acquired control; and the last bit is the system normal operation flag.

[0015] Furthermore, the logic control module is configured with 8 modes. The system status control word defaults to 0000000001 after the power-on self-test passes. Before entering the following modes, the corresponding flag bits are set to 1 by the logic judgment module. When exiting a mode, the system status control word restores its default value:

[0016] 1) Manual selection mode for motor angle sets: The system status control word is 0000100011; When the motion detection module detects that the user's action is a right swipe, it enters the manual selection mode; In the manual selection mode, the automatic switching of motor angle sets is canceled. When the user is detected to continue to swipe right, it switches to the next motor angle set. When the motion detection module does not detect the user's next right swipe action for n consecutive seconds, it exits the manual selection mode.

[0017] 2) Motor Follow-up Tracking Human Arm Angle Mode: The system status control word is 0000001011; when the motion detection module detects that the user's action is waving, it enters the human arm angle mode. The human arm angle mode reads human joint data from the motion detection module, with the face facing forward as the Y-axis and vertical upward as the Z-axis. Using a right-hand coordinate system, two spatial vectors are constructed for the spatial coordinates of the left and right palms, elbow joints, and shoulder joints. The angles between each vector and the unit vector in the negative direction of the Z-axis are calculated, and the larger one is taken as the angle between the arm and the vertical plane, called the arm angle. The arm angle is calculated as the motor input for the motor to follow up and track the human arm angle. The calculation is as follows: Assuming N columns of motors, for the left N / 2 columns of motors: the motor angle is 180 + arm angle; for the right N / 2 columns of motors: the motor angle is 180 - arm angle. If the change in arm angle is within 10 degrees and the count exceeds 2000 times, or if the motion detection system detects that the user's action has stopped, the mode is exited.

[0018] 3) Motor-guided human body relative position mode: The system status control word is 0000010011; the motion detection module detects that after the user exits the motor-guided human arm angle mode, it automatically enters the human body relative position mode. Using the depth direction of the motion sensor as the Z-axis and the vertical direction upwards as the Y-axis, a right-handed coordinate system is used. The human body relative position mode reads the three-dimensional coordinates of the human body center from the motion detection module and takes the x-component as the offset direction. When the absolute value of the difference between the x-component of the previous frame and the x-component of the next frame is less than 0.2, it is considered that the human body has not moved significantly, and the motor does not move. Correspondingly, when the absolute value of the phase difference is greater than 0.2, it is considered that the human body is walking. At this time, the offset data is calculated and used as the motor input for the motor to follow the relative position of the human body. The calculation is as follows: the x component of the most recent frame that did not move significantly is prevMag, the x component of the current frame is nowMag, then the relative position of the human body is Mag = abs(nowMag - prevMag), and the motor angle is uniformly input as 180 + Mag * 100. When the user who currently has control walks out of the control range, this mode is automatically exited, and the system status control word is automatically restored to the default value.

[0019] 4) Specific human body movements trigger specific motor angle set mode: The system status control word is 00000001X1; when the motion detection module detects a user movement, it enters this mode. After entering this mode, the system automatically switches to the corresponding motor angle set and automatically exits after execution.

[0020] 5) Single motor angle correction mode: The system status control word is 1100000000; when the Bluetooth receiving module receives a message frame sent by the client with a single motor angle correction instruction, it enters this mode. After entering this mode, the motor returns to its encoder relative to the origin; the Bluetooth receiving module returns the instruction including the row number, column number, and correction relative value to the logic instruction module, and the logic instruction module sends the correction data to the lower-level machine for single motor angle fine-tuning correction.

[0021] 6) Cancel Control Mode: When the motion detection module detects that the user has swiped left, it triggers the cancel control mode; this cancel control mode unconditionally sets the system status control word to the default value, and the human body will lose the right to control the motion.

[0022] 7) Automatic Angle Set Rotation Mode: Automatic angle set rotation mode is the system default mode. The system status word is the inverse set of the intersection of the above system status words. In automatic angle set rotation mode, the logic control module automatically rotates the angle set according to the order of the motor array angle set loaded by the angle control module.

[0023] Furthermore, the angle control module is responsible for loading, storing, and querying the preset static multi-motor angle set, and is also responsible for real-time calculation of the dynamic multi-motor angle set sequence;

[0024] The angle control module is divided into two types: static angle control and dynamic angle control. Static angle control establishes a key-value mapping between the motor array angle set and its name. This mapping is defined by the motor angle set description file, loaded during system initialization, and stored in system memory.

[0025] When the user layer calls the static motor array angle set, the user layer function uses the name to query and calls the corresponding motor array angle set through mapping. Dynamic angle control establishes a series of coherent motor array angle set transformations, which are defined by function methods, establishing a mapping between function names and implementation methods. When the user layer function calls the dynamic motor array angle set, it uses the function name to query and calls the motor array angle set through the function to achieve dynamic transformation.

[0026] Furthermore, the motor angle set description file in the angle control module has four main elements: angle set name - motor angle set key-value pairs, ordered list of angle set names, ordered list of Boolean values ​​indicating whether it is a dynamic angle set, and delay time key-value pairs; the motor angle set description file is read and loaded into the angle control module during system initialization.

[0027] Furthermore, the motion detection module is responsible for driving and loading the Kinect motion detection sensor to detect the human body. Using a right-handed coordinate system with the human face facing forward as the Y-axis and vertical upward as the Z-axis, it tracks the detected human body and acquires the joint spatial coordinates. The acquired data is then compared with prior knowledge or a pose classification network using a multilayer perceptron to determine the final pose. The human joint and skeletal data determination modes include the following six:

[0028] 1) Heart shape: The difference between the Y-components of the spatial coordinates of the left and right palms and the Y-components of the spatial coordinates of the left and right shoulders of the human body is greater than 0 and less than 0.4; and the Y-components of the spatial coordinates of the left and right elbow joints are greater than the Y-components of the spatial coordinates of the left and right shoulders of the human body.

[0029] 2) Waving: The absolute value of the difference between the Y component of the left palm spatial coordinate and the Y component of the left shoulder joint spatial coordinate is less than 0.25, and the absolute value of the difference between the Y component of the left elbow joint spatial coordinate and the Y component of the left shoulder joint spatial coordinate is less than 0.25, and the X component of the left palm joint spatial coordinate is less than the X component of the left elbow joint spatial coordinate; or the absolute value of the difference between the Y component of the right palm spatial coordinate and the Y component of the right shoulder joint spatial coordinate is less than 0.25, and the absolute value of the difference between the Y component of the right elbow joint spatial coordinate and the Y component of the right shoulder joint spatial coordinate is less than 0.25, and the X component of the right palm joint spatial coordinate is less than the X component of the left elbow joint spatial coordinate.

[0030] 3) Stop: The difference between the Y-components of the left palm joint and the left hip joint is less than 0.15 and the difference between the Y-components of the left palm joint and the left hip joint is less than 0.15.

[0031] 4) Left swipe: The X-coordinate component of the right palm starts from the right side of the body and changes to the left side of the body within 1 second;

[0032] 5) Right swipe: The X-coordinate component of the left palm changes from the left side of the body to the right side within 1 second;

[0033] 6) Raise your right hand: The difference between the Y-coordinate component of the right palm and the right shoulder joint is greater than 0.1 and the difference between the Y-coordinate component of the left palm and the left shoulder joint is less than 0.

[0034] Furthermore, the pose determination using a multilayer perceptron is as follows: the input of the multilayer perceptron is the 17 joint key points output by the pose estimation stage, and the output is the probability of each pre-trained pose. A three-layer perceptron structure is used, consisting of an input layer, three fully connected layers, and an output layer. The user records the pose types to be detected separately, creates a dataset, and uses this dataset to train the three-layer perceptron structure. After verification, the weights are saved and loaded into the system for inference.

[0035] Furthermore, the Bluetooth communication module is used to establish a Bluetooth connection with the mobile control terminal and receive control words sent by Bluetooth as a control message queue. The Bluetooth control word is 8 bits long (0#0#0#00), using ASCII encoding. Each bit starts from the most significant bit and has the following meanings: control status, separator, row number of the motor to be calibrated, separator, column number of the motor to be calibrated, separator, and single motor calibration angle. The Bluetooth control word has the following four possible values:

[0036] 1) 1#0#0#0: Normal operation;

[0037] 2) 2#0#0#0: Enter single motor angle correction mode;

[0038] 3) 3#motor row number#motor column number#motor correction angle: Send specific single motor correction data, where the motor correction angle value ranges from -50 to +50;

[0039] 4)4#0#0#0: Enter power off lock mode.

[0040] The Bluetooth message receiving strategy is as follows: a message queue is established, and multiple threads asynchronously detect whether a Bluetooth control command is being received; when a command is received, if the command is consistent with the message at the end of the queue, the command is discarded; otherwise, a message is filled in from the end of the message queue; the logic control module retrieves the command from the head of the queue, and if the length of the command queue is not 1, the command that has been retrieved from the head of the queue is popped.

[0041] Furthermore, the serial communication module is used to establish a connection with the lower-level machine via serial communication, receive angle data input from the angle control module, perform corresponding data processing, and package the data into a data frame according to the agreed protocol. Then, the data frame is transmitted to the lower-level machine for further control. Assuming that the motor array has M rows and N columns, the data frame is defined as an M x (N+1) two-dimensional array, where the data frame of each row is defined as: motor speed level + N motor angles.

[0042] Furthermore, the logical process of packaging data frames is as follows: the serial port sending module receives data from the angle control module, puts (motor angle / 2) into a two-dimensional array according to the motor row number and column number, and then fills the speed code into the frame header; where the speed code takes values ​​of 0-70, 101, 102, 103; the speed gear algorithm is as follows: if the motor has N rows, then the motor row numbers are 1, 2, 3, ..., N., then the speed code for each row is (row number + (speed gear selection - 1) * 7); if the speed gear selection is 101, 102, 103, then it is filled in directly, and the serial port sending module will send data frames line by line.

[0043] Compared with existing technologies, the beneficial effects of the present invention are as follows:

[0044] Based on the characteristics of the application scenario, the control system of this invention abstracts business functions into five modules. These five modules are combined into three layers to form a control protocol, creating a unified standard control method. The modules are loosely coupled, highly maintainable, and allow for flexible and agile development under the standard control method. The control method of this invention employs motion sensing control, achieving contactless control while maintaining good control accuracy. It offers numerous expansion interfaces and a superior user interaction mode, representing an innovative application in multi-motor array host computer control systems. This invention compresses control words and control logic to form a communication protocol, increasing the information entropy of control signals and reducing the amount of control signal data transmitted between host and slave computers, achieving efficient real-time data transmission. This enables efficient real-time multi-motor angle control, offering significant advantages. Attached Figure Description

[0045] Figure 1 A schematic diagram of the control flow of a multi-motor array host computer control system based on motion sensing according to an embodiment of the present invention;

[0046] Figure 2 A schematic diagram of the system modules and hierarchical architecture of this invention embodiment;

[0047] Figure 3 A schematic diagram of a serial port transmission frame according to an embodiment of the present invention;

[0048] Figure 4 Simulation diagram of multi-motor array angle control according to an embodiment of the present invention;

[0049] Figure 5 A schematic diagram of a physical control system for a multi-motor array host computer based on motion sensing, according to an embodiment of the present invention;

[0050] Figure 6 This invention provides a flowchart for obtaining human posture using a monocular camera and neural network inference. Detailed Implementation

[0051] The present invention will be further described below with reference to the embodiments and the accompanying drawings.

[0052] Example 1

[0053] This invention provides a motion-sensing-based multi-motor array host computer control system, such as... Figure 1 As shown, the control architecture consists of three layers: a user layer, a middle layer, and a machine layer. The user layer includes a logic control module; the middle layer includes an angle control module and a motion detection module; and the machine layer includes a serial communication module, a Bluetooth communication module, and the motion detection module.

[0054] The angle control module is responsible for loading, storing, and querying preset static multi-motor angle sets, and also for real-time calculation of dynamic multi-motor angle set sequences. The angle control module is divided into two types: static control and dynamic control. Static angle control establishes a key-value mapping between motor array angle sets and their names, defined by a motor angle set description file, loaded during system initialization, and stored in system memory. The motor angle set description file is defined as follows: the name of the motor angle set, an array storing the corresponding motor arrangement dimensions for each angle, and a boolean value indicating whether it is a dynamic pattern.

[0055] The motor angle set description file contains four main elements: angle set name - motor angle set key-value pairs, an ordered list of angle set names, an ordered list of Boolean values ​​indicating whether it is a dynamic angle set, and delay time key-value pairs. This file is read and loaded into the angle control module during system initialization.

[0056] When the user layer calls the static motor array angle set, that is, calls the corresponding motor arrangement dimension array that stores the motor angles when the "whether it is a dynamic pattern" boolean attribute is "no", the user can query by name and call the corresponding motor array angle set through mapping.

[0057] Dynamic angle control establishes a series of coherent transformations of motor array angle sets, defined by functions, establishing a mapping between function names and implementation methods. Calling the dynamic motor array angle set involves calling the corresponding motor arrangement dimension array storing the motor angles, where the boolean attribute "Is it a dynamic pattern?" is set to "Yes". The arrays are called sequentially based on their names. Users can query by function name to access the motor array angle set and achieve dynamic transformation.

[0058] The motion detection module is responsible for driving and loading the Kinect motion detection sensor to detect the human body. A right-handed coordinate system is established with the human face facing forward as the Y-axis and vertically upward as the Z-axis, in meters. The module tracks the detected human body and acquires the joint spatial coordinates, comparing this data with prior knowledge to ultimately obtain a specific posture determination result. The human joint and skeleton data determination modes include the following six types:

[0059] 1) "Heart shape": The difference between the Y-component of the spatial coordinates of the left and right palms and the Y-component of the spatial coordinates of the left and right shoulders of the human body is greater than 0 and less than 0.4; and the Y-component of the spatial coordinates of the left and right elbow joints is greater than the Y-component of the spatial coordinates of the left and right shoulders of the human body.

[0060] 2) "Waving": The absolute value of the difference between the Y component of the left palm spatial coordinate and the Y component of the left shoulder joint spatial coordinate is less than 0.25, and the absolute value of the difference between the Y component of the left elbow joint spatial coordinate and the Y component of the left shoulder joint spatial coordinate is less than 0.25, and the X component of the left palm joint spatial coordinate is less than the X component of the left elbow joint spatial coordinate; or the absolute value of the difference between the Y component of the right palm spatial coordinate and the Y component of the right shoulder joint spatial coordinate is less than 0.25, and the absolute value of the difference between the Y component of the right elbow joint spatial coordinate and the Y component of the right shoulder joint spatial coordinate is less than 0.25, and the X component of the right palm joint spatial coordinate is less than the X component of the left elbow joint spatial coordinate.

[0061] 3) "Stop": The difference between the Y-components of the left palm joint and the left hip joint is less than 0.15 and the difference between the Y-components of the left palm joint and the left hip joint is less than 0.15.

[0062] 4) "Left swipe": The X-coordinate component of the right palm changes from the right side of the body to the left side of the body within 1 second;

[0063] 5) "Right swipe": The X-coordinate component of the left palm changes from the left side of the body to the right side within 1 second;

[0064] 6) "Raise right hand": The difference between the Y-coordinate component of the right palm and the right shoulder joint is greater than 0.1 and the difference between the Y-coordinate component of the left palm and the left shoulder joint is less than 0;

[0065] The Bluetooth communication module is responsible for establishing a Bluetooth connection with the mobile phone control terminal and receiving control words sent by Bluetooth to form a control message queue. The Bluetooth control word, as shown in Table 1, is 8 bits long (0#0#0#00), using ASCII encoding. Each bit starts from the most significant bit and has the following meanings: control status, separator, row number of the motor to be calibrated, separator, column number of the motor to be calibrated, separator, and single motor calibration angle.

[0066] Table 1 Bluetooth Control Words

[0067] function number # Single motor row number # Single motor column number # Correction angle

[0068] The Bluetooth control word value set has the following four types:

[0069] 1) 1#0#0#0: Normal operation;

[0070] 2) 2#0#0#0: Enter single motor angle correction mode;

[0071] 3) 3#motor row number#motor column number#motor correction angle: Send specific single motor correction data, where the motor correction angle value ranges from -50 to +50;

[0072] 4)4#0#0#0: Enter power off lock mode.

[0073] The Bluetooth message receiving strategy is as follows: A message queue is established, and multiple threads asynchronously detect whether Bluetooth control commands are being received. When a command is received, if the command matches the message at the end of the queue, the command is discarded; otherwise, a message is added from the end of the message queue. The logic control module retrieves commands from the head of the queue. If the command queue length is not 1, the command already retrieved from the head of the queue is popped.

[0074] The logic control module is responsible for updating the system status control word by combining data from the motion detection module and the Bluetooth communication module. Different system status control words call different functions. This module is the core scheduling module of the system, using the motion detection module and the Bluetooth communication module as inputs to directly control the angle control module. The system status control word consists of 10 bits, "0000000000". Each bit, starting from the most significant bit, has the following meanings: the first two bits are the Bluetooth control protocol control number; the third bit is the human body tracking cancellation flag; the fourth bit is the power-off confirmation flag; the fifth bit is the manual selection mode flag for the motor angle set; the sixth bit is the flag for the motor following the relative position of the human body; the seventh bit is the flag for the motor following the angle of the human arm; the eighth bit is the flag for the specific motor angle set triggered by a specific human body movement; the ninth bit is the flag for whether a human body has acquired control; and the last bit is the system normal operation flag.

[0075] The logic control module has 8 control modes. The system status control word defaults to 0000000001 after the power-on self-test passes. Before entering the following modes, the corresponding flag bits are set to 1 by the logic judgment module. When exiting the mode, the system status control word returns to its default value:

[0076] 1) Manual selection mode for motor angle set: The system status control word is 0000100011. This mode is entered when the motion detection module detects a "swipe right" action from the user. In this mode, the motor angle set is fixed. When the user is detected to continue swiping right, the mode switches to the next motor angle set. If the motion detection module does not detect the user's next "swipe right" action for 10 consecutive seconds, the mode exits.

[0077] 2) Motor Follow-up Tracking Human Arm Angle Mode: The system status control word is 0000001011. This mode is entered when the motion detection module detects a "waving" action from the user. This mode reads human joint data from the motion detection module, using a right-handed coordinate system with the user facing forward as the Y-axis and vertically upward as the Z-axis (unit: meters). Two spatial vectors are constructed for the left and right palms, elbows, and shoulders. The angles between each vector and the negative Z-axis unit vector are calculated, and the larger angle is taken as the angle between the arm and the vertical plane, called the arm angle. This arm angle is used as the motor input for tracking the human arm angle. The calculation is as follows: Assuming N columns of motors, for the left N / 2 column motors: the motor angle is 180 + arm angle; for the right N / 2 column motors: the motor angle is 180 - arm angle. If the arm angle change is within 10 degrees and the count exceeds 2000 times, or if the motion detection system detects a "stop" action from the user, the mode exits.

[0078] 3) Motor Follow-up Tracking Human Relative Position Mode: The system status control word is 0000010011. The motion detection module automatically enters this mode after the user exits the motor follow-up tracking human arm angle mode. A right-handed coordinate system is established with the depth direction of the motion sensor as the Z-axis and the vertical direction upwards as the Y-axis, in meters. This mode reads the three-dimensional coordinates of the human body center from the motion detection module and takes the x-component as the offset direction. When the absolute difference between the x-components of the current frame and the next frame is less than 0.2, it is assumed that the human body has not moved significantly, and the motor does not respond. When the absolute difference is greater than 0.2, it is assumed that the human body is moving, and the offset data is calculated and used as the motor input for tracking the relative position of the human body. The calculation is as follows: Let the x-component of the most recent frame without significant movement be prevMag, and the x-component of the current frame be nowMag. Then, the relative position of the human body is Mag = abs(nowMag - prevMag), and the motor angle is uniformly input as 180 + Mag * 100. When the user who currently holds control leaves the control area, the system automatically exits this mode, and the system status control word automatically reverts to its default value.

[0079] 4) Specific human body movement triggers specific motor angle set mode: The system status control word is 00000001X1. This mode is entered when the motion detection module detects a specific user movement. After entering this mode, the system automatically switches to the specific motor angle set and automatically exits after execution.

[0080] 5) Single Motor Angle Correction Mode: The system status control word is 1100000000. This mode is entered when the Bluetooth receiver module receives a message frame from the client containing a single motor angle correction command. Upon entering this mode, the motor returns to its encoder relative to the origin. The Bluetooth receiver module returns an instruction containing the row number, column number, and relative correction value to the logic instruction module, which then sends the correction data to the lower-level machine for single-motor angle fine-tuning.

[0081] 6) Cancel Control Mode: This mode is triggered when the motion detection module detects a "swipe left" action from the user. This mode unconditionally resets the system status control word to its default value. The user will lose control of the motion sensors.

[0082] 7) Automatic Angle Set Rotation Mode: This is the system default mode, and the system status word is the inverse set of the intersection of the above system status words. In this mode, the logic control module automatically rotates the angle set according to the order of the motor array angle sets loaded by the angle control module.

[0083] The serial communication module is responsible for establishing a connection with the lower-level machine via serial communication, receiving angle data input from the angle control module, processing the data accordingly, packaging it into frames according to a pre-defined protocol, and then transmitting the data frames to the lower-level machine for further control. Assuming the motor array has M rows and N columns, the definition of the data frame is as follows: Figure 3 As shown, there is an M x (N+1) two-dimensional array, where each row of data is defined as: motor speed code + N bits of motor angle. The logic process of packaging data frames is as follows: the serial port sending module receives data from the angle control module, puts (motor angle / 2) into the two-dimensional array according to the motor row number and column number, and then fills the speed code into the frame header. The speed code takes values ​​from 0-70, 101, 102, 103. The speed gear algorithm is: if the motor has N rows, then the motor row numbers are 1, 2, 3, ..., N. Then the speed code for each row is (row number + (speed gear selection - 1) * 7). If the speed gear selection is 101, 102, 103, it is directly filled in. The serial port sending module will send data frames line by line.

[0084] like Figure 2 As shown, the control protocol structure consisting of the above five modules is as follows:

[0085] User layer: Logic control module; Middle layer: Angle control module and motion detection module; Machine layer: Serial communication module, Bluetooth communication module, and motion detection module. The user layer is directly object-oriented; users modify the logic control of the entire system at this layer. The middle layer faces the user layer, processing angle and human skeletal joint data, presenting the processed data to the user layer for operation, and receiving user commands and query requests. It then transparently transmits the data to the machine layer after querying. The machine layer faces the machine, receiving data from the middle layer, packaging it into frames according to a predetermined protocol, and transmitting it to the lower-level machine via serial port. Simultaneously, it receives data from sensors, unpacks it, and transmits it to the middle layer for parsing and processing.

[0086] This invention integrates motion sensing detection, Bluetooth control, and logic control, proposing a three-layer architecture and five modules for a multi-motor array control system. It decouples and standardizes the functional modules of multi-motor control, and uses the logic control module as a hub to achieve efficient, real-time, and multifunctional control of the multi-motor array system. Testing has shown that this invention is adaptable to the control and scheduling of large-scale motor arrays, achieving a scientific trade-off between algorithm complexity and scheduling efficiency.

[0087] The present invention discloses a process for establishing a motion-sensing-based multi-motor array host computer control system, comprising the following steps:

[0088] 1) Start the program, load the multi-motor angle set in the configuration description file and establish the mapping, initialize the angle control module, and establish the intermediate layer in the control architecture;

[0089] 2) Detect the USB to TTL serial port chip, detect the motion detection sensor and other hardware, initialize the serial communication module, Bluetooth communication module, motion detection module, establish a communication link with the lower-level machine, establish a data link with the motion detection sensor, broadcast the upper-level machine's Bluetooth information to the public space, verify that messages can be sent and received normally, establish the machine layer in the control architecture, and establish a connection with the middle layer.

[0090] 3) Initialize the logic control module and the logic control word in the module, establish the user layer in the control architecture, establish a connection with the middle layer, run the power-on self-test program, and make the motor array angle increase from 0 degrees to 180 degrees in 45-degree increments and then decrease to 0 degrees. Each time the angle changes, the motor array performs an angle transformation.

[0091] 4) Switch to normal operation cycle. The motion detection module and Bluetooth module return data to the logic control module. The logic control module updates the logic state control word according to the data returned by the module, and redirects to different function entry points according to the updated logic state control word.

[0092] 5) In different functional implementations, after the system controls and selects the angle set, the intermediate layer constructs the specific angle according to the mapping relationship, and then passes the angle set to the machine layer. The machine layer packages the angle set into a frame according to the protocol, transmits the data frame to the lower-level machine, and performs a delay according to the settings. After the delay, it loops back to 4) to detect the new frame.

[0093] The simulation effect of the control angle of the present invention is as follows: Figure 4 As shown. The motion-sensing-based multi-motor array host computer control system provided by this invention can be adapted to large-scale motor array control and has been successfully applied in physical devices.

[0094] Example 2

[0095] Regarding the motion detection module, due to space and cost limitations, installing the Kinect motion sensor is not suitable for all situations. To address this, the method used in Example 1 to acquire human joint data via the Kinect sensor in the motion detection module is modified to use a monocular camera to acquire human data, followed by inference using the lightweight pose estimation network MoveNet to obtain the human joint data. Figure 5 As shown, the monocular camera module connects to the host computer via a USB 3.0 cable to acquire images for the motion detection module in the control system. After acquiring the image, the motion detection module feeds the frame into the MoveNet neural network for inference. The input to the MoveNet neural network is an image of a specified size, and the output is a tensor of [1, 1, 17, 3], representing the two-dimensional spatial positions of the 17 key joints of the person closest to the camera. The motion detection module rearranges the output of the neural network module according to the joint order and then inputs it into the posture determination stage for judgment. This improvement enhances the robustness of motion posture detection, enabling the control system to successfully implement motion detection functionality in a smaller space.

[0096] Example 3

[0097] In motion detection, due to the complexity and high difficulty of determining certain user-expected movements, traditional pose determination methods using prior knowledge of joint coordinates are insufficient for describing relative positions, resulting in a high false positive rate. To address this, the pose determination step using prior knowledge in Example 1 is replaced with a pose determination step using a multilayer perceptron. This step takes 17 joint keypoints from the pose estimation step as input and outputs the probability of each pre-trained pose. It employs a three-layer perceptron structure: input layer – three fully connected layers – output layer, as shown in the diagram. Figure 6As shown, the user records the different pose types to be detected, creates a dataset, and uses this dataset to train the three-layer perceptron structure. After verification, the weights are saved and loaded into the control system for inference. This improvement enhances the accuracy of pose determination under complex poses, enabling users to perform higher-level semantic control using more complex haptic tasks.

[0098] Example 4

[0099] Due to varying installation environments, Bluetooth communication modules are unsuitable for remote communication in some situations. To address this, the Bluetooth communication module in Example 1 is replaced with a network remote control module. This module connects the host computer system to the Internet of Things (IoT) gateway platform, acting as a client to listen for and receive messages. Mobile clients connect to the network, connect to the corresponding port, and send data packets containing control words to the IoT gateway platform. Upon receiving the data packets, the IoT gateway platform forwards them to the network remote control module of the multi-motor array host computer control system according to its rules. The network remote control module unpacks the data packets according to the same data frame semantics and sends the corresponding control words to the logic control module for further processing. This improvement enhances the breadth of control signal transmission, enabling the control system to connect to the IoT and supporting more advanced remote control capabilities.

[0100] The above embodiments are only used to illustrate the technical solutions of the present invention in detail to help understand the present invention. Any improvements and substitutions made by those skilled in the art without departing from the principles of the present invention shall fall within the protection scope of the present invention.

Claims

1. A motion-sensing-based multi-motor array host computer control system, characterized in that, include: The control architecture consists of three layers: user layer, middle layer, and machine layer. The user layer includes a logic control module; the middle layer includes an angle control module and a motion detection module; and the machine layer includes a serial communication module, a Bluetooth communication module, and the motion detection module. Users modify and control the system logic in the logic control module at the user level; The middle layer is used to process angle data and human bone and joint data, and transmits the processed data to the user layer for user operation. It also receives user layer instructions and query requests, and transparently transmits the data sent down to the machine layer after querying. The machine layer is used to receive data from the middle layer and package the received data into frames according to the protocol, and transmit them to the lower-level machine through the serial communication module. At the same time, it receives data transmitted by sensors or camera frames, unpacks the sensor data or performs morphological processing on the frame data, and transmits it to the middle layer for parsing and further processing. Specifically, the logic control module combines data from the motion detection module and the Bluetooth communication module to update the system status control word, and different system status control words call different functions to achieve the desired results. The logic control module is the core scheduling module, using the motion detection module and Bluetooth communication module as inputs to directly control the angle control module. The system status control word consists of 10 bits (0000000000), with each bit starting from the most significant bit and having the following meanings: the first two bits are the Bluetooth control protocol control number; the third bit is the human body tracking cancellation flag; the fourth bit is the power-off confirmation flag; the fifth bit is the manual selection mode flag for the motor angle set; the sixth bit is the flag for the motor following the relative position of the human body; the seventh bit is the flag for the motor following the angle of the human arm; the eighth bit is the flag for a specific human body action triggering a specific motor angle set; the ninth bit is the flag for whether a human body has acquired control; and the last bit is the system normal operation flag.

2. The logic control module in a motion-sensing-based multi-motor array host computer control system according to claim 1, characterized in that, The logic control module has 8 modes. The system status control word defaults to 0000000001 after the power-on self-test passes. Before entering the following modes, the corresponding flag bits are set to 1 by the logic judgment module. When exiting a mode, the system status control word returns to its default value: 1) Manual selection mode for motor angle sets: The system status control word is 0000100011; When the motion detection module detects that the user's action is a right swipe, it enters the manual selection mode; In the manual selection mode, the automatic switching of motor angle sets is canceled. When the user is detected to continue to swipe right, it switches to the next motor angle set. When the motion detection module does not detect the user's next right swipe action for n consecutive seconds, it exits the manual selection mode. 2) Motor-guided tracking of human arm angle mode: System status control word is 0000001011; When the motion detection module detects a user waving hand, it enters the human arm angle mode. This mode reads human joint data from the motion detection module, using a right-handed coordinate system with the user facing forward as the Y-axis and vertically upward as the Z-axis. Two spatial vectors are constructed for the left and right palms, elbows, and shoulders, respectively. The angles between each vector and the negative Z-axis unit vector are calculated, and the larger of the two is taken as the angle between the arm and the vertical plane, called the arm angle. This arm angle is then used as the motor input for tracking the human arm angle. The calculation is as follows: With N columns of motors, for the left N / 2 columns: the motor angle is 180 + arm angle; for the right N / 2 columns: the motor angle is 180 - arm angle. If the arm angle change is within 10 degrees and the count exceeds 2000 times, or if the motion detection system detects that the user's movement has stopped, the mode exits. 3) Motor-guided human body relative position mode: The system status control word is 0000010011; the motion detection module detects that the user has exited the motor-guided human arm angle mode and automatically enters the human body relative position mode. Using the depth direction of the motion sensor as the Z-axis and the vertical direction upwards as the Y-axis, a right-handed coordinate system is used. The human body relative position mode reads the three-dimensional coordinates of the human body center from the motion detection module and takes the x-component as the offset direction. When the absolute difference between the x-components of the previous frame and the next frame is less than 0.2, it is considered that the human body has not moved significantly, and the motor does not output anything; when the absolute difference is greater than 0.2, it is considered that the human body is moving. At this time, the offset data is calculated and used as the motor input for the motor-guided human body relative position mode. The calculation is as follows: the x-component of the most recent frame without significant movement is prevMag, and the x-component of the current frame is nowMag. Then the relative position of the human body is Mag = abs(nowMag - prevMag), and the motor angle is uniformly input as 180 + Mag * 100; When the user who currently holds control leaves the control area, this mode will automatically exit, and the system status control word will automatically revert to its default value. 4) Specific human body movements trigger specific motor angle set mode: The system status control word is 00000001X1; when the motion detection module detects a user movement, it enters this mode. After entering this mode, the system automatically switches to the corresponding motor angle set and automatically exits after execution. 5) Single motor angle correction mode: The system status control word is 1100000000; when the Bluetooth receiving module receives a message frame sent by the client with a single motor angle correction instruction, it enters this mode. After entering this mode, the motor returns to its encoder relative to the origin; the Bluetooth receiving module returns the instruction including the row number, column number, and correction relative value to the logic instruction module, and the logic instruction module sends the correction data to the lower-level machine for single motor angle fine-tuning correction. 6) Cancel Control Mode: When the motion detection module detects that the user has swiped left, it triggers the cancel control mode; this cancel control mode unconditionally sets the system status control word to the default value, and the human body will lose the right to control the motion. 7) Automatic Angle Set Rotation Mode: Automatic angle set rotation mode is the system default mode. The system status word is the inverse set of the intersection of the above system status words. In automatic angle set rotation mode, the logic control module automatically rotates the angle set according to the order of the motor array angle set loaded by the angle control module.

3. The multi-motor array host computer control system based on motion sensing according to claim 1, characterized in that, The angle control module is responsible for loading, storing, and querying the preset static multi-motor angle set, and is also responsible for real-time calculation of the dynamic multi-motor angle set sequence; The angle control module is divided into two types: static angle control and dynamic angle control. Static angle control establishes a key-value mapping between the motor array angle set and its name. This mapping is defined by the motor angle set description file, loaded during system initialization, and stored in system memory. When the user layer calls the static motor array angle set, the user layer function uses the name to query and calls the corresponding motor array angle set through mapping. Dynamic angle control establishes a series of coherent motor array angle set transformations, which are defined by function methods, establishing a mapping between function names and implementation methods. When the user layer function calls the dynamic motor array angle set, it uses the function name to query and calls the motor array angle set through the function to achieve dynamic transformation.

4. The multi-motor array host computer control system based on motion sensing according to claim 3, characterized in that, The motor angle set description file in the angle control module has four main elements: angle set name - motor angle set key-value pairs, ordered list of angle set names, ordered list of Boolean values ​​indicating whether it is a dynamic angle set, and delay time key-value pairs. The motor angle set description file is read and loaded into the angle control module during system initialization.

5. The multi-motor array host computer control system based on motion sensing according to claim 1, characterized in that, The motion detection module is responsible for driving and loading the Kinect motion detection sensor to detect the human body. Using a right-handed coordinate system with the human face facing forward as the Y-axis and vertical upward as the Z-axis, it tracks the detected human body and acquires the joint space coordinates. The acquired data is then compared with prior knowledge or a pose classification network using a multilayer perceptron to determine the final pose. The human joint and skeletal data determination modes include the following six: 1) Heart shape: The difference between the Y-components of the spatial coordinates of the left and right palms and the Y-components of the spatial coordinates of the left and right shoulders of the human body is greater than 0 and less than 0.4; and the Y-components of the spatial coordinates of the left and right elbow joints are greater than the Y-components of the spatial coordinates of the left and right shoulders of the human body. 2) Waving: The absolute value of the difference between the Y component of the left palm spatial coordinate and the Y component of the left shoulder joint spatial coordinate is less than 0.25, and the absolute value of the difference between the Y component of the left elbow joint spatial coordinate and the Y component of the left shoulder joint spatial coordinate is less than 0.25, and the X component of the left palm joint spatial coordinate is less than the X component of the left elbow joint spatial coordinate; or the absolute value of the difference between the Y component of the right palm spatial coordinate and the Y component of the right shoulder joint spatial coordinate is less than 0.25, and the absolute value of the difference between the Y component of the right elbow joint spatial coordinate and the Y component of the right shoulder joint spatial coordinate is less than 0.25, and the X component of the right palm joint spatial coordinate is less than the X component of the left elbow joint spatial coordinate. 3) Stop: The difference between the Y-components of the left palm joint and the left hip joint is less than 0.15; 4) Left swipe: The X-coordinate component of the right palm changes from the right side of the body to the left side within 1 second; 5) Right swipe: The X-coordinate component of the left palm changes from the left side of the body to the right side within 1 second; 6) Raise your right hand: The difference between the Y-coordinate component of the right palm and the right shoulder joint is greater than 0.1 and the difference between the Y-coordinate component of the left palm and the left shoulder joint is less than 0.

6. The multi-motor array host computer control system based on motion sensing according to claim 5, characterized in that, The pose determination using a multilayer perceptron in the motion detection module is as follows: the input of the multilayer perceptron is the 17 joint key points output by the pose estimation stage, and the output is the probability of each pre-trained pose. It uses a three-layer perceptron structure with an input layer, three fully connected layers, and an output layer. The user records the pose types to be detected separately, creates a dataset, and uses this dataset to train the three-layer perceptron structure. After verification, the weights are saved and loaded into the system for inference.

7. The multi-motor array host computer control system based on motion sensing according to claim 1, characterized in that, The Bluetooth communication module is used to establish a Bluetooth connection with the mobile control terminal and receive control words sent by Bluetooth, forming a control message queue. The Bluetooth control word is 8 bits long (0#0#0#00), using ASCII encoding. Each bit starts from the most significant bit and represents the following: control status, separator, row number of the motor to be calibrated, separator, column number of the motor to be calibrated, separator, and single motor calibration angle. The Bluetooth control word can take the following four values: 1) 1#0#0#0: Normal operation; 2) 2#0#0#0: Enter single motor angle correction mode; 3) 3#motor row number#motor column number#motor correction angle: Send specific single motor correction data, where the motor correction angle value ranges from -50 to +50; 4) 4#0#0#0: Enter power-off lock mode; The Bluetooth message receiving strategy is as follows: a message queue is established, and multiple threads asynchronously detect whether a Bluetooth control command is being received; when a command is received, if the command is consistent with the message at the end of the queue, the command is discarded; otherwise, a message is filled in from the end of the message queue; the logic control module retrieves the command from the head of the queue, and if the length of the command queue is not 1, the command that has been retrieved from the head of the queue is popped.

8. A motion-sensing-based multi-motor array host computer control system according to any one of claims 1 to 7, characterized in that, The serial communication module is used to establish a connection with the lower-level machine via serial communication, receive angle data input from the angle control module, perform corresponding data processing, package the data into a data frame according to the agreed protocol, and then transmit the data frame to the lower-level machine for further control. If the motor array has M rows and N columns, then the data frame is defined as an M x (N+1) two-dimensional array, where the data frame of each row is defined as: motor speed level + N motor angles.

9. A motion-sensing-based multi-motor array host computer control system according to claim 8, characterized in that, The logical process of packaging data frames is as follows: The serial port sending module receives data from the angle control module, puts (motor angle / 2) into a two-dimensional array according to the motor row number and column number, and then fills the speed code into the frame header; where the speed code takes values ​​of 0-70, 101, 102, 103; the speed gear algorithm is as follows: if the motor has M rows, then the motor row numbers are 1, 2, 3, ..., M, then the speed code for each row is (row number + (speed gear selection - 1) * 7); if the speed gear selection is 101, 102, 103, then it is filled in directly, and the serial port sending module will send data frames line by line.

Images