Industrial AGV and method based on multi-mode gesture action recognition signal control

Abstract

The invention discloses an industrial AGV and method based on multi-mode gesture action recognition signal control, and relates to the technical field of gesture recognition, and the industrial AGV comprises a myoelectric arm ring, a control module, a Bluetooth remote control assembly and an action module. The control module comprises a training program and a control program; the training program comprises a myoelectric armlet signal acquisition program and a model training program; the control program comprises a myoelectric armlet signal receiving program, a gesture recognition program and a motor control program; the myoelectric armlet collects a multi-mode gesture signal and sends the multi-mode gesture signal to the Bluetooth remote control assembly, and a control program processes the signal and recognizes a gesture action; and the action module completes corresponding actions according to the recognized different gestures. According to the invention, the industrial AGV equipment is controlled by using the multi-mode gesture signal, and the method has the advantages of simple operation mode, low training cost, light control equipment, single-hand operation and operability for hand disability.

Description

Technical Field

[0001] The present invention relates to the field of gesture recognition technology, and in particular to an industrial AGV and method based on multi-modal gesture action recognition signal control. Background Art

[0002] Surface electromyography (sEMG) has been widely used in the medical field and research in the field of rehabilitation for the disabled. There are myoelectric armband devices on the market that can be worn directly on the forearm and can accurately identify multi-channel surface electromyography signals and original inertial measurement unit (IMU) signals. However, the research and application of such devices are mostly limited to gesture recognition, signal acquisition, screen control and other fields for the disabled, while ignoring the advantages of myoelectric armbands in wearability, learnability, simplicity and one-handed operation compared to traditional control methods. The research on control systems for multimodal signal-controlled movable devices based on surface electromyography signals has not yet been perfected. Summary of the invention

[0003] The purpose of the present invention is to provide an industrial AGV and method based on multi-modal gesture action recognition signal control, which can realize gesture recognition control and one-handed simple control.

[0004] To achieve the above object, the present invention provides the following solutions:

[0005] In a first aspect, the present invention provides an industrial AGV based on multimodal gesture motion recognition signal control, wherein the industrial AGV based on multimodal gesture motion recognition signal control comprises: an electromyographic armband, a control module, a Bluetooth remote control component and an action module.

[0006] The myoelectric armband is used to collect multimodal gesture signals and send the multimodal gesture signals to the Bluetooth remote control component; the multimodal gesture signals include: surface electromyography signals and IMU signals.

[0007] The control module includes a training program and a control program; the training program includes: an electromyogram armband signal acquisition program and a model training program; the electromyogram armband signal acquisition program is used to control the electromyogram armband to acquire multimodal gesture signals; the model training program is used to train the machine learning model to obtain a gesture recognition model; the control program includes: an electromyogram armband signal receiving program, a gesture recognition program and a motor control program; the electromyogram armband signal receiving program is used to receive multimodal gesture signals sent by the Bluetooth remote control component; the gesture recognition program is used to recognize gesture actions according to the multimodal gesture signals; the motor control program is used to control the action module to complete the action according to the gesture action.

[0008] Optionally, the control module includes: a control single chip microcomputer and a host computer; the control single chip microcomputer includes: a host computer control program and an independent control program.

[0009] The host computer control program is used to passively receive the control signal transmitted by the host computer through the Bluetooth remote control component.

[0010] The independent control program is used to identify gesture actions according to the multimodal gesture signal, and control the action module to complete the action according to the gesture action.

[0011] Optionally, the model training program is used to train the machine learning model to obtain a gesture recognition model, specifically including:

[0012] Acquire a training set; the training set includes: different preset actions and preset multimodal gesture signals corresponding to the different preset actions.

[0013] The machine learning model is trained with the preset multimodal gesture signal as input and the preset action corresponding to the preset multimodal gesture signal as output to obtain a gesture recognition model.

[0014] Optionally, obtain a training set, including:

[0015] Data is collected according to different preset actions to obtain a number of preset multimodal gesture signals.

[0016] Each of the preset multimodal gesture signals is normalized and formatted to obtain a plurality of processed preset multimodal gesture signals.

[0017] Different preset actions and processed preset multimodal gesture signals corresponding to the different preset actions are used as training sets.

[0018] Optionally, the gesture recognition program is used to recognize gesture actions according to the myoelectric armband signal, specifically including:

[0019] Output several groups of gesture recognition tags and cache them;

[0020] The tags in the cache that are greater than or equal to the preset threshold are taken as the final recognized gesture actions.

[0021] In a second aspect, the present invention provides a method for controlling a multimodal gesture recognition signal, the method comprising:

[0022] Acquire a multimodal gesture signal to be tested; the multimodal gesture signal to be tested includes: a surface electromyography signal and an IMU signal.

[0023] The multimodal gesture signal to be tested is input into the gesture recognition model to obtain a gesture action; the gesture recognition model is a machine learning model.

[0024] Based on the gesture action, the action module is controlled to complete the action.

[0025] Optionally, the method for determining the gesture recognition model specifically includes:

[0026] Acquire a training set; the training set includes: different preset actions and preset multimodal gesture signals corresponding to the different preset actions.

[0027] The machine learning model is trained with the preset multimodal gesture signal as input and the preset action corresponding to the preset multimodal gesture signal as output to obtain a gesture recognition model.

[0028] Optionally, obtain a training set, including:

[0029] Data is collected according to different preset actions to obtain a number of preset multimodal gesture signals.

[0030] Each of the preset multimodal gesture signals is normalized and formatted to obtain a plurality of processed preset multimodal gesture signals.

[0031] Different preset actions and processed preset multimodal gesture signals corresponding to the different preset actions are used as training sets.

[0032] Optionally, inputting the multimodal gesture signal to be tested into the gesture recognition model to obtain a gesture action specifically includes:

[0033] The multimodal gesture signal to be tested is input into the gesture recognition model, and a plurality of groups of gesture recognition labels are output and cached.

[0034] The tags in the cache that are greater than or equal to the preset threshold are taken as the final recognized gesture actions.

[0035] Optionally, different gesture actions have a one-to-one correspondence with different actions of the action module.

[0036] According to the specific embodiments provided by the present invention, the present invention discloses the following technical effects:

[0037] The present invention provides an industrial AGV and method based on multimodal gesture recognition signal control, the industrial AGV based on multimodal gesture recognition signal control comprises: an electromyographic armband, a control module, a Bluetooth remote control component and an action module; the electromyographic armband is used to collect multimodal gesture signals and send the multimodal gesture signals to the Bluetooth remote control component; the control module comprises a training program and a control program; the training program comprises: an electromyographic armband signal collection program and a model training program; the control program comprises: an electromyographic armband signal receiving program, a gesture recognition program and a motor control program; the electromyographic armband collects multimodal gesture signals and sends them to the Bluetooth remote control component, and the control module processes the signals and recognizes gestures; the action module performs actions according to different gestures recognized by the gesture recognition program. The present invention uses multimodal gesture signals to control industrial AGV equipment, which has the advantages of simple operation, low training cost, light control equipment, and single-handed operation and operability for hand disabilities. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0039] Figure 1 A schematic diagram of a process framework for using a multimodal electromyographic gesture signal acquisition, training and recognition program provided in one embodiment of the present invention.

[0040] Figure 2 A schematic diagram of a control system and device structure provided in one embodiment of the present invention.

[0041] Figure 3 A conceptual diagram of gesture control provided by an embodiment of the present invention.

[0042] Figure 4 A flowchart of a multi-modal gesture recognition signal control method provided by an embodiment of the present invention.

[0043] Figure 5 A schematic diagram of the structure of a computer device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0044] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0045] The purpose of the present invention is to provide a control system (including control program and circuit) based on surface electromyography signals and IMU signals collected by an electromyography armband, and to apply this control system to gesture control of industrial AGVs. The electromyography armband device is used to identify multimodal gesture signals composed of surface electromyography signals and IMU signals of the operator's forearm, and the industrial AGV is manipulated to move, turn and perform other actions through preset different gestures. Using multimodal gesture signals to control industrial AGV equipment has the advantages of simple operation, low training cost, light control equipment, and the ability to be operated with one hand or even with hand disabilities.

[0046] In order to make the above-mentioned objects, features and advantages of the present invention more obvious and easy to understand, the present invention is further described in detail below with reference to the accompanying drawings and specific embodiments.

[0047] In an exemplary embodiment, an industrial AGV based on multimodal gesture action recognition signal control is provided, and the industrial AGV based on multimodal gesture action recognition signal control includes: an electromyographic armband, a control module, a Bluetooth remote control component and an action module.

[0048] The myoelectric armband is used to collect multimodal gesture signals and send the multimodal gesture signals to the Bluetooth remote control component; the multimodal gesture signals include: surface electromyography signals and IMU signals.

[0049] The control module includes a training program and a control program; the training program includes: an electromyographic armband signal acquisition program and a model training program; the electromyographic armband signal acquisition program is used to control the myoelectric armband to acquire multimodal gesture signals; the model training program is used to train the machine learning model to obtain a gesture recognition model.

[0050] The EMG armband signal acquisition program is based on the EMG armband open source library, which can control the Bluetooth module adapted by the EMG armband to receive surface EMG signals and IMU signals, and collect and store the raw data of each channel in real time, and finally store it as a csv file. During the acquisition process, the user maintains several preset actions and collects them for a period of time. Finally, the program labels the multimodal signal data files of different actions and merges them into the file format required for model training.

[0051] The model training program reads the saved csv raw data and performs preprocessing and segmentation, then trains the multimodal data based on machine learning models such as random forests, and automatically saves the optimal weights. After the training, the optimal weights are saved as weights.hdf5 files, and the accuracy is tested and displayed using the test set.

[0052] After receiving the multimodal gesture signal, the Bluetooth remote control component sends the multimodal gesture signal to the control program, and the control program processes the signal and recognizes the gesture action.

[0053] The control program includes: an electromyographic armband signal receiving program, a gesture recognition program and a motor control program; the electromyographic armband signal receiving program is used to receive the multimodal gesture signal sent by the Bluetooth remote control component; the gesture recognition program is used to recognize gesture actions according to the multimodal gesture signal; the motor control program is used to control the action module to complete the action according to the gesture action; the action module includes: a wheel group and other power components (such as a lifting component, etc.).

[0054] Among them, the myoelectric armband signal receiving program runs based on the myoelectric armband open source library, and can receive a combined signal consisting of a group of surface electromyography signals and IMU signals at a fixed frequency and continuously refresh them. Each group of signals will be input into the gesture recognition program as an input feature.

[0055] The gesture recognition program reads the weight file saved by the training program to obtain the model recognition weight, receives the combined signal input by the myoelectric armband signal receiving program, performs real-time training through a machine learning model such as a random forest, and outputs a gesture recognition label corresponding to each group of the combined signals.

[0056] The present invention utilizes multimodal gesture signals to control industrial AGV equipment, which has the advantages of simple operation, low training cost, light control equipment, and single-handed operation and operability for people with hand disabilities.

[0057] In another exemplary embodiment of the present invention, the control module includes: a control single chip microcomputer and a host computer (PC); the industrial AGV receives and processes multimodal signals based on the control single chip microcomputer or the host computer.

[0058] The control single chip microcomputer comprises: a host computer control program and an independent control program, and the control single chip microcomputer can switch between two control modes.

[0059] The host computer control program does not perform gesture recognition, but passively receives the control signal transmitted by the host computer (PC) through the Bluetooth remote control component.

[0060] The independent control program runs directly on the control single chip microcomputer without the intervention of a fixed position host computer (PC), and the library files used are all adapted to the Linux system.

[0061] The independent control program is used to identify gesture actions according to the multimodal gesture signal, and control the action module to complete the action according to the gesture action. Specifically, the wheel set and the other power components change the moving direction and perform different actions according to the different gesture actions identified by the control microcontroller and compare them with the preset gesture actions.

[0062] In addition, the motor control program receives the gesture signal tag output by the host computer control program or the host computer (PC) to control the operation, stop and direction of the wheel set and other power components. Different gesture actions have a one-to-one correspondence with different actions of the industrial AGV.

[0063] In another exemplary embodiment of the present invention, the model training program is used to train the machine learning model to obtain a gesture recognition model, specifically including:

[0064] A1: Obtain a training set; the training set includes: different preset actions and preset multimodal gesture signals corresponding to the different preset actions.

[0065] A2: Taking the preset multimodal gesture signal as input and the preset action corresponding to the preset multimodal gesture signal as output, the machine learning model is trained to obtain a gesture recognition model.

[0066] Wherein, in step A1, specifically including:

[0067] A11: Data is collected according to different preset actions to obtain a number of preset multimodal gesture signals.

[0068] A12: Perform normalization processing and format processing on each of the preset multimodal gesture signals to obtain a plurality of processed preset multimodal gesture signals.

[0069] A13: Different preset actions and processed preset multimodal gesture signals corresponding to the different preset actions are used as training sets.

[0070] In another exemplary embodiment of the present invention, the gesture recognition program is used to recognize gesture actions according to the myoelectric armband signal, specifically including:

[0071] B1: Output several groups of gesture recognition tags and cache them;

[0072] B2: The tags in the cache that are greater than or equal to the preset threshold are taken as the final recognized gesture actions.

[0073] Specifically, the gesture recognition program outputs the gesture recognition label to be cached, identifies ten sets of cached data, and outputs labels greater than or equal to a threshold (such as 7) among the ten sets of data as the final recognized gesture signal label.

[0074] It should be noted that: the prediction window length is 10, and the range of the threshold setting must be greater than 5; there is only one label for each prediction result. If one type of label is greater than the threshold, then the total number of labels in other categories will not exceed 4, and there will not be two labels that exceed the threshold; the size of the threshold is variable, and the sensitivity can be increased or decreased according to actual conditions.

[0075] The purpose of the present invention is to provide a control system (including control program and circuit) based on surface electromyography signals and IMU signals collected by an electromyography armband, and to apply this control system to industrial AGV gesture control. The present invention provides an industrial AGV based on multimodal gesture action signal control, which has an electromyography armband, a control single chip microcomputer, a host computer (PC), a Bluetooth remote control component, a training program, a control program, a wheel group and other power components (such as a lifting component, etc.). The industrial AGV receives and processes multimodal signals based on the control single chip microcomputer or the host computer (PC); the training program includes an electromyography armband signal collection program and a model training program; the control program includes an electromyography armband signal receiving program, a gesture recognition program and a motor control program; the electromyography armband collects surface electromyography signals and IMU signals and sends them to the Bluetooth remote control component, and the control program processes the signals and recognizes gesture actions; the wheel group and other power components change the moving direction and make different actions according to the different gesture actions recognized by the control single chip microcomputer and compare them with the preset gesture actions. Using multimodal gesture signals to control industrial AGV equipment has the advantages of simple operation, low training cost, lightweight control equipment, and one-handed operation and operability for people with hand disabilities.

[0076] In an exemplary embodiment, Figure 1 As shown, a use process of a multimodal gesture signal acquisition, training, and recognition program system is provided, and the program system includes a training program and a control program. The training program consists of an electromyographic armband signal acquisition program and a model training program; the control program consists of an electromyographic armband signal receiving program, a gesture recognition program, and a motor control program.

[0077] The training program and the control program correspond to the two stages of the control system usage process: the training stage and the control stage.

[0078] The first is the training stage: the operator first collects multimodal gesture signals, and then conducts training based on the multimodal gesture signals collected by the operator.

[0079] During the training phase, the multimodal gesture signal collection work is completed by the myoelectric armband signal collection program. The operator wears the myoelectric armband and collects data according to several different gestures preset in the data set. The program autonomously records, processes and saves the obtained data. The multimodal gesture signal includes 8-channel surface electromyography signals and 4-channel IMU signal data. Each channel data is saved in a csv file by column, and the corresponding label is obtained according to the selected gesture action.

[0080] Next, the program calls the weight training model to train the training data set collected and processed by the operator. The data processing process includes normalization and format processing, and finally obtains the training set data required by the machine learning model. The model training program uses machine learning models such as random forests, which contain functions that automatically adjust parameters, such as grid search or random search, and can automatically save the optimal weights (such as the adjustable parameters of the random forest model include n_estimators, max_depth and max_features). After the training is completed, the program saves the best weights of the entire process and uses it to identify the control program.

[0081] The second is the control stage: after the training stage is completed, the operator can use the collected different gestures to control the industrial AGV to complete the corresponding actions. During this process, the host computer (PC) or the control microcontroller receives a single set of multimodal gesture signal data at a specific frequency and outputs a gesture command label. Finally, the motor completes the corresponding action according to the gesture command label.

[0082] In the control stage, the signal receiving program runs based on the open source library of the myoelectric armband, and has the function of receiving the Bluetooth signal sent by the myoelectric armband and reading the data of each channel. After reading the original data, the signal receiving program distinguishes the channel type and number according to the key in the data dictionary, and compiles the value corresponding to the key into a temporary storage list, which is a multimodal signal composed of a group of 8-channel surface electromyography signals and 4-channel IMU signals.

[0083] After processing, the list is input into the gesture recognition program module, and the input data is predicted through the same model and weight as in the training phase, and the predicted gesture label is output, and the prediction result is temporarily stored in a list of length 10, and so on, and the 10-digit label numbers in the list are continuously refreshed. When the number of a specific label in the 10-digit list is greater than or equal to the set threshold (such as 7), the current gesture is predicted to be the gesture corresponding to the label, and the corresponding gesture and gesture label are printed.

[0084] Finally, the motor control program module receives the gesture or gesture label predicted and output by the gesture recognition program module, controls the corresponding motor switch, forward rotation or reverse rotation, thereby completing the control of the industrial AGV.

[0085] In another exemplary embodiment of the present invention, a practical use of the control system process in the above embodiment in an industrial AGV is provided, including the main components of the control system circuit, control module selection, circuit element composition, etc.

[0086] The circuit and module structure of the whole control system are shown in the figure Figure 2 As shown. The operator wears an electromyographic armband device with two sensors, a surface electromyographic signal sensor and an IMU sensor, such as the Myo armband launched by Canadian company Thalmic Labs, which has built-in 8-channel surface electromyographic signal sensors and 4-channel inertial measurement units, and transmits multimodal gesture signals via Bluetooth. When using the armband, a Bluetooth receiver needs to be inserted at the receiving end. The armband signal receiving program can be written in python, and there is a relatively mature python library available on the open source platform.

[0087] exist Figure 2 In the figure, the reference numerals are: 1. power wheel; 2. steering control servo motor; 3. motion motor and bogie; 4. power supply; 5. control microcontroller; 6. counterweight block; 7. lifting motor; 8. chain and gear; 9. guide rod; 10. safety wheel; 11. forklift arm; 12. universal wheel; 13. myoelectric arm ring; 14. host computer (PC).

[0088] When the AGV is controlled to move in the horizontal space, the connection relationship is as follows: the power wheel 1 is connected to the motion motor and the motion motor in the bogie 3, and the motion motor directly provides linear motion power for the power wheel 1 inside the bogie; the large gear on the top of the bogie cooperates with the small gear of the steering control servo motor 2, and the steering control servo motor 2 provides 360-degree steering power; a pair of universal wheels 12 and a pair of safety wheels 10 embedded in the forklift arm 11 facilitate steering and support.

[0089] When the AGV is controlled to vertically lift the forklift arm, the forklift arm 11 can move vertically along the guide rod 9, and the lifting motor 7 provides vertical power through the chain and gear 8; two counterweight blocks 6 are installed inside the vehicle body to keep the center of gravity of the AGV at the front to prevent the risk of rollover.

[0090] When the myoelectric control system is implemented, the power supply 4 provides power for the entire control system and power system. In this case, the power supply is a removable rechargeable battery; the control system emits electromyographic signals by the myoelectric armband 13, and the host computer (PC) 14 or the control microcontroller 5 receives and processes the signals; the control microcontroller 5 uses the obtained gesture tags to control the motion system, and the directly connected motion system units include the steering control servo motor 2, the motion motor and bogie 3, and the lifting motor 7.

[0091] The host computer (PC) or control microcontroller is equipped with signal acquisition, receiving program and gesture training and recognition program. Its function is to directly receive multi-channel gesture signals transmitted by the electromyographic armband, process the data and predict gesture movements.

[0092] Using a host computer (PC) as a transfer station for industrial AGV control is an option, which has obvious advantages and disadvantages: the host computer (PC) has a significantly better data processing speed than general control microcontrollers, which can improve the sensitivity of industrial AGV control; but its position is relatively fixed, which limits the control scene to a certain space and cannot control the industrial AGV to move long distances. The host computer (PC) can use a control and recognition program adapted to the Windows system.

[0093] You can also directly use a control microcontroller (such as Raspberry Pi 4B) to burn the control program for gesture recognition and control. The advantage is that no other equipment other than the industrial AGV is needed as a transfer station, and the operator can directly control the industrial AGV. However, after testing, the algorithm processing efficiency of control microcontrollers such as Data Pi 4B is lower than that of the host computer (PC), and the recognition refresh rate needs to be reduced to maintain the accuracy of recognition. Therefore, there is a greater delay compared to using the host computer (PC) control. The control microcontroller uses a control and recognition program adapted to the Linux system. At the same time, the control microcontroller is also cheaper than the host computer (PC) in price.

[0094] In addition to the above two types of control modules, you can also choose a lightweight host computer such as a microcomputer as a control module. The advantage is that it can be directly installed on the industrial AGV and has higher processing efficiency than control microcontrollers such as the Raspberry Pi 4B.

[0095] After the control module receives, processes, and recognizes the gestures, it sends instructions to the functional modules for control. The main functional modules of industrial AGV include the mobile module and the handling module. The main functions of the mobile module include forward, backward, left turn, right turn, and stop; the main functions of the handling module include the lifting and lowering of the forklift arm.

[0096] The forward and backward functions of the mobile module can be completed by a DC reduction motor, which receives a PWM signal from the control microcontroller and controls the power wheel to run or stop at a certain speed through the power system; the left turn and right turn functions are completed by a steering gear, which receives a control signal from the control microcontroller and controls the steering wheel to turn left or right through the power system.

[0097] The handling module controls the lifting and lowering of the forklift arm, which can be completed by a stepper motor. The stepper motor receives a control signal from the control microcontroller and controls the lifting or lowering of the forklift arm through a power system.

[0098] When using a DC reduction motor, a motor driver board (such as L298N) is required; when using a stepper motor and a servo, a corresponding driver module is required. The motor driver module must be connected to the GND, VCC, and signal serial port of the control microcontroller according to the instructions.

[0099] In another exemplary embodiment of the present invention, Figure 3 As shown, a set of intuitive and easy-to-learn gesture recognition control cases are provided. The correspondence between intuitive gestures and industrial AGV actions can make the operation process more concise and efficient. The control system can also customize gesture information during the training process to ensure the comfort of the operator; however, it is recommended to open a relatively large gap between different gestures to ensure the accuracy of the gesture recognition program.

[0100] The present invention has the following beneficial effects:

[0101] 1) The present invention provides an application of a multimodal information acquisition device in a control system, and the established gesture recognition program has good recognition accuracy on the control microcontroller.

[0102] 2) The present invention uses the multimodal signal composed of surface electromyography signal and IMU signal as the control signal of industrial AGV, which has higher flexibility and convenient operation compared with the traditional control system, and has good one-hand operability and is suitable for operators with hand disabilities; gesture control conforms to objective cognition, making the operation method easy to learn and reducing the error rate of operation. Gesture control can be customized to conform to the operator's operating habits.

[0103] In another exemplary embodiment of the present invention, Figure 4 As shown, a method for controlling a multimodal gesture recognition signal is provided, and the method for controlling a multimodal gesture recognition signal comprises:

[0104] S1: Acquire a multimodal gesture signal to be measured; the multimodal gesture signal to be measured includes: a surface electromyography signal and an IMU signal.

[0105] S2: Inputting the multimodal gesture signal to be tested into the gesture recognition model to obtain a gesture action; the gesture recognition model is a machine learning model.

[0106] S3: Based on the gesture action, control the action module to complete the action.

[0107] As an optional implementation, in step S2, the method for determining the gesture recognition model specifically includes:

[0108] C1: Acquire a training set; the training set includes: different preset actions and preset multimodal gesture signals corresponding to the different preset actions.

[0109] C2: Taking the preset multimodal gesture signal as input and the preset action corresponding to the preset multimodal gesture signal as output, the machine learning model is trained to obtain a gesture recognition model.

[0110] Specifically, in step C1, it specifically includes:

[0111] C11: Data is collected according to different preset actions to obtain a number of preset multimodal gesture signals.

[0112] C12: performing normalization processing and format processing on each of the preset multimodal gesture signals to obtain a plurality of processed preset multimodal gesture signals.

[0113] C13: different preset actions and the processed preset multimodal gesture signals corresponding to the different preset actions are used as training sets.

[0114] As an optional implementation, in step S2, specifically including:

[0115] S21: inputting the multimodal gesture signal to be tested into the gesture recognition model, outputting several groups of gesture recognition labels and caching them.

[0116] S22: taking the tags in the cache that are greater than or equal to the preset threshold as the finally recognized gesture action.

[0117] As an optional implementation, different gesture actions have a one-to-one correspondence with different actions of the action module.

[0118] The present invention also provides an application scenario, which applies the above-mentioned control method based on multimodal gesture action recognition signal. Specifically: The control method based on multimodal gesture action recognition signal provided in this embodiment can be applied in the industrial AGV control scenario. The industrial AGV control scenario includes: a multimodal gesture signal acquisition link, a gesture recognition link and an action link; after acquiring the multimodal gesture signal, it is input into the gesture recognition model to obtain the gesture action, and then according to the one-to-one correspondence between different gesture actions and different actions of the action module, the industrial AGV can complete the corresponding action.

[0119] In an exemplary embodiment, a computer device is provided. The computer device may be a server or a terminal. The internal structure diagram thereof may be as follows: Figure 5As shown. The computer device includes a processor, a memory, an input / output interface (Input / Output, referred to as I / O) and a communication interface. The processor, the memory and the input / output interface are connected through a system bus, and the communication interface is connected to the system bus through the input / output interface. The processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The database of the computer device is used to store multimodal gesture signals. The input / output interface of the computer device is used to exchange information between the processor and an external device. The communication interface of the computer device is used to communicate with an external terminal through a network connection. When the computer program is executed by the processor, a control method based on multimodal gesture action recognition signals is implemented.

[0120] Those skilled in the art will understand that Figure 5 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present invention, and does not constitute a limitation on the computer device to which the solution of the present invention is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0121] In an exemplary embodiment, a computer device is further provided, including a memory and a processor, wherein a computer program is stored in the memory, and the above-mentioned method embodiments are implemented when the processor executes the computer program.

[0122] In an exemplary embodiment, a computer-readable storage medium is provided, storing a computer program, and when the computer program is executed by a processor, the above-mentioned method embodiments are implemented.

[0123] In an exemplary embodiment, a computer program product is provided, including a computer program, and when the computer program is executed by a processor, the above-mentioned method embodiments are implemented.

[0124] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in the present invention are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with relevant regulations.

[0125] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiments can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to the memory, database or other medium used in the embodiments provided by the present invention can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetoresistive random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM may be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM).

[0126] The database involved in each embodiment provided by the present invention may include at least one of a relational database and a non-relational database. The non-relational database may include a distributed database based on blockchain, etc., but is not limited thereto. The processor involved in each embodiment provided by the present invention may be a general-purpose processor, a central processing unit, a graphics processor, a digital signal processor, a programmable logic device, a data processing logic device based on quantum computing, etc., but is not limited thereto.

[0127] The technical features of the above embodiments may be arbitrarily combined. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0128] The principles and implementation methods of the present invention are described in this article using specific examples. The description of the above embodiments is only used to help understand the method and core idea of ​​the present invention. At the same time, for those skilled in the art, according to the idea of ​​the present invention, there will be changes in the specific implementation methods and application scope. In summary, the content of this specification should not be understood as limiting the present invention.

Claims

1. An industrial AGV based on multi-modal gesture recognition signal control, characterized in that: The industrial AGV based on multi-modal gesture recognition signal control includes: an electromyographic armband, a control module, a Bluetooth remote control component and an action module; The myoelectric armband is used to collect multimodal gesture signals and send the multimodal gesture signals to the Bluetooth remote control component; the multimodal gesture signals include: surface electromyography signals and IMU signals; The control module includes a training program and a control program; the training program includes: an electromyogram armband signal acquisition program and a model training program; the electromyogram armband signal acquisition program is used to control the electromyogram armband to acquire multimodal gesture signals; the model training program is used to train the machine learning model to obtain a gesture recognition model; the control program includes: an electromyogram armband signal receiving program, a gesture recognition program and a motor control program; the electromyogram armband signal receiving program is used to receive multimodal gesture signals sent by the Bluetooth remote control component; the gesture recognition program is used to recognize gesture actions according to the multimodal gesture signals; the motor control program is used to control the action module to complete the action according to the gesture action.

2. The industrial AGV based on multi-modal gesture recognition signal control according to claim 1 is characterized in that: The control module includes: a control single chip microcomputer and a host computer; the control single chip microcomputer includes: a host computer control program and an independent control program; The host computer control program is used to passively receive the control signal transmitted by the host computer through the Bluetooth remote control component; The independent control program is used to identify gesture actions according to the multimodal gesture signal, and control the action module to complete the action according to the gesture action.

3. The industrial AGV based on multi-modal gesture recognition signal control according to claim 1 is characterized in that: The model training program is used to train the machine learning model to obtain a gesture recognition model, which specifically includes: Acquire a training set; the training set includes: different preset actions and preset multimodal gesture signals corresponding to the different preset actions; The machine learning model is trained with the preset multimodal gesture signal as input and the preset action corresponding to the preset multimodal gesture signal as output to obtain a gesture recognition model.

4. The industrial AGV based on multi-modal gesture recognition signal control according to claim 3 is characterized in that: Get the training set, including: Data is collected according to different preset actions to obtain a number of preset multimodal gesture signals; Performing normalization processing and format processing on each of the preset multimodal gesture signals to obtain a plurality of processed preset multimodal gesture signals; Different preset actions and processed preset multimodal gesture signals corresponding to the different preset actions are used as training sets.

5. The industrial AGV based on multi-modal gesture recognition signal control according to claim 1, characterized in that: The gesture recognition program is used to recognize gesture actions according to the myoelectric armband signal, specifically including: Output several groups of gesture recognition tags and cache them; The tags in the cache that are greater than or equal to the preset threshold are taken as the final recognized gesture actions.

6. A method for controlling multimodal gesture recognition signals, characterized in that: The multi-modal gesture recognition signal control method includes: Acquire a multimodal gesture signal to be tested; the multimodal gesture signal to be tested includes: a surface electromyography signal and an IMU signal; Inputting the multimodal gesture signal to be tested into the gesture recognition model to obtain a gesture action; the gesture recognition model is a machine learning model; Based on the gesture action, the action module is controlled to complete the action.

7. The method for controlling a multi-modal gesture recognition signal according to claim 6, characterized in that: The method for determining the gesture recognition model specifically includes: Acquire a training set; the training set includes: different preset actions and preset multimodal gesture signals corresponding to the different preset actions; The machine learning model is trained with the preset multimodal gesture signal as input and the preset action corresponding to the preset multimodal gesture signal as output to obtain a gesture recognition model.

8. The method for controlling a multi-modal gesture recognition signal according to claim 7, characterized in that: Get the training set, including: Data is collected according to different preset actions to obtain a number of preset multimodal gesture signals; Performing normalization processing and format processing on each of the preset multimodal gesture signals to obtain a plurality of processed preset multimodal gesture signals; Different preset actions and processed preset multimodal gesture signals corresponding to the different preset actions are used as training sets.

9. The method for controlling a multi-modal gesture recognition signal according to claim 6, characterized in that: Inputting the multimodal gesture signal to be tested into the gesture recognition model to obtain a gesture action specifically includes: Inputting the multimodal gesture signal to be tested into the gesture recognition model, outputting several groups of gesture recognition labels and caching them; The tags in the cache that are greater than or equal to the preset threshold are taken as the final recognized gesture actions.

10. The method for controlling multimodal gesture recognition signals according to claim 6, characterized in that: There is a one-to-one correspondence between different gesture actions and different actions of the action module.

Images