Two-wheeled electric vehicle and gesture recognition method and device for interactive control of two-wheeled electric vehicle

Abstract

The invention provides a two-wheeled electric vehicle and a gesture recognition method and device for interaction control of the two-wheeled electric vehicle, and relates to the technical field of two-wheeled electric vehicle man-machine interaction control. The method comprises the steps that if a recognition device recognizes lens change of a camera, a target image frame is read, hand key node detection is carried out, and a target image is obtained; according to the hand key node coordinates and the threshold range of each preset gesture, gesture recognition is carried out; the threshold range is composed of at least one sub-threshold range of a vector included angle, and the vector is composed of coordinates of two hand key nodes; when the recognition device successfully recognizes the target gesture, a corresponding target operation is determined, a report instruction is generated, and the report instruction is sent to the control device; and when the control device determines to execute the target operation according to the reported instruction, the working state of the two-wheeled electric vehicle is controlled according to the target operation. According to the system, high reliability and safety can be provided, the requirements of modern consumers for intelligence and convenience can be met, the development cost is reduced, and the development period is shortened.

Description

Technical Field

[0001] The present invention relates to the technical field of human-computer interaction control for two-wheeled electric vehicles, and particularly to a two-wheeled electric vehicle and a gesture recognition method and device for its interaction control. Background Art

[0002] With the popularization of two-wheeled electric vehicles, many problems have gradually emerged in the traditional mechanical key and remote control key interaction methods. Although mechanical keys have high reliability, their physical characteristics make them easy to lose and copy, and they cannot meet the needs of modern consumers for intelligence and convenience. Although remote control keys provide a convenient unlocking method, they rely on battery power supply and signal transmission, have a risk of failure, and lack intelligent functions.

[0003] In recent years, the interaction method between intelligent APPs and two-wheeled electric bicycles has gradually emerged. However, due to compatibility problems between different brands and models, as well as the complexity and safety hazards of the operation interface, its popularization and application are restricted. In addition, although new interaction methods such as NFC (Near Field Communication) card swiping and fingerprint unlocking provide higher security, their binary nature, that is, they usually can only recognize binary results such as "yes" or "no", "matched" or "unmatched", limits their application in complex scenarios.

[0004] Therefore, there is an urgent need for a new type of interaction control method for two-wheeled electric vehicles that can provide high reliability and security, and meet the needs of modern consumers for intelligence and convenience. Summary of the Invention

[0005] In view of this, in order to solve the above technical problems, the present invention provides a two-wheeled electric vehicle and a gesture recognition method and device for its interaction control.

[0006] The present invention adopts the following technical solutions: In a first aspect, the present invention provides a gesture recognition method for two-wheeled electric vehicle interaction control, including: After the recognition device determines that the user authentication is passed, if it recognizes a change in the lens of the camera of the two-wheeled electric vehicle, it controls the camera to capture a real-time video stream; The recognition device reads the target image frame in the video stream; The recognition device uses the MediaPipe model to detect the hand key nodes of the target image frame to obtain a detection result; When the recognition device determines that the detection result shows that the hand key node coordinates are successfully obtained, gesture recognition is performed according to the hand key node coordinates and the threshold ranges of each preset gesture to obtain a gesture recognition result; the threshold range is composed of at least one sub-threshold range of the vector angle, and the vector is a vector formed according to the coordinates of two of the hand key nodes; When the recognition device determines that the gesture recognition result shows that the target gesture is successfully recognized, it determines the target operation corresponding to the target gesture; the target gesture is one of the preset gestures; The recognition device generates a reporting instruction according to the target operation and sends the reporting instruction to the control device; When the control device determines to execute the target operation according to the reporting instruction, it controls the working state of the two-wheeled electric vehicle according to the target operation.

[0007] Optionally, performing gesture recognition according to the hand key node coordinates and the threshold ranges of each preset gesture to obtain a gesture recognition result specifically includes: The following operations are sequentially performed for each of the preset gestures until the target gesture is recognized or all the preset gestures are polled: According to the hand key node coordinates, calculate the target vector angles corresponding to each of the sub-threshold ranges in the threshold range of the current preset gesture; When all the target vector angles belong to the corresponding sub-threshold ranges, determine that the current preset gesture is the target gesture; When there is a target vector angle that does not belong to the corresponding sub-threshold range among all the target vector angles, determine that the current preset gesture is not the target gesture.

[0008] Optionally, after performing gesture recognition according to the hand key node coordinates and the threshold ranges of each preset gesture to obtain a gesture recognition result, the gesture recognition method for two-wheeled electric vehicle interaction control of the present invention further includes: When the recognition device determines that the gesture recognition result shows that an unknown gesture is recognized, it determines whether it receives a gesture setting instruction sent by the control device; the control device generates the gesture setting instruction according to the user's gesture setting operation; the gesture setting instruction includes at least one gesture and its corresponding operation; If the gesture setting instruction sent by the control device is received, the recognition device performs corresponding settings according to the gesture setting instruction; The recognition device feeds back a gesture setting result to the control device.

[0009] Optionally, after determining whether the gesture setting instruction sent by the control device is received, the gesture recognition method for two-wheeled electric vehicle interaction control of the present invention further includes: If the gesture setting instruction sent by the control device is not received, the recognition device determines that there is no need to set an operation for the unknown gesture; The recognition device re-executes the step of reading the target image frame in the video stream.

[0010] Optionally, before controlling the camera to capture a real-time video stream if a lens change of the camera of the two-wheeled electric vehicle is recognized, the gesture recognition method for two-wheeled electric vehicle interaction control of the present invention further includes: When the control device detects that the intensity of the Bluetooth signal sent by a Bluetooth device is greater than a preset signal intensity threshold, the control device obtains the MAC address of the Bluetooth device; The control device compares the MAC address with the pre-stored vehicle owner information; If the MAC address matches the vehicle owner information successfully, the control device determines that the corresponding user authentication is passed.

[0011] Optionally, the recognition device generates a reporting instruction according to the target operation and sends the reporting instruction to the control device, specifically including: The recognition device obtains the current working state of the two-wheeled electric vehicle through the control device; The recognition device determines whether there is a logical conflict between the current working state and the target operation; If there is a logical conflict between the current working state and the target operation, the recognition device generates a first reporting instruction carrying error prompt information and sends the first reporting instruction to the control device; If there is no logical conflict between the current working state and the target operation, the recognition device generates a second reporting instruction carrying the target operation and sends the second reporting instruction to the control device.

[0012] Optionally, after the recognition device uses the MediaPipe model to detect the hand key nodes of the target image frame and obtains the detection result, the gesture recognition method for two-wheeled electric vehicle interaction control of the present invention further includes: When the recognition device determines that the detection result shows that the hand key node coordinates are not successfully obtained, the recognition device re-executes the step of reading the target image frame in the video stream.

[0013] Based on a general inventive concept, the present invention also provides a gesture recognition device for two-wheeled electric vehicle interaction control, including: a recognition device, a control device, a first communication device, and a camera; The recognition device is communicatively connected to the control device through the first communication device, and the recognition device is also connected to the camera; The recognition device, the control device, the first communication device, and the camera are used to execute the gesture recognition method for two-wheeled electric vehicle interaction control as described above.

[0014] Optionally, the gesture recognition device for two-wheeled electric vehicle interaction control of the present invention further includes: a second communication device connected to the control device; The second communication device is a wireless communication device for supporting wireless communication between the control device and external devices.

[0015] Based on a general inventive concept, the present invention also provides a two-wheeled electric vehicle, including: the gesture recognition device for two-wheeled electric vehicle interaction control as described above.

[0016] The present invention adopts the above technical solutions. Since gesture recognition does not rely on physical media or signal transmission, it avoids failure problems caused by signal interference, battery depletion, or physical damage. Moreover, gesture recognition can adapt to different lighting conditions, background environments, and the diversity of user gestures, and has high robustness, which makes the present invention have high reliability; secondly, gesture recognition is difficult to be replicated or forged, making the present invention have high security; furthermore, gesture recognition can not only be used for unlocking, but also support various intelligent interaction functions, such as gesture control of vehicle startup, mode switching, and volume adjustment, etc., providing a richer intelligent experience for users, so that the present invention can meet the user's demand for intelligence; finally, gesture recognition enables users not to carry keys, remote controls, or mobile phones, etc., and there is no need to learn complex operation procedures, so that the present invention can meet the user's demand for convenience and improve the user's interaction experience.

[0017] In addition, the present invention pre-sets the threshold range of each preset gesture. Subsequently, when obtaining the coordinates of the key hand nodes, gesture recognition is performed according to the coordinates of the key hand nodes and the threshold range of each preset gesture, so that the present invention does not require a complex machine training process, reduces the development cost, and shortens the development cycle. Brief Description of the Drawings

[0018] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the following will briefly introduce the accompanying drawings required for the description of the embodiments or the prior art. Obviously, the accompanying drawings in the following description are only some embodiments of the present invention. For those of ordinary skill in the art, without creative efforts, other accompanying drawings can be obtained based on these drawings.

[0019] Figure 1 is a schematic flowchart of a gesture recognition method for two-wheeled electric vehicle interaction control provided by an embodiment of the present invention; Figure 2 is a schematic diagram of 21 hand key nodes; Figure 3 is a schematic diagram of a gesture provided by an embodiment of the present invention; Figure 4 is a schematic structural diagram of a gesture recognition device for two-wheeled electric vehicle interaction control provided by an embodiment of the present invention. Detailed implementation manners

[0020] To make the purpose, technical solutions and advantages of the present invention clearer, the technical solutions of the present invention will be described in detail below. Obviously, the described embodiments are only a part of the embodiments of the present invention, rather than all the embodiments. Based on the embodiments of the present invention, all other implementation manners obtained by those of ordinary skill in the art without creative efforts belong to the scope protected by the present invention.

[0021] With the popularization of two-wheeled electric vehicles, many problems have gradually emerged in the traditional mechanical key and remote control key interaction methods. Although mechanical keys have high reliability, their physical characteristics make them easy to lose and copy, and they cannot meet the needs of modern consumers for intelligence and convenience. Although remote control keys provide a convenient unlocking method, they rely on battery power supply and signal transmission. When the battery runs out or the signal is interfered, there is a risk of failure, and they lack intelligent functions.

[0022] In recent years, the interaction method between smart APPs and two-wheeled electric bicycles has gradually emerged. However, due to compatibility issues between different brands and models, as well as the complexity of the operation interface, its popularization and application are restricted. Especially when users are not convenient to operate with mobile phones, there may be potential safety hazards on the operation page during riding. In addition, although new interaction methods such as NFC card swiping and fingerprint unlocking provide higher security, their binary nature, that is, they usually can only recognize binary results such as "yes" or "no", "match" or "no match", limits their application in complex scenarios. Also, fingerprint unlocking may be affected by environmental factors such as wet, dirty or injured fingers, resulting in a decrease in recognition accuracy, and NFC card swiping may also be affected by physical obstacles or signal interference.

[0023] Based on this, the present invention provides a two-wheeled electric vehicle and a gesture recognition method and device for its interactive control, which can not only provide high reliability and safety, but also meet the needs of modern consumers for intelligence and convenience. The technical solution of the present invention will be described in detail below with reference to the accompanying drawings.

[0024] Figure 1 It is a schematic flowchart of a gesture recognition method for the interactive control of a two-wheeled electric vehicle provided by an embodiment of the present invention. As Figure 1 shown, the gesture recognition method for the interactive control of the present two-wheeled electric vehicle includes: Step 101: After the recognition device determines that the user authentication is passed, if it recognizes a change in the lens of the camera of the two-wheeled electric vehicle, it controls the camera to capture a real-time video stream.

[0025] Specifically, when the user needs to send a control instruction to the two-wheeled electric vehicle, the user can make a corresponding gesture in front of the lens of the camera of the two-wheeled electric vehicle to send a control instruction to the two-wheeled electric vehicle. At this time, the lens of the camera will change. After the recognition device determines that the user authentication is passed, if it recognizes a change in the lens of the camera of the two-wheeled electric vehicle, it controls the camera to exit the low-power mode and capture a real-time video stream.

[0026] Step 102: The recognition device reads the target image frame in the video stream.

[0027] Specifically, the target image frame can be any frame in the video stream.

[0028] Step 103: The recognition device uses the MediaPipe model to detect the hand key nodes of the target image frame to obtain a detection result.

[0029] Specifically, after the recognition device reads the target image frame in the video stream, it performs image preprocessing such as adjusting the resolution, cropping the ROI (Region of Interest), and removing noise on the target image frame, and then sends the preprocessed target image frame to the MediaPipe model to use the MediaPipe model to detect the hand key nodes of the target image frame to obtain a detection result.

[0030] The MediaPipe model provides a series of pre-trained neural network models, including a hand key node detection model, and 21 hand key nodes are included in the hand key node detection model. Figure 2 It is a schematic diagram of 21 hand key nodes. As Figure 2As shown in the figure, the 21 key hand nodes include: wrist 0, thumb carpometacarpal joint 1, thumb metacarpophalangeal joint 2, thumb proximal interphalangeal joint 3, thumb fingertip 4, index metacarpophalangeal joint 5, index proximal interphalangeal joint 6, index distal interphalangeal joint 7, index fingertip 8, middle metacarpophalangeal joint 9, middle proximal interphalangeal joint 10, middle distal interphalangeal joint 11, middle fingertip 12, ring metacarpophalangeal joint 13, ring proximal interphalangeal joint 14, ring distal interphalangeal joint 15, ring fingertip 16, little metacarpophalangeal joint 17, little proximal interphalangeal joint 18, little distal interphalangeal joint 19, little fingertip 20.

[0031] The construction of the above-mentioned key hand node detection model of the present invention is applied to the hand detection model, gesture recognition algorithm and gesture library of the present invention. The specific construction method is the prior art and will not be elaborated here.

[0032] The hand detection model is aimed at obtaining the coordinates of key hand nodes and performs key hand node detection on the target image frame. During the detection process, first, specific interfaces and methods are used to process the target image frame to convert it into a data format suitable for model analysis. Then, the target image frame after format conversion is input into the hand detection model. The hand detection model is used to detect the coordinates of key hand nodes of the target image frame and output the detection result. When the detection result shows that the coordinates of key hand nodes are successfully obtained, the recognition device reports a gesture acquisition success instruction to the control device. The control device controls the instrument of the vehicle to display a gesture acquisition success prompt according to the gesture acquisition success instruction. When the detection result shows that the coordinates of key hand nodes are not successfully obtained, the recognition device reports a gesture acquisition failure instruction to the control device. The control device controls the instrument of the vehicle to display a gesture acquisition failure prompt according to the gesture acquisition failure instruction, so that the user can stop making gestures or continue making gestures according to the prompt.

[0033] In addition, after the control camera captures the real-time video stream, the recognition device also continuously monitors the lens change state of the camera. When it is detected that the lens of the camera remains unchanged for more than the first preset duration, if it is determined that the coordinates of key hand nodes have been successfully obtained, or other situations indicating that there is no need to continue capturing the video stream are determined, all occupied camera resources are released.

[0034] Conversely, when it is detected that the lens of the camera is continuously changing, if it is determined that the coordinates of key hand nodes have not been successfully obtained, or other situations indicating that it is necessary to continue capturing the video stream are determined, step 102 is re-executed: reading the target image frame in the video stream.

[0035] Based on this, in the embodiment of the present invention, after the recognition device uses the MediaPipe model to perform key hand node detection on the target image frame and obtains the detection result, the gesture recognition method for two-wheeled electric vehicle interaction control of the present invention may further include: When the recognition device determines that the detection result shows that the coordinates of the key hand nodes have not been successfully obtained, the recognition device re-executes the step of reading the target image frame in the video stream.

[0036] Step 104: When the recognition device determines that the detection result shows that the coordinates of the key hand nodes have been successfully obtained, gesture recognition is performed according to the coordinates of the key hand nodes and the threshold ranges of each preset gesture to obtain a gesture recognition result; the threshold range is composed of at least one sub-threshold range of the vector angle, and the vector is a vector formed according to the coordinates of two key hand nodes.

[0037] Specifically, the encoding and threshold range of each preset gesture are stored in the gesture library.

[0038] Based on the hand detection model, a gesture recognition algorithm can be further implemented. The gesture recognition algorithm is a classification algorithm that can recognize the detected hand image area as different gesture categories, such as being recognized as Figure 3 one of the three gestures shown.

[0039] In traditional gesture recognition machine learning, usually a complex and large number of gesture libraries and gesture models are required to train the machine, and this process is extremely cumbersome, usually requiring investment of manpower, cost and a large amount of time. Based on this, the present invention extracts the unique features of each preset gesture and plans a unique threshold range for each preset gesture according to the extracted features. Subsequently, gesture recognition can be performed according to the coordinates of the key hand nodes and the threshold ranges of each preset gesture. This enables the present invention to avoid a complex machine training process, reduces the development cost, and shortens the development cycle. It should be noted that the present invention does not specifically limit the threshold range herein, and those skilled in the art can set the threshold range according to the actual situation.

[0040] For example, referring to Figure 2 and Figure 3 , when judging Gesture 1, Gesture 2 and Gesture 3, assuming under the two-dimensional plane condition, it is necessary to use Figure 2Among them, there are 13 hand key nodes numbered 0, 5 - 16. Among them, for gesture 1, when we stretch out the index finger alone, the vector angle between vector <0,5> and vector <0,6> is less than 5 degrees, and the vector angle between vector <0,6> and vector <0,7> is also less than 5 degrees. Therefore, the threshold range of gesture 1 includes at least two sub - threshold ranges. One is that the vector angle between vector <0,5> and vector <0,6> is less than 5 degrees, and the other is that the vector angle between vector <0,6> and vector <0,7> is less than 5 degrees. It should be noted that vector <0,5> is the vector pointing from the hand key node numbered 0 to the hand key node numbered 5, and the modulus of this vector is equal to the distance between the two hand key nodes. The same applies to other vectors and will not be elaborated here. It can be seen that the present invention uses the hand key node numbered 0 as the coordinate origin.

[0041] Moreover, when the user makes gesture 2, the index finger joint will be squeezed to the side due to the extension of the middle finger. At this time, the vector angle between vector <0,5> and vector <0,6> is greater than 5 degrees, and the vector angle between vector <0,6> and vector <0,7> is also greater than 5 degrees. And at this time, the vector angle between vector <0,8> and vector <0,9> generates a unique sub - threshold range because there is no other finger squeezing. The same applies to gesture 3 and will not be elaborated here. Those skilled in the art can plan a unique threshold range for each preset gesture according to the actual situation. The threshold range is composed of at least one sub - threshold range of vector angles, and the vector is formed according to the coordinates of two hand key nodes.

[0042] In the embodiment of the present invention, gesture recognition is performed according to the hand key node coordinates and the threshold ranges of each preset gesture to obtain a gesture recognition result, which specifically may include: Perform the following operations for each preset gesture in sequence until the target gesture is recognized or all preset gestures are polled: (1041)According to the hand key node coordinates, calculate the target vector angles corresponding to each sub - threshold range in the threshold range of the current preset gesture.

[0043] (1042)When all target vector angles belong to the corresponding sub - threshold ranges, determine that the current preset gesture is the target gesture; when there is a target vector angle that does not belong to the corresponding sub - threshold range among all target vector angles, determine that the current preset gesture is not the target gesture.

[0044] Specifically, a function named recognize_complex_gesture (recognize complex gestures) is deployed in the recognition device to compare and recognize hand gestures. In addition, a function named vectorial_angle (vector angle) is deployed in the recognition device to calculate the target vector angle (in degrees). The gesture recognition algorithm includes the recognize_complex_gesture function and the vectorial_angle function.

[0045] The working process of the recognize_complex_gesture function is as follows: (1) Construct a dictionary named hand_landmarks (hand key nodes) based on the hand key node coordinates and input it into the recognize_complex_gesture function.

[0046] (2) Perform the following operations for each preset gesture in sequence until the target gesture is recognized or all preset gestures are polled: (2.1) Call the vectorial_angle function to calculate the target vector angles corresponding to each sub-threshold range in the threshold range of the current preset gesture.

[0047] In the calculation process of any target vector angle, first, determine the two vectors corresponding to the current target vector angle, namely vector V1 and vector V2, and input these two vectors into the vectorial_angle function. Here, taking vector V1 as vector <0, 5> and vector V2 as vector <0, 6> as an example for illustration. As can be seen from the foregoing, these two vectors are vectors under two-dimensional plane conditions. Therefore, each vector consists of two elements, respectively representing the components of the vector on the x-axis and y-axis. In addition, the hand key node coordinates can also be coordinates in a three-dimensional rectangular coordinate system, and those skilled in the art can select the coordinate system according to the actual situation.

[0048] Then, extract the x component from vector V1 and the y component . Similarly, extract the x component from vector V2 and the y component . Calculate the dot product dot_product of the two vectors, and the calculation formula is: . Then, calculate the magnitudes of the two vectors. Among them, the magnitude magnitude_v1 of vector V1 is: , and the magnitude magnitude_v2 of vector V2 is: 。Next, by dividing the dot product by the product of the magnitudes of the two vectors, the cosine value cos_angle of the angle between the two vectors is obtained, i.e., cos_angle = dot_product / (magnitude_v1 *magnitude_v2).

[0049] Next, the math.acos() function is used to calculate the radian value angle of the current angle, i.e., angle = math.acos(cos_angle). Finally, the math.degrees() function is used to convert the calculated radian value to degrees (i.e., the angle between the target vectors). The vectorial_angle function finally returns this degree, representing the angle size between vector V1 and vector V2.

[0050] (2.2) Compare each target vector angle with the corresponding sub-threshold range. When all target vector angles belong to the corresponding sub-threshold range, determine that the current preset gesture is the target gesture; when there is a target vector angle that does not belong to the corresponding sub-threshold range among all target vector angles, determine that the current preset gesture is not the target gesture.

[0051] (3) When the target gesture is recognized, return the gesture code of the target gesture; when it is determined that there is no preset gesture matching the current gesture after polling all preset gestures, return an unknown gesture prompt.

[0052] Step 105: When the recognition device determines that the gesture recognition result shows that the target gesture is successfully recognized, determine the target operation corresponding to the target gesture; the target gesture is a preset gesture.

[0053] Specifically, the present invention pre-stores the operations corresponding to each preset gesture. Subsequently, after the target gesture is determined, the target operation corresponding to the target gesture can be determined according to this corresponding relationship.

[0054] The target operation may include: ACC (Accessory) power on, ACC power off, disarming, arming, turning on TCS (Traction Control System), turning off TCS, turning on hill descent control, turning off hill descent control, turning on seat heating, turning off seat heating, turning on cloud power, turning off cloud power, turning on assisted push, turning off assisted push, turning on reverse, turning off reverse, turning on ramp parking and turning off ramp parking, etc.

[0055] Step 106: The recognition device generates a reporting instruction according to the target operation and sends the reporting instruction to the control device.

[0056] In the embodiment of the present invention, the recognition device generates a reporting instruction according to the target operation and sends the reporting instruction to the control device, which may specifically include: (1) The recognition device obtains the current working state of the two-wheeled electric vehicle through the control device.

[0057] Specifically, the recognition device and the control device communicate through the RS-485 communication method. The control device is used to obtain the current working state of the two-wheeled electric vehicle. Moreover, the control device broadcasts a working state broadcast packet carrying the current working state of the two-wheeled electric vehicle every second preset time period (such as 5S). In the working state broadcast packet, the RS-485 address is 0x00, the communication protocol version number is fixed at 0xA0, the current working state is represented by the hexadecimal byte 0x00. The hexadecimal byte can be converted into binary, and each bit in the binary can be used to represent a state of the corresponding function. 0 represents the state off, and 1 represents the state on.

[0058] The recognition device will only obtain the current working state of the two-wheeled electric vehicle according to the byte position in the broadcast packet when it receives a broadcast packet starting with 0x00. For example, the broadcast packet is: 0x00 0xA0 0x00 0x00 sum, where 0x00, 0xA0, 0x00, 0x00, sum are the RS-485 address, communication protocol version number, current working state, current working state, and checksum respectively. According to this broadcast packet, it can be determined that the current working state is the state where all functions are turned off.

[0059] (2) The recognition device determines whether there is a logical conflict between the current working state and the target operation.

[0060] Specifically, after obtaining the current working state of the vehicle, the recognition device determines whether there is a logical conflict between the current working state and the target operation.

[0061] In a specific example, when the entire vehicle is in the ACC power-on state and the target operation is ACC power-on, it is determined that there is a logical conflict between the current working state and the target operation at this time.

[0062] In another specific example, when the entire vehicle has a wheel movement signal and the target operation is ACC power-off or anti-theft setting, it is determined that there is a logical conflict between the current working state and the target operation at this time.

[0063] (3) If there is a logical conflict between the current working state and the target operation, the recognition device generates a first reporting instruction carrying error prompt information and sends the first reporting instruction to the control device; the control device controls the instrument of the vehicle to display an operation conflict prompt according to the first reporting instruction so that the user can know the current conflict situation according to the operation conflict prompt.

[0064] Conversely, if there is no logical conflict between the current working state and the target operation, the recognition device generates a second reporting instruction carrying the target operation and sends the second reporting instruction to the control device.

[0065] Specifically, the second reporting instruction may include a gesture packet and an operation packet. The gesture packet contains the encoding of the target gesture, and the operation packet contains the encoding of the target operation. In the second reporting instruction, the RS-485 address is 0x01, the communication protocol version number is fixed at 0xA0, the gesture encoding is one byte, and the operation encoding is one byte. In a specific example, assuming there are a total of 15 preset gestures, in the second reporting instruction, the gesture encoding byte can be 0x00, 0x01, 0x02,..., or 0x0E. Among them, 0x00, 0x01, 0x02,..., 0x0E represent gesture No. 0, gesture No. 1, gesture No. 2,..., gesture No. 14 respectively, and gesture No. 14 is a user-defined gesture.

[0066] In another specific example, the target gesture is gesture No. 0, and the corresponding target operation is operation No. 1, that is, the encoding of the target gesture is 0, and the encoding of the target operation is 1. Then the second reporting instruction can be: 0x01 0xA0 0x00 0x01 sum. Among them, 0x01, 0xA0, 0x00, 0x01, sum are the RS-485 address, communication protocol version number, gesture encoding, operation encoding, and checksum respectively.

[0067] In the embodiment of the present invention, before the recognition device generates a reporting instruction according to the target operation and sends the reporting instruction to the control device, the gesture recognition method for two-wheeled electric vehicle interactive control of the present invention may further include: After the recognition device determines the target operation corresponding to the target gesture, it starts a delay of a third preset duration, which may be equal to 500 ms, for filtering possible interference signals. After the delay ends, the recognition device generates a reporting instruction according to the target operation and sends the reporting instruction to the control device.

[0068] Step 107: When the control device determines to execute the target operation according to the reporting instruction, it controls the working state of the two-wheeled electric vehicle according to the target operation.

[0069] Specifically, if the target operation involves operations of the control device itself such as ACC power-on, it will be completed by the control device itself. If it involves operations related to other control devices such as turning off TCS and turning on hill descent control, the control device sends corresponding control instructions to the corresponding devices to achieve corresponding control.

[0070] With the above technical solutions, since gesture recognition does not rely on physical media or signal transmission, it avoids the failure problems caused by signal interference, battery depletion or physical damage. Moreover, gesture recognition can adapt to different lighting conditions, background environments and the diversity of user gestures, and has high robustness, which makes the present invention have high reliability. Secondly, gesture recognition is difficult to be copied or forged, making the present invention have high security. Furthermore, gesture recognition can not only be used for unlocking, but also support a variety of intelligent interaction functions, such as gesture control of vehicle start, mode switching and volume adjustment, etc., providing a richer intelligent experience for users, so that the present invention can meet the user's demand for intelligence. Finally, gesture recognition enables users not to carry keys, remote controls or mobile phones, etc., and there is no need to learn complex operation processes, so that the present invention can meet the user's demand for convenience and improve the user's interaction experience.

[0071] In addition, the present invention pre-sets the threshold ranges of each preset gesture. Subsequently, when the coordinates of the key hand nodes are obtained, gesture recognition is performed according to the coordinates of the key hand nodes and the threshold ranges of each preset gesture, so that the present invention does not need to perform a complex machine training process, reduces the development cost, and shortens the development cycle.

[0072] In the embodiment of the present invention, after gesture recognition is performed according to the coordinates of the key hand nodes and the threshold ranges of each preset gesture to obtain a gesture recognition result, the gesture recognition method for two-wheeled electric vehicle interaction control of the present invention may further include: (1) When the recognition device determines that the gesture recognition result shows that an unknown gesture is recognized, it judges whether it receives a gesture setting instruction sent by the control device; the control device generates a gesture setting instruction according to the user's gesture setting operation; the gesture setting instruction includes at least one gesture and its corresponding operation.

[0073] Specifically, when the user customizes a gesture, relevant operations can be performed in the mobile phone APP so that the mobile phone sends a gesture setting request carrying the operation corresponding to the unknown gesture to the control device. When the control device receives the gesture setting request, it obtains the unknown gesture, analyzes the unknown gesture to generate the threshold range of the unknown gesture, generates a gesture setting instruction according to the threshold range of the unknown gesture and the operation corresponding to the unknown gesture, and sends the gesture setting instruction to the recognition device.

[0074] In the gesture setting instruction, the RS-485 address is 0x0E, the communication protocol version number is fixed at 0xA0, there are a total of 15 setting bytes, and the high 4 bits of each setting byte represent the gesture encoding, while the low 4 bits represent the operation encoding. In a specific example, the gesture setting instruction is: 0x0E 0xA0 0x00 0x11 0x22 0x33 0x44 0x55 0x66 0x77 0x88 0x99 0xAA 0xBB 0xCC 0xDD 0xFF, where 0x00, 0x11, 0x22, 0x33, 0x44, 0x55, 0x66, 0x77, 0x88, 0x99, 0xAA, 0xBB, 0xCC, 0xDD, 0xFF are the 15 setting bytes, and 0xFF is the setting byte for the user-defined gesture. When the user has not defined a gesture, this byte is 0xFF.

[0075] (2) If the gesture setting instruction sent by the control device is received, the recognition device makes corresponding settings according to the gesture setting instruction and feeds back the gesture setting result to the control device.

[0076] Specifically, the gesture setting result (i.e., the response packet) includes the current set state of the recognition device and the byte indicating whether the setting is successful. When the current set state is consistent with the corresponding state in the gesture setting instruction, it is determined that the setting is successful, and the corresponding byte is 0x01; otherwise, the setting is not successful, and the corresponding byte is 0x00.

[0077] After receiving the gesture setting result, the control device generates a personalized version, which contains unknown gestures and their operation information. Then, the control device sends the personalized version to the user's mobile phone through OTA (Over-the-Air Technology) to update the personalized gesture function of the user's mobile phone.

[0078] (3) If the gesture setting instruction sent by the control device is not received, the recognition device determines that there is no need to set an operation for the unknown gesture and re-executes the step of reading the target image frame in the video stream.

[0079] In the embodiment of the present invention, before controlling the camera to capture a real-time video stream if a change in the lens of the camera of the two-wheeled electric vehicle is recognized, the gesture recognition method for two-wheeled electric vehicle interactive control of the present invention may further include: (1) When the control device detects that the intensity of the Bluetooth signal sent by a Bluetooth device is greater than the preset signal intensity threshold, it obtains the MAC (Media Access Control) address of the Bluetooth device.

[0080] (2) The control device compares the MAC address with the pre-stored owner information; (3) If the MAC address matches the vehicle owner information successfully, the control device determines that the corresponding user authentication is passed.

[0081] Specifically, when the user approaches the two-wheeled electric vehicle with a smart phone, if the phone Bluetooth is in the on state, the control device will detect the Bluetooth signal emitted by the phone. The control device detects the strength of the Bluetooth signal. When the detected strength of the Bluetooth signal is greater than the preset signal strength threshold, it obtains the MAC address of the Bluetooth device, compares this MAC address with the pre-stored vehicle owner information. If this MAC address matches the vehicle owner information successfully, it determines that the user passes the authentication.

[0082] In addition, the user can also control the two-wheeled electric vehicle in other ways. In a specific example, when the user approaches the two-wheeled electric vehicle with a smart phone, if the phone Bluetooth is in the off state, the user can use existing technical means such as a mechanical key to unlock the two-wheeled electric vehicle; if the phone Bluetooth is in the on state, the user identity is verified based on the Bluetooth MAC address. After the user passes the authentication, the user can choose to long-press the EPB (Electronic Parking Brake) button on the vehicle or rotate the knob lock on the vehicle to unlock the vehicle. Secondly, the user can also choose to make an unlocking gesture to the vehicle to unlock the vehicle.

[0083] Based on a general inventive concept, the present invention also provides a gesture recognition device for two-wheeled electric vehicle interaction control. Figure 4 It is a schematic structural diagram of a gesture recognition device for two-wheeled electric vehicle interaction control provided by an embodiment of the present invention. As Figure 4 shown, this device includes: a recognition device 41, a control device 42, a first communication device 43, and a camera 44.

[0084] The recognition device 41 is communicatively connected to the control device 42 through the first communication device 43, and the recognition device 41 is also connected to the camera 44.

[0085] The recognition device 41, the control device 42, the first communication device 43, and the camera 44 are used to execute the above-mentioned gesture recognition method for two-wheeled electric vehicle interaction control.

[0086] Specifically, the hardware of the recognition device uses a single-chip microcomputer, that is, an MCU (Microcontroller Unit). Specifically, an MCU with the model of STM32F4ZGT6 can be used. The hardware of the control device is the host of an existing two-wheeled electric vehicle. The first communication device 43 can be an RS-485 communication device or a one-line communication device.

[0087] In the physical layer of the present invention, a standard RS-485 serial electrical interface is adopted. An integrated instrument with an ECU (Electronic Control Unit) or a TBOX (Telematics Box) inside is the master device in the 485 bus, and other devices are slave devices.

[0088] In addition, the gesture recognition device for two-wheeled electric vehicle interaction control of the present invention further includes: a second communication device.

[0089] The second communication device is a wireless communication device. For example, the second communication device can be a 4G communication device, which is used for communication connection with the control device 42, so that the control device 42 can communicate with the user's mobile phone and the cloud server through the second communication device. For example, when the control device determines an abnormal signal, it can send the abnormal signal to the user's mobile phone and / or the cloud server through the second communication device.

[0090] Based on a general inventive concept, the present invention also provides a two-wheeled electric vehicle. Compared with the two-wheeled electric vehicles in the prior art, the two-wheeled electric vehicle of the present invention further includes the gesture recognition device for two-wheeled electric vehicle interaction control as described above.

[0091] It can be understood that the same or similar parts in the above embodiments can be referred to each other, and the content not detailed in some embodiments can be seen in the same or similar content of other embodiments.

[0092] It should be noted that in the description of the present invention, terms such as "first" and "second" are only used for descriptive purposes and cannot be understood as indicating or implying relative importance. In addition, in the description of the present invention, unless otherwise specified, the meaning of "a plurality of" refers to at least two.

[0093] Any process or method description shown in the flowchart or described in other ways herein can be understood as representing a module, segment, or part of code including one or more executable instructions for implementing a specific logical function or process. The scope of the preferred embodiments of the present invention includes additional implementations, where the functions can be executed in a manner that is not shown or discussed, including in a substantially simultaneous manner according to the involved functions or in a reverse order, which should be understood by those skilled in the technical field to which the embodiments of the present invention belong.

[0094] It should be understood that each part of the present invention can be implemented by hardware, software, firmware, or a combination thereof. In the above embodiments, multiple steps or methods can be implemented by software or firmware stored in a memory and executed by a suitable instruction execution system. For example, if implemented by hardware, as in another embodiment, any one or a combination of the following techniques well known in the art can be used: discrete logic circuits having logic gate circuits for implementing logical functions on data signals, application specific integrated circuits having appropriate combinational logic gate circuits, programmable gate arrays (PGAs), field programmable gate arrays (FPGAs), and the like.

[0095] Those of ordinary skill in the art can understand that all or part of the steps carried by the method of the above embodiments can be completed by instructing relevant hardware through a program, and the program can be stored in a computer-readable storage medium. When the program is executed, it includes one or a combination of the steps of the method embodiments.

[0096] In addition, in each embodiment of the present invention, each functional unit can be integrated into a processing module, or each unit can exist physically alone, or two or more units can be integrated into one module. The above integrated module can be implemented in the form of hardware or in the form of a software functional module. When the above integrated module is implemented in the form of a software functional module and sold or used as an independent product, it can also be stored in a computer-readable storage medium.

[0097] The above-mentioned storage medium can be a read-only memory, a magnetic disk, an optical disk, or the like.

[0098] In the description of this specification, the description with reference to the terms "one embodiment", "some embodiments", "example", "specific example", or "some examples", etc. means that the specific features, structures, materials, or characteristics described in connection with the embodiment or example are included in at least one embodiment or example of the present invention. In this specification, the schematic representations of the above terms do not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials, or characteristics described can be combined in any one or more embodiments or examples in a suitable manner.

[0099] Although the embodiments of the present invention have been shown and described above, it can be understood that the above embodiments are exemplary and should not be construed as limiting the present invention. Those of ordinary skill in the art can make changes, modifications, substitutions, and variations to the above embodiments within the scope of the present invention.

Claims

1. A gesture recognition method for interactive control of a two-wheeled electric vehicle, characterized in that: include: After determining that the user identity authentication is passed, the recognition device controls the camera to capture a real-time video stream if it recognizes a change in the lens of the camera of the two-wheeled electric vehicle; The recognition device reads the target image frame in the video stream; The recognition device uses the MediaPipe model to perform hand key node detection on the target image frame to obtain a detection result; When the recognition device determines that the detection result shows that the coordinates of the key nodes of the hand are successfully acquired, the recognition device performs gesture recognition according to the coordinates of the key nodes of the hand and the threshold range of each preset gesture to obtain a gesture recognition result; the threshold range is composed of a sub-threshold range of at least one vector angle, and the vector is a vector composed of the coordinates of two of the key nodes of the hand; The recognition device determines a target operation corresponding to the target gesture when determining that the gesture recognition result shows that the target gesture is successfully recognized; The target gesture is one of the preset gestures; The identification device generates a reporting instruction according to the target operation, and sends the reporting instruction to the control device; When the control device determines to execute the target operation according to the reporting instruction, the control device controls the working state of the two-wheeled electric vehicle according to the target operation.

2. The gesture recognition method for interactive control of a two-wheeled electric vehicle according to claim 1, characterized in that: According to the coordinates of the key hand nodes and the threshold range of each preset gesture, gesture recognition is performed to obtain a gesture recognition result, which specifically includes: The following operations are performed for each of the preset gestures in sequence until the target gesture is recognized or all the preset gestures are polled: Calculating the target vector angle corresponding to each sub-threshold range in the threshold range of the current preset gesture according to the coordinates of the key nodes of the hand; When all the target vector angles are within the corresponding sub-threshold range, determining that the current preset gesture is the target gesture; When the target vector angle among all the target vector angles does not fall within the corresponding sub-threshold range, it is determined that the current preset gesture is not the target gesture.

3. The gesture recognition method for interactive control of a two-wheeled electric vehicle according to claim 1, characterized in that: After performing gesture recognition according to the coordinates of the key hand nodes and the threshold ranges of the preset gestures to obtain the gesture recognition results, the method further includes: When the recognition device determines that the gesture recognition result shows that an unknown gesture is recognized, the recognition device determines whether a gesture setting instruction sent by the control device is received; the control device generates the gesture setting instruction according to the gesture setting operation of the user; the gesture setting instruction includes at least one gesture and its corresponding operation; If the gesture setting instruction sent by the control device is received, the recognition device performs corresponding settings according to the gesture setting instruction; The recognition device feeds back the gesture setting result to the control device.

4. The gesture recognition method for interactive control of a two-wheeled electric vehicle according to claim 3, characterized in that: After determining whether the gesture setting instruction sent by the control device is received, the method further includes: If the gesture setting instruction sent by the control device is not received, the recognition device determines that there is no need to set an operation for the unknown gesture; The recognition device re-executes the step of reading the target image frame in the video stream.

5. The gesture recognition method for interactive control of a two-wheeled electric vehicle according to claim 1, characterized in that: If a change in the lens of the camera of the two-wheeled electric vehicle is identified, before controlling the camera to capture a real-time video stream, the method further includes: The control device acquires the MAC address of the Bluetooth device when detecting that the strength of the Bluetooth signal sent by the Bluetooth device is greater than a preset signal strength threshold; The control device compares the MAC address with pre-stored vehicle owner information; If the MAC address successfully matches the vehicle owner information, the control device determines that the corresponding user identity authentication is passed.

6. The gesture recognition method for interactive control of a two-wheeled electric vehicle according to claim 1, characterized in that: The identification device generates a reporting instruction according to the target operation, and sends the reporting instruction to the control device, specifically including: The identification device obtains the current working state of the two-wheeled electric vehicle through the control device; The identification device determines whether there is a logical conflict between the current working state and the target operation; If there is a logical conflict between the current working state and the target operation, the identification device generates a first reporting instruction carrying error prompt information, and sends the first reporting instruction to the control device; If there is no logical conflict between the current working state and the target operation, the identification device generates a second reporting instruction carrying the target operation, and sends the second reporting instruction to the control device.

7. The gesture recognition method for interactive control of a two-wheeled electric vehicle according to claim 1, characterized in that: The recognition device uses the MediaPipe model to detect the key nodes of the hand on the target image frame, and after obtaining the detection result, it also includes: When the recognition device determines that the detection result shows that the coordinates of the key nodes of the hand are not successfully acquired, the recognition device re-executes the step of reading the target image frame in the video stream.

8. A gesture recognition device for interactive control of a two-wheeled electric vehicle, characterized in that: include: An identification device, a control device, a first communication device and a camera; The identification device is connected to the control device through the first communication device, and the identification device is also connected to the camera; The recognition device, the control device, the first communication device and the camera are used to execute the gesture recognition method for interactive control of a two-wheeled electric vehicle as described in any one of claims 1 to 7.

9. The gesture recognition device for interactive control of a two-wheeled electric vehicle according to claim 8, characterized in that: Also included: a second communication device connected to the control device; The second communication device is a wireless communication device, which is used to support the control device to communicate wirelessly with external equipment.

10. A two-wheeled electric vehicle, characterized in that: include: A gesture recognition device for interactive control of a two-wheeled electric vehicle as described in claim 8 or 9.

Images