Mediape gesture recognition-based control method and system for Solidworks software

Abstract

The invention discloses a Mediape gesture recognition-based Solidworks software control system, and relates to the technical field of computer vision, and the Mediape gesture recognition-based Solidworks software control system comprises the following steps: defining a function starting gesture and a function control gesture, and collecting a gesture video of a user; detecting and recognizing the gesture video, if a function starting gesture is detected, reminding the user that the gesture control is started in a character manner, and carrying out the next step, and if the function starting gesture is not detected, reminding the user that the function starting gesture is not detected in the character manner; according to the Solidworks control method, a gesture is recognized, the recognized function control gesture is mapped into a Solidworks control signal, the Solidworks control signal is transmitted to Solidworks software, and Solidworks are controlled.

Description

Technical Field

[0001] The present invention relates to the technical field of computer vision, and in particular to a control method and system for Solidworks software based on Mediapipe gesture recognition. Background Art

[0002] The gesture recognition model integrated by Mediapipe can quickly and accurately identify the 21 key points of a person's hand and their coordinates, and each key point has a corresponding number.

[0003] Solidworks software is a powerful 3D mechanical design software system. With its excellent performance, ease of use and innovation, it has greatly improved the design efficiency of mechanical engineers. Solidworks software is powerful. In addition to being able to design 3D models simply and quickly, it can also directly generate 2D engineering drawings that meet design standards from 3D models. The 3D model and the engineering drawing are interrelated. When the 3D model is changed as needed, the engineering drawing will also change accordingly. In addition, Solidworks software also has excellent rendering functions. Even if the designed product is not actually processed, the rendering function can be used to preview the realistic effect. In addition, Solidworks software also has common functions such as motion simulation, force analysis, and CAM numerical control simulation. It is precisely because of its powerful functions that Solidworks has become the only choice for people engaged in design work in many industries including aerospace, industrial machinery, automobiles and transportation, consumer products, design and engineering services, electronics, mold design, petroleum, packaging machinery, education, etc. However, under some special working conditions, it is inconvenient for users to control Solidworks software using general external devices such as mouse and keyboard. Summary of the invention

[0004] In order to solve the deficiencies mentioned in the above background technology, the purpose of the present invention is to provide a control method and system for Solidworks software based on Mediapipe gesture recognition.

[0005] In a first aspect, the purpose of the present invention can be achieved by the following technical solution: a control method for Solidworks software based on Mediapipe gesture recognition, the method comprising the following steps:

[0006] Define function activation gestures and function control gestures, and collect user gesture videos;

[0007] Detect and identify the gesture video. If a function-enabling gesture is detected, the user is reminded in text that gesture control is enabled and the next step is performed. If no function-enabling gesture is detected, the user is reminded in text that no function-enabling gesture is detected.

[0008] The gestures are recognized, the recognized function control gestures are mapped into Solidworks control signals, and transmitted to the Solidworks software to control Solidworks.

[0009] In combination with the first aspect, in some implementations of the first aspect, the method further includes: the image acquisition module acquires the user's hand video frame by capturing the human hand through Python OpenCV.

[0010] In combination with the first aspect, in certain implementations of the first aspect, the method also includes: the function activation gesture is that the palm faces the user, the thumb is vertically upward, and the other four fingers are bent toward the palm; the function control gesture includes: rotating the view to the left, the palm faces the camera, and the five fingers are stretched and upward; rotating the view to the right, the palm faces the user, and the five fingers are stretched and upward.

[0011] In combination with the first aspect, in some implementations of the first aspect, the method further includes: the function activation gesture and the function control gesture are defined by the relative position and distance relationship between the key points, specifically:

[0012] Function activation gesture: must satisfy y4>y3, l8 <l6、l 12 <l 10 , l 16 <l 14 , l 20 <l 18 ; Where P3 (x3, y3) is the coordinate of key point 3; P4 (x4, y4) is the coordinate of key point 4;

[0013] l i , (i=6, 8, 10, 12, 14, 16, 18, 20) is the distance between key point i and key point 0, that is:

[0014]

[0015] Among them, P0(x0, y0) is the coordinate of key point 0;

[0016] P i (x i ,y i ) are the coordinates of key point i, (i=6, 8, 10, 12, 14, 16, 18, 20);

[0017] Function control gesture: Left rotation needs to satisfy x 17 >x3、l8>l6、l 12 >l 10 , l 16 >l 14 , l 20 >l 18 ; Where P3(x3, y3) is the coordinate of key point 3; P 17 (x 17 ,y 17 ) are the coordinates of key point 17;

[0018] l i (i=6, 8, 10, 12, 14, 16, 18, 20) is the distance between key point i and key point 0, that is:

[0019]

[0020] Among them, P0(x0, y0) is the coordinate of key point 0;

[0021] P i (x i ,y i ) are the coordinates of key point i, (i=6, 8, 10, 12, 14, 16, 18, 20).

[0022] Right rotation requires x 17 <x3、l8> l6、l 12 >l 10 , l 16 >l 14 , l 20 >1 18 ; Where P3(x3, y3) is the coordinate of key point 3; P 17 (x 17 ,y 17 ) are the coordinates of key point 17;

[0023] l i (i=6, 8, 10, 12, 14, 16, 18, 20) is the distance between key point i and key point 0, that is:

[0024]

[0025] Among them, P0(x0, y0) is the coordinate of key point 0;

[0026] P i (x i ,y i ) are the coordinates of key point i, (i=6, 8, 10, 12, 14, 16, 18, 20).

[0027] In combination with the first aspect, in certain implementations of the first aspect, the method further includes: the Solidworks software control module calls different Solidworks API commands according to different types of gestures, and to implement the Solidworks view rotation, it is necessary to call the RotateAboutAxis() function in the Solidworks API, and the rotation angle and rotation direction are changed by changing the parameter values ​​in the RotateAboutAxis() function.

[0028] In a second aspect, in order to achieve the above-mentioned purpose, the present invention discloses a control system for Solidworks software based on Mediapipe gesture recognition, comprising:

[0029] Image acquisition module: used to collect user gesture videos in real time;

[0030] Gesture recognition module: used to detect and recognize the collected gesture video to determine whether there is a function activation gesture;

[0031] Gesture processing module: used to identify and detect the gesture video, determine whether there is a function control gesture, and process it into a control signal of Solidworks.

[0032] Solidworks software control module: used to receive the control signal and realize the control of Solidworks software.

[0033] In another aspect of the present invention, in order to achieve the above-mentioned purpose, a terminal device is disclosed, including a memory, a processor, and a computer program stored in the memory and capable of running on the processor, wherein the memory stores a computer program capable of running on the processor, and when the processor loads and executes the computer program, the control method for Solidworks software based on Mediapipe gesture recognition as described above is adopted.

[0034] In another aspect of the present invention, in order to achieve the above-mentioned purpose, a computer-readable storage medium is disclosed, in which a computer program is stored, characterized in that when the computer program is loaded and executed by a processor, the control method for Solidworks software based on Mediapipe gesture recognition as described above is adopted.

[0035] Beneficial effects of the present invention:

[0036] The present invention can overcome the situation that it is inconvenient or impossible to use general external devices such as a mouse and a keyboard to operate Solidworks software under special working conditions, thereby improving the work efficiency of Solidworks software users. BRIEF DESCRIPTION OF THE DRAWINGS

[0037] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, for those skilled in the art, other drawings can be obtained based on these drawings without creative work.

[0038] Figure 1 is a flow chart of the method of the present invention;

[0039] Figure 2 It is a schematic diagram of the system framework of the present invention;

[0040] Figure 3 This is a numbered diagram of the 21 key points in the Mediapipe gesture recognition model;

[0041] Figure 4 This is the effect diagram of the function activation gesture;

[0042] Figure 5 It is a left-handed effect diagram;

[0043] Figure 6 It is a right-hand effect diagram. DETAILED DESCRIPTION

[0044] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0045] Embodiment 1:

[0046] The following is an introduction to the relevant terms involved in the embodiments of the present application:

[0047] OpenCV: OpenCV, short for Open Source Computer Vision, is a free computer vision library that allows manipulating images and videos to perform various tasks.

[0048] Computer Vision: Computer vision is about enabling computers and systems to extract meaningful information from images, videos, and other visual inputs, and to take actions or provide recommendations based on that information. If artificial intelligence gives computers the ability to think, then computer vision gives computers the ability to discover, observe, and understand.

[0049] Generally speaking, computer vision is a simulation of human vision using computers and related equipment. Computers use various imaging systems instead of visual organs as input, and the computer's processing unit replaces the brain to complete processing and interpretation. With the development of computer hardware systems, artificial intelligence and deep learning technologies, computer vision technology is playing an increasingly important role in many fields such as business, entertainment, and medicine.

[0050] Human-computer interaction: Human-computer interaction is a discipline that studies the interactive relationship between systems and users. Systems can be various machines or computerized systems and software. For computers, traditional human-computer interaction methods mainly rely on various hardware, such as mice, keyboards, etc. Gesture recognition, as an important branch of computer vision, provides a new human-computer interaction method, that is, the computer can collect hand images through image acquisition devices and match them with specific gestures for similarity. When the comparison is successful, the computer executes the specific command represented by the specific gesture, thereby realizing the human-computer interaction process.

[0051] Mediapipe: Mediapipe is an integrated machine learning visual algorithm tool library that includes various models such as face detection, facial key point recognition, gesture recognition, image segmentation, and posture recognition.

[0052] In this embodiment, the function activation gesture and the function control gesture are predefined, such as Figure 1 As shown, this embodiment is to control the Solidworks software view to rotate left, including the following steps:

[0053] Step 1: Define the function start gesture and function control gesture, such as Figure 4 , the function activation gesture is defined as the palm facing the user, the thumb pointing vertically upward, and the other four fingers bent toward the palm; Figure 5 ,The function control gesture is defined as : the palm faces the camera and the five fingers are extended and upward;

[0054] Preferably, in step 1, defining the function activation gesture and the function control gesture is implemented by the relative position and distance relationship between the key points, specifically:

[0055] Function activation gesture: Figure 4 As shown, it is necessary to satisfy: y4>y3, l8 <l6、l 12 <l 10 , l 16 <l 14 , l 20 <l 18 ; Where P3 (x3, y3) is the coordinate of key point 3; P4 (x4, y4) is the coordinate of key point 4;

[0056] l i , (i=6, 8, 10, 12, 14, 16, 18, 20) is the distance between key point i and key point 0, that is:

[0057]

[0058] Where P0(x0, y0) is the coordinate of key point 0;

[0059] P i (x i ,y i ) are the coordinates of key point i, (i=6, 8, 10, 12, 14, 16, 18, 20);

[0060] Function control gestures: Figure 5 As shown, left rotation needs to satisfy x 17 >x3、l8>l6、l 12 >l 10 , l 16 >l 14 , l 20 >l 18 ; Where P3(x3, y3) is the coordinate of key point 3; P 17 (x 17 ,y 17 ) are the coordinates of key point 17;

[0061] l i (i=6, 8, 10, 12, 14, 16, 18, 20) is the distance between key point i and key point 0, that is:

[0062]

[0063] Where P0(x0, y0) is the coordinate of key point 0;

[0064] P i (x i ,y i ) are the coordinates of key point i, (i=6, 8, 10, 12, 14, 16, 18, 20).

[0065] Step 2: The image acquisition module acquires the hand video frames of the current user and transmits the video frames to the gesture recognition module and the gesture processing module.

[0066] OpenCV is the abbreviation of Open Source Computer Vision, which is a free computer vision library that allows the manipulation of images and videos to complete various tasks. The image acquisition module in the present invention uses Python OpenCV to acquire the hand video frame of the current user.

[0067] Step 3: The gesture recognition module recognizes the received video frames. If a function activation gesture is recognized, the user is prompted with text that "gesture control is enabled" and the next step is to start the recognition of function control gestures. If no function activation gesture is recognized, the user is prompted with text that "no function activation gesture is detected".

[0068] In the present invention, the gesture recognition module uses the Mediapipe model. Mediapipe is a machine learning framework developed by Google, which is mainly used to process image and video data. It has a series of built-in computer vision models that can realize functions such as face detection, gesture recognition, and posture estimation. The present invention uses the gesture recognition model in Mediapipe to obtain the key points of the user's hands in the video and their coordinates, and uses the position and distance relationship between these coordinates to realize the definition and recognition of the gesture.

[0069] Step 4: The gesture processing module receives the video frame transmitted by the image acquisition module, and recognizes the function control gesture. If the recognition is successful, the function control gesture is mapped to a Solidwork control signal, and the control signal is sent to the Solidworks software control module.

[0070] Specifically, in this embodiment, the Solidworks control signal mapped by the function control gesture is:

[0071] swModelView.RotateAboutAxis(-2 / 180*np.pi, 0, 0, 0, 0, 1, 0)

[0072] Step 5: The Solidworks software control module receives the control signal in step 4 and controls the Solidworks model view to rotate left.

[0073] Embodiment 2:

[0074] In this embodiment, the function activation gesture and the function control gesture are predefined, based on the control system for the Solidworks software of Mediapipe gesture recognition, such as Figure 2 As shown, this embodiment is to control the Solidworks software view to rotate right, including:

[0075] Step 1: Define the function start gesture and function control gesture, such as Figure 4 , the function activation gesture is defined as the palm facing the user, the thumb pointing vertically upward, and the other four fingers bent toward the palm; Figure 6 ,The function control gesture is defined as : the palm faces the user and the five fingers are extended and upward;

[0076] Preferably, in step 1, defining the function activation gesture and the function control gesture is implemented by the relative position and distance relationship between the key points, specifically:

[0077] Function activation gesture: Figure 4 As shown, it is necessary to satisfy y4>y3, l8<l6, l 12 <l 10 , l 16 <l 14 , l 20 <l 18 ; Where P3 (x3, y3) is the coordinate of key point 3; P4 (x4, y4) is the coordinate of key point 4;

[0078] l i , (i=6, 8, 10, 12, 14, 16, 18, 20) is the distance between key point i and key point 0, that is:

[0079]

[0080] Where P0(x0, y0) is the coordinate of key point 0;

[0081] P i (x i ,y i ) are the coordinates of key point i, (i=6, 8, 10, 12, 14, 16, 18, 20);

[0082] Function control gestures: Figure 6 As shown, right rotation must satisfy x 17 <x3、l8> l6、l 12 >l 10 , l 16 >l 14 , l 20 >l 18 ; Where P3(x3, y3) is the coordinate of key point 3; P 17 (x 17 ,y 17 ) are the coordinates of key point 17;

[0083] l i (i=6, 8, 10, 12, 14, 16, 18, 20) is the distance between key point i and key point 0, that is:

[0084]

[0085] Where P0(x0, y0) is the coordinate of key point 0;

[0086] P i (xi， y i ) are the coordinates of key point i, (i=6, 8, 10, 12, 14, 16, 18, 20).

[0087] Step 2: The image acquisition module acquires the hand video frames of the current user and transmits the video frames to the gesture recognition module and the gesture processing module.

[0088] OpenCV is the abbreviation of Open Source Computer Vision, which is a free computer vision library that allows the manipulation of images and videos to complete various tasks. The image acquisition module in the present invention uses Python OpenCV to acquire the hand video frame of the current user.

[0089] Step 3: The gesture recognition module recognizes the received video frames. If a function activation gesture is recognized, the user is prompted with text that "gesture control is enabled" and the next step is to start the recognition of function control gestures. If no function activation gesture is recognized, the user is prompted with text that "no function activation gesture is detected".

[0090] In the present invention, the gesture recognition module uses the Mediapipe model. Mediapipe is a machine learning framework developed by Google, which is mainly used to process image and video data. It has a series of built-in computer vision models, which can realize functions such as face detection, gesture recognition, and posture estimation. The present invention uses the gesture recognition model in Mediapipe to obtain the key points of the user's hand in the video and their coordinates, and uses the position and distance relationship between these coordinates to realize the definition and recognition of the gesture.

[0091] Step 4: The gesture processing module receives the video frame transmitted by the image acquisition module, and recognizes the function control gesture. If the recognition is successful, the function control gesture is mapped to a Solidwork control signal, and the control signal is sent to the Solidworks software control module.

[0092] Specifically, in this embodiment, the Solidworks control signal mapped by the function control gesture is:

[0093] swModelView.RotateAboutAxis(2 / 180*np.pi,0,0,0,0,1,0)

[0094] Step 5: The Solidworks software control module receives the control signal in step 4 and controls the Solidworks model view to rotate right.

[0095] Based on the same inventive concept, the present invention also provides a computer device, which includes: one or more processors, and a memory for storing one or more computer programs; the program includes program instructions, and the processor is used to execute the program instructions stored in the memory. The processor may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components, etc. It is the computing core and control core of the terminal, which is used to implement one or more instructions, specifically for loading and executing one or more instructions in a computer storage medium to implement the above method.

[0096] It needs to be further explained that, based on the same inventive concept, the present invention also provides a computer storage medium, on which a computer program is stored, and the computer program is executed by a processor to execute the above method. The storage medium can adopt any combination of one or more computer-readable media. The computer-readable medium can be a computer-readable signal medium or a computer-readable storage medium. The computer-readable storage medium can be, for example, but not limited to, an electrical, magnetic, optical, electrical, magnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples (non-exhaustive list) of computer-readable storage media include: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the present invention, a computer-readable storage medium can be any tangible medium containing or storing a program, which can be used by an instruction execution system, device or device or used in combination with it.

[0097] In the description of this specification, the description with reference to the terms "one embodiment", "example", "specific example", etc. means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present disclosure. In this specification, the schematic representation of the above terms does not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described can be combined in any one or more embodiments or examples in a suitable manner.

[0098] The above shows and describes the basic principles, main features and advantages of the present disclosure. Those skilled in the art should understand that the present disclosure is not limited by the above embodiments, and the above embodiments and descriptions are only for explaining the principles of the present disclosure. Without departing from the spirit and scope of the present disclosure, the present disclosure may have various changes and improvements, and these changes and improvements fall within the scope of the present disclosure to be protected.

Claims

1. A control method for Solidworks software based on Mediapipe gesture recognition, characterized in that: The method comprises the following steps: Define function activation gestures and function control gestures, and collect user gesture videos; Detect and identify the gesture video. If a function-enabling gesture is detected, the user is reminded in text that gesture control is enabled and the next step is performed. If no function-enabling gesture is detected, the user is reminded in text that no function-enabling gesture is detected. The gestures are recognized, the recognized function control gestures are mapped into Solidworks control signals, and transmitted to the Solidworks software to control Solidworks.

2. The control method for Solidworks software based on Mediapipe gesture recognition according to claim 1, characterized in that: The user's hand video is collected using the Python OpenCV module.

3. The control method for Solidworks software based on Mediapipe gesture recognition according to claim 1, characterized in that: The gesture detection and recognition is performed using the Mediapipe gesture recognition model.

4. The control method for Solidworks software based on Mediapipe gesture recognition according to claim 1, characterized in that: The function activation gesture is to have the palm of your hand facing the user, with the thumb pointing vertically upward and the other four fingers bent toward the palm; The function control gestures include: rotating the view to the left so that the palm faces the camera, with five fingers extended and pointing upward; The view is rotated right so that the palm faces the user, with the fingers extended and pointing upward.

5. The control method for Solidworks software based on Mediapipe gesture recognition according to claim 4, characterized in that: The function activation gesture and function control gesture are defined through the relative position and distance relationship between the key points, specifically: Function activation gesture: must satisfy y4>y3, l8 <l6、l 12 <l 10 , l 16 <l 14 , l 20 <l 18 ; Where P3 (x3, y3) is the coordinate of key point 3; P4 (x4, y4) is the coordinate of key point 4; l i , (i=6, 8, 10, 12, 14, 16, 18, 20) is the distance between key point i and key point 0, that is: Among them, P0(x0, y0) is the coordinate of key point 0; P i (x i ,y i ) are the coordinates of key point i, (i=6, 8, 10, 12, 14, 16, 18, 20); Function control gesture: Left rotation needs to satisfy x 17 >x3、l8>l6、l 12 >l 10 , l 16 >l 14 , l 20 >l 18 ; Where P3(x3, y3) is the coordinate of key point 3; P 17 (x 17 ,y 17 ) are the coordinates of key point 17; l i (i=6, 8, 10, 12, 14, 16, 18, 20) is the distance between key point i and key point 0, that is: Among them, P0(x0, y0) is the coordinate of key point 0; P i (x i ,y i ) are the coordinates of key point i, (i=6, 8, 10, 12, 14, 16, 18, 20); Right rotation requires x 17 <x3、l8> l6、l 12 >l 10 , l 16 >l 14 , l 20 >l 18 ; Where P3(x3, y3) is the coordinate of key point 3; P 17 (x 17 ,y 17 ) are the coordinates of key point 17; l i (i=6, 8, 10, 12, 14, 16, 18, 20) is the distance between key point i and key point 0, that is: Where P0(x0, y0) is the coordinate of key point 0; P i (x i ,y i ) are the coordinates of key point i, (i=6, 8, 10, 12, 14, 16, 18, 20).

6. The control method for Solidworks software based on Mediapipe gesture recognition according to claim 1, characterized in that: The control of Solidworks is achieved by calling the API function of Solidworks, and the control signal is realized by the API function swModelView.RotateAboutAxis(). The rotation angle and rotation direction are changed by changing the parameter value in the RotateAboutAxis() function.

7. A control system for Solidworks software based on Mediapipe gesture recognition, characterized in that: include: Image acquisition module: used to collect user gesture videos in real time; Gesture recognition module: used to detect and recognize the collected gesture video to determine whether there is a function activation gesture; Gesture processing module: used to identify and detect the gesture video, determine whether there is a function control gesture, and process it into a control signal of Solidworks; Solidworks software control module: used to receive the control signal and realize the control of Solidworks software.

8. A terminal device, comprising a memory, a processor, and a computer program stored in the memory and capable of running on the processor, characterized in that: The memory stores a computer program that can be run on the processor. When the processor loads and executes the computer program, the control method for Solidworks software based on Mediapipe gesture recognition described in any one of claims 1 to 6 is adopted.

9. A computer-readable storage medium having a computer program stored therein, characterized in that: When the computer program is loaded and executed by the processor, the control method for Solidworks software based on Mediapipe gesture recognition described in any one of claims 1 to 6 is adopted.

Images