Human-computer interaction method and system integrating AI voice and intelligent feedback

Abstract

The present invention discloses a human-computer interaction method and system integrating AI voice and intelligent feedback, including the following steps: obtaining the voice information of the operator and performing identity recognition through an AI voiceprint recognition model; obtaining the corresponding interaction rate parameter of the operator after identity recognition; obtaining photos containing the left and right finger slides on the mouse in real time through the middle camera and the left and right reflectors on the mouse; dividing the photos into left and right sides, and separately using a visual detection model for each divided photo on each side to detect the fingertip contours of the left and right fingers. The present invention uses an AI voiceprint recognition model to identify the user's voiceprint and determine the interaction rate parameter, realizing intelligent feedback for different users, and then uses the middle camera to detect the gestures of the two fingers respectively, and correspondingly can control different areas of the operating system. At this time, the hand does not need to move away from the mouse, and the instruction control can be realized by moving the left and right fingers back and forth, improving the user experience.

Description

Technical Field

[0001] The present invention relates to the technical field of human-computer interaction software, and particularly to a human-computer interaction method and system integrating AI voice and intelligent feedback. Background Art

[0002] As a traditional human-computer interaction device, a mouse can achieve basic operations such as pointer movement and left and right clicks. On the basis of these basic operations, currently, shortcut keys can also be set on the mouse. By installing mouse management software on the operating system, when the shortcut key is pressed, it can simulate the pressing of a keyboard combination key, thereby achieving more complex shortcut key functions. Currently, gesture recognition and interaction can also be achieved based on the camera of the operating system to collect gestures. However, this depends on the camera of the operating system, and at the same time, the hand needs to switch back and forth between the mouse and the camera during gesture operations, affecting the user experience. And it cannot achieve adaptation for different users and cannot achieve intelligent feedback. Summary of the Invention

[0003] Therefore, there is a need to provide a human-computer interaction method and system integrating AI voice and intelligent feedback to solve the problems that the hand needs to switch back and forth between the mouse and the camera during existing gesture operations and that intelligent feedback cannot be achieved for different users.

[0004] To achieve the above object, the present invention provides a human-computer interaction method integrating AI voice and intelligent feedback, including the following steps:

[0005] Obtain the voice information of the operator and perform identity recognition through an AI voiceprint recognition model;

[0006] After identity recognition, obtain the interaction rate parameter corresponding to the operator;

[0007] Real-time obtain photos containing left and right finger slides on the mouse through the middle camera and left and right reflectors on the mouse;

[0008] Divide the photo into left and right sides, and separately use a visual detection model to detect the fingertip contours of the left and right fingers for each divided photo on each side;

[0009] After obtaining that the bottom positions of the fingertip contours are less than a preset distance from the mouse surface, calculate the scrolling distances and directions of the left and right sides in real time according to the interaction rate parameter, finger sliding distance and direction, and separately send control commands containing the scrolling distances and directions of the left and right sides to the first area and the second area of the operating system interface.

[0010] Further, it further includes the step of:

[0011] After obtaining that the bottom positions of the left and right fingertip contours simultaneously contact the mouse surface;

[0012] After the bottom positions of the left and right fingertip contours and the mouse surface are both greater than a preset distance within a preset first interval time;

[0013] After the bottom positions of the left and right fingertip contours and the mouse surface come into contact again within a preset second interval time;

[0014] After the bottom positions of the left and right fingertip contours and the mouse surface are both greater than a preset distance again within a preset third interval time;

[0015] Trigger a preset two - finger control instruction to the operating system.

[0016] Furthermore, it also includes an interaction rate parameter initialization step:

[0017] Obtain an initialization instruction, obtain the voice information of the operator and the input interaction rate parameter, extract the voiceprint feature information through an AI voiceprint recognition model and save the corresponding relationship with the input interaction rate parameter.

[0018] Furthermore, after obtaining that the bottom position of the fingertip contour and the mouse surface are greater than a preset distance, obtain the change situation of the bending angles of the left and right fingers, and send corresponding control instructions to the operating system according to the change situation of the finger bending angles.

[0019] Furthermore, it also includes steps:

[0020] After obtaining the voice recognition instruction triggered by the mouse button, the mouse obtains the voice information of the operator and performs voice recognition through an AI voice recognition model. After the voice recognition is a preset control instruction, send the control instruction to the operating system.

[0021] Furthermore, the first region is the left region of the mouse pointer, and the second region is the right region of the mouse pointer.

[0022] Furthermore, the first region is the left region in the middle of the screen, and the second region is the right region in the middle of the screen.

[0023] Furthermore, the first region is the first screen region, the second region is the second screen region, and the first screen and the second screen are different screens.

[0024] Furthermore, the AI voiceprint recognition model is a local voiceprint recognition model or an online API voiceprint recognition model.

[0025] The present invention provides a human - computer interaction system integrating AI voice and intelligent feedback, including a memory and a processor. A computer program is stored on the memory, and when the computer program is executed by the processor, it implements the steps of the method described in any one of the present invention.

[0026] Different from the prior art, the above technical solution uses an AI voiceprint recognition model to identify the user's voiceprint and determine the interaction rate parameter, realizing intelligent feedback for different users. Then, through the middle camera, the side images of the left and right fingers placed on the human-computer interaction device (mouse) can be collected respectively. After determining the fingertip contour according to the visual detection model, the distance from the mouse surface is judged, so as to realize the gesture detection of the two fingers respectively, and correspondingly control different areas of the operating system. At this time, the hand does not need to move away from the mouse, and the instruction control can be realized by moving the left and right fingers back and forth, improving the user experience. BRIEF DESCRIPTION OF THE DRAWINGS

[0027] Figure 1 is a flowchart of the method according to an embodiment of the present invention;

[0028] Figure 2 is a physical diagram of the human-computer interaction device of the present invention;

[0029] Figure 3 is a sectional view of the human-computer interaction device of the present invention;

[0030] Figure 4 is a flowchart of the method according to another embodiment of the present invention. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0031] In order to describe the technical content, structural features, achieved objectives and effects of the technical solution in detail, the following will be described in detail with specific embodiments in conjunction with the accompanying drawings.

[0032] Please refer to Figures 1 to 4 , the present invention provides a human-computer interaction method integrating AI voice and intelligent feedback, which can be applied to a human-computer interaction device (such as a mouse) such as Figures 2 - 3 . The human-computer interaction device includes a device body 1. On the surface of the device body 1 protruding in the middle position at the upper part, there is an image acquisition module 2. The image acquisition module 2 includes left and right reflectors 20 at the top and a bottom camera 21. The bottom camera 21 is located at the middle position directly below the left and right reflectors. In order to collect the images of the left and right fingers into the bottom camera 21, a convex lens can also be included in front of the reflector. The bottom of the device body 1 can also include a photoelectric sensor 3 for collecting displacement at the bottom and a main control chip 4. The main control chip is used to collect the information of the camera and the sensor and convert it into corresponding control instructions, and then send them to the operating system by wire (USB cable) or wireless (Bluetooth or 2.4G wireless) means.

[0033] Please refer to Figure 1As shown, the method of the present invention includes the following steps: Step S101 obtains the voice information of the operator and performs identity recognition through an AI voiceprint recognition model. Here, voice collection can be performed through a camera connected to the operating system or a microphone can be set on the human-computer interaction device. After the AI voiceprint recognition model obtains the voice, it performs recognition, and the recognized voiceprint feature information is compared with the pre-stored voiceprint feature information to determine whether there is the same voiceprint information. It should be noted that the AI voiceprint recognition model is a local voiceprint recognition model or an online API voiceprint recognition model. The local voiceprint model can be deployed on the operating system or on the main control chip 4 of the human-computer interaction device. The local voiceprint model can be a single-state hidden Markov model (HMM) or a Gaussian mixture Markov model (GMM). The online API voiceprint recognition model such as Yunzhisheng or the Alibaba Cloud voiceprint recognition API interface can send the voice to this API interface to achieve voiceprint recognition. The voice information here can be the voice information for the operating system to guide the user to say a preset piece of text, so as to achieve fast recognition.

[0034] After voiceprint recognition, it can enter step S102. After identity recognition, the corresponding interaction rate parameter of the operator is obtained. Each operator can store its corresponding interaction rate parameter. Step S103 obtains photos containing the left and right finger slides on the mouse in real time through the middle camera and the left and right reflectors on the mouse. As Figure 3As shown, the left finger (index finger) operating the left mouse button is reflected through the left mirror into the left half of the lens of the lower camera, and the right finger (middle finger) operating the right mouse button is reflected through the right mirror into the right half of the lens of the lower camera, achieving bilateral acquisition with a single lens. Here, the images collected are of the sides of the fingers, that is, the images between the fingers. Then, it enters step S104 to divide the photo into left and right sides, and the visual detection model is used to detect the fingertip contours of the left and right fingers separately for each side of the divided photo. Here, preferably, one side of the image needs to be mirror-flipped so that the fingers on both sides face the same direction, facilitating the rapid processing of the visual detection model. The visual detection model can use the YOLOv8 model. The trained YOLOv8 model can be deployed in the main control chip, and the main control chip can use the RK3588 chip of Rockchip. After deployment, the collected pictures are segmented and mirrored and then input into the YOLOv8 model, and then the fingertip contours are obtained. The training process of the YOLOv8 model can be operated as follows: First, the side images of fingers at different positions, different curvatures, and different users are collected through the camera. Then, after opening each image with labelImg, manual annotation is performed, and the contour of the last joint of the finger is marked with a square and saved as an annotation file corresponding to the image. Then, most (such as 90%) of the images are placed in the training folder of the YOLOv8 model, and a small part of the images are placed in the validation folder of the YOLOv8 model. The saved annotation files are placed in the marked training folder and validation folder corresponding to the images. Then, the YOLOv8 model training starts. After training, the trained model file can be saved. The model file and the YOLOv8 model are placed in the memory of the main control chip together, and then by calling this model, the position of the fingertip contour can be achieved.

[0035] After obtaining the bottom position of the fingertip contour and the mouse surface with a distance less than a preset distance (such as less than 3 pixel points) in step S105, it indicates that the finger contacts the mouse surface, and the instruction triggering process starts. The left and right scrolling distances and directions are calculated in real time according to the interaction rate parameter, the finger sliding distance and direction, and control instructions containing the left and right scrolling distances and directions are respectively sent to the first area and the second area of the operating system interface. For example, if the left finger slides down 5 pixel points and the interaction rate parameter is 10, the result of multiplication is 50 pixel points, and the first area of the operating system can be scrolled down 50 pixel points. If the right finger slides up 7 pixel points and the interaction rate parameter is 10, the result of multiplication is 70 pixel points, and the second area of the operating system can be scrolled up 70 pixel points. Thus, operations on different areas are achieved. For different operators, if the interaction rate parameter is 12 and the left finger slides down 5 pixel points, the result of multiplication is 60 pixel points, and the first area of the operating system can be scrolled down 60 pixel points. Intelligent feedback for different users is achieved.

[0036] The above solution uses an AI voiceprint recognition model to recognize the user's voiceprint and determine the interaction rate parameter, achieving intelligent feedback for different users. Then, the side images of the left and right fingers placed on the human-computer interaction device (mouse) can be respectively collected through the middle camera. After determining the fingertip contour according to the visual detection model, the distance from the mouse surface is judged, so as to achieve the gesture detection of the two fingers respectively, and different areas of the operating system can be controlled correspondingly. At this time, the hand does not need to move away from the mouse, and the instruction control can be achieved by moving the left and right fingers back and forth, improving the user experience.

[0037] In some embodiments, gesture operations can also be achieved by clicking the left and right fingers, such as Figure 4As shown, it further includes steps: Step S401: After the bottom positions of the left and right fingertip contours come into contact with the mouse surface simultaneously. Here, "simultaneously" does not mean exactly the same moment, but rather within a certain interval, such as less than 100 milliseconds, which can be considered simultaneous. Then, in step S402, within a preset first interval time (such as 200 milliseconds), when the bottom positions of the left and right fingertip contours are both greater than a preset distance from the mouse surface simultaneously; that is, the fingertips leave the mouse surface, but the hand does not leave the mouse. Then, in step S403, within a preset second interval time (such as 200 milliseconds), when the bottom positions of the left and right fingertip contours come into contact with the mouse surface again simultaneously; and in step S404, within a preset third interval time (such as 200 milliseconds), when the bottom positions of the left and right fingertip contours are both greater than the preset distance from the mouse surface again simultaneously; enter step S405 to trigger a preset two-finger control instruction to the operating system. The two-finger control instruction can be preset as a shortcut key, a combination key, or other control instructions, such as page turning, returning to the desktop, etc. In some embodiments, this two-finger control instruction can be used as Figure 1 the trigger instruction for step 105 of the embodiment. After such an operation with the two fingers, the control instruction judgment and sending operation for the scrolling in step 105 are performed. Otherwise, the judgment of the control instruction for scrolling is not performed to avoid misoperations. Then, after repeating steps S401 to S405, the trigger instruction can be cancelled, and the judgment of the control instruction for scrolling in step S105 is not performed. The Figure 1 trigger instruction for step 105 of the embodiment can also be implemented by a button set on the human-computer interaction device. Press the button to trigger, and press the button again to cancel the trigger.

[0038] In order to have corresponding interaction rate parameters after identity recognition, the present invention further includes an interaction rate parameter initialization step: obtaining an initialization instruction, such as when the user clicks the initialization button on the software, obtaining the voice information of the operator and the input interaction rate parameter, extracting the voiceprint feature information through an AI voiceprint recognition model, and saving the corresponding relationship with the input interaction rate parameter. In this way, the voiceprint feature information and the corresponding interaction rate parameter can be saved. Of course, if no corresponding voiceprint feature information is detected, the default interaction rate parameter (such as 10) can also be used.

[0039] Furthermore, the present invention can also implement more gesture instructions. After obtaining that the bottom position of the fingertip contour is greater than a preset distance from the mouse surface, that is, after the two fingers are lifted, the change in the bending angles of the left and right fingers is obtained, and corresponding control instructions are sent to the operating system according to the change in the bending angles of the fingers. The bending of the fingertips can be detected by a visual detection model, such as the YOLOv8 model. First, the rectangle corresponding to the finger contour is obtained, and this rectangle will enclose the entire finger. Since the aspect ratio (height divided by width) of the rectangle will be close to a larger value when the finger is bent, and the rectangle becomes flattened when the finger is straight, and the aspect ratio becomes smaller. By judging the change in the aspect ratio of the rectangle of the entire finger obtained by detection, the detection of finger bending can be achieved. For example, when the finger changes from straight to bent, the next page operation can be triggered. The specific solution is to detect that within a preset time, the change amount of the aspect ratio of the rectangle from small to large is greater than a certain value, then the next page operation is triggered, and the next page operation is sent to the first area corresponding to the left finger, and the next page operation is sent to the second area corresponding to the right finger. If it is detected that within a preset time, the change amount of the aspect ratio of the rectangle from large to small is greater than a certain value, then the previous page operation is triggered, that is, when the finger changes from bent to straight, it is the previous page, and vice versa for the next page, thus realizing the page turning operation of the finger.

[0040] In order to further implement voice operations, it further includes the step: after obtaining the voice recognition instruction triggered by the mouse button, such as pressing the voice recognition button. Then the mouse obtains the voice information of the operator and performs voice recognition through an AI voice recognition model (different from the voiceprint model). After the voice recognition is a preset control instruction, the control instruction is sent to the operating system. Voice recognition is now relatively mature. The focus of the present invention is that triggering and voice recognition can be achieved through the buttons and microphones on the mouse, such as controlling the volume of the operating system to increase or decrease.

[0041] It should be noted that the present invention does not limit the positions of the first area and the second area, which can be set according to the user. In some embodiments, the first area is the left area of the mouse pointer, and the second area is the right area of the mouse pointer. In this way, after moving the mouse and the mouse pointer moves, the first area and the second area can be changed at any time, which is convenient for the user to operate.

[0042] In some embodiments, the first area is the left area in the middle of the screen, and the second area is the right area in the middle of the screen. In this way, the operating system can quickly place different documents or web pages in the left half-screen area and the right half-screen area of the screen. The left finger can control the documents in the left half-screen, and the right finger can control the documents in the right half-screen, which is convenient for operation.

[0043] In some embodiments, the first region is a first screen region, the second region is a second screen region, and the first screen and the second screen are different screens. This enables separate control of multiple connected screens, facilitating user control. It should be noted that the scrolling in the present invention can also be carousel scrolling of pictures. In this case, the first region can play multiple pictures, and the second region can also play multiple pictures. By moving the left and right fingers, the pictures can be quickly scrolled to the one the user wants to view.

[0044] The present invention provides a human - machine interaction system integrating AI voice and intelligent feedback, including a memory and a processor. A computer program is stored on the memory, and when the computer program is executed by the processor, the steps of the method described in any one of the present invention are implemented. The memory in this embodiment can be a memory set in an electronic device. The electronic device can read the content of the memory and achieve the effects of the present invention. The memory can also be a separate memory. When this memory is connected to the electronic device, the electronic device can read the content in the memory and implement the method steps of the present invention. The system of the present invention uses an AI voiceprint recognition model to identify the user's voiceprint and determine the interaction rate parameter, achieving intelligent feedback for different users. Then, through an intermediate camera, the side images of the left and right fingers placed on the human - machine interaction device (mouse) can be collected respectively. After determining the fingertip contour according to the visual detection model, the distance from the mouse surface is judged, thereby realizing the gesture detection of the two fingers respectively, and correspondingly controlling different regions of the operating system. At this time, the hand does not need to leave the mouse, and the instructions can be controlled by moving the left and right fingers back and forth, improving the user experience.

[0045] It should be noted that although the above embodiments have been described in this article, this does not limit the patent protection scope of the present invention. Therefore, based on the innovative concept of the present invention, changes and modifications made to the embodiments described in this article, or equivalent structural or equivalent process transformations using the content of the specification and drawings of the present invention, directly or indirectly applying the above technical solutions to other related technical fields are all included in the patent protection scope of the present invention.

Claims

1. A human-computer interaction method integrating AI voice and intelligent feedback, characterized in that: The steps include: Acquire the operator's voice information and perform identity recognition through the AI voiceprint recognition model; After identification, the interaction rate parameters corresponding to the operator are obtained; The middle camera and left and right mirrors on the mouse are used to obtain photos containing the left and right finger sliding on the mouse in real time; Segment the photo into left and right sides, and use the visual detection model to detect the fingertip contours of the left and right fingers in the segmented photo of each side. After obtaining the bottom position of the fingertip contour and the mouse surface less than a preset distance, the scrolling distance and direction on the left and right sides are calculated in real time according to the interaction rate parameter, the finger sliding distance and direction, and control instructions containing the scrolling distance and direction on the left and right sides are sent to the first area and the second area of the operating system interface respectively.

2. The human-computer interaction method integrating AI voice and intelligent feedback according to claim 1, characterized in that: Also includes the steps: After obtaining the bottom positions of the left and right fingertip contours and contacting the mouse surface at the same time; After the bottom positions of the left and right fingertip contours are simultaneously greater than a preset distance from the mouse surface within a preset first interval; After the bottom positions of the left and right fingertip contours touch the mouse surface again simultaneously within the preset second interval; After the bottom positions of the left and right fingertip contours are simultaneously greater than the preset distance from the mouse surface within the preset third interval; Trigger the preset two-finger control command to the operating system.

3. The human-computer interaction method integrating AI voice and intelligent feedback according to claim 1, characterized in that: It also includes the interaction rate parameter initialization step: Get the initialization instruction, obtain the operator's voice information and the recorded interaction rate parameters, extract the voiceprint feature information through the AI voiceprint recognition model and save the corresponding relationship with the recorded interaction rate parameters.

4. The human-computer interaction method integrating AI voice and intelligent feedback according to claim 1, characterized in that: After obtaining the bottom position of the fingertip contour and the mouse surface greater than the preset distance, the bending angle changes of the left and right fingers are obtained, and corresponding control instructions are sent to the operating system according to the bending angle changes of the fingers.

5. The human-computer interaction method integrating AI voice and intelligent feedback according to claim 1, characterized in that: Also includes the steps: After obtaining the voice recognition command triggered by the mouse button, the mouse obtains the operator's voice information and performs voice recognition through the AI voice recognition model. After the voice recognition is the preset control command, the control command is sent to the operating system.

6. The human-computer interaction method integrating AI voice and intelligent feedback according to claim 1, characterized in that: The first area is the left area of the mouse pointer, and the second area is the right area of the mouse pointer.

7. The human-computer interaction method integrating AI voice and intelligent feedback according to claim 1, characterized in that: The first area is the left area in the middle of the screen, and the second area is the right area in the middle of the screen.

8. The human-computer interaction method integrating AI voice and intelligent feedback according to claim 1, characterized in that: The first area is a first screen area, the second area is a second screen area, and the first screen and the second screen are different screens.

9. The human-computer interaction method integrating AI voice and intelligent feedback according to claim 1, characterized in that: The AI voiceprint recognition model is a local voiceprint recognition model or an online API voiceprint recognition model.

10. A human-computer interaction system integrating AI voice and intelligent feedback, featuring: The method comprises a memory and a processor, wherein a computer program is stored in the memory, and when the computer program is executed by the processor, the steps of the method according to any one of claims 1 to 9 are implemented.

Images