Camera integrated with gesture control function

Abstract

The invention discloses a camera integrated with a gesture control function, and the camera comprises an image sensor which is used for capturing an image of a hand of a user; the chip is used for processing and analyzing the image, identifying a specific gesture input by a user and converting the specific gesture into a standard mouse or touch screen equipment signal; the standard interface is used for outputting a standard mouse equipment signal to a PC (Personal Computer) or other electronic equipment; the gesture recognition and signal processing method of the camera integrated with the gesture control function comprises the steps that when a user makes a specific gesture within the view field of the camera, the image sensor captures an image of the hand of the user and transmits the image to the chip; the chip processes and analyzes the image, identifies a specific gesture input by a user, and converts the specific gesture into a standard mouse device signal. According to the method, a gesture control function supported by additional hardware or software is not needed, and a user can control a PC or other electronic equipment only by making a specific gesture in the view field of the camera.

Description

Technical Field

[0001] The present invention relates to computer vision and human-computer interaction, and in particular to a camera with integrated gesture control function. Background Art

[0002] Existing gesture control devices usually require additional hardware or software support. Generally, in the case of touch devices, if additional hardware devices are added, the cost is high, and if software is used, the operation is often inconvenient or inaccurate.

[0003] In order to solve these problems, the present invention is proposed.

[0004] SUMMARY OF THE INVENTION

[0005] The object of the present invention is to provide a camera with an integrated gesture control function, which converts a specific gesture input by a user into a standard mouse device signal and outputs it to a PC or other electronic devices through a standard interface.

[0006] The purpose of the present invention can be achieved through the following technical solutions:

[0007] A camera with integrated gesture control function includes: an image sensor for capturing images of a user's hand; a chip for processing and analyzing the image, identifying specific gestures input by the user, and converting them into standard mouse or touch screen device signals.

[0008] Furthermore, a standard interface is included for outputting a standard mouse device signal to a PC or other electronic devices.

[0009] Furthermore, the gesture recognition and signal processing method of the camera with integrated gesture control function is as follows: when the user makes a specific gesture within the field of view of the camera, the image sensor captures the image of the user's hand and transmits it to the chip; the chip processes and analyzes the image, recognizes the specific gesture input by the user, and converts it into a standard mouse device signal.

[0010] Preferably, the camera gesture control method comprises the following specific steps:

[0011] S1: Connect the camera to a standard interface of a PC or other electronic device; S2: Open a gesture control program of the PC or other electronic device; S3: Make a specific gesture within the field of view of the camera; S4: Observe the feedback from the PC or other electronic device.

[0012] Furthermore, in step S1, the camera is connected to the USB-C port of a PC or other electronic device, the power switch of the camera is turned on, and the camera and gesture control on mode is selected. At this time, the indicator light will display blue, indicating that the gesture control function has been turned on.

[0013] Furthermore, in step S2, a gesture control program of a PC or other electronic device is opened, and the program can receive image data from the camera and mouse / touch screen signals through the USB-C interface, and perform corresponding operations according to the user's settings.

[0014] Furthermore, in the step S4: outputting a control signal to a PC or other electronic device; outputting the control signal to a PC or other electronic device via a standard interface, and causing it to perform corresponding operations via a driver.

[0015] Furthermore, the step S3 includes activating the gesture recognition function, and activating the gesture recognition function includes the following steps:

[0016] S31: the gesture recognition program periodically obtains images from the image buffer. Before the gesture recognition state is activated, the gesture recognition program works at a low frequency and obtains a frame of image at a certain interval.

[0017] S32: performing target detection on the acquired image to detect whether a hand appears in the image, and marking the hand area in the image with a rectangular frame. If no hand appears in the image, continue to wait for the next frame of image;

[0018] S33: performing gesture recognition on the detected hand to detect whether it is an activation gesture. If the gesture type is an activation gesture, recording the sequence number of the frame image and continuing to wait for the next frame image; if the gesture type is not an activation gesture, clearing the recorded sequence number and continuing to wait for the next frame image;

[0019] S34: Determine whether N consecutive frames are detected as activation gestures, compare the recorded serial numbers, if they are continuous and the number reaches N, enter the gesture recognition activation state and switch to the execution of gesture control function; if they are not continuous or the number is less than N, continue to wait for the next frame image.

[0020] Furthermore, the step S3 further includes executing a gesture control function, and executing the gesture control function includes the following steps:

[0021] S35: acquiring images from the image buffer at a high frequency frame by frame; when the gesture recognition is activated, acquiring one frame of image at a certain interval;

[0022] S36: Identify the gesture action of the first frame of the activation state to determine the gesture type and gesture position; perform gesture recognition on the image of the first frame of the activation state to obtain the gesture type and gesture position; wherein the gesture type determines which type of control signal will be triggered in the next series of frames, and the gesture position determines the starting position coordinates of the triggering control signal;

[0023] S37: Perform gesture recognition on the images of subsequent frames to generate a control signal, perform gesture recognition on the images of subsequent frames to obtain the gesture type and gesture position of the current frame; calculate the corresponding control signal according to the gesture type and gesture position of the current frame and the previous frame.

[0024] Furthermore, the steps S33, S36 and S37 include a gesture recognition method based on deep learning, the gesture recognition method includes a palm direction determination method, and the specific training steps are as follows:

[0025] S100: Using the idea of ​​transfer learning, the gesture dataset is supplemented with multiple sets of data with different palm orientations, which are classified into front, back, left, right, up, and down;

[0026] S200: Select a "hand key point detection" pre-training model as a training starting point;

[0027] S300: replacing the original neural network key point detection output layer with a Softmax layer to output the probability that the palm belongs to front, back, left, right, top, or bottom;

[0028] S400: training and verifying the neural network model;

[0029] S500: When using the model for reasoning, the network end is simultaneously connected to the "hand key point detection" layer and the "palm direction recognition" layer;

[0030] S600: Calculate the coordinates of the key points of the hand and the direction of the palm simultaneously according to the hand image;

[0031] S700: Based on the above model, access the subsequent gesture classification algorithm, which accepts the aforementioned hand key point coordinates and palm direction, and outputs a specific "gesture type".

[0032] Beneficial technical effects:

[0033] 1. The present invention realizes the gesture control function without the need for additional hardware or software support by integrating a camera with gesture control function. The user only needs to make specific gestures within the field of view of the camera to control a PC or other electronic devices. The operation is simple, convenient and efficient.

[0034] 2. The present invention realizes the transmission of images and control signals simultaneously by multiplexing a single interface, so that it can be easily adapted to various electronic devices such as PCs, mobile phones, tablet computers, etc., and realize a wider range of uses.

[0035] 3. The present invention integrates the gesture recognition function directly into the camera, while saving the resources occupied by the gesture algorithm; that is, the image is directly read through the cache area of ​​the camera processing chip, and the computing chip can be equipped with dedicated neural network units such as NPU to effectively improve computing efficiency and reduce resource consumption, and the entire process does not need to occupy the resources of the user terminal device.

[0036] The present invention is further described below in conjunction with the accompanying drawings and embodiments. BRIEF DESCRIPTION OF THE DRAWINGS

[0037] Figure 1 This is a flow chart of the method for performing gesture control on a camera of the present invention.

[0038] Figure 2 This is a flow chart of activating the gesture recognition function of the present invention.

[0039] Figure 3 This is a flow chart of executing the gesture control function of the present invention.

[0040] Figure 4 The figure is a flow chart of specific training steps of the palm direction discrimination method of the present invention. DETAILED DESCRIPTION

[0041] The following will be combined with the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0042] A camera with integrated gesture control function includes: an image sensor for capturing images of a user's hand; a chip for processing and analyzing the image, identifying specific gestures input by the user, and converting them into standard mouse / touch screen device signals; and a standard interface for outputting the standard mouse device signals to a PC or other electronic devices.

[0043] The chip is used for image acquisition, storage and transmission, receiving image data acquired by the image sensor and obtained after analog-to-digital conversion. On the one hand, these images are directly output to a PC or other electronic devices (like a traditional camera), and on the other hand, the images are stored in the chip cache for subsequent gesture recognition.

[0044] The chip includes a non-volatile storage area and a cache area. The non-volatile storage area is equipped with a gesture recognition method. The cache area temporarily stores images. When performing the gesture recognition function, the chip non-volatile storage area is equipped with a gesture recognition method, loads the image in the cache area for gesture recognition, and converts it into a standard input device signal and sends it to a PC or other electronic device.

[0045] In addition, an indicator light and a status switch may be provided.

[0046] The standard interface can be a multiplexed interface for transmitting the camera image and the standard input device signal at the same time, or two interfaces can be used for transmission respectively. If it is the former, the interface needs to support the alternative transmission function.

[0047] The state switching switch of the present invention has the state switching function of turning on gesture control and turning off gesture control. The indicator light of the present invention has the function of status prompting, for example, the camera is turned on / off, the gesture function is turned on / off, there is an active gesture in the picture, and there is no active gesture, which can be prompted by the indicator light.

[0048] The method of gesture recognition and signal processing is as follows: when the user makes a specific gesture within the field of view of the camera, the image sensor captures the image of the user's hand and transmits it to the chip. The chip processes and analyzes the image, recognizes the specific gesture input by the user, and converts it into a standard mouse device signal. The chip then outputs the standard mouse device signal to a PC or other electronic device through a standard interface, thereby realizing the gesture control function.

[0049] Reference Figure 1 The user can perform gesture control by using the camera of the present invention, and the specific steps are as follows:

[0050] S1: Connect the camera to a standard interface of a PC or other electronic device; S2: Open the gesture control program of the PC or other electronic device; S3: Make specific gestures within the field of view of the camera, such as swiping left, swiping right, swiping up, swiping down, clicking, double-clicking, etc.; S4: Observe the feedback from the PC or other electronic device, such as cursor movement, page switching, file opening, etc.

[0051] Specifically, in step S1, connect the camera to the USB-C port of a PC or other electronic device, turn on the power switch of the camera, select the camera and gesture control on mode, and the indicator light will turn blue, indicating that the gesture control function has been turned on.

[0052] Specifically, in step S2, a gesture control program of a PC or other electronic device is opened, and the program can receive image data from the camera and mouse / touch screen signals through the USB-C interface, and perform corresponding operations according to user settings.

[0053] Specifically, in step S4: output control signals to a PC or other electronic device. The program will output control signals to a PC or other electronic device through a standard interface (such as USB-C), and the driver program will cause it to perform corresponding operations, such as cursor movement, page switching, file opening, etc.

[0054] Reference Figure 2 Specifically, step S3 includes activating the gesture recognition function, and activating the gesture recognition function includes the following steps:

[0055] S31: The gesture recognition program periodically obtains images from the image buffer. Before the gesture recognition state is activated, the gesture recognition program works at a low frequency and obtains a frame of image every certain time (such as 0.5 seconds) to save computing resources and power.

[0056] S32: Perform target detection on the acquired image to detect whether a hand appears in the image, and mark the hand area in the image with a rectangular frame. If no hand appears in the image, continue to wait for the next frame of image.

[0057] S33: Perform gesture recognition on the detected hand to detect whether it is an activation gesture. If the gesture type is an activation gesture, record the serial number of the frame image and continue to wait for the next frame image; if the gesture type is not an activation gesture, clear the recorded serial number and continue to wait for the next frame image.

[0058] S34: Determine whether N consecutive frames (N is a preset value, such as 3) are detected as activation gestures, and compare the recorded serial numbers. If they are continuous and the number reaches N, it means that the user has entered the activation gesture, and the gesture recognition activation state will be entered, and the gesture control function will be executed; if they are not continuous, or the number is less than N, it means that the user has not entered the activation gesture, and the next frame image will continue to be waited for.

[0059] Reference Figure 3 In step S3, the gesture control function is also executed, and the gesture control function is executed including the following steps:

[0060] S35: acquiring images from the image buffer at a high frequency frame by frame; when the gesture recognition is activated, this step operates at a high frequency and acquires a frame of image at a certain interval (such as 0.1 seconds) to improve the accuracy and sensitivity of gesture recognition;

[0061] S36: Identify the gesture action of the first frame of the activation state to determine the gesture type and gesture position; perform gesture recognition on the image of the first frame of the activation state to obtain the gesture type and gesture position; wherein the gesture type determines which type of control signal will be triggered in the next series of frames, and the gesture position determines the starting position coordinates of the triggering control signal;

[0062] S37: Perform gesture recognition on the images of subsequent frames to generate a control signal. Perform gesture recognition on the images of subsequent frames to obtain the gesture type and gesture position of the current frame; calculate the corresponding control signal based on the gesture type and gesture position of the current frame and the previous frame. For example, if the "gesture type" is "move", the movement distance and direction of the mouse are calculated according to the change of the "gesture position", and a control signal for the movement of the mouse is generated; if the "gesture type" is "click", then based on the stability of the "gesture position", it is determined whether it is a valid click action, and a control signal for the left mouse button to be clicked is generated; if the "gesture type" is "double-click", then based on the stability of the "gesture position" and the time interval, it is determined whether it is a valid double-click action, and a control signal for the left mouse button to be double-clicked is generated; and so on.

[0063] In steps S33, S36 and S37, a gesture recognition method based on deep learning is used to effectively recognize hand postures and movements and classify them into different gesture types.

[0064] The gesture recognition method introduces "palm direction identification" to assist gesture recognition. This is because the camera used in this solution cannot collect depth information, and the direction of the palm is one of the factors to achieve more diverse and more accurate gesture recognition. Therefore, the palm direction data is introduced separately to more accurately identify gestures. Figure 4 , the specific training steps of the palm direction discrimination method are as follows:

[0065] S100: Using the idea of ​​transfer learning, the gesture dataset is supplemented with multiple sets of data with different palm orientations, which are classified into front, back, left, right, up, and down;

[0066] S200: Select a "hand key point detection" pre-training model as a training starting point;

[0067] S300: replacing the original neural network key point detection output layer with a Softmax layer to output the probability that the palm belongs to front, back, left, right, top, or bottom;

[0068] S400: training and verifying the neural network model;

[0069] S500: When using the model for reasoning, the network end is simultaneously connected to the "hand key point detection" layer and the "palm direction recognition" layer;

[0070] S600: Calculate the coordinates of the key points of the hand and the direction of the palm simultaneously according to the hand image;

[0071] S700: Based on the above model, access the subsequent gesture classification algorithm, which accepts the aforementioned hand key point coordinates and palm direction, and outputs a specific "gesture type".

[0072] In step S700, the gesture classification algorithm is KNN clustering or other machine learning algorithms.

[0073] The technical purpose of the present invention is to provide a simple, convenient and efficient gesture control solution, so that users can control PC or other electronic devices through gestures. The present invention can be used as an accessory device of an ordinary camera, and can realize standardized gesture control of the device without the user having to purchase special software and hardware equipment.

[0074] In addition, it should be understood that although the present specification is described according to implementation modes, not every implementation mode contains only one independent technical solution. This description of the specification is only for the sake of clarity. Those skilled in the art should regard the specification as a whole. The technical solutions in each embodiment may also be appropriately combined to form other implementation modes that can be understood by those skilled in the art.

Claims

1. A camera with integrated gesture control function, characterized in that: include: An image sensor that captures images of the user's hand; a chip that processes and analyzes the image, recognizes specific gestures input by the user, and converts them into signals for a standard mouse or touch screen device.

2. The camera with integrated gesture control function according to claim 1, characterized in that: Also included is a standard connector for outputting standard mouse device signals to a PC or other electronic device.

3. The camera with integrated gesture control function according to claim 1, characterized in that: The method of gesture recognition and signal processing of the camera with integrated gesture control function is as follows: when the user makes a specific gesture within the field of view of the camera, the image sensor captures the image of the user's hand and transmits it to the chip; the chip processes and analyzes the image, recognizes the specific gesture input by the user, and converts it into a standard mouse device signal.

4. The camera with integrated gesture control function according to claim 1, characterized in that: The camera gesture control method comprises the following specific steps: S1: Connect the camera to a standard interface of a PC or other electronic device; S2: Open a gesture control program of the PC or other electronic device; S3: Make a specific gesture within the field of view of the camera; S4: Observe the feedback from the PC or other electronic device.

5. The camera with integrated gesture control function according to claim 4, characterized in that: In step S1, the camera is connected to the USB-C port of a PC or other electronic device, the power switch of the camera is turned on, and the camera and gesture control on mode is selected. At this time, the indicator light will display blue, indicating that the gesture control function has been turned on.

6. The camera with integrated gesture control function according to claim 4, characterized in that: In step S2, a gesture control program of a PC or other electronic device is opened, and the program can receive image data from the camera and mouse / touch screen signals through the USB-C interface, and perform corresponding operations according to the user's settings.

7. The camera with integrated gesture control function according to claim 4, characterized in that: In the step S4: outputting a control signal to a PC or other electronic device; outputting the control signal to a PC or other electronic device via a standard interface, and causing the PC or other electronic device to perform corresponding operations via a driver.

8. The camera with integrated gesture control function according to claim 4, characterized in that: The step S3 includes activating the gesture recognition function, which includes the following steps: S31: the gesture recognition program periodically obtains images from the image buffer. Before the gesture recognition state is activated, the gesture recognition program works at a low frequency and obtains a frame of image at a certain interval. S32: performing target detection on the acquired image to detect whether a hand appears in the image, and marking the hand area in the image with a rectangular frame. If no hand appears in the image, continue to wait for the next frame of image; S33: performing gesture recognition on the detected hand to detect whether it is an activation gesture. If the gesture type is an activation gesture, recording the sequence number of the frame image and continuing to wait for the next frame image; if the gesture type is not an activation gesture, clearing the recorded sequence number and continuing to wait for the next frame image; S34: Determine whether N consecutive frames are detected as activation gestures, compare the recorded serial numbers, if they are continuous and the number reaches N, enter the gesture recognition activation state and switch to the execution of gesture control function; if they are not continuous or the number is less than N, continue to wait for the next frame image.

9. The camera with integrated gesture control function according to claim 8, characterized in that: The step S3 also includes executing a gesture control function, and executing the gesture control function includes the following steps: S35: acquiring images from the image buffer at a high frequency frame by frame; when the gesture recognition is activated, acquiring one frame of image at a certain interval; S36: Identify the gesture action of the first frame of the activation state to determine the gesture type and gesture position; perform gesture recognition on the image of the first frame of the activation state to obtain the gesture type and gesture position; wherein the gesture type determines which type of control signal will be triggered in the next series of frames, and the gesture position determines the starting position coordinates of the triggering control signal; S37: Perform gesture recognition on the images of subsequent frames to generate a control signal, perform gesture recognition on the images of subsequent frames to obtain the gesture type and gesture position of the current frame; calculate the corresponding control signal according to the gesture type and gesture position of the current frame and the previous frame.

10. The camera with integrated gesture control function according to claim 1, characterized in that: The steps S33, S36 and S37 include a gesture recognition method based on deep learning, the gesture recognition method includes a palm direction determination method, and the specific training steps are as follows: S100: Using the idea of ​​transfer learning, the gesture dataset is supplemented with multiple sets of data with different palm orientations, which are classified into front, back, left, right, up, and down; S200: Select a "hand key point detection" pre-training model as a training starting point; S300: replacing the original neural network key point detection output layer with a Softmax layer to output the probability that the palm belongs to front, back, left, right, top, or bottom; S400: training and verifying the neural network model; S500: When using the model for reasoning, the network end is simultaneously connected to the "hand key point detection" layer and the "palm direction recognition" layer; S600: Calculate the coordinates of the key points of the hand and the direction of the palm simultaneously according to the hand image; S700: Based on the above model, access the subsequent gesture classification algorithm, which accepts the aforementioned hand key point coordinates and palm direction, and outputs a specific "gesture type".

Images