A method of positioning a hand region and a display device

Abstract

The application discloses a hand region positioning method and a display device. The method first uses the features of a first color image to preliminarily segment a first depth image, and then uses the depth values of the depth image itself to secondarily segment a second depth image obtained through the preliminary segmentation, reduces first interference data through twice segmentation, and obtains a third depth image. Finally, the hand region determined according to the third depth image is positioned, and the accuracy of gesture recognition is improved. The method comprises the following steps: acquiring a first color image and a corresponding first depth image; using the pixel values of pixel points in the first color image to preliminarily segment the first depth image, and obtaining a second depth image; using the depth values of pixel points of the second depth image to secondarily segment the second depth image, and obtaining a third depth image; and positioning a hand region according to the third depth image.

Description

Technical Field

[0001] This application relates to the technical field of locating hand areas, and more particularly to a method for locating hand areas and a display device. Background Technology

[0002] With the rapid development of artificial intelligence technology, 3D gesture recognition has become an important branch of human-computer interaction and a crucial technological field in virtual technology. 3D gesture recognition technology can be applied not only to games but also plays a vital role in emerging technology areas such as intelligent control, for example, controlling smart home devices through gestures.

[0003] Gesture recognition technology has evolved from initially using motion sensors and other devices to acquire 3D gesture posture information, to directly estimating 3D gesture posture information based on a single color image, and finally to using depth data to participate in deep convolutional neural networks to estimate 3D gesture posture information, with accuracy gradually improving. However, even with current methods using depth data to participate in deep convolutional neural networks to estimate 3D gesture posture information, users still perceive gesture recognition as inaccurate, affecting their gaming experience and intelligent control effects.

[0004] Therefore, how to further improve the accuracy of gesture recognition has become an urgent problem to be solved by those skilled in the art. Summary of the Invention

[0005] Some embodiments of this application provide a method and display device for locating a hand region. The method first uses the features of a first color image to initially segment the first depth image. Then, it uses the depth value of the depth image itself to perform a second segmentation on the second depth image obtained from the initial segmentation. After the two segments reduce the first interference data, a third depth image is obtained. Finally, the hand region determined based on the third depth image is located, thereby improving the accuracy of gesture recognition.

[0006] In a first aspect, a display device is provided, comprising:

[0007] A monitor is used to display the user interface.

[0008] User interface, used to receive input signals;

[0009] The controllers, connected to the display and user interface respectively, are used to perform:

[0010] Acquire a first color image and a corresponding first depth image, wherein the first color image is determined by the original color image and the first depth image is determined by the original depth image;

[0011] Using the pixel values ​​of the pixels in the first color image, the first depth image is initially segmented to obtain a second depth image, so that the second depth image reduces the first interference data relative to the first depth image. The first interference data indicates the external environment image corresponding to the non-hand image in the original depth image. The second depth image is then segmented a second time using the depth values ​​of its own pixels to obtain a third depth image, so that the third depth image reduces the first interference data relative to the second depth image. The hand region is located based on the third depth image.

[0012] In some embodiments, the controller is configured to perform the following steps to determine a first color image and a corresponding first depth image:

[0013] The original color image is cropped to obtain a first color image; cropping data is determined based on the first color image, wherein the cropping data is used to indicate a first positional relationship between the first color image and the original color image; the original depth image is cropped to obtain a first depth image, such that a second positional relationship between the first depth image and the original depth image is the same as the first positional relationship.

[0014] In some embodiments, the controller is configured to perform preliminary segmentation of the first depth image using pixel values ​​of pixels in the first color image to obtain a second depth image, according to the following steps:

[0015] The pixel value includes an HSV color value; the HSV color value of the pixel is compared with a first mask, wherein the first mask is determined according to second interference data, and the second interference data indicates the external environment image corresponding to the non-hand image in the original color image; the pixel position of the pixel with the same HSV color value as the first mask is determined; the pixels corresponding to the pixel positions in the first depth image are eliminated to obtain the second depth image.

[0016] In some embodiments, the controller is configured to perform secondary segmentation of the second depth image using the depth values ​​of its own pixels to obtain a third depth image according to the following steps:

[0017] The image at the center of the second depth image is selected as the reference image;

[0018] A reference depth is determined based on the depth values ​​of pixels in the reference image, and the reference depth is used to determine a second mask; the second depth image is segmented using the second mask to obtain a third depth image.

[0019] In some embodiments, the center of the reference image is set to the center of the second depth image; the length of the reference image is 1 / 2 of the second depth image, and the width of the reference image is 1 / 2 of the second depth image.

[0020] In some embodiments, the controller is configured to perform the following steps to determine a reference depth based on the depth values ​​of pixels in the reference image:

[0021] From the depth values ​​of pixels in the reference image, non-zero depth values ​​are selected, and an average depth is calculated for the non-zero depth values; the average depth is then used as the reference depth.

[0022] In some embodiments, the controller is configured to determine a second mask based on the reference depth according to the following steps:

[0023] The range of the second mask is determined based on preset prior information and the average depth, wherein the prior information is the maximum distance between the human hand and the palm, the maximum value in the range of the second mask is the sum of the average depth and the maximum distance, and the minimum value in the range of the second mask is the difference between the average depth and the maximum distance.

[0024] In some embodiments, the controller is configured to perform the following steps to segment the second depth image using the second mask to obtain a third depth image:

[0025] Compare the depth values ​​of the pixels in the second depth image with the second mask;

[0026] The depth values ​​in the second depth image that are not within the range of the second mask are eliminated to obtain the third depth image.

[0027] In some embodiments, the controller is configured to perform the following steps to locate the hand region based on the third depth image:

[0028] For the third depth image, multiple hand region localization methods are used to determine the corresponding hand region to be determined;

[0029] From all the undetermined hand regions, select the one with the largest outline and use it as the hand region for positioning.

[0030] Secondly, a method for locating a hand region is provided, including:

[0031] Acquire a first color image and a corresponding first depth image, wherein the first color image and the first depth image are determined by the original color image and the original depth image captured by the depth camera;

[0032] Using the pixel values ​​of the pixels in the first color image, the first depth image is initially segmented to obtain a second depth image, so that the second depth image reduces the first interference data relative to the first depth image. The first interference data indicates the external environment image corresponding to the non-hand image in the original depth image. The second depth image is then segmented a second time using the depth values ​​of its own pixels to obtain a third depth image, so that the third depth image reduces the first interference data relative to the second depth image. The hand region is located based on the third depth image.

[0033] In the above embodiments, a method and display device for locating a hand region are disclosed. The method first uses features of a first color image to initially segment a first depth image. Second, it uses the depth values ​​of the depth image itself to perform a second segmentation on the second depth image obtained from the initial segmentation. After these two segmentations, first interference data is reduced, resulting in a third depth image. Finally, the hand region determined based on the third depth image is located, improving the accuracy of gesture recognition. The method includes: acquiring a first color image and a corresponding first depth image; performing initial segmentation of the first depth image using the pixel values ​​of pixels in the first color image to obtain a second depth image; performing a second segmentation of the second depth image using the depth values ​​of its own pixels to obtain a third depth image; and locating the hand region based on the third depth image. Attached Figure Description

[0034] Figure 1 An operational scenario between a display device and a control device according to some embodiments is illustrated;

[0035] Figure 2 A hardware configuration block diagram of a control device 100 according to some embodiments is shown;

[0036] Figure 3 A hardware configuration block diagram of a display device 200 according to some embodiments is shown;

[0037] Figure 4 A software configuration diagram of a display device 200 according to some embodiments is shown;

[0038] Figure 5 An exemplary schematic diagram of a hand region is shown according to some embodiments;

[0039] Figure 6 An exemplary schematic diagram showing the locations of all the joints of a hand according to some embodiments is provided;

[0040] Figure 7 An exemplary flowchart is shown for a method of locating a hand region according to some embodiments;

[0041] Figure 8 An example of a user interaction diagram according to some embodiments is shown;

[0042] Figure 9 An exemplary schematic diagram of a user interface according to some embodiments is shown;

[0043] Figure 10 An exemplary schematic diagram of an original color image according to some embodiments is shown;

[0044] Figure 11 An exemplary schematic diagram of a first color image according to some embodiments is shown;

[0045] Figure 12 A schematic diagram of a first coordinate system according to some embodiments is shown as an example;

[0046] Figure 13 An exemplary schematic diagram illustrates the position of a first color image in a first coordinate system according to some embodiments;

[0047] Figure 14 A schematic diagram of a second coordinate system according to some embodiments is shown as an example;

[0048] Figure 15 An exemplary schematic diagram of yet another first color image according to some embodiments is shown;

[0049] Figure 16 An exemplary schematic diagram of a first depth image according to some embodiments is shown;

[0050] Figure 17 An exemplary schematic diagram of a second color image according to some embodiments is shown;

[0051] Figure 18 An exemplary schematic diagram of a second depth image according to some embodiments is shown;

[0052] Figure 19 A schematic diagram of a reference image according to some embodiments is shown by way of example;

[0053] Figure 20 An exemplary schematic diagram of a third depth image according to some embodiments is shown;

[0054] Figure 21 An exemplary schematic diagram of a hand region according to some embodiments is shown. Detailed Implementation

[0055] To make the objectives and implementation methods of this application clearer, the exemplary implementation methods of this application will be clearly and completely described below with reference to the accompanying drawings of the exemplary embodiments of this application. Obviously, the exemplary embodiments described are only some embodiments of this application, and not all embodiments.

[0056] It should be noted that the brief descriptions of terms in this application are only for the convenience of understanding the embodiments described below, and are not intended to limit the embodiments of this application. Unless otherwise stated, these terms should be understood in their ordinary and common meaning.

[0057] The terms "first," "second," "third," etc., used in the specification, claims, and accompanying drawings of this application are used to distinguish similar or related objects or entities, and do not necessarily imply a specific order or sequence, unless otherwise specified. It should be understood that such terms are interchangeable where appropriate.

[0058] The terms “include” and “have”, and any variations thereof, are intended to cover but not exclusively include, for example, a product or device that includes a range of components is not necessarily limited to all of the components that are clearly listed, but may include other components that are not clearly listed or that are inherent to such product or device.

[0059] The display device provided in this application can have various implementation forms, such as a television, a smart television, a laser projection device, a monitor, an electronic bulletin board, an electronic table, etc. Figure 1 and Figure 2 This is one specific embodiment of the display device of this application.

[0060] Figure 1 This is a schematic diagram illustrating the operational scenario between the display device and the control unit according to the embodiment. Figure 1 As shown, the user can operate the display device 200 through the smart device 300 or the control device 100.

[0061] In some embodiments, the control device 100 may be a remote control. Communication between the remote control and the display device includes infrared protocol communication, Bluetooth protocol communication, and other short-range communication methods, controlling the display device 200 wirelessly or via wired means. Users can control the display device 200 by inputting user commands through buttons on the remote control, voice input, control panel input, etc.

[0062] In some embodiments, a smart device 300 (such as a mobile terminal, tablet computer, computer, laptop computer, etc.) may also be used to control the display device 200. For example, an application running on the smart device may be used to control the display device 200.

[0063] In some embodiments, the display device may receive instructions not through the aforementioned smart devices or control devices, but through touch or gestures.

[0064] In some embodiments, the display device 200 can also be controlled in ways other than the control device 100 and the smart device 300. For example, it can be controlled by directly receiving the user's voice commands through a module configured inside the display device 200 for acquiring voice commands, or it can be controlled by receiving the user's voice commands through a voice control device set outside the display device 200.

[0065] In some embodiments, the display device 200 also communicates with the server 400. The display device 200 may communicate via a local area network (LAN), wireless local area network (WLAN), and other networks. The server 400 may provide various content and interactive features to the display device 200. The server 400 may be a cluster or multiple clusters, and may include one or more types of servers.

[0066] Figure 2 An exemplary block diagram of the configuration of the control device 100 according to an exemplary embodiment is shown. Figure 2 As shown, the control device 100 includes a controller 110, a communication interface 130, a user input / output interface 140, a memory, and a power supply. The control device 100 can receive user input operation commands and convert the operation commands into commands that the display device 200 can recognize and respond to, thus acting as an intermediary for interaction between the user and the display device 200.

[0067] like Figure 3 The display device 200 includes at least one of the following: a tuner 210, a communicator 220, a detector 230, an external device interface 240, a controller 250, a display 260, an audio output interface 270, a memory, a power supply, and a user interface.

[0068] In some embodiments, the controller includes a processor, a video processor, an audio processor, a graphics processor, RAM, ROM, and a first interface to an nth interface for input / output.

[0069] The display 260 includes a display screen assembly for presenting images, a driving assembly for driving image display, a component for receiving image signals from the controller output, and a user control UI interface for displaying video content, image content, menu control interface, and user control UI interface.

[0070] The display 260 can be an LCD display, an OLED display, or a projection display, and can also be a projection device and a projection screen.

[0071] The communicator 220 is a component used to communicate with external devices or servers according to various communication protocol types. For example, the communicator may include at least one of the following: a Wi-Fi module, a Bluetooth module, a wired Ethernet module, other network communication protocol chips or near-field communication protocol chips, and an infrared receiver. The display device 200 can establish the transmission and reception of control signals and data signals with the external control device 100 or the server 400 through the communicator 220.

[0072] The user interface can be used to receive control signals from the control device 100 (such as an infrared remote control).

[0073] Detector 230 is used to collect signals from the external environment or to interact with the external environment. For example, detector 230 includes a light receiver, a sensor for collecting ambient light intensity; or, detector 230 includes an image acquisition device, such as a camera, which can be used to collect external environmental scenes, user attributes, or user interaction gestures; or, detector 230 includes a sound acquisition device, such as a microphone, for receiving external sounds.

[0074] The external device interface 240 may include, but is not limited to, one or more of the following: High Definition Multimedia Interface (HDMI), analog or high-definition component input interface (component), composite video input interface (CVBS), USB input interface (USB), RGB port, etc. It may also be a composite input / output interface formed by multiple interfaces mentioned above.

[0075] The tuner / demodulator 210 receives broadcast television signals via wired or wireless means, and demodulates audio and video signals, such as EPG data signals, from multiple wireless or wired broadcast television signals.

[0076] In some embodiments, the controller 250 and the tuner 210 may be located in different separate devices, that is, the tuner 210 may also be located in an external device of the main device where the controller 250 is located, such as an external set-top box.

[0077] The controller 250 controls the operation of the display device and responds to user operations through various software control programs stored in the memory. The controller 250 controls the overall operation of the display device 200. For example, in response to receiving a user command to select a UI object to display on the monitor 260, the controller 250 can execute operations related to the object selected by the user command.

[0078] In some embodiments, the controller includes at least one of a central processing unit (CPU), a video processor, an audio processor, a graphics processing unit (GPU), RAM (random access memory), ROM (read-only memory), a first to an nth interface for input / output, a communication bus, etc.

[0079] Users can input commands through a graphical user interface (GUI) displayed on the monitor 260, and the user input interface receives the user input commands through the GUI. Alternatively, users can input commands by entering specific sounds or gestures, and the user input interface receives the user input commands by recognizing the sounds or gestures through sensors.

[0080] A "user interface" is the medium through which an application or operating system interacts and exchanges information with the user. It converts information from its internal form to a form that the user can accept. A common form of user interface is the graphical user interface (GUI), which refers to a user interface related to computer operation displayed graphically. It can be an icon, window, control, or other interface element displayed on the screen of an electronic device. Controls can include visual interface elements such as icons, buttons, menus, tabs, text boxes, dialog boxes, status bars, navigation bars, and widgets.

[0081] See Figure 4 In some embodiments, the system is divided into four layers, from top to bottom: the Applications layer (referred to as the "Application Layer"), the Application Framework layer (referred to as the "Framework Layer"), the Android runtime and system library layer (referred to as the "System Runtime Library Layer"), and the kernel layer.

[0082] In some embodiments, at least one application runs in the application layer. These applications may be Windows programs, system settings programs, or clock programs that come with the operating system; they may also be applications developed by third-party developers. In specific implementations, the application packages in the application layer are not limited to the examples above.

[0083] The framework layer provides application programming interfaces (APIs) and a programming framework for applications. The application framework layer includes predefined functions. It acts as a central processing unit, determining the actions taken by applications within the application layer. Through the API, applications can access system resources and obtain system services during execution.

[0084] like Figure 4 As shown, the application framework layer in this embodiment includes managers, content providers, etc., wherein the managers include at least one of the following modules: ActivityManager, which interacts with all activities running in the system; LocationManager, which provides access to system location services for system services or applications; PackageManager, which retrieves various information related to application packages currently installed on the device; NotificationManager, which controls the display and clearing of notification messages; and WindowManager, which manages icons, windows, toolbars, wallpapers, and desktop widgets on the user interface.

[0085] In some embodiments, the Activity Manager manages the lifecycle of individual applications and common navigation and back functions, such as controlling application exit, opening, and back actions. The Window Manager manages all window programs, such as obtaining the screen size, determining if a status bar is present, locking the screen, capturing the screen, and controlling display window changes (e.g., shrinking the display window, shaking the display, distorting the display, etc.).

[0086] In some embodiments, the system runtime library layer provides support for the upper layer, namely the framework layer. When the framework layer is used, the Android operating system runs the C / C++ libraries contained in the system runtime library layer to implement the functions that the framework layer needs to perform.

[0087] In some embodiments, the kernel layer is a layer between hardware and software. For example... Figure 4 As shown, the kernel layer includes at least one of the following drivers: audio driver, display driver, Bluetooth driver, camera driver, WIFI driver, USB driver, HDMI driver, sensor driver (such as fingerprint sensor, temperature sensor, pressure sensor, etc.), and power driver.

[0088] With the rapid development of artificial intelligence technology, 3D gesture recognition has become an important branch of human-computer interaction and a crucial technological field in virtual technology. 3D gesture recognition technology can be applied not only to games but also plays a vital role in emerging technology areas such as intelligent control, for example, controlling smart home devices through gestures.

[0089] 3D gesture recognition technology includes gesture recognition using motion sensors, data gloves, and other 3D gesture technologies; estimation of 3D gesture pose information based on a single color image (RGB image) using a deep convolutional neural network; and estimation of 3D gesture pose information based on depth data using a deep convolutional neural network. The color image can be understood as an image with color; images captured by traditional cameras are color images. The depth data can be determined from depth images captured by a depth camera. The depth data (hereinafter referred to as depth value) refers to the distance between the object being photographed and the camera.

[0090] However, products that use motion sensors and other devices to acquire 3D gesture posture information cannot provide users with a more natural experience. Estimating 3D gesture posture information directly from a single color image is not accurate enough, and ultimately only the relative depth of the hand gesture can be estimated, resulting in poor gesture recognition. Estimating 3D gesture posture information using depth data in conjunction with deep convolutional neural networks is more accurate than the previous two methods. However, even with this method, users still perceive inaccurate gesture recognition, impacting the user experience.

[0091] Therefore, how to further improve the accuracy of gesture recognition has become an urgent problem to be solved by those skilled in the art.

[0092] Research has shown that in the process of estimating 3D gesture pose information using deep convolutional neural networks based on depth data, i.e., recognizing gestures based on depth data, it is necessary to first detect the hand region, obtain the hand region, and then determine the gesture by identifying the joint information in the hand region.

[0093] It should be noted that the hand region is a rectangular image formed by including the hand image. For example, the hand region corresponding to the most accurate hand image refers to a rectangular image drawn that closely matches the hand's contour, such as... Figure 5 As shown, Figure 5 An exemplary schematic diagram of a hand region according to some embodiments is shown. Figure 5 The four sides of the hand region 500 described herein are respectively aligned with the outline of the hand, and at this time the hand region 500 is the optimal hand region of the hand image.

[0094] In some embodiments, when the user's hand is fully extended, disregarding the six degrees of freedom of the entire hand relative to space, the human hand has 21 degrees of freedom. In this text, this can be understood as the total number of hand joints being 21, including the fingertips, joints, and the base of the palm. For example... Figure 6 As shown, Figure 6 An exemplary schematic diagram showing the positions of all hand joints according to some embodiments is provided. In this embodiment, the gesture displayed by the user is ultimately determined by predicting the hand image joints.

[0095] However, when determining gestures, the requirements for the hand area are quite strict, requiring a clean image of the hand. During gesture recognition, the user can place their hand within the field of view of the depth camera, which then captures both the hand and its surrounding environment. Since the user is in the external environment, the depth camera inevitably captures the surrounding environment as well. Therefore, the resulting depth image contains not only the hand image but also the external environment image (also known as the first interference data). For example, since the user's 3D gesture game interaction scenario is primarily indoors, and the user is directly facing the depth camera, the images of the rest of the user's body, excluding the hand area, naturally become the first interference data. If this first interference data is not processed, during 3D hand pose estimation, the hand area will include not only the hand image but also the first interference data. This first interference data will severely affect the accuracy of joint prediction from the hand image, ultimately leading to inaccurate gesture recognition. Therefore, whether the hand image in the final hand area is clearly separated from the first interference data directly affects the recognition of joints from the hand image, ultimately impacting the gesture recognition performance.

[0096] In order to completely separate the hand image from the first interference data in the depth image, this application provides a method for hand region. The method first uses the features of the first color image to perform preliminary segmentation of the first depth image. Second, it uses the depth value of the depth image itself to perform secondary segmentation of the second depth image obtained from the preliminary segmentation. After the two segmentations reduce the first interference data, a third depth image is obtained. Finally, the hand region determined based on the third depth image is located, thereby improving the accuracy of gesture recognition.

[0097] It should be noted that the method in this application embodiment can be applied to display devices, as well as computers, VR glasses, AR glasses, and other terminals that can be equipped with depth cameras.

[0098] The method, such as Figure 7 As shown, Figure 7 An exemplary flowchart illustrates a method for locating a hand region according to some embodiments, including:

[0099] S100. Obtain a first color image and a corresponding first depth image, wherein the first color image is determined by the original color image and the first depth image is determined by the original depth image.

[0100] In this embodiment, the depth camera can be built into the display device or an external device connected to the display device. For example, such as... Figure 1 As shown, a depth camera 231 is installed on the display device 200. The depth camera can simultaneously capture color images and depth images. Therefore, in this embodiment, gesture recognition is performed using the two types of images captured by the depth camera.

[0101] The color image in this embodiment is an RGB image, where each pixel has a corresponding pixel value. Each pixel in the depth image has a corresponding depth value, which indicates the distance of the object being photographed from the depth camera.

[0102] In this application embodiment, there are multiple scenarios in which the depth camera is activated for shooting.

[0103] In one example, a first preset application is launched to acquire raw color and raw depth images captured by a depth camera, and a method for recognizing gestures is executed. For example, the first preset application could be a game application, in which a controlled object moves according to the recognized gestures. When the gesture changes position, the controlled object also changes accordingly.

[0104] like Figure 8 As shown, Figure 8 An exemplary diagram of a user interaction according to some embodiments is shown. Figure 8 Figure (a) illustrates a user interface including game application controls, which are displayed on the user interface. In some embodiments, when the display device displays the game application controls, other controls that can implement the game application controls may also be displayed, without limitation. The user can move the focus to the game application control using a control device and press the confirmation button on the control device, causing the user interface to jump to, for example, Figure 8 As shown in (b), Figure 8 Figure (b) shows a user interface for a game screen, where the user can control the movement of a controlled object (airplane) using gestures. For example, a right-pointing gesture moves the controlled object to the right, and a down-pointing gesture moves the controlled object downwards.

[0105] It should be noted that, Figure 8The user interface containing the game screen in section (b) generally does not appear during actual gameplay. Two controlled objects will not appear simultaneously; only the moved controlled object will exist, i.e., the controlled object pointed to by the dotted arrow in the user interface. Figure 8 The purpose of marking the controlled object before and after the movement in (b) is to clearly show the movement process of the controlled object. Of course, in some embodiments, depending on the needs of the game settings, the controlled object before the movement, that is, the controlled object at the starting point of the dashed arrow, can also be retained.

[0106] In another example, a second preset application is launched, which includes a project. Launching this project activates a depth camera to capture a raw color image and a raw depth image. For example, this project could be an effects project that adds effects based on a hand; when this effects project is launched, the depth camera is activated to capture a raw color image and a raw depth image.

[0107] For example, such as Figure 9 As shown, Figure 9 An exemplary schematic diagram of a user interface according to some embodiments is shown. This user interface displays special effects controls corresponding to special effects items. The user uses a control device to move the focus to the special effects control and presses an confirmation button on the control device to activate a depth camera to capture a raw color image and a raw depth image. After recognizing the gesture, the gesture is used to add special effects to the user's hand. In some embodiments, when the display device displays the special effects controls, controls that can perform other functions may also be displayed; this is not limited to these embodiments.

[0108] In some embodiments, the special effects control may be displayed on a floating layer of the user interface, and the special effects control is not located at the bottom of the user interface to avoid the special effects control obscuring other display content on the user interface.

[0109] In some embodiments, if you want to cancel adding an effect, you can move the focus back to the effect control and press the confirmation button on the control device to cancel the effect display.

[0110] In some embodiments, the special effects controls can be invoked and displayed by pressing a specific button on the control device. Specifically, upon entering the second preset application, the special effects controls are not directly displayed on the user interface. This is because continuously displaying the special effects controls might obscure part of the user interface, affecting the integrity of the content displayed in the user interface. In this embodiment, when the user presses a specific button on the control device, the display interface is controlled to show the special effects controls. This allows the special effects controls to be displayed when needed, avoiding the special effects controls from continuously obscuring the user interface.

[0111] In some embodiments, when a special effects control is selected, the special effects project is activated, and the special effects control is no longer displayed. In other embodiments, the user can press a specific button again to control the display device to not display the special effects control.

[0112] The method described in this application can also be applied to fields such as air keyboards and gesture-based touchless teaching. An air keyboard can be understood as a virtual keyboard without a physical keyboard, where gestures are used to determine the keys pressed by the user, thus enabling input. Gesture-based touchless teaching can be understood as allowing users to write text and draw on a virtual blackboard using gestures.

[0113] It should be noted that in this embodiment, moving the focus to a control can thicken the control's border. Alternatively, other methods can be used to indicate that the control is selected. For example, when the focus moves to a control, the control can change its shape, such as from a square to a circle. Alternatively, when a control is selected, it can be enlarged according to a preset ratio. For instance, if the display area of ​​video controls on the user interface is the same, when a control is selected, its display area increases by 1.2 times compared to the original control's display area. Since this application does not restrict the shape of the control when the focus is on it, other forms that facilitate user identification of selected video controls are acceptable.

[0114] In this embodiment of the application, the first color image is determined by the original color image, and the first depth image is determined by the original depth image.

[0115] In some embodiments, a raw color image and a corresponding raw depth image captured by a depth camera are acquired. The raw color image is directly used as the first color image, and the raw depth image is directly used as the first depth image.

[0116] In other embodiments, since the object being captured by the depth camera may include the external environment surrounding the human hand during the shooting process, both the original color image and the corresponding original depth image contain a large amount of external environment images. These external environment images in the original depth image can also be referred to as first interference data. Because of the large amount of first interference data, it can affect the recognition of joint points in the hand image. Therefore, in this embodiment, the original depth image is cropped to reduce the first interference data. To reduce the first interference data, the steps for determining the first color image and the corresponding first depth image include:

[0117] The original color image is cropped to obtain a first color image. In this embodiment, the purpose of cropping the original color image is mainly to obtain a first color image with less second interference data, which includes the hand image. In this embodiment, the external environment image in the original color image is referred to as the second interference data.

[0118] In some embodiments, the step of cropping the original color image includes: identifying a hand region in the original color image, identifying the identified hand region, the hand region comprising a rectangular image of the hand image, and using the hand region as a first color image.

[0119] This application does not limit the method of locating the hand region in the original color image. In some embodiments, the step of cropping the original color image includes: inputting the original color image into a preset deep convolutional neural network to obtain the hand region; wherein the deep convolutional neural network is learned through a large number of samples, the samples including several original color images and the corresponding relative parameters of the hand region. In related technologies, since the hand region is determined based on features in the hand image, the hand region is centered on the hand image.

[0120] In this embodiment of the application, the first color image obtained by cropping includes a hand image. For example, as shown... Figure 10 As shown, Figure 10 An exemplary schematic diagram of an original color image according to some embodiments is shown, where dashed lines are used to represent the second interference data in the original color image because it is too complex. Figure 11 As shown, Figure 11 An exemplary schematic diagram of a first color image 600 according to some embodiments is shown, the first color image 600 including a hand image and a small amount of second interference data compared to the original color image.

[0121] It should be noted that the embodiments of this application do not limit the specific method of cropping the original color image to obtain the first color image. Any method that can obtain the first color image that reduces the second interference data of the original color image and includes the hand image is acceptable.

[0122] Based on the first color image, cropping data is determined, wherein the cropping data is used to indicate a first positional relationship between the first color image and the original color image.

[0123] In some embodiments, a coordinate system can be used to determine the cropping data corresponding to the first color image. The coordinate system can describe the positional relationship between the first color image and the original color image.

[0124] In some embodiments, a first coordinate system is established with the top left corner of the original color image as the origin, the horizontal direction to the right as the positive x-axis, and the vertical direction downwards as the positive y-axis. For example, ... Figure 12 As shown, Figure 12 A schematic diagram of a first coordinate system according to some embodiments is shown as an example.

[0125] The position of the first color image in the first coordinate system is determined, and this position indicates the positional relationship between the original color image and the first color image.

[0126] In some embodiments, the position of the first color image in the first coordinate system can be represented by the coordinates of the four vertices of the first color image, see again. Figure 12 ,exist Figure 12 The position of the first color image in the first coordinate system is determined by vertices A, B, C, and D. The coordinates of these four vertices are used as clipping data. In other embodiments, the position of the first color image in the first coordinate system can be represented by the coordinates of the top-left corner vertex of the first color image, as well as the width and height of the first color image, such as... Figure 13 As shown, Figure 13 An exemplary diagram illustrates the position of a first color image in a first coordinate system according to some embodiments. Figure 13 Using vertex A, and the width W and height H of the first color image, the position of the first color image in the first coordinate system is determined. Vertex A, and the width W and height H of the first color image are used as clipping data.

[0127] In other embodiments, the center of the original color image is used as the origin, and a second coordinate system is set with the horizontal direction to the right as the positive x-axis and the vertical direction downwards as the positive y-axis, such as... Figure 14 As shown, Figure 14 A schematic diagram of a second coordinate system according to some embodiments is illustrated. The position of a first color image in the second coordinate system is determined, and the position determined in the second coordinate system is used as cropping data. The position determined in the second coordinate system can use the same description method as the position in the first coordinate system, and will not be repeated here.

[0128] In this embodiment, the positional relationship between the first color image and the original color image can be determined by cropping the data. It is understood that the original color image and the original depth image have a one-to-one correspondence, having the same number of pixels, and each pixel has a corresponding position. Therefore, the first depth image can be obtained by cropping from the original depth image according to the first positional relationship. In this embodiment, the original depth image is cropped to obtain the first depth image, so that the second positional relationship between the first depth image and the original depth image is the same as the first positional relationship.

[0129] For example, see Figure 15 and Figure 16 As shown, Figure 15 An exemplary schematic diagram of yet another first color image according to some embodiments is shown. Figure 16 An exemplary schematic diagram of a first depth image according to some embodiments is shown.

[0130] In some embodiments, due to the influence of different lighting conditions, holes may exist in the original depth image captured by the depth camera, and consequently, the first depth image may also contain holes. This has a significant impact on predicting gestures from hand joints in the hand image. If the depth value of a joint happens to fall within a hole area, the depth value becomes invalid. For example, if a user is using gestures to grasp a virtual object, the holes in the depth image cause the hand joint recognition to be invalid or significantly inaccurate, preventing the user from grasping the virtual object and affecting the user experience. For example, see again... Figure 16 , Figure 16 Holes exist in the first depth image.

[0131] In some embodiments, the hole can be filled by the following method: Hole filling is performed on the obtained depth image. A 5x5 convolution kernel is used to perform image morphological transformation on the depth data to fill the hole, and then Gaussian filtering is used to finally obtain the filled depth image. The filled image is then used in subsequent steps, that is, the filled image is used instead of the first depth image to perform subsequent preliminary segmentation steps. S200: Using the pixel values ​​of the pixels in the first color image, the first depth image is initially segmented to obtain a second depth image, so that the second depth image reduces first interference data relative to the first depth image. The first interference data indicates the external environment image corresponding to the non-hand image in the original depth image.

[0132] In this embodiment, the color image has more colors than the depth image. Using the pixel values ​​of pixels in the first color image corresponding to the first depth image, second interference data in the first color image is determined. Then, the positional relationship between the second interference data and the first color image is determined, and the first interference data in the first depth image is removed according to this positional relationship.

[0133] In some embodiments, the step of performing preliminary segmentation of the first depth image using the pixel values ​​of pixels in the first color image to obtain a second depth image includes:

[0134] The pixel values ​​include HSV color values.

[0135] In this embodiment, the parameters of the HSV color value are: hue (H), saturation (S), and lightness (V). The RGB values ​​are obtained by varying the red (R), green (G), and blue (B) color channels and superimposing them. RGB represents the colors of the red, green, and blue channels. When describing the color of a pixel, the HSV color value, compared to the RGB value, adds saturation and lightness elements, which can more accurately describe the color of the pixel, and thus make it easier to distinguish the hand image and the second interference data in the first color image.

[0136] The HSV color values ​​of the pixels are compared with a first mask, wherein the first mask is determined based on second interference data. The first mask is also represented by HSV color values. In this embodiment, the method of determining the first mask using the second interference data is not limited, as long as the purpose of setting the first mask is to reduce the second interference data in the first color image.

[0137] For example, such as Figure 17 As shown, Figure 17 An exemplary schematic diagram of a second color image according to some embodiments is shown. Figure 17 The image in the image is the second color image after removing some of the second interference data from the first color image.

[0138] The pixel position corresponding to the HSV color value that is the same as the first mask is determined; the pixels corresponding to the pixel positions in the first depth image are eliminated to obtain the second depth image. Since the second interference data in the first color image exists in the first depth image as first interference data, after determining the pixel position, the pixels corresponding to that pixel position in the first depth image are eliminated, thus reducing the first interference data in the first depth image. For example, as shown... Figure 18 As shown, Figure 18 An exemplary schematic diagram of a second depth image according to some embodiments is shown. Figure 18 The image in the image is the second depth image obtained after filling in the holes and performing preliminary segmentation.

[0139] In some embodiments, the pixel position can be represented by the row number and column number of the pixel within the entire image. For example, the pixel position can be represented as row number 1 and column number 2. The row number can be determined by arranging the image from top to bottom or bottom to top. The column number can be determined by arranging the image from left to right or right to left.

[0140] In this embodiment, since the first depth image and the first color image correspond, the pixels corresponding to the pixel positions obtained from the first color image can be eliminated to obtain the second depth image. For example, if the pixel with row number 1 and column number 2 in the first color image is determined to be second interference data, then the pixel with row number 1 and column number 2 in the first depth image is eliminated.

[0141] It is worth noting that in some embodiments, in order to obtain the hand region and accurately recognize the gesture, it is necessary to ensure that the effect of gesture recognition is not affected when performing the steps of the method in the embodiments of this application. Therefore, the premise of minimizing interference data is to avoid eliminating key images used for gesture recognition from the hand image. The same principle applies to the steps below.

[0142] S300: The second depth image is segmented a second time using the depth values ​​of its own pixels to obtain a third depth image, so that the third depth image reduces the first interference data relative to the second depth image.

[0143] In this embodiment, the first depth image is initially segmented using the features of the first color image, namely pixel values. Then, to further reduce the first interference data, the second depth image obtained from the initial segmentation is further segmented using the depth values ​​of its own pixels. After the initial and secondary segmentation, a more accurate hand image can be obtained. The depth value determines the distance between the object being photographed and the depth camera. It is understood that the distance between the external environment and the distance between the hand and the depth camera are different; therefore, this application uses the depth value to eliminate the first interference data corresponding to the external environment.

[0144] In some embodiments, the step of performing secondary segmentation on the second depth image using the depth values ​​of its own pixels to obtain a third depth image is applicable when the source of the second depth image is a first depth image cropped from the original depth image, and includes:

[0145] The image at the center of the second depth image is selected as a reference image. In this embodiment, the original depth image is cropped to obtain a first depth image including the hand image. Generally, the hand image occupies the center position of the first depth image. After the first depth image is initially segmented to obtain the second depth image, the hand image will still appear at the center position of the second depth image. This embodiment uses depth values ​​to perform secondary segmentation of the second depth image, that is, by using the distance between the hand and the depth camera, and the distance between the external environment and the depth camera, secondary segmentation is achieved, ultimately separating the hand image and the first interference image.

[0146] In some embodiments, the center of the reference image is set to the center of the second depth image; the length of the reference image is half the length of the second depth image, and the width of the reference image is half the width of the second depth image. For example, as... Figure 19 As shown, Figure 19 An exemplary schematic diagram of a reference image according to some embodiments is shown. Figure 20 Reference image 700 is displayed.

[0147] A reference depth is determined based on the depth values ​​of pixels in the reference image, and this reference depth is used to determine the second mask. It is understood that the distances between the hand and the external environment and the depth camera are different. For example, when a user is standing in a living room, the distances of each finger from the depth camera, the distances between the hand and the face, and the distances between the hand and furniture in the living room are all different. In this embodiment, the second depth image is segmented secondaryly using the different distances between each object in the photographed object and the depth camera.

[0148] Since the reference region contains a significant proportion of hand images, a reference depth corresponding to this region can be used as a reference for the depth value of the hand image, determining the likely depth value of the distance between the hand and the depth camera. By filtering out external environment images that are significantly farther from the hand and depth camera, and eliminating pixels from these images, the first interference data affecting hand region localization can be further reduced, resulting in a third depth image. For example, as shown... Figure 20 As shown, Figure 20 An exemplary schematic diagram of a third depth image according to some embodiments is shown.

[0149] In some embodiments, the step of determining the reference depth based on the depth values ​​of pixels in the reference image includes: filtering out non-zero depth values ​​from the depth values ​​of pixels in the reference image, and calculating an average depth for the non-zero depth values; and using the average depth as the reference depth.

[0150] In this embodiment, due to different user gestures, there may be cases where the depth value of a pixel in the reference image is zero. For example, when a user makes an "OK" gesture, a hollow circle is formed between the thumb and forefinger, and the external environment image is actually displayed within this hollow circle. In some embodiments, the depth value of the external environment image, which is far from the depth camera, is set to zero in the depth image. Therefore, this application filters out non-zero depth values ​​and calculates an average depth for these non-zero depth values. This average depth can reflect the average depth value of the hand when displayed in the second depth image.

[0151] The reference depth D can be calculated according to the following formula:

[0152]

[0153] Where Di,j represents the depth value of a pixel in the second depth image, i represents the row number of the pixel, j represents the column number of the pixel, N represents the number of pixels with non-zero depth in the reference region, H1 represents the total number of rows of pixels in the reference image, and W1 represents the total number of columns of pixels in the reference image.

[0154] Using the second mask, the second depth image is segmented to obtain a third depth image. In this embodiment, the second mask is used to further segment the first interference data in the second depth image to obtain the third depth image.

[0155] In some embodiments, the step of determining the second mask based on the reference depth includes:

[0156] The range of the second mask is determined based on preset prior information and the average depth. The prior information is the maximum distance between the human hand and the palm. The maximum value within the range of the second mask is the sum of the average depth and the maximum distance, and the minimum value is the difference between the average depth and the maximum distance. In this embodiment, since user gestures are diverse (e.g., the thumb can point forward or backward), to avoid destroying the hand image when using the second mask to eliminate external environment images, the second mask can be set as a range. This range can be determined by preset prior information and the average depth. This range describes the depth range of the hand image. For example, the distance corresponding to the prior information can be 150mm, meaning the distance from the center of the human palm to any area of ​​the hand is less than 150mm.

[0157] In some embodiments, the step of segmenting the second depth image using the second mask to obtain a third depth image includes:

[0158] Compare the depth values ​​of the pixels in the second depth image with the second mask;

[0159] Depth values ​​in the second depth image that are not within the range of the second mask are removed to obtain a third depth image. For example, review again. Figure 20 It can be seen Figure 20 The third depth image in the image, with Figure 18 Compared to the second image, the first interference data is reduced again.

[0160] In this embodiment, the depth values ​​in the second depth image that are not within the range of the second mask correspond to interference data, so they are deleted to obtain the third depth image.

[0161] S400. Locate the hand region based on the third depth image. In this embodiment, the third depth image contains the least amount of first interference data, and the hand image therein is the depth image that is closest to the real hand. By using this third depth image, the hand region can be located more accurately.

[0162] In some embodiments, the step of locating the hand region based on the third depth image includes:

[0163] For the third depth image, multiple hand region localization methods are used to determine the corresponding undetermined hand regions. Various hand region localization methods in related technologies are used to localize the hand regions in the third depth image, resulting in multiple undetermined hand regions. For example, multiple convolutional neural network models are used to localize the hand regions in the third depth image to obtain the undetermined hand regions.

[0164] To avoid inaccurate hand region positioning and disruption of the hand's outline, the largest possible hand region is selected from all undefined hand regions and used as the positioned hand region. For example... Figure 21 As shown, Figure 21 An exemplary schematic diagram of a hand region according to some embodiments is shown.

[0165] The following is a brief description of the process of estimating 3D hand gesture pose information based on depth data using a deep convolutional neural network: First, the hand region localization method of this embodiment is executed on the first color image and the first depth image. Then, the localized hand region, for example, is... Figure 21 The hand region shown is input into the hand joint prediction network model, which predicts 21 hand joints. These hand joints are then visualized on a display device. For example, a virtual hand can be created using these hand joints and displayed on the display device.

[0166] This application embodiment also provides a display device, including:

[0167] A monitor is used to display the user interface.

[0168] User interface, used to receive input signals;

[0169] The controllers, connected to the display and user interface respectively, are used to perform:

[0170] Acquire a first color image and a corresponding first depth image, wherein the first color image is determined by the original color image and the first depth image is determined by the original depth image;

[0171] Using the pixel values ​​of the pixels in the first color image, the first depth image is initially segmented to obtain a second depth image, so that the second depth image reduces the first interference data relative to the first depth image. The first interference data indicates the external environment image corresponding to the non-hand image in the original depth image. The second depth image is then segmented a second time using the depth values ​​of its own pixels to obtain a third depth image, so that the third depth image reduces the first interference data relative to the second depth image. The hand region is located based on the third depth image.

[0172] In this embodiment of the application, a storage medium is provided that stores computer instructions, which, when executed by a processor, cause a computer device to perform the following:

[0173] Acquire a first color image and a corresponding first depth image, wherein the first color image is determined by the original color image and the first depth image is determined by the original depth image;

[0174] Using the pixel values ​​of the pixels in the first color image, the first depth image is initially segmented to obtain a second depth image, so that the second depth image reduces the first interference data relative to the first depth image. The first interference data indicates the external environment image corresponding to the non-hand image in the original depth image. The second depth image is then segmented a second time using the depth values ​​of its own pixels to obtain a third depth image, so that the third depth image reduces the first interference data relative to the second depth image. The hand region is located based on the third depth image.

[0175] In the above embodiments, a method and display device for locating a hand region are disclosed. The method first uses features of a first color image to initially segment a first depth image. Then, it uses the depth values ​​of the depth image itself to perform a second segmentation on the second depth image obtained from the initial segmentation. After two segmentations to reduce first interference data, a third depth image is obtained. Finally, the hand region determined based on the third depth image is located, improving the accuracy of gesture recognition. The method includes: acquiring a first color image and a corresponding first depth image; performing initial segmentation of the first depth image using the pixel values ​​of pixels in the first color image to obtain a second depth image; performing a second segmentation of the second depth image using the depth values ​​of its own pixels to obtain a third depth image; and locating the hand region based on the third depth image.

[0176] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some or all of the technical features therein. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of this application.

[0177] For ease of explanation, the above description has been provided in conjunction with specific embodiments. However, the above exemplary discussion is not intended to be exhaustive or to limit the embodiments to the specific forms disclosed above. Various modifications and variations can be obtained based on the above teachings. The selection and description of the above embodiments are for the purpose of better explaining the principles and practical applications, thereby enabling those skilled in the art to better utilize the described embodiments and various different variations of embodiments suitable for specific use considerations.

Claims

1. A display device, characterized in that, include: A monitor is used to display the user interface. User interface, used to receive input signals; The controllers, connected to the display and user interface respectively, are used to perform: Acquire a first color image and a corresponding first depth image, wherein the first color image is determined by the original color image and the first depth image is determined by the original depth image; The first depth image is filled with holes by performing image morphological transformation on the first depth image using a convolution kernel, and the filled first depth image is obtained by using Gaussian filtering. Using the HSV color values ​​of pixels in the completed first color image, the HSV color values ​​of the pixels are compared with a first mask to determine the pixel position of the pixel with the same HSV color value as the first mask. The pixels corresponding to the pixel positions in the first depth image are eliminated to obtain a second depth image. This reduces the first interference data in the second depth image relative to the first depth image. The first interference data indicates the external environment image corresponding to the non-hand image in the original depth image. The first mask is determined based on the second interference data, which indicates the external environment image corresponding to the non-hand image in the original color image. The image at the center of the second depth image is selected as the reference image; A reference depth is determined based on the depth values ​​of pixels in the reference image. The reference depth is used to determine a second mask. The range of the second mask is determined based on preset prior information and average depth, wherein the prior information is the maximum distance between the human hand and the palm. The second depth image is segmented using the second mask to obtain a third depth image, so that the third depth image reduces the first interference data relative to the second depth image; The hand region is located based on the third depth image.

2. The display device according to claim 1, characterized in that, The controller is configured to determine the first color image and the corresponding first depth image by performing the following steps: The original color image is cropped to obtain a first color image; cropping data is determined based on the first color image, wherein the cropping data is used to indicate a first positional relationship between the first color image and the original color image; the original depth image is cropped to obtain a first depth image, such that a second positional relationship between the first depth image and the original depth image is the same as the first positional relationship.

3. The display device according to claim 1, characterized in that, The controller is configured to perform the following actions: setting the center of the reference image to be the center of the second depth image; the length of the reference image being half the length of the second depth image; and the width of the reference image being half the width of the second depth image.

4. The display device according to claim 1, characterized in that, The controller is configured to perform the following steps to determine a reference depth based on the depth values ​​of pixels in the reference image: From the depth values ​​of pixels in the reference image, non-zero depth values ​​are selected, and an average depth is calculated for the non-zero depth values; the average depth is then used as the reference depth.

5. The display device according to claim 1, characterized in that, The minimum value in the range of the second mask is the difference between the average depth and the maximum distance.

6. The display device according to claim 1, characterized in that, The controller is configured to perform the following steps: segmenting the second depth image using the second mask to obtain a third depth image: Compare the depth values ​​of the pixels in the second depth image with the second mask; The depth values ​​in the second depth image that are not within the range of the second mask are eliminated to obtain the third depth image.

7. The display device according to claim 1, characterized in that, The controller is configured to perform the following steps to locate the hand region based on the third depth image: For the third depth image, multiple hand region localization methods are used to determine the corresponding undetermined hand regions; from all undetermined hand regions, the undetermined hand region with the largest area is selected as the localized hand region.

8. The display device according to claim 1, characterized in that, The controller is also configured to: The hand joints in the located hand area are visualized on the display, and a virtual hand is created using the hand joints and displayed on the display.

9. A method for locating a hand region, characterized in that, include: Acquire a first color image and a corresponding first depth image, wherein the first color image and the first depth image are determined by the original color image and the original depth image captured by the depth camera; The first depth image is filled with holes by performing image morphological transformation on the first depth image using a convolution kernel, and the filled first depth image is obtained by using Gaussian filtering. Using the HSV color values ​​of pixels in the completed first color image, the HSV color values ​​of the pixels are compared with a first mask to determine the pixel position of the pixel with the same HSV color value as the first mask. The pixels corresponding to the pixel positions in the first depth image are eliminated, and the first depth image is initially segmented to obtain a second depth image. This reduces the first interference data in the second depth image relative to the first depth image. The first interference data indicates the external environment image corresponding to the non-hand image in the original depth image. The first mask is determined based on the second interference data, which in turn indicates the external environment image corresponding to the non-hand image in the original color image. The image at the center of the second depth image is selected as the reference image; A reference depth is determined based on the depth values ​​of pixels in the reference image. The reference depth is used to determine a second mask. The range of the second mask is determined based on preset prior information and average depth, wherein the prior information is the maximum distance between the human hand and the palm. The second depth image is segmented using the second mask to obtain a third depth image, so that the third depth image reduces the first interference data relative to the second depth image; The hand region is located based on the third depth image.

Images