A method and display device for displaying a hand image

Abstract

This application discloses a method and device for displaying a hand image. The method uses the center point of the hand region as the root node and calculates the corresponding absolute depth value. Since the center point of the hand region is more stable than the wrist, the accuracy of determining the depth information of the hand image can be improved. The method includes: acquiring a first depth image, which is determined by an original depth image captured by an image acquisition device; determining a first parameter corresponding to a first hand region in the first depth image; using the first parameter to determine a detection box; acquiring the depth values ​​corresponding to all first pixels in the detection box that are in the same row and column as the center point of the first hand region; and determining the absolute depth value corresponding to the center point of the first hand region based on the depth values ​​corresponding to the first pixels, so as to control the display of the hand graphic based on the absolute depth value corresponding to the center point.

Description

Technical Field

[0001] This application relates to the technical field of gesture recognition, and more particularly to a method and device for displaying hand images. Background Technology

[0002] With the rapid development of artificial intelligence technology, 3D gesture recognition has become an important branch of human-computer interaction and a crucial technological field in virtual technology. 3D gesture recognition technology can be applied not only to games but also plays a vital role in emerging technology areas such as intelligent control, for example, controlling smart home devices through gestures.

[0003] In related technologies, the effectiveness of air keyboards largely reflects the performance of 3D gesture recognition algorithms. Specifically, a virtual keyboard is displayed on the user interface of a display device. Users can touch this virtual keyboard to simulate hand input on a physical keyboard. The user interface displays both the virtual keyboard and an image of the user's hand. However, it is frequently observed that when a user touches the virtual keyboard, the hand image may pass through the virtual keyboard without triggering a key press, or a key press may be triggered even when the hand image is still some distance from the virtual keyboard.

[0004] The above situations all differ from the effect of a physical hand pressing a physical keyboard, resulting in a poor user experience. Therefore, how to accurately determine the depth information of a hand image has become a problem that urgently needs to be solved by those skilled in the art. Summary of the Invention

[0005] This application provides a method and device for displaying a hand image. In this method, the center point of the hand region is used as the root node, and the corresponding absolute depth value is calculated. Since the center point of the hand region is more stable than the wrist, the accuracy of determining the depth information of the hand image can be improved.

[0006] In a first aspect, a display device is provided, comprising:

[0007] A monitor is used to display the user interface.

[0008] User interface, used to receive input signals;

[0009] The controllers, connected to the display and user interface respectively, are used to perform:

[0010] A first depth image is acquired, which is determined by the original depth image captured by the image acquisition device; a first parameter corresponding to the first hand region in the first depth image is determined, wherein the first parameter includes the coordinates of the center point of the first hand region, the width and height of the first hand region; a detection box is determined using the first parameter; the depth values ​​corresponding to all first pixels in the detection box that are in the same row and column as the center point of the first hand region are acquired; the absolute depth value corresponding to the center point of the first hand region is determined based on the depth values ​​corresponding to the first pixels, so as to display the hand image based on the absolute depth value corresponding to the center point.

[0011] In some embodiments, the controller is configured to determine a first depth image from the raw depth image captured by the image acquisition device by performing the following steps:

[0012] In the original depth image, second pixels whose depth values ​​are outside the preset depth value range are selected; the second pixels in the original depth image are removed to obtain the first depth image.

[0013] In some embodiments, the controller is configured to perform the following steps to determine a first parameter corresponding to a first hand region in the first depth image:

[0014] Acquire the original color image captured by the image acquisition device, wherein the pixels in the original color image and the original depth image correspond to each other; input the original color image into a preset convolutional neural network model to obtain the second parameter corresponding to the second hand region in the original color image, wherein the second parameter includes the coordinates of the center point of the second hand region, the width and height of the second hand region; set the second parameter as the first parameter.

[0015] In some embodiments, the controller is configured to determine a detection frame using the first parameter according to the following formula:

[0016] The detection frame includes a horizontal range (cols) and a vertical range (rows);

[0017]

[0018]

[0019] Wherein, the x-coordinate c of the center point of the first hand region x c x The ordinate c is an integer representing the y-coordinate of the center point of the first hand region. y c y The integer represents w, the width of the first hand region, and h, the height of the first hand region. The int function represents rounding down.

[0020] In some embodiments, the controller is configured to perform the following steps to determine the absolute depth value corresponding to the center point of the first hand region based on the depth value corresponding to the first pixel:

[0021] Take the first median value of the depth values ​​of all pixels in the same column as the center point of the first hand region, and use the first median value as the first depth value.

[0022] Take the second median value of the depth values ​​of all pixels in the same row as the center point of the first hand region from the first pixel point, and use the second median value as the second depth value;

[0023] Based on the first depth value and the second depth value, determine the absolute depth value corresponding to the center point of the first hand region.

[0024] In some embodiments, the controller is configured to calculate, based on the first depth value and the second depth value, an absolute depth value corresponding to the center point of the first hand region according to the following formula:

[0025]

[0026] Here, avgsd(cx, cy) represents the absolute depth value, and the int function represents rounding down. Represents the first depth value. This represents the second depth value.

[0027] Secondly, a method for displaying a hand image is provided, including:

[0028] Acquire a first depth image, which is determined by the original depth image captured by the image acquisition device;

[0029] A first parameter is determined corresponding to the first hand region in the first depth image, wherein the first parameter includes the coordinates of the center point of the first hand region, the width and height of the first hand region; a detection box is determined using the first parameter; the depth values ​​corresponding to all first pixels in the detection box that are in the same row and column as the center point of the first hand region are obtained; the absolute depth value corresponding to the center point of the first hand region is determined based on the depth values ​​corresponding to the first pixels, so as to display the hand image based on the absolute depth value corresponding to the center point.

[0030] In some embodiments, the step of determining the first depth image from the raw depth image captured by the image acquisition device includes:

[0031] In the original depth image, second pixels whose depth values ​​are outside the preset depth value range are selected; the second pixels in the original depth image are removed to obtain the first depth image.

[0032] In some embodiments, the method for determining the first parameter corresponding to the first hand region in the first depth image includes:

[0033] Acquire the original color image captured by the image acquisition device, wherein the pixels in the original color image and the original depth image correspond to each other; input the original color image into a preset convolutional neural network model to obtain the second parameter corresponding to the second hand region in the original color image, wherein the second parameter includes the coordinates of the center point of the second hand region, the width and height of the second hand region; set the second parameter as the first parameter.

[0034] In some embodiments, the detection box is determined using the first parameter according to the following formula:

[0035] The detection frame includes a horizontal range (cols) and a vertical range (rows);

[0036]

[0037]

[0038] Wherein, the x-coordinate c of the center point of the first hand region x c x The ordinate c is an integer representing the y-coordinate of the center point of the first hand region. y c y The integer represents w, the width of the first hand region, and h, the height of the first hand region. The int function represents rounding down.

[0039] In some embodiments, the step of determining the absolute depth value corresponding to the center point of the first hand region based on the depth value corresponding to the first pixel includes:

[0040] Take the first median value of the depth values ​​of all pixels in the same column as the center point of the first hand region, and use the first median value as the first depth value.

[0041] Take the second median value of the depth values ​​of all pixels in the same row as the center point of the first hand region from the first pixel point, and use the second median value as the second depth value;

[0042] Based on the first depth value and the second depth value, determine the absolute depth value corresponding to the center point of the first hand region.

[0043] In some embodiments, the controller is configured to calculate, based on the first depth value and the second depth value, an absolute depth value corresponding to the center point of the first hand region according to the following formula:

[0044]

[0045] Here, avgsd(cx, cy) represents the absolute depth value, and the int function represents rounding down. Represents the first depth value. This represents the second depth value.

[0046] In the above embodiments, a method and device for displaying a hand image are disclosed. In this method, the center point of the hand region is used as the root node, and the corresponding absolute depth value is calculated. Since the center point of the hand region is more stable than the wrist, the accuracy of determining the depth information of the hand image can be improved. The method includes: acquiring a first depth image, which is determined by an original depth image captured by an image acquisition device; determining a first parameter corresponding to a first hand region in the first depth image, wherein the first parameter includes the coordinates of the center point of the first hand region, the width of the first hand region, and the height of the first hand region; determining a detection box using the first parameter; acquiring the depth values ​​corresponding to all first pixels in the detection box that are in the same row and column as the center point of the first hand region; and determining the absolute depth value corresponding to the center point of the first hand region based on the depth values ​​corresponding to the first pixels, so as to display the hand image based on the absolute depth value corresponding to the center point. Attached Figure Description

[0047] Figure 1 An operational scenario between a display device and a control device according to some embodiments is illustrated;

[0048] Figure 2 A hardware configuration block diagram of a control device 100 according to some embodiments is shown;

[0049] Figure 3 A hardware configuration block diagram of a display device 200 according to some embodiments is shown;

[0050] Figure 4 A software configuration diagram of a display device 200 according to some embodiments is shown;

[0051] Figure 5 A schematic diagram of a user interface according to some embodiments is shown;

[0052] Figure 6 Another user interface diagram according to some embodiments is shown;

[0053] Figure 7An exemplary schematic diagram showing the locations of all the joints of a hand according to some embodiments is provided;

[0054] Figure 8 A flowchart of a method for displaying a hand image according to some embodiments is shown as an example;

[0055] Figure 9 An example of a user interaction diagram according to some embodiments is shown;

[0056] Figure 10 An exemplary schematic diagram of a hand region is shown according to some embodiments;

[0057] Figure 11 A schematic diagram of a first coordinate system according to some embodiments is shown as an example;

[0058] Figure 12 A schematic diagram of a second coordinate system according to some embodiments is shown as an example;

[0059] Figure 13 A schematic diagram of a check box according to some embodiments is shown as an example. Detailed Implementation

[0060] To make the objectives and implementation methods of this application clearer, the exemplary implementation methods of this application will be clearly and completely described below with reference to the accompanying drawings of the exemplary embodiments of this application. Obviously, the exemplary embodiments described are only some embodiments of this application, and not all embodiments.

[0061] It should be noted that the brief descriptions of terms in this application are only for the convenience of understanding the embodiments described below, and are not intended to limit the embodiments of this application. Unless otherwise stated, these terms should be understood in their ordinary and common meaning.

[0062] In this application, the terms "first," "second," "third," etc., used in the specification, claims, and accompanying drawings are used to distinguish similar or related objects or entities, and do not necessarily imply a specific order or sequence, unless otherwise specified. It should be understood that such terms are interchangeable where appropriate.

[0063] The terms “include” and “have”, and any variations thereof, are intended to cover but not exclusively include, for example, a product or device that includes a range of components is not necessarily limited to all of the components that are clearly listed, but may include other components that are not clearly listed or that are inherent to such product or device.

[0064] The display device provided in this application can have various implementation forms, such as a television, a smart television, a laser projection device, a monitor, an electronic bulletin board, an electronic table, etc. Figure 1 and Figure 2 This is one specific embodiment of the display device of this application.

[0065] Figure 1 This is a schematic diagram illustrating the operational scenario between the display device and the control unit according to the embodiment. Figure 1 As shown, the user can operate the display device 200 through the smart device 300 or the control device 100.

[0066] In some embodiments, the control device 100 may be a remote control. Communication between the remote control and the display device includes infrared protocol communication, Bluetooth protocol communication, and other short-range communication methods, controlling the display device 200 wirelessly or via wired means. Users can control the display device 200 by inputting user commands through buttons on the remote control, voice input, control panel input, etc.

[0067] In some embodiments, a smart device 300 (such as a mobile terminal, tablet computer, computer, laptop computer, etc.) may also be used to control the display device 200. For example, an application running on the smart device may be used to control the display device 200.

[0068] In some embodiments, the display device may receive instructions not through the aforementioned smart devices or control devices, but through touch or gestures.

[0069] In some embodiments, the display device 200 can also be controlled in ways other than the control device 100 and the smart device 300. For example, it can be controlled by directly receiving the user's voice commands through a module configured inside the display device 200 for acquiring voice commands, or it can be controlled by receiving the user's voice commands through a voice control device set outside the display device 200.

[0070] In some embodiments, the display device 200 also communicates with the server 400. The display device 200 may communicate via a local area network (LAN), wireless local area network (WLAN), and other networks. The server 400 may provide various content and interactive features to the display device 200. The server 400 may be a cluster or multiple clusters, and may include one or more types of servers.

[0071] Figure 2 An exemplary block diagram of the configuration of the control device 100 according to an exemplary embodiment is shown. Figure 2 As shown, the control device 100 includes a controller 110, a communication interface 130, a user input / output interface 140, a memory, and a power supply. The control device 100 can receive user input operation commands and convert the operation commands into commands that the display device 200 can recognize and respond to, thus acting as an intermediary for interaction between the user and the display device 200.

[0072] like Figure 3 The display device 200 includes at least one of the following: a tuner 210, a communicator 220, a detector 230, an external device interface 240, a controller 250, a display 260, an audio output interface 270, a memory, a power supply, and a user interface.

[0073] In some embodiments, the controller includes a processor, a video processor, an audio processor, a graphics processor, RAM, ROM, and a first interface to an nth interface for input / output.

[0074] The display 260 includes a display screen assembly for presenting images, a driving assembly for driving image display, a component for receiving image signals from the controller output, and a user control UI interface for displaying video content, image content, menu control interface, and user control UI interface.

[0075] The display 260 can be an LCD display, an OLED display, or a projection display, and can also be a projection device and a projection screen.

[0076] The communicator 220 is a component used to communicate with external devices or servers according to various communication protocol types. For example, the communicator may include at least one of the following: a Wi-Fi module, a Bluetooth module, a wired Ethernet module, other network communication protocol chips or near-field communication protocol chips, and an infrared receiver. The display device 200 can establish the transmission and reception of control signals and data signals with the external control device 100 or the server 400 through the communicator 220.

[0077] The user interface can be used to receive control signals from the control device 100 (such as an infrared remote control).

[0078] Detector 230 is used to collect signals from the external environment or to interact with the external environment. For example, detector 230 includes a light receiver, a sensor for collecting ambient light intensity; or, detector 230 includes an image acquisition device, such as a camera, which can be used to collect external environmental scenes, user attributes, or user interaction gestures; or, detector 230 includes a sound acquisition device, such as a microphone, for receiving external sounds.

[0079] The external device interface 240 may include, but is not limited to, one or more of the following: High Definition Multimedia Interface (HDMI), analog or high-definition component input interface (component), composite video input interface (CVBS), USB input interface (USB), RGB port, etc. It may also be a composite input / output interface formed by multiple interfaces mentioned above.

[0080] The tuner / demodulator 210 receives broadcast television signals via wired or wireless means, and demodulates audio and video signals, such as EPG data signals, from multiple wireless or wired broadcast television signals.

[0081] In some embodiments, the controller 250 and the tuner 210 may be located in different separate devices, that is, the tuner 210 may also be located in an external device of the main device where the controller 250 is located, such as an external set-top box.

[0082] The controller 250 controls the operation of the display device and responds to user operations through various software control programs stored in the memory. The controller 250 controls the overall operation of the display device 200. For example, in response to receiving a user command to select a UI object to display on the monitor 260, the controller 250 can execute operations related to the object selected by the user command.

[0083] In some embodiments, the controller includes at least one of a central processing unit (CPU), a video processor, an audio processor, a graphics processing unit (GPU), RAM (random access memory), ROM (read-only memory), a first to an nth interface for input / output, a communication bus, etc.

[0084] Users can input commands through a graphical user interface (GUI) displayed on the monitor 260, and the user input interface receives the user input commands through the GUI. Alternatively, users can input commands by entering specific sounds or gestures, and the user input interface receives the user input commands by recognizing the sounds or gestures through sensors.

[0085] A "user interface" is the medium through which an application or operating system interacts and exchanges information with the user. It converts information from its internal form to a form that the user can accept. A common form of user interface is the graphical user interface (GUI), which refers to a user interface related to computer operation displayed graphically. It can be an icon, window, control, or other interface element displayed on the screen of an electronic device. Controls can include visual interface elements such as icons, buttons, menus, tabs, text boxes, dialog boxes, status bars, navigation bars, and widgets.

[0086] See Figure 4 In some embodiments, the system is divided into four layers, from top to bottom: the Applications layer (referred to as the "Application Layer"), the Application Framework layer (referred to as the "Framework Layer"), the Android runtime and system library layer (referred to as the "System Runtime Library Layer"), and the kernel layer.

[0087] In some embodiments, at least one application runs in the application layer. These applications may be Windows programs, system settings programs, or clock programs that come with the operating system; they may also be applications developed by third-party developers. In specific implementations, the application packages in the application layer are not limited to the examples above.

[0088] The framework layer provides application programming interfaces (APIs) and a programming framework for applications. The application framework layer includes predefined functions. It acts as a central processing unit, determining the actions taken by applications within the application layer. Through the API, applications can access system resources and obtain system services during execution.

[0089] like Figure 4As shown, the application framework layer in this embodiment includes managers, content providers, etc., wherein the managers include at least one of the following modules: ActivityManager, which interacts with all activities running in the system; LocationManager, which provides access to system location services for system services or applications; PackageManager, which retrieves various information related to application packages currently installed on the device; NotificationManager, which controls the display and clearing of notification messages; and WindowManager, which manages icons, windows, toolbars, wallpapers, and desktop widgets on the user interface.

[0090] In some embodiments, the Activity Manager manages the lifecycle of individual applications and common navigation and back functions, such as controlling application exit, opening, and back actions. The Window Manager manages all window programs, such as obtaining the screen size, determining if a status bar is present, locking the screen, capturing the screen, and controlling display window changes (e.g., shrinking the display window, shaking the display, distorting the display, etc.).

[0091] In some embodiments, the system runtime library layer provides support for the upper layer, namely the framework layer. When the framework layer is used, the Android operating system runs the C / C++ libraries contained in the system runtime library layer to implement the functions that the framework layer needs to perform.

[0092] In some embodiments, the kernel layer is a layer between hardware and software. For example... Figure 4 As shown, the kernel layer includes at least one of the following drivers: audio driver, display driver, Bluetooth driver, camera driver, WIFI driver, USB driver, HDMI driver, sensor driver (such as fingerprint sensor, temperature sensor, pressure sensor, etc.), and power driver.

[0093] With the rapid development of artificial intelligence technology, 3D gesture recognition has become an important branch of human-computer interaction and a crucial technological field in virtual technology. 3D gesture recognition technology can be applied not only to games but also plays a vital role in emerging technology areas such as intelligent control, for example, controlling smart home devices through gestures.

[0094] In related technologies, the effectiveness of air keyboards largely reflects the performance of 3D gesture recognition algorithms. Specifically, a virtual keyboard is displayed on the user interface of the display device. Users can touch the virtual keyboard to simulate the input operations of their hands on a physical keyboard. At this time, the user interface displays both the virtual keyboard and an image of the user's hand, which is the corresponding image of the physical hand displayed on the user interface.

[0095] However, it is frequently observed that when users touch the virtual keyboard with their hands, the hand image on the user interface may cross the virtual keyboard without triggering any key presses. For example, as shown in... Figure 5 As shown, Figure 5 An exemplary schematic diagram of a user interface according to some embodiments is shown, in which a hand image displayed on the user interface has passed through a virtual keyboard, but no key press signal is generated. Alternatively, a key press signal is generated even when the hand image displayed on the user interface is still some distance from the virtual keyboard, for example, as shown in... Figure 6 As shown, Figure 6 An exemplary view of another user interface according to some embodiments is shown, in which a hand image is displayed at a considerable distance from the virtual keyboard, yet key presses are already triggered. These situations differ from the effect of a physical hand pressing a physical keyboard, resulting in a poor user experience.

[0096] Therefore, how to make the display position of the hand image on the user interface more accurate has become a problem that urgently needs to be solved by those skilled in the art.

[0097] Research has revealed that the aforementioned problem may be due to low accuracy in gesture recognition. In some embodiments, the hand image displayed on the user interface is determined based on the recognized gesture.

[0098] The process of recognizing gestures includes determining the absolute depth value of each hand joint. Using the absolute depth value of each hand joint, in some embodiments, when the user's hand is fully extended, without considering the six degrees of freedom of the entire hand relative to space, the human hand has 21 degrees of freedom. In this document, this can be understood as meaning there are a total of 21 hand joint positions, including the fingertips, joints, and the base of the palm. For example... Figure 7 As shown, Figure 7 An exemplary schematic diagram showing the locations of all the hand joints according to some embodiments is provided.

[0099] In some embodiments, the process of determining the pose representation of the hand joints includes: First, the pose representation of the hand joints adopts a 2.5D method, that is, the 2D position of the hand joints and the relative depth value (Z) of each hand joint relative to the root node. root n The relative depth refers to the depth difference between the hand joint and the root node. 2D position can be represented using coordinates (U, V), and the hand joint pose represented in 2.5D is ultimately expressed as (U... n V n , Z root n (where n represents the sequence number of the hand joint). It should be noted that the sequence numbers of the hand joints can be sorted as needed, and this application does not impose any restrictions.

[0100] If the absolute depth value (Z) of the root node is determined root It can be obtained through formula Z. n =Z root +Z root n Calculate the absolute depth value (Z) of each hand joint. n This transforms the hand joint positions represented in 2.5D into hand joint positions in the camera coordinate system (U). n V n , Z n Where n represents the index of the hand joint, the hand joint position in the camera coordinate system (U) is then calculated. n V n , Z n The position information (Xn, Yn, Zn) in the world coordinate system is converted into position information (Xn, Yn, Zn). The position information (Xn, Yn, Zn) in the world coordinate system is used as the pose of the hand joint, where Zn represents the absolute depth value of the hand joint.

[0101] Generally, the wrist is used as the root node among the hand joints. The position information of the wrist is used to determine the absolute depth of other hand joints. However, the wrist is prone to occlusion, which leads to inaccurate calculation of the absolute depth of other hand joints, affecting the accuracy of gesture recognition and ultimately reducing the accuracy of the hand image displayed on the user interface.

[0102] To address the aforementioned technical problems, this application provides a method for displaying hand images. In this method, the center point of the hand region is used as the root node, and the corresponding absolute depth value is calculated. Since the center point of the hand region is more stable than the wrist, the accuracy of gesture recognition can be improved, thereby improving the accuracy of the hand image display on the user interface.

[0103] It should be noted that the method in this application embodiment can be applied to display devices, as well as computers, VR glasses, AR glasses, and other terminals that can be equipped with depth cameras.

[0104] like Figure 8 As shown, Figure 8 A flowchart of a method for displaying a hand image according to some embodiments is illustrated. The method includes:

[0105] S100. Acquire a first depth image, which is determined by the original depth image captured by the image acquisition device. In this embodiment, the image acquisition device can be a depth camera, which can be built into the display device or an external device connected to the display device. For example, Figure 1 As shown, a depth camera 231 is installed on the display device 200.

[0106] In one example, the preset application is launched, and the process begins acquiring the raw depth image captured by the depth camera, then executing the method for displaying the hand image. For example... Figure 9 As shown, Figure 9 An exemplary diagram of a user interaction according to some embodiments is shown. Figure 9 A user interface including an air keyboard application control is shown, on which the air keyboard application control is displayed. In some embodiments, other controls may also be displayed when the air keyboard application control is displayed on the display device; this is not limited to these embodiments. The user can move the focus to the air keyboard application control using a control device and press the confirmation key on the control device, causing the user interface to jump to a page such as... Figure 9 As shown in (b), Figure 9 Figure (b) shows a user interface that includes an air keyboard, on which a user can press keys using their hands. For example, a user can touch the "space" key on the air keyboard.

[0107] It should be noted that in this embodiment, moving the focus to a control can thicken the control's border. Alternatively, other methods can be used to indicate that the control is selected. For example, when the focus moves to a control, the control can change its shape, such as from a square to a circle. Alternatively, when a control is selected, it can be enlarged according to a preset ratio. For instance, if the display area of ​​video controls on the user interface is the same, when a control is selected, its display area increases by 1.2 times compared to the original control's display area. Since this application does not restrict the shape of the control when the focus is on it, other forms that facilitate user identification of selected video controls are acceptable.

[0108] The method described in this application can also be applied to fields such as gesture-based touchless teaching. Gesture-based touchless teaching can be understood as users writing text and drawing on a virtual blackboard using their hands. Similar to an air keyboard, the virtual blackboard requires the hand to be kept at an appropriate distance when writing or drawing on it. If the hand image passes through the blackboard, or if there is a significant distance between the hand image and the blackboard, text or drawings will appear on the virtual blackboard, resulting in a poor user experience.

[0109] It should be noted that the method in this application embodiment can be applied not only in scenarios where virtual objects exist, but also in scenarios where no virtual objects exist in order to ensure the accuracy of the hand image display.

[0110] In some embodiments, the step of determining a first depth image from an original depth image captured by an image acquisition device includes: filtering out second pixels whose depth values ​​are outside a preset depth value range in the original depth image; removing the second pixels from the original depth image to obtain the first depth image.

[0111] In this embodiment, each pixel in the depth image corresponds to a depth value, which represents the distance between the object and the image sensor. It is understood that to ensure a clear display of the hand image on the user interface, the user needs to position their hand within a certain range of the image sensor. This preset depth value range is a pre-defined depth value determined based on the distance between the hand and the image sensor.

[0112] In some embodiments, when a user places their hand within the field of view of a depth camera, the depth camera captures images of the hand and its surrounding environment. It is understood that the depth camera inevitably captures the external environment, resulting in a depth image containing not only the hand image but also the external environment image. For example, since user 3D gesture interaction scenarios are primarily indoors, the raw depth image captured by the depth camera includes images of the surrounding environment, i.e., the external environment image, in addition to the hand area.

[0113] It is understandable that the distance between the external environment and the hand and the depth camera are different. Therefore, this embodiment uses depth values ​​to eliminate the second pixel corresponding to the external environment. If the second pixel is not processed, the first depth image will include not only the pixel corresponding to the hand image but also the second pixel, affecting the accuracy of the absolute depth value calculation in subsequent steps. In some embodiments, the elimination of the second pixel can be achieved by setting the corresponding depth value to 0.

[0114] In some embodiments, the preset distance range can be from 20cm to 150cm. Second pixels corresponding to objects within 20cm and greater than 150cm of the depth camera are removed.

[0115] S200. Determine the first parameter corresponding to the first hand region in the first depth image, wherein the first parameter includes the coordinates of the center point of the first hand region, the width and height of the first hand region. It should be noted that the hand region mentioned herein refers to a rectangular image including the hand image. For example, the most accurate hand region corresponding to the image refers to a rectangular image drawn to fit the hand contour, such as... Figure 10 As shown, Figure 10 An exemplary schematic diagram of a hand region according to some embodiments is shown. Figure 10 The four sides of the hand region 500 described above respectively conform to the contour of the hand, at which point the hand region 500 is the optimal hand region in the image. The first hand region in the first depth image is the hand region determined in the depth image.

[0116] In some embodiments, the top-left corner of the first depth image is used as the origin, and a first coordinate system is set with the horizontal direction to the right as the positive x-axis and the vertical direction downwards as the positive y-axis. For example, ... Figure 11 As shown, Figure 11 A schematic diagram of a first coordinate system according to some embodiments is illustrated. The coordinates corresponding to the center point 601 of the first hand region 600 can be determined by this first coordinate system. Figure 11 The position of the first hand region in the first coordinate system is determined using the coordinates of the center point of the first hand region, as well as the width W and height H of the first hand region. It should be noted that, due to the complexity of the content in the first depth image, all images except the depth image of the first hand region are represented by dashed lines.

[0117] In some embodiments, color images are more convenient for locating hand regions compared to depth images. This application embodiment utilizes a color image to determine first hand region parameters. The steps for determining the first hand region parameters of the first depth image include:

[0118] The image acquisition device captures a raw color image, wherein the pixels in the raw color image and the raw depth image correspond to each other. For example, the raw color image has m*n pixels, and the raw depth image also has m*n pixels. The raw color image can be understood as an image with color; images captured by a traditional camera are color images.

[0119] Using the original color image, a second parameter corresponding to the second hand region in the original color image is determined. This second parameter is then set as the first parameter. The second parameter includes the coordinates of the center point of the second hand region, and the width and height of the second hand region.

[0120] This application does not limit the method of locating the second hand region in the original color image. In some embodiments, the method for determining the second parameter includes: inputting the original color image into a preset convolutional neural network model to obtain the second parameter corresponding to the second hand region; wherein the preset convolutional neural network model is obtained through learning from a large number of samples, and the samples include several original color images and corresponding second parameters.

[0121] In some embodiments, the top-left corner of the original color image is used as the origin, and a second coordinate system is set with the horizontal direction to the right as the positive x-axis and the vertical direction downwards as the positive y-axis. For example, such as... Figure 12 As shown, Figure 12 A schematic diagram of a second coordinate system according to some embodiments is illustrated. Figure 12 The position of the second hand region in the second coordinate system is determined using the coordinates of the center point 701 of the second hand region 700, as well as the width W′ and height H′ of the second hand region. At this time, the second parameter includes the coordinates of the center point of the second hand region in the second coordinate system, the width W′ of the second hand region, and the height H′ of the second hand region. The coordinates of the center point of the second hand region in the second coordinate system are set as the coordinates of the center point of the first hand region, the width W′ of the second hand region is set as the width W of the first hand region, and the height H of the second hand region is set as the height H′ of the first hand region.

[0122] S300. Using the first parameter, determine the detection box. In this embodiment, the detection box has an odd number of pixels in both the horizontal and vertical directions. In some embodiments, the detection box includes a horizontal range cols and a vertical range rows, and the detection box contains cols * rows of pixels;

[0123]

[0124]

[0125] Wherein, the x-coordinate c of the center point of the first hand region x c x The ordinate c is an integer representing the y-coordinate of the center point of the first hand region. y c y The integer represents w, the width of the first hand region, and h, the height of the first hand region. The int function represents rounding down.

[0126] For example, the int function represents rounding down. For instance, the value 5.6 will be rounded down to 5 after being calculated using the int function.

[0127] It should be noted that the detection box may be the same as or different from the first hand region. Using the horizontal and vertical ranges determined by the above formula, it can be guaranteed that there are an odd number of pixels horizontally and an odd number of pixels vertically within the detection box. This is because in subsequent steps, it is necessary to find the median value associated with the detection box within its range.

[0128] For example, the center point coordinates (c) of the first hand region x c y The coordinates of the first hand region are (15, 20). The width W of the first hand region is 20 pixels, the height H is 30 pixels, the horizontal range cols is [15-int(20 / 2), 15+int(20 / 2)], i.e. [5, 25], and the vertical range rows is [20-int(30 / 2), 20+int(30 / 2)], i.e. [5, 35]. For example, the center point coordinates (c...) of the first hand region... x c y The first hand region has a width W of 21 pixels and a height H of 31 pixels. Its horizontal range (cols) is [15 - int(21 / 2), 15 + int(21 / 2)], which is [5, 25]. Its vertical range (rows) is [20 - int(31 / 2), 20 + int(31 / 2)], which is [5, 35]. It should be noted that the horizontal or vertical range includes not only the two endpoints but also the integer values ​​between them. For example, [5, 25] includes both 5 and 25, and also all integer values ​​between 5 and 25.

[0129] S400. Obtain the depth values ​​of all first pixels in the detection frame that are in the same row and column as the center point of the first hand region.

[0130] In this embodiment of the application, among all pixels in the detection frame, all pixels that are in the same row as the center point of the first hand region are searched. For example... Figure 13 The above, Figure 13 An exemplary diagram of a check box according to some embodiments is shown. Figure 13 The center point 601 is marked with all pixels 801 in the same column as the center point 601, and the center point 602 is marked with all pixels 802 in the same row as the center point 602. For example, the center point coordinates (c...) of the first hand region... x c yIf the coordinates of the first hand region are (15, 20) and the vertical range of row is [5, 35], then the coordinates of all pixels in the same row as the center point of the first hand region include (15, 5), (15, 6)...(15, 35), for a total of 31 pixels.

[0131] In this embodiment of the application, among all pixels in the detection frame, all pixels that are in the same column as the center point of the first hand region are searched. For example, the coordinates of the center point of the first hand region are (c x c y If the coordinates of the first hand region are (15, 20) and the horizontal range of cols is [5, 25], then the coordinates of all pixels in the same row as the center point of the first hand region include (5, 20), (6, 20)...(25, 20), for a total of 21 pixels.

[0132] S500: Based on the depth value corresponding to the first pixel, determine the absolute depth value corresponding to the center point of the first hand region, and display the hand image based on the absolute depth value corresponding to the center point. In this embodiment, the median filtering idea is adopted. Using the center coordinates of the first hand region as the base point, horizontal and vertical coordinate lines are drawn in the detection box, and the depth value of each pixel is obtained. Median filtering is used to calculate the corresponding median row depth (cd), column depth (rd), and average square root (avgsd) depth. The average square root is used as the absolute depth value, which makes the absolute depth value of the center point more accurate. It should be noted that median filtering is very effective in smoothing impulse noise and can also protect sharp edges of the image by selecting appropriate points to replace the values ​​of contaminated points. In the method of this embodiment, the center point of the hand region is used as the root node, and the corresponding absolute depth value is calculated. Since the center point of the hand region is more stable than the wrist, the accuracy of gesture recognition can be improved, thereby improving the accuracy of the hand image display on the user interface.

[0133] In some embodiments, the step of determining the absolute depth value corresponding to the center point of the first hand region based on the depth value corresponding to the first pixel includes:

[0134] The first median of the depth values ​​corresponding to all pixels in the same column as the center point of the first hand region is taken as the first depth value. In this embodiment, the depth values ​​corresponding to all pixels in the same column as the center point of the first hand region are arranged from largest to smallest or from smallest to largest, and the depth value in the middle of the sorted list is taken as the first median. In this embodiment, since the number of pixels in the same column as the center point of the first hand region is an odd number, a unique depth value in the middle can be determined. For example, the depth values ​​corresponding to all pixels in the same column include 2, 3, and 4, and 3 is determined as the first median.

[0135] The second median of the depth values ​​corresponding to all pixels in the same row as the center point of the first hand region is taken as the second depth value. In this embodiment, the depth values ​​corresponding to all pixels in the same row as the center point of the first hand region are arranged from largest to smallest or smallest to largest, and the depth value in the middle of the sorted list is taken as the second median. In this embodiment, since the number of pixels in the same column as the center point of the first hand region is an odd number, a unique depth value in the middle can be determined. For example, the depth values ​​corresponding to all pixels in the same row include 19, 30, and 35, and 30 is determined as the second median.

[0136] Based on the first depth value and the second depth value, determine the absolute depth value corresponding to the center point of the first hand region.

[0137] In some embodiments, the absolute depth value corresponding to the center point of the first hand region is calculated based on the first depth value and the second depth value according to the following formula:

[0138]

[0139] Among them, avgsd(c x c y () represents the absolute depth value, and the int function represents rounding down. Represents the first depth value. This represents the second depth value.

[0140] In this embodiment, the center point of the hand region is taken as the root node, and the corresponding absolute depth value is calculated using median filtering. The pixels in the local region are sorted according to their depth values, and the median of the depth values ​​in the region is taken as the gray value of the current pixel. This effectively removes noise, resulting in much lower blurring compared to small-sized linear smoothing filters, and is also very effective in handling impulse noise.

[0141] Adding this median filtering scheme to the 3D gesture interaction technology framework can effectively avoid false detection and over-detection of hand. Over-detection refers to the detection of hand area data that is not needed in actual application. At the same time, it makes the application of 3D gesture interaction technology more stable and the actual interaction process more natural.

[0142] In this embodiment of the application, after using the method, the gesture including the absolute depth value of the hand joint can be accurately determined, and the hand image can be accurately displayed on the user interface.

[0143] In the method of this application embodiment, the center point of the hand region is used as the root node, and the corresponding absolute depth value is calculated. Since the center point of the hand region is more stable than the wrist, the accuracy of gesture recognition can be improved, thereby improving the accuracy of the hand image display on the user interface.

[0144] In this embodiment of the application, a display device is provided, including:

[0145] A monitor is used to display the user interface.

[0146] User interface, used to receive input signals;

[0147] The controllers, connected to the display and user interface respectively, are used to perform:

[0148] A first depth image is acquired, which is determined by the original depth image captured by the image acquisition device; a first parameter corresponding to the first hand region in the first depth image is determined, wherein the first parameter includes the coordinates of the center point of the first hand region, the width and height of the first hand region; a detection box is determined using the first parameter; the depth values ​​corresponding to all first pixels in the detection box that are in the same row and column as the center point of the first hand region are acquired; the absolute depth value corresponding to the center point of the first hand region is determined based on the depth values ​​corresponding to the first pixels, so as to display the hand image based on the absolute depth value corresponding to the center point.

[0149] In this embodiment of the application, a storage medium is provided that stores computer instructions, which, when executed by a processor, cause a computer device to perform the following:

[0150] A first depth image is acquired, which is determined by the original depth image captured by the image acquisition device; a first parameter corresponding to the first hand region in the first depth image is determined, wherein the first parameter includes the coordinates of the center point of the first hand region, the width and height of the first hand region; a detection box is determined using the first parameter; the depth values ​​corresponding to all first pixels in the detection box that are in the same row and column as the center point of the first hand region are acquired; the absolute depth value corresponding to the center point of the first hand region is determined based on the depth values ​​corresponding to the first pixels, so as to display the hand image based on the absolute depth value corresponding to the center point.

[0151] In the above embodiments, a method and device for displaying a hand image are disclosed. The method uses the center point of the hand region as the root node and calculates the corresponding absolute depth value. Since the center point of the hand region is more stable than the wrist, the accuracy of determining the depth information of the hand image can be improved. The method includes: acquiring a first depth image, which is determined by an original depth image captured by an image acquisition device; determining a first parameter corresponding to a first hand region in the first depth image, wherein the first parameter includes the coordinates of the center point of the first hand region, the width of the first hand region, and the height of the first hand region; determining a detection box using the first parameter; acquiring the depth values ​​corresponding to all first pixels in the detection box that are in the same row and column as the center point of the first hand region; and determining the absolute depth value corresponding to the center point of the first hand region based on the depth values ​​corresponding to the first pixels, so as to display the hand image based on the absolute depth value corresponding to the center point.

[0152] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some or all of the technical features therein. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of this application.

[0153] For ease of explanation, the above description has been provided in conjunction with specific embodiments. However, the above exemplary discussion is not intended to be exhaustive or to limit the embodiments to the specific forms disclosed above. Various modifications and variations can be obtained based on the above teachings. The selection and description of the above embodiments are for the purpose of better explaining the principles and practical applications, thereby enabling those skilled in the art to better utilize the described embodiments and various different variations of embodiments suitable for specific use considerations.

Claims

1. A display device, characterized in that, include: A monitor is used to display the user interface. User interface, used to receive input signals; The controllers, connected to the display and user interface respectively, are used to perform: A first depth image is acquired, which is determined by the original depth image captured by the image acquisition device; a first parameter corresponding to the first hand region in the first depth image is determined, wherein the first parameter includes the coordinates of the center point of the first hand region, the width and height of the first hand region; a detection box is determined using the first parameter; the depth values ​​corresponding to all first pixels in the detection box that are in the same row and column as the center point of the first hand region are acquired; the absolute depth value corresponding to the center point of the first hand region is determined based on the depth values ​​corresponding to the first pixels, so as to display the hand image based on the absolute depth value corresponding to the center point.

2. The display device according to claim 1, characterized in that, The controller is configured to determine a first depth image from the raw depth image captured by the image acquisition device by performing the following steps: In the original depth image, second pixels whose depth values ​​are outside the preset depth value range are selected; the second pixels in the original depth image are removed to obtain the first depth image.

3. The display device according to claim 1, characterized in that, The controller is configured to perform the following steps to determine a first parameter corresponding to a first hand region in the first depth image: Acquire the original color image captured by the image acquisition device, wherein the pixels in the original color image and the original depth image correspond to each other; input the original color image into a preset convolutional neural network model to obtain the second parameter corresponding to the second hand region in the original color image, wherein the second parameter includes the coordinates of the center point of the second hand region, the width and height of the second hand region; set the second parameter as the first parameter.

4. The display device according to claim 1, characterized in that, The controller is configured to determine the detection frame using the first parameter according to the following formula: The detection frame includes a horizontal range (cols) and a vertical range (rows); Wherein, the x-coordinate c of the center point of the first hand region x c x The ordinate c is an integer representing the y-coordinate of the center point of the first hand region. y c y The integer represents w, the width of the first hand region, and h, the height of the first hand region. The int function represents rounding down.

5. The display device according to claim 1, characterized in that, The controller is configured to perform the following steps to determine the absolute depth value corresponding to the center point of the first hand region based on the depth value corresponding to the first pixel: Take the first median value of the depth values ​​of all pixels in the same column as the center point of the first hand region, and use the first median value as the first depth value. Take the second median value of the depth values ​​of all pixels in the first pixel that are in the same row as the center point of the first hand region, and use the second median value as the second depth value. Based on the first depth value and the second depth value, determine the absolute depth value corresponding to the center point of the first hand region.

6. The display device according to claim 5, characterized in that, The controller is configured to calculate, based on the first depth value and the second depth value, the absolute depth value corresponding to the center point of the first hand region according to the following formula: Among them, avgsd(c x c y () represents the absolute depth value, and the int function represents rounding down. Represents the first depth value. This represents the second depth value.

7. A method for displaying a hand image, characterized in that, include: Acquire a first depth image, which is determined by the original depth image captured by the image acquisition device; A first parameter is determined corresponding to the first hand region in the first depth image, wherein the first parameter includes the coordinates of the center point of the first hand region, the width and height of the first hand region; a detection box is determined using the first parameter; the depth values ​​corresponding to all first pixels in the detection box that are in the same row and column as the center point of the first hand region are obtained; the absolute depth value corresponding to the center point of the first hand region is determined based on the depth values ​​corresponding to the first pixels, so as to display the hand image based on the absolute depth value corresponding to the center point.

8. The method according to claim 7, characterized in that, The steps for determining the first depth image from the raw depth image captured by the image acquisition device include: In the original depth image, second pixels whose depth values ​​are outside the preset depth value range are selected; the second pixels in the original depth image are removed to obtain the first depth image.

9. The method according to claim 7, characterized in that, The method for determining the first parameter corresponding to the first hand region in the first depth image includes: Acquire the original color image captured by the image acquisition device, wherein the pixels in the original color image and the original depth image correspond to each other; input the original color image into a preset convolutional neural network model to obtain the second parameter corresponding to the second hand region in the original color image, wherein the second parameter includes the coordinates of the center point of the second hand region, the width and height of the second hand region; set the second parameter as the first parameter.

10. The method according to claim 7, characterized in that, The step of determining the absolute depth value corresponding to the center point of the first hand region based on the depth value corresponding to the first pixel includes: Take the first median value of the depth values ​​of all pixels in the same column as the center point of the first hand region, and use the first median value as the first depth value. Take the second median value of the depth values ​​of all pixels in the same row as the center point of the first hand region from the first pixel point, and use the second median value as the second depth value; Based on the first depth value and the second depth value, determine the absolute depth value corresponding to the center point of the first hand region.

Images