A method of positioning a hand region and a display apparatus

Abstract

The application discloses a method for positioning a hand region and a display device. A preset convolutional neural network model used in the method is optimized by using a loss function determined by a hand joint point, and joint point information is added to assist in positioning the hand region, so that the accuracy of the hand region output by the convolutional neural network model is improved. The method comprises the following steps: when an instruction for positioning the hand region is received, an image captured by an image acquisition device is input into a preset convolutional neural network model to determine a hand region, wherein the hand region is a rectangular region formed by an outer edge of the hand, and the preset convolutional neural network model is optimized by using a loss function determined by a hand joint point.

Description

Technical Field

[0001] This application relates to the technical field of locating a hand area, and more particularly to a method and display device for locating a hand area. Background Technology

[0002] Currently, 2D and 3D gesture technologies are developing rapidly, expanding the functionality of display devices. For example, display devices can implement special effects based on human hands, and can also interact with users through gestures.

[0003] In achieving the aforementioned functions, the localization of the human hand area is involved. However, due to the high degree of freedom of the fingers—they can move in multiple directions and angles—the detection of the hand area is often inaccurate, affecting the effectiveness of the functions. For example... Figure 5 The user interface shown represents the ideal effect of the special effect, where the user's index fingertip points upwards, and a rotating virtual basketball appears on the fingertip. However, due to inaccurate hand area positioning, the hand area is either too large or too small, resulting in an actual effect that appears as shown below. Figure 6 As shown and as Figure 7 The situation is shown below. Specifically, in... Figure 6 The virtual basketball in the image is still some distance from the index finger. Figure 7 In the case of a virtual basketball partially obscuring the index finger, the special effects cannot be displayed well in either situation, affecting the user experience. Furthermore, when users interact with the display device via gestures, hand area detection is a prerequisite for hand posture estimation during interaction. If the hand area cannot be accurately located, the interaction effect with the display device through gestures will be affected.

[0004] Therefore, how to improve the accuracy of locating the hand area has become an urgent problem to be solved by those skilled in the art. Summary of the Invention

[0005] Some embodiments of this application provide a method and display device for locating a hand region. The method uses a preset convolutional neural network model that is optimized by utilizing a loss function determined by hand joints, which can improve the accuracy of the hand region output by the convolutional neural network model.

[0006] In a first aspect, a display device is provided, comprising:

[0007] A monitor is used to display the user interface.

[0008] User interface, used to receive input signals;

[0009] The controllers, connected to the display and user interface respectively, are used to perform:

[0010] When receiving the instruction of locating the hand region, the image captured by the image acquisition device is input into the preset convolutional neural network model to determine the hand region, wherein the hand region is a rectangular region formed by the outer edge of the hand, and the preset convolutional neural network model is obtained by optimizing the loss function determined by the hand joint.

[0011] In some embodiments, the controller is configured to obtain the preset convolutional neural network model by optimizing the loss function determined by the hand joint according to the following steps:

[0012] Obtain a training set, wherein the training set includes a plurality of sample images and true information corresponding to the sample images, and the true information includes hand true classification, hand joint true position and hand true region frame;

[0013] Preprocess each of the sample images; input all the preprocessed sample images into an initial convolutional neural network model in sequence to output corresponding prediction information, and the prediction information includes hand prediction classification, hand joint prediction position and hand prediction region frame;

[0014] Input the true information and the prediction information into a loss function to obtain a loss value, wherein the loss function includes a detection frame regression loss function, a detection classification loss function, a hand joint detection regression loss function and a hand joint detection penalty term, the joint detection penalty term is determined according to a hand prediction region frame and a joint bounding rectangle, and the joint bounding rectangle is the largest rectangle surrounded by the hand joint true position;

[0015] Adjust the initial convolutional neural network model by using the loss value to determine an adjusted initial convolutional neural network model as the preset convolutional neural network model.

[0016] In some embodiments, the controller is configured to determine the joint detection penalty term (1-IOLBR) according to the hand prediction region frame and the joint bounding rectangle according to the following formula:

[0017]

[0018] wherein, Area lbr is the area where the joint bounding rectangle is located, Area pre is the area where the hand prediction region frame is located; and Intersection(Area pre , Area lbr ) is the overlapping area of the area where the hand region detection frame is located and the area where the joint bounding rectangle is located.

[0019] In some embodiments, the controller is configured to perform the preprocessing on each sample image according to the following steps:

[0020] Subtract the revised value from the first RGB value of all pixels in the sample image to obtain a second RGB value;

[0021] The sample image with the second RGB value is taken as the preprocessed sample image.

[0022] In some embodiments, the formula of the loss function is:

[0023] loss total =loss box +loss class +alpha*loss landmark +beta*(1-ILOBR);

[0024] Wherein, loss total is a loss function, loss box is a bounding box regression loss function, loss class is a detection classification loss function, loss landmark is a hand joint point detection regression loss function, 1-ILOBR is a hand joint point detection penalty term, alpha is the weight of the hand joint point detection regression loss function, and beta is the weight of the hand joint point detection penalty term.

[0025] In a second aspect, a method for positioning a hand region is provided, comprising:

[0026] When an instruction for positioning a hand region is received, an image captured by an image acquisition device is used to determine a hand region by using a preset convolutional neural network model, wherein the hand region is a rectangular region formed by the outer edge of the hand, and the preset convolutional neural network model is obtained by optimizing a loss function for hand joint point determination.

[0027] In some embodiments, the step of obtaining the preset convolutional neural network model by optimizing the loss function for hand joint point determination comprises:

[0028] Obtaining a training set, wherein the training set comprises a plurality of sample images and corresponding true information, and the true information comprises hand true classification, hand joint point true position, and hand true region bounding box;

[0029] Preprocessing each sample image; and sequentially inputting all preprocessed sample images into an initial convolutional neural network model to output corresponding prediction information, wherein the prediction information comprises hand prediction classification, hand joint point prediction position, and hand prediction region bounding box.

[0030] inputting the real information and the predicted information into a loss function to obtain a loss value, wherein the loss function comprises a bounding box regression loss function, a detection classification loss function, a hand joint point detection regression loss function, and a hand joint point detection penalty term, the joint point detection penalty term being determined according to a hand prediction region bounding box and a joint point bounding rectangle, the joint point bounding rectangle being a maximum rectangle surrounded by a real position of a hand joint point;

[0031] adjusting the initial convolutional neural network model by using the loss value, and determining an adjusted initial convolutional neural network model as the preset convolutional neural network model.

[0032] In some embodiments, a calculation formula of the joint point detection penalty term (1-IOLBR) is determined according to the hand prediction region bounding box and the joint point bounding rectangle as follows:

[0033]

[0034] wherein, Area lbr is an area where the joint point bounding rectangle is located, Area pre is an area where the hand prediction region bounding box is located, and Intersection(Area pre , Area lbr ) is an overlapping area of the area where the hand region detection bounding box is located and the area where the joint point bounding rectangle is located.

[0035] In some embodiments, the step of pre-processing the sample image comprises:

[0036] subtracting a revision value from the first RGB value of all pixel points in the sample image to obtain a second RGB value;

[0037] taking the sample image with the second RGB value as the pre-processed sample image.

[0038] In some embodiments, a formula of the loss function is as follows:

[0039] loss total =loss box +loss class +alpha*loss landmark +beta*(1-ILOBR);

[0040] wherein, loss total is the loss function, loss box is the bounding box regression loss function, loss class is the detection classification loss function, and loss landmarkThe hand joint point detection regression loss function is 1-ILOBR, alpha is the weight of the hand joint point detection regression loss function, and beta is the weight of the hand joint point detection penalty term.

[0041] In a third aspect, a storage medium is provided, and the storage medium has computer instructions stored thereon. When the computer instructions are executed by a processor, the computer instructions cause a computer device to perform the following operations:

[0042] When the instruction of locating the hand region is received, the image captured by the image acquisition device is input into the preset convolutional neural network model to determine the hand region, wherein the hand region is a rectangular region formed by the outer edge of the hand, and the preset convolutional neural network model is obtained by optimizing the loss function of the hand joint point determination.

[0043] In the above embodiment, a method of locating a hand region and a display device are provided. The preset convolutional neural network model used in the method is obtained by optimizing the loss function of the hand joint point determination, and the joint point information is added to assist in locating the hand region, so that the accuracy of the convolutional neural network model outputting the hand region can be improved. The method comprises: when an instruction of locating a hand region is received, inputting an image captured by an image acquisition device into a preset convolutional neural network model to determine a hand region, wherein the hand region is a rectangular region formed by the outer edge of the hand, and the preset convolutional neural network model is obtained by optimizing the loss function of the hand joint point determination. BRIEF DESCRIPTION OF DRAWINGS

[0044] Figure 1 An operating scenario between a display device and a control device according to some embodiments is shown;

[0045] Figure 2 A hardware configuration block diagram of the control device 100 according to some embodiments is shown;

[0046] Figure 3 A hardware configuration block diagram of the display device 200 according to some embodiments is shown;

[0047] Figure 4 A software configuration diagram in the display device 200 according to some embodiments is shown;

[0048] Figure 5 A user interface including a special effect of an ideal effect according to some embodiments is exemplarily shown;

[0049] Figure 6 A user interface including a special effect when the hand region locating range is too large according to some embodiments is exemplarily shown;

[0050] Figure 7An exemplary illustration of a user interface including a hand region positioning too small special effect, in accordance with some embodiments;

[0051] Figure 8 An exemplary flowchart of a method of positioning a hand region, in accordance with some embodiments;

[0052] Figure 9 An exemplary illustration of a user interaction diagram, in accordance with some embodiments;

[0053] Figure 10 An exemplary illustration of a user interface diagram, in accordance with some embodiments;

[0054] Figure 11 An exemplary illustration of a hand region diagram, in accordance with some embodiments;

[0055] Figure 12 An exemplary illustration of yet another hand region diagram, in accordance with some embodiments;

[0056] Figure 13 An exemplary illustration of a coordinate system diagram, in accordance with some embodiments;

[0057] Figure 14 An exemplary illustration of another coordinate system diagram, in accordance with some embodiments;

[0058] Figure 15 An exemplary illustration of a hand's full hand joint position diagram, in accordance with some embodiments;

[0059] Figure 16 An exemplary illustration of a joint's bounding rectangle diagram, in accordance with some embodiments;

[0060] Figure 17 An exemplary illustration of another joint's bounding rectangle diagram, in accordance with some embodiments;

[0061] Figure 18 An exemplary illustration of an overlap region diagram, in accordance with some embodiments;

[0062] Figure 19 An exemplary illustration of a hand region diagram, in accordance with some embodiments;

[0063] Figure 20 An exemplary illustration of another hand region diagram, in accordance with some embodiments;

[0064] Figure 21 An exemplary illustration of a hand region superimposed on a hand, in accordance with some embodiments; Figure 19 and Figure 20 An exemplary illustration of a hand region superimposed on a hand, in accordance with some embodiments. DETAILED DESCRIPTION

[0065] For the purpose of making the purpose and embodiments of the present application more clear, the exemplary embodiments of the present application will be described clearly and completely below in combination with the drawings of the exemplary embodiments of the present application. Obviously, the described exemplary embodiments are only some of the embodiments of the present application, but not all the embodiments of the present application.

[0066] It should be noted that the brief description of the terms in the present application is only for the convenience of understanding the subsequently described embodiments, and is not intended to limit the embodiments of the present application. Unless otherwise specified, these terms should be understood according to their ordinary and general meanings.

[0067] The terms "first", "second", "third", etc. in the specification and claims of the present application and the above drawings are used to distinguish similar or identical objects or entities, and do not necessarily mean to limit the specific order or sequence, unless otherwise specified. It should be understood that the terms used in this way can be interchanged under appropriate circumstances.

[0068] The terms "include" and "have" and any variations thereof are intended to cover but not exclusive inclusion, for example, a product or device including a series of components does not have to be limited to all the components clearly listed, but can include other components not clearly listed or inherent to these products or devices.

[0069] The display device provided by the embodiments of the present application can have various implementation forms, for example, can be a television, a smart television, a laser projection device, a monitor, an electronic bulletin board, an electronic table, etc. Figure 1 And Figure 2 As a specific embodiment of the display device of the present application.

[0070] Figure 1 As shown in FIG. 1, the display device 200 can be connected to the control device 100, and the control device 100 can be connected to the smart device 300. Figure 1 As shown, the user can operate the display device 200 through the smart device 300 or the control device 100.

[0071] In some embodiments, the control device 100 can be a remote controller, and the communication between the remote controller and the display device includes infrared protocol communication or Bluetooth protocol communication, and other short-distance communication modes, to control the display device 200 through wireless or wired mode. The user can input user instructions through the keys on the remote controller, voice input, control panel input, etc., to control the display device 200.

[0072] In some embodiments, a smart device 300 (such as a mobile terminal, tablet computer, computer, laptop computer, etc.) may also be used to control the display device 200. For example, an application running on the smart device may be used to control the display device 200.

[0073] In some embodiments, the display device may receive instructions not through the aforementioned smart devices or control devices, but through touch or gestures.

[0074] In some embodiments, the display device 200 can also be controlled in ways other than the control device 100 and the smart device 300. For example, it can be controlled by directly receiving the user's voice commands through a module configured inside the display device 200 for acquiring voice commands, or it can be controlled by receiving the user's voice commands through a voice control device set outside the display device 200.

[0075] In some embodiments, the display device 200 also communicates with the server 400. The display device 200 may communicate via a local area network (LAN), wireless local area network (WLAN), and other networks. The server 400 may provide various content and interactive features to the display device 200. The server 400 may be a cluster or multiple clusters, and may include one or more types of servers.

[0076] Figure 2 An exemplary block diagram of the configuration of the control device 100 according to an exemplary embodiment is shown. Figure 2 As shown, the control device 100 includes a controller 110, a communication interface 130, a user input / output interface 140, a memory, and a power supply. The control device 100 can receive user input operation commands and convert the operation commands into commands that the display device 200 can recognize and respond to, thus acting as an intermediary for interaction between the user and the display device 200.

[0077] like Figure 3 The display device 200 includes at least one of the following: a tuner 210, a communicator 220, a detector 230, an external device interface 240, a controller 250, a display 260, an audio output interface 270, a memory, a power supply, and a user interface.

[0078] In some embodiments, the controller includes a processor, a video processor, an audio processor, a graphics processor, RAM, ROM, and a first interface to an nth interface for input / output.

[0079] The display 260 includes a display screen assembly for presenting images, a driving assembly for driving image display, a component for receiving image signals from the controller output, and a user control UI interface for displaying video content, image content, menu control interface, and user control UI interface.

[0080] The display 260 can be a liquid crystal display, an OLED display, and a projection display, and can also be a projection device and a projection screen.

[0081] The communicator 220 is a component for communicating with external devices or servers according to various communication protocol types. For example, the communicator can include at least one of a Wifi module, a Bluetooth module, a wired Ethernet module, and other network communication protocol chips or near field communication protocol chips, and an infrared receiver. The display device 200 can establish transmission and reception of control signals and data signals with the external control device 100 or the server 400 through the communicator 220.

[0082] The user interface can be used to receive control signals of the control device 100 (such as an infrared remote controller, etc.).

[0083] The detector 230 is used to collect signals of the external environment or interaction with the outside. For example, the detector 230 includes a light receiver for collecting ambient light intensity, or an image collector such as a camera for collecting external environment scenes, user attributes, or user interaction gestures, or a sound collector such as a microphone for receiving external sounds.

[0084] The external device interface 240 can include, but is not limited to, any one or more of the following: a high-definition multimedia interface (HDMI), an analog or digital high-definition component input interface (component), a composite video input interface (CVBS), a USB input interface (USB), an RGB port, etc. It can also be a composite input / output interface formed by the above multiple interfaces.

[0085] The tuner demodulator 210 receives broadcast television signals through wired or wireless reception, and demodulates audio and video signals and EPG data signals from multiple wireless or wired broadcast television signals.

[0086] In some embodiments, the controller 250 and the tuner demodulator 210 can be located in different split devices, that is, the tuner demodulator 210 can also be in an external device of the main device where the controller 250 is located, such as an external set-top box, etc.

[0087] The controller 250 controls the operation of the display device and responds to user operations through various software control programs stored in the memory. The controller 250 controls the overall operation of the display device 200. For example, in response to receiving a user command for selecting a UI object displayed on the display 260, the controller 250 can perform an operation related to the object selected by the user command.

[0088] In some embodiments, the controller includes at least one of a Central Processing Unit (CPU), a video processor, an audio processor, a Graphics Processing Unit (GPU), a RAM (Random Access Memory), a ROM (Read-Only Memory), a first interface to an n-th interface for input / output, a communication bus, and the like.

[0089] The user can input a user command through a Graphical User Interface (GUI) displayed on the display 260, and the user input interface receives the user input command through the Graphical User Interface (GUI). Alternatively, the user can input a user command by inputting a specific sound or gesture, and the user input interface receives the user input command by recognizing the sound or gesture through a sensor.

[0090] A "user interface" is a medium interface for interaction and information exchange between an application or an operating system and a user, which realizes conversion between an internal form of information and a form acceptable by the user. A commonly used form of the user interface is a Graphical User Interface (GUI), which refers to a user interface related to computer operation displayed in a graphical manner. The user interface can be an icon, a window, a control, and the like interface elements displayed in a display screen of an electronic device, wherein the control can include an icon, a button, a menu, a tab, a text box, a dialog box, a status bar, a navigation bar, a Widget, and the like visible interface elements.

[0091] Referring to Figure 4 In some embodiments, the system is divided into four layers from top to bottom, namely, an Applications layer (referred to as "application layer" for short), an Application Framework layer (referred to as "framework layer" for short), an Android runtime and system library layer (referred to as "system runtime library layer" for short), and a kernel layer.

[0092] In some embodiments, at least one application program is run in the application layer, which can be a window (Window) program, a system setting program, or a clock program, and the like provided by the operating system; or can be an application program developed by a third-party developer. In a specific implementation, the application program package in the application layer is not limited to the above examples.

[0093] The framework layer provides application programming interfaces (APIs) and programming frameworks for applications. The application framework layer includes some pre-defined functions. The application framework layer is equivalent to a processing center that decides which application in the application layer to act. The application can access resources in the system and obtain services of the system through the API interface in the execution.

[0094] As shown in Figure 4 In the embodiments of the present application, the application framework layer includes managers (Managers) and content providers (Content Provider), wherein the managers include at least one of the following modules: an activity manager (ActivityManager) for interacting with all activities running in the system; a location manager (Location Manager) for providing access to system location services for system services or applications; a package manager (Package Manager) for retrieving various information related to application packages currently installed on the device; a notification manager (NotificationManager) for controlling the display and clearing of notification messages; and a window manager (Window Manager) for managing icons, windows, toolbars, wallpapers, and desktop components on the user interface.

[0095] In some embodiments, the activity manager is used to manage the life cycle of each application and the general navigation back function, such as controlling the exit, opening, back, etc. of the application. The window manager is used to manage all window programs, such as obtaining the size of the display screen, determining whether there is a status bar, locking the screen, intercepting the screen, controlling the display window change (such as reducing the display window, shaking the display, twisting the display, etc.), etc.

[0096] In some embodiments, the system runtime library layer provides support for the upper layer, i.e. the framework layer. When the framework layer is used, the Android operating system runs the C / C++ library contained in the system runtime library layer to realize the functions of the framework layer.

[0097] In some embodiments, the kernel layer is a layer between hardware and software. As shown in Figure 4 The kernel layer at least includes at least one of the following drivers: an audio driver, a display driver, a Bluetooth driver, a camera driver, a WIFI driver, a USB driver, an HDMI driver, a sensor driver (such as a fingerprint sensor, a temperature sensor, a pressure sensor, etc.), and a power supply driver, etc.

[0098] In some embodiments, the display device can expand the display device functions through the application installed thereon. For example, the display device can provide special effect display functions and functions for interacting with the user.

[0099] In implementing the display device to perform special effect display based on the human hand and to implement the function of interacting with the user through the user gesture, the positioning of the human hand region is involved. In the related art, due to the high degree of freedom of the hand and fingers, i.e., the hand and fingers can move in multiple directions and at multiple angles, the hand movement range is large, which results in inaccurate detection of the hand region and affects the implementation effect of the above functions.

[0100] For example, the ideal effect of the special effect is to display a rotating virtual basketball on the tip of the index finger when the tip of the index finger is upward, simulating the scene that the user is turning the ball. The specific implementation manner can be to position the hand region based on the recognized image, the lower end of the virtual basketball is attached to the upper end of the hand region, and the tip of the index finger points to the center of the virtual basketball. As shown in Figure 5 , Figure 5 An example shows a user interface including a special effect with an ideal effect according to some embodiments, when the hand region is accurately positioned, i.e., the hand region is drawn close to the edge of the hand, the uppermost end of the tip of the index finger is in contact with the upper end of the hand region 500, and the tip of the index finger is attached to the virtual basketball. As can be seen from Figure 5 , the current displayed special effect can reflect the scene that the user actually turns the ball.

[0101] However, in the related art, the hand region is not accurately positioned, as shown in Figure 6 , Figure 6 An example shows a user interface including a special effect when the positioning range of the hand region is too large according to some embodiments. The positioning range of the hand region 500 is too large, the hand region 500 is not attached to the edge of the human hand, and the uppermost end of the hand region 500 is away from the upper end of the tip of the index finger, and thus the virtual basketball is also away from the tip of the index finger, the entire virtual basketball floats on the tip of the index finger, which is different from the scene in the actual user turning the ball, and affects the display effect of the special effect.

[0102] As shown in Figure 7 , Figure 7 An example shows a user interface including a special effect when the positioning range of the hand region is too small according to some embodiments. The positioning range of the hand region 500 is too small, and the hand region 500 covers part of the human hand, and thus the virtual basketball covers part of the index finger, which also affects the effect of the special effect.

[0103] In the above two cases, the special effect cannot be well displayed, affecting the user's use experience.

[0104] In addition, when a user interacts with a display device through gestures, hand region positioning is a prerequisite for hand pose estimation involved in the interaction process. If the hand region cannot be accurately positioned, the interaction effect with the display device through gestures will be affected.

[0105] Therefore, in some scenarios, how to improve the accuracy of positioning the hand region has become a problem to be solved by those skilled in the art.

[0106] To solve the above technical problems, the embodiments of the present application provide a method for positioning a hand region. The preset convolutional neural network model used in the method is optimized by using a loss function determined by a hand joint point, and the joint point information is added to assist in positioning the hand region, so that the accuracy of the convolutional neural network model outputting the hand region can be improved.

[0107] It should be noted that the method in the embodiments of the present application can be applied to a display device, and can also be applied to a computer, a VR glasses, an AR glasses and other terminals that can carry a camera.

[0108] As shown in Figure 8 , Figure 8 An exemplary flowchart of a method for positioning a hand region according to some embodiments is shown, which comprises:

[0109] S100, receiving an instruction for positioning a hand region. In the embodiments of the present application, the hand region needs to be positioned in various scenarios, mainly for positioning the hand region of the user's hand in the obtained image.

[0110] In some embodiments, starting a first preset application can generate an instruction for positioning a hand region. Exemplarily, the first preset application can be a game application, and the controlled object in the game application moves according to the recognized hand region. When the position of the hand region changes, the controlled object also changes. In the embodiments of the present application, when the game application is started, the instruction for positioning the hand region is directly generated.

[0111] As shown in Figure 9 , Figure 9 An exemplary user interaction diagram according to some embodiments is shown, Figure 9 Fig. (a) shows a user interface containing game application controls, and the user interface displays game application controls. In some embodiments, when the display device displays game application controls, other controls can also be displayed, which are not limited herein. The user can move the focus to the game application controls through the control device and press the confirmation key on the control device. The user interface jumps, and jumps to Fig. (b) as shown. Figure 9 Fig. (b) shows a user interface containing game application controls and a hand region, Figure 9The user interface in (b) shows a game screen, and the user can move the hand to control the movement of the controlled object (aircraft) in the user interface. It should be noted that, Figure 9 The user interface in (b) shows a game screen, and the user can move the hand to control the movement of the controlled object (aircraft) in the user interface. It should be noted that, Figure 9 The user interface in (b) shows a game screen, and the user can move the hand to control the movement of the controlled object (aircraft) in the user interface. It should be noted that,

[0112] In some embodiments, the second preset application is started, and the second preset application includes a certain item. Starting the item can generate an instruction for positioning the hand region. For example, the item can be a special effect item, and a special effect can be added based on the hand. When the special effect item is started, the instruction for positioning the hand region is generated.

[0113] For example, as shown in Figure 10 For example, as shown in Figure 10 For example, as shown in the user interface schematic diagram according to some embodiments, a special effect control corresponding to a special effect item is displayed in the user interface. The user moves the focus to the special effect control by using the control device, and presses the confirmation key on the control device to generate an instruction for positioning the hand region. After positioning the hand region, the hand region is used to add a special effect to the hand of the user. In some embodiments, when the display device displays the special effect control, controls that can realize other functions can also be displayed, which are not limited herein.

[0114] In some embodiments, the special effect control can be displayed on the floating layer of the user interface, and the special effect control is not in the lower part of the user interface, so as to avoid that the special effect control blocks other display contents on the user interface.

[0115] In some embodiments, if it is desired to cancel the addition of the special effect, the focus can be moved to the special effect control again, and the confirmation key on the control device is pressed, so that the special effect display is cancelled.

[0116] In some embodiments, the special effect control can be displayed by pressing a specific key on the control device. Specifically, when entering the second preset application, the special effect control is not directly displayed on the user interface. This is because if the special effect control is always displayed, part of the user interface can be blocked, which affects the integrity of the content display in the user interface. In the embodiments of the present application, when the user presses the specific key on the control device, the control display interface displays the special effect control. In this way, the special effect control can be displayed when needed, and the special effect control can not always block the user interface.

[0117] In some embodiments, when the special effect control is selected, the special effect item is started, and the special effect control is no longer displayed. In other embodiments, the user can press the certain button again to control the display device to stop displaying the special effect control.

[0118] It should be noted that in the embodiments of the present application, when the focus moves to the control, the border of the control can be thickened, and in addition, other forms can also be used to indicate that the control is selected. For example, when the focus moves to the control, the control can change its shape, for example, from a square to a circle, etc. When the control is selected, the control can be enlarged according to a preset ratio. For example, the display areas of the video controls on the user interface are the same, and when a certain control is selected, the display area of the control is increased by 1.2 times compared with the display area of the original control. Since the present application does not limit the shape of the control when the focus is on the control, other forms that can facilitate the user to distinguish that the video control is selected can also be accepted.

[0119] S200, when receiving the instruction of positioning the hand region, determining the hand region by using a preset convolutional neural network model on an image captured by an image capturing device.

[0120] In the embodiments of the present application, the image capturing device can be a device built in the display device, and can also be an external device connected with the display device. For example, as shown in Figure 1 As shown in FIG. 2, the display device 200 is installed with an image capturing device 231. In some embodiments, the image capturing device can be an RGB camera, which usually uses three independent CCD (Charge-coupled Device) sensors to obtain three color signals.

[0121] In the embodiments of the present application, in order to determine the best hand region, i.e. the rectangular region formed by the outer edge of the hand, a preset convolutional neural network model is used to process the image. For example, as shown in Figures 11-12 As shown in FIG. 3, Figure 11 and Figure 12 all show a hand region diagram according to some embodiments, Figure 11 and Figure 12 show the best hand region 500 corresponding to different postures of the hand.

[0122] In the embodiments of the present application, the preset convolutional neural network model is obtained by using a loss function determined by a hand joint point. Since the preset convolutional neural network model is optimized by using the hand joint point related information, the setting of the parameters of the preset convolutional neural network is more accurate, and the accuracy of the output data of the preset convolutional neural network model is improved.

[0123] In some embodiments, the step of optimizing the preset convolutional neural network model by using a loss function determined by the hand joint points comprises:

[0124] obtaining a training set, wherein the training set comprises a plurality of sample images and real information corresponding to the sample images, the real information comprising hand real classification, hand joint point real position, and hand real region frame.

[0125] In the embodiments of the present application, the sample images in the training set are photographed by an image acquisition device. In some embodiments, in order to make the preset convolutional neural network model more accurate, the same image acquisition device as mentioned above is used when photographing the sample images, that is, the image acquisition device used to photograph the images input into the preset convolutional neural network model, so that the difference in images photographed by different image acquisition devices when photographing the same scene can be avoided, thereby interfering with the process of learning the preset convolutional neural network model.

[0126] In the embodiments of the present application, the hand classification comprises left hand and right hand. In some embodiments, when the hand classification is left hand, "1" can be used as the identifier, and when the hand classification is right hand, "0" can be used as the identifier. It can be understood that other ways can be used to identify left hand and right hand, and the way of distinguishing left hand and right hand is acceptable in the embodiments of the present application.

[0127] In some embodiments, the hand joint point position is shown in the form of coordinates of the hand joint point.

[0128] In some embodiments, the hand real region frame is represented in the form of coordinates of four vertices.

[0129] In some embodiments, the coordinates are set according to coordinate system A, as shown in Figure 13 , and Figure 13 an exemplary diagram of a coordinate system according to some embodiments is shown. The coordinate system A takes the upper left corner of the image as the origin, sets the positive direction of the x-axis along the horizontal direction to the right, and sets the positive direction of the y-axis along the vertical direction downward. In some embodiments, the coordinates are set according to coordinate system B, as shown in Figure 14 , and Figure 14 an exemplary diagram of another coordinate system according to some embodiments is shown. The coordinate system B takes the center of the image as the origin, sets the positive direction of the x-axis along the horizontal direction to the right, and sets the positive direction of the y-axis along the vertical direction downward.

[0130] In some embodiments, when the user's hand is fully extended, the six degrees of freedom of the whole hand relative to the space are not considered, and there are 21 degrees of freedom of the human hand, which can be understood in this paper as the total number of hand joint point positions, including the palm of each finger, the joint, and the palm root. As Figure 15As shown, Figure 15 An exemplary diagram of all hand joint positions of a hand is shown according to some embodiments.

[0131] In some embodiments, the method for determining the real information corresponding to the sample image comprises: determining the hand class, the hand joint position and the hand region frame in an automatic manner. The automatic manner can directly output the hand class, the hand joint position and the hand region frame corresponding to the sample image. However, since the information in the training set needs to be accurately determined, so that the preset convolutional neural network model obtained by training is more accurate, after determining the above information in an automatic manner, the determined information also needs to be corrected manually. Finally, the corrected hand class, hand joint position and hand region frame are determined, which are the real hand class, real hand joint position and real hand region frame. The real hand class, real hand joint position and real hand region frame are the real information corresponding to the sample image in the real situation. In the embodiments of the present application, the automatic manner and the manual manner are combined to finally determine the real information corresponding to the sample image, so as to ensure the accuracy of the real information.

[0132] In the embodiments of the present application, since the number of sample images is large, if the real information corresponding to the sample image is directly determined by a manual manner, the labor is huge, so in the first round, the hand class, the hand joint position and the hand region frame are determined by an automatic manner, and then modified by a manual manner, so as to not only reduce the workload of manual operation, but also ensure the accuracy of the obtained real information.

[0133] In some embodiments, the database stores pre-stored images, and the hand class, the hand joint position and the hand region frame corresponding to the pre-stored images. The automatic manner can match the sample image with the pre-stored images in the database, calculate the similarity between the sample image and the pre-stored images, according to the similarity, select the pre-stored image with the largest similarity from the database, and take the hand class, the hand joint position and the hand region frame corresponding to the pre-stored image as the hand class, the hand joint position and the hand region frame corresponding to the sample image obtained by the automatic manner. Of course, the specific manner of the automatic manner is not limited in the embodiments of the present application, and other manners can also be used to determine the hand class, the hand joint position and the hand region frame.

[0134] There are many methods for calculating the similarity between images, for example, the histogram matching method can be used to calculate the histogram of the sample image and the pre-stored image respectively, and then calculate the normalized correlation coefficient of the two histograms, such as Bhattacharyya distance and histogram intersection distance. The specific method for calculating the similarity between images is not limited in the present application, and will not be enumerated here.

[0135] In the embodiments of the present application, in order to reduce the interference of the environment in the sample image on the hand image, each sample image is preprocessed. After the preprocessing process, the interference of the non-hand region in each sample image in the training set can be removed, which is beneficial to quickly determine the hand classification, the hand joint position and the hand region frame.

[0136] In some embodiments, the step of preprocessing each sample image includes: subtracting a revised value from the first RGB value of all pixel points in the sample image to obtain a second RGB value. The sample image with the second RGB value is taken as the preprocessed sample image. In the embodiments of the present application, the sample image is captured by an RGB camera, and each pixel in the sample image has a corresponding RGB value, i.e., the first RGB value.

[0137] In the embodiments of the present application, the revised value (means) is obtained through a large number of calculations and researches, and the revised value may be different in different scenarios.

[0138] For example, the revised value can be (104.117, 123). If the first RGB value is (255, 123, 204), the second RGB value is (255-104, 123-117, 204-123), i.e., (151, 6, 81).

[0139] In some embodiments, the determination step of the revised value can include: pre-calculating preset revised values in different environments, which can be understood as other regions in the sample image except the hand. The environment image and the corresponding preset revised value are stored in the database. In the database, the stored environment image identical to the environment image in the sample image is searched, and the corresponding preset revised value is extracted as the revised value in the preprocessing step.

[0140] If the identical environment image is not found in the database, the preset revised value corresponding to the most similar environment image in the database can be extracted as the revised value in the preprocessing step according to the similarity ranking.

[0141] All the preprocessed sample images are sequentially input into the initial convolutional neural network model, and the corresponding prediction information is output, which includes the hand prediction classification, the hand joint prediction position and the hand prediction region frame.

[0142] In the embodiments of the present application, the input data of the initial convolutional neural network model is an image, and the output data is a hand classification, a hand joint position and a hand region frame, and the initial convolutional neural network model is designed accordingly. In order to distinguish from the real data in the above, the output data of the initial convolutional neural network model is called the hand prediction classification, the hand joint prediction position and the hand prediction region frame.

[0143] In some embodiments, the initial convolutional neural network model refers to a model that is not adjusted by a loss function, and the parameters in the initial convolutional neural network model are random values.

[0144] The predicted information obtained by the initial convolutional neural network model after inputting the sample image may be different from the real information. Therefore, in order to reduce the difference between the preset information and the real information, that is, to make the output data closer to the real data, in the embodiments of the present application, the parameters in the initial convolutional neural network model are continuously adjusted by the loss value corresponding to the loss function. The smaller the loss value is, the closer the output data of the preset convolutional neural network model is to the real data. Finally, the initial convolutional neural network model after multiple parameter adjustments is used as the preset convolutional neural network model.

[0145] In some embodiments, the real information and the predicted information are input into a loss function to obtain a loss value, wherein the loss function includes a detection box regression loss function, a detection classification loss function, a hand joint detection regression loss function, and a hand joint detection penalty term. The joint detection penalty term is determined according to a hand prediction region box and a joint bounding rectangle. The joint bounding rectangle is the largest rectangle surrounded by the real position of the hand joint. In the embodiments of the present application, the detection box regression loss function and the detection classification loss function are used when training the convolutional neural network model in different fields. In addition to the detection box regression loss function and the detection classification loss function, the embodiments of the present application also include the hand joint detection regression loss function and the hand joint detection penalty term. The information of the hand joint is added to the loss function, so that the loss value calculated by the loss function can make the preset convolutional neural network model obtained after adjusting the parameters of the initial convolutional neural network model more accurate.

[0146] The initial convolutional neural network model is adjusted by using the loss value, and the adjusted initial convolutional neural network model is determined as the preset convolutional neural network model. In the embodiments of the present application, the parameters of the initial convolutional neural network model are adjusted by the loss value. After the adjustment of the loss value corresponding to a large amount of sample data, the output effect of the preset convolutional neural network model obtained by the initial convolutional neural network model is more accurate.

[0147] In some embodiments, the joint detection penalty term (1-IOLBR) is determined according to the hand prediction region box and the joint bounding rectangle according to the following formula:

[0148]

[0149] wherein, Area lbr is the area of the joint bounding rectangle, and Area preArea pre , Area lbr ) is the intersection area of the area where the hand region detection frame is located and the area where the joint outer rectangle is located.

[0150] It should be noted that the joint outer rectangle is the largest rectangle surrounded by the real positions of the hand joints, and examples are shown in Figure 16 and Figure 17 . Figure 16 and Figure 17 respectively show a schematic diagram of a joint outer rectangle according to some embodiments, which can be determined according to the four most edge hand joint real positions among all hand joint real positions. The four edges of the joint outer rectangle 600 pass through the four most edge hand joints respectively. In Figure 16 and Figure 17 , only the four most edge hand joints are labeled, and the other joints are not labeled.

[0151] In some embodiments, the joint outer rectangle is represented in the form of coordinates of four vertices.

[0152] In some embodiments, the method of screening the four most edge hand joint real positions can be to determine the positions of the hand joints with the maximum vertical coordinate, the minimum vertical coordinate, the maximum horizontal coordinate, and the minimum horizontal coordinate among all hand joint real positions.

[0153] In one example, the position of the hand joint with the maximum vertical coordinate is (2, 16), the position of the hand joint with the minimum vertical coordinate is (1, 2), the position of the hand joint with the maximum horizontal coordinate is (6, 13), and the position of the hand joint with the minimum horizontal coordinate is (-1, 9). Therefore, the four most edge hand joint real positions are determined as (2, 16), (1, 2), (6, 13), and (-1, 9), and the four vertices of the determined joint outer rectangle are (-1, 2), (-1, 16), (6, 2), and (6, 16).

[0154] In another example, the position of the hand joint with the maximum vertical coordinate is (2, 16), the position of the hand joint with the minimum vertical coordinate is (-1, 1), the position of the hand joint with the maximum horizontal coordinate is (6, 13), and the position of the hand joint with the minimum horizontal coordinate is (-1, 1). Therefore, the four most edge hand joint real positions are determined as (2, 16), (-1, 1), (6, 13), and (-1, 1), and the four vertices of the finally determined joint outer rectangle are (-1, 1), (-1, 16), (6, 16), and (6, 1) with (-1, 1) as one vertex.

[0155] As shown in the example, Figure 18 Figure 18 As shown in the example, a schematic diagram of an overlapping area according to some embodiments is shown, showing an overlapping area 800 of the area where the hand region detection frame 700 is located and the area where the joint outer-approximate rectangle 600 is located, and the intersection area (Area pre, Area br) of the overlapping area 800.

[0156] In the embodiments of the present application, the joint outer-approximate rectangle is necessarily within the best hand region. The greater the overlapping area of the area where the joint outer-approximate rectangle is located and the area where the hand prediction region frame is located, the more accurate the data output in the preset convolutional neural network model at this time, and thus the loss value corresponding to the loss function should be smaller, so the joint detection penalty term (1-IOLBR) is used as part of the loss function here. The greater the overlapping area of the two areas, the greater the value of IOLBR, and the smaller the value of the joint detection penalty term (1-IOLBR).

[0157] In some embodiments, the formula of the loss function is:

[0158] loss total = loss box + loss class + alpha * loss landmark + beta * (1-ILOBR) ;

[0159] Wherein, loss total is the loss function, loss box is the detection frame regression loss function, loss class is the detection classification loss function, loss landmark is the hand joint detection regression loss function, 1-ILOBR is the hand joint detection penalty term, alpha is the weight of the hand joint detection regression loss function, and beta is the weight of the hand joint detection penalty term.

[0160] In the embodiments of the present application, in order to adapt to different scenarios, the hand joint detection regression loss function and the hand joint detection penalty term in the loss function are both provided with corresponding weights, that is, alpha is the weight of the hand joint detection regression loss function, and beta is the weight of the hand joint detection penalty term. When the preset convolutional neural network model needs to output more accurate hand joint prediction positions, the weight of the hand joint detection regression loss function can be increased; when the preset convolutional neural network model needs to output more accurate hand prediction region frames, the weight of the hand joint detection penalty term can be increased.

[0161] ​In some embodiments, compared with the related art, after using the method for positioning the hand region in the above, the hand region positioning is more accurate, as shown in Figure 19 Figure 19 An exemplary hand region diagram according to some embodiments is shown, Figure 19 The hand region 900 in the diagram is determined using the method for positioning the hand region in the embodiments of the present application. As shown in Figure 20 Figure 20 An exemplary hand region diagram according to some embodiments is shown, Figure 20 The hand region 1000 in the diagram is not determined using the method for positioning the hand region in the embodiments of the present application. The Figure 19 and Figure 20 are superimposed and displayed, as shown in Figure 21 By comparison, it can be seen that the hand region obtained using the method for positioning the hand region in the embodiments of the present application is more fitted to the palm and the positioning effect is more accurate.

[0162] The embodiments of the present application also provide a display device, comprising:

[0163] a display for displaying a user interface;

[0164] a user interface for receiving an input signal;

[0165] a controller connected with the display and the user interface respectively, for performing:

[0166] when receiving an instruction for positioning the hand region, determining a hand region from an image captured by an image acquisition device using a preset convolutional neural network model, wherein the hand region is a rectangular region formed by the outer edge of the hand, and the preset convolutional neural network model is obtained by optimizing a loss function determined using a hand joint.

[0167] In the embodiments of the present application, a storage medium having computer instructions stored thereon is provided, and the computer instructions are executed by a processor to make a computer device perform:

[0168] when receiving an instruction for positioning the hand region, determining a hand region from an image captured by an image acquisition device using a preset convolutional neural network model, wherein the hand region is a rectangular region formed by the outer edge of the hand, and the preset convolutional neural network model is obtained by optimizing a loss function determined using a hand joint.

[0169] ​​In the above embodiment, a method for positioning a hand region and a display device, the preset convolutional neural network model used in the method is optimized by a loss function determined by a hand joint point, and joint point information is added to assist in positioning the hand region, so that the accuracy of the convolutional neural network model outputting the hand region can be improved. The method comprises: when an instruction for positioning a hand region is received, an image captured by an image acquisition device is input into a preset convolutional neural network model to determine a hand region, wherein the hand region is a rectangular region formed by an outer edge of a hand, and the preset convolutional neural network model is optimized by a loss function determined by a hand joint point.

[0170] Finally, it should be noted that: the above embodiments are only used to illustrate the technical solutions of the present application, and not to limit them; although the present application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that: it can still modify the technical solutions recorded in the foregoing embodiments, or make equivalent replacement to part or all of the technical features; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the scope of the technical solutions of the embodiments of the present application.

[0171] For the convenience of explanation, the above description has been made in combination with specific embodiments. However, the above exemplary discussion is not intended to exhaust or limit the embodiments to the specific forms disclosed above. Various modifications and variations can be derived according to the above teachings. The selection and description of the above embodiments are to better explain the principles and practical applications, so that those skilled in the art can better use the embodiments and various different modified embodiments suitable for specific use considerations.

Claims

1. A display device, characterized by comprising: The method comprises the following steps: a display for displaying a user interface; a user interface for receiving an input signal; a controller connected with the display and the user interface respectively, for performing the following steps: when receiving an instruction of locating a hand region, determining a hand region from an image captured by an image capturing device based on a preset convolutional neural network model, wherein the hand region is a rectangular region formed by the outer edge of the hand, and the preset convolutional neural network model is obtained by optimizing a loss function determined by hand joint points; the loss function comprises a hand joint point detection penalty term, and the joint point detection penalty term is determined according to a hand prediction region frame and a joint point bounding rectangle, and the joint point bounding rectangle is a maximum rectangle surrounded by the real positions of the hand joint points; the joint point bounding rectangle is determined according to the hand joint points with the maximum vertical coordinate, the minimum vertical coordinate, the maximum horizontal coordinate and the minimum horizontal coordinate among all the real positions of the hand joint points.

2. The display device of claim 1, wherein, the controller is configured to perform the optimization of the preset convolutional neural network model by the loss function determined by the hand joint points according to the following steps: obtaining a training set, wherein the training set comprises a plurality of sample images and real information corresponding to the sample images, and the real information comprises a hand real classification, real positions of hand joint points and a hand real region frame; preprocessing each of the sample images; inputting all the preprocessed sample images into an initial convolutional neural network model in sequence to output corresponding prediction information, wherein the prediction information comprises a hand prediction classification, prediction positions of hand joint points and a hand prediction region frame; inputting the real information and the prediction information into a loss function to obtain a loss value, wherein the loss function comprises a detection frame regression loss function, a detection classification loss function, a hand joint point detection regression loss function and a hand joint point detection penalty term; adjusting the initial convolutional neural network model by using the loss value to determine an adjusted initial convolutional neural network model as the preset convolutional neural network model.

3. The display device of claim 2, wherein, the controller is configured to determine the joint point detection penalty term (1-IOLBR) according to the hand prediction region frame and the joint point bounding rectangle according to the following formula: Wherein, Area lbr is an area where the joint outer rectangle is located, Area pre is an area where the hand prediction area frame is located; Intersection(Area pre , Area lbr ) is an overlapping area of the area where the hand area detection frame is located and the area where the joint outer rectangle is located.

4. The display device of claim 2, wherein, the controller is configured to perform the preprocessing of each sample image according to the following steps: subtracting a revision value from the first RGB value of all the pixel points in the sample image to obtain a second RGB value; taking the sample image with the second RGB value as the preprocessed sample image.

5. The display device of claim 2, wherein, the formula of the loss function is: loss total = loss box + loss class + alpha * loss landmark + beta * (1 - ILOBR); wherein loss total is a loss function, loss box is a bounding box regression loss function, loss class is a bounding box classification loss function, loss landmark is a hand joint detection regression loss function, 1-ILOBR is a hand joint detection penalty term, and alpha is a weight of the hand joint detection regression loss function and beta is a weight of the hand joint detection penalty term.

6. A method of locating a hand region, characterized by, The method comprises the following steps: when receiving an instruction of locating a hand region, determining a hand region from an image captured by an image capturing device based on a preset convolutional neural network model, wherein the hand region is a rectangular region formed by the outer edge of the hand, and the preset convolutional neural network model is obtained by optimizing a loss function determined by hand joint points; the loss function comprises a hand joint point detection penalty term, and the joint point detection penalty term is determined according to a hand prediction region frame and a joint point bounding rectangle, and the joint point bounding rectangle is a maximum rectangle surrounded by the real positions of the hand joint points; The joint outside rectangle is determined according to the maximum longitudinal coordinate, the minimum longitudinal coordinate, the maximum horizontal coordinate and the minimum horizontal coordinate of the hand joint real positions.

7. The method of claim 6, wherein, The step of obtaining the preset convolutional neural network model by optimizing the loss function determined by the hand joint includes: Obtaining a training set, wherein the training set includes a plurality of sample images and real information corresponding to the sample images, the real information including hand real classification, hand joint real position and hand real region frame; Pretreating each sample image; inputting all the pretreated sample images into an initial convolutional neural network model in sequence to output corresponding prediction information, the prediction information including hand prediction classification, hand joint prediction position and hand prediction region frame; Inputting the real information and the prediction information into a loss function to obtain a loss value, wherein the loss function includes a detection frame regression loss function, a detection classification loss function, a hand joint detection regression loss function and a hand joint detection penalty term, the joint detection penalty term being determined according to the hand prediction region frame and the joint outside rectangle, the joint outside rectangle being the largest rectangle surrounded by the hand joint real positions; Adjusting the initial convolutional neural network model by using the loss value to determine an adjusted initial convolutional neural network model as the preset convolutional neural network model.

8. The method of claim 7, wherein, The calculation formula of the joint detection penalty term (1-IOLBR) is determined according to the hand prediction region frame and the joint outside rectangle. Wherein, Area lbr is an area where the joint outer rectangle is located, Area pre is an area where the hand prediction area frame is located; Intersection(Area pre , Area lbr ) is an overlapping area of the area where the hand area detection frame is located and the area where the joint outer rectangle is located.

9. The method of claim 7, wherein, The step of pretreating the sample image includes: Subtracting a revision value from the first RGB value of all pixel points in the sample image to obtain a second RGB value; Taking the sample image with the second RGB value as the pretreated sample image.

10. The method of claim 7, wherein, The formula of the loss function is: loss total = loss box + loss class + alpha * loss landmark + beta * (1 - ILOBR); wherein loss total is a loss function, loss box is a bounding box regression loss function, loss class is a bounding box classification loss function, loss landmark is a hand joint detection regression loss function, 1-ILOBR is a hand joint detection penalty term, and alpha is a weight of the hand joint detection regression loss function and beta is a weight of the hand joint detection penalty term.

Images