Gesture-based display equipment control method and device, storage medium and equipment

Abstract

The invention discloses a gesture-based display equipment control method and device, a storage medium and computer equipment. The method comprises the following steps: continuously acquiring a hand image of a user; performing image recognition on the hand image to obtain first point coordinates corresponding to each joint point of the hand of the user; determining whether a user gesture in the hand image belongs to a preset gesture type or not through a preset gesture recognition model according to the first point coordinates corresponding to the joint points; and when the user gesture belongs to the type of the preset gesture, determining a first screen control instruction based on the user gesture, and controlling the display equipment through the first screen control instruction. The hand image of the user is collected through the camera, and then the display equipment is controlled according to the gesture of the user in the hand image, so that the two hands of the user can be liberated, the activity range of the user when the display equipment is controlled is expanded, and the operation flexibility and efficiency are improved.

Description

Technical Field

[0001] The present application relates to the field of computer technology, and in particular to a gesture-based display device control method and device, a storage medium, and a computer device. Background Art

[0002] With the rapid development of science and technology, portable electronic devices and large display devices have become an indispensable part of modern society, and they are widely used in daily life, work, education, entertainment and other fields. However, although these technologies have brought unprecedented convenience and interactivity, they also face some significant limitations, especially in terms of user interaction experience and display efficiency.

[0003] In the prior art, when using portable electronic devices such as smart phones and tablet computers, the portable electronic devices are usually controlled by single-finger or multi-finger operations such as finger clicking, long pressing, and dragging. However, holding these devices for a long time and watching them at close range often leads to fatigue of the hands and vision, which is particularly prominent for users who need to demonstrate, study or entertain for a long time. In addition, for large screens used in public places such as large conferences, press conferences, and exhibitions, it is usually necessary to set up a counter in front of the user to place external devices (such as computers, keyboards, mice, etc.). The user uses these external devices to control the display content of the large screen. However, this method limits the user's range of activities and requires them to stand at the counter or frequently go back and forth between the counter and the screen, reducing the flexibility and efficiency of the operation. In summary, when using existing portable electronic devices and large screens, there are limitations to varying degrees in terms of user experience, operational flexibility, and display efficiency. Summary of the invention

[0004] In view of this, the present application provides a gesture-based display device control method and apparatus, storage medium, and computer equipment, which captures the user's hand image through a camera, and then controls the display device according to the user's gesture in the hand image, thereby freeing the user's hands and increasing the user's range of motion when controlling the display device, helping to improve the flexibility and efficiency of the operation.

[0005] According to one aspect of the present application, a gesture-based display device control method is provided, comprising:

[0006] Continuously obtain the user's hand image;

[0007] Performing image recognition on the hand image to obtain first point coordinates corresponding to each joint point of the user's hand;

[0008] According to the first point coordinates corresponding to each joint point, determining whether the user gesture in the hand image belongs to the type of preset gesture by using a preset gesture recognition model;

[0009] When the user gesture belongs to the type of preset gesture, a first screen control instruction is determined based on the user gesture, and the display device is controlled by the first screen control instruction.

[0010] According to another aspect of the present application, a gesture-based display device control apparatus is provided, comprising:

[0011] A hand image acquisition module, used to continuously acquire a user's hand image;

[0012] An image recognition module, used to perform image recognition on the hand image to obtain first point coordinates corresponding to each joint point of the user's hand;

[0013] A gesture recognition module, configured to determine whether the user gesture in the hand image belongs to a preset gesture type according to the first point coordinates corresponding to each joint point and through a preset gesture recognition model;

[0014] A control module is used to determine a first screen control instruction based on the user gesture when the user gesture belongs to the type of preset gesture, and control the display device through the first screen control instruction.

[0015] According to another aspect of the present application, a storage medium is provided, on which a computer program is stored, and when the program is executed by a processor, the above-mentioned gesture-based display device control method is implemented.

[0016] According to another aspect of the present application, a computer device is provided, including a storage medium, a processor, and a computer program stored on the storage medium and executable on the processor, wherein the processor implements the above-mentioned gesture-based display device control method when executing the program.

[0017] By means of the above technical scheme, the present application provides a gesture-based display device control method and device, storage medium, and computer device. First, the user's hand image can be continuously collected through a camera. Here, the hand image can include both hands or one hand of the user. After each frame of hand image is obtained, the frame of hand image can be processed and analyzed to identify the first point coordinates of each joint point of the hand. Further, according to the first point coordinates of each joint point obtained, a preset gesture recognition model can be used to determine whether the current user gesture matches the type of the preset gesture. If it is recognized that the user gesture belongs to a preset gesture type, then the first screen control instruction can be determined according to the user gesture, and then the display device is controlled using the first screen control instruction. The embodiment of the present application collects the user's hand image through a camera, and then controls the display device according to the user gesture in the hand image, which can free the user's hands, increase the user's range of activity when controlling the display device, and help improve the flexibility and efficiency of the operation.

[0018] The above description is only an overview of the technical solution of the present application. In order to more clearly understand the technical means of the present application, it can be implemented in accordance with the contents of the specification. In order to make the above and other purposes, features and advantages of the present application more obvious and easy to understand, the specific implementation methods of the present application are listed below. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] The drawings described herein are used to provide a further understanding of the present application and constitute a part of the present application. The illustrative embodiments of the present application and their descriptions are used to explain the present application and do not constitute an improper limitation on the present application. In the drawings:

[0020] Figure 1 A flowchart of a gesture-based display device control method provided in an embodiment of the present application is shown;

[0021] Figure 2 A schematic diagram of various joints of a hand provided in an embodiment of the present application is shown;

[0022] Figure 3 A schematic diagram of a hand trajectory provided by an embodiment of the present application is shown;

[0023] Figure 4 A schematic diagram of an updated three-dimensional hand model provided in an embodiment of the present application is shown;

[0024] Figure 5 A schematic diagram showing a method of adjusting a preset three-dimensional hand model according to a hand rotation matrix provided in an embodiment of the present application is shown;

[0025] Figure 6A schematic diagram of an updated three-dimensional hand model obtained after adjustment according to a joint rotation matrix provided in an embodiment of the present application is shown;

[0026] Figure 7 A schematic structural diagram of a gesture-based display device control device provided in an embodiment of the present application is shown;

[0027] Figure 8 A schematic diagram of the device structure of a computer device provided in an embodiment of the present application is shown. DETAILED DESCRIPTION

[0028] The present application will be described in detail below with reference to the accompanying drawings and in combination with embodiments. It should be noted that the embodiments and features in the embodiments of the present application can be combined with each other without conflict.

[0029] In this embodiment, a gesture-based display device control method is provided. Figure 1 As shown, the method includes:

[0030] Step 101, continuously acquiring the user's hand image.

[0031] The embodiment of the present application provides a gesture-based display device control method. When a user operates and controls the display screen of an electronic device, a large display device, and other display contents, etc., it can be separated from the traditional operation method. After the user's hand image is collected by the camera, the display content of the display device can be controlled according to the user's gesture after the user's gesture is recognized. Here, the display device can be a mobile phone or a tablet computer, etc., or a large screen used in exhibitions, various press conferences, etc. For example, when you feel tired after using a mobile phone or a tablet computer for a long time, you can place the mobile phone or tablet computer on a bracket and keep the camera facing the user. The user can operate the mobile phone or tablet computer without holding the device, relying only on one-handed or two-handed gestures; at various press conferences or exhibitions, when there is a camera facing the user, the user can control the large screen without relying on other devices using only one-handed or two-handed gestures. It should be noted that in this embodiment, the method supports camera permission application, and then the user's hand image can be continuously collected through the camera. Here, the hand image can include the user's hands or one hand.

[0032] Step 102: Perform image recognition on the hand image to obtain first point coordinates corresponding to each joint point of the user's hand.

[0033] In this embodiment, after each frame of hand image is acquired, the frame of hand image can be processed and analyzed to identify the first point coordinates of each hand joint. Here, when identifying the first point coordinates of each hand joint, the model algorithm for identifying hand joints in the prior art can be used, which is not limited here. It should be noted that although the hand image is a two-dimensional image, the first point coordinates can be three-dimensional coordinates. Specifically, the hand image can be input into the model algorithm for identifying hand joints, and the model algorithm outputs the three-dimensional coordinates. The model algorithm for identifying hand joints is obtained by training a large number of two-dimensional hand image samples, and the two-dimensional hand image samples may include the three-dimensional coordinates (x-axis coordinates, y-axis coordinates, and depth coordinates) of each manually marked hand joint.

[0034] Step 103: Determine whether the user gesture in the hand image belongs to a preset gesture type according to the first point coordinates corresponding to each joint point and through a preset gesture recognition model.

[0035] In this embodiment, the preset gesture recognition model can be used to determine whether the current user gesture matches the type of the preset gesture based on the first point coordinates of each joint point obtained in the previous step. These preset gestures can be single-handed gestures such as zoom in, zoom out, click, OK, etc., or more complex two-handed combination gestures. Here, the preset gesture recognition model may be built based on deep learning algorithms, which can accurately distinguish different gesture types by learning from a large amount of sample data.

[0036] Step 104: When the user gesture belongs to a preset gesture type, a first screen control instruction is determined based on the user gesture, and the display device is controlled by the first screen control instruction.

[0037] In this embodiment, once it is recognized that the user gesture belongs to a certain preset gesture type, the first screen control instruction can be determined according to the user gesture, and then the first screen control instruction can be transmitted through a specific interface, and the generated first screen control instruction is sent to the display device, thereby achieving an effect similar to operating on a touch screen or a mouse. In this way, the user can control various functions of the display device through simple gestures, such as turning pages, zooming, switching channels, etc., thereby achieving a more natural and convenient interactive experience. Here, the mapping relationship between different preset gestures and the first screen control instruction can be preset.

[0038] By applying the technical solution of this embodiment, first, the user's hand image can be continuously collected through the camera. Here, the hand image can include both hands or one hand of the user. After each frame of hand image is obtained, the frame of hand image can be processed and analyzed to identify the first point coordinates of each joint point of the hand. Further, according to the first point coordinates of each joint point obtained, a preset gesture recognition model can be used to determine whether the current user gesture matches the type of preset gesture. If it is recognized that the user gesture belongs to a certain preset gesture type, then the first screen control instruction can be determined according to the user gesture, and then the display device is controlled using the first screen control instruction. The embodiment of the present application collects the user's hand image through the camera, and then controls the display device according to the user gesture in the hand image, which can free the user's hands, increase the user's range of activities when controlling the display device, and help improve the flexibility and efficiency of the operation.

[0039] In an embodiment of the present application, optionally, after "when the user gesture belongs to the type of preset gesture" described in step 104, the method also includes: when the type of preset gesture corresponding to the user gesture is the trigger type of the hand trajectory monitoring instruction, obtaining the hand image of the user for a preset number of frames after the current hand image; determining the hand trajectory corresponding to the user's hand according to the first point coordinates corresponding to each joint point of the user's hand in the current hand image, and the first point coordinates corresponding to each joint point of the user's hand in each frame of the user's hand image, and based on the hand trajectory, determining the second screen control instruction, and controlling the display device through the second screen control instruction.

[0040] In this embodiment, if the user gesture in the nth frame of hand images acquired continuously belongs to the type of preset gesture, then, further, it can be determined whether the user gesture in the frame of hand images is the trigger type of the preset hand trajectory monitoring instruction. The hand trajectory monitoring instruction is used to indicate the motion trajectory of the hand to be tracked next. If the user gesture in the frame of hand images is the trigger type of the preset hand trajectory monitoring instruction, then the continuous preset frame number of hand images after the frame of hand images can be acquired from the hand images continuously acquired by the camera. The preset frame number here is to ensure that enough hand joint position information can be captured so that the hand trajectory can be accurately drawn later. The preset frame number can be determined based on experience. For the nth frame, and the continuous preset frame number of hand images after the nth frame, the first point coordinates of each joint point of the user's hand can be identified and recorded. Here, each hand joint point can include a wrist joint point, 4 joint points corresponding to the thumb, 4 joint points corresponding to the index finger, 4 joint points corresponding to the middle finger, 4 joint points corresponding to the ring finger, and 4 joint points corresponding to the little finger, such as Figure 2As shown, a schematic diagram of various hand joints provided by an embodiment of the present application is shown. After obtaining the first point coordinates of each joint point in the above-mentioned hand image frames, the movement trajectory of the hand between consecutive frames, that is, the hand trajectory, can be drawn by connecting these points (or performing smoothing). This hand trajectory reflects the movement path of the hand in space. Afterwards, based on the determined hand trajectory, the second screen control instruction corresponding to the hand trajectory can be further analyzed, and the display device can be subsequently controlled to perform corresponding operations such as sliding, zooming, etc. according to the second screen control instruction. Figure 3 As shown, a schematic diagram of a hand trajectory provided by an embodiment of the present application is shown, in which the two arrows on the far left and the far right represent the hand trajectory of the "zoom in" operation, and the two arrows in the middle represent the hand trajectory of the "zoom out" operation. Here, the mapping relationship between different hand trajectories and the second screen control instructions can also be preset. By monitoring and analyzing the user's hand trajectory, the embodiment of the present application realizes an intuitive and natural interactive control method for the display device, thereby improving the user experience.

[0041] In an embodiment of the present application, optionally, the hand image includes both hands of the user, and step 103 includes: inputting the first point coordinates corresponding to each joint point of the user's hands into a preset gesture recognition model, and determining whether the two-hand gesture in the hand image belongs to the type of preset gesture through the preset gesture recognition model; accordingly, before inputting the first point coordinates corresponding to each joint point of the user's hands into the preset gesture recognition model, the method also includes: determining the key joint points of the hands corresponding to each preset gesture, and obtaining training samples corresponding to each preset gesture based on the key joint points of the hands, wherein the key joint points are necessary joint points for identifying the preset gestures; and training the initial gesture recognition model based on the training samples corresponding to different preset gestures to obtain the preset gesture recognition model.

[0042] In this embodiment, the user's hand image may include both hands of the user, so that the user's gesture can be identified by detecting and analyzing the coordinates of the joint points of both hands to identify whether the user's gesture belongs to the type of preset gesture, thereby realizing a more complex preset gesture. First, the first point coordinates corresponding to the detected joint points of both hands are input into the preset gesture recognition model, and the preset hand recognition model can identify whether the two-hand gesture contained in the hand image belongs to the type of preset gesture based on these first point coordinates. Here, the first point coordinates of each joint point identified by the hand image can be marked with the hand number and the joint point number, wherein the hand number is used to indicate which hand the first point coordinate belongs to, and the joint point number is used to indicate which joint point the first point coordinate belongs to. The first point coordinates can be input into the preset gesture recognition model in the form of two coordinate sequences, and the arrangement order of the first point coordinates in each coordinate sequence can be predetermined according to the connection relationship of the joint points of the hand. For example, sequence 1 corresponds to the sequence of the left hand, sequence 2 corresponds to the sequence of the right hand, and the joint point corresponding to the first first point coordinate in sequence 1 is the wrist joint point, etc., which can greatly improve the recognition efficiency of the preset gesture recognition model. The embodiment of the present application arranges the first point coordinates in a fixed order and then inputs them into a preset gesture recognition model, so that the calculation unit inside the model can be optimized for this fixed order, reducing unnecessary calculation overhead, so that the model can process input data faster.

[0043] The training process of the preset gesture recognition model can be as follows:

[0044] First, determine the key joints. The key joints are the joints necessary for identifying the preset gestures. For example, for the preset gesture of "pinch", the preset gesture can usually be determined by the four joints corresponding to the thumb and the four joints corresponding to the index finger, and the influence of the first point coordinates of the remaining joints on the judgment of the preset gesture can be ignored, so these eight joints can be used as the key joints of the preset gesture. Then, according to the key joints corresponding to each preset gesture, select or generate corresponding training samples from the existing data set. Among them, each training sample should contain a hand image, a gesture type label corresponding to the hand image, and point coordinate information, wherein the point coordinate information includes the point coordinates of all key joints corresponding to the hand image. Then, use these training samples to train the initial gesture recognition model. During the training process, after the point coordinate information in each training sample is input into the initial gesture recognition model, the initial gesture recognition model can learn features from the input point coordinate information and match the learned features with the preset gesture type. Through a large number of iterations and optimizations, the model can gradually improve its accuracy and robustness in gesture recognition tasks, and finally, a preset gesture recognition model can be obtained. It should be noted that the key joint points can be the joint points of a single hand or the joint points of both hands, which are not limited here. When the key joint points are the joint points of a single hand, it means that the gesture of only one hand of the user matches the key joint points, then it belongs to the type of preset gesture. The embodiment of the present application determines the key joint points of a single hand or both hands corresponding to each preset gesture, and the core features of the user's gesture can be captured and understood more accurately in the future. These key joint points are necessary conditions for identifying specific gestures, which help to reduce misjudgment and improve recognition accuracy. In addition, obtaining training samples corresponding to each preset gesture according to the key joint points can ensure the pertinence and effectiveness of the training data. This customized training method helps the model better learn the characteristics of different gestures, thereby achieving more accurate recognition in practical applications.

[0045] In an embodiment of the present application, optionally, when the type of preset gesture corresponding to the user gesture is a joint control type, the method further includes: determining a first display position of a target guide mark on the display device according to first point coordinates corresponding to target joint points of both hands of the user, and displaying the target guide mark at the first display position of the display device; accordingly, the "controlling the display device through the first screen control instruction" in step 104 includes: determining a target interaction object from the display content of the display device based on the first screen control instruction and the first display position, and controlling the display content of the display device according to the first screen control instruction and the target interaction object.

[0046] In this embodiment, the type of preset gesture may include an independent control type and a joint control type. When the user gesture belongs to the independent control type, the display content of the display device can be controlled only according to the recognized user gesture, such as turning pages. When the user gesture belongs to the joint control type, not only the display content of the display device is controlled according to the user gesture, but also the content that the user wants to interact with needs to be recognized.

[0047] When the type of preset gesture corresponding to the user gesture is a joint control type, the first display position of the target guide mark on the display device can be calculated according to the first point coordinates corresponding to the target joint points of the user's hands. These target guide marks can be virtual icons, cursors or highlight areas, which are used to indicate the mapping position of the user's current gesture on the display device. Here, the target joint point refers to the joint point used to determine the first display position of the target guide mark on the display device, which can be one joint point in both hands, or multiple joint points, and can be pre-set.

[0048] Subsequently, the target guide mark can be displayed at the first display position of the display device. In this way, the user can intuitively understand whether their gesture is pointing to the correct target or whether it is on the correct operation path by viewing the target guide mark on the display device, which can help the user quickly move the user gesture to the location where the interactive content is located. Under the joint control type, the target interactive object can be determined from the current display content of the display device according to the first screen control instruction, combined with the display position of the previously determined target guide mark. The target interactive object here can be an icon of an application, a file, a play control button, a game element, a specific content in a document, etc. After determining the target interactive object, the display content of the display device can be controlled accordingly according to the first screen control instruction and the target interactive object. For example, if the first screen control instruction corresponding to the user gesture indicates that a video is to be played, the corresponding video can be located in the video list according to the first display position where the target guide mark is located, and playback can be started. The embodiment of the present application can enable the user to interact with the display device in a more natural and intuitive way, improving the user experience and the flexibility of operation.

[0049] In an embodiment of the present application, optionally, when the type of preset gesture corresponding to the user gesture is a joint control type, the method further includes: determining the second display position of the wrist joint point of the preset hand three-dimensional model of each hand on the display device according to the first point coordinates corresponding to the joint points of both hands of the user, and determining the hand rotation matrix and joint rotation matrix corresponding to each of the preset hand three-dimensional models, performing a rotation transformation on the corresponding preset hand three-dimensional model according to the hand rotation matrix and the joint rotation matrix to obtain an updated hand three-dimensional model, and displaying the updated hand three-dimensional model of each hand on the display device based on the second display position; accordingly, the "controlling the display device through the first screen control instruction" in step 104 includes: determining the target interaction object from the display content of the display device based on the first screen control instruction and the third display position of the remaining joint points except the wrist joint point in the updated hand three-dimensional model of each hand on the display device, and controlling the display content of the display device according to the first screen control instruction and the target interaction object.

[0050] In this embodiment, when the type of the preset gesture corresponding to the user gesture is a joint control type, not only can the type of the user gesture be identified, but also the corresponding hand three-dimensional model can be rendered and displayed in real time on the display device according to the actual position and posture of the user's hands, so that the user's experience and immersion are stronger. First, the second display position of the wrist joint point of the preset hand three-dimensional model of each hand on the display device can be calculated according to the first point coordinates corresponding to each joint point of the user's hands. This second display position is the anchor point of the hand three-dimensional model on the display device, which is used to determine the overall position of the hand three-dimensional model. Then, the corresponding hand rotation matrix and joint rotation matrix can be calculated according to the first point coordinates of the joint point of each hand. These matrices describe the rotation angle and direction of the hand and its various joints. Then, these matrices are used to rotate and transform the preset hand three-dimensional model to match the actual posture of the user's hands. In this way, the updated hand three-dimensional model can accurately reflect the current position and posture of the user's hands. Finally, the updated hand three-dimensional model of each hand can be displayed on the display device based on the calculated second display position. In this way, users can intuitively understand their gesture status by viewing the hand three-dimensional model on the display device. like Figure 4 As shown, a schematic diagram of an updated three-dimensional hand model of an embodiment of the present application is given.

[0051] Under the joint control type, the target interactive object can be determined from the current display content of the display device according to the first screen control instruction, combined with the third display position of each joint point in the previously determined updated hand three-dimensional model of each hand on the display device. Because the actual position and posture of the user's hands are taken into account, this method can more accurately understand the user's intention. After determining the target interactive object, the display content of the display device can be controlled accordingly according to the first screen control instruction and the target interactive object. For example, if the first screen control instruction corresponding to the user's gesture indicates that a file is to be selected and dragged, the file can be located in the file list according to the third display position of each joint point in the hand three-dimensional model on the display device, and the user is allowed to drag it by gesture. The embodiment of the present application can provide users with a more immersive interactive experience by rendering and displaying the three-dimensional model of the user's hands in real time. Users can intuitively see the reflection of their gestures on the display device, so as to more easily understand and control their operations. In addition, the target interactive object is determined in combination with the real-time position and posture of the hand three-dimensional model, which can significantly improve the efficiency and accuracy of the interaction.

[0052] In an embodiment of the present application, optionally, when the current display page of the display device is a virtual scene interaction page, the "determining the target interaction object from the display content of the display device based on the first screen control instruction and the third display position of the remaining joints except the wrist joint in the updated three-dimensional hand model of each hand, and controlling the display content of the display device according to the first screen control instruction and the target interaction object" includes: determining the third display position of the remaining joints of each hand of the user except the wrist joint on the display device based on the updated three-dimensional hand model of each hand, and controlling the display content of the display device according to the third display position of the remaining joints of both hands and the distance between different interactive objects in the virtual scene interaction page. ; According to the first screen control instruction, the target distance condition is determined; when the distance between the third display position of the remaining joints of both hands and any interactive object meets the target distance condition, the any interactive object is determined to be the target interactive object, and based on the continuous multiple frames of hand images after the hand image, the moving trajectory of the target interactive object is determined; according to the first screen control instruction and the moving trajectory, the fourth display position of the target interactive object on the display device is updated until any interactive object does not meet the triggering interaction condition, wherein the triggering interaction condition includes that the user gesture belongs to the type of preset gesture, and the distance between the third display position of the remaining joints of both hands and any interactive object meets the target distance condition.

[0053] In this embodiment, the display page of the display device may be a virtual scene interaction page. Here, the virtual scene interaction page may be a page that displays a variety of interactive objects. By displaying an updated three-dimensional hand model of each hand on the display device and interacting with the interactive objects on the virtual scene interaction page, the real interaction process between the user and the object can be simulated. For example, the interactive object is a ball, and the updated three-dimensional hand model shows actions such as dragging, grabbing, and lifting, then the actions of dragging, grabbing, and lifting the ball can be displayed on the display device.

[0054] When the current display page of the display device is a virtual scene interaction page, first determine the third display positions of the remaining joints of each hand of the user except the wrist joints on the display device based on the updated three-dimensional hand models of the user's hands. These third display positions reflect the user's current hand posture and position, and are the basis for subsequent interaction judgments. Then, based on the distance between the third display positions of the remaining joints of the hands and the different interactive objects in the virtual scene interaction page, it can be determined which interactive object may become the target interactive object.

[0055] Then, according to the first screen control instruction, a target distance condition can be determined. The target distance condition is used to screen out the interactive objects whose distances from the hand joints meet the specific distance requirements set under the condition under the first screen control instruction. When the distances between the third display positions of the remaining joints of both hands and a certain interactive object meet the target distance condition, the interactive object is determined as the target interactive object.

[0056] After the target interactive object is determined, the moving trajectory of the target interactive object can be further tracked based on the continuous multiple frames of hand images after the hand image, in order to more accurately reflect the dynamic interaction process between the user's hand and the target interactive object. Subsequently, the fourth display position of the target interactive object on the display device can be continuously updated according to the first screen control instruction and the tracked moving trajectory, so that the target interactive object shows the effect of being moved by the user on the display device. The interaction process can continue until the target interactive object no longer meets the triggering interaction condition. The triggering interaction condition here can include two aspects: one is that the user gesture belongs to the type of preset gesture, indicating that the user is performing a certain specific interactive operation; the other is that the distance between the third display position of each joint point of both hands and the target interactive object meets the target distance condition, ensuring that the user's hand and the target interactive object are effectively interacted. The embodiment of the present application tracks the three-dimensional model of the user's hand in real time, and judges and triggers the interaction event according to the distance between the user's hand posture and position and the interactive object in the virtual scene interaction page, thereby realizing the user's natural and intuitive interactive experience in the virtual scene.

[0057] In an embodiment of the present application, optionally, the "determining a second display position of a wrist joint of a preset three-dimensional hand model of each hand on the display device according to first point coordinates corresponding to each joint point of both hands of the user" includes: acquiring first target information and second target information, wherein the first target information includes a preset virtual position of a perspective camera for collecting hand images, a preset forward direction vector, and a real distance between the user's hand and the perspective camera, and the second target information includes a field of view angle and a field of view aspect ratio of the perspective camera; according to the first target information, calculating the near plane center of the near plane corresponding to the perspective camera, and According to the actual distance between the user's hand and the perspective camera and the second target information, the near plane height and the near plane width of the near plane corresponding to the perspective camera are calculated; according to a preset coordinate conversion algorithm, the first point coordinates corresponding to the wrist joint point of each hand of the user are converted into normalized device coordinates; based on the near plane center, the near plane height, the near plane width and the normalized device coordinates, the second point coordinates of the first point coordinates corresponding to the wrist joint point of each hand on the near plane are calculated, and according to the second point coordinates, the second display position of the wrist joint point of the preset hand three-dimensional model of each hand on the display device is determined.

[0058] In this embodiment, the first target information includes the preset virtual position of the perspective camera for collecting hand images, the preset front direction vector, and the real distance between the user's hand and the perspective camera. This information is the basis for understanding the spatial relationship between the perspective camera and the user's hand. The second target information includes the field of view angle (FOV, English full name: Field of View) and the aspect ratio of the perspective camera. These parameters determine the field of view range that the camera can capture and the aspect ratio of the image. Among them, the preset virtual position of the perspective camera for collecting hand images can be the placement position of the perspective camera set in the application, and the preset front direction vector can be the front direction vector of the placement position of the perspective camera set in the application. In this way, no matter where the perspective camera is located in the application, the calculation of the near plane here is always calculated in front of the perspective camera. The real distance between the user's hand and the perspective camera can be fixed or dynamically changed. If the distance of the user in front of the perspective camera remains unchanged, then the real distance is fixed; if the distance of the user in front of the perspective camera is constantly changing, then the real distance can be dynamically changing. Here, a laser radar can be set on the perspective camera to measure the real distance between the user and the perspective camera by the laser radar. In addition, the real distance between the user and the perspective camera can also be measured by using a monocular camera, a depth camera, etc. The specific distance measurement method is not limited here. The field of view angle and the field of view aspect ratio of the perspective camera can also be pre-set according to the perspective camera. When the user's hand is within the field of view angle of the perspective camera, the three-dimensional hand model corresponding to the user's gesture can be displayed on the display device; when the user's hand is outside the field of view angle of the perspective camera, the user's hand cannot be detected from the hand image, and the three-dimensional hand model is not displayed on the display device.

[0059] Afterwards, the center point position of the near plane (ie, an imaginary plane in front of the camera, used to project the three-dimensional scene onto the two-dimensional image) in the perspective camera's field of view may be calculated based on the first target information.

[0060] Specifically, the following formula can be used for calculation:

[0061] nearCenter=cameraPosition+cameraForward*viewportPoint.z;

[0062] Among them, nearCenter represents the near plane center coordinates of the near plane corresponding to the perspective camera;

[0063] cameraPosition represents the preset virtual position of the perspective camera; cameraForward represents the preset forward direction vector of the perspective camera; viewportPoint.z represents the real distance between the user's hand and the perspective camera.

[0064] In addition, the height and width of the near plane can be calculated using the real distance between the user's hand and the perspective camera, the camera's field of view angle, and the field of view aspect ratio. These parameters are crucial for the subsequent projection of the three-dimensional coordinates onto the two-dimensional plane. Specifically, the following formula can be used for calculation:

[0065] fovRad = fov*M_PI / 180.0;

[0066] nearHeight=2.0*tan(fovRad / 2.0)*viewportPoint.z;

[0067] nearWidth=nearHeight*aspectRatio;

[0068] Among them, nearHeight represents the height of the near plane; nearWidth represents the width of the near plane; fovRad represents the field of view arc of the perspective camera; aspectRatio represents the aspect ratio of the field of view of the perspective camera; fov represents the field of view angle of the perspective camera.

[0069] Afterwards, the first point coordinates corresponding to the wrist joints of each hand of the user can be converted into normalized device coordinates (NDC) through a preset coordinate conversion algorithm. Normalized device coordinates are a standardized coordinate system used to uniformly process display devices of different sizes and resolutions in a graphics rendering pipeline. Specifically, the formula of the preset coordinate conversion algorithm can be as follows:

[0070] ndcX = viewportPoint.x*2-1;

[0071] ndcY = viewportPoint.y*2-1;

[0072] Among them, ndcX is the normalized device coordinate of the x-axis of the wrist joint point; ndcY is the normalized device coordinate of the y-axis of the wrist joint point; viewportPoint.x and viewportPoint.y are the x-coordinate value and y-coordinate value in the first point coordinate of the wrist joint point.

[0073] Next, based on the near plane center, near plane height, near plane width and normalized device coordinates, the normalized device coordinates of the wrist joint point can be mapped to a specific position on the near plane to obtain the second point coordinates. The calculation formula for the second point coordinates can be as follows:

[0074] worldPointOnNearPlane=nearCenter+right*(ndcX*nearWidth / 2.0)+up*(ndcY*nearHeight / 2.0);

[0075] Among them, worldPointOnNearPlane can be the second point coordinate of the wrist joint on the near plane; right and up can be the standard direction vectors determined by the engine coordinate system, right can be (1,0,0), and up can be (0,1,0). The second point coordinate is also the coordinate under the orthogonal camera. The embodiment of the present application collects the hand image through a perspective camera and restores the position of the hand to the coordinates of the applied perspective camera, so that the hand in the reference and in the real space are the same; then it is converted to the orthogonal camera coordinate system because there is a situation in the perspective camera where the near is large and the far is small, and the unit distance moved in the plane at different positions from the perspective camera has different distances on the projection of the screen (the farther from the camera, the smaller the position moved on the screen), so when converted to the orthogonal camera coordinate system, the distance can be ignored, and as long as it is facing the camera, the distance moved on the screen is consistent with the distance moved in the real world.

[0076] Finally, since the display on the display device has the same scaling principle as the display in the orthogonal camera coordinate system, the final display position of the wrist joint point of the preset hand three-dimensional model of each hand on the display device can be further determined according to the calculated second point coordinates, that is, the second display position. This second display position can be used as the basis for rendering and displaying the user's hand model on the display device in the future, so as to facilitate user interaction or observation. The embodiment of the present application converts the first point coordinates of the wrist joint point into the second point coordinates in the orthogonal coordinate system, and then determines the second display position of the wrist joint point of the preset hand three-dimensional model of each hand on the display device according to the second point coordinates, so as to ensure that the position of the hand three-dimensional model on the display device is spatially consistent with the position of the user's real hand in front of the perspective camera, so that the user can intuitively feel the corresponding relationship between the virtual hand and the real hand in space, which helps to improve the user experience; in the orthogonal camera coordinate system, no matter how far the hand is from the perspective camera, as long as the hand is within the field of view of the perspective camera and facing the camera, the moving distance of the hand on the screen is consistent with the moving distance in the real world. This eliminates the visual distortion caused by distance changes in the perspective camera, making the movement of the hand model more natural and accurate. In addition, the embodiment of the present application supports the dynamic change of the distance between the user's hand and the perspective camera, as long as the user's hand is within the field of view of the perspective camera, which can further enhance the flexibility of the user to control the display device.

[0077] It should be noted that if the actual distance between the user's hand and the perspective camera is fixed, then the steps of obtaining the first target information and the second target information, and calculating the near plane center, and calculating the near plane height and width can be calculated once after starting the application, and there is no need to calculate repeatedly, because the above results are fixed values. That is, each time the second display position of the wrist joint point on the display device is determined, the steps of converting the first point coordinates corresponding to the wrist joint point into normalized device coordinates and the subsequent steps can be directly executed.

[0078] In an embodiment of the present application, optionally, according to the first point coordinates corresponding to each joint point of the user's hands, the hand rotation matrix and the joint rotation matrix corresponding to each of the preset hand three-dimensional models are determined, including: based on the first point coordinates corresponding to each joint point of the user's hands, calculating the first direction vector from the wrist joint point to the first joint point of the middle finger of each hand of the user, the second direction vector from the first joint point of the little finger to the first joint point of the index finger, and the third direction vector in front of the palm, and based on the first direction vector, the second direction vector and the third direction vector, calculating the hand rotation matrix corresponding to each of the preset hand three-dimensional models; for each joint point of the user's hands, calculating the relevant rotation matrix according to the first point coordinates corresponding to the joint point of each hand and its parent node. The current upward vector corresponding to the parent node of the node is cross-multiplied with the current upward vector and the preset original right-facing vector of the parent node to obtain the forward direction vector of the parent node corresponding to the joint point, and the joint rotation matrix of the joint point of each hand relative to the corresponding parent node is calculated according to the current upward vector and the forward direction vector of the parent node of the joint point; accordingly, the corresponding preset hand three-dimensional model is rotated and transformed according to the hand rotation matrix and the joint rotation matrix to obtain an updated hand three-dimensional model, including: according to the hand rotation matrix, the corresponding preset hand three-dimensional model is rotated, and according to the joint rotation matrix, the bending state corresponding to each finger in the preset hand three-dimensional model after the rotation is updated to obtain an updated hand three-dimensional model.

[0079] In this embodiment, the posture of the preset three-dimensional hand model is fixed. In order to ensure consistency between the three-dimensional hand model displayed on the display device and the user's actual gesture, the preset three-dimensional hand model needs to be updated using the hand rotation matrix and the joint rotation matrix. The hand rotation matrix can be calculated by the following steps. First, determine the first direction vector, the second direction vector and the third direction vector. Here, the first direction vector refers to the direction vector from the wrist joint point to the first joint point of the middle finger; the second direction vector refers to the direction vector from the first joint point of the little finger to the first joint point of the index finger; the third direction vector refers to the direction vector in front of the palm, representing the forward facing direction of the palm. Specifically, the above three direction vectors can be calculated using the following formulas:

[0080] V_wrist2middle=normalize(P_middle–P_wrist);

[0081] V_pinky2index=normalize(P_pinky–P_index);

[0082] V_palm=normalize(cross(V_wrist2middle,V_pinky2index));

[0083] Among them, V_wrist2middle represents the first direction vector; P_middle represents the first point coordinate corresponding to the first joint point of the middle finger; P_wrist represents the first point coordinate corresponding to the wrist joint point; V_pinky2index represents the second direction vector; P_pinky represents the first point coordinate corresponding to the first joint point of the little finger; P_index represents the first point coordinate corresponding to the first joint point of the index finger; V_palm represents the third direction vector; cross represents cross product; normalize represents normalization processing.

[0084] Next, using the above three direction vectors, a rotation matrix representing the global orientation of the hand can be constructed. This matrix will be used to rotate the preset three-dimensional hand model to the same orientation as the user's real hand. Specifically, the hand rotation matrix can be calculated by the following method:

[0085] M_rotation1=mat4f::lookAt(float3{0,0,0},V_wrist2middle,V_palm);

[0086] Among them, M_rotation1 represents the hand rotation matrix, which is a 4x4 floating point matrix (mat4f); mat4f::lookAt(...) represents a static member function belonging to the mat4f class, which is used to generate a "view" matrix according to the provided parameters; float3{0,0,0} represents the first parameter of the lookAt function, representing the observation point (i.e. the position of the perspective camera or the eye), which is set to the origin (0,0,0) in the embodiment of the present application; V_wrist2middle represents the second parameter of the lookAt function, which is the first direction vector; V_palm represents the third parameter of the lookAt function, which is the third direction vector. Through the above three parameters, the hand rotation matrix can be generated through the lookAt function.

[0087] In addition, the joint rotation matrix can also be calculated by the following steps. First, for each joint point in each hand, determine the first point coordinates of the joint point and its parent node, and then calculate the current upward vector of the parent node of the joint point based on these two first point coordinates. Specifically, the current upward vector can be calculated using the following formula:

[0088] Vup=normalize(P_child–P);

[0089] Among them, Vup represents the current upward vector; P_child represents the first point coordinate corresponding to the joint point; P represents the first point coordinate corresponding to the parent node of the joint point. It should be noted that the parent node of a joint point can be a joint point adjacent to the joint point and close to the wrist joint point. For example, for the index finger, there are 4 joint points on the index finger, numbered from the fingertip to the wrist, namely joint point 1 to joint point 4. The joint point corresponding to the fingertip is the last joint point, and its parent node is joint point 2. The parent node of joint point 2 is joint point 3, and so on. The above-mentioned model algorithm for identifying hand joint points can obtain the number of the hand and the joint point number of each joint point, as well as the corresponding first point coordinates after identifying the hand image. Therefore, after identifying the first point coordinates of each joint point, it is possible to identify which joint point the first point coordinate corresponds to, and then the current upward vector of the parent node of each joint point can be calculated.

[0090] Next, a cross multiplication operation can be used to calculate the forward direction vector of the parent node corresponding to each joint point by cross-multiplying the current upward vector by a preset original right-facing vector of the parent node. The vector represents the rotation direction of the joint point relative to its parent node. Specifically, the forward direction vector can be calculated using the following formula:

[0091] Vforward=normalize(cross(V_up,V_right));

[0092] Among them, Vforward represents the forward direction vector; V_right represents the original right direction vector of the preset parent node, which can be set according to the original situation of the preset hand 3D model.

[0093] Finally, based on the current upward vector and forward direction vector of each joint point, the joint rotation matrix of the joint point relative to its parent node can be calculated. This matrix will be used to adjust the bending direction and angle of the finger to match the user's real hand posture. Specifically, the joint rotation matrix can be calculated using the following formula:

[0094] M_rotation2=mat4f::lookAt(float3{0,0,0},Vup,Vforward);

[0095] Among them, M_rotation2 represents the joint rotation matrix of each joint point relative to the corresponding parent node.

[0096] After obtaining the hand rotation matrix and the joint rotation matrix, the two matrices can be used to perform a rotation transformation on the preset hand 3D model to obtain an updated hand 3D model. Specifically, the calculated hand rotation matrix can be applied to the preset hand 3D model to adjust its global orientation to be consistent with the orientation of the user's real hand, such as Figure 5 As shown in FIG. 1 , a schematic diagram is given after adjusting the preset three-dimensional hand model according to the hand rotation matrix; then, for each finger in the model, its bending state and angle are adjusted according to its corresponding joint rotation matrix to ensure that the posture of the finger matches the posture of the user's real finger, such as Figure 6 As shown, a schematic diagram of an updated 3D hand model obtained after adjustment according to the joint rotation matrix is ​​given. Through the above steps, the final 3D hand model will accurately reflect the position and posture of the user's real hand, including the global orientation and the bending state of each finger.

[0097] In an embodiment of the present application, optionally, the method also includes: in response to a gesture customization instruction, outputting a gesture customization interface, and determining target customized gesture information based on the customized gesture image uploaded by the user based on the gesture customization interface, and the key joint points marked by the user on the customized gesture image, and sending the target customized gesture information to a target server, so that the target server retrains the preset gesture recognition model based on the target customized gesture information to obtain an updated preset gesture recognition model; and receiving the updated preset gesture recognition model.

[0098] In this embodiment, the user can trigger the gesture customization instruction by clicking a button in the application, issuing a specific voice command, or performing a specific gesture. Afterwards, in response to this gesture customization instruction, a gesture customization interface can be displayed to the user, which can include functional areas such as uploading pictures, marking key points, and previewing gestures. Then, the user can upload the customized gesture images they want to be recognized by the system on this interface. These customized gesture images can be static or dynamic, for example, extracted through video frames. The user can then mark the location of the key joint points on the uploaded customized gesture image. Subsequently, the customized gesture images uploaded by the user and the marked key points can be encapsulated into target customized gesture information so as to be sent to the target server for further processing. After the target server receives the target customized gesture information, it can use this information to retrain the already trained preset gesture recognition model. This process may include adjusting the parameters of the model, etc., so that the model can better recognize the user-defined customized gestures. After training, the target server obtains an updated preset gesture recognition model, which includes the recognition capability of the user-defined customized gestures.

[0099] Furthermore, the updated preset gesture recognition model transmitted back by the target server can be received and then redeployed. After successful deployment, the updated preset gesture recognition model can be used to recognize a wider set of gestures, including user-defined custom gestures. The embodiment of the present application expands the recognition capability of gesture recognition by allowing users to customize gesture images and key joints; by allowing users to upload and mark customized gesture information, this information can be used to train and improve the preset gesture recognition model, thereby achieving accurate recognition of user-defined gestures. This approach improves the flexibility and personalization of gesture recognition, thereby better adapting to the needs and scenarios of different users.

[0100] Further, as Figure 1 The specific implementation of the method, the embodiment of the present application provides a display device control device based on gestures, such as Figure 7 As shown, the device comprises:

[0101] A hand image acquisition module, used to continuously acquire a user's hand image;

[0102] An image recognition module, used to perform image recognition on the hand image to obtain first point coordinates corresponding to each joint point of the user's hand;

[0103] A gesture recognition module, configured to determine whether the user gesture in the hand image belongs to a preset gesture type according to the first point coordinates corresponding to each joint point and through a preset gesture recognition model;

[0104] A control module is used to determine a first screen control instruction based on the user gesture when the user gesture belongs to the type of preset gesture, and control the display device through the first screen control instruction.

[0105] Optionally, the device further comprises:

[0106] A monitoring module, for obtaining a hand image of the user at a preset number of frames after a current hand image when the user gesture belongs to a preset gesture type and when the preset gesture type corresponding to the user gesture is a trigger type of a hand trajectory monitoring instruction;

[0107] A hand trajectory determination module is used to determine the hand trajectory corresponding to the user's hand based on the first point coordinates corresponding to each joint point of the user's hand in the current hand image and the first point coordinates corresponding to each joint point of the user's hand in each frame of the user's hand image, and based on the hand trajectory, determine the second screen control instruction, and control the display device through the second screen control instruction.

[0108] Optionally, the hand image includes both hands of the user, and the gesture recognition module is used to:

[0109] Inputting first point coordinates corresponding to joints of both hands of the user into a preset gesture recognition model, and determining whether the two-hand gesture in the hand image belongs to the type of preset gesture through the preset gesture recognition model;

[0110] Accordingly, the device further comprises:

[0111] A key joint point determination module, used for determining the key joint points of the hands corresponding to each preset gesture before inputting the first point coordinates corresponding to each joint point of the hands of the user into the preset gesture recognition model, and obtaining training samples corresponding to each preset gesture according to the key joint points of the hands, wherein the key joint points are necessary joint points for recognizing the preset gesture;

[0112] The training module is used to train the initial gesture recognition model according to the training samples corresponding to different preset gestures to obtain the preset gesture recognition model.

[0113] Optionally, when the type of the preset gesture corresponding to the user gesture is a joint control type, the device further includes:

[0114] A first display module, configured to determine a first display position of a target guide mark on the display device according to first point coordinates corresponding to target joint points of both hands of the user, and to display the target guide mark at the first display position of the display device;

[0115] Accordingly, the control module is used to:

[0116] Based on the first screen control instruction and the first display position, a target interactive object is determined from display content of the display device, and the display content of the display device is controlled according to the first screen control instruction and the target interactive object.

[0117] Optionally, when the type of the preset gesture corresponding to the user gesture is a joint control type, the device further includes:

[0118] A second display module is used to determine the second display position of the wrist joint point of the preset hand three-dimensional model of each hand on the display device according to the first point coordinates corresponding to the joint points of the two hands of the user, and determine the hand rotation matrix and the joint rotation matrix corresponding to each of the preset hand three-dimensional models, perform a rotation transformation on the corresponding preset hand three-dimensional model according to the hand rotation matrix and the joint rotation matrix to obtain an updated hand three-dimensional model, and display the updated hand three-dimensional model of each hand on the display device based on the second display position;

[0119] Accordingly, the control module is used to:

[0120] Based on the first screen control instruction and the third display position of the joints except the wrist joints in the updated three-dimensional hand model of each hand on the display device, the target interaction object is determined from the display content of the display device, and the display content of the display device is controlled according to the first screen control instruction and the target interaction object.

[0121] Optionally, when the current display page of the display device is a virtual scene interaction page, the control module is further used to:

[0122] Based on the updated three-dimensional hand model of each hand, determine the third display position of each joint point of each hand of the user except the wrist joint point on the display device, and according to the distance between the third display position of the remaining joint points of both hands and different interactive objects in the virtual scene interaction page;

[0123] According to the first screen control instruction, a target distance condition is determined; when the distance between the third display position of the remaining joints of both hands and any interactive object meets the target distance condition, the any interactive object is determined to be the target interactive object; and based on multiple consecutive frames of hand images after the hand image, a moving trajectory of the target interactive object is determined; according to the first screen control instruction and the moving trajectory, a fourth display position of the target interactive object on the display device is updated until any interactive object does not meet the triggering interaction condition, wherein the triggering interaction condition includes that the user gesture belongs to a preset gesture type, and the distance between the third display position of the remaining joints of both hands and any interactive object meets the target distance condition.

[0124] Optionally, the second display module is used to:

[0125] Acquire first target information and second target information, wherein the first target information includes a preset virtual position of a perspective camera for collecting hand images, a preset forward direction vector, and a real distance between the user's hand and the perspective camera, and the second target information includes a field of view angle and a field of view aspect ratio of the perspective camera;

[0126] Calculate the near plane center of the near plane corresponding to the perspective camera according to the first target information, and calculate the near plane height and near plane width of the near plane corresponding to the perspective camera according to the real distance between the user's hand and the perspective camera and the second target information;

[0127] According to a preset coordinate conversion algorithm, the first point coordinates corresponding to the wrist joint points of each hand of the user are converted into normalized device coordinates;

[0128] Based on the near plane center, the near plane height, the near plane width and the normalized device coordinates, the second point coordinates on the near plane corresponding to the first point coordinates of the wrist joint point of each hand are calculated, and based on the second point coordinates, the second display position of the wrist joint point of the preset hand three-dimensional model of each hand on the display device is determined.

[0129] Optionally, the second display module is further used for:

[0130] Based on the first point coordinates corresponding to the joints of both hands of the user, a first direction vector from the wrist joint point to the first joint point of the middle finger, a second direction vector from the first joint point of the little finger to the first joint point of the index finger, and a third direction vector in front of the palm of each hand of the user are calculated, and based on the first direction vector, the second direction vector and the third direction vector, a hand rotation matrix corresponding to each of the preset three-dimensional hand models is calculated;

[0131] For each joint point of the user's hands, according to the joint point of each hand and the first point coordinates corresponding to the parent node of the joint point, calculate the current upward vector corresponding to the parent node of the joint point, perform cross multiplication processing on the current upward vector and the preset original right-facing vector of the parent node to obtain the forward direction vector of the parent node corresponding to the joint point, and according to the current upward vector and the forward direction vector of the parent node of the joint point, calculate the joint rotation matrix of the joint point of each hand relative to the corresponding parent node;

[0132] Accordingly, the second display module is further used for:

[0133] According to the hand rotation matrix, the corresponding preset three-dimensional hand model is rotated, and according to the joint rotation matrix, the bending state corresponding to each finger in the preset three-dimensional hand model after the rotation is updated to obtain an updated three-dimensional hand model.

[0134] Optionally, the device further comprises:

[0135] an instruction response module, configured to respond to a gesture customization instruction, output a gesture customization interface, and determine target customized gesture information according to a customized gesture image uploaded by the user based on the gesture customization interface and key joint points marked by the user on the customized gesture image, and send the target customized gesture information to a target server, so that the target server retrains the preset gesture recognition model based on the target customized gesture information to obtain an updated preset gesture recognition model;

[0136] The receiving module is used to receive the updated preset gesture recognition model.

[0137] It should be noted that for other corresponding descriptions of the functional units involved in the gesture-based display device control device provided in the embodiment of the present application, reference can be made to Figures 1 to 6 The corresponding description in the method will not be repeated here.

[0138] The present application also provides a computer device, which may be a personal computer, a server, a network device, etc. Figure 8 As shown, the computer device includes a bus, a processor, a memory and a communication interface, and may also include an input and output interface and a display device. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The database of the computer device is used to store location information. The network interface of the computer device is used to communicate with an external terminal through a network connection. When the computer program is executed by the processor, the steps in each method embodiment are implemented.

[0139] Those skilled in the art will understand that Figure 8 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0140] In one embodiment, a computer-readable storage medium is provided. The computer-readable storage medium may be non-volatile or volatile, and stores a computer program thereon. When the computer program is executed by a processor, the steps in the above-mentioned method embodiments are implemented.

[0141] In one embodiment, a computer program product is provided, including a computer program, which implements the steps in the above method embodiments when executed by a processor.

[0142] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties.

[0143] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to the memory, database or other medium used in the embodiments provided in the present application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetoresistive random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. As an illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM). The database involved in each embodiment provided in this application may include at least one of a relational database and a non-relational database. Non-relational databases may include distributed databases based on blockchains, etc., but are not limited to this. The processor involved in each embodiment provided in this application may be a general-purpose processor, a central processing unit, a graphics processor, a digital signal processor, a programmable logic device, a data processing logic device based on quantum computing, etc., but are not limited to this.

[0144] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0145] The above-described embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the present application. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the present application shall be subject to the attached claims.

Claims

1. A gesture-based display device control method, characterized in that: include: Continuously obtain the user's hand image; Performing image recognition on the hand image to obtain first point coordinates corresponding to each joint point of the user's hand; Determining whether the user gesture in the hand image belongs to a preset gesture type by using a preset gesture recognition model according to the first point coordinates corresponding to each joint point; When the user gesture belongs to the type of preset gesture, a first screen control instruction is determined based on the user gesture, and the display device is controlled by the first screen control instruction.

2. The method according to claim 1, characterized in that When the user gesture belongs to the type of preset gesture, the method further includes: When the type of the preset gesture corresponding to the user gesture is a trigger type of a hand trajectory monitoring instruction, obtaining a hand image of the user at a preset number of frames after a current hand image; According to the first point coordinates corresponding to each joint of the user's hand in the current hand image and the first point coordinates corresponding to each joint of the user's hand in each frame of the user's hand image, the hand trajectory corresponding to the user's hand is determined, and based on the hand trajectory, a second screen control instruction is determined, and the display device is controlled by the second screen control instruction.

3. The method according to claim 1, characterized in that The hand image includes both hands of the user, and determining whether the user gesture in the hand image belongs to the type of preset gesture by a preset gesture recognition model according to the first point coordinates corresponding to each joint point includes: Inputting first point coordinates corresponding to joints of both hands of the user into a preset gesture recognition model, and determining whether the two-hand gesture in the hand image belongs to the type of preset gesture through the preset gesture recognition model; Accordingly, before inputting the first point coordinates corresponding to the joints of the user's hands into the preset gesture recognition model, the method further includes: Determine the key joint points of both hands corresponding to each preset gesture, and obtain training samples corresponding to each preset gesture according to the key joint points of both hands, wherein the key joint points are necessary joint points for identifying the preset gesture; The initial gesture recognition model is trained according to the training samples corresponding to different preset gestures to obtain the preset gesture recognition model.

4. The method according to claim 3, characterized in that When the type of the preset gesture corresponding to the user gesture is a joint control type, the method further includes: Determining a first display position of a target guide mark on the display device according to first point coordinates corresponding to target joint points of both hands of the user, and displaying the target guide mark at the first display position of the display device; Correspondingly, controlling the display device through the first screen control instruction includes: Based on the first screen control instruction and the first display position, a target interactive object is determined from display content of the display device, and the display content of the display device is controlled according to the first screen control instruction and the target interactive object.

5. The method according to claim 3, characterized in that: When the type of the preset gesture corresponding to the user gesture is a joint control type, the method further includes: Determine the second display position of the wrist joint point of the preset three-dimensional hand model of each hand on the display device according to the first point coordinates corresponding to the joint points of the two hands of the user, and determine the hand rotation matrix and joint rotation matrix corresponding to each of the preset three-dimensional hand models, perform rotation transformation on the corresponding preset three-dimensional hand model according to the hand rotation matrix and the joint rotation matrix to obtain an updated three-dimensional hand model, and display the updated three-dimensional hand model of each hand on the display device based on the second display position; Correspondingly, controlling the display device through the first screen control instruction includes: Based on the first screen control instruction and the third display position of the joints except the wrist joints in the updated three-dimensional hand model of each hand on the display device, the target interaction object is determined from the display content of the display device, and the display content of the display device is controlled according to the first screen control instruction and the target interaction object.

6. The method according to claim 5, characterized in that When the current display page of the display device is a virtual scene interaction page, the target interaction object is determined from the display content of the display device based on the first screen control instruction and the third display position of the remaining joints except the wrist joint in the updated three-dimensional hand model of each hand on the display device, and the display content of the display device is controlled according to the first screen control instruction and the target interaction object, including: Based on the updated three-dimensional hand model of each hand, determine the third display position of each joint point of each hand of the user except the wrist joint point on the display device, and according to the distance between the third display position of the remaining joint points of both hands and different interactive objects in the virtual scene interaction page; According to the first screen control instruction, a target distance condition is determined; when the distance between the third display position of the remaining joints of both hands and any interactive object meets the target distance condition, the any interactive object is determined to be the target interactive object; and based on multiple consecutive frames of hand images after the hand image, a moving trajectory of the target interactive object is determined; according to the first screen control instruction and the moving trajectory, a fourth display position of the target interactive object on the display device is updated until any interactive object does not meet the triggering interaction condition, wherein the triggering interaction condition includes that the user gesture belongs to a preset gesture type, and the distance between the third display position of the remaining joints of both hands and any interactive object meets the target distance condition.

7. The method according to claim 5, characterized in that The step of determining the second display position of the wrist joint point of the preset three-dimensional hand model of each hand on the display device according to the first point coordinates corresponding to the joint points of the two hands of the user includes: Acquire first target information and second target information, wherein the first target information includes a preset virtual position of a perspective camera for collecting hand images, a preset forward direction vector, and a real distance between the user's hand and the perspective camera, and the second target information includes a field of view angle and a field of view aspect ratio of the perspective camera; Calculate the near plane center of the near plane corresponding to the perspective camera according to the first target information, and calculate the near plane height and near plane width of the near plane corresponding to the perspective camera according to the real distance between the user's hand and the perspective camera and the second target information; According to a preset coordinate conversion algorithm, the first point coordinates corresponding to the wrist joint points of each hand of the user are converted into normalized device coordinates; Based on the near plane center, the near plane height, the near plane width and the normalized device coordinates, the second point coordinates on the near plane corresponding to the first point coordinates of the wrist joint point of each hand are calculated, and based on the second point coordinates, the second display position of the wrist joint point of the preset hand three-dimensional model of each hand on the display device is determined.

8. The method according to claim 5, characterized in that Determining the hand rotation matrix and the joint rotation matrix corresponding to each of the preset hand three-dimensional models according to the first point coordinates corresponding to the joint points of the two hands of the user, including: Based on the first point coordinates corresponding to the joints of both hands of the user, a first direction vector from the wrist joint point to the first joint point of the middle finger, a second direction vector from the first joint point of the little finger to the first joint point of the index finger, and a third direction vector in front of the palm of each hand of the user are calculated, and based on the first direction vector, the second direction vector and the third direction vector, a hand rotation matrix corresponding to each of the preset three-dimensional hand models is calculated; For each joint point of the user's hands, according to the joint point of each hand and the first point coordinates corresponding to the parent node of the joint point, calculate the current upward vector corresponding to the parent node of the joint point, perform cross multiplication processing on the current upward vector and the preset original right-facing vector of the parent node to obtain the forward direction vector of the parent node corresponding to the joint point, and according to the current upward vector and the forward direction vector of the parent node of the joint point, calculate the joint rotation matrix of the joint point of each hand relative to the corresponding parent node; Correspondingly, the rotation transformation of the corresponding preset hand three-dimensional model is performed according to the hand rotation matrix and the joint rotation matrix to obtain an updated hand three-dimensional model, including: According to the hand rotation matrix, the corresponding preset three-dimensional hand model is rotated, and according to the joint rotation matrix, the bending state corresponding to each finger in the preset three-dimensional hand model after the rotation is updated to obtain an updated three-dimensional hand model.

9. The method according to claim 1, characterized in that: The method further comprises: In response to the gesture customization instruction, a gesture customization interface is output, and according to the customized gesture image uploaded by the user based on the gesture customization interface and the key joint points marked by the user on the customized gesture image, target customized gesture information is determined, and the target customized gesture information is sent to a target server, so that the target server retrains the preset gesture recognition model based on the target customized gesture information to obtain an updated preset gesture recognition model; Receive updated preset gesture recognition models.

10. A gesture-based display device control device, characterized in that: include: A hand image acquisition module, used to continuously acquire a user's hand image; An image recognition module, used to perform image recognition on the hand image to obtain first point coordinates corresponding to each joint point of the user's hand; A gesture recognition module, configured to determine whether the user gesture in the hand image belongs to a preset gesture type according to the first point coordinates corresponding to each joint point and through a preset gesture recognition model; A control module is used to determine a first screen control instruction based on the user gesture when the user gesture belongs to the type of preset gesture, and control the display device through the first screen control instruction.

11. A storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the method according to any one of claims 1 to 9 is implemented.

12. A computer device comprising a storage medium, a processor, and a computer program stored in the storage medium and executable on the processor, characterized in that: When the processor executes the computer program, the method according to any one of claims 1 to 9 is implemented.

Images