A large-screen interactive method, device and equipment based on hardware combination

Image frames are collected by the camera device for hand recognition and classification, hardware type and posture are determined, and the operating system is controlled to execute corresponding instructions, which realizes the interaction between the user and the large screen, solves the problem of limited hardware transformation, and improves the interactive experience.

CN113448443BActive Publication Date: 2024-11-15TSINGHUA UNIVERSITY
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202110785576.3
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2021-07-12
Publication Date
2024-11-15
Estimated Expiration
2041-07-12

AI Technical Summary

Technical Problem

The prior art is difficult to realize the interaction between users and large screens without modifying the hardware, especially due to the limitations of the hardware's own body type.

Method used

By obtaining image frames collected by the camera device on the large screen, performing hand recognition and classification, determining the hardware type and grip posture, controlling the operating system to execute corresponding functional instructions, and realizing the interaction between the user and the large screen.

Benefits of technology

Without any form of hardware transformation, users' interactive experience with large screens is effectively improved, and the problem of limited hardware transformation is solved.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN113448443B_ABST
    Figure CN113448443B_ABST
Patent Text Reader

Abstract

The present application discloses a large-screen interaction method, device and equipment based on hardware combination, which obtains image frames collected by a camera device preset on a large screen, and the image frames are used to indicate people. The hand recognition of the person is performed to obtain a hand image, and the hand image is used to indicate the hand holding the hardware and the various key points of the hand. The hand image is input into the classification model to obtain the classification result output by the classification model. The hardware type with the highest probability is used as the hardware type of the hardware shown in the image frame. When the user's trigger operation on the large screen is obtained, the operating system is controlled to execute the functional instructions corresponding to the hardware type of the hardware shown in the image frame and the holding posture of the hand shown in the image frame. It can be seen that by using the scheme described in the present application, the user's interaction with the large screen can be realized in combination with the hardware without any form of hardware modification, thereby effectively improving the user's experience.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present application relates to the field of human-computer interaction, and in particular to a large-screen interaction method, device and equipment based on hardware integration. Background Art

[0002] The current usage of large screens (usually touch screens) is that users trigger the large screens, causing the operating system to execute functional instructions corresponding to the triggering operations. When using large screens, users often hold hardware such as pens and erasers in their hands. How to combine hardware to achieve user interaction with large screens is a hot topic in this field.

[0003] Currently, existing technical solutions usually involve hardware modification, such as installing sensors inside the hardware. Some hardware is limited by its size and is difficult to modify.

[0004] Therefore, how to combine the hardware user's interaction with the large screen without any form of hardware modification has become an urgent problem to be solved in this field. Summary of the invention

[0005] The present application provides a large-screen interaction method, device and equipment based on hardware combination, which is used to realize user interaction with a large screen in combination with hardware without any form of hardware modification.

[0006] In order to achieve the above objectives, this application provides the following technical solutions:

[0007] A large screen interaction method based on hardware combination, comprising:

[0008] Acquire an image frame captured by a camera device preset on a large screen; the image frame is used to indicate a person;

[0009] Performing hand recognition on the person to obtain a hand image; the hand image is used to indicate the hand holding the hardware and various key points of the hand;

[0010] The hand image is input into a classification model to obtain a classification result output by the classification model; the classification model is trained based on taking the hand image of the sample image frame as input and taking the hardware type of the hardware shown in the sample image frame and the holding posture of the hand shown in the sample image frame as training targets; the classification result includes the probability of each hardware type and the holding posture of the hand;

[0011] Using the hardware type with the highest probability as the hardware type of the hardware shown in the image frame;

[0012] When a trigger operation of the user on the large screen is obtained, the operating system is controlled to execute a functional instruction corresponding to the hardware type of the hardware shown in the image frame and corresponding to the holding posture of the hand shown in the image frame.

[0013] Optionally, also include:

[0014] The hand image is input into a hardware detection model to obtain a detection result output by the hardware detection model; the hardware detection model is trained based on taking the hand image of the sample image frame as input and taking the mask of the hardware shown in the sample image frame marked in advance as a training target; the detection result includes the mask of the hardware shown in the image frame;

[0015] Based on each hand key point of the hand shown in the target image frame and the mask of the hardware shown in the target image frame, determining whether an interaction occurs between the user and the hardware shown in the target image frame; the target image frame is an image frame collected by the camera device within a preset time period;

[0016] When the interaction occurs between the user and the hardware shown in the target image frame, the operating system is controlled to execute functional instructions corresponding to the hardware type of the hardware shown in the target image frame and corresponding to the holding posture of the hand shown in the target image frame.

[0017] Optionally, the target image frame includes a first image frame, a second image frame, and a third image frame; a timestamp of the first image frame is earlier than a timestamp of the second image frame, and a timestamp of the second image frame is earlier than a timestamp of the third image frame;

[0018] The determining whether an interaction occurs between the user and the hardware shown in the target image frame based on each hand key point of the hand shown in the target image frame and the mask of the hardware shown in the target image frame includes:

[0019] Calculate the shortest distance between the target hand key point shown in the first image frame and the mask shown in the first image frame to obtain a first value; the target hand key point is: a hand key point indicating the tip of the index finger among the various hand key points of the hand;

[0020] Calculate the shortest distance between the target hand key point shown in the second image frame and the mask shown in the second image frame to obtain a second value;

[0021] Calculate the shortest distance between the target hand key point shown in the third image frame and the mask shown in the third image frame to obtain a third value;

[0022] Calculating a time difference between a timestamp of the third image frame and a timestamp of the first image frame to obtain a fourth value;

[0023] When the fourth value is less than the first preset threshold, and the first value is less than the second preset threshold, and the second value is greater than the third preset threshold, and the third value is less than the second preset threshold, it is determined that an interaction occurs between the user and the hardware shown in the target image frame; the interaction is used to characterize that the user uses the index finger to perform a click operation on the hardware shown in the target image frame.

[0024] Optionally, the performing hand recognition on the person to obtain a hand image includes:

[0025] The image frame is input into a preset hand recognition model to obtain a hand image output by the hand recognition model; wherein the hand recognition model is used to: perform hand recognition on the person shown in the image frame, mark each hand key point in the image frame, and select the hand image using a minimum bounding box algorithm.

[0026] A large-screen interactive device based on hardware combination, comprising:

[0027] An acquisition unit, used to acquire image frames collected by a camera device preset on the large screen; the image frames are used to indicate a person;

[0028] A recognition unit, used to perform hand recognition on the person to obtain a hand image; the hand image is used to indicate the hand holding the hardware and various key points of the hand;

[0029] A classification unit, configured to input the hand image into a classification model to obtain a classification result output by the classification model; the classification model is trained based on taking the hand image of a sample image frame as input and taking the hardware type of the hardware shown in the sample image frame and the holding posture of the hand shown in the sample image frame as training targets; the classification result includes the probability of each hardware type and the holding posture of the hand;

[0030] a determining unit, configured to use the hardware type with the highest probability as the hardware type of the hardware shown in the image frame;

[0031] The first control unit is used to control the operating system to execute functional instructions corresponding to the hardware type of the hardware shown in the image frame and the holding posture of the hand shown in the image frame when a trigger operation of the user on the large screen is obtained.

[0032] Optionally, also include:

[0033] A detection unit, configured to input the hand image into a hardware detection model to obtain a detection result output by the hardware detection model; the hardware detection model is trained based on taking the hand image of the sample image frame as input and taking the mask of the hardware shown in the sample image frame marked in advance as a training target; the detection result includes the mask of the hardware shown in the image frame;

[0034] A judging unit, configured to judge whether an interaction occurs between the user and the hardware shown in the target image frame based on each hand key point of the hand shown in the target image frame and a mask of the hardware shown in the target image frame; the target image frame is an image frame collected by the camera device within a preset time period;

[0035] A second control unit is used to control the operating system to execute functional instructions corresponding to the hardware type of the hardware shown in the target image frame and to the holding posture of the hand shown in the target image frame when the interaction behavior occurs between the user and the hardware shown in the target image frame.

[0036] Optionally, the target image frame includes a first image frame, a second image frame, and a third image frame; a timestamp of the first image frame is earlier than a timestamp of the second image frame, and a timestamp of the second image frame is earlier than a timestamp of the third image frame;

[0037] The judging unit is specifically used for:

[0038] Calculate the shortest distance between the target hand key point shown in the first image frame and the mask shown in the first image frame to obtain a first value; the target hand key point is: a hand key point indicating the tip of the index finger among the various hand key points of the hand;

[0039] Calculate the shortest distance between the target hand key point shown in the second image frame and the mask shown in the second image frame to obtain a second value;

[0040] Calculate the shortest distance between the target hand key point shown in the third image frame and the mask shown in the third image frame to obtain a third value;

[0041] Calculating a time difference between a timestamp of the third image frame and a timestamp of the first image frame to obtain a fourth value;

[0042] When the fourth value is less than the first preset threshold, and the first value is less than the second preset threshold, and the second value is greater than the third preset threshold, and the third value is less than the second preset threshold, it is determined that an interaction occurs between the user and the hardware shown in the target image frame; the interaction is used to characterize that the user uses the index finger to perform a click operation on the hardware shown in the target image frame.

[0043] Optionally, the identification unit is specifically used for:

[0044] The image frame is input into a preset hand recognition model to obtain a hand image output by the hand recognition model; wherein the hand recognition model is used to: perform hand recognition on the person shown in the image frame, mark each hand key point in the image frame, and select the hand image using a minimum bounding box algorithm.

[0045] A computer-readable storage medium includes a stored program, wherein the program executes the large-screen interaction method based on hardware combination.

[0046] A large-screen interactive device based on hardware combination, comprising: a processor, a memory and a bus; the processor and the memory are connected via the bus;

[0047] The memory is used to store programs, and the processor is used to run programs, wherein the large-screen interaction method based on hardware integration is executed when the program is running.

[0048] The technical solution provided by the present application obtains an image frame collected by a camera device preset on a large screen, and the image frame is used to indicate a person. The hand recognition of the person is performed to obtain a hand image, and the hand image is used to indicate the hand holding the hardware and each key point of the hand. The hand image is input into the classification model to obtain the classification result output by the classification model. The classification model is based on taking the hand image of the sample image frame as input, and taking the hardware type of the hardware shown in the pre-annotated sample image frame and the holding posture of the hand shown in the sample image frame as training targets, and training is obtained. The classification result includes the probability of each hardware type and the holding posture of the hand. The hardware type with the highest probability is taken as the hardware type of the hardware shown in the image frame. When the user's trigger operation on the large screen is obtained, the operating system is controlled to execute the functional instructions corresponding to the hardware type of the hardware shown in the image frame and the holding posture of the hand shown in the image frame. It can be seen that by using the solution described in the present application, the user's interaction with the large screen can be realized in combination with the hardware without any form of hardware modification, effectively improving the user's experience. BRIEF DESCRIPTION OF THE DRAWINGS

[0049] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0050] Figure 1a A schematic diagram of a large-screen interaction method based on hardware integration provided in an embodiment of the present application;

[0051] Figure 1b A schematic diagram of a large screen provided in an embodiment of the present application;

[0052] Figure 1c A schematic diagram of another large screen provided in an embodiment of the present application;

[0053] Figure 1d A schematic diagram of hand recognition provided in an embodiment of the present application;

[0054] Figure 1e A schematic diagram of a hand image provided in an embodiment of the present application;

[0055] Figure 1f A schematic diagram of a sample image frame provided in an embodiment of the present application;

[0056] Figure 1g A schematic diagram of another sample image frame provided in an embodiment of the present application;

[0057] Figure 1h A schematic diagram of a holding posture provided in an embodiment of the present application;

[0058] Figure 1i A schematic diagram of another holding posture provided in an embodiment of the present application;

[0059] Figure 1j A schematic diagram of another holding posture provided in an embodiment of the present application;

[0060] Figure 2 A schematic diagram of another large-screen interaction method based on hardware integration provided in an embodiment of the present application;

[0061] Figure 3 A schematic diagram of the structure of a large-screen interactive device based on hardware integration provided in an embodiment of the present application. DETAILED DESCRIPTION

[0062] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0063] like Figure 1a As shown, a large-screen interaction method based on hardware combination provided in an embodiment of the present application includes the following steps:

[0064] S101: Acquire sample image frames captured by a camera device preset on a large screen.

[0065] The camera device includes but is not limited to being placed directly above the large screen, and the camera device can capture environmental information in front of the large screen. For details, the installation location of the camera device and the environmental information that can be captured can be found in Figure 1b and Figure 1c As shown (in Figure 1b and Figure 1c In the example, “fisheye lens camera” represents the camera device, “large screen” represents the large screen, and “Interactive space” represents the environment. For example, when a user clicks the large screen, from raising the hand to touching it, the camera device has a field of view of 180 degrees and captures images at a frame rate of 30 frames per second.

[0066] It should be noted that the camera device includes but is not limited to an RGB fisheye camera. In the embodiment of the present application, the sample image frame is used to indicate a person.

[0067] S102: Label the sample image frame with a type identifier.

[0068] The type identifier is used to indicate the hardware type of the hardware shown in the sample image frame.

[0069] S103: Inputting the sample image frame into a preset hand recognition model to obtain a hand image output by the hand recognition model.

[0070] Among them, the preset hand recognition model includes but is not limited to the hand recognition neural network model in the Mediapipe framework (the model has been pre-trained on a wide range of human posture image data sets, so the official model parameters can be directly used without retraining the model). The hand recognition neural network model is used to recognize the hands of the person shown in the sample image frame, and mark the various hand key points of the person's hands in the sample image frame, that is, output the plane coordinates of each hand key point of the person (including horizontal and vertical coordinates), and use the minimum bounding box algorithm to select the person's hand image.

[0071] In addition, in order to ensure that the hand image can cover the hardware, the rectangular box output by the minimum bounding box algorithm is enlarged, and the area selected by the enlarged rectangular box is used as the hand image of the person. Compared with the area selected by the rectangular box before enlargement, the area selected by the enlarged rectangular box is larger and has a higher possibility of covering the hardware.

[0072] Specifically, the hand recognition neural network model is used to mark the key points of the hand of the person in the sample image frame, and to select the hand image of the person using the minimum bounding box algorithm, such as Figure 1d shown.

[0073] It should be noted that the hand image is used to indicate the hand holding the hardware and various key points of the hand, specifically, hand images of a hand holding a pen, a hand holding an eraser, and an empty hand, such as Figure 1e shown.

[0074] S104: Taking the hand image of the sample image frame as input, taking the hardware type of the hardware shown in the sample image frame as a training target, and training to obtain a classification model.

[0075] The network structure of the classification model includes but is not limited to: VGG19 (usually used as a skeleton structure), 16 convolutional layers, and 3 fully connected layers. Specifically, when training the classification model, the Adam optimizer can be used to reduce the cross entropy of the loss function to improve the training efficiency of the classification model.

[0076] S105: marking a mask for the hardware shown in the sample image frame.

[0077] The mask represents a polygon, and the polygon contains hardware. Specifically, assuming that the hardware shown in the sample image frame is an eraser, the eraser annotation mask is used, and the annotation result is as follows: Figure 1f As shown in the figure, assuming that the hardware shown in the sample image frame is a pen, the pen annotation mask is used, and the annotation result is as follows Figure 1g shown.

[0078] S106: Taking the hand image of the sample image frame as input, and taking the mask of the hardware shown in the sample image frame as a training target, training to obtain a hardware detection model.

[0079] The hardware detection model includes but is not limited to a Mask R-CNN network model. During the training of the hardware detection model, the training efficiency of the hardware detection model can be improved by reducing the average binary cross entropy.

[0080] S107: When it is detected that the operating system is turned on, image frames captured by the camera device are acquired according to a preset time interval.

[0081] When the operating system is turned on, it is determined that the large screen has entered the human-computer interaction state. The preset time interval can also be understood as a preset frame rate. In the field of videography, the frame rate is usually used to represent the time interval.

[0082] In the embodiment of the present application, the image frame is used to indicate a person. The person shown in the image frame can also be understood as a user.

[0083] S108: Input the image frame into the hand recognition model to obtain a hand image output by the hand recognition model.

[0084] Among them, if the total number of characters indicated by the image frame is n (that is, there are n characters in front of the large screen), and n is a positive integer greater than 1, then the total number of hand images ranges from n to 2n (that is, both hands of the character), and each hand image corresponds to a character.

[0085] S109: Input the hand image of the image frame into the classification model to obtain the classification result output by the classification model.

[0086] The classification results include the probability of each hardware type and the hand holding posture.

[0087] In an embodiment of the present application, the classification result output by the classification model can be specifically represented by a set of vectors. The number of elements contained in the vector represents the total number of hardware types, and each element represents the probability of a hardware type appearing in an image frame.

[0088] S110: Taking the hardware type with the highest probability as the hardware type of the hardware shown in the image frame.

[0089] The hardware type and holding posture of the hardware shown in each image frame may be stored in a preset queue in the order of the timestamps of the image frames from early to late.

[0090] S111: Inputting the hand image of the image frame into the hardware detection model to obtain the detection result output by the hardware detection model.

[0091] The detection result includes a mask of the hardware shown in the image frame and various hand key points of the hand shown in the image frame.

[0092] It should be noted that if the image frame contains n hand images and the n hand images are input into the hardware detection model, the mask shown in each hand image and the various hand key points of the hand shown in each hand image will be obtained.

[0093] S112: When a trigger operation of the user on the large screen is obtained, the operating system is controlled to execute a function instruction corresponding to the hardware type of the hardware shown in the image frame and corresponding to the holding posture of the hand shown in the image frame.

[0094] The corresponding relationship between the hardware type, the holding posture, and the function instruction can be set by the technician according to the actual situation. For example, if the hardware type is a pen and the holding posture is holding the pen head (such as Figure 1h ), the corresponding function instruction is to switch to the brush. If the hardware type is a pen and the holding posture is to hold the pen end (such as Figure 1i ), the corresponding function instruction is to switch to the fluorescent brush. If the hardware type is a pen and the holding posture is to hold the pen and click the large screen (such as Figure 1j As shown), the corresponding function instruction is to switch to eraser mode. If the hardware type is an eraser and the holding posture is to hold an eraser, the corresponding function instruction is to switch to eraser mode. If the hardware type is a red pen and the holding posture is to hold a pen, the corresponding function instruction is to change the brush color bar to red.

[0095] Optionally, when it is detected that the image frame contains multiple hand images (i.e., there are multiple characters in front of the large screen, and each character holds a piece of hardware), the hand position of each character is calculated based on the plane coordinates of the key hand points indicated by each hand image. From each hand position, the hand position closest to the trigger position (i.e., the position where the user performs a trigger operation on the large screen) is selected as the target hand position. The hand image to which the target hand position belongs is identified as the target hand image, and the operating system is controlled to execute functional instructions corresponding to the hardware type of the hardware shown in the target hand image and corresponding to the holding posture of the hand shown in the target hand image.

[0096] Specifically, the correspondence between hardware type, holding posture, and function instruction is shown in Table 1.

[0097] Table 1

[0098]

[0099]

[0100] Specifically, user A holds a red pen, and user B holds a black pen. User A and user B stand on the left and right sides of the large screen, respectively, and perform trigger operations on the large screen at the same time. At this time, two touch positions can be captured, one touch position on the left side of the large screen, and the other touch position on the right side of the large screen. At this time, the touch position on the left side of the large screen is closest to the hand position of user A. According to the hardware type (i.e., red pen) and holding state (i.e., holding pen) identified by the hand image of user A, the operating system is controlled to execute the functional instructions corresponding to the "red pen" and the "holding pen" (i.e., adjusting the pen color to red). The touch position on the right side of the large screen is closest to the hand position of user B. According to the hand image of user B, the hardware type (i.e., black pen) and holding state (i.e., holding pen) identified, the operating system is controlled to execute the functional instructions corresponding to the "black pen" and the "holding pen" (i.e., adjusting the pen color to black).

[0101] It should be noted that the above specific implementation content is only used for illustration.

[0102] S113: Based on each hand key point of the hand shown in the target image frame and the mask of the hardware shown in the target image frame, determine whether an interaction occurs between the user and the hardware shown in the target image frame.

[0103] The target image frame is an image frame captured by the camera device within a preset time period.

[0104] It should be noted that the specific process of determining whether an interaction occurs between the user and the hardware shown in the target image frame based on the various hand key points of the hand shown in the target image frame and the mask of the hardware shown in the target image frame includes:

[0105] 1. Obtain detection results corresponding to the first image frame, the second image frame, and the third image frame.

[0106] It should be noted that the first image frame, the second image frame, and the third image frame are image frames continuously captured by the camera device within a preset time, and the timestamp t1 of the first image frame is earlier than the timestamp t2 of the second image frame, and the timestamp t2 of the second image frame is earlier than the timestamp t3 of the third image frame.

[0107] 2. Calculate the shortest distance between the target hand key point shown in the first image frame and the mask shown in the first image frame to obtain a first value d_t1.

[0108] It should be noted that the target hand key point is the hand key point used to indicate the tip of the index finger among all the hand key points.

[0109] 3. Calculate the shortest distance between the target hand key point shown in the second image frame and the mask shown in the second image frame to obtain a second value d_t2.

[0110] 4. Calculate the shortest distance between the target hand key point shown in the third image frame and the mask shown in the third image frame to obtain a third value d_t3.

[0111] It should be noted that the specific implementation method of calculating the shortest distance between the target hand key point and the mask in the image frame is common knowledge familiar to those skilled in the art and will not be described in detail here.

[0112] 5. Calculate the time difference between the timestamp t3 of the third image frame and the timestamp t1 of the first image frame to obtain a fourth value.

[0113] 6. When the fourth value is less than the first preset threshold th_t, the first value d_t1 is less than the second preset threshold th_dl, the second value d_t2 is greater than the third preset threshold th_dh, and the third value d_t3 is less than the second preset threshold th_dl, it is determined that an interaction occurs between the user and the hardware shown in the target image frame.

[0114] In the embodiment of the present application, the interactive behavior is used to represent: the user uses the index finger to perform a click operation on the hardware shown in the target image frame.

[0115] It should be noted that the first value d_t1 is less than the second preset threshold th_dl, indicating that the user's index finger touches the hardware surface at time t1, the second value d_t2 is greater than the third preset threshold th_dh, indicating that the user's index finger leaves the hardware surface at time t2, and the third value d_t3 is less than the second preset threshold th_dl, indicating that the user's index finger touches the hardware surface at time t3. Within the preset time period, the user's index finger goes through the process of touch-leave-touch, and it can be determined that the user uses the index finger to perform a click operation on the hardware, which represents the user's triggering operation on the large screen.

[0116] S114: When an interaction occurs between the user and the hardware shown in the target image frame, control the operating system to execute a functional instruction corresponding to the hardware type of the hardware shown in the target image frame and corresponding to the holding posture of the hand shown in the target image frame.

[0117] In summary, by utilizing the solution described in this embodiment, it is possible to realize user interaction with the large screen in combination with the hardware without any modification to the hardware, thereby effectively improving the user experience.

[0118] It should be noted that S101 mentioned in the above embodiment is an optional implementation of the large-screen interaction method based on hardware combination described in this application. In addition, S102 mentioned in the above embodiment is also an optional implementation of the large-screen interaction method based on hardware combination described in this application. To this end, the process mentioned in the above embodiment can be summarized as follows: Figure 2 The method described.

[0119] like Figure 2 FIG. 1 is a schematic diagram of another large-screen interaction method based on hardware integration provided in an embodiment of the present application, comprising the following steps:

[0120] S201: Acquire image frames captured by a camera device preset on a large screen.

[0121] The image frame is used to indicate a person.

[0122] S202: Perform hand recognition on the person to obtain a hand image.

[0123] The hand image is used to indicate the hand holding the hardware and various key points of the hand.

[0124] S203: Input the hand image into the classification model to obtain the classification result output by the classification model.

[0125] The classification model is trained based on taking the hand image of the sample image frame as input, and taking the hardware type of the hardware shown in the pre-labeled sample image frame and the holding posture of the hand shown in the sample image frame as training targets. The classification result includes the probability of each hardware type and the holding posture of the hand.

[0126] S204: Taking the hardware type with the highest probability as the hardware type of the hardware shown in the image frame.

[0127] S205: When a trigger operation of the user on the large screen is obtained, the operating system is controlled to execute a function instruction corresponding to the hardware type of the hardware shown in the image frame and corresponding to the holding posture of the hand shown in the image frame.

[0128] In summary, by utilizing the solution described in this embodiment, it is possible to realize user interaction with the large screen in combination with the hardware without any modification to the hardware, thereby effectively improving the user experience.

[0129] Corresponding to the large-screen interaction method based on hardware combination provided in the above-mentioned embodiment of the present application, the embodiment of the present application also provides a large-screen interaction device based on hardware combination.

[0130] like Figure 3FIG. 1 is a schematic diagram of a large-screen interactive device based on hardware combination provided in an embodiment of the present application, including:

[0131] The acquisition unit 100 is used to acquire image frames collected by a camera device preset on the large screen; the image frames are used to indicate people.

[0132] The recognition unit 200 is used to perform hand recognition on a person to obtain a hand image; the hand image is used to indicate the hand holding the hardware and various key points of the hand.

[0133] Among them, the recognition unit 200 is specifically used to: input the image frame into a preset hand recognition model to obtain a hand image output by the hand recognition model; wherein the hand recognition model is used to: perform hand recognition on the person shown in the image frame, and mark each hand key point of the hand in the image frame, and use the minimum bounding box algorithm to select the hand image.

[0134] The classification unit 300 is used to input the hand image into the classification model to obtain the classification result output by the classification model; the classification model is based on taking the hand image of the sample image frame as input, and taking the hardware type of the hardware shown in the pre-labeled sample image frame and the holding posture of the hand shown in the sample image frame as training targets, and is trained; the classification result includes the probability of each hardware type and the holding posture of the hand.

[0135] The determination unit 400 is configured to use the hardware type with the highest probability as the hardware type of the hardware shown in the image frame.

[0136] The first control unit 500 is used to control the operating system to execute functional instructions corresponding to the hardware type of the hardware shown in the image frame and the holding posture of the hand shown in the image frame when obtaining the user's trigger operation on the large screen.

[0137] The detection unit 600 is used to input the hand image into the hardware detection model to obtain the detection result output by the hardware detection model; the hardware detection model is trained based on the hand image of the sample image frame as input and the mask of the hardware shown in the pre-labeled sample image frame as the training target; the detection result includes the mask of the hardware shown in the image frame.

[0138] The judgment unit 700 is used to judge whether an interaction occurs between the user and the hardware shown in the target image frame based on the various hand key points of the hand shown in the target image frame and the mask of the hardware shown in the target image frame; the target image frame is an image frame captured by the camera device within a preset time period.

[0139] The target image frames include a first image frame, a second image frame, and a third image frame; the timestamp of the first image frame is earlier than the timestamp of the second image frame, and the timestamp of the second image frame is earlier than the timestamp of the third image frame.

[0140] The judgment unit 700 is specifically used to: calculate the shortest distance between the target hand key point shown in the first image frame and the mask shown in the first image frame to obtain a first value; the target hand key point is: the hand key point used to indicate the fingertip of the index finger among the various hand key points of the hand; calculate the shortest distance between the target hand key point shown in the second image frame and the mask shown in the second image frame to obtain a second value; calculate the shortest distance between the target hand key point shown in the third image frame and the mask shown in the third image frame to obtain a third value; calculate the time difference between the timestamp of the third image frame and the timestamp of the first image frame to obtain a fourth value; when the fourth value is less than the first preset threshold, and the first value is less than the second preset threshold, and the second value is greater than the third preset threshold, and the third value is less than the second preset threshold, it is determined that an interaction behavior occurs between the user and the hardware shown in the target image frame; the interaction behavior is used to characterize: the user uses the index finger to click on the hardware shown in the target image frame.

[0141] The second control unit 800 is used to control the operating system to execute functional instructions corresponding to the hardware type of the hardware shown in the target image frame and the holding posture of the hand shown in the target image frame when an interaction occurs between the user and the hardware shown in the target image frame.

[0142] In summary, by utilizing the solution described in this embodiment, it is possible to realize user interaction with the large screen in combination with the hardware without any modification to the hardware, thereby effectively improving the user experience.

[0143] The present application also provides a computer-readable storage medium, which includes a stored program, wherein the program executes the large-screen interaction method based on hardware combination provided by the present application.

[0144] The present application also provides a large-screen interactive device based on hardware combination, including: a processor, a memory and a bus. The processor and the memory are connected via a bus, the memory is used to store programs, and the processor is used to run programs, wherein when the program is running, the large-screen interactive method based on hardware combination provided by the present application is executed, including the following steps:

[0145] Acquire an image frame captured by a camera device preset on a large screen; the image frame is used to indicate a person;

[0146] Performing hand recognition on the person to obtain a hand image; the hand image is used to indicate the hand holding the hardware and various key points of the hand;

[0147] The hand image is input into a classification model to obtain a classification result output by the classification model; the classification model is trained based on taking the hand image of the sample image frame as input and taking the hardware type of the hardware shown in the sample image frame and the holding posture of the hand shown in the sample image frame as training targets; the classification result includes the probability of each hardware type and the holding posture of the hand;

[0148] Using the hardware type with the highest probability as the hardware type of the hardware shown in the image frame;

[0149] When a trigger operation of the user on the large screen is obtained, the operating system is controlled to execute a functional instruction corresponding to the hardware type of the hardware shown in the image frame and corresponding to the holding posture of the hand shown in the image frame.

[0150] Optionally, also include:

[0151] The hand image is input into a hardware detection model to obtain a detection result output by the hardware detection model; the hardware detection model is trained based on taking the hand image of the sample image frame as input and taking the mask of the hardware shown in the sample image frame marked in advance as a training target; the detection result includes the mask of the hardware shown in the image frame;

[0152] Based on each hand key point of the hand shown in the target image frame and the mask of the hardware shown in the target image frame, determining whether an interaction occurs between the user and the hardware shown in the target image frame; the target image frame is an image frame collected by the camera device within a preset time period;

[0153] When the interaction occurs between the user and the hardware shown in the target image frame, the operating system is controlled to execute functional instructions corresponding to the hardware type of the hardware shown in the target image frame and corresponding to the holding posture of the hand shown in the target image frame.

[0154] Optionally, the target image frame includes a first image frame, a second image frame, and a third image frame; a timestamp of the first image frame is earlier than a timestamp of the second image frame, and a timestamp of the second image frame is earlier than a timestamp of the third image frame;

[0155] The determining whether an interaction occurs between the user and the hardware shown in the target image frame based on each hand key point of the hand shown in the target image frame and the mask of the hardware shown in the target image frame includes:

[0156] Calculate the shortest distance between the target hand key point shown in the first image frame and the mask shown in the first image frame to obtain a first value; the target hand key point is: a hand key point indicating the tip of the index finger among the various hand key points of the hand;

[0157] Calculate the shortest distance between the target hand key point shown in the second image frame and the mask shown in the second image frame to obtain a second value;

[0158] Calculate the shortest distance between the target hand key point shown in the third image frame and the mask shown in the third image frame to obtain a third value;

[0159] Calculating a time difference between a timestamp of the third image frame and a timestamp of the first image frame to obtain a fourth value;

[0160] When the fourth value is less than the first preset threshold, and the first value is less than the second preset threshold, and the second value is greater than the third preset threshold, and the third value is less than the second preset threshold, it is determined that an interaction occurs between the user and the hardware shown in the target image frame; the interaction is used to characterize that the user uses the index finger to perform a click operation on the hardware shown in the target image frame.

[0161] Optionally, the performing hand recognition on the person to obtain a hand image includes:

[0162] The image frame is input into a preset hand recognition model to obtain a hand image output by the hand recognition model; wherein the hand recognition model is used to: perform hand recognition on the person shown in the image frame, mark each hand key point in the image frame, and select the hand image using a minimum bounding box algorithm.

[0163] If the functions described in the method of the embodiment of the present application are implemented in the form of software functional units and sold or used as independent products, they can be stored in a storage medium readable by a computing device. Based on this understanding, the part of the embodiment of the present application that contributes to the prior art or the part of the technical solution can be embodied in the form of a software product, which is stored in a storage medium and includes several instructions to enable a computing device (which can be a personal computer, server, mobile computing device or network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk and other media that can store program code.

[0164] The various embodiments in this specification are described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the various embodiments can be referenced to each other.

[0165] The above description of the disclosed embodiments enables those skilled in the art to implement or use the present application. Various modifications to these embodiments will be apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present application. Therefore, the present application will not be limited to the embodiments shown herein, but will conform to the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A large-screen interactive method based on hardware integration, It is characterized in that include: Obtain image frames captured by a camera device preset on a large screen; The image frame is used to indicate a person; Performing hand recognition on the person to obtain a hand image; the hand image is used to indicate the hand holding the hardware and various key points of the hand; The hand image is input into a classification model to obtain a classification result output by the classification model; the classification model is trained based on taking the hand image of the sample image frame as input and taking the hardware type of the hardware shown in the sample image frame and the holding posture of the hand shown in the sample image frame as training targets; the classification result includes the probability of each hardware type and the holding posture of the hand; Using the hardware type with the highest probability as the hardware type of the hardware shown in the image frame; When a trigger operation of the user on the large screen is obtained, the operating system is controlled to execute a functional instruction corresponding to the hardware type of the hardware shown in the image frame and corresponding to the holding posture of the hand shown in the image frame.

2. The method according to claim 1, It is characterized in that Also includes: Inputting the hand image into a hardware detection model to obtain a detection result output by the hardware detection model; The hardware detection model is trained based on taking the hand image of the sample image frame as input and taking the mask of the hardware shown in the sample image frame marked in advance as a training target; the detection result includes the mask of the hardware shown in the image frame; Based on each hand key point of the hand shown in the target image frame and the mask of the hardware shown in the target image frame, determining whether an interaction behavior occurs between the user and the hardware shown in the target image frame; The target image frame is an image frame captured by the camera device within a preset time period; When the interaction occurs between the user and the hardware shown in the target image frame, the operating system is controlled to execute functional instructions corresponding to the hardware type of the hardware shown in the target image frame and corresponding to the holding posture of the hand shown in the target image frame.

3. The method according to claim 2, It is characterized in that The target image frame includes a first image frame, a second image frame, and a third image frame; the timestamp of the first image frame is earlier than the timestamp of the second image frame, and the timestamp of the second image frame is earlier than the timestamp of the third image frame; The determining whether an interaction occurs between the user and the hardware shown in the target image frame based on each hand key point of the hand shown in the target image frame and the mask of the hardware shown in the target image frame includes: Calculate the shortest distance between the target hand key point shown in the first image frame and the mask shown in the first image frame to obtain a first value; the target hand key point is: a hand key point indicating the tip of the index finger among the various hand key points of the hand; Calculate the shortest distance between the target hand key point shown in the second image frame and the mask shown in the second image frame to obtain a second value; Calculate the shortest distance between the target hand key point shown in the third image frame and the mask shown in the third image frame to obtain a third value; Calculating a time difference between a timestamp of the third image frame and a timestamp of the first image frame to obtain a fourth value; When the fourth value is less than the first preset threshold, and the first value is less than the second preset threshold, and the second value is greater than the third preset threshold, and the third value is less than the second preset threshold, it is determined that an interaction occurs between the user and the hardware shown in the target image frame; the interaction is used to characterize that the user uses the index finger to perform a click operation on the hardware shown in the target image frame.

4. The method according to claim 1, It is characterized in that The step of performing hand recognition on the person to obtain a hand image includes: The image frame is input into a preset hand recognition model to obtain a hand image output by the hand recognition model; wherein the hand recognition model is used to: perform hand recognition on the person shown in the image frame, mark each hand key point in the image frame, and select the hand image using a minimum bounding box algorithm.

5. A large-screen interactive device based on hardware combination, It is characterized in that include: An acquisition unit, used for acquiring image frames collected by a camera device preset on the large screen; The image frame is used to indicate a person; A recognition unit, used to perform hand recognition on the person to obtain a hand image; the hand image is used to indicate the hand holding the hardware and various key points of the hand; A classification unit, configured to input the hand image into a classification model to obtain a classification result output by the classification model; the classification model is trained based on taking the hand image of a sample image frame as input and taking the hardware type of the hardware shown in the sample image frame and the holding posture of the hand shown in the sample image frame as training targets; the classification result includes the probability of each hardware type and the holding posture of the hand; a determining unit, configured to use the hardware type with the highest probability as the hardware type of the hardware shown in the image frame; The first control unit is used to control the operating system to execute functional instructions corresponding to the hardware type of the hardware shown in the image frame and the holding posture of the hand shown in the image frame when a trigger operation of the user on the large screen is obtained.

6. The device according to claim 5, It is characterized in that Also includes: A detection unit, used for inputting the hand image into a hardware detection model to obtain a detection result output by the hardware detection model; The hardware detection model is trained based on taking the hand image of the sample image frame as input and taking the mask of the hardware shown in the sample image frame marked in advance as a training target; the detection result includes the mask of the hardware shown in the image frame; A judging unit, configured to judge whether an interaction occurs between the user and the hardware shown in the target image frame based on each hand key point of the hand shown in the target image frame and a mask of the hardware shown in the target image frame; The target image frame is an image frame captured by the camera device within a preset time period; A second control unit is used to control the operating system to execute functional instructions corresponding to the hardware type of the hardware shown in the target image frame and to the holding posture of the hand shown in the target image frame when the interaction behavior occurs between the user and the hardware shown in the target image frame.

7. The device according to claim 6, It is characterized in that The target image frame includes a first image frame, a second image frame, and a third image frame; the timestamp of the first image frame is earlier than the timestamp of the second image frame, and the timestamp of the second image frame is earlier than the timestamp of the third image frame; The judging unit is specifically used for: Calculate the shortest distance between the target hand key point shown in the first image frame and the mask shown in the first image frame to obtain a first value; the target hand key point is: a hand key point indicating the tip of the index finger among the various hand key points of the hand; Calculate the shortest distance between the target hand key point shown in the second image frame and the mask shown in the second image frame to obtain a second value; Calculate the shortest distance between the target hand key point shown in the third image frame and the mask shown in the third image frame to obtain a third value; Calculating a time difference between a timestamp of the third image frame and a timestamp of the first image frame to obtain a fourth value; When the fourth value is less than the first preset threshold, and the first value is less than the second preset threshold, and the second value is greater than the third preset threshold, and the third value is less than the second preset threshold, it is determined that an interaction occurs between the user and the hardware shown in the target image frame; the interaction is used to characterize that the user uses the index finger to perform a click operation on the hardware shown in the target image frame.

8. The device according to claim 5, It is characterized in that The identification unit is specifically used for: The image frame is input into a preset hand recognition model to obtain a hand image output by the hand recognition model; wherein the hand recognition model is used to: perform hand recognition on the person shown in the image frame, mark each hand key point in the image frame, and select the hand image using a minimum bounding box algorithm.

9. A computer-readable storage medium, It is characterized in that The computer-readable storage medium includes a stored program, wherein the program executes the large-screen interaction method based on hardware combination as described in any one of claims 1-4.

10. A large-screen interactive device based on hardware combination, It is characterized in that include: processor, memory, and bus; The processor is connected to the memory via the bus; The memory is used to store programs, and the processor is used to run programs, wherein the program executes the large-screen interaction method based on hardware combination described in any one of claims 1-4 when running.