Model training method, and control method and device of large-screen display equipment

Abstract

The invention provides a model training method and a control method and device of large-screen display equipment, and relates to the technical field of remote control. The method comprises the following steps: acquiring multiple frames of continuous first sample images including the face and the hand of at least one sample user, and labeling each frame of first sample image; and then acquiring annotation information of each frame of the first sample image, and performing model training according to the multiple frames of continuous first sample images and the annotation information of each frame of the first sample image to obtain a gesture recognition model. Further, continuous multi-frame user images including the host in front of the large-screen display device are obtained, and the multi-frame user images are input into the gesture recognition model to obtain a target gesture of the host. And finally, sending a target control instruction corresponding to the target gesture to the large-screen display equipment according to the mapping relationship between the gesture and the control instruction. According to the method provided by the invention, the controllability of the large-screen display equipment is improved.

Description

Technical Field

[0001] The present application relates to the field of remote control technology, and in particular to a model training method, a control method and a device for a large-screen display device. Background Art

[0002] Large-screen display devices can not only display data, information or content in a graphical and dynamic way, but also display data changes in real time to help users make timely decisions. With the continuous advancement of biometric technology, more possibilities are provided for the display and control of large-screen display devices.

[0003] In most cases, the existing technology still uses mouse operation to control large-screen display devices. When a large-screen display device needs to be remotely controlled, wearable devices such as smart gloves and smart bracelets are used to capture the user's gestures to control the computer or other large-screen display devices.

[0004] However, the prior art has the technical problem of poor controllability of large-screen display devices. Summary of the invention

[0005] The present application provides a model training method, a control method and a device for a large-screen display device, so as to solve the technical problem of poor controllability of the large-screen display device in the prior art.

[0006] In a first aspect, the present application provides a model training method, comprising:

[0007] Acquire a plurality of continuous first sample image frames, each of which includes a face and a hand of at least one sample user;

[0008] Annotate each frame of the first sample image to obtain annotation information of each frame of the first sample image, where the annotation information is used to indicate a facial region and a gesture of each sample user in the first sample image;

[0009] Model training is performed according to multiple frames of continuous first sample images and the annotation information of each frame of the first sample image to obtain a gesture recognition model.

[0010] Optionally, the annotation information of each frame of the first sample image includes first sub-annotation information and second sub-annotation information, the first sub-annotation information is used to indicate a facial area of ​​each sample user in the first sample image, and the second sub-annotation information is used to indicate a gesture of each sample user in the first sample image;

[0011] Accordingly, the method of performing model training according to multiple frames of continuous first sample images and the annotation information of each frame of the first sample image to obtain a gesture recognition model includes:

[0012] According to the plurality of frames of continuous first sample images and the first sub-annotation information of each frame of the first sample image, an initial face recognition sub-model of the pre-trained model is trained to obtain a face recognition sub-model;

[0013] According to the multiple frames of continuous first sample images and the second sub-annotation information of each frame of the first sample image, the initial gesture recognition sub-model of the pre-trained model is trained to obtain the gesture recognition sub-model;

[0014] The gesture recognition model includes the facial recognition sub-model and the gesture recognition sub-model.

[0015] Optionally, the training of the initial gesture recognition sub-model of the pre-trained model according to the multiple frames of continuous first sample images and the second sub-annotation information of each frame of the first sample image to obtain the gesture recognition sub-model includes:

[0016] Crop each frame of the first sample image to obtain a second sample image corresponding to each sample user;

[0017] Determining third sub-annotation information of each frame of the second sample image according to the second sub-annotation information of each frame of the first sample image;

[0018] For each sample user, the initial gesture recognition sub-model is trained according to a plurality of frames of continuous second sample images and the third sub-annotation information of each frame of the second sample image to obtain the gesture recognition sub-model.

[0019] Optionally, the pre-trained model is a YOLOv8 model.

[0020] In a second aspect, the present application provides a control method for a large-screen display device, comprising:

[0021] Acquire a continuous multi-frame user image, wherein the multi-frame user image includes a host standing in front of a large-screen display device;

[0022] Inputting the multiple frames of user images into a gesture recognition model to obtain the host's target gesture, wherein the gesture recognition model is obtained by model training using the model training method according to any one of the first aspects;

[0023] According to the mapping relationship between gestures and control instructions, a target control instruction corresponding to the target gesture is sent to the large-screen display device.

[0024] Optionally, inputting the multiple frames of user images into a gesture recognition model to obtain the host's target gesture includes:

[0025] Determine the facial vectors of all users in each frame of the user image through the facial recognition sub-model of the gesture recognition model;

[0026] Matching the facial vectors of all the users with the facial vectors of the host stored in the preset host library, and determining the user with successful matching as the host;

[0027] Cropping the multiple frames of user images to obtain multiple frames of user sub-images corresponding to the host;

[0028] The target gestures of the multiple frames of user sub-images are determined by the gesture recognition sub-model of the gesture recognition model.

[0029] In a third aspect, the present application provides a model training device, comprising:

[0030] An acquisition module, used to acquire a plurality of consecutive frames of first sample images, each frame of the first sample image including a face and a hand of at least one sample user;

[0031] a labeling module, used to label each frame of the first sample image and obtain labeling information of each frame of the first sample image, wherein the labeling information is used to indicate the facial area and gesture of each sample user in the first sample image;

[0032] The training module is used to perform model training based on multiple frames of continuous first sample images and annotation information of each frame of the first sample image to obtain a gesture recognition model.

[0033] Optionally, the annotation information of each frame of the first sample image includes first sub-annotation information and second sub-annotation information, the first sub-annotation information is used to indicate a facial area of ​​each sample user in the first sample image, and the second sub-annotation information is used to indicate a gesture of each sample user in the first sample image;

[0034] Optionally, the training module is also used for:

[0035] According to the plurality of frames of continuous first sample images and the first sub-annotation information of each frame of the first sample image, an initial face recognition sub-model of the pre-trained model is trained to obtain a face recognition sub-model;

[0036] According to the multiple frames of continuous first sample images and the second sub-annotation information of each frame of the first sample image, the initial gesture recognition sub-model of the pre-trained model is trained to obtain the gesture recognition sub-model;

[0037] The gesture recognition model includes the facial recognition sub-model and the gesture recognition sub-model.

[0038] Optionally, the training module further includes: a cutting module and a determining module.

[0039] The cropping module is used to crop each frame of the first sample image to obtain a second sample image corresponding to each sample user;

[0040] The determining module is used to determine the third sub-annotation information of each frame of the second sample image according to the second sub-annotation information of each frame of the first sample image;

[0041] The training module is further used to train the initial gesture recognition sub-model for each sample user according to multiple frames of continuous second sample images and the third sub-annotation information of each frame of the second sample image to obtain the gesture recognition sub-model.

[0042] Optionally, the pre-trained model is a YOLOv8 model.

[0043] In a fourth aspect, the present application provides a control device for a large-screen display device, comprising:

[0044] An acquisition module, used to acquire a continuous multi-frame user image, wherein the multi-frame user image includes a host standing in front of a large-screen display device;

[0045] An input module, used to input the multiple frames of user images into a gesture recognition model to obtain the target gesture of the host, wherein the gesture recognition model is obtained by model training using the model training method according to any one of the first aspects;

[0046] The sending module is used to send the target control instruction corresponding to the target gesture to the large-screen display device according to the mapping relationship between the gesture and the control instruction.

[0047] Optionally, the input module further includes: a determination module, a matching module, and a cutting module;

[0048] The determination module is used to determine the facial vectors of all users in each frame of the user image by using the facial recognition sub-model of the gesture recognition model;

[0049] The matching module is used to match the facial vectors of all the users with the facial vectors of the host stored in the preset host library, and determine the user who successfully matches as the host;

[0050] The cropping module is used to crop the multiple frames of user images to obtain multiple frames of user sub-images corresponding to the host;

[0051] The determination module is further configured to determine the target gesture of the multiple frames of user sub-images by using the gesture recognition sub-model of the gesture recognition model.

[0052] In a fifth aspect, the present application provides an electronic device, including:

[0053] Processor and memory;

[0054] Wherein, the memory stores computer-executable instructions;

[0055] The processor executes the computer-executable instructions stored in the memory to implement the method as described in the first aspect, the second aspect and various possible implementation methods.

[0056] In a sixth aspect, the present application provides a computer-readable storage medium, in which computer execution instructions are stored. When the computer execution instructions are executed by a processor, they are used to implement the methods described in the first aspect, the second aspect and various possible implementation methods mentioned above.

[0057] In a seventh aspect, the present application provides a computer program product, which includes a computer program, and when the computer program is executed by a processor, it is used to implement the methods described in the first aspect, the second aspect and various possible implementation methods mentioned above.

[0058] The model training method, control method and device of a large-screen display device provided by the present application obtain multiple frames of continuous first sample images including the face and hands of at least one sample user, and annotate each frame of the first sample image. Then, the annotation information for indicating the facial area and gesture of each sample user in each frame of the first sample image is obtained, and the model training is performed according to the multiple frames of continuous first sample images and the annotation information of each frame of the first sample image to obtain a gesture recognition model. It can be seen that by performing multi-dimensional training on a large number of first sample images, the gesture recognition model obtained after training can better adapt to the needs of large-screen command operations, improve the accuracy of the model, and make the command recognition more in line with the actual project needs. Next, a continuous multi-frame user image including the host in front of the large-screen display device is obtained, and the multi-frame user image is input into the gesture recognition model to obtain the host's target gesture. Finally, according to the mapping relationship between the gesture and the control instruction, the target control instruction corresponding to the target gesture is sent to the large-screen display device. The position of the host in the venue is accurately located by target tracking and positioning, and then the host's target gesture is obtained by the gesture recognition model, which avoids the interference of a series of factors such as light changes and background environment on gesture recognition, and improves the controllability of the large-screen display device. BRIEF DESCRIPTION OF THE DRAWINGS

[0059] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present application and, together with the description, serve to explain the principles of the present application.

[0060] Figure 1 The control method flow of the large screen display device provided in the embodiment of the present application Figure 1 ;

[0061] Figure 2 A flow chart of the model training method provided in the embodiment of the present application;

[0062] Figure 3 The control method flow of the large screen display device provided in the embodiment of the present application Figure 2 ;

[0063] Figure 4 A schematic diagram of the structure of the model training device provided in this application;

[0064] Figure 5 A schematic diagram of the structure of a control device for a large-screen display device provided in this application;

[0065] Figure 6 This is a hardware structure diagram of the electronic device provided in this application.

[0066] The above drawings have shown clear embodiments of the present application, which will be described in more detail later. These drawings and text descriptions are not intended to limit the scope of the present application in any way, but to illustrate the concept of the present application to those skilled in the art by referring to specific embodiments. DETAILED DESCRIPTION

[0067] Exemplary embodiments will be described in detail herein, examples of which are shown in the accompanying drawings. When the following description refers to the drawings, the same numbers in different drawings represent the same or similar elements unless otherwise indicated. The implementations described in the following exemplary embodiments do not represent all implementations consistent with the present application. Instead, they are merely examples of devices and methods consistent with some aspects of the present application as detailed in the appended claims.

[0068] The terms "first", "second", "third", "fourth", etc. (if any) in the specification and claims of the present invention and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the numbers used in this way can be interchanged where appropriate, so that the embodiments of the present invention described herein can be implemented in sequences other than those illustrated or described herein, for example.

[0069] In the embodiments of the present application, the words "exemplary" or "for example" are used to indicate examples, illustrations or descriptions. Any embodiment or design described as "exemplary" or "for example" in the present application should not be interpreted as being more preferred or more advantageous than other embodiments or designs. Specifically, the use of words such as "exemplary" or "for example" is intended to present related concepts in a specific way.

[0070] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with relevant laws, regulations and standards, and provide corresponding operation entrances for users to choose to authorize or refuse.

[0071] Terminology explanation:

[0072] YOLO model: is a deep learning-based target detection algorithm that can transform the target detection task into a regression problem. The core idea of ​​the YOLO model is to use the entire image as the input of the network and obtain the location of the bounding box and its category through the neural network.

[0073] Roboflow platform: is a platform focused on computer vision and machine learning data annotation. It helps users process and annotate multimedia data such as images and videos for training machine learning models. The main functions of the Roboflow platform include data uploading, annotation, verification, and export, aiming to improve the efficiency and accuracy of data annotation, thereby accelerating the development process of machine learning projects.

[0074] Google Remote Procedure Call (gRPC): is a modern, open source, high-performance framework that allows communication between clients and servers and supports multiple programming languages. It can efficiently connect services within and across data centers, supporting load balancing, tracking, health checks, and authentication.

[0075] The control technology of large-screen display devices is used to display content and perform interactive operations on large-screen display devices, aiming to improve user experience and enable users to interact with the large screen more intuitively and efficiently.

[0076] In most cases, although gesture control and other advanced interaction technologies are gradually developing and being applied, mouse operation is still one of the main ways to control large-screen display devices due to its maturity, reliability and universality. However, mouse operation also has some disadvantages and limitations, especially in large screens and specific application scenarios.

[0077] For example, in front of a large screen or projection screen, the operator usually needs to stand a certain distance away, and it is inconvenient to use a mouse for long-distance operation. In addition, the mouse is usually designed for a single user. In a scenario that requires multi-person collaboration, multiple users may need to frequently use the mouse alternately, resulting in inconvenient operation and low efficiency.

[0078] In some scenarios where large-screen display devices need to be remotely controlled, existing technologies use wearable devices such as smart gloves and smart bracelets to capture user gestures and control computers or other large-screen display devices. Wearable devices usually require additional hardware support and rely on battery power. In addition, misidentification or missed identification may occur.

[0079] Especially when performing complex gestures or moving quickly, the accuracy of gesture recognition and user experience may be affected. In addition, certain environmental factors such as light changes and background clutter may also affect the accuracy of gesture recognition, resulting in mismatches between gesture commands and controls.

[0080] It can be seen that the above-mentioned series of problems will lead to poor controllability of large-screen display devices.

[0081] In response to the above technical problems, the inventor discovered during the process of studying the control of large-screen display devices that when remotely controlling large-screen display devices, the accuracy of gesture recognition is an important factor affecting the operability of large-screen display devices, while light changes and background environment are the main factors affecting the accuracy of gesture recognition. Based on this, the inventor began to focus on studying how to improve the accuracy of gesture recognition. Considering that model training can achieve high-precision recognition effects and reduce misidentification and missed recognition, the inventor chose to perform gesture recognition through model training. Through continuous learning and optimization, the inventor can recognize a variety of gestures and actions of different users, provide rich control instructions, meet different operating requirements, and thus improve the operability of large-screen display devices.

[0082] The technical solution of the present application and how the technical solution of the present application solves the above-mentioned technical problems are described in detail below with specific embodiments. The following specific embodiments can be combined with each other, and the same or similar concepts or processes may not be repeated in some embodiments. The embodiments of the present application will be described below in conjunction with the accompanying drawings.

[0083] Figure 1 The control method flow of the large screen display device provided in the embodiment of the present application Figure 1 .like Figure 1 As shown, a control method for a large-screen display device provided in an embodiment of the present application includes:

[0084] S101: Acquire multiple frames of continuous user images.

[0085] For example, tracking technology can be used to automatically track the user's face in the venue, so as to obtain continuous multiple frames of user images. The tracking technology combines multiple technologies such as computer vision, machine learning and image processing, and can achieve efficient and accurate automatic tracking of the user's face.

[0086] Explanatory, each frame of the user image may include only the host in the venue, or only other audiences in the venue, or may include both the host and other audiences.

[0087] S102: Input multiple frames of user images into a gesture recognition model to obtain the host's target gesture.

[0088] Among them, the gesture recognition model can be obtained through training to adapt to different users, environments and gesture changes.

[0089] In one possible implementation, Figure 2 The flow chart of the model training method provided in the embodiment of the present application is as follows: Figure 2 As shown in the figure, the training process of the gesture recognition model specifically includes the following steps:

[0090] S201, acquiring a plurality of continuous frames of first sample images.

[0091] Each frame of the first sample image includes a face and a hand of at least one sample user.

[0092] Specifically, the gestures to be defined are collected offline and collected on site in the venue. According to business needs, the main variables of the first sample image are the number of sample users, scene size, distance between the sample user and the camera, light intensity, etc. At the same time, it is also necessary to collect images of the host standing in different positions to perform gesture control.

[0093] It is worth noting that the first sample image can be in the form of a photo or a video stream.

[0094] S202: Label each frame of the first sample image to obtain labeling information of each frame of the first sample image.

[0095] The annotation information is used to indicate the facial region and gesture of each sample user in the first sample image. Specifically, the annotation information of each frame of the first sample image includes first sub-annotation information and second sub-annotation information, the first sub-annotation information is used to indicate the facial region of each sample user in the first sample image, and the second sub-annotation information is used to indicate the gesture of each sample user in the first sample image.

[0096] Exemplarily, an image annotation tool (such as labelimg) is used to annotate each frame of the first sample image to generate a corresponding eXtensible Markup Language (XML) file, wherein the XML file stores annotation information of the first sample image, specifically including information such as the category of the sample user, the gesture category corresponding to the sample user, and the coordinates of the bounding box.

[0097] In a possible implementation, a collection data folder mydata is prepared, wherein an xml folder and an image folder JPEGImages are set under the mydata folder. The XML file is placed in the xml folder, and the first sample image is placed in the JPEGImages folder. In addition, by writing a configuration file, a dataset folder dataset is set under the mydata folder to store the training set, the validation set, and the test set.

[0098] S203: Perform model training according to multiple frames of continuous first sample images and annotation information of each frame of the first sample image to obtain a gesture recognition model.

[0099] It is understandable that in order to reduce the model training time and improve the training efficiency and model performance, the pre-trained model that has been trained on similar tasks can be further trained to obtain a gesture recognition model. Specifically, the pre-trained model used is the latest version of the YOLO model, the YOLOv8 model.

[0100] Therefore, the YOLOv8 model needs to be fully pre-trained first to obtain a trained pre-trained model. The training process includes facial recognition training and gesture recognition training.

[0101] For example, images of the face and hands of at least one sample user in various light, different sizes, and different angles can be obtained from the data set for sign recognition on the Roboflow platform. These images are used as input and input into the YOLOv8 model for training to obtain a trained pre-trained model. It is worth noting that the pre-trained model includes an initial face recognition sub-model and an initial gesture recognition sub-model.

[0102] Next, the gesture recognition model is obtained by further training the pre-trained model, wherein the gesture recognition model includes a facial recognition sub-model and a gesture recognition sub-model. In a possible implementation, obtaining the gesture recognition model through model training includes the following steps:

[0103] Step a1: train the initial facial recognition sub-model of the pre-trained model according to multiple frames of continuous first sample images and the first sub-annotation information of each frame of the first sample image to obtain the facial recognition sub-model.

[0104] Step a2: training the initial gesture recognition sub-model of the pre-trained model according to multiple frames of continuous first sample images and the second sub-annotation information of each frame of the first sample image to obtain the gesture recognition sub-model.

[0105] Specifically, the initial face recognition sub-model and the initial gesture recognition sub-model of the pre-trained model are trained in sequence by writing a training script.

[0106] For example, the batch size can be set to 100, the number of loops can be set to 200, and the validation set: training set ratio can be set to 1:5 for training. In addition, the gesture categories can be calibrated as follows: 0 corresponds to page up, 1 corresponds to page down, 2 corresponds to open pop-up box, 3 corresponds to color switching, 4 corresponds to animation start, 5 corresponds to animation end, etc.

[0107] It is worth noting that after the initial facial recognition sub-model training is completed, each frame of the first sample image needs to be cropped to obtain the second sample image corresponding to each sample user. Afterwards, the third sub-annotation information of each frame of the second sample image is determined based on the second sub-annotation information of each frame of the first sample image. Then, for each sample user, the initial gesture recognition sub-model is trained based on multiple frames of continuous second sample images and the third sub-annotation information of each frame of the second sample image to obtain the gesture recognition sub-model.

[0108] In a possible implementation, the first sub-annotation information, the second sub-annotation information, and the third sub-annotation information are stored in the XML file mentioned in step S202. Since the annotation format supported by YOLOv8 is the yolo_txt format, the first sub-annotation information and the second sub-annotation information in the XML file need to be format converted. Specifically, each frame of the first sample image corresponds to a txt file, and each line of the txt file is the information of a sample user, including the category of the sample user, key point information, normalized center point x coordinate, normalized center point y coordinate, normalized width w, and normalized height y. The calculation formula is as follows:

[0109]

[0110] Among them, X max , X min , Y max , Y min are the coordinates of the four sides of the bounding box; W is the width of the bounding box before normalization; H is the height of the bounding box before normalization.

[0111] It should be noted that the resolution of the first sample image should also be adjusted to meet the resolution supported by the YOLOv8 model. For example, the resolution of the first sample image is fixed at 640*640 by using the affine transformation function warpAffine in the Open Source Computer Vision Library (OpenCV). In addition, NVIDIA's Compute Unified Device Architecture (CUDA) can also be used for acceleration.

[0112] Explanation: The YOLOv8 model is a state-of-the-art (SOTA) model that builds on the success of previous versions of the YOLO model, introduces new features and improvements to further improve performance and flexibility, and the YOLOv8 model has greatly improved accuracy. Specifically, the optimization updates of the YOLOv8 model include the following aspects:

[0113] (1) The YOLOv8 model is implemented based on the instance segmentation algorithm YOLACT.

[0114] (2) The backbone network of the YOLOv8 model uses cross-stage feature fusion (C2F) with richer gradient flow, adjusts the number of channels for models of different scales, and greatly improves the model performance.

[0115] (3) The YOLOv8 model uses an anchor-free method to replace the anchor-based method: The anchor-free method and the anchor-based method are two different target detection methods. The difference between the two is whether a predefined anchor box is used to match the real target box. The predefined anchor box is a set of rectangular boxes clustered on the training set using methods such as k-means before training, representing the length and width scales of the main distribution of targets in the data set. The advantage of the anchor-free method is that it does not require a predefined anchor box. It only needs to regress the target center point and width and height of feature maps of different scales, which reduces time and computing power. At the same time, the anchor-free method can also avoid some problems of missed detection or repeated detection caused by unreasonable anchor box settings.

[0116] (4) Loss function calculation: The YOLOv8 model uses the TaskAligned strategy for the distribution of positive and negative samples. The TaskAligned strategy selects positive samples based on the weighted scores of classification and regression. In order to improve the alignment of the two tasks, a simple allocation rule is used to select training samples: for each instance, m anchor boxes with the largest t value are selected as positive samples, and the remaining anchor boxes are selected as negative samples. Next, training is performed using the loss function (a loss function designed for the alignment of classification and positioning).

[0117] Specifically, the varifocal loss function (VFL) is introduced as the classification loss, and the distribution focal loss (DFL) loss function and the complete intersection over union (CIOU) loss function are used as the regression loss. Among them, the three loss functions mentioned above are weighted with a certain weight ratio.

[0118] Specifically, VFL introduces asymmetric weighting, the formula is as follows:

[0119]

[0120] Where p represents the predicted IACS (intersection over union) score, and q represents the IOU (intersection over union) between the candidate box and the true box. For negative samples, that is, candidate boxes that are not true boxes, their q value is 0.

[0121] DFL loss function:

[0122] DFL(S i ,S i+1 )=-((y i+1 -y)lg(S i )+(yy i )lg(S i+1 ))

[0123] in, The meaning of DFL is to optimize the two positions closest to label y in the form of cross entropy (y i ,y i+1 ), so that the network can focus on the distribution of the neighboring area of ​​the target location more quickly.

[0124] In addition, the advantage of choosing the CIOU loss function is that the CIOU loss function takes into account the center point distance and diagonal distance of the bounding box, so it can more accurately measure the similarity between two bounding boxes. Especially when the target shape and size are irregular, the use of the CIOU loss function can better adapt to the target detection tasks of various sizes and shapes.

[0125] It can be seen that the use of YOLOv8 model as a pre-training model has the characteristics of high speed, high accuracy, and strong robustness, which further improves the recognition ability.

[0126] Furthermore, after the final gesture recognition model is obtained through the above training and tuning, the facial vectors of all users in each frame of the user image are first determined through the facial recognition sub-model of the gesture recognition model. Secondly, the facial vectors of all users are matched with the facial vectors of the host stored in the preset host library, and the user who successfully matches is determined as the host. Finally, the multiple frames of user images are cropped to obtain the multiple frames of user sub-images corresponding to the host, and the target gestures of the multiple frames of user sub-images are determined through the gesture recognition sub-model of the gesture recognition model.

[0127] Specifically, the preset host library has saved the host's facial vector, and the preset host library is used to perform a matching operation on the facial vectors of all users, and the user image of the matched host is cropped to obtain multiple frames of user sub-images. Exemplarily, the preset host library can choose to use dlib_face_recognition_resnet_model_v1, which is a deep learning-based facial recognition model widely used in image processing and computer vision tasks.

[0128] S103: Sending a target control instruction corresponding to the target gesture to the large-screen display device according to the mapping relationship between the gesture and the control instruction.

[0129] In one possible implementation, Figure 3 The control method flow of the large screen display device provided in the embodiment of the present application Figure 2 ,like Figure 3 As shown, the specific implementation process of controlling the large-screen display device through the target gesture is as follows:

[0130] S301, tracking the user's face and determining the user's face vector;

[0131] S302, matching the user's facial vector;

[0132] S303, identifying the target gesture corresponding to the successfully matched host;

[0133] S304, classifying the target gesture, wherein the classification results include page up, page down, pop-up window opening, color switching, animation start, and animation end;

[0134] S305, large screen automated operation.

[0135] For example, a gRPC service interface is designed, including methods such as display operation update and data query, where gRPC transmits data based on the Protobuf message format. When the backend data is updated, it is pushed to the frontend in time through gRPC to ensure the real-time and accuracy of the data.

[0136] In one possible implementation, the front end performs the final operation after receiving the target control instruction, and the user interface (UI) increases the adaptive display capability, which is implemented by the front end using responsive design and media query. As a UI layout degradation solution, several common large-screen resolutions can also be given according to the specific exhibition hall scene, and different display styles can be designed by the UI and implemented through front-end development.

[0137] The model training method and the control method of the large-screen display device provided by the present application obtain multiple frames of continuous first sample images including the face and hands of at least one sample user, and annotate each frame of the first sample image. Then, the first sub-annotation information for indicating the facial area of ​​each sample user and the second sub-annotation information for indicating the gesture of each sample user are obtained for each frame of the first sample image. The initial facial recognition submodel of the pre-trained model is trained according to the first sub-annotation information to obtain the facial recognition submodel. Then, each frame of the first sample image is cropped, and the initial gesture recognition submodel of the pre-trained model is trained according to the cropped first sample image to obtain the gesture recognition submodel. It can be seen that by performing multi-dimensional training on a large number of first sample images, the gesture recognition model obtained after training can better adapt to the needs of large-screen command operations, improve the accuracy of the model, and make the command recognition more in line with the actual project needs. Next, a continuous multi-frame user image including the host in front of the large-screen display device is obtained, and the facial vectors of all users in each frame of the user image are determined by the facial recognition submodel of the gesture recognition model. The facial vector is matched with the host facial vector stored in the preset host library, and the user who successfully matches is determined as the host. Afterwards, the multi-frame user images are cropped to obtain the multi-frame user sub-images corresponding to the host, and then the target gesture of the multi-frame user sub-images is determined by the gesture recognition sub-model of the gesture recognition model. Finally, according to the mapping relationship between gestures and control instructions, the target control instruction corresponding to the target gesture is sent to the large-screen display device. The position of the host in the venue is accurately located by target tracking and positioning, and then the host's target gesture is obtained by the gesture recognition model, avoiding the interference of a series of factors such as light changes and background environment on gesture recognition, and improving the controllability of the large-screen display device.

[0138] Figure 4 This is a schematic diagram of the structure of the model training device provided in this application. Figure 4 As shown, the model training device 400 includes: an acquisition module 401, a labeling module 402, and a training module 403;

[0139] The acquisition module 401 is used to acquire multiple frames of continuous first sample images, each frame of the first sample image includes the face and hand of at least one sample user;

[0140] A labeling module 402 is used to label each frame of the first sample image and obtain labeling information of each frame of the first sample image, where the labeling information is used to indicate the facial area and gesture of each sample user in the first sample image;

[0141] The training module 403 is used to perform model training according to multiple frames of continuous first sample images and the annotation information of each frame of the first sample image to obtain a gesture recognition model.

[0142] Optionally, the annotation information of each frame of the first sample image includes first sub-annotation information and second sub-annotation information, the first sub-annotation information is used to indicate a facial area of ​​each sample user in the first sample image, and the second sub-annotation information is used to indicate a gesture of each sample user in the first sample image;

[0143] Optionally, the training module 403 is further used for:

[0144] According to the plurality of frames of continuous first sample images and the first sub-annotation information of each frame of the first sample image, an initial face recognition sub-model of the pre-trained model is trained to obtain a face recognition sub-model;

[0145] According to the multiple frames of continuous first sample images and the second sub-annotation information of each frame of the first sample image, the initial gesture recognition sub-model of the pre-trained model is trained to obtain the gesture recognition sub-model;

[0146] The gesture recognition model includes the facial recognition sub-model and the gesture recognition sub-model.

[0147] Optionally, the training module 403 further includes: a cutting module 404, a determining module 405,

[0148] The cropping module 404 is used to crop each frame of the first sample image to obtain a second sample image corresponding to each sample user;

[0149] The determining module 405 is used to determine the third sub-annotation information of each frame of the second sample image according to the second sub-annotation information of each frame of the first sample image;

[0150] The training module 403 is further used to train the initial gesture recognition sub-model for each sample user according to multiple frames of continuous second sample images and the third sub-annotation information of each frame of the second sample image to obtain the gesture recognition sub-model.

[0151] Optionally, the pre-trained model is a YOLOv8 model.

[0152] The model training device provided in the embodiments of the present application can be used to execute the model training method in any of the above embodiments. Its implementation principles and technical effects are similar and will not be repeated here.

[0153] Figure 5 This is a schematic diagram of the structure of the control device of the large-screen display device provided in this application. Figure 5As shown, the control device 500 of the large-screen display device includes: an acquisition module 501, an input module 502, and a sending module 503;

[0154] The acquisition module 501 is used to acquire a continuous multi-frame user image, wherein the multi-frame user image includes a host standing in front of the large-screen display device;

[0155] An input module 502 is used to input the multiple frames of user images into a gesture recognition model to obtain the host's target gesture, wherein the gesture recognition model is obtained by model training using the model training method according to any one of the first aspects;

[0156] The sending module 503 is used to send the target control instruction corresponding to the target gesture to the large-screen display device according to the mapping relationship between the gesture and the control instruction.

[0157] Optionally, the input module 502 further includes: a determination module 504, a matching module 505, and a cutting module 506;

[0158] The determination module 504 is used to determine the facial vectors of all users in each frame of the user image by using the facial recognition sub-model of the gesture recognition model;

[0159] The matching module 505 is used to match the facial vectors of all the users with the facial vectors of the host stored in the preset host library, and determine the user who successfully matches as the host;

[0160] The cropping module 506 is used to crop the multiple frames of user images to obtain multiple frames of user sub-images corresponding to the host;

[0161] The determination module 502 is further configured to determine the target gesture of the multiple frames of user sub-images by using the gesture recognition sub-model of the gesture recognition model.

[0162] The control device for a large-screen display device provided in an embodiment of the present application may be used to execute a control method for a large-screen display device in any of the above embodiments, and its implementation principle and technical effect are similar and will not be described in detail here.

[0163] It should be noted that it should be understood that the division of the various modules of the above device is only a division of logical functions. In actual implementation, they can be fully or partially integrated into one physical entity, or they can be physically separated. And these modules can all be implemented in the form of software calling through processing elements; they can also be all implemented in the form of hardware; some modules can also be implemented in the form of software calling through processing elements, and some modules can be implemented in the form of hardware. In addition, all or part of these modules can be integrated together, or they can be implemented independently. The processing element described here can be an integrated circuit with signal processing capabilities. In the implementation process, each step of the above method or each module above can be completed by an integrated logic circuit of hardware in the processor element or instructions in the form of software.

[0164] Figure 6 The hardware structure diagram of the electronic device provided in this application. Figure 6 As shown, the electronic device 600 includes:

[0165] Processor 601 and memory 602;

[0166] Memory stores computer-executable instructions;

[0167] The processor executes the computer-executable instructions stored in the memory 602 , so that the electronic device executes the above-mentioned method.

[0168] It should be understood that the processor 601 can be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), ASICs, etc. A general-purpose processor can be a microprocessor or any conventional processor. The steps of the method disclosed in the invention can be directly implemented as a hardware processor, or can be implemented by a combination of hardware and software modules in the processor. The memory 602 may include a high-speed random access memory (RAM), and may also include a non-volatile memory (NVM), such as at least one disk storage, and may also be a USB flash drive, a mobile hard disk, a read-only memory, a disk or an optical disk.

[0169] The electronic device provided in the embodiment of the present application can be used to execute the method provided in any of the above method embodiments. Its implementation principles and technical effects are similar and will not be repeated here.

[0170] The embodiment of the present application also provides a computer-readable storage medium, in which computer-executable instructions are stored. When the computer-executable instructions are executed by a processor, they are used to implement the above method.

[0171] The embodiment of the present application also provides a computer program product, which includes a computer program. When the computer program is executed by a processor, it is used to implement the method as described above.

[0172] It should be noted that, for the aforementioned method embodiments, for the sake of simplicity, they are all expressed as a series of action combinations, but those skilled in the art should be aware that the present application is not limited by the described order of actions, because according to the present application, certain steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also be aware that the embodiments described in the specification are all optional embodiments, and the actions and modules involved are not necessarily required by the present application.

[0173] It should be further noted that, although the various steps in the flowchart are displayed in sequence according to the indication of the arrows, these steps are not necessarily executed in sequence in the order indicated by the arrows. Unless there is a clear description in this article, the execution of these steps is not strictly limited in order, and these steps can be executed in other orders. Moreover, at least a portion of the steps in the flowchart may include multiple sub-steps or multiple stages, and these sub-steps or stages are not necessarily executed at the same time, but can be executed at different times, and the execution order of these sub-steps or stages is not necessarily to be carried out in sequence, but can be executed in turn or alternately with other steps or at least a portion of the sub-steps or stages of other steps.

[0174] It should be understood that the above-mentioned device embodiments are only illustrative, and the device of the present application can also be implemented in other ways. For example, the division of units / modules in the above-mentioned embodiments is only a logical function division, and there may be other division methods in actual implementation. For example, multiple units, modules or components can be combined, or can be integrated into another system, or some features can be ignored or not executed.

[0175] In addition, unless otherwise specified, each functional unit / module in each embodiment of the present application may be integrated into one unit / module, each unit / module may exist physically separately, or two or more units / modules may be integrated together. The above-mentioned integrated unit / module may be implemented in the form of hardware or in the form of a software program module.

[0176] If the integrated unit / module is implemented in the form of hardware, the hardware may be a digital circuit, an analog circuit, etc. The physical implementation of the hardware structure includes but is not limited to transistors, memristors, etc. If not specifically stated, the processor may be any appropriate hardware processor, such as a CPU, a GPU, an FPGA, a DSP, an ASIC, etc. If not specifically stated, the storage unit may be any appropriate magnetic storage medium or magneto-optical storage medium, such as a resistive random access memory RRAM (Resistive Random Access Memory), a dynamic random access memory DRAM (Dynamic Random Access Memory), a static random access memory SRAM (Static Random-Access Memory), an enhanced dynamic random access memory EDRAM (Enhanced Dynamic Random Access Memory), a high-bandwidth memory HBM (High-Bandwidth Memory), a hybrid memory cube HMC (Hybrid Memory Cube), etc.

[0177] If the integrated unit / module is implemented in the form of a software program module and sold or used as an independent product, it can be stored in a computer-readable memory. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art or all or part of the technical solution, can be embodied in the form of a software product, which is stored in a memory and includes several instructions for a computer device (which can be a personal computer, a server or a network device, etc.) to perform all or part of the steps of the various embodiments of the present application. The aforementioned memory includes: U disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), mobile hard disk, disk or optical disk and other media that can store program codes.

[0178] In the above embodiments, the description of each embodiment has its own emphasis. For the part not described in detail in a certain embodiment, please refer to the relevant description of other embodiments. The technical features of the above embodiments can be combined arbitrarily. In order to make the description concise, all possible combinations of the technical features in the above embodiments are not described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0179] Those skilled in the art will readily appreciate other embodiments of the present application after considering the specification and practicing the invention disclosed herein. The present application is intended to cover any modification, use or adaptation of the present application, which follows the general principles of the present application and includes common knowledge or customary techniques in the art that are not disclosed in the present application. The specification and examples are intended to be exemplary only, and the true scope and spirit of the present application are indicated by the following claims.

[0180] It should be understood that the present application is not limited to the precise structures that have been described above and shown in the drawings, and that various modifications and changes may be made without departing from the scope thereof. The scope of the present application is limited only by the appended claims.

Claims

1. A model training method, characterized in that: include: Acquire a plurality of continuous first sample image frames, each of which includes a face and a hand of at least one sample user; Annotate each frame of the first sample image to obtain annotation information of each frame of the first sample image, where the annotation information is used to indicate a facial region and a gesture of each sample user in the first sample image; Model training is performed according to multiple frames of continuous first sample images and the annotation information of each frame of the first sample image to obtain a gesture recognition model.

2. The method according to claim 1, characterized in that: The annotation information of each frame of the first sample image includes first sub-annotation information and second sub-annotation information, wherein the first sub-annotation information is used to indicate a facial region of each sample user in the first sample image, and the second sub-annotation information is used to indicate a gesture of each sample user in the first sample image; Accordingly, the model training is performed according to the multiple frames of continuous first sample images and the annotation information of each frame of the first sample image to obtain the gesture recognition model, including: According to the plurality of frames of continuous first sample images and the first sub-annotation information of each frame of the first sample image, an initial face recognition sub-model of the pre-trained model is trained to obtain a face recognition sub-model; According to the multiple frames of continuous first sample images and the second sub-annotation information of each frame of the first sample image, the initial gesture recognition sub-model of the pre-trained model is trained to obtain the gesture recognition sub-model; The gesture recognition model includes the facial recognition sub-model and the gesture recognition sub-model.

3. The method according to claim 2, characterized in that The training of the initial gesture recognition sub-model of the pre-trained model according to the plurality of frames of continuous first sample images and the second sub-annotation information of each frame of the first sample image to obtain the gesture recognition sub-model includes: Crop each frame of the first sample image to obtain a second sample image corresponding to each sample user; Determining third sub-annotation information of each frame of the second sample image according to the second sub-annotation information of each frame of the first sample image; For each sample user, the initial gesture recognition sub-model is trained according to a plurality of frames of continuous second sample images and the third sub-annotation information of each frame of the second sample image to obtain the gesture recognition sub-model.

4. The method according to claim 2 or 3, characterized in that: The pre-trained model is a YOLOv8 model.

5. A control method for a large-screen display device, characterized in that: The method comprises: Acquire a continuous multi-frame user image, wherein the multi-frame user image includes a host standing in front of a large-screen display device; Inputting the multiple frames of user images into a gesture recognition model to obtain the host's target gesture, wherein the gesture recognition model is obtained by performing model training using the model training method according to any one of claims 1 to 4; According to the mapping relationship between gestures and control instructions, a target control instruction corresponding to the target gesture is sent to the large-screen display device.

6. The method according to claim 5, characterized in that The step of inputting the multiple frames of user images into a gesture recognition model to obtain the host's target gesture includes: Determine the facial vectors of all users in each frame of the user image through the facial recognition sub-model of the gesture recognition model; Matching the facial vectors of all the users with the facial vectors of the host stored in the preset host library, and determining the user with successful matching as the host; Cropping the multiple frames of user images to obtain multiple frames of user sub-images corresponding to the host; The target gestures of the multiple frames of user sub-images are determined by the gesture recognition sub-model of the gesture recognition model.

7. A model training device, characterized in that: include: An acquisition module, used to acquire a plurality of consecutive frames of first sample images, each frame of the first sample image including a face and a hand of at least one sample user; a labeling module, used to label each frame of the first sample image and obtain labeling information of each frame of the first sample image, wherein the labeling information is used to indicate the facial area and gesture of each sample user in the first sample image; The training module is used to perform model training based on multiple frames of continuous first sample images and annotation information of each frame of the first sample image to obtain a gesture recognition model.

8. A control device for a large-screen display device, characterized in that: include: An acquisition module, used to acquire a continuous multi-frame user image, wherein the multi-frame user image includes a host standing in front of a large-screen display device; An input module, used for inputting the multiple frames of user images into a gesture recognition model to obtain the target gesture of the host, wherein the gesture recognition model is obtained by model training using the model training method according to any one of claims 1 to 4; The sending module is used to send the target control instruction corresponding to the target gesture to the large-screen display device according to the mapping relationship between the gesture and the control instruction.

9. An electronic device, characterized in that: include: A processor, and a memory communicatively connected to the processor; The memory stores computer-executable instructions; The processor executes the computer-executable instructions stored in the memory to implement the method according to any one of claims 1 to 6.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer-executable instructions, which are used to implement the method according to any one of claims 1 to 6 when executed by a processor.

11. A computer program product, characterized in that The invention comprises a computer program, which is used to implement the method according to any one of claims 1 to 6 when being executed by a processor.

Images