Method and device for controlling distributed display equipment and readable medium

Abstract

The invention mainly discloses a method for controlling distributed display equipment, which comprises the following steps of: performing target detection on a current frame image of an acquired operation area to obtain a human body detection frame; comparing a first distance between each human body detection frame in the current frame image and a prefabricated frame of an operator in the current frame image with a first threshold value, and calculating a ratio of an intersection to a union of the area of the human body detection frame and the area of the prefabricated frame when the first distance is smaller than the first threshold value; determining a human body detection frame of an operator in the current frame image; and according to the human body detection frame of the operator in the current frame image and the face feature information of the operator, determining the left hand or the right hand of the operator and acquiring the gesture posture of the operator, and responding to the operation corresponding to the gesture posture through the display equipment. And the distributed display equipment is accurately controlled.

Description

Technical Field

[0001] The present invention relates to the field of electronic information technology, and in particular to a method, device and readable medium for controlling a distributed display device. Background Art

[0002] With the rapid development of information technology, distributed display devices have been widely used in public places, commercial displays, home entertainment and other fields. However, traditional display device control methods such as physical buttons and remote controls are cumbersome and not intuitive enough. In recent years, although control methods based on visual recognition have emerged, when there are multiple people present, non-operator misoperation events often occur, and their accuracy and robustness still need to be improved. Summary of the invention

[0003] To solve at least one of the aforementioned technical problems, the present disclosure proposes, in a first aspect, a method for controlling a distributed display device, comprising: performing target detection on a current frame image of an operation area to obtain a human body detection frame; comparing a first distance and a first threshold between each human body detection frame in the current frame image and a prefabricated frame of an operator in the current frame image, calculating a ratio of an intersection and a union of an area of ​​the human body detection frame and an area of ​​the prefabricated frame when the first distance is less than the first threshold, when there is one human body detection frame with the largest ratio, determining the human body detection frame with the largest ratio as the human body detection frame of the operator in the current frame image, when there are more than one human body detection frame with the largest ratio, comparing the lengths and widths of multiple human body detection frames with the largest ratios, determining the human body detection frame with the largest length and width as the human body detection frame of the operator in the current frame image; determining the left hand or right hand of the operator and obtaining a gesture of the operator based on the human body detection frame of the operator in the current frame image and the facial feature information of the operator, and responding to the operation corresponding to the gesture through a display device.

[0004] Preferably, the method further includes: acquiring previous frames of images of the operation area, and obtaining a prefabricated frame of the operator in the current frame of image through the trajectory of the operator in the previous frames of images of the operation area.

[0005] Preferably, skeleton recognition is performed on the current frame image in which the operator's human body detection frame is determined, and a plurality of skeleton node information and corresponding plurality of skeleton frames are obtained; the second distance and the second threshold are compared, the second distance being the distance between each skeleton frame in the current frame image and the operator's human body detection frame determined in the current frame image, and the ratio of the intersection and union of the area of ​​the skeleton frame and the area of ​​the human body detection frame when the second distance is less than the second threshold is calculated; when there is one skeleton frame with the largest ratio, the skeleton frame with the largest ratio is determined to be the skeleton frame of the operator in the current frame image; when the skeleton frame with the largest ratio is greater than one, the lengths and widths of multiple skeleton frames with the largest ratios are compared, and the skeleton frame with the largest length and width is determined to be the skeleton frame of the operator in the current frame image.

[0006] Preferably, the skeleton nodes corresponding to the skeleton frame of the operator in the current frame image are determined as the skeleton nodes of the operator in the current frame image; and the skeleton face frame of the operator in the current frame image is obtained according to the skeleton nodes of the operator in the current frame image.

[0007] Preferably, face detection is performed on the current frame image in which the operator's human body detection frame is determined to obtain a plurality of face frames; a third distance and a third threshold are compared, the third distance being the distance between each face frame in the current frame image and the operator's skeleton face frame in the current frame image, and the ratio of the intersection and union of the area of ​​the face frame and the area of ​​the skeleton face frame when the third distance is less than the third threshold is calculated; when there is one face frame with the largest ratio, the face frame with the largest ratio is determined to be the face frame of the operator in the current frame image; when the face frame with the largest ratio is greater than one, the lengths and widths of multiple face frames with the largest ratios are compared to determine the face frame with the largest length and width to be the face frame of the operator in the current frame image; and feature extraction is performed on the face frame of the operator in the current frame image to obtain the facial features of the operator.

[0008] Preferably, multiple facial features and corresponding multiple facial frames of the current frame image are obtained through face detection and recognition, and multiple facial features and corresponding multiple facial frames are screened out according to the facial features of the operator; the fourth distance and the fourth threshold are compared, the fourth distance is the distance between each screened facial frame and the prefabricated frame, and the ratio of the intersection and the union of the area of ​​the facial frame and the area of ​​the prefabricated frame when the fourth distance is less than the fourth threshold is calculated; when there is one facial frame with the largest ratio, the facial frame with the largest ratio is determined to be the facial frame of the operator in the current frame image; when the facial frame with the largest ratio is greater than one, the lengths and widths of multiple facial frames with the largest ratios are compared, and the facial frame with the largest length and width is determined to be the facial frame of the operator in the current frame image.

[0009] Preferably, skeleton recognition is performed on the current frame image in which the operator's face frame is determined to obtain a plurality of skeleton node information and corresponding plurality of skeleton frames; the fifth distance and the fifth threshold between each skeleton frame in the current frame image and the operator's face frame determined in the current frame image are compared according to preset conditions, and the ratio of the intersection and the union of the area of ​​the skeleton frame and the area of ​​the face frame when the fifth distance is less than the fifth threshold is calculated; when there is one skeleton frame with the largest ratio, the skeleton frame with the largest ratio is determined to be the skeleton frame of the operator in the current frame image; when the skeleton frame with the largest ratio is greater than one, the lengths and widths of multiple skeleton frames with the largest ratios are compared, and the skeleton frame with the largest length and width is determined to be the skeleton frame of the operator in the current frame image.

[0010] Preferably, the skeleton face frame of the operator in the current frame image is obtained based on the skeleton frame of the operator in the current frame image; when the center distance between the face frame of the operator in the current frame image and the skeleton face frame of the operator in the current frame image is less than a threshold value or the intersection area is greater than a threshold value, it is determined whether the relationship between the face frame of the operator in the current frame image and the skeleton face frame of the operator satisfies an adjustable preset condition, and if so, the current frame image is determined to be a valid frame image; hand skeleton recognition is performed on the valid frame image to obtain multiple hand node information, and the hand node information of the left hand or right hand of the current operator is determined based on the hand node information of the valid frame and the skeleton node information corresponding to the skeleton frame of the current operator; and the gesture of the left hand or right hand of the current operator is determined based on the hand node information of the left hand or right hand of the current operator.

[0011] In a second aspect, the present disclosure proposes the following: an acquisition module, which is used to perform target detection on a current frame image of an operation area to obtain a human body detection frame; a judgment module, which is used to compare the first distance and the first threshold between each human body detection frame in the current frame image and the prefabricated frame of the operator in the current frame image, calculate the ratio of the intersection and the union of the area of ​​the human body detection frame and the area of ​​the prefabricated frame when the first distance is less than the first threshold, and when there is one human body detection frame with the largest ratio, determine the human body detection frame with the largest ratio as the human body detection frame of the operator in the current frame image; when there are more than one human body detection frame with the largest ratio, compare the lengths and widths of multiple human body detection frames with the largest ratios, and determine the human body detection frame with the largest length and width as the human body detection frame of the operator in the current frame image; and a determination module, which is used to determine the left hand or right hand of the operator and obtain the gesture posture of the operator based on the human body detection frame of the operator in the current frame image and the facial feature information of the operator, and respond to the operation corresponding to the gesture posture through the display device.

[0012] The present disclosure proposes a computer-readable medium in a third aspect, wherein a computer program is stored in the computer-readable medium. The computer program is loaded and executed by a processing module to implement the steps of any of the above-mentioned methods.

[0013] Some technical effects of the present disclosure are: a method for controlling a distributed display device, comprising: Perform target detection on the current frame image of the operation area to obtain a human body detection frame; compare the first distance and the first threshold between each human body detection frame in the current frame image and the prefabricated frame of the operator in the current frame image, calculate the ratio of the intersection and union of the area of ​​the human body detection frame and the area of ​​the prefabricated frame when the first distance is less than the first threshold, when there is only one human body detection frame with the largest ratio, determine the human body detection frame with the largest ratio as the human body detection frame of the operator in the current frame image, when the human body detection frame with the largest ratio is greater than one, compare the lengths and widths of multiple human body detection frames with the largest ratios, and determine the human body detection frame with the largest length and width as the human body detection frame of the operator in the current frame image; determine the left or right hand of the operator and obtain the gesture posture of the operator according to the human body detection frame of the operator in the current frame image and the facial feature information of the operator, and respond to the operation corresponding to the gesture posture through the display device. In this way, the only operator's only operating hand, that is, the only left or right hand of the operator, is determined to operate the display device, avoiding the situation of accidental touch in the case of multiple people. Finally, the distributed display device can be accurately controlled. BRIEF DESCRIPTION OF THE DRAWINGS

[0014] To better understand the technical solution of the present disclosure, you may refer to the following drawings for auxiliary explanation of the prior art or embodiments. These drawings will selectively display the products or methods involved in the prior art or some embodiments of the present disclosure. The basic information of these drawings is as follows: Figure 1 The present invention is a flowchart of an embodiment of a method for controlling a distributed display device according to the present invention. DETAILED DESCRIPTION

[0015] The technical means or technical effects involved in the present disclosure will be further described below. Obviously, the embodiments (or implementation methods) provided are only some of the implementation methods covered by the present disclosure, but not all of them. Based on the embodiments in the present disclosure and the explicit or implicit descriptions of the figures and texts, all other embodiments that can be obtained by those skilled in the art without making creative efforts will be within the scope of protection requested by the present disclosure.

[0016] In the existing methods of controlling display devices, for example, when controlling the volume, it is basically controlled by one hand, so when both hands of the controller appear, it is impossible to determine which hand of the controller controls the volume of the display device, so the volume cannot be adjusted or the volume is accidentally adjusted by mistake. In another case, for example, when multiple people appear, other people's hands may also be recognized and control the volume, and the incident of non-operator mistakenly controlling the volume may occur, resulting in a bad experience. Or when there are other people standing behind the operator, the other people behind the operator can also be easily mistakenly identified as the operator by some existing technologies.

[0017] In this regard, the present application discloses a method, device and readable medium for controlling a distributed display device. The method for controlling a distributed display device provided in the embodiment of the present application can be applied to various industries and other scenarios where it is necessary to achieve fast and accurate human-computer interaction by controlling the large screen of the display device, such as emergency command and dispatch center, public security command and dispatch center, traffic command and dispatch center, energy command and dispatch center, smart city command and dispatch center and other fields, and control the dispatch system by controlling the large screen, such as switching distributed signal sources, taking over the mouse in the signal source, and thus performing any operation on the content in the signal source. As the central brain of command and dispatch control, the command center plays an extremely important role in social governance and people's livelihood development, and requires high accuracy and speed of control. In this regard, the method for controlling the large screen in the embodiment of the present disclosure does not require the use of any complex control equipment or wearable sensors, and can quickly and accurately complete the takeover and control of the control of the large screen of the command center only by the behavior recognition of living people, and efficiently realize the fast and accurate interactive operation control of people and large screen content such as signal on-screen, switching, zooming, etc. through simple air gesture operation. Of course, the method disclosed in the present invention can also be applied to the above-mentioned general application scenarios with low requirements to perform operations such as turning pages of PPT.

[0018] An exemplary system architecture to which the method for controlling a distributed display device or an apparatus for controlling a distributed display device disclosed herein can be applied. The system architecture may include a camera with a pan / tilt head, a server, and a display device. Exemplarily, the display device is a large screen, and the camera with a pan / tilt head is connected to the server via a serial cable and a USB cable, and the server is then connected to the display device via a network cable. The image information captured by the camera is transmitted to the server via a USB cable, and the server generates information or commands after processing, analyzing, and making decisions on the received image information. The server may also send the information to a distributed scheduling and image comprehensive management platform via a network port. The distributed scheduling and image comprehensive management platform receives the information and displays corresponding operation feedback on the large screen.

[0019] Display devices generally require large screen, multi-color, high brightness, and high resolution display effects. Exemplarily, the display device is a large screen display device, which refers to a large screen in a direct-view color TV or a rear projection TV. Usually, the diagonal size of the screen is more than 40 inches. The display surface of the large screen display device can be flat or curved. The large screen display device can also be spliced, without limitation.

[0020] In this embodiment, the camera with a pan-tilt head is located directly above the large screen. The pan-tilt head is a device that carries the camera.

[0021] The target user can use gestures to interact with the server through the camera, and then the server and the large screen can interact with each other, thereby realizing gesture control of the large screen. Among them, the server can be a single server, or a server cluster or cloud computing center composed of several servers. The server can provide various services for the display device. For different applications on the display device, the server can be considered as a background server that provides corresponding network services. It can be considered that the method disclosed in this application is mainly executed by the server side.

[0022] The application scenario under one embodiment of the present disclosure includes a large screen, a camera with a pan / tilt system located above the large screen, and an operable range located in front of the large screen. The operable range is roughly a ring-shaped area. The operator can operate the large screen within the operating range. If it is not within the operable range, for example, if the distance to the large screen is too far, the gesture recognition may be wrong, which may cause an operation error. If the distance to the large screen is too close, the operator may not be able to observe the entire content of the large screen, which is not conducive to operating the large screen. In this embodiment, the width of the large screen is 10 meters, and the operable range in this embodiment is a ring range of 3 to 12 meters from the large screen.

[0023] like Figure 1 , which shows a flow chart of an embodiment of a method for controlling a distributed display device according to the present disclosure. The method comprises the following steps: S10: Performing target detection on the current frame image of the operation area to obtain a human body detection frame; S20: comparing the first distance between each human body detection frame in the current frame image and the prefabricated frame of the operator in the current frame image with the size of the first threshold, calculating the ratio of the intersection and the union of the area of ​​the human body detection frame and the area of ​​the prefabricated frame when the first distance is less than the first threshold, when there is one human body detection frame with the largest ratio, determining the human body detection frame with the largest ratio as the human body detection frame of the operator in the current frame image, when there are more than one human body detection frame with the largest ratio, comparing the lengths and widths of multiple human body detection frames with the largest ratios, and determining the human body detection frame with the largest length and width as the human body detection frame of the operator in the current frame image; S30: Determine the left hand or right hand of the operator and obtain the gesture of the operator according to the human body detection frame of the operator in the current frame image and the facial feature information of the operator, and respond to the operation corresponding to the gesture through the display device.

[0024] The camera captures the image of the operation area and transmits it to the server. The camera obtains the previous frames of the operation area, and obtains the pre-made frame of the operator in the current frame image based on the trajectory of the operator in the previous frames of the operation area. The current frame image of the operation area is obtained, and the target detection is performed on the current frame image to obtain the human body detection frame; when no person is detected in the current frame image, the detection of the current frame image is terminated, the next frame image is obtained, and the above steps of performing target detection on the current frame image are repeated. When a person is detected in the pre-made frame of the current frame, the human body detection frame of all people in the operation area is obtained.

[0025] S20: comparing the first distance between each human body detection frame in the current frame image and the prefabricated frame of the operator in the current frame image with the size of the first threshold, calculating the ratio of the intersection and the union of the area of ​​the human body detection frame and the area of ​​the prefabricated frame when the first distance is less than the first threshold, when there is one human body detection frame with the largest ratio, determining the human body detection frame with the largest ratio as the human body detection frame of the operator in the current frame image, when there are more than one human body detection frame with the largest ratio, comparing the lengths and widths of multiple human body detection frames with the largest ratios, and determining the human body detection frame with the largest length and width as the human body detection frame of the operator in the current frame image; In this embodiment, the first distance refers to the center distance between each human body detection frame detected above and the pre-made frame of the operator in the current frame image. The first threshold may be preset, and the size of the first threshold may be obtained according to the relationship between the maximum center distance and the minimum center distance. The human body detection frame whose center distance to the pre-made frame of the operator in the current frame image is within the first threshold range is retained, and the human body detection frame whose center distance to the pre-made frame of the operator in the current frame image is not within the first threshold range is discarded.

[0026] Then, the human body detection frame is screened based on the intersection and union ratio of the areas of the human body detection frame and the prefabricated frame. The ratio of the intersection and union of the areas of the human body detection frame and the prefabricated frame when the first distance is less than the first threshold is calculated, and the human body detection frame with the largest ratio result is retained.

[0027] If the human detection frame obtained in the previous step is still a complex value, further screening is performed to retain the human detection frame with the largest length and width, because the length and width of the human detection frame reflect the distance relationship between different people in front of the camera and the camera, and the larger one is closer to the camera. This will prevent other people standing behind the operator who are closer to the center of the prefabricated frame from being misjudged as operators. Moreover, the length and width are both maximum or minimum at the same time. That is, the length and width of the human detection frame of the operator in front are respectively greater than the length and width of the human detection frame of other people behind. In particular, the operator is often misjudged in existing actual scenes. For example, when the person standing behind the controller is closer to the center of the prefabricated frame than the controller, the wrong person will be selected. In particular, when using the loss function DIOU Loss involving detection frame screening in the field of target detection, the distance relationship in DIOU Loss is the real frame and the detection frame, and the square of the center distance between the two frames is divided by the square of the diagonal of the circumscribed frame surrounding the two frames.

[0028] S30: Determine the left hand or right hand of the operator and obtain the gesture of the operator according to the human body detection frame of the operator in the current frame image and the facial feature information of the operator, and respond to the operation corresponding to the gesture through the display device.

[0029] The current frame image in which the operator's human body detection frame is determined is subjected to skeleton recognition to obtain multiple skeleton node information and corresponding multiple skeleton frames; the second distance and the second threshold value are compared, the second distance is the distance between each skeleton frame in the current frame image and the operator's human body detection frame determined in the current frame image, and the second distance can be the center distance between the two, the skeleton frame with the second distance greater than the second threshold value is discarded, and the skeleton frame with the second distance less than the second threshold value is retained. The ratio of the intersection and the union of the area of ​​the skeleton frame and the area of ​​the operator's human body detection frame when the second distance is less than the second threshold value is calculated; when there is one skeleton frame with the maximum ratio, the skeleton frame with the maximum ratio is determined to be the skeleton frame of the operator in the current frame image; when the skeleton frame with the maximum ratio is greater than one, the length and width of multiple skeleton frames with the maximum ratio are compared, and the skeleton frame with the largest length and width is determined to be the skeleton frame of the operator in the current frame image. The maximum length and width refer to the maximum length or the maximum width, and generally the two are the maximum at the same time. Because the operator is closest to the camera, the width and length of the operator's human body detection frame are both the largest.

[0030] Each person's skeleton frame has a corresponding skeleton node. The skeleton node corresponding to the operator's skeleton frame in the current frame image is determined as the operator's skeleton node in the current frame image. According to the operator's skeleton node in the current frame image, the operator's skeleton face frame in the current frame image is obtained. The skeleton nodes include eye nodes, nose nodes, etc., and the approximate skeleton face frame can be inferred.

[0031] Perform face detection on the current frame image in which the operator's human body detection frame is determined to obtain a plurality of face frames; compare the third distance and the third threshold, the third distance is the distance between each face frame in the current frame image and the skeleton face frame of the operator in the current frame image, the third distance can be the center distance between the two frames, discard the face frame when the third distance is greater than the third threshold, calculate the ratio of the intersection and the union of the area of ​​the face frame and the area of ​​the skeleton face frame when the third distance is less than the third threshold; when there is one face frame with the largest ratio, determine the face frame with the largest ratio as the face frame of the operator in the current frame image; when the face frame with the largest ratio is greater than one, compare the lengths and widths of multiple face frames with the largest ratios, determine the face frame with the largest length and width as the face frame of the operator in the current frame image; perform feature extraction on the face frame of the operator in the current frame image to obtain the facial features of the operator.

[0032] After acquiring the operator's facial features as described above, the gesture of the operator's unique operating hand can be determined for any frame of the image. The multiple facial features and corresponding multiple facial frames of the current frame image are acquired through face detection and recognition, and a portion of the multiple facial features and corresponding multiple facial frames are screened out based on the operator's facial features. For example, the screening can be performed based on the similarity between each detected facial feature and the operator's facial features determined as described above; the facial frames corresponding to the facial features that do not meet the similarity conditions are discarded, and the facial frames corresponding to some facial features that meet the similarity conditions are retained. The fourth distance is compared with the fourth threshold, where the fourth distance is the distance between each screened face frame and the prefabricated frame, and the ratio of the intersection and union of the area of ​​the face frame and the area of ​​the prefabricated frame when the fourth distance is less than the fourth threshold is calculated; when there is one face frame with the largest ratio, the face frame with the largest ratio is determined to be the face frame of the operator in the current frame image; when the face frame with the largest ratio is greater than one, the lengths and widths of multiple face frames with the largest ratios are compared, and the face frame with the largest length and width is determined to be the face frame of the operator in the current frame image.

[0033] The skeleton recognition is performed on the current frame image in which the operator's face frame is determined, and the plural skeleton node information and the corresponding plural skeleton frames are obtained; the fifth distance and the fifth threshold value between each skeleton frame in the current frame image and the operator's face frame determined in the current frame image are compared according to the preset conditions, and the ratio of the intersection and the union of the area of ​​the skeleton frame and the area of ​​the face frame when the fifth distance is less than the fifth threshold value is calculated; when there is one skeleton frame with the largest ratio, the skeleton frame with the largest ratio is determined to be the skeleton frame of the operator in the current frame image; when the skeleton frame with the largest ratio is greater than one, the length and width of multiple skeleton frames with the largest ratio are compared, and the skeleton frame with the largest length and width is determined to be the skeleton frame of the operator in the current frame image. The preset conditions can be adjusted according to the on-site conditions of the distributed system, and the conditions are not limited to the center distance of the two frames being less than the threshold value or the intersection area of ​​the two frames being greater than the threshold value.

[0034] According to the skeleton frame of the operator in the current frame image, the skeleton face frame of the operator in the current frame image is obtained; the calculation method for obtaining the skeleton face frame is the same as the above method. When the center distance between the operator's face frame in the current frame image and the operator's skeleton face frame in the current frame image is less than the threshold or the intersection area is greater than the threshold, it is determined whether the relationship between the operator's face frame in the current frame image and the operator's skeleton face frame meets the adjustable preset condition. If it meets the condition, the current frame image is determined to be a valid frame image; the hand skeleton is recognized for the valid frame image to obtain multiple hand node information, and the hand node information of the left or right hand of the current operator is determined according to the hand node information of the valid frame and the skeleton node information corresponding to the skeleton frame of the current operator; according to the hand node information of the left or right hand of the current operator, the gesture of the left or right hand of the current operator is determined. The operation mode corresponding to the gesture posture is obtained. And the operation corresponding to the gesture posture is responded to through the display device. For example, operations such as turning pages of PPT are performed.

[0035] In a second aspect, the present disclosure provides a device for controlling a distributed display device, comprising: The acquisition module is used to perform target detection on the current frame image of the operation area to obtain a human body detection frame; the judgment module is used to compare the first distance between each human body detection frame in the current frame image and the prefabricated frame of the operator in the current frame image and the size of the first threshold, calculate the ratio of the intersection and union of the area of ​​the human body detection frame and the area of ​​the prefabricated frame when the first distance is less than the first threshold, when there is one human body detection frame with the largest ratio, determine the human body detection frame with the largest ratio as the human body detection frame of the operator in the current frame image, when the human body detection frame with the largest ratio is greater than one, compare the lengths and widths of multiple human body detection frames with the largest ratios, and determine the human body detection frame with the largest length and width as the human body detection frame of the operator in the current frame image; the determination module is used to determine the left hand or right hand of the operator and obtain the gesture posture of the operator based on the human body detection frame of the operator in the current frame image and the facial feature information of the operator, and respond to the operation corresponding to the gesture posture through the display device.

[0036] The present disclosure proposes a computer-readable medium in a third aspect, wherein a computer program is stored in the computer-readable medium, and the computer program is loaded and executed by a processing module to implement the steps of the acquisition method. It can be understood by those skilled in the art that all or part of the steps in the embodiment can be implemented by instructing related hardware through a computer program, and the program can be stored in a computer-readable medium, and the readable medium can include various media that can store program codes, such as a flash drive, a mobile hard disk, a read-only memory, a random access memory, a magnetic disk or an optical disk.

[0037] Within the scope of the knowledge and ability level of those skilled in the art, the various embodiments or technical features mentioned herein may be combined with each other as other optional embodiments without conflict. These limited number of optional embodiments, which are not listed one by one and are formed by combining a limited number of technical features, still fall within the technical scope disclosed in the present disclosure and can be understood or inferred by those skilled in the art in combination with the drawings and the above text.

[0038] In addition, the description of most embodiments is based on different focuses. If you need to further understand what is not described in detail, you can refer to the relevant content of the prior art, other relevant descriptions in this article or the purpose of the invention for reasonable reasoning.

[0039] It is emphasized again that the embodiments listed above are typical and preferred embodiments of the present disclosure, which are only used to explain and interpret the technical solutions of the present disclosure in detail to facilitate the understanding of the readers, and are not used to limit the scope or application of the present disclosure. Any technical solutions obtained by modification, equivalent substitution, improvement, etc. made within the spirit and principle of the present disclosure shall be included in the scope of the present disclosure.

Claims

1. A method for controlling a distributed display device, characterized in that: include: Perform target detection on the current frame image of the operation area to obtain a human body detection frame; Compare the first distance between each human body detection frame in the current frame image and the prefabricated frame of the operator in the current frame image with the size of the first threshold, calculate the ratio of the intersection and the union of the area of ​​the human body detection frame and the area of ​​the prefabricated frame when the first distance is less than the first threshold, when there is one human body detection frame with the largest ratio, determine the human body detection frame with the largest ratio as the human body detection frame of the operator in the current frame image, when the human body detection frame with the largest ratio is greater than one, compare the lengths and widths of multiple human body detection frames with the largest ratios, and determine the human body detection frame with the largest length and width as the human body detection frame of the operator in the current frame image; According to the human body detection frame of the operator in the current frame image and the facial feature information of the operator, the left hand or right hand of the operator is determined and the gesture posture of the operator is obtained, and the operation corresponding to the gesture posture is responded to through the display device.

2. The method according to claim 1, characterized in that Also includes: The previous frames of images of the operation area are obtained, and the prefabricated frame of the operator in the current frame image is obtained through the trajectory of the operator in the previous frames of images of the operation area.

3. The method according to claim 1, characterized in that Performing skeleton recognition on the current frame image in which the operator's human body detection frame is determined, and obtaining a plurality of skeleton node information and a corresponding plurality of skeleton frames; Compare the second distance and the second threshold, where the second distance is the distance between each skeleton frame in the current frame image and the human body detection frame of the operator determined in the current frame image, and calculate the ratio of the intersection and union of the area of ​​the skeleton frame and the area of ​​the human body detection frame when the second distance is less than the second threshold; When there is one skeleton frame with the largest ratio, the skeleton frame with the largest ratio is determined to be the skeleton frame of the operator in the current frame image; When there is more than one skeleton frame with the largest ratio, the lengths and widths of the multiple skeleton frames with the largest ratios are compared to determine the skeleton frame with the largest length and width as the skeleton frame of the operator in the current frame image.

4. The method according to claim 3, characterized in that Determine the bone node corresponding to the bone frame of the operator in the current frame image as the bone node of the operator in the current frame image; According to the skeleton nodes of the operator in the current frame image, the skeleton face frame of the operator in the current frame image is obtained.

5. The method according to claim 4, characterized in that Performing face detection on the current frame image in which the operator's human body detection frame is determined to obtain a plurality of face frames; Comparing the third distance with the third threshold, where the third distance is the distance between each face frame in the current frame image and the skeleton face frame of the operator in the current frame image, and calculating the ratio of the intersection and union of the area of ​​the face frame and the area of ​​the skeleton face frame when the third distance is less than the third threshold; When there is one face frame with the largest ratio, the face frame with the largest ratio is determined to be the face frame of the operator in the current frame image; When there is more than one face frame with the largest ratio, the lengths and widths of the multiple face frames with the largest ratios are compared, and the face frame with the largest length and width is determined to be the face frame of the operator in the current frame image; The facial features of the operator are extracted from the face frame of the operator in the current frame image to obtain the facial features of the operator.

6. The method according to claim 5, characterized in that Acquire multiple facial features and corresponding multiple facial frames of the current frame image through face detection and recognition, and select multiple facial features and corresponding multiple facial frames according to the facial features of the operator; comparing a fourth distance with a fourth threshold, where the fourth distance is the distance between each screened face frame and the prefabricated frame, and calculating a ratio of an intersection and a union of an area of ​​the face frame and an area of ​​the prefabricated frame when the fourth distance is less than the fourth threshold; When there is one face frame with the largest ratio, the face frame with the largest ratio is determined to be the face frame of the operator in the current frame image; When there is more than one face frame with the largest ratio, the lengths and widths of the multiple face frames with the largest ratios are compared to determine the face frame with the largest length and width as the face frame of the operator in the current frame image.

7. The method according to claim 6, characterized in that Performing skeleton recognition on the current frame image with the operator's face frame determined, and obtaining a plurality of skeleton node information and a corresponding plurality of skeleton frames; Comparing the fifth distance between each skeleton frame in the current frame image and the face frame of the operator determined in the current frame image with a fifth threshold value according to a preset condition, and calculating the ratio of the intersection and union of the area of ​​the skeleton frame and the area of ​​the face frame when the fifth distance is less than the fifth threshold value; When there is one skeleton frame with the largest ratio, the skeleton frame with the largest ratio is determined to be the skeleton frame of the operator in the current frame image; When there is more than one skeleton frame with the largest ratio, the lengths and widths of the multiple skeleton frames with the largest ratios are compared to determine the skeleton frame with the largest length and width as the skeleton frame of the operator in the current frame image.

8. The method according to claim 7, characterized in that According to the skeleton frame of the operator in the current frame image, the skeleton face frame of the operator in the current frame image is obtained; When the center distance between the operator's face frame in the current frame image and the operator's skeleton face frame in the current frame image is less than a threshold value or the intersection area is greater than a threshold value, it is determined whether the relationship between the operator's face frame in the current frame image and the operator's skeleton face frame meets an adjustable preset condition, and if so, the current frame image is determined to be a valid frame image; Perform hand skeleton recognition on the valid frame image to obtain multiple hand node information, and determine the hand node information of the left hand or right hand of the current operator according to the hand node information of the valid frame and the skeleton node information corresponding to the skeleton frame of the current operator; The left hand or right hand gesture of the current operator is determined according to the hand node information of the left hand or right hand of the current operator.

9. A device for controlling a distributed display device, characterized in that: include: An acquisition module is used to perform target detection on the current frame image of the operation area to obtain a human body detection frame; A judgment module is used to compare the first distance between each human body detection frame in the current frame image and the prefabricated frame of the operator in the current frame image with a first threshold value, calculate the ratio of the intersection and union of the area of ​​the human body detection frame and the area of ​​the prefabricated frame when the first distance is less than the first threshold value, and when there is one human body detection frame with the largest ratio, determine the human body detection frame with the largest ratio as the human body detection frame of the operator in the current frame image; when there are more than one human body detection frame with the largest ratio, compare the lengths and widths of multiple human body detection frames with the largest ratios, and determine the human body detection frame with the largest length and width as the human body detection frame of the operator in the current frame image; The determination module is used to determine the operator's left hand or right hand and obtain the operator's gesture posture according to the operator's human body detection frame in the current frame image and the operator's facial feature information, and respond to the operation corresponding to the gesture posture through the display device.

10. A computer readable medium, characterized in that: The computer readable medium stores a computer program, which is loaded and executed by the processing module to implement the steps of any one of the methods of claims 1 to 8.

Images