Video conference system for privacy protection and privacy protecting method using the same

The video conferencing system uses hand gesture detection and machine learning to generate real-time privacy-protecting videos, addressing the lack of immediate privacy protection in online conferencing by adjusting user content and audio output based on predefined gestures.

KR103046137B1Active Publication Date: 2026-11-10KOREA UNIV OF TECH & EDUCATION IND UNIV COOPERATION FOUND +1
View PDF 4 Cites 0 Cited by

Patent Information

Application Number
KR1020240133862
Authority / Receiving Office
KR · KR
Patent Type
Patents
Current Assignee / Owner
Filing Date
2024-10-02
Publication Date
2026-11-10
Estimated Expiration
2044-10-02

AI Technical Summary

Technical Problem

Existing online video conferencing platforms lack the ability to immediately protect user privacy by reacting to real-time user gestures, leading to potential exposure of personal space, background, and sound information.

Method used

A video conferencing system that detects hand gestures using machine learning algorithms to generate privacy-protecting videos by separating user faces and backgrounds or controlling audio output based on predefined hand gestures, and transmits these adjustments to counterpart terminals in real-time.

Benefits of technology

Enables immediate privacy protection by generating and transmitting privacy-protecting videos in response to user hand gestures, effectively shielding sensitive information from view or sound.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure 112024107432164-PAT00003_ABST
    Figure 112024107432164-PAT00003_ABST
Patent Text Reader

Abstract

The present invention relates to a video conferencing system for privacy protection and a privacy protection method using the same, comprising: a user terminal requesting privacy protection via a hand gesture while conducting an online video conference through a video conferencing app; a counterpart terminal providing a privacy protection video in response to the privacy protection while attending the online video conference through the video conferencing app; and a video conferencing server providing the online video conference through the video conferencing app, detecting the hand gesture in a user video transmitted from the user terminal, and, if the detected hand gesture is a privacy protection hand gesture for privacy protection, generating a privacy protection video corresponding to the privacy hand gesture and transmitting it to the counterpart terminal; thereby enabling privacy protection by immediately reflecting the user's real-time gesture.
Need to check novelty before this filing date? Find Prior Art

Description

Technology Field

[0001] The present invention relates to a video conferencing system for privacy protection and a privacy protection method using the same, which can protect privacy by immediately reflecting a user's real-time gestures by detecting a hand gesture in a user video transmitted from a user terminal while providing an online video conference through a video conferencing app, and, if the detected hand gesture is a privacy-protecting hand gesture for privacy protection, generating a privacy-protecting video corresponding to the privacy hand gesture and transmitting it to a counterpart terminal.

[0002] This invention is a research project conducted with funding from the government (Ministry of Science and ICT) and supported by the Institute of Information and Communications Planning and Evaluation - Regional Intelligence Innovation Talent Development Project (IITP-2024-2710008868). Background Technology

[0003] As is well known, online video conferencing platforms have been actively utilized for video conferences and remote learning since the COVID-19 pandemic; while these platforms offer users the convenience of connectivity, they also bring with them issues regarding privacy infringement.

[0004] For example, there are problems where the user's personal space and background can be exposed as is, such as personal belongings or information in the home or office being unintentionally revealed through the camera, or unnecessary sound information being exposed through the microphone.

[0005] Meanwhile, in representative online video conferencing platforms such as Zoom, users can select options such as blurring the background by choosing a separate menu; however, since this method requires navigating to a menu to change settings, there is a problem in that it is difficult to immediately change the user environment depending on the user's situation.

[0006] Furthermore, while privacy protection technologies that blur pedestrians' faces in surveillance systems such as CCTV are being extensively researched, no technology has yet been proposed to protect privacy in online video conferencing platforms by reacting immediately to the user's situation or intention.

[0007] Accordingly, there is a need for the development of technology that can protect privacy by immediately reflecting users' real-time gestures while using online video conferencing platforms. Prior art literature

[0008] 1. Korean Registered Patent No. 10-1270780 (Registered on May 28, 2013) The problem to be solved

[0009] The present invention aims to provide a video conferencing system for privacy protection and a privacy protection method using the same, which can protect privacy by immediately reflecting a user's real-time gestures, by detecting a hand gesture in a user video transmitted from a user terminal while providing an online video conference through a video conferencing app, and if the detected hand gesture is a privacy-protecting hand gesture for privacy protection, generating a privacy-protecting video corresponding to the privacy hand gesture and transmitting it to a counterpart terminal.

[0010] The purposes of the embodiments of the present invention are not limited to those mentioned above, and other unmentioned purposes will be clearly understood by those skilled in the art from the description below. means of solving the problem

[0011] According to one aspect of the present invention, a video conferencing system for privacy protection may be provided, comprising: a user terminal requesting privacy protection via a hand gesture while conducting an online video conference through a video conferencing app; a counterpart terminal providing a privacy protection video in response to the privacy protection while attending the online video conference through the video conferencing app; and a video conferencing server providing the online video conference through the video conferencing app, detecting the hand gesture in a user video transmitted from the user terminal, and, if the detected hand gesture is a privacy protection hand gesture for privacy protection, generating a privacy protection video corresponding to the privacy hand gesture and transmitting it to the counterpart terminal.

[0012] In addition, according to one aspect of the present invention, a video conferencing system for privacy protection may be provided, comprising: a communication module unit responsible for communication with the user terminal and the counterpart terminal; a video conferencing providing unit that provides a video service for the online video conferencing by providing the video conferencing app; a hand gesture detection unit that detects the hand gesture in the user image and determines whether the detected hand gesture is the privacy protection hand gesture; a face background separation unit that separates the user face and the user background in the user image when the hand gesture is the video protection gesture among the privacy protection hand gestures; and a protection image generation unit that generates the privacy protection image corresponding to the video protection gesture, while controlling the image output of at least one of the user face and the user background.

[0013] In addition, according to one aspect of the present invention, a video conferencing system for privacy protection may be provided, wherein the hand gesture detection unit detects and identifies the hand gesture by detecting hand feature points and classifying finger shapes using a machine learning algorithm.

[0014] Additionally, according to one aspect of the present invention, a video conferencing system for privacy protection may be provided, wherein the hand gesture detection unit provides a privacy protection release signal to the video conferencing providing unit to transmit the user image transmitted from the user terminal to the counterpart terminal when the hand gesture is a privacy protection release hand gesture while providing the privacy protection image.

[0015] In addition, according to one aspect of the present invention, a video conferencing system for privacy protection may be provided, which further comprises a voice control unit that controls voice output in the user video or privacy protection video when the hand gesture is a voice protection gesture among the privacy protection hand gestures.

[0016] According to another aspect of the present invention, a video conferencing system and a privacy protection method using the same may be provided, comprising: a step of providing an online video conference to a user terminal and a counterpart terminal, respectively, through a video conferencing app at a video conferencing server; a step of requesting privacy protection through a hand gesture while conducting the online video conference at the user terminal; a step of detecting the hand gesture in a user video transmitted from the user terminal at the video conferencing server; a step of determining at the video conferencing server whether the detected hand gesture is a privacy protection hand gesture for privacy protection; a step of generating and transmitting a privacy protection video corresponding to the privacy hand gesture at the video conferencing server; and a step of providing a privacy protection video in response to privacy protection while attending the online video conference at the counterpart terminal.

[0017] In addition, according to another aspect of the present invention, the step of detecting the hand gesture in the user image may be provided by a video conferencing system that detects the hand gesture through hand feature point detection and finger shape classification using a machine learning algorithm, and a privacy protection method using the same.

[0018] Additionally, according to another aspect of the present invention, a video conferencing system and a privacy protection method using the same may be provided, wherein the step of determining whether the hand gesture is a privacy protection hand gesture determines whether the hand gesture is a privacy protection hand gesture, and determines whether the hand gesture is a video protection gesture or a voice protection gesture among the privacy protection hand gestures.

[0019] Additionally, according to another aspect of the present invention, the step of generating and transmitting the privacy-protecting image may be provided by separating the user face and the user background from the user image when the hand gesture is the image-protecting gesture, and then generating the privacy-protecting image in correspondence with the image-protecting gesture, while controlling the image output of at least one of the user face and the user background. A video conferencing system and a privacy protection method using the same may be provided.

[0020] In addition, according to another aspect of the present invention, a video conferencing system and a privacy protection method using the same may be provided, further comprising the step of providing a privacy protection release signal when the hand gesture is a privacy protection release hand gesture while providing the privacy protection image, thereby transmitting the user image transmitted from the user terminal to the counterpart terminal.

[0021] In addition, according to another aspect of the present invention, a video conferencing system and a privacy protection method using the same may be provided, further comprising the step of controlling voice output in the user video or privacy protection video when the hand gesture is the voice protection gesture. Effects of the invention

[0022] The present invention can protect privacy by immediately reflecting the user's real-time gestures while providing an online video conference through a video conferencing app, by detecting a hand gesture in a user video transmitted from a user terminal, and if the detected hand gesture is a privacy-protecting hand gesture for privacy protection, by generating a privacy-protecting video corresponding to the privacy hand gesture and transmitting it to a counterpart terminal. Brief explanation of the drawing

[0023] FIG. 1 is a block diagram of a video conferencing system for privacy protection according to one embodiment of the present invention, and FIGS. 2 to 5 are drawings for explaining the detailed configuration of a video conferencing system for privacy protection according to an embodiment of the present invention, and FIG. 6 is a flowchart illustrating the process of protecting privacy using a video conferencing system according to another embodiment of the present invention. Specific details for implementing the invention

[0024] The advantages and features of the embodiments of the present invention, and the methods for achieving them, will become clear by referring to the embodiments described below in detail together with the accompanying drawings. However, the present invention is not limited to the embodiments disclosed below but may be implemented in various different forms. These embodiments are provided merely to ensure that the disclosure of the present invention is complete and to fully inform those skilled in the art of the scope of the invention, and the present invention is defined only by the scope of the claims. Throughout the specification, the same reference numerals refer to the same components.

[0025] In describing the embodiments of the present invention, specific descriptions of known functions or configurations will be omitted if it is determined that such detailed descriptions could unnecessarily obscure the essence of the invention. Furthermore, the terms described below are defined in consideration of their functions in the embodiments of the present invention, and these definitions may vary depending on the intentions or practices of the user or operator. Therefore, such definitions should be based on the content throughout this specification.

[0026] Hereinafter, embodiments of the present invention will be described in detail with reference to the attached drawings.

[0027] FIG. 1 is a block diagram of a video conferencing system for privacy protection according to one embodiment of the present invention, and FIG. 2 to 5 are drawings for explaining the detailed configuration of a video conferencing system for privacy protection according to one embodiment of the present invention.

[0028] Referring to FIGS. 1 to 5, a video conferencing system for privacy protection according to one embodiment of the present invention may include a user terminal (100), a video conferencing server (200), a counterpart terminal (300), a communication network (400), etc.

[0029] The user terminal (100) is a terminal that requests privacy protection through hand gestures while conducting an online video conference through a video conferencing app, and may include, for example, a smartphone, a mobile phone, a navigation device, a computer, a laptop, a digital broadcasting terminal, a PDA (Personal Digital Assistants), a PMP (Portable Multimedia Player), a tablet PC, a game console, a wearable device, an IoT (Internet of Things) device, a VR (Virtual Reality) device, an AR (Augmented Reality) device, etc.

[0030] This user terminal (100) can connect to a video conferencing server (200) via a communication network (400), download and install a video conferencing app (e.g., Zoom, Microsoft Teams, Google Meet, etc.), run the installed video conferencing app, and create a user account to log in as needed.

[0031] Here, to explain the process of conducting an online video conference, a host who will host the online video conference (e.g., a user of a user terminal (100), a participant of a counterpart terminal (300), etc.) can set the meeting schedule (e.g., meeting date and time, meeting topic, etc.) through menus such as creating a new meeting or scheduling a meeting via a video conference app installed on the terminal, and can share video conference connection information such as a meeting link, ID, and password with the participants via a landline phone, mobile phone, email, message, etc., and the user and the participant can select the meeting link provided by the host through their own terminal, and then enter the ID, password, etc. through the running video conference app or automatically join the online video conference.

[0032] The user terminal (100) described above can take a user video containing video and audio through a camera, microphone, etc. while participating in an online video conference and transmit it to a video conference server (200) via a communication network (400), and can receive a participant video of a counterpart terminal (300) from the video conference server (200) and output it through a video conference app.

[0033] Meanwhile, the user terminal (100) can transmit a user video containing a hand gesture to a video conferencing server (200) via a video conferencing app when the user expresses a hand gesture for privacy protection, and the hand gesture can be expressed in various ways, such as using the right hand, using the left hand, or finger shapes.

[0034] The video conferencing server (200) provides online video conferencing through a video conferencing app and, as illustrated in FIG. 2, detects a hand gesture in a user video transmitted from a user terminal (100), and if the detected hand gesture is a privacy-protecting hand gesture for privacy protection, generates a privacy-protecting video corresponding to the privacy hand gesture and transmits it to a counterpart terminal (300). The server may include a communication module (210), a video conferencing provision unit (220), a hand gesture detection unit (230), a face background separation unit (240), a protection video generation unit (250), a voice control unit (260), etc.

[0035] Here, the communication module (210) is a component responsible for communication with the user terminal (100) and the counterpart terminal (300), and can transmit installation data for installing a video conferencing app according to each request from the user terminal (100) and the counterpart terminal (300), and can receive a user video transmitted through the video conferencing app from the user terminal (100) and transmit it to the counterpart terminal (300), and can receive a participant video transmitted through the video conferencing app from the counterpart terminal (300) and transmit it to the user terminal (100).

[0036] Additionally, the communication module (210) can transmit a privacy-protected video generated through the protection video generation unit (250) to the counterpart terminal (300) when a privacy-protected hand gesture is detected in a user video transmitted from the user terminal (100).

[0037] This communication module (210) can use encryption technology for data protection during the transmission and reception of user video and participant video. For example, data security can be performed through data encryption, AES encryption, etc., during data transmission and reception using SSL / TLS, thereby preventing unauthorized access to transmitted and received data.

[0038] The video conferencing provider (220) is a component that provides a video service for online video conferencing by providing a video conferencing app. It provides a video conferencing app for providing online video conferencing services, and can provide installation data for installing the video conferencing app in response to a request from a user terminal (100) and a counterpart terminal (300), and transmit it to the user terminal (100) and the counterpart terminal (300) respectively through a communication module (210).

[0039] In addition, the video conference provider (220) can manage accounts, authentication, etc. corresponding to the user corresponding to the user terminal (100) and the participant corresponding to the counterpart terminal (300), and can set each authority by managing user information and participant information, and can provide a meeting schedule management function so that when the host sets a meeting schedule, video conference connection information including a meeting link, ID, password, etc. can be generated and stored and managed.

[0040] Additionally, the video conferencing provider (220) can process and transmit user video and participant video (including video, voice, chat messages, etc.) transmitted from the user terminal (100) and the counterpart terminal (300), respectively, connected via a video conferencing app, by compressing, transmitting, decoding, etc., to transmit them to the counterpart in real time, and can process and transmit real-time video data and real-time voice data through, for example, WebRTC.

[0041] In the manner described above, the video conferencing provider (220) can provide an online video conference to the user terminal (100) and the counterpart terminal (300) respectively through a video conferencing app, and the video conference data generated therefrom can be recorded and stored, and then extracted as needed (e.g., a download request) after the online video conference has ended and provided to the user terminal (100) and the counterpart terminal (300).

[0042] The hand gesture detection unit (230) is a component that detects hand gestures in a user image and determines whether the detected hand gesture is a privacy protection hand gesture. It can detect and determine hand gestures by detecting hand feature points and classifying finger shapes using a machine learning algorithm, and can determine whether the hand gesture is a video protection gesture or a voice protection gesture among privacy protection hand gestures so as to perform image processing or voice processing for privacy protection.

[0043] This hand gesture detection unit (230) can be implemented, for example in Python, using Google’s Mediapipe library, for hand gesture recognition including hand feature point detection and finger shape classification, and can extract 21 types of hand feature points in real time from a user video through machine learning as shown in FIG. 3.

[0044] For example, first, a palm area is extracted from the user image, and hand feature points are extracted within the extracted palm area using a feature point extraction algorithm. Then, the hand gesture can be classified into three types to enable selective image protection through the extracted hand feature points. As shown in Fig. 4, it can be classified into a type where the thumb is extended (i.e., the thumb is extended toward the face), a type where the little finger is extended, and a type where both the thumb and little finger are extended.

[0045] Here, if hand feature points are extracted as a type where the thumb is extended, it can be identified as a video protection gesture that protects only the user's face; if hand feature points are extracted as a type where the little finger is extended, it can be identified as a video protection gesture that protects only the user's background; and if hand feature points are extracted as a type where both the thumb and little finger are extended, it can be identified as a video protection gesture that protects both the user's face and the user's background.

[0046] The feature point extraction algorithm described above (e.g., Mediapipe algorithm, etc.) can effectively classify hand gestures into three types by performing hand feature point extraction and outputting the hand gestures in the current user video (i.e., video frame) in the form of an array. Specifically, if only the thumb of the right hand is extended, it can be output as [1,0,0,0,0]; if only the little finger is extended, it can be output as [0,0,0,0,1]; and if both the thumb and little finger are extended, it can be output as [1,0,0,0,1].

[0047] Meanwhile, the hand gesture detection unit (230) may provide a privacy protection release signal to the video conference providing unit (220) to transmit a user video transmitted from the user terminal (100) to the counterpart terminal (300) when the hand gesture is a privacy protection release hand gesture while providing a privacy protection video.

[0048] For example, the hand gesture detection unit (230) can extract a palm area from a user image, and if the extracted palm area is a right hand area, it can be determined as a video protection gesture through hand feature point extraction as shown in FIG. 4, and if the extracted palm area is a left hand area, it can be determined as a privacy protection release hand gesture (i.e., when the same finger type is detected with the left hand), and can provide a privacy protection release signal corresponding to this to the video conference providing unit (220).

[0049] Of course, in one embodiment of the present invention, the right-hand area was described as the privacy protection hand gesture and the privacy release hand gesture as the left-hand area; however, the settings for this can be configured by switching between the right-hand area and the left-hand area according to the user's choice.

[0050] Meanwhile, the hand gesture detection unit (230) extracts a palm area from a user image and extracts hand feature points within the extracted palm area using a feature point extraction algorithm. After that, it can recognize hand gestures to perform selective voice protection through the extracted hand feature points, and can determine that two fingers (e.g., index finger and middle finger) are spread out as a voice protection gesture.

[0051] For example, when an index finger (i.e., landmark 8 in FIG. 3) and a middle finger (i.e., landmark 12 in FIG. 3) are detected in a user image (i.e., a video frame), they can be identified as a voice protection gesture, and a gap corresponding to the straight-line distance of each end of the two fingers can be measured as shown in FIG. 5, and normalization can be performed as shown in Equation 1 below so that the volume value can be determined between 1 and 100.

[0052]

[0053] Here, minval and maxval represent the minimum and maximum values ​​of the gap between the ends of the index finger and middle finger, respectively. Since setting the maximum value too high causes inconvenience as the two fingers must be spread too far apart to create a volume of 100, the maximum value must be set appropriately in correspondence with the maximum gap that can be spread. By applying clipping as shown in Equation 2 below, it is possible to ensure that the volume clip (vol_clip) does not exceed 100.

[0054]

[0055] For example, as illustrated in FIG. 5, a voice protection gesture can be identified to apply mute or a specific volume value depending on the spacing of the fingers. If the spacing between two fingers of the right hand (i.e., the index finger and the middle finger) changes, it can be identified as a voice protection gesture intended to immediately adjust the volume value. If a hand gesture, such as clenching a fist while the left hand is visible on the screen, is extracted after the voice output control (i.e., mute, volume control, etc.) has been adjusted to the desired volume value, it can be identified as a voice output determination gesture to complete the voice output control.

[0056] Through the process described above, the hand gesture detection unit (230) can provide the determination result to the face background separation unit (240) to adjust the video output for privacy protection when the hand gesture is determined to be a video protection gesture, and can provide the determination result to the voice control unit (260) to adjust the voice output for privacy protection when the hand gesture is determined to be a voice protection gesture.

[0057] The face background separation unit (240) is a component that separates the user's face and user background from the user image when the hand gesture is a video protection gesture among privacy protection hand gestures. It can detect the face area through a face area detection algorithm (e.g., Mediapipe's face detection, etc.), and as shown in FIG. 2, the face area can be detected as a rectangular box, and four major feature points of the face (e.g., left eye, right eye, tip of the nose, lips) can be extracted as red dots.

[0058] In addition, the face background separation unit (240) can separate a circular area of ​​radius size covering the face area centered on the tip of the nose among the feature points marked in the detected face area as the user's face, and can separate the background area excluding the person area as the user background through a person area separation algorithm (e.g., Python's Pixellib library, etc.).

[0059] The detection results for these user faces and user backgrounds can be provided to a protection image generation unit (250) for generating a privacy protection image.

[0060] The protection image generation unit (250) is a component that generates a privacy protection image in response to an image protection gesture and can control the image output of at least one of the user's face and user background. When the image protection gesture is a hand gesture that protects the user's face (i.e., when the thumb is extended), it can generate and provide a privacy protection image in which a circular area according to the determination result of the face background separation unit (240) is color-processed with a preset color (e.g., green).

[0061] Additionally, the protection image generation unit (250) can generate and provide a privacy protection image in which the background area excluding the person area is blurred according to the determination result of the face background separation unit (240) when the image protection gesture is a hand gesture that protects the user background (i.e., when the little finger is extended).

[0062] Additionally, the protection image generation unit (250) can generate and provide a privacy protection image with the entire area of ​​the user image blurred when the image protection gesture is a hand gesture that protects both the user's face and the user's background (i.e., when both the thumb and little finger are extended).

[0063] Meanwhile, the protection image generation unit (250) can blur the user background or, when processing the entire user image, detect the human region using a human region segmentation algorithm (e.g., Python's Pixellib library, etc.) and define it in pixel units (Human region segmentation of FIG. 2), and can process the background area excluding the human region by dividing it into pixel units or process the entire user image by dividing it into pixel units using the change_bg.blur_frame() function, and can set the degree of blur processing to three levels (e.g., low, moderate, extreme).

[0064] The voice control unit (260) is a component that controls the voice output in the user video or privacy protection video when the hand gesture is a voice protection gesture among privacy protection hand gestures. When two set fingers are spread out, it can be determined as a voice protection gesture through the hand gesture detection unit (230). When it is a voice protection gesture, depending on the distance between the ends of the two fingers, it can perform voice output control such as removing the voice or adjusting the volume in the user video transmitted by the video conferencing provider (220) or the privacy protection video transmitted by the protection video generation unit (250).

[0065] As shown in Fig. 4, when a voice protection gesture is determined to apply mute or a specific volume value according to the finger spacing, the normalized volume value can be displayed on the user image as a bar graph and a number (e.g., 35, 98, etc.) using the cv2.rectangle() and cv2.putText() functions of the OpenCV library.

[0066] Meanwhile, voice output control that immediately adjusts the volume value can be performed when the distance between two fingers of the right hand (i.e., the index finger and the middle finger) changes. After the voice output control (i.e., mute, volume adjustment, etc.) is adjusted to the desired volume value, voice output control can be completed if a hand gesture, such as clenching a fist while the left hand is visible on the screen, is identified as a voice output determining gesture.

[0067] The counterpart terminal (300) is a terminal that provides privacy protection video in response to privacy protection while attending an online video conference through a video conferencing app, and may include, for example, a smartphone, a mobile phone, a navigation device, a computer, a laptop, a digital broadcasting terminal, a PDA (Personal Digital Assistants), a PMP (Portable Multimedia Player), a tablet PC, a game console, a wearable device, an IoT (Internet of Things) device, a VR (Virtual Reality) device, an AR (Augmented Reality) device, etc.

[0068] This counterpart terminal (300) can connect to the video conferencing server (200) via the communication network (400), download and install a video conferencing app (e.g., Zoom, Microsoft Teams, Google Meet, etc.), run the installed video conferencing app, and create a user account to log in as needed.

[0069] The counterpart terminal (300) described above can participate in an online video conference, receive user video, and while the conference is in progress, capture a video of the participant including video and audio through a camera, microphone, etc., and transmit it to the video conference server (200) via the communication network (400).

[0070] Meanwhile, the counterpart terminal (300) may receive a privacy protection video corresponding to the user's privacy protection request from the video conferencing server (200) while transmitting the participant video and attending an online video conference through a video conferencing app, and output it through the video conferencing app. Here, the privacy protection video may be output in the form of a blurred image of the face, background, etc., or an image with adjusted audio output.

[0071] The communication network (400) provides a mutual communication environment for each of the user terminal (100), the video conferencing server (200), and the counterpart terminal (300), and may include, for example, a mobile communication network, a wired internet communication network, a wireless internet communication network, a broadcasting network, an administrative network (closed network), and specifically, may include one or more networks such as a PAN (personal area network), LAN (local area network), CAN (campus area network), MAN (metropolitan area network), WAN (wide area network), BBN (broadband network), the Internet, and LoRaWAN, and may include one or more network topologies such as a bus network, a star network, a ring network, a mesh network, a star-bus network, a tree network, and a hierarchical network.

[0072] Through this communication network (400), a communication environment for transmitting and receiving wired and wireless data between each of the user terminal (100), video conferencing server (200), and counterpart terminal (300) can be provided.

[0073] Meanwhile, in one embodiment of the present invention as described above, a hand gesture requesting privacy protection is expressed at the user terminal (100), and in response, a privacy protection video is provided at the counterpart terminal (300) of the participant attending the online video conference. However, it is also possible to express a hand gesture requesting privacy protection at the counterpart terminal (300) and in response, provide a privacy protection video at the user terminal (100) of the user attending the online video conference. Furthermore, it is obvious that if a hand gesture requesting privacy protection is expressed at the user terminal (100) and the counterpart terminal (300) respectively, a privacy protection video can be provided at both the user terminal (100) and the counterpart terminal (300) in response.

[0074] In addition, although the present invention has been described in one embodiment as having only one counterpart terminal (300), it is obvious that a privacy-protected video can be provided through a similar process even when there are multiple counterpart terminals (300) (i.e., when there are multiple participants).

[0075] Accordingly, according to one embodiment of the present invention, while providing an online video conference through a video conferencing app, a hand gesture is detected in a user video transmitted from a user terminal, and if the detected hand gesture is a privacy-protecting hand gesture for privacy protection, a privacy-protecting video corresponding to the privacy hand gesture is generated and transmitted to a counterpart terminal, thereby enabling immediate reflection of the user's real-time gesture and protection of privacy.

[0076] FIG. 6 is a flowchart illustrating the process of protecting privacy using a video conferencing system according to another embodiment of the present invention. Here, since the specific details regarding the process of protecting privacy using a video conferencing system have been described in detail in one embodiment of the present invention, the process will be described in a general manner below.

[0077] Referring to FIG. 6, the video conferencing server (200) can provide an online video conference to the user terminal (100) and the counterpart terminal (300) respectively through a video conferencing app (step 510).

[0078] Here, the video conferencing server (200) can provide an online video conference by processing a user video transmitted from a user terminal (100) and transmitting it to a counterpart terminal (300), and by processing a participant video transmitted from a counterpart terminal (300) and transmitting it to a user terminal (100).

[0079] And, while conducting an online video conference on the user terminal (100), a request for privacy protection can be made via hand gesture (step 520).

[0080] Next, the video conferencing server (100) can detect hand gestures in the user video transmitted from the user terminal (100) (step 530).

[0081] In the step (530) of detecting the hand gesture in the above user video, the hand gesture can be detected by detecting hand feature points and classifying finger shapes using a machine learning algorithm.

[0082] And, it can determine whether the hand gesture detected by the video conferencing server (200) is a privacy protection hand gesture for privacy protection (step 540).

[0083] In the step (540) of determining whether the hand gesture is a privacy protection hand gesture, the hand gesture is determined to be a privacy protection hand gesture, and the hand gesture can be determined to be a video protection gesture or a voice protection gesture among the privacy protection hand gestures.

[0084] Based on the determination result in the above step (540), the video conferencing server (200) can generate a privacy protection video corresponding to the privacy hand gesture and transmit it to the counterpart terminal (300) (step 550).

[0085] In the step (550) of generating and transmitting the privacy protection video, if the hand gesture is a video protection gesture, the user's face and user background are separated from the user video, and a privacy protection video is generated in correspondence with the video protection gesture, and at least one of the user's face and user background can be adjusted for video output.

[0086] Meanwhile, if the hand gesture is a voice protection gesture, the audio output can be controlled in the user video or privacy protection video.

[0087] Next, while attending an online video conference on the other terminal (300), a privacy-protected video transmitted from the video conference server (200) in response to privacy protection can be provided (step 560).

[0088] Meanwhile, while providing a privacy-protected video, if the hand gesture detected by the video conferencing server (200) is a privacy-protected release hand gesture, a privacy-protected release signal can be provided to transmit the user video transmitted from the user terminal (100) to the counterpart terminal (300).

[0089] Accordingly, according to another embodiment of the present invention, while providing an online video conference through a video conferencing app, a hand gesture is detected in a user video transmitted from a user terminal, and if the detected hand gesture is a privacy-protecting hand gesture for privacy protection, a privacy-protecting video corresponding to the privacy hand gesture is generated and transmitted to a counterpart terminal, thereby enabling immediate reflection of the user's real-time gesture and protection of privacy.

[0090] Although various embodiments of the present invention have been presented and described in the above description, the present invention is not necessarily limited thereto, and those skilled in the art will readily understand that various substitutions, modifications, and changes are possible within the scope of the technical concept of the present invention. Explanation of the symbols

[0091] 100 : User terminal 200 : Video conferencing server 210 : Communication Module 220 : Video conferencing provider 230 : Hand gesture detection unit 240 : Face background separation section 250 : Protection Video Generation Unit 260 : Voice control unit 300 : Counter terminal 400 : Communication network

Claims

Claim 1 A user terminal requesting privacy protection via hand gesture while conducting an online video conference through a video conferencing app; a counterpart terminal providing a privacy protection video in response to the privacy protection while attending the online video conference through the video conferencing app; A video conferencing server that provides the online video conference through the video conferencing app, detects the hand gesture in the user video transmitted from the user terminal, and, if the detected hand gesture is a privacy-protecting hand gesture for privacy protection, generates the privacy-protecting video corresponding to the privacy-protecting hand gesture and transmits it to the counterpart terminal; wherein the video conferencing server comprises: a communication module unit responsible for communication with the user terminal and the counterpart terminal; a video conferencing provision unit that provides the video conferencing app to provide a video service for the online video conference; a hand gesture detection unit that detects the hand gesture in the user video and determines whether the detected hand gesture is a privacy-protecting hand gesture; and a face-background separation unit that separates the user face and the user background in the user video if the hand gesture is a video-protecting gesture among the privacy-protecting hand gestures. A protection image generation unit that generates the privacy protection image in response to the above image protection gesture, and controls the image output of at least one of the user face and user background;The hand gesture detection unit includes, wherein the hand gesture detection unit detects and identifies the hand gesture by detecting hand feature points and classifying finger shapes using a machine learning algorithm, and extracts 21 types of hand feature points in real time from the user image using the machine learning algorithm, wherein the palm area is extracted from the user image, and after extracting the hand feature points within the extracted palm area using a feature point extraction algorithm, the hand gesture is classified into a type where the thumb is extended, a type where the little finger is extended, and a type where both the thumb and little finger are extended so as to perform selective image protection through the extracted hand feature points, and is identified as an image protection gesture that protects only the user face, only the user background, or both the user face and the user background according to the classified type, wherein the feature point extraction algorithm uses a Mediapipe algorithm, and in the case where the hand gesture is only the right thumb extended A video conferencing system for privacy protection that outputs [1,0,0,0,0], outputs [0,0,0,0,1] when only the little finger of the right hand is extended, and outputs [1,0,0,0,1] when both the thumb and little finger of the right hand are extended; wherein the hand gesture detection unit, while providing the privacy protection video, provides a privacy protection release signal to the video conferencing unit to transmit the user video transmitted from the user terminal to the counterpart terminal when the hand gesture is a privacy protection release hand gesture, and after extracting the palm area from the user video, determines it as the image protection gesture through hand feature point extraction if the extracted palm area is either the right hand area or the left hand area, and determines it as the privacy protection release hand gesture if the extracted palm area is the other of the right hand area and the left hand area. Claim 2 delete Claim 3 delete Claim 4 delete Claim 5 A video conferencing system for privacy protection according to claim 1, wherein the video conferencing server further comprises a voice control unit that controls voice output in the user video or privacy protection video when the hand gesture is a voice protection gesture among the privacy protection hand gestures. Claim 6 A step of providing an online video conference to a user terminal and a counterpart terminal, respectively, via a video conference app on a video conference server; a step of requesting privacy protection via a hand gesture while the online video conference is being conducted on the user terminal; a step of detecting the hand gesture in a user video transmitted from the user terminal on the video conference server; a step of determining whether the detected hand gesture is a privacy protection hand gesture for privacy protection on the video conference server; a step of generating and transmitting a privacy protection video corresponding to the privacy protection hand gesture on the video conference server; and a step of providing the privacy protection video in response to privacy protection while the counterpart terminal is attending the online video conference.The method includes the step of detecting the hand gesture in the user video, wherein the hand gesture is detected by detecting hand feature points and classifying finger shapes using a machine learning algorithm, and the step of determining whether it is a privacy-protecting hand gesture is determined whether the hand gesture is a privacy-protecting hand gesture, wherein the hand gesture is determined whether it is a video-protecting gesture or a voice-protecting gesture among the privacy-protecting hand gestures, wherein the hand gesture detection unit of the video conferencing server extracts 21 types of hand feature points in real time from the user video using the machine learning algorithm, wherein the palm area is extracted from the user video, and after extracting the hand feature points within the extracted palm area using a feature point extraction algorithm, the hand gesture is classified into a type where the thumb is extended, a type where the little finger is extended, and a type where both the thumb and little finger are extended so as to perform selective video protection through the extracted hand feature points, and the classified type Accordingly, the image protection gesture is determined to protect only the user's face, only the user's background, or both the user's face and the user's background; in the case of the feature point extraction algorithm, the Mediapipe algorithm is used, wherein the hand gesture is output as [1,0,0,0,0] if only the right thumb is extended, as [0,0,0,0,1] if only the right little finger is extended, and as [1,0,0,0,1] if both the right thumb and little finger are extended; and while providing the privacy protection image, if the hand gesture is a privacy protection release hand gesture, a privacy protection release signal is provided to transmit the user image transmitted from the user terminal to the counterpart terminal;A video conferencing system and a privacy protection method using the same, comprising: a hand gesture detection unit extracting the palm area from the user image; determining the image protection gesture through hand feature point extraction if the extracted palm area is either a right hand area or a left hand area; and determining the privacy protection release hand gesture if the extracted palm area is the other of the right hand area and the left hand area. Claim 7 delete Claim 8 delete Claim 9 In claim 6, the step of generating and transmitting the privacy-protecting image comprises, when the hand gesture is the image-protecting gesture, separating the user face and the user background from the user image, and then generating the privacy-protecting image in correspondence with the image-protecting gesture, wherein at least one image output of the user face and the user background is adjusted. A video conferencing system and a privacy protection method using the same. Claim 10 delete Claim 11 In claim 9, the privacy protection method further comprises the step of adjusting voice output in the user video or privacy protection video when the hand gesture is the voice protection gesture; a video conferencing system and a privacy protection method using the same.

Citation Information

Patent Citations

  • System and method for digital image processing by wearable glass device

    KR1020160011302A

  • System and method for deep learning based hand gesture recognition in first person view

    US20190107894A1

  • Capturing and displaying images with multiple focal planes

    US20200106952A1

  • Gesture-initiated actions in videoconferences

    US9445048B1