Using inclusion zones in video conferencing

Abstract

Devices, systems, and methods are provided for using a containment zone for a video conference. The method comprises: capturing an image of a location; applying a subject detector model to the image to identify chamber coordinates for each subject detected in the image; and defining an inclusion region for the location. The inclusion region is based on a top-down view of the location. The method further includes determining whether chamber coordinates for each subject are within the inclusion region; filtering data associated with a subject determined not to be within the inclusion region; and processing data associated with the subject determined to be within the inclusion region.

Description

Background Art

[0001] Various techniques attempt to provide an acoustic fence around a video conferencing area to help reduce external noise, such as from ambient noise or from other individuals. In one variation, microphones are arranged in a perimeter around the video conferencing area and used to detect background or far-field noise, which can then be subtracted or used to mute or unmute the main microphone audio. This technology requires multiple microphones located in various places around the video conferencing area. In another variation, the acoustic fence is set to be within a certain angle of the centerline of the video conferencing camera or a certain angle of the sensing microphone array. If the microphone array is located in the camera body, the centerline of the microphone array and the camera can match. This results in the acoustic fence blocking the area outside a certain angle of the centerline of the array, which is generally an angle related to the camera field of view, and it is expected that the capture angle can be manually changed. BRIEF DESCRIPTION OF THE DRAWINGS

[0002] Figure 1 is a top view of an example conference room according to some aspects of the present disclosure.

[0003] Figure 2 is an isometric schematic diagram of another example conference room with three individuals located at different coordinate positions relative to a video conferencing camera.

[0004] Figure 3 yes Figure 2 Top view of a sample conference room.

[0005] Figure 4 is an isometric diagram of yet another example conference room with three individuals located at different coordinate positions according to some examples of the present disclosure.

[0006] Figure 5 is a schematic diagram of a camera and a two-dimensional image plane with example determination of room coordinates for a head bounding box according to an example of the present disclosure.

[0007] Figure 6 is a front view of yet another example conference room with three individuals located at different coordinate positions according to some examples of the present disclosure.

[0008] Figure 7 is an isometric schematic diagram of another example conference room with a single person located at four different coordinate positions relative to a video conferencing camera according to some examples of the present disclosure.

[0009] Figure 8 yes Figure 7 Top view of a sample conference room.

[0010] Fig. 9is an example screen shot of a GUI for configuring room dimensions according to some examples of the present disclosure.

[0011] Fig.10 is another example screen shot of a GUI for configuring the size of an inclusion zone according to some examples of the present disclosure.

[0012] Fig.11 is a front view of yet another example conference room with four individuals located in the conference room and two individuals located outside the conference room according to some examples of the present disclosure.

[0013] Fig.12 is drawn on the room size chart Fig.11 A top-down view of an example conference room.

[0014] Fig.13 The invention is a flowchart of a method for implementing a video conferencing system containing an area in a conference room according to an example of the present disclosure.

[0015] Fig.14 is a front view of a camera according to an example of the present disclosure.

[0016] Fig.15 is a flow chart of a method of determining image plane coordinates for a detected subject according to an example of the present disclosure.

[0017] Fig.16 is a schematic diagram of an example codec according to an example of the present disclosure. DETAILED DESCRIPTION

[0018] In a video conferencing system, framing of individuals in a video conference room can be improved by determining the positions of individual participants in the room relative to each other or specific reference points. For example, if person A is sitting 2.5 meters from the camera and person B is sitting 4 meters from the camera, the ability to detect this position information can enable various advanced framing and tracking experiences. For example, participant position information can be used to define an inclusion zone in the camera's field of view (FOV) that excludes people outside the inclusion zone from being framed and tracked in the video conference.

[0019] When the microphone array of a video conferencing system is used in a public place or a large conference room with two or more participants, background sound, side conversations or distracting noise may be present in the audio signal recorded by the microphone array and output to other participants in the video conference. This is particularly true when the background sound, side conversations or distracting noise originate from within the field of view (FOV) of the camera used to record visual data for the video conferencing system. When the microphone array is being used to capture the user's voice as audio for use in a teleconference, one or more other participants in the conference may hear background sound, side conversations or distracting noise on their corresponding audio devices or speakers. Further, no industrial standard or specification has been developed to reduce unwanted sounds in a video conferencing system based on the following distance: the unwanted sound is determined to originate from this distance relative to the video conferencing camera.

[0020] For many applications, it is useful to know the horizontal and vertical positions of participants in the room to provide a more comprehensive and complete understanding of the video conferencing room environment. For example, some techniques operate in only the width (i.e., horizontal) dimension. On the other hand, the ability to determine two-dimensional room parameters (e.g., width and depth) for each conference participant can be achieved by using depth estimation / detection sensors or computationally intensive machine learning-based monocular depth estimation models, but such schemes impose significant hardware and / or processing costs without providing accuracy for measuring participant positions. Further, such schemes do not take into account the effects of each participant's distance from the camera or lens distortion on the detection technology.

[0021] For example, some techniques attempt to incorporate filters or borders onto images captured by a camera to limit unwanted sounds from being transmitted to the far end of a video conference. However, such techniques require multiple microphones and / or do not account for the distance of a person from the camera or the effects of lens distortion on the image when calculating the person's position on the image plane coordinate system. Thus, such calculations erroneously include or exclude people detected in the image, which can cause confusion in the video conference and result in a less desirable experience for the participants.

[0022] Accordingly, in some examples, the present disclosure provides methods and apparatus for implementing inclusion zones to remove or reduce background sounds, side conversations, or other distracting noises from a video conference. In particular, the present disclosure provides methods for calibrating inclusion zones for images captured by a video conferencing system to select data (e.g., audio or visual data) associated with the subject of the video conference for downstream processing in the video conference. By utilizing the inclusion zone methods and apparatus discussed herein, communications between participants in a conference call can be clearer, and the overall video conferencing experience can be more enjoyable for the video conference participants. Further, the methods and apparatus discussed herein are applicable to a wide variety of different locations and room designs, meaning that the disclosed methods can be easily assembled and applied to any particular location, including, for example, conference rooms, enclosed rooms, and open concept workspaces.

[0023] As an example, Figure 1 An example conference room 10 for use in a video conference is illustrated. The conference room 10 includes a conference table 12 and a series of chairs 14. People 16 sit in the chairs 14 around the conference table 12, and additional people 16 may be located outside the conference room 10. Figure 1 In the non-limiting example illustrated in FIG. 1 , the first person 16A, the second person 16B, the third person 16C, and the fourth person 16D are seated around the conference table 12, while the fifth person 16E and the sixth person 16F are located outside the conference room 10. Figure 1 An example of a video conference room 10 with six persons 16 is illustrated, but more or fewer persons 16 may be seated around a conference table 12 or otherwise located within the conference room 10 at any given time. Further, more or fewer persons 16 may be located outside of the conference room 10 at any given time. Additional examples of video conference rooms, positions of persons relative to the video conference room, and arrangements of containment zones in a video conference room are discussed in greater detail below.

[0024] Still refer to Figure 1 In some aspects, the video conferencing system 18 may include a camera 20, a microphone array 22, and a monitor 24. More specifically, as Figure 1As shown in the example of , the video conferencing system 18 may include a main or front camera 20. However, it is contemplated that the video conferencing system 18 may include additional cameras (e.g., a secondary or left camera, a tertiary or right camera, and / or other cameras). The camera 20 has a field of view (FOV) 25 (horizontal and vertical) and an axis or centerline (CL) 26 extending in a direction corresponding to the direction in which the camera 20 is pointing (i.e., the straight line of sight of the camera 20 at 90 degrees from its focal point). For example, the camera 20 may have: a horizontal FOV 25A, which pans horizontally (i.e., along the width dimension) in the conference room 10; and a vertical FOV 25B, which pans vertically (i.e., along the height dimension) in the conference room 10. In some aspects, the camera 20 includes: a corresponding microphone array 22, which can be used to record and transmit audio data in the video conference using sound source localization (SSL). In some examples, SSL is used in a manner similar to that described in International Application No. PCT / US2023 / 016764 and U.S. Patent Application Publication No. 2023 / 0053202, which are incorporated herein by reference in their entirety. In some examples, microphone array 22 is housed on or within a housing of camera 20. In addition, video conferencing system 18 may include: a monitor 24 or television that is provided to display one or more remote meeting locations and generally provides speaker output. Monitor 24 can be coupled to front camera 20 and microphone array 22, although it is contemplated that the monitor can be located anywhere in conference room 10 and that video conferencing system 18 can include an additional monitor (not shown) located in conference room 10.

[0025] Further, the centerline 26 of the camera 20 is centered along the conference table 12. In some examples, a center microphone 28 is provided to capture the speaker (i.e., the person speaking) for transmission to the far end of the video conference. In some aspects, a person 16 can be located within the FOV of the camera 20 and / or create noise recorded by the microphone array 22 even if the person 16 is located outside the conference room 10. Figure 1 In the non-limiting example illustrated in , the fifth and sixth persons 16E, 16F are located outside the conference room 10, but may still be within the FOV of the camera 20. For example, the left wall 30 of the conference room 10 is a transparent (e.g., glass) wall, and the fifth and sixth persons 16E, 16F may be visible through the left wall 30. In some aspects, the sounds and / or movements created by the persons 16E, 16F may cause confusion in the video conference and cause distracting noise to be transmitted to the far end of the video conference. Therefore, it may be advantageous to screen or filter the subject data associated with each person 16 based on the location of each person (i.e., the location from the camera 20). This, in turn, may reduce confusion in the video conference by ensuring that only persons actively participating in the video conference are recorded and transmitted to the far end of the video conference.

[0026] Furthermore, the camera 20 and microphone array 22 may be used in combination to define a containment boundary or zone 32, such that data associated with each person 16A, 16B, 16C, 16D within the containment zone 32 may be processed for transmission to the far end of the video conference via the microphone array 22. In this manner, data associated with persons 16E, 16F outside of the containment zone 32 may be filtered, e.g., not relayed to the far end of the video conference.

[0027] To this end, the inclusion zone can act as a boundary for the video conferencing system 18 to distinguish data originating within the boundary from data originating outside the boundary. A variety of different video conferencing technologies can incorporate this distinction to enhance the user experience during a video conference. For example, incorporating an inclusion zone into a video conferencing system can be used to select data to be transmitted to the far end of the video conference and / or select data to be filtered (e.g., muted, blurred, cropped, etc.). In some examples, the inclusion zone can be used to mute audio data (i.e., sounds) originating outside the inclusion zone to achieve the effect of a 2D acoustic fence, such as those described in International Application No. PCT / US2023 / 016764, which is incorporated herein by reference in its entirety. Correspondingly, the inclusion zone can be used to blur video data (i.e., images) that contains a person located outside the inclusion zone, such as a Figure 1 In the example of persons 16E, 16F. In addition, data originating from within the inclusion zone that is selected for transmission to the far end of the video conference can be processed normally, for example using optimal view selection techniques, such as those described in International Application No. PCT / US2023 / 075906 filed on October 4, 2023, which is incorporated herein by reference in its entirety. In addition, the inclusion zone can act as a movement zone, meaning that after a person enters the inclusion zone, the video conferencing system can perform a specified function. As a non-limiting example, after the video conferencing system realizes that a person has entered the inclusion zone, the video conferencing system can display a greeting message or emit a voice prompt. Additional applications of inclusion zones in video conferencing will be discussed in more detail below.

[0028] Accordingly, the use of inclusion zones in a video conferencing setting can prevent and / or eliminate distractions originating from outside the conference room, thereby providing a more desirable video conferencing experience to the far-end participants. The process described herein allows a video conferencing system to define inclusion zones in a conference room to selectively filter data associated with each person based on the position of each person detected by the camera relative to the camera. This is accomplished using an artificial intelligence (AI) or machine learning human head detector model, as discussed below.

[0029] In some aspects, the AI ​​human head detector model, which may also be referred to herein as a subject detector model, is substantially similar to the model described in International Application No. PCT / US2023 / 016764, which is incorporated herein by reference in its entirety. For example, now refer to Figure 2 and 3 , illustrates a conference room 40 having three video conference participants 42, 44, 46 located at different coordinate locations. In the conference room 40, the front camera 20 has a horizontal and vertical FOV, and the camera position relative to the room 40 is indicated by the three-dimensional (3D) coordinates {0, 0, 0}. Further, the front camera 20 captures views of all three participants 42, 44, 46, having positions that can be characterized in terms of a roll angle ΦPAN relative to the centerline 26 of the front camera 20 and a distance metric between the front camera 20 and each participant 42, 44, 46. In particular, the first participant 42 has a position defined by a first roll angle 48 and a first distance 50. Additionally, the second participant 44 has a position defined by a roll angle 52 and a second distance 54, and the third participant 44 has a position defined by a roll angle 56 and a third distance metric 58.

[0030] Now specific reference Figure 3 , which illustrates Figure 2 A top view of an example conference room 40. In some examples, the position of each participant 42, 44, 46 may be determined by a roll angle 48, 52, 56 and from x ROOM Dimensions or axes 60 and y ROOM The distances 50, 54, 58 derived from the dimensions or axes 62 are characterized by the front camera 20 being located at {x ROOM ,y ROOM In particular, the first participant 42 has a position defined by a first roll angle 48 and a first distance metric 50, the first distance metric 50 being characterized by a two-dimensional room distance parameter {-0.5, 1} to indicate that the participant is located along y ROOM Axis 62 is at a "vertical" distance (relative to the top view) of 1 meter measured from front camera 20 and is located along the y axis. ROOM Axis 62 vertical x ROOM In addition, the second participant 44 has a position defined by a second roll angle 52 and a second distance metric 54, which is characterized by the two-dimensional room distance parameters {0, 3} to indicate that the participant is located at a vertical distance of 3 meters (along the y axis). ROOM axis 62) and is located at a horizontal distance of 0 meters (along the x ROOMThe third participant 44 has a position defined by a third roll angle 56 and a third distance metric 58, characterized by the two-dimensional room distance parameters {1, 2.5}, indicating that the participant is located at a vertical distance of 2.5 meters (along the y axis 60). ROOM axis 62) and located at a horizontal distance of 1 meter (along the x ROOM Axis 60 measurement).

[0031] The roll angle value (ΦPAN) and the two-dimensional chamber distance parameter {x ROOM ,y ROOM}, in this reference coordinate table, for Figure 2 and 3 Example of different coordinate positions {x ROOM ,y ROOM The pan angle ΦPAN value for the front camera 20 of the video conference can be calculated for the coordinate position {-x} in the example conference room 40. ROOM ,y ROOM} and calculates the same table (not shown) for the negative roll angle ΦPAN value (eg, -ΦPAN). Therefore, it should be understood that the value of the negative roll angle ΦPAN will be calculated for the centerline 26 (eg, x ROOM = 0) at any depth metric (e.g., y ROOM =0.5-8)) for the conference participants. Similarly, ROOM =y ROOM The same roll angle ΦPAN value (eg, ΦPAN=45) is generated for conference participants at any coordinate position of the conference participant. As illustrated, the roll angle ΦPAN alone may not be used to determine the two-dimensional room distance parameter {x ROOM ,y ROOM}. For example, the first participant 42 may appear to be at a greater distance from the front camera 20 than the second participant 44 due to vanishing point perspective. Thus, as the conference participants move farther from the front camera 20, the apparent height and width of the participants become closer to the video conferencing system, and when projected onto the camera image sensor 64, the conference participants are represented using a smaller number of pixels than participants closer to the front camera 20. Further, if two heads are viewed as the same size by the front camera 20, they are not necessarily located at the same distance, and as Figure 3 As shown in the figure, they are in two dimensions x ROOM -y ROOMThe position in plane 66 may differ due to roll angle Φ PAN and distortion in height and width.

[0032] In particular, the statistical distribution of human head height and width measurements can be used to determine a minimum-median-maximum metric for participant head sizes in centimeters. Additionally, by knowing the FOV resolution of the front camera 20 in both the horizontal and vertical directions with corresponding horizontal and vertical pixel counts, the measured angular range of each head can be used to calculate the percentage of the overall frame occupied by the head and the number of pixels for the head height and width metrics. Using this information to calculate a lookup table for the minimum-median-maximum head sizes (height and width) at various distances, an artificial (AI) human head detector model can be applied to detect the position of each head in a two-dimensional viewing plane with specified image plane coordinates and the associated width and height metrics for the head frame or bounding box (e.g., {x box ,y box , width, height}). By using a reverse lookup table operation, the distance between the front camera 20 and each head located on the center line 26 of the front camera 20 can be determined.

[0033] In some examples, the subject detection process is similar to the AI ​​head detection process disclosed in U.S. Patent Application No. 17 / 971,564, filed on October 22, 2022, which is incorporated herein by reference in its entirety. Figure 4 , the front camera 20 is used to provide images of conference participants taken along a two-dimensional image plane 110. Conference participants can be located at a first central position 112 and at x ROOM In the first centered position, the conference participant is located at a distance d0=Y meters along the centerline 26 of the camera 20 (e.g., ΦPAN=0), so the two-dimensional room distance parameter for the first centered position 112 is {x ROOM =0,y ROOM =Y}. In the second roll position, at x ROOM The conference participant is displaced laterally by the roll angle ΦPAN in the direction and is positioned at d1>d0 meters. Therefore, the two-dimensional room distance parameter for the second roll position 114 is {x ROOM =P,y ROOM=Y}. Further, the same vertical head height metric V / 2 for conference participant positions 112, 114 will result in an angular range θFRAME_V1 / 2 for the first conference participant position 112 that is greater than the angular range θFRAME_V2 / 2 for the second conference participant position 114. In fact, the fact that the second pan position 114 is located farther from the front camera 20 than the first centered position 112 (d1>d0) causes the angular range for the second pan position 114 to appear smaller than the angular range for the first centered position 112, such that θFRAME_V1 / 2>θFRAME_V2 / 2.

[0034] Based on the above, the problem is to find the angle range θ for the entire head height HH , and then express it as a percentage of the full frame vertical field of view (VFrame_Percentage), which is then converted into the number of pixels that the head will occupy at a specific distance and at a roll angle ΦPAN (VHead_Pixel_Count). To do this, we can use the equation tan(θ HH1 / 2)=(V / 2) / d0 to calculate the angle range θ for the entire head height of the first conference participant position 112 HH1 . Solving the angle range θ1, the angle range θ for the entire head height can be HH1 Calculated as θ HH1 =2arctan((V / 2) / d0). In a similar manner, the equation tan(θ HH2 / 2)=(V / 2) / d1 to calculate the angular range θ of the entire head height for the second conference participant position 114 at the roll angle ΦPAN HH2 ,in Solving angle range θ HH2 , the angle range θ for the entire head height can be HH2 Calculated as Based on this calculation, the percentage of the frame occupied by the head height for the second conference participant position 114 can be calculated as VFrame_Percentage=θ HH2 / Vertical FOV. Additionally, the corresponding number of pixels for the head height of the second conference participant position 114 can be calculated as VHead_Pixel_Count in pixels = VFrame_Percentage x Vertical FOV. Based on the above calculation, x equivalent to the various angles of pan angle ΦPAN that can be listed in a lookup table (not shown) can be calculated. ROOM and ROOMThe angular range θ for the entire head height is calculated at discrete distances of, for example, 0.5 meters in each of the directions HH =θFRAME_V.

[0035] Figure 5 A front camera 20 and a video conference room 200 including a two-dimensional image plane 210 are illustrated to illustrate how the distance metric X can be calculated by computing a direct distance metric HYP between the front camera 20 and the location of the conference participants. ROOM (m) Calculate the vertical or depth distance Y to the meeting participant's location ROOM (meters). The two-dimensional image plane 210 includes: a plurality of two-dimensional coordinate points 212, 214, 216, which are obtained by using the coordinates {x i ,y i} defined. In addition, reference is made to the head bounding box 218, the width dimension (along the x i axis) and the height dimension (measured along the y i The head bounding box 218 is defined by the starting coordinate point {x1, y1} (measured from the axis). In order to locate the vertical or depth distance Y from the front camera 20 ROOM (meters), the vertical angular extent (θ) for the head bounding box 218 is calculated as θ=Height*V_FOV / V_PIXELS, where Height is the height of the head bounding box in pixels, where V_FOV is the vertical FOV in degrees, and where V_PIXELS is the vertical FOV in pixels. Next, the vertical angular extent (θ / 2) is calculated for the upper half of the head bounding box and used to derive the direct distance metric HYP between the front camera 20 and the conference participant location: HYP=V_HEAD / (2 x tan(θ / 2)), where HYP is the direct distance metric to the conference participant location at the roll angle ΦPAN. Finally, the Pythagorean theorem is used to derive the distance metric HYP from the direct distance metric and the distance metric x ROOM (m) Derived vertical or depth room distance Y ROOM (rice):

[0036] Utilizing this understanding of the AI ​​human head detector model, the present disclosure provides methods, devices, systems, and computer-readable media for accurately determining whether the source of subject data (e.g., audio or visual data) originates from within an inclusion zone defined by a video conferencing system. The position of each person with respect to the camera's FOV is determined by the AI ​​human head detector model using room distance parameters, as discussed above. In particular, coordinates, such as image and / or world coordinates, are determined for each person in the camera view. In some aspects, the world coordinates identified by the AI ​​human detector model are referred to as world coordinate points. The world coordinates of the human head are then compared with the room parameters corresponding to (one or more) inclusion zones defined by the video conferencing system. In this way, it becomes possible to determine whether the data (e.g., an image of a specific head captured by a camera or a sound recorded by a microphone array) has originated from within an area defined by an inclusion zone or from outside an area defined by an inclusion zone. If the data is determined to have originated from within an inclusion zone, the video conferencing system processes the data and transmits the data to the far end of the video conference. However, if the data is determined to have originated from outside the inclusion zone, the video conferencing system processes the data in a different manner (e.g., filters the data) and may not transmit the data to the remote participants in the video conference. Any suitable filtering techniques may be used to prevent or adjust data originating from outside the inclusion zone from being transmitted downstream in the video conference, such as, for example, audio muting, video blurring, video cropping, etc. In some examples, filtering subject data may also include preventing people located outside the inclusion zone from being framed or tracked, such as using composition frames, person framing, active speaker framing, and tracking techniques. In addition, it is contemplated that multiple inclusion zones, boundary lines, and / or exclusion zones may be defined using the methods discussed herein.

[0037] Generally, in some aspects, calibration methods may be used to determine the dimensions of a video conference room and / or define an inclusion zone in a video conference room. For example, a calibration method may be used to determine the dimensions of a video conference room, and the entire video conference room may be considered an inclusion zone or a portion of the video conference room may be defined as an inclusion zone. As another example, a calibration method may be used to determine an inclusion zone without first determining the dimensions of the video conference room. These calibration methods may be automatic or manual, and may be initially completed at the time of setting up the video conference room and / or periodically completed while using the video conferencing system.

[0038] According to one example, the video conference room dimensions may be defined during an automatic calibration phase, in which the video conferencing system may use the positions of the conference participants to automatically determine the maximum world coordinates of the video conference room and further optionally automatically determine the maximum world coordinates of the containing area. Figure 6A picture image 300 of another example conference room 302 is illustrated. Three subjects or participants 304, 306, 308 are located at different coordinate locations in the room 302, and wherein corresponding head frames or bounding boxes 310, 312, 314 are identified according to the coordinate location for each of the participants 304, 306, 308. The bounding boxes 310, 312, 314 may be overlaid on the image 300 using a subject detector model, as discussed above. Further, the room width dimension x may be referenced. ROOM Japanese-style room depth ROOM To measure the coordinate position. Chamber width x ROOM The width of the cross chamber 302 is from the front camera 20 (see Figure 1 ) center line 26 (see Figure 1 ) is extended so that x ROOM The negative value is located at the center line 26 (see Figure 1 ) on the left and x ROOM The positive value is located at the center line 26 (see Figure 1 ) on the right. In addition, the chamber depth dimension y ROOM Extends downwardly along the length of chamber 302 parallel to centerline 26 of front camera 20 (see Figure 2 ).

[0039] By applying computer vision processing to image 300, a first conference participant 304 is detected in the back left corner of room 302, and a region of interest around the head of first conference participant 304 is framed using a first head bounding box 310, where first conference participant 304 is located within the two-dimensional room distance parameter (x ROOM =-3,y ROOM =21). In a similar manner, a second conference participant 306 sitting at table 316 is detected, wherein the head of the second conference participant 306 is framed by a second head bounding box 312, wherein the second conference participant 306 is located at the two-dimensional room distance parameter (x ROOM =-1,y ROOM =13). Finally, the third conference participant 308 standing on the right is detected, wherein the head of the third conference participant 308 is framed by a third head bounding box 314, wherein the third conference participant 308 is located at the two-dimensional room distance parameter (x ROOM =5,y ROOM =14).

[0040] During the automatic calibration phase, the maximum and minimum room parameters {x ROOM ,y ROOM} to automatically determine the video conference room 302. In particular, the automatic calibration phase can use the coordinates of the participants 304, 306, 308 to measure the maximum and minimum room width parameters x ROOM and the maximum chamber depth parameter y ROOM .exist Figure 6 In the non-limiting example illustrated in FIG, participants 304, 306, 308 are located at room dimensions {x ROOM ,y ROOM Thus, following the automatic calibration, the video conferencing system may determine that the video conference room 302 has a minimum and maximum room width dimension x of (-3, 5) respectively. ROOM Further, the video conference room 302 may have a room depth dimension y of (0, 21). ROOM In other words, video conference room 302 may have room dimensions of at least 8 units wide and 21 units deep. Based on these video conference room 302 dimensions, a containment zone may be defined as the entire room or a portion of the room.

[0041] Accordingly, it should be understood that the video conference room 302 dimensions can be defined in the conference room based on the participant positions measured during the calibration phase. In some aspects, the automatic calibration phase is activated by a host or participant of the video conference, such as using a controller or pressing a calibration phase button on a camera, or the automatic calibration phase can be automatically activated when a first participant enters the FOV of the camera, as will be discussed in more detail below. In addition, the automatic calibration phase can be activated for a predetermined amount of time, such as 30 seconds, 60 seconds, 120 seconds, 300 seconds, etc., or the automatic calibration phase can be continuously active. For example, the automatic calibration phase can track the participant positions in the conference room for a longer period of time (e.g., hours or days) to generate a predictable model of the participant positions in the conference room, which means that the inclusion zone can be automatically updated or changed over time.

[0042] In other examples, the video conference room dimensions may be defined during a manual calibration phase in which a human installer (e.g., a host or video conference participant) manually sets the shape and size of the video conference room, and optionally the shape and size of the containment zone. Figure 7 and 8Another example conference room 400 is illustrated with a single installer 402 walking around the perimeter 404 of the room 400 to define the dimensions of the room 400 and optionally the dimensions of the containment zone 406 in the room 400 while the front camera 20 is in the manual calibration phase. In particular, the installer 402 may activate the manual calibration mode and continue walking between different locations 408 in the room. The camera 20 may track the installer 402 as the installer 402 moves through the room 400 to define the boundaries or dimensions of the room 400 and / or the containment zone 406. It should be apparent that the installer 402 may move around the room 400 to define any particular shape, meaning that the containment zone 406 may also be defined in any one or more particular shapes (e.g., triangles, rectangles, quadrilaterals, circles, etc.). For example, the installer 402 may walk between a first location 408A, a second location 408B, a third location 408C, and a fourth location 408D (which may correspond to the corners of the room 400). Using the subject detector model as discussed above, camera 20 may determine and record world coordinates for each location 408 that installer 402 walks past, or camera 20 may continuously determine and record world coordinates of installer 402 during a manual calibration phase. In other words, installer 402 may map room 400 and / or containment zone 406 by walking around room 400 while camera 20 is in manual calibration mode.

[0043] Specific reference Figure 8 , which illustrates Figure 7 A top-down view of conference room 400 of FIG. 400. In some aspects, camera 20 may use an AI head detector model to determine the distance between camera 20 and installer 402 in terms of horizontal roll angle and from x ROOM Dimension or axis 414 and y ROOM The depth distance derived from the dimension or axis 416 is used to accurately define the size of the chamber 400 and / or the containment zone 406, where the camera 20 is located {x ROOM ,y ROOM} coordinate position (0, 0). For example, camera 20 may determine that the participant is at a first distance 418A in first position 408A, a second distance 418B in second position 408B, a third distance 418C in third position 408C, and a fourth distance 418D in fourth position 408D. Accordingly, camera 20 may define inclusion zone 406 based on distances 418 measured as installer 402 moves through room 400 during the manual calibration phase. In some aspects, installer 402 may choose not to walk around perimeter 404 of room 400, e.g., around a smaller portion of room 400 or in a shape that is different from the shape of room 400. Further, installer 402 may activate manual calibration mode before the video conference occurs, or installer 402 may activate manual calibration mode and define inclusion zone 406 at the beginning of the video conference (i.e., after all participants have entered room 400).

[0044] In other examples, an installer or user may manually input the coordinates of the chamber and containment zone during a manual calibration phase using, for example, a graphical user interface (GUI) on a computer monitor screen or tablet screen. Fig. 9 , illustrates a chamber configuration GUI 500 that includes: a top view of a chamber 502; and a "Setup Chamber" page 504 that can be selected by a user to at least define the dimensions of the chamber 502 and / or adjust the placement of camera pins (not shown). Although the GUI 500 is illustrated as including a rectangular chamber 502, the chamber 502 can be arranged in any suitable shape, such as an oval chamber, a circular chamber, a triangular chamber, etc. Further, a variety of different inputs can be used to allow the user to control certain aspects of the GUI 500, including any acceptable human interface device, such as a touch-enabled device, button input, keyboard, mouse, trackball, joystick, touchpad, etc.

[0045] Still refer to Fig. 9 , the GUI 500 may include a first field box 506, a second field box 508, a third field box 510, a "next" icon 512, and a "cancel" icon 514. However, it is contemplated that the "Set Room" page 504 of the GUI 500 may include more than one field box. Fig. 9 506, 508, 510. In some aspects, each of the field boxes 506, 508, 510 can be a text field box in which a user manually enters numbers or text (e.g., using a keyboard), or each of the field boxes 506, 508, 510 can be configured as a drop-down list (DDL). The field boxes 506, 508, 510 can be used to define the dimensions of the chamber 502 in specific units of measure (e.g., feet, meters, etc.), such as length, width, depth, radius, curvature, etc.

[0046] like Fig. 9 As shown in the non-limiting example of FIG. 1 , the first field box 506 may correspond to a value along y ROOM Dimension or axis 516 measures the depth of chamber 502, and second field box 508 may correspond to the depth of chamber 502 along x. ROOM The width of the chamber 502 measured relative to the y-axis 518. For example, the first and second field boxes 506, 508 may be DDLs of numbers (e.g., 1, 2, 3, etc.), and the third field box 508 may be DDLs of different measurement units (e.g., feet (ft) and meters (m)). Accordingly, the user may define the length and width dimensions of the chamber 502 by filling in the field boxes 506, 508, 510. For example, the user may fill in the first field box 506 with "18", fill in the second field box 508 with "12", and fill in the third field box 510 with "feet (ft)" to define the length and width dimensions of the chamber 502 relative to the y-axis 518. ROOM and x ROOM Axes 516, 518 are chambers that are 18 feet long and 12 feet wide, respectively.

[0047] In addition, a grid 520 can be overlaid on the top view of the chamber 502 in the GUI 500, wherein the grid 520 can change shape according to the dimensions of the chamber, and the grid 520 can be sized according to the units selected in the third field box 510. In some aspects, the user can draw the chamber 502 rather than manually entering the dimensions in the field boxes 506, 508, 510, which can be advantageous, for example, if the chamber 502 is an irregular shape. In addition, the user can place a "pin" (not shown) anywhere along the grid 520 that corresponds to the location of the camera within the chamber 502. After the dimensions have been set for the chamber 502 (i.e., using the field boxes 506, 508, 510), the user can select the "next" icon 512 to move to the "set perimeter" page 524 (see Fig.10 ), or the user may select “Cancel” icon 514 to reset the chamber dimensions and / or return to the home page of GUI 500 (not shown).

[0048] Reference now Fig.10 , illustrates a “Set Perimeter” page 524 of the GUI 500, which includes a top view of the room 502, an inclusion zone 526 overlaid on the room 502, a first slider 528, a second slider 530, a third slider 532, a “Save and Exit” icon 534, and a “Cancel” icon 536. Specifically, the area of ​​the room 502 enclosed by the perimeter or virtual boundary line 538 can define the inclusion zone 526. Correspondingly, the area of ​​the room 502 outside the boundary line 538 can be defined as the exclusion zone 540. In this way, the boundary line 538 can be used to determine what data or what type of data to transmit to the far end of the video conference, as will be discussed in more detail below.

[0049] To define the boundary line 538 on the chamber 502, the user can manually draw the boundary line 538 within the grid 520, or the user can use the sliders 528, 530, 532 to adjust the boundary line 538 relative to the size of the chamber 502. However, it is contemplated that the "Set Perimeter" page 534 may include more than Fig.10 Further, the "Set Perimeter" page 524 may include a field box with DDL instead of a slide bar, or the "Set Perimeter" page 524 may include both a field box and a slide bar.

[0050] In some aspects, the sliders 528, 530, 532 can be used to adjust the inclusion zone boundary lines corresponding to the sides of the chamber 502 (e.g., the left or first side 542, the rear or second side 544, and the right or third side 546). For example, the first slider 528 can be used to move the first boundary line 538A inward from the first side 542 of the chamber 502 or move the first boundary line 538A outward to the first side 524 of the chamber 502, the second slider 530 can be used to move the second boundary line 538B inward from the second side 544 of the chamber 502 or move the second boundary line 538B outward to the second side 544 of the chamber 502, and the third slider 528 can be used to move the third boundary line 538C inward from the third side 546 of the chamber 502 or move the third boundary line 538C outward to the third side 546 of the chamber 502. Accordingly, the size of the inclusion zone 526 can be incrementally adjusted as desired. Fig.10 In the non-limiting example illustrated in , boundary lines 538A, 538B, 538C are each two feet away from sides 542, 544, 546 of chamber 502, respectively, as indicated by slider bars 528, 530, 532.

[0051] Once boundary line 538 is adjusted as desired and inclusion zone 526 is defined in room 502, the user can select "Save and Exit" icon 534 to save the configuration of inclusion zone 526, which means that inclusion zone 526 is active in room 502. Alternatively, the user can select "Cancel" icon 536 to reset boundary line 538 size and / or return to the home page (not shown) of GUI 500. In some examples, the user may desire to adjust inclusion zone 526 after the video conference has begun due to, for example, people entering or exiting the conference room, changes in environmental conditions, or another reason. Accordingly, the user can re-enter manual calibration mode at any point during the video conference and readjust the inclusion zone using, for example, slider bars 528, 530, 532. Further, it is contemplated that the above-described calibration methods can be used together during a video conference. Figure 6-8Manual calibration mode and automatic calibration mode discussed. For example, the user can initially use the manual calibration mode to define the inclusion zone 526 before switching to the automatic calibration mode after the video conference has begun. Alternatively, the user can use the automatic calibration mode to define the inclusion zone 526 before switching to the manual calibration mode to adjust the boundaries of the inclusion zone 526.

[0052] Continue to refer Fig.10 , by camera 20 (see Figure 1 ) and microphones (eg, microphone array 22 (see Figure 1 )) All data captured originating from outside the inclusion zone 526 (i.e., within the exclusion zone 540) may be filtered (e.g., muted or blurred), while the data captured by the camera 20 and microphone (see Figure 1 ) All data captured from within the inclusion zone 526 can be provided to the far end of the video conference. In this way, data from outside the inclusion zone 526 can be distinguished from data from within the inclusion zone 526.

[0053] In addition, GUI 500 may be used to track people in room 502 in real time to determine if they are within inclusion zone 526. Specifically, the AI ​​head detector model may be used to determine room or world coordinates of people in room 502, and the world coordinates may then be compared to the world coordinates of inclusion zone 526 to determine if the people are within inclusion zone 526. For example, first person 548A and second person 548B may be located in room 502, and the AI ​​head detector model may be applied to the room detected by camera 20 (see FIG. Figure 1 ) to determine the coordinates for each person 548. As illustrated, the first person 548A is located within the boundary line 538 (i.e., within the containment zone 526), ​​and therefore, the camera 20 and / or microphone (see Figure 1 ) Any data originating from first person 548A recorded by the video conference system can be processed by the video conference system and transmitted to the far end of the video conference. Relatedly, second person 548B is at least partially outside inclusion zone 526, i.e., partially within exclusion zone 540. Thus, if person 548 is at least partially on boundary line 538 and / or at least partially within exclusion zone 540, person 548 can be considered to be outside inclusion zone 526. Alternatively, person 548 can be considered to be outside inclusion zone 526 only if person 548 is outside boundary line 538 and completely within exclusion zone 540.

[0054] Still refer to Fig.10 As a result of determining that the second person 548B is outside the inclusion zone 526, data originating from the second person 548B may still be received by the camera 20 and / or microphone (see Figure 1) recorded, but the data may also be filtered before being transmitted to the far end of the video conference. For example, the data originating from the second person 548B may be blurred, muted, reduced in volume, and / or otherwise filtered using another suitable audio or visual filtering technique. In some examples, the data originating from the second person 548B may not be transmitted to the far end of the video conference.

[0055] Thus, more generally, data originating from persons within inclusion zone 526 is processed differently than data originating from persons outside of inclusion zone 526. Additionally, in some applications, different filtering techniques may be used with respect to different inclusion zones 526. That is, if multiple inclusion zones 526 are defined within a video conference room, a user may be able to specify a certain type of filtering or action to be taken when a participant is detected in a particular inclusion zone 526. As an example, a "greeting zone" type of inclusion zone 526 may be defined, where upon detecting that a participant has entered the greeting zone, the video conferencing system may start video or ask the participant if they would like video to begin on monitor 24 (see FIG. 24 ). Figure 1 In another example, a containment zone 526 of the "privacy zone" type may be defined, where the video conferencing system transmits data to the far end of the video conference so that the video is focused only within the privacy zone.

[0056] It should be apparent that the method of using containment boundaries to filter subject data based on location within a conference room can be used in a variety of different conference rooms and with any number of people. Fig.11 , illustrates a picture image 600 of yet another example conference room 602, having Fig.12 A top-down schematic diagram of a conference room 602 is shown in FIG. The conference room 602 includes: a camera 604 (see Fig.12), with microphone arrays located at the front, left wall 606, right wall 608, and back wall 610 of room 602. In some aspects, left wall 606 may be a transparent wall (e.g., a glass wall) such that a foyer 612 adjacent to conference room 602 is visible. In addition, a first person 614A, a second person 614B, a third person 614C, and a fourth person 614D are seated around a conference table 616 in conference room 602, while a fifth person 614E and a sixth person 614F are located outside conference room 602, i.e., in foyer 612 adjacent to transparent left wall 606 of conference room 602. Each of the persons 614 is located at a different coordinate position in picture image 600, and is recognized as a person by applying the AI ​​head detector model to picture image 600. As discussed above, the AI ​​head detector model can generate a head frame or bounding box 618 around each person 614, such as a first bounding box 618A, a second bounding box 618B, a third bounding box 618C, etc., and the bounding box 618 can be used to determine the world coordinate position for the person 614. Further, the room width dimension x can be referenced. ROOM Japanese-style room depth ROOM To measure the world coordinate position. Room width dimension x ROOM The width of the chamber 602 may be viewed from the centerline 620 of the camera 604 (see Fig.12 ) is extended so that x ROOM The negative value is located to the left of the center line 620 (see Fig.12 ) and x ROOM The positive value is located to the right of the center line 620 (see Fig.12 ). In addition, the chamber depth dimension y ROOM Extends downwardly along the length of chamber 602 parallel to the centerline 620 of front camera 20 (see Fig.12 ).

[0057] In addition, the people 614 in the conference room 602 (i.e., the first person 614A, the second person 614B, the third person 614C, and the fourth person 614D) may be participants in the video conference, and the people 614 outside the conference room 602 (i.e., the fifth person 614E and the sixth person 614F) may not be participants in the video conference. However, the fifth and sixth persons 614E, 614F are captured by the camera 604 (see Fig.12 ) capture, which means that subject data (i.e., audio and / or visual data associated with a subject or person) can be recorded and transmitted to the far end of the video conference. Accordingly, the sounds and / or movements created by the fifth and sixth persons 614E, 614F may cause confusion and / or distraction to the far end participants in the video conference.

[0058] To prevent distracting noise or movement from being transmitted to the far end of the video conference, the calibration techniques discussed above may be used to define a containment zone 622 in image 600. Specifically, referring to Fig.12 , we can use the room distance parameter {x ROOM ,y ROOM One or more boundary lines 624 are overlaid on the image to separate the inclusion zone 622 from the exclusion zone 626. In this way, the subject data originating from the persons 614A, 614B, 614C, 614D within the inclusion zone 622 can be processed and transmitted to the far end of the video conference, while the subject data originating from the persons 614E, 614F within the exclusion zone 626 can be filtered, for example, and not transmitted to the far end of the video conference. Fig.11 and 12 In a non-limiting example, a boundary line 624 may be drawn along the left wall 606, such that an inclusion zone 622 may be defined between the left wall 606 and the right wall 610. Correspondingly, an exclusion zone 626 may be defined by the left wall 606, meaning that any object or person visible through the left wall 606 is within the exclusion zone 626.

[0059] Reference now Fig.12 , a top view of the image 600 is represented using a world coordinate system 628. The world coordinate system 628 includes an x ​​coordinate system corresponding to the width of the conference room 602. ROOM Axis 630 and y corresponding to the depth of conference room 602 ROOM Axis 632. Correspondingly, the camera 604 is located at {x ROOM ,y ROOM} coordinate position (0, 0). As discussed above, boundary line 624 is defined along left wall 606 so that inclusion zone 622 can be defined to the right of boundary line 624 and exclusion zone 626 can be defined to the left of boundary line 624. After zones 622, 626, and boundary line 624 have been defined and the room coordinates of persons 614 have been determined, the room coordinates associated with each person 614 can be compared to boundary line 624, as discussed above.

[0060] Still refer to Fig.12 , the first person 614A may be located in a two-dimensional room at a distance parameter (x ROOM =2,y ROOM =6), the second person 614B may be located at the two-dimensional room distance parameter (x ROOM =2,y ROOM =8), the third person 614C may be located at the two-dimensional room distance parameter (x ROOM =2,y ROOM =11), and the fourth person 614D may be located at the two-dimensional room distance parameter (x ROOM =-1,y ROOM=11). In addition, the fifth person 614D may be located at the two-dimensional room distance parameter (x ROOM =-5,y ROOM =6), and the sixth person 614F may be located at the two-dimensional room distance parameter (x ROOM =-5,y ROOM =9). In addition, the left wall 606 can be at a width distance {x ROOM =-3.5} at the same ROOM A boundary line 624 may be defined along left wall 606 such that inclusion zone 622 may extend between width distances {-3.5, 2.25} and exclusion zone 626 may extend between width distances {-3.5, 2.25} and ... ROOM The width measured from the axis 630 extends between the distances {-5.75, -3.5}. Accordingly, the first, second, third, and fourth persons 614A, 614B, 614C, 614D may be located in the inclusion zone 622, while the fifth and sixth persons 614E, 614F may be located in the exclusion zone 626.

[0061] As discussed above, if person 614 is determined to be located within inclusion zone 622, data associated with person 614 may be processed normally, and video conferencing framing techniques may be used to appropriately frame or track person 614. For example, person 614 may be processed normally and / or transmitted to the far end of the video conference. Conversely, if person 614 is determined to be at least partially located outside of inclusion zone 622, data associated with the person may be filtered or blocked from being transmitted to the far end of the video conference, and person 614 may not be processed by video conferencing framing or tracking techniques.

[0062] Thus, the inclusion zone videoconferencing system disclosed herein is able to distinguish between data originating from within a containment zone and data originating from outside of the containment zone, where a zone is defined in terms of width and depth dimensions relative to a top-down view of a videoconference room or area. Correspondingly, the containment zone videoconferencing system can prevent distracting movement and / or sound from being provided to the far end of a videoconference, which in turn can reduce confusion in the videoconference. In some aspects, the containment zone videoconferencing system disclosed herein is particularly advantageous in open concept workspaces and / or conference rooms with transparent walls. Further, it is contemplated that Figure 6-12 A non-limiting example of an inclusion zone video conferencing system is illustrated, and the inclusion zone video conferencing system may be applied to a variety of different conference rooms and be compatible with a variety of different camera arrangements.

[0063] In view of the above, Fig.13A method 700 for implementing an inclusion zone video conferencing system is illustrated. At step 702, (one or more) cameras are used to capture (one or more) images of a location. As discussed above, the camera can be arranged at the front of the conference room, and the camera can communicate and / or be connected to a monitor and / or codec including a memory and a processor, as discussed in more detail below. At step 704, an AI head detection model is used to detect human heads in the image (i.e., heads of people in the conference room), as described above. For example, the AI ​​head detection model is applied to the image captured by the camera so that a head bounding box with specified room and / or pixel coordinates is generated for each detected human head. At step 706, (one or more) inclusion zones are defined for the location based on a top-down view of the location, for example using world coordinates. The inclusion zone can be defined during a calibration phase (such as the automatic or manual calibration phase discussed above). Alternatively, if the inclusion zone is previously defined, step 706 can include retrieving the previously set inclusion zone boundaries from the memory.

[0064] At step 708, the system determines whether the room coordinates and size information for each detected human head are within the boundaries of the inclusion zone. In other words, the room coordinates of each detected human head are checked against the world coordinates of the inclusion zone to determine whether any of the human heads are at least partially located outside the inclusion zone. At step 710, if the subject data (i.e., data associated with or generated by a specific person in the location) is determined to not originate from within the inclusion zone, the system filters the subject data. This can be done using a variety of different filtering techniques, such as but not limited to audio muting and video blurring, as discussed above. In addition, in some applications, step 710 may further include filtering any data originating from outside the inclusion zone, such as, for example, blurring all video outside the inclusion zone or muting any audio outside the inclusion zone even when the subject is not detected outside the inclusion zone. At step 712, if the subject data is determined to have originated from within the inclusion zone, the system processes the subject data. Processing the subject data may include, for example, transmitting the subject data to the far end of the video conference. Alternatively, subject data determined to have originated within the inclusion zone may also be filtered before it is transmitted to the far end of the video conference, albeit in a different manner than subject data outside the inclusion zone. Operation returns to step 702, such that the distinction between subject data originating outside the inclusion zone and subject data originating within the inclusion zone is automatic when the camera captures an image of the location.

[0065] Generally, method 700 may be performed in real time or near real time. For example, in some aspects, steps 702, 704, 706, 708, 710, 712 of method 700 are repeated continuously or after a period of time has passed (such as, for example, at least every 30 seconds, or at least every 15 seconds, or at least every 10 seconds, or at least every 5 seconds, or at least every 3 seconds, or at least every second, or at least every 0.5 seconds). Accordingly, method 700 allows for real-time or near real-time tracking of participants in a bird's-eye view to determine whether the participant is in or outside the inclusion zone. It is contemplated that all of method 700 (including any of the other methods described above) may be performed in the camera, and / or method 700 may be executable via machine-readable instructions stored on a codec and / or executed on a processing unit. Therefore, it should be understood that the methods described herein may be computationally lightweight and may be performed entirely in the main camera, thereby reducing the need for a resource-heavy GPU and / or other dedicated computing machinery.

[0066] Fig.14 An example camera 820 that may be similar to the front camera 20 and the microphone array 22 (see Figure 1 ) is similar to an example microphone array 822. The camera 820 has a housing 824 with a lens 826 provided at the center to operate with an imager 828. A series of microphone openings 830 (such as five openings 830) are provided as ports facing the microphones in the microphone array 822. In some examples, the openings 830 form a horizontal line 832 to provide desired angle determination for the SSL process, as discussed above. Fig.14is an example illustration of a camera 820, although many other configurations are possible, with different camera lenses and microphone configurations. Additionally, in some examples, aspects of the technology, including computerized implementations of methods according to the technology, may be implemented as a system, method, apparatus, or article of manufacture using standard programming or engineering techniques to produce software, firmware, hardware, machine-readable instructions, or any combination thereof, to control a processor device (e.g., a serial or parallel general or special purpose processor chip, a single or multi-core chip, a microprocessor, a field programmable gate array; any kind of combination of control units, arithmetic logic units, and processor registers, etc.), a computer (e.g., a processor device operatively coupled to a memory), or another electronically operated controller to implement the aspects detailed herein. Accordingly, for example, the technology may be implemented as an instruction set tangibly embodied on a non-transitory computer-readable medium, such that a processor device may implement the instructions based on reading the instructions from the computer-readable medium. Some examples of the technology may include (or utilize) a control device, such as, for example, an automation device, a special or general purpose computer including various computer hardware, software, firmware, etc., consistent with the discussion below. As specific examples, the control device may include a processor, a microcontroller, a field programmable gate array, a programmable logic controller, logic gates, etc., as well as other suitable components for implementing appropriate functions (e.g., memory, communication systems, power sources, user interfaces and other inputs, etc.).

[0067] The above description assumes that the front camera 20 and the microphone array 22 (see Figure 1 ) are matched. If the axes are displaced, the displacement is used when converting the determined sound angles from the microphone array 822 to the camera reference frame.

[0068] As described above, the methods of some aspects of the present disclosure include: using an AI human head detector model to detect the positions of individual conference participants. Fig.15, illustrates an example process 900 for determining the coordinates of a detected human head using such an AI human head detector process. The AI ​​human head detector process utilizes a machine learning AI human head detector model 904 to analyze incoming room-view video frame images 902 of a conference room scene to detect and display human heads with corresponding head bounding boxes 906, 908, 910. As depicted, each incoming room-view video frame image 902 may be captured by a front camera 20 in a video conferencing system. Each incoming room-view video frame image 902 may be processed using an on-device AI human head detector model 904, which may be located at the corresponding camera that captured the video frame image. However, in other examples, the AI ​​human head detector model 904 may be located at a remote or centralized location or at only a single camera. Regardless of where it is positioned, the AI ​​human head detector model 904 can include: multiple processing modules 912, 914, 916, 918 that implement a machine learning model that is trained to detect or classify human heads from incoming video frame images and identify a head bounding box with specified image plane coordinates and size information for each detected human head.

[0069] In this example, the AI ​​human head detector model 904 may include: a first pre-processing module 912, which applies image pre-processing (such as color conversion, image scaling, image enhancement, image resizing, etc.) so that the input video frame image is ready for subsequent AI processing. In addition, the second module 914 may include: training data parameters and / or model architecture definitions, which may be pre-defined and used to train and define the human head detection module 904 to accurately detect or classify human heads from incoming video frame images. In the selected example, the human head detection model module 916 may be implemented as model inference software or a machine learning model, such as a convolutional neural network (CNN) model, which is specially trained for video codec operation to detect heads in the input image by generating a pixel-by-pixel position for each detected head and by generating a corresponding head bounding box for each detected head to frame the detected head. Finally, the AI ​​human head detector model 904 may include: a post-processing module 918, which applies image post-processing to the output from the AI ​​human head detector model module 916 to make the processed image suitable for human viewing and understanding. In addition, the post-processing module 918 can also reduce the size of the data output generated by the human head detection model module 916, such as by consolidating or grouping multiple head bounding boxes or frames generated from a single meeting participant so that a single head bounding box or frame is designated.

[0070] Based on the results of the processing modules 912, 916, 918, the AI ​​human head detector model 904 can generate an output video frame image 902 in which the detected human heads are framed with corresponding head bounding boxes 906, 908, 910. As depicted, the first output video frame image 902a includes: head bounding boxes 906a, 906b and 906c, which are superimposed around each detected human head. In addition, the second output video frame image 902b includes head bounding boxes 908a, 908b and 908c superimposed around each detected human head, and the third output video frame image 902c includes head bounding boxes 910a, 910b superimposed around each detected human head. The AI ​​human head detector model 904 can specify each head bounding box using any suitable pixel-based parameters, such as defining the x and y pixel coordinates of the head bounding box or frame in combination with the height and width dimensions of the head bounding box or frame. In addition, the AI ​​human head detector model 904 can use any suitable measurement technique to specify a distance metric between the camera position and the position of the detected human head. The AI ​​human head detector model 904 can also calculate a corresponding confidence metric or score for each head bounding box, which quantifies the model's confidence in detecting a human head.

[0071] In some examples of the present disclosure, the AI ​​human head detector model 904 may specify all head detections in a data structure that holds the coordinates of each detected human head along with their detection confidence. More specifically, a human head data structure for a number n of human heads may be generated as follows:

[0072]

[0073] In this example, x i and i denotes the image plane coordinates of the i-th detected head, and where the width i and height i Refers to the width and height information of the head bounding box for the i-th detected head. In addition, the score iis in the range [0, 100] and reflects the confidence as a percentage for the ith detected head. The data structure can be used as input to various applications such as framing, tracking, writing, recording, switching, reporting, encoding, etc. In this example data structure, the first detected head is in the image frame in a head bounding box located at pixel position parameter x1, y1 and extending horizontally to a width of 1 and vertically downward to a height of 1. In addition, the second detected head is in the image frame in a head bounding box located at pixel position parameter x2, y2 and extending horizontally to a width of 2 and vertically downward to a height of 2, and the nth detected head is in the image frame at pixel position parameter x2, y2 and extending horizontally to a width of 2 and vertically downward to a height of 2. n ,y n and the horizontal extension width n And vertically extending downward n In some aspects, the center of each head bounding box is determined using the following equation:

[0074] This human head data structure can then be used as input to the distance estimation process, which takes the {width, height} parameters of each head bounding box to calculate the distance between the room coordinates {x ROOM ,y ROOM} to pick the best matching distance: first use one of the width or height parameters on the first lookup table, and then if multiple meeting room coordinates {x ROOM ,y ROOM}, then use the other parameters as tie-breakers. The human head data structure itself can then be modified to also embed distance information about each head, resulting in a modified human head data structure that looks like the following:

[0075]

[0076] where {x ROOM1 ,y ROOM1}、{x ROOM2 ,y ROOM2}、……、{x ROOMn ,y ROOMn}Specify head1, head2, ..., head in two-dimensional coordinates n The distance from the camera respectively.

[0077] Fig.16Aspects of a codec 1000 according to some examples of the present disclosure are illustrated. As discussed above, the codec 1000 can be a separate device of a video conferencing system, or can be incorporated into (one or more) cameras (such as a primary camera) within a video conferencing system. Generally, the codec 1000 includes machine-readable instructions for maintaining a video call with a video conferencing endpoint, receiving streams from a secondary camera (and, if not integrated with the primary camera, the primary camera), and encoding and compositing the streams for transmission to the endpoint according to the methods described herein.

[0078] like Fig.16 As shown in , the codec 1000 may include speaker(s) 1002, although in many cases the speaker 1002 is provided in the monitor 1004. The codec 1000 may include microphone(s) 1006 interfaced via a bus 1008. The microphone 1006 is connected via an analog-to-digital (A / D) converter 1010, and the speaker 1002 is connected via a digital-to-analog (D / A) converter 1012. The codec 1000 also includes a processing unit 1014, a network interface 1016, a flash or other non-volatile memory 1018, a RAM 1020, and an input / output (I / O) general interface 1022, all coupled via the bus 1008. A camera 1024 is connected to the I / O general interface 1022. The microphone(s) 1006 are connected to the network interface 1016. An HDMI interface 1026 is connected to the bus 1008 and to an external display or monitor 1004. The bus 1008 is illustrative, and any interconnect between the elements may be used, such as a Peripheral Component Interconnect Express (PCie) link and switch, a Universal Serial Bus (USB) link and hub, and combinations thereof. The camera 1024 and microphones 1006, 1006 may be contained within a housing containing the other components, or may be external and removable, connected via a wired or wireless connection.

[0079] Processing unit 1014 may include a digital signal processor (DSP), a central processing unit (CPU), a graphics processing unit (GPU), dedicated hardware elements such as a neural network accelerator, and a hardware codec.

[0080] The flash memory 1018 stores modules of different functions in the form of software and firmware, generally programs or machine-readable instructions, for controlling the codec 1000. The illustrated modules include video codec 1028, camera control 1030, framing 1032, other video processing 1034, audio codec 1036, audio processing 1038, network operation 1040, user interface 1042 and operating system, and various other modules 1044. In some examples, an AI head detector module is included with the modules included in the flash memory 1018. In addition, in some examples, machine-readable instructions may be stored in the flash memory 1018, which cause the processing unit 1014 to implement any of the methods described above. The RAM 1020 is used to store any of the modules in the flash memory 1018 when the module executes, stores video images of the video stream and audio samples of the audio stream, and can be used for scratch pad operations of the processing unit 1014.

[0081] The network interface 1016 enables communication between the codec 1000 and other devices and can be wired, wireless, or a combination. In one example, the network interface 1016 is connected or coupled to the Internet 1046 to communicate with a remote endpoint 1048 in a video conference. In one example, the general interface 1022 provides data transmission with a local device (not shown), such as a keyboard, mouse, printer, projector, display, external speakers, additional camera and microphone pod, etc.

[0082] In one example, the camera 1024 and the microphone 1006 capture video and audio, respectively, in a video conferencing environment and generate video and audio streams or signals that are transmitted to the processing unit 1014 via the bus 1008. As discussed herein, capturing a "view" or "image" of a location can include capturing individual frames and / or frames within a video stream. For example, the camera 1024 can be instructed to continuously capture a particular view of a location (e.g., an image within a video stream) for the duration of the video conference. In one example of the present disclosure, the processing unit 1014 processes the video and audio using processes in a module stored in the flash memory 1018. The processed audio and video streams can be sent to a remote device coupled to the network interface 1016 and a device coupled to the universal interface 1022, and the processed audio and video streams can be received from a remote device coupled to the network interface 1016 and a device coupled to the universal interface 1022.

[0083] The microphones in the microphone array used for SSL may be used as microphones to provide voice to a remote location, or a separate microphone, such as microphone 1006, may be used.

[0084] Certain operations of methods according to the technology or systems performing those methods may be schematically represented in the figures or otherwise discussed herein. Unless otherwise specified or limited, the representation of specific operations in a particular spatial order in the figures does not necessarily require that those operations be performed in a particular order corresponding to the particular spatial order. Correspondingly, certain operations represented in the figures or otherwise disclosed herein may be performed in an order different from the order explicitly illustrated or described, as appropriate for a particular example of the technology. Further, in some examples, certain operations may be performed in parallel, including by dedicated parallel processing devices or separate computing devices that interoperate as part of a larger system.

[0085] The disclosed technology is not limited in its application to the construction details and component arrangements set forth in the following description or illustrated in the following drawings. Other examples of the disclosed technology are possible, and the examples described and / or illustrated herein can be practiced or implemented in various ways.

[0086] Multiple hardware and software-based devices and multiple different structural components can be used to implement the disclosed technology. In addition, the examples of the disclosed technology may include hardware, software and electronic components or modules that can be illustrated and described as most of the components are implemented only in hardware for the purpose of discussion. However, in one example, the electronic-based aspects of the disclosed technology can be implemented with software that can be executed by a processor (for example, stored on a non-transient computer-readable medium). Although some drawings illustrate the hardware and software located in a specific device, these depictions are used for illustrative purposes. In some examples, the illustrated components can be combined or divided into separate software, firmware, hardware or a combination thereof. As an example, instead of being located in a single electronic processor and being executed by a single electronic processor, logic and processing can be distributed among multiple electronic processors. Regardless of how they are combined or divided, hardware and software components can be located on the same computing device, or can be distributed among different computing devices connected by a network or other suitable communication links.

[0087] Any suitable non-transient computer-usable or computer-readable medium may be utilized. A computer-usable or computer-readable medium may be, for example, but not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or apparatus. More specific examples (non-exhaustive list) of computer-readable media would include the following: a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), a portable compact disk read-only memory (CD-ROM), an optical storage device, or a magnetic storage device. In the context of the present disclosure, a computer-usable or computer-readable medium may be any medium that may contain, store, transmit, or transfer a program for use with or in conjunction with an instruction execution system, device, or device.

[0088] As used herein in the context of computer implementation, unless otherwise specified or limited, the terms "component", "system", "module", "block", etc. are intended to cover part or all of a computer-related system including hardware, software, a combination of hardware and software, or software in execution. For example, a component can be, but is not limited to, a processor device, a process executed (or executable) by a processor device, an object, an executable file, a thread of execution, a computer program, or a computer. As an illustration, both an application running on a computer and the computer can be components. A component (or system, module, etc.) can reside within a process or thread of execution, can be localized on one computer, can be distributed between two or more computers or other processor devices, or can be included in another component (or system, module, etc.).

Claims

1. A method of using a containment zone for a video conference, the method comprising: Capture an image of the location; applying a subject detector model to the image to identify room coordinates for each subject detected in the image; defining an inclusion zone for the location, the inclusion zone based on a top-down view of the location; determining whether the room coordinates for each subject are within the inclusion zone; filtering data associated with subjects determined not to be within the contained zone; and Data associated with a subject determined to be within the containment zone is processed.

2. The method of claim 1 , wherein capturing an image of the location comprises: Capture images of sections of an enclosed room or sections of an open concept workspace.

3. The method of claim 1 , wherein applying the subject detector model comprises: A bounding box is defined for each human head of each subject detected in the image.

4. The method of claim 1, wherein defining the inclusion zone further comprises: The chamber coordinates of the inclusion zone were manually entered using a graphical user interface during the manual calibration phase.

5. The method of claim 1, wherein defining the inclusion zone further comprises: The subject's world coordinates are recorded during the calibration phase to create the boundaries of the containment zone.

6. The method of claim 5, wherein defining the inclusion zone further comprises: The maximum and minimum chamber parameters for the locations are determined in an automatic calibration phase.

7. The method of claim 6, wherein the maximum and minimum chamber parameters include a maximum chamber width parameter, a minimum chamber width parameter, and a maximum chamber depth parameter.

8. The method of claim 1, wherein filtering data associated with subjects determined not to be within the containment zone comprises at least one of: muting audio included in the data; and Video included in the data is blurred.

9. The method of claim 1, wherein processing the data comprises: The data is transmitted to a far end of the video conference.

10. A video conferencing system using a containment zone, the system comprising: A camera to capture images of the location; Microphone, used to receive sound; a processor, connected to the camera and the microphone, the processor executing a program for performing video conferencing operations including transmitting data to a remote video conferencing location; as well as A memory is coupled to the processor, the memory storing instructions that, when executed by the processor, cause the processor to perform the following operations: identifying room coordinates for each subject detected in the image; defining an inclusion zone for the location, the inclusion zone based on a top-down view of the location; determining whether the room coordinates for each subject are within the inclusion zone; filtering data associated with subjects determined not to be within the contained zone; and Data associated with a subject determined to be within the containment zone is processed.

11. The system of claim 10, wherein the processor for identifying room coordinates for each subject detected in the image comprises a processor for defining a bounding box for each human head for each subject detected in the image.

12. The system of claim 10, wherein the processor for defining the inclusion zone further comprises a processor for manually using input room coordinates of the inclusion zone via a graphical user interface during a manual calibration phase.

13. The system of claim 10, wherein the processor for defining an inclusion zone for the location comprises a processor for using the camera to record world coordinates of a subject during a calibration phase to create a boundary line of the inclusion zone.

14. The system of claim 13, wherein the processor for defining the inclusion zone further comprises a processor for determining maximum and minimum chamber parameters for the location in an auto-calibration phase.

15. The system of claim 10, wherein the processor for filtering data associated with subjects determined not to be within the containment zone comprises a processor for performing at least one of the following operations: muting audio included in the data; and Video included in the data is blurred.

16. The system of claim 10, wherein the processor for processing the data comprises a processor for transmitting the data to the remote video conferencing location.

17. The system of claim 10, wherein the processor is further caused to define a virtual boundary line separating the inclusion zone from an exclusion zone.

18. The system of claim 17, wherein data in the inclusion zone is processed differently than data in the exclusion zone.

19. A non-transitory computer readable medium containing instructions that, when executed, cause a processor to perform the following operations: instruct a camera to capture an image of a location; applying a machine learning human head detector model to the image to detect human heads in the image and identify coordinates for each human head detected; defining an inclusion region for the image based on a top-down view of the location; and A determination is made as to whether each detected human head is within the inclusion zone.

20. The non-transitory computer readable medium of claim 19, wherein the processor is further configured to: filtering data associated with subjects determined not to be within the contained zone; and Data associated with a subject determined to be within the containment zone is processed.

Images