Participant re-identification in multi-camera video conference including central camera

Abstract

A system, method, and media for identifying participants in a venue using a front camera and a central camera are provided. The method includes capturing images of the venue using the cameras and applying a machine learning subject detector model to the images to identify coordinates and extract embeddings for each participant detected in the images. The method also includes determining a centroid of the coordinates for each participant detected in the images, applying an identification label to each participant in each image, and identifying a first participant in a first image captured by the front camera as a primary anchor point. The method further includes measuring distances between the embeddings in the images to match the primary anchor point to a secondary anchor point detected in a second image captured by the central camera, and aligning the identification labels associated with each participant across each of the images based on the match.

Description

Background Technology

[0001] Video conferencing systems typically connect people at one video conferencing endpoint (such as a video conference room) to people at other video conferencing endpoints. In some video conferencing modes, all participants detected in the video conference room are separated and placed in a gallery view to create equality with remote participants. Attached Figure Description

[0002] Figure 1 This is a top view of an exemplary conference room according to some aspects of this disclosure.

[0003] Figure 2 This is a schematic isometric view of another exemplary conference room with three participants, who are located at different coordinate positions relative to the video conferencing camera.

[0004] Figure 3 yes Figure 2 A top view of an exemplary conference room.

[0005] Figure 4 This is a schematic isometric view of yet another exemplary meeting room with two participants located at different coordinate positions, according to some examples of this disclosure.

[0006] Figure 5 This is a schematic illustration of a camera and a two-dimensional image plane according to an example of this disclosure, with an exemplary determination of the room coordinates of the head bounding box.

[0007] Figure 6 This is a front view of another exemplary conference room with seven individuals, which are located at different coordinate positions relative to the front camera and the center camera, according to some examples of this disclosure.

[0008] Figure 7 It was captured by the central camera. Figure 6 A front view of an exemplary conference room.

[0009] Figure 8 It was captured by the central camera. Figure 6 A left-side view of an exemplary meeting room.

[0010] Figure 9 It was captured by the central camera. Figure 6 A rear view of an exemplary conference room.

[0011] Figure 10 It was captured by the central camera. Figure 6 A right-side view of an exemplary meeting room.

[0012] Figure 11 This is an example of an image stripe, which includes... Figure 6 A single view of an individual in an exemplary meeting room, the single view being taken from... Figure 6-10 The view captured by the central camera.

[0013] Figure 12A It was captured by the front camera. Figure 6 A pixel map of an image of a single individual in an exemplary meeting room.

[0014] Figure 12B It is performed after background segmentation. Figure 12A The pixel image of the individual.

[0015] Figure 13 Is Figure 11 Embeddings associated with each individual in the image stripes and with Figure 12B A plot of the Euclidean distance between the embeddings associated with individuals (identified as participant two).

[0016] Figure 14 yes Figure 6 A front view of an exemplary conference room, where images of individuals captured by a front camera are aligned with images of individuals captured by a central camera, as shown in... Figure 11 As shown in the image stripes.

[0017] Figure 15 This is a flowchart illustrating a method for implementing an inside-out identification system in a conference room, according to an example of this disclosure.

[0018] Figure 16 This is a front view of a camera according to an example of this disclosure.

[0019] Figure 17 This is a flowchart of a method for determining image plane coordinates for a detected subject, according to an example of this disclosure.

[0020] Figure 18 This is a schematic diagram of an exemplary codec based on examples of this disclosure. Detailed Implementation

[0021] In video conferencing rooms equipped with multiple cameras, the same participant may appear in the field of view of more than one camera. Therefore, a problem in video conferencing using multiple cameras is participant duplication, which involves transmitting more than one image of a particular participant to the far end of the video conference. Accordingly, framing of individuals in a video conferencing room can be improved by determining the location of individual participants in the room relative to each other or a specific reference point. For example, the ability to detect location information, such as if person A is sitting 2.5 meters from the camera and person B is sitting 4 meters from the camera, enables a variety of advanced framing and tracking experiences. For instance, participant location information can be used to compare multiple camera views and re-identify the same participant in each camera view to determine the optimal camera view for that participant.

[0022] More specifically, when multiple cameras in a video conferencing system are used in a conference room with two or more participants, multiple views of the participants are captured, such as multiple views from different angles. This is especially true when the video conferencing system includes a front camera and a central camera (such as a 360-degree camera) for recording visual data for the video conferencing system. For example, the main camera may capture a first or frontal view of the participants, and the central camera may capture views of the participants from an inside-out angle. As a result, multiple images of the same one and / or multiple participants can be transmitted to the far end of the video conference, which can in turn confuse other participants in the video conference. Accordingly, it is undesirable to transmit a particular view to the far end of the video conference if a participant's face is not sufficiently visible in that particular view. Furthermore, adequate industry standards or specifications have not yet been developed for tracking individuals across multiple camera views in a video conferencing system based on their relative distance to each other or their relative position in the conference room.

[0023] For many applications, knowing the horizontal and vertical positions of participants in the room is useful to provide a more comprehensive and complete understanding of the video conferencing environment. While it is possible to determine two-dimensional room parameters (e.g., width and height) for each participant and identify each participant using these parameters by employing external feature properties or monocular depth estimation models based on computationally intensive machine learning, such methods impose significant hardware and / or processing costs without providing accuracy in identifying participants across multiple camera views. Furthermore, due to their reliance on specialized hardware, such methods are limited in their adaptability to a wide variety of room geometries and locations.

[0024] For example, various techniques attempt to perform so-called person re-identification (ReID) to match a participant captured by one camera with the same participant captured by another camera. In one variant, a pre-trained deep learning ReID model is used to identify the embeddings of a participant in a particular camera view and match those embeddings with those found in other camera views. However, this technique relies on resource-intensive computation that requires considerable memory to execute, and such computation still proves ineffective for distinguishing participants with similar clothing (such as participants wearing uniforms). Furthermore, deep learning ReID models are primarily trained on upright pedestrians, so their performance is further compromised when applied to seated participants.

[0025] In another variation, ReID-based methods involve mapping a three-dimensional (3D) world coordinate system to a two-dimensional (2D) camera plane by using external object or eigenvector matching to determine the transformation matrix. The reliance on external reference points within the room limits the applicability of such models to the physical room in which they are located. As a result, ReID techniques that depend on external reference points to deduce the transformation matrix lack adaptability and cannot be used in a wide variety of video conferencing room settings.

[0026] Therefore, in some examples, this disclosure provides apparatus, methods, and media for re-identifying participants across multiple camera views in a video conference. Specifically, this disclosure provides methods for: identifying each participant in a video conference room from multiple camera views (including a central camera view), thereby preventing duplicate participant images from being transmitted to the remote end of the video conference; and / or identifying the best view of the participant to be transmitted to the remote end, rather than another less desirable duplicate view. Thus, although the conference room is covered by multiple cameras, the remote site can be shown a single manipulated data stream that does not contain duplicate people and has the best view of each person in the video conference room. By utilizing the disclosed ReID method, communication between participants in a video conference can be clearer, and the overall video conference experience can be more enjoyable for the participants. Furthermore, the methods discussed herein are applicable to a wide variety of different location and room designs, meaning that the disclosed methods can be easily fitted and applied to any particular conference room.

[0027] As an example, Figure 1 An exemplary meeting room 10 used in a video conference is illustrated. Meeting room 10 includes a conference table 12 and a series of chairs 14. Personnel 16 are seated in the chairs 14 surrounding the conference table 12. Figure 1 In the non-limiting example illustrated, the first person 16A, the second person 16B, the third person 16C, and the fourth person 16D are seated around the conference table 12. Although Figure 1 The illustration shows an example of a video conference room 10 with four people 16, but more or fewer people 16 may be seated around the conference table 12 or otherwise located within the conference room 10 at any given time. Furthermore, more or fewer people 16 may be outside the conference room 10 at any given time. Further examples of video conference rooms, the locations of participants within them, and camera arrangements will be discussed in more detail below.

[0028] Still referencing Figure 1 In some aspects, the video conferencing system 18 may include a camera 20, a microphone array 22, and a monitor 24. More specifically, as in Figure 1 As shown in the example, video conferencing system 18 may include a first or front camera 20A and a second or central camera 20B. However, it is conceivable that video conferencing system 18 may include additional cameras (e.g., a third or left camera, a fourth or right camera, and / or other cameras). The front camera 20A has a field of view (FOV) 25 (horizontal and vertical) and an axis or centerline (CL) 26 that extends along a direction corresponding to the direction pointed to by the front camera 20A (i.e., the line of sight of the front camera 20A perpendicular to its focal point). For example, as in Figure 1 As shown, the front camera 20A may have a horizontal FOV 25A that pans horizontally (i.e., along the width dimension) within the conference room 10 and a vertical FOV 25B that pans vertically (i.e., along the height dimension) within the conference room 10. The central camera 20B may be positioned on the conference table 12 (e.g., at the center of the conference table 12 or at another location on the conference table 12), and the central camera 20B may have a 360-degree FOV 25C. In some examples, the central camera 20B may be arranged as a camera array comprising multiple cameras, such as front-facing cameras, side-facing cameras, rear-facing cameras, etc. Furthermore, the front camera 20A may include a first microphone array 22A and the central camera 20B may include a second microphone array 22B, and the microphone array 22B may use source localization (SSL) for recording and transmitting audio data during the video conference. In some examples, the first microphone array 22A is mounted on or inside the housing of the front camera 20A, and the second microphone array 22B is mounted on or inside the housing of the central camera 20B.

[0029] Additionally, the video conferencing system 18 may include a monitor 24 or television provided to display one or more remote conferencing sites and generally provide speaker output. The monitor 24 may be coupled to a front-facing camera 20A and a first microphone array 22A, although it is contemplated that the monitor 24 may be placed anywhere within the conference room 10, and the video conferencing system 18 may include additional monitors (not shown) placed within the conference room 10.

[0030] Furthermore, the centerline 26 of the front camera 20A is centered along the conference table 12. In some embodiments, the central camera 20B is positioned along the centerline 26 of the front camera 20A and on the conference table 12. In some aspects, the person 16 may be located within more than one FOV 25 of the camera 20, meaning that when the view captured by the camera 20 is transmitted to the remote site of the video conference, the person 16 may be duplicated. This can, in turn, cause confusion in the video conference and / or result in a suboptimal view of the person 16 being transmitted to the remote end of the video conference. Therefore, it is advantageous to re-identify the person 16 across each view captured by the camera 20, thereby reducing confusion in the video conference and ensuring that only one optimal view of each person 16 is shown in the video conference.

[0031] Accordingly, when some individuals 16 are looking forward (i.e., towards the front of the meeting room 10), the front camera 20A can provide a better view of their faces, and when some individuals 16 are facing the center of the table 12, the central camera 20B can provide a better view of their faces. In some aspects, the cameras 20 and microphone array 22 are combined to provide multiple views of the meeting room 10, and the process described herein can be applied to each of the views to re-identify participants across said views and prevent duplicate images of participants from being transmitted to the remote end of the video conference. For example, the systems, processes, and media described herein allow for the identification and selection of the best view for each meeting participant for transmission, while non-ideal views may not be transmitted to the remote end of the video conference. This is accomplished using artificial intelligence (AI) or machine learning subject detector models, as discussed below. As used herein, the “best” view can be the view that provides the best frontal view of the participant (i.e., the best view of the participant’s face compared to other views). For example, the best view can be determined by applying facial recognition technology to the images. In particular, the best view can be a view in which facial features or embedded information recognized by facial recognition technology are of higher quality than any other view.

[0032] An exemplary AI or machine learning human head detector model (also referred to as a subject detector model in this article) will now be referenced. Figure 2-5 To describe. More specifically, now refer to Figure 2and 3 The illustration depicts another exemplary conference room 40 with three video conference participants 42, 44, and 46, located at different coordinate positions. In conference room 40, a front camera 20A has both horizontal and vertical FOVs, and its location relative to room 40 is indicated by three-dimensional (3D) coordinates {0, 0, 0}. Furthermore, the front camera 20A captures views of all three participants 42, 44, and 46 with locations characterized by a panning angle ΦPAN relative to the centerline 26 of the front camera 20A and a distance metric between the front camera 20A and each participant 42, 44, and 46. Specifically, the first participant 42 has a location defined by a first panning angle 48 and a first distance 50. Additionally, the second participant 44 has a location defined by a panning angle 52 and a second distance 54, and the third participant 44 has a location defined by a panning angle 56 and a third distance metric 58.

[0033] Now for specific reference Figure 3 The illustration shows Figure 2 A top view of an exemplary meeting room 40. In some examples, the location of each participant 42, 44, 46 can be characterized by panning angles 48, 52, 56 and distances 50, 54, 58, which are from x 房间 Dimensions or axis 60 and y 房间 The dimension or axis 62 is derived, where the front camera 20A is located at {0, 0} in {x}. 房间 y 房间 At the coordinate position. Specifically, the first participant 42 has a location defined by a first panning angle 48 and a first distance metric 50, the location being characterized by a two-dimensional room distance parameter {-0.5, 1} to indicate that the participant is located along the y-axis. 房间 Axis 62 is located 1 meter from the "vertical" distance (relative to the top view) measured by the front camera 20A and is situated along the y-axis. 房间 x-axis perpendicular to axis 62 房间 The "horizontal" distance measured by axis 60 is -0.5 meters. Additionally, the second participant 44 has a location defined by a second panning angle 52 and a second distance metric 54, characterized by two-dimensional room distance parameters {0, 3} to indicate that the participant is located at a vertical distance of 3 meters (along the y-axis). 房间 (Measured at axis 62) and located at a horizontal distance of 0 meters (along x) 房间The second person is positioned at a point (measured along axis 60) to indicate that they are located at the centerline 26 of the front camera 20A, resulting in a zero-degree panning angle 52. Finally, the third participant 44 has a location defined by a third panning angle 56 and a third distance metric 58, characterized by two-dimensional room distance parameters {1, 2.5}, indicating that the participant is located at a vertical distance of 2.5 meters (along the y-axis). 房间 (Measured at axis 62) and located at a horizontal distance of 1 meter (along x) 房间 (Measured at axis 60).

[0034] Panning angle value (ΦPAN) and two-dimensional room distance parameter {x 房间 y 房间 The relationship between} can be determined using a reference coordinate table (not shown), in which, for} Figure 2 and 3 In the exemplary meeting room 40, the different coordinate positions {x 房间 y 房间 For the meeting participants, calculate the panning angle ΦPAN value of the front camera 20A in the video conferencing room 40. For the coordinate position {-x} in the exemplary meeting room 40... 房间 y 房间 The same table (not shown) can be used to calculate the negative panning angle ΦPAN value (e.g., -ΦPAN). Therefore, it will be understood that for the centerline 26 of the front camera 20A (e.g., x... 房间 =0) lies in any depth metric (e.g., y 房间 For meeting participants with a panning angle of 0.5-8, the same panning angle ΦPAN value will be generated (e.g., ΦPAN = 0). Similarly, for participants located at x... 房间 =y 房间 For any meeting participant at any coordinate location, the same panning angle ΦPAN value will be generated (e.g., ΦPAN = 45). As illustrated, the panning angle ΦPAN alone may not be sufficient information to determine the two-dimensional room distance parameter {x} of the participant's location. 房间 y 房间 For example, due to vanishing perspective, the first participant 42 may appear larger than the second participant 44 relative to the front camera 20A. Therefore, as participants move further away from the front camera 20A, their apparent height and width become smaller for the video conferencing system, and when projected onto the camera image sensor 64, participants are represented by fewer pixels compared to participants closer to the front camera 20A. Furthermore, if two heads are perceived as having the same size by the front camera 20A, they are not necessarily located at the same distance, and as... Figure 3 As illustrated in the figure, due to the panning angle ΦPAN and distortions in height and width, the two heads appear in two dimensions x... 房间 -y房间 The locations in plane 66 may differ.

[0035] In particular, the statistical distribution of human head height and width measurements can be used to determine the minimum-median-maximum measures of participant head size in centimeters. Additionally, by knowing the FOV resolution of the front camera 20A in both the horizontal and vertical directions, and the corresponding horizontal and vertical pixel counts, the angular width of each measured head can be used to calculate the percentage of the head that occupies the entire frame, as well as the number of pixels for head height and width measurements. Using this information to calculate a lookup table of minimum-median-maximum head sizes (height and width) at various distances, an artificial intelligence (AI) subject detector model can be applied to detect the location of each head in the two-dimensional viewing plane, with specified image plane coordinates for the head frame or bounding box and associated width and height measures (e.g., {x 框 y 框 (width, height). By using a reverse lookup table operation, the distance between the front camera 20A and each head located on the centerline 26 of the front camera 20A can be determined.

[0036] For specific references Figure 4 The front camera 20A is used to provide images of the meeting participants captured along a two-dimensional image plane 110. The meeting participants can be positioned at a first centered position 112 and a second panning position 114, the second panning position 114 being in the x-axis... 房间 Lateral displacement in the direction. In the first centered position, the meeting participants are located at a certain distance (d0 = Y meters) along the centerline 26 of the front camera 20A (e.g., ΦPAN = 0), therefore the two-dimensional room distance parameter of the first centered position 112 is {x 房间 =0, y 房间 =Y}. In the second panning position, the meeting participants pan at the panning angle ΦPAN at x 房间 The direction is shifted laterally and it is located at d1 > d0 meters, therefore the two-dimensional room distance parameter of the second panning position 114 is {x 房间 =P,y 房间 =Y}. Furthermore, the same vertical head height measurement V / 2 for meeting participant positions 112 and 114 will result in the angular breadth θFRAME_V1 / 2 of the first meeting participant position 112 being greater than the angular breadth θFRAME_V2 / 2 of the second meeting participant position 114. In fact, the fact that the second panning position 114 is located further from the front camera 20A (d1 > d0) than the first centering position 112 causes the angular breadth of the second panning position 114 to appear smaller than the angular breadth of the first centering position 112, so that θFRAME_V1 / 2 > θFRAME_V2 / 2.

[0037] Based on the foregoing, the problem is to find the angular breadth θ of the entire head height. HH This is then expressed as a percentage of the entire frame's vertical field of view (VFrame_Percentage), which is subsequently converted into the number of pixels the head would occupy at a specific distance and panning angle ΦPAN (VHead_Pixel_Count). This is done by starting with the equation tan(θ) HH1 / 2)=(V / 2) / d0, which can be used to calculate the angular span θ of the entire head height at location 112 of the first meeting participant. HH1 Solve for the angular breadth θ1, and the angular breadth θ of the entire head height. HH1 It can be deduced as θ HH1 =2arctan((V / 2) / d0). Similarly, starting with the equation tan(θ) HH2 / 2)=(V / 2) / d1(where ), which can be used to calculate the angular breadth θ of the entire head height of the second meeting participant location 114 at the panning angle ΦPAN. HH2 Solving for the angular breadth θ HH2 The angular breadth θ of the entire head height HH2 It can be estimated as Based on this calculation, the percentage of the frame occupied by the head height of the second meeting participant at location 114 can be calculated as VFrame_Percentage = θ HH2 / Vertical FOV. Additionally, the number of pixels corresponding to the head height of the second meeting participant at location 114 can be calculated as VHead_Pixel_Count = VFrame_Percentage x Vertical FOV in pixels. Based on the aforementioned calculation, x 房间 and y 房间 The angular span θ for estimating the total head height at discrete distances, such as 0.5 meters, in each direction. HH =θFRAME_V, where the discrete distance is equivalent to the panning angles ΦPAN of various angles, which can be listed in a lookup table (not shown).

[0038] Figure 5 The illustration shows a front-facing camera 20A and an exemplary video conference room 200, which includes a two-dimensional image plane 210 for illustrating how to calculate the direct distance metric X between the front-facing camera 20A and the locations of the meeting participants. 房间 (meters) The vertical or depth room distance Y calculated from the location of the meeting participants. 房间(meters). Two-dimensional image plane 210 includes coordinates {x} of image plane 210. i y i Define multiple two-dimensional coordinate points 212, 214, and 216 as described above. Additionally, the head bounding box 218 is the starting coordinate point {x} of the reference head bounding box 218. i y i}, Width dimension (along x) i (axis measurement) and height dimension (along the y-axis) i Defined by axis measurement. To locate the vertical or depth room distance Y from the front camera (20A). 房间 The vertical angular width (θ) of the head bounding box 218 is calculated as θ = height * V_FOV / V_PIXELS, where height is the height of the head bounding box in pixels, V_FOV is the vertical FOV in degrees, and V_PIXELS is the vertical FOV in pixels. Next, the vertical angular width (θ / 2) of the upper half of the head bounding box is calculated and used to derive the direct distance metric HYP between the front camera 20A and the meeting participants' locations, HYP = V_HEAD / (2 x tan(θ / 2)), where HYP is the direct distance metric to the meeting participants' locations at the panning angle ΦPAN. Finally, using Pythagorean theorem, the direct distance metric HYP and the distance metric x are derived. 房间 (meters) Derive the vertical or depth room distance Y 房间 (rice), In some examples, the width and height dimensions of the head bounding box 218 can also be manipulated (e.g., scaled) to define the upper body bounding box 220 of the meeting participant. For example, the width and height dimensions of the head bounding box 218 can be enlarged to define the upper body bounding box 220, which can surround the head bounding box 218 (i.e., the meeting participant's head) as well as the meeting participant's upper body and clothing. This, in turn, allows for the extraction of more robust embeddings from the image, as will be discussed in more detail below.

[0039] Given this understanding of the AI ​​subject detector model, this disclosure provides methods, apparatus, systems, and computer-readable media for accurately detecting and re-identifying participants in a video conferencing system using multiple cameras (e.g., front and center cameras in an inside-out video conferencing system). The location of each participant or subject is determined by the AI ​​subject detector model using room distance parameters, as discussed above. Specifically, coordinates (e.g., image and / or world coordinates) are determined for each participant in each camera view. In some aspects, the world coordinates identified by the AI ​​subject detector model are referred to as world coordinate points. Further, each participant detected by the AI ​​subject detector model is assigned an identification (ID) tag, and each identification tag in a first view captured by a first camera is grouped or paired with each identification tag in a second view captured by a second camera. In some aspects, identification tags in the second image are paired with identification tags in the first image based on the distance between the identification tags in the first and second images. In another example, an ID label is assigned to each human head in each camera view based on the corresponding coordinates (angle and / or distance) of each human head from the centroid, which is calculated for each corresponding image using the coordinates of each participant.

[0040] Additionally, a reference subject or anchor point can be selected for each camera view, which may be known as the same participant in each camera view. If this is unknown, the primary anchor point in the first image captured by the front camera can be analyzed for embedding, and then compared with the embedding associated with each subject detected in the second image captured by the central camera. Specifically, the distance between embeddings in each image can be measured, and the subject associated with the smallest distance relative to other subjects in the second image can be matched with the primary anchor point, i.e., selected as the secondary anchor point of the second image. By reordering the ID labels in clockwise or counterclockwise order with respect to each centroid (starting from the anchor point in each image), the ID labels can be aligned across all camera views to identify each participant across all camera views. Therefore, it will be understood that multiple methods can be used to identify human heads across different camera views without departing from the scope of this disclosure.

[0041] In view of the above, Figure 6-14 The diagram illustrates an example of an inside-out recognition system and its corresponding operations. In particular, Figure 6-10 The illustration shows another exemplary meeting room 300, which consists of a front wall 302 (see diagram). Figure 7The table 310 is defined by the left wall 304, right wall 306, and rear wall 308. The table 310 is located in the center of the meeting room 300, and seven participants 312 are located within the meeting room 300. For example, the first participant 312A, the second participant 312B, the third participant 312C, the fourth participant 312D, the fifth participant 312E, the sixth participant 312F, and the seventh participant 312G are located within the meeting room 300. Specifically, the first, second, and third participants 312A, 312B, and 312C are seated along the left side 314 of the table 310, the fourth participant 312D is seated along the rear side 316 of the table 310, and the fifth, sixth, and seventh participants 312E, 312F, and 312G are seated along the right side 318 of the table 310. An inside-out recognition system may include a first or front camera 320 (see [link to relevant documentation]). Figure 7 ) and the second or central camera 322 (see Figure 6 In some aspects, the front camera 320 is coupled to the monitor 324 (e.g., fixed to the top of the monitor 324), and the monitor 324 is located at the front of the conference room 300, adjacent to the front wall 302 of the conference room 300 (see...). Figure 7 In some aspects, the central camera 322 is mounted on table 310 (e.g., the center of table 310), and the central camera 322 can be an omnidirectional or 360-degree camera (see...). Figure 6 The inside-out identification system may further include a processor and a memory coupled to the processor, and the memory stores program instructions that, when executed by the processor, cause the processor to perform certain operations described herein, including performing video conferencing operations such as transmitting data to a remote video conferencing site. The processor and the memory may be part of the front camera 320, the central camera 322, and / or a separate codec, as described below. Therefore, some operations of the inside-out identification system may be performed within the front camera 320, the central camera 322, and / or the separate codec.

[0042] Now for specific reference Figure 6 The illustration shows the front camera 320 (see...) Figure 7The first image 326, or front view of the meeting room 300, is captured. As illustrated, the left wall 304, right wall 306, and rear wall 308 are visible in the first image 326, and each participant among the participants 312 is also visible in the first image 326. In other words, walls 304, 306, 308, and participants 312 are within the field of view (FOV) of the front camera 320. After capturing the first image 326, as discussed above, an AI subject detector model can be applied to the first image 326 to determine the world and / or image coordinates of each participant among the participants 312 and generate corresponding bounding boxes for participants 312. In particular, the AI ​​subject detector model can generate head bounding boxes 328A, 328B, 328C, 328D, 328E, 328F, and 328G corresponding to the respective heads of participants 312. The AI ​​subject detector model can also manipulate the coordinates of the head bounding box 328 to further generate upper body bounding boxes 330A, 330B, 330C, 330D, 330E, 330F, and 330G corresponding to the upper body of each participant in the participant 312. The upper body bounding box 330 allows for consideration of the upper body and / or clothing of each participant 312 when extracting or embedding features from the first image 326, as will be discussed in more detail below.

[0043] Once the coordinates of each bounding box in bounding boxes 328 and 330 are known, the center of each bounding box (e.g., head bounding box 328, upper body bounding box 330, or both) is calculated and stored in the inside-out recognition system. For example, as in Figure 6 As shown, the first centroid 332 can be calculated using the centers of bounding boxes 328 and 330 using the following formula:

[0044] and

[0045] In the above formula, (x k y k ), k = 0, 1, ..., n corresponds to the pixel coordinates of the center of each bounding box 328, 330 and (x c y c The coordinates of the first centroid 332 are indicated.

[0046] Once the first centroid 332 has been calculated, ID labels (not shown) (e.g., 0, 1, 2, etc.) can be arbitrarily assigned to each participant 312 by the AI ​​subject detector model, or they can be assigned to each participant 312 based on a predetermined order. In one example, ID labels can be assigned to each participant 312 based on the order in which bounding boxes 328, 330 are calculated and generated (such as a clockwise or counterclockwise order starting from the participant closest to the front camera 320 (e.g., the first participant 312A)). Therefore, it will be understood that various methods can be used to assign ID labels to participants 312. Once an ID label is assigned to a participant 312, the distance (e.g., Euclidean distance) from the center of each bounding box 328, 330 to the first centroid 332 is measured by an inside-out recognition system. In some respects, the inside-out identification can be performed by sorting each participant in the participants 312 in a clockwise or counterclockwise order with respect to the first centroid 332 (starting with the participant 312 with the minimum Euclidean distance to the first centroid 332 (e.g., the fourth participant 312D)).

[0047] Continue to refer to Figure 6 Based on the position of bounding boxes 328, 330 in the first image 326 and / or image quality, the inside-out recognition system can further identify participant 312 as a reference person or (one or more) anchor point in the first image 326. In some aspects, an anchor point is one of the participants 312 detected in the first image 326 that will undergo image processing, such as feature extraction and / or background segmentation, before being compared with participants 312 detected by the central camera 322 to align the images captured by the front camera 320 and the central camera 322. In particular, an anchor point can be a bounding box 328, 330 associated with any participant 312. In some aspects, the known coordinates of participant 312 are used to select the anchor point. It is conceivable that the specific method of selecting the anchor point may depend on the geometry of the room, meaning that the method of selecting the anchor point can be modified to best suit a particular meeting room. For example, the inside-out recognition system may apply the equations in Table 1 below to the first image 326 to identify the anchor point as the one closest to the front wall 302 (see Table 1). Figure 7 (one or more) participants.

[0048] Table 1

[0049] Front view - bottom left <![CDATA[argmin(x i -and i )]]> Front view - bottom right <![CDATA[argmax(x i +and i )]]>

[0050] In the table above, argmin(x) i -y i ) was used to identify the participant furthest from the bottom left corner 336 of the first image 326, and argmax(x)i +y i The first image 326 is used to identify the participant 312 furthest from the bottom right corner 338 of the first image 326 (e.g., the seventh participant 312G). As a result, the inside-out recognition system can identify the front participants (e.g., the first and seventh participants 312A, 312F) as anchor points in the first image 326.

[0051] Additionally, based on the Intersection over Union (IOU) score associated with each participant 312 detected in the first image 326, the inside-out recognition system can select anchor points. To this end, the AI ​​subject detection model can also calculate an IOU score for each participant 312, where the IOU score can be a measure of the modeling accuracy of bounding boxes 328 and 330 for the actual head and upper body of the participant 312, respectively. After calculating the IOU score for each participant 312, the inside-out recognition system can select the participant 312 with the lowest IOU score as an anchor point, or the inside-out recognition system can select any participant 312 with an IOU score less than approximately 50%, less than approximately 40%, less than approximately 25%, less than approximately 10%, or less than approximately 5% as an anchor point. Figure 6 In the non-limiting example illustrated, the inside-out identification system can determine that the third and fourth participants 312C and 312D both have IOU scores of less than approximately 10%, so the inside-out identification system can subsequently select the third and fourth participants 312C and 312D as anchor points.

[0052] Therefore, according to this example, the inside-out identification system can select multiple participants 312 (e.g., first, third, fourth, and seventh participants 312A, 312C, 312D, 312E) as anchor points for further image processing. By selecting a subgroup of participants 312 as anchor points for downstream processing, the inside-out system can streamline subject re-identification, thereby reducing the amount of resources (e.g., power, time, memory space, etc.) required to identify video conferencing participants across different camera images. In some aspects, the inside-out identification system can identify one of the selected anchor points (i.e., first, third, fourth, and seventh participants 312A, 312C, 312D, 312E) as the master anchor point 340. For example, the master anchor point 340 can be selected based on the Euclidean distance of participant 312 relative to the first centroid 332. Figure 6In the non-limiting example illustrated, the fourth participant 312D has the smallest Euclidean distance to the first centroid 332 (i.e., is closest to the first centroid 332), and therefore can be chosen as the primary anchor point 340. However, it is conceivable that the selection of the primary anchor point 340 could also be based on various other factors, such as, for example, the coordinate positions of the bounding box 328 and 330, the ID label order, pre-stored data associated with participant 312, etc. Further aspects of the primary anchor point 340 will be discussed in more detail below.

[0053] Figure 7-10 The illustration shows the central camera 322 (see...) Figure 6 Additional images 342 of the conference room 300 are captured. As discussed above, the central camera 322 may be configured as an omnidirectional camera and / or a camera array with multiple cameras to capture additional images 342. For example, Figure 7 The illustration shows a forward-facing or second image 342A of the meeting room 300 captured by a central camera 322 mounted on a table 310. In the second image 342A, the first participant 312A and the seventh participant 312G, along with a front camera 320 mounted along the front wall 302, are visible. In some aspects, the front camera 320 may be configured to rotate to focus on the active speaker, or the front camera 320 may be fixed in its position and not rotated. As discussed above, the AI ​​subject detector model may generate head bounding boxes 328A, 328G and upper body bounding boxes 330A, 330G for the first and seventh participants 312A, 312G respectively in the second image 342A. It should also be noted that, according to some examples, the AI ​​subject detector model may perform operations to natively generate bounding boxes 328, 330 within the specific cameras 320, 322 that capture the respective views.

[0054] Accordingly, Figure 8 The illustration shows a left-side view or third image 342B of the meeting room 300 captured by a central camera 322 mounted on a table 310. In the third image 342B, the first, second, and third participants 312A, 312B, and 312C are visible. As discussed above, the AI ​​subject detector model can generate head bounding boxes 328A, 328B, and 328C and upper body bounding boxes 330A, 330B, and 330C for the first, second, and third participants 312A, 312B, and 312C in the third image 342B, respectively.

[0055] in addition, Figure 9The illustration shows a rear view or fourth image 342C of the meeting room 300 captured by a central camera 322 mounted on a table 310. In the fourth image 342C, the third, fourth, and fifth participants 312C, 312D, and 312E are visible. As discussed above, the AI ​​subject detector model can generate head bounding boxes 328C, 328D, and 328E and upper body bounding boxes 330C, 330D, and 330E for the third, fourth, and fifth participants 312C, 312D, and 312E in the fourth image 342C, respectively.

[0056] further, Figure 10 The illustration shows a right-side view, or fifth image 342D, of the meeting room 300 captured by a central camera 322 mounted on table 310. In the fifth image 342D, the fifth, sixth, and seventh participants 312E, 312F, and 312G are visible. As discussed above, the AI ​​subject detector model can generate head bounding boxes 328AE, 328F, and 328G and upper body bounding boxes 330E, 330F, and 330G for the fifth, sixth, and seventh participants 312E, 312F, and 312G in the fifth image 342D, respectively.

[0057] You will understand: Figure 7-10 An example image captured by a central camera 322 is illustrated; and depending on user preferences and / or camera configuration, the central camera 322 may capture more or fewer images of the meeting room 300. In some aspects, the inside-out recognition system is configured to: re-identify participants 312 in the additional image 342 to eliminate duplicate views of participants 312 before comparing the additional image 342 with the first image 326. In particular, the inside-out system may compare the embeddings in the additional image 342 to determine whether any participant 312 has been repeated, and subsequently eliminate any duplication. For example, specifically refer to... Figure 7 and 8 Both the second and third images 342A and 342B include a view of the first participant 312A. Therefore, the inside-out recognition system can extract embeddings (e.g., facial features, clothing, posture, etc.) associated with the first participant 312A to determine if the rightmost participant 312 (i.e., the first participant 312A) in the second image 342A and the leftmost participant 312 (i.e., the first participant 312A) in the second image 342B are the same person. It is conceivable that this comparison process could be similarly applied to each of the other images 342 to identify whether any participant among the other participants 312 has been repeated.

[0058] Still referencing Figure 7 and 8After performing the comparison process discussed above, the inside-out recognition system can select a single view of the first participant 312A for downstream processing. In some aspects, the view of the first participant 312A is selected if the first participant 312A is sufficiently visible in the view and / or if the center of the head bounding box 328A of the first participant 312A is closer to the center of the corresponding upper body bounding box 330A in that view than in another view. For example, if the first participant 312A is sufficiently visible in the second image 342A but may only be partially visible in the third image 342B, then the second image 342A can be selected to show the first participant 312A. Furthermore, in the second image 342A, the center of the head bounding box 328A is closer to the center of the upper body bounding box 330A than in the third image 342B, therefore the second image 342A can be selected instead of the third image 342B to show the first participant 312A. As discussed above, it is conceivable that this view selection process could be similarly applied to each of the other images 342 to select a single view of any repeated participant 312.

[0059] Therefore, the inside-out recognition system can eliminate duplicate views of participant 312 captured by the central camera 322, thus preventing them from being transmitted to the remote end of the video conference. In some aspects, the inside-out recognition system can create or stitch together stripes of selected views of participant 312. Now refer to Figure 11 The illustration shows an exemplary central camera strip 344, which includes a single view of each of the participants 312, selected for downstream processing due to the comparison and view selection process described above. Specifically, strip 344 may include views of the first and seventh participants 312A, 312G taken from the second image 342A (see...). Figure 7 ), a view of the second participant 312B taken from the third image 342B (see Figure 8 ), views of the third and fourth participants 312C, 312D taken from the fourth image 342C (see Figure 9 ) and views of the fifth and sixth participants 312E, 312F taken from the fifth image 342D (see Figure 10 In this way, only a single view of each participant 312 can be further processed downstream.

[0060] After eliminating duplicate views of participant 312 captured by central camera 322, the first image 326 (see...) Figure 6The view in strip 344 and the view within it can be subjected to image processing techniques before being compared to re-identify participant 312 across different camera views. Various image processing techniques can be used, including background segmentation and feature or embedding extraction. For this purpose, background segmentation can be used to isolate participant 312 (e.g., participant 312's head and upper body) from the background (e.g., meeting room 300, walls 302, 304, 306, 308, table 310, etc.). This, in turn, allows the embedding extraction process, as discussed below, to generate stronger embeddings, resulting in more accurate and efficient re-identification of participant 312 across different views.

[0061] Now for reference Figure 12A The illustration shows a sample taken from the first image 326 (see image 326). Figure 6 An exemplary pixel map 400 is a view of the fourth participant 312D (i.e., the main anchor point 340). Specifically, pixel map 400 includes the head 402 and upper body 404 of the fourth participant 312D, and also includes a back wall 308 and a table 310. In some aspects, the fourth participant 312D may be identified as foreground 406 in pixel map 400, and the back wall 308 and table 310 may be jointly identified as background 408 in pixel map 400. An inside-out recognition system may apply a background segmentation or removal model to pixel map 400 to isolate the foreground 406 from the background 408, i.e., remove the background 408. Relatedly, Figure 12B The diagram illustrates how the background segmentation model can be applied to... Figure 12A Another exemplary pixel image 410 is generated after pixel image 400. It is worth noting that, in Figure 12A The background 408 shown in pixel map 400 has been removed, so that the foreground 406 (i.e., the fourth participant 312D) is in Figure 12B The pixels in image 410 are isolated. It is conceivable that a background segmentation model can be applied to image 326 (see...). Figure 6 ), 342 (see) Figure 7-10 All images in the first image 326 can be used to isolate all participants in the detected participant 312, or the background segmentation model can be applied only to the main anchor point 340 in the first image 326 and in the second image 312. Figure 11 Participant 312 in band 344.

[0062] As discussed above, embeddings associated with a specific participant 312 can be generated, extracted, and compared with subsequent embeddings associated with other participants to re-identify the participant across different images. Embeddings may include, for example, facial features, clothing, pose, color, etc., associated with a specific participant. In some aspects, AI models can be used to generate embeddings for images. After the embeddings have been generated, various techniques (e.g., K-means clustering and / or Euclidean distance determination) can be used to classify the embeddings and compare clusters of embeddings with each other. In some examples, neural networks can also be used to generate and / or compare embeddings across different images; these neural networks include, for example, deep learning neural networks, convolutional neural networks, feedforward neural networks, recurrent neural networks, radial basis function neural networks, etc.

[0063] Now for reference Figure 13 An exemplary graph 450 is illustrated, which compares the measured Euclidean distance between the embedding in a first image and the embedding in a second image. Specifically, this can be done in the first image 326 (see...). Figure 6 The anchor point detected in the ) or at least the primary anchor point 340 (i.e., the fourth participant 312D) (see Figure 12B Generate and extract the first embedding set. Additionally, for another image 342 (see...) Figure 7-10 Participant 312 detected in ) (e.g., in Figure 11 For each participant (participant 312) in strip 344, a second embedding or set of second embeddings is generated and extracted. After the embeddings have been extracted, the inside-out identification system can compare the first and second embeddings to determine which participant (participant 312) in strip 344 is most similar to the master anchor 340. For example, the Euclidean distance between the first embedding associated with the master anchor 340 and the second embedding associated with each participant (participant 312) in strip 344 can be measured, and the Euclidean distances can then be compared with each other.

[0064] exist Figure 13 In the non-limiting example illustrated, chart 450 includes: an x-axis 452, which may be an index of participant 312 in the strips 344 (i.e., participant 0, 1, 2, etc.); a y-axis 454, which may define the range of Euclidean distances; and data lines 456 for displaying each of the measured Euclidean distances. In some aspects, Euclidean distances can be compared with, for example, in... Figure 12B The main anchor point 340 shown in the figure and in Figure 11The similarity measure among participants 312 in strip 344 shown is inversely proportional to the similarity measure. In this way, a participant 312 in strip 344 with the minimum Euclidean distance relative to the primary anchor 340 can be selected as a secondary anchor 458. For example, a fourth participant 312D in strip 344 can return the minimum Euclidean distance as indicated by circle 460, meaning that the fourth participant 312D in strip 344 has a similarity to the primary anchor 340 (see [reference]). Figure 12B The most similar one is the fourth participant 312D in strip 344. Therefore, the inside-out recognition system can select the fourth participant 312D in strip 344 as a secondary anchor point 458. After the secondary anchor point 458 has been identified, the participants 312 can be rearranged starting from the primary anchor point 340 and the secondary anchor point 458 respectively to generate aligned front and central camera views of the participants 312. In some aspects, repeating this process for each anchor point identified in the first image 326 means that multiple secondary anchor points can be selected due to the comparison of the Euclidean distance between the embedding in the first image 326 and the embedding in strip 344.

[0065] In some examples, the inside-out recognition system can be configured to use a multi-anchor scheme to confirm that the secondary anchor 458 has correctly matched the primary anchor 340. To this end, as described above, the inside-out recognition system can initially identify the primary anchor 340 as well as other primary anchors 340, such as three additional primary anchors 340 from the first image. This can produce, for example, in... Figure 13 The diagram 450 shows a comparison of the measured Euclidean distances between the embeddings in the first image and the embeddings in the second image. The participant 312 with the minimum Euclidean distance relative to the primary anchor 340 in strip 344 can be selected as a potential secondary anchor 458, as described above. Furthermore, two other participants 312 with the second minimum Euclidean distances in image strip 344 can be selected as potential secondary anchors 458.

[0066] Based on this example, refer to Figure 13 In strip 344, the first participant 312A, the fourth participant 312D, and the seventh participant 312G, which have three minimum Euclidean distances relative to the main anchor point 340, can be considered as potential matches or secondary anchor points 458 relative to the main anchor point 340. Participants 312 can then be rearranged in three different orders (i.e., each order begins with one of the three potential secondary anchor points 458), and subsequently, for example, in a table (e.g., ...). Figure 14In Table 462 (illustrated in the figure), these orders can be matched (e.g., aligned) with the order of participants 312 in the first image 326 (starting from the main anchor point 340). For each of these three orders (e.g., for each corresponding Table 462), the Euclidean distance between each of the three additional main anchor points 340 from the first image 326 and its assumed matching participant 312 in strip 344 can be measured, and these three distances are then summed together with the first Euclidean distance between the main anchor point 340 and the corresponding potential secondary anchor point 458. This results in the sum of the Euclidean distances for each potential secondary anchor point 458 being the sum of those four distances. The inside-out identification can then compare the sums of the three Euclidean distances to identify their minimum, and the order of participants 312 corresponding to the minimum Euclidean distance sum can be selected as the correct order. For example, based on this multi-anchor method, an inside-out identification system can identify the order starting with the fourth participant 312D as an order with the minimum sum of Euclidean distances, meaning that the fourth participant 312D can be selected as the true secondary anchor 458. Therefore, it will be understood that various different methods can be used to generate aligned views of participants 312 by the inside-out identification system.

[0067] Figure 14 Table 462 illustrates a first image 326 of the meeting room 300 and an aligned view of each participant in the participant 312. As discussed above, the participants 312 captured by the front camera 320 and the central camera 322 can be rearranged, starting from the main anchor point 340 and the true secondary anchor point 458 (i.e., the fourth participant 312D), respectively. As an example, in the first image 326 (see... Figure 6 ) and stripe 344 (see Figure 11 Images of each of the participants 312 captured by the central camera 322 are arranged in a table 462 above the meeting room 300. The first row 464 of table 462 may include images of each of the participants 312 captured by the central camera 322, and the second row 466 of table 462 may include images of the participants captured by the front camera 320. The first row 464 begins at a secondary anchor point 458, and the second row 466 begins at a primary anchor point 340. The remaining participants 312 are then arranged in rows 464, 466 in a clockwise order about the first centroid 332 (i.e., the order in which they are seated around the table 310).

[0068] Therefore, it will be understood that by sorting participants 312 in a clockwise order starting from anchor points (e.g., primary anchor point 340 and secondary anchor point 458), the order of participants 312 can be aligned across different images. Once the sorting is aligned, the inside-out recognition system can rearrange the ID labels in table 462 to reflect the aligned order of participants 312. In other words, it is possible to align participants across images (e.g., Figure 6 First image 326 and Figure 7-10 Each image in the other image 342) is aligned with the ID tag associated with participant 312. Therefore, although the fourth participant 312D (minor anchor 458) is initially assigned the third ID tag (tag 2) in strip 344, as in... Figure 13 As shown, however, when realigning in Table 462, the fourth participant 312D is assigned the first ID label (label 0), as in Figure 14 As shown in the diagram. Similarly, although the second and third participants 312B and 312C were initially assigned first and second ID tags (tags 0 and 1) respectively in the stripe, as in... Figure 13 As shown, however, when realigned using a clockwise order in Table 462, the second and third participants 312B and 312C are shifted to the end of row 464 and assigned sixth and seventh ID tags (tags 5 and 6) respectively, as in Figure 14 As shown in the image.

[0069] Therefore, the inside-out recognition system disclosed herein is capable of re-identifying video conference participants across different camera views. Accordingly, the inside-out recognition system prevents multiple views of the same participant from being transmitted to the remote end of the video conference, which in turn reduces confusion in video conferences. In some aspects, the inside-out recognition system disclosed herein is particularly advantageous in crowded conference rooms (where there is almost no distance between participants in the conference room, and / or the conference room includes front and center cameras). Furthermore, it is conceivable that... Figure 6-14 The illustration depicts a non-limiting example of an inside-out recognition system, which can be applied to various meeting rooms and is compatible with various camera arrangements. For example, in some applications, an inside-out recognition system can incorporate a front camera, a central camera, and additional outside-in viewing cameras, such as a rear camera, a left camera, and / or a right camera, wherein the views captured by those additional outside-in cameras can be processed in a manner similar to that described above.

[0070] Figure 15The illustration depicts a method 500 for implementing the inside-out recognition system discussed above. In step 502, images of the location are captured using a first or front camera and (one or more) second or central cameras. As discussed above, the front camera may be positioned at the front of a meeting room, and (one or more) central cameras may be positioned on a table in the meeting room and / or configured as omnidirectional cameras. In some aspects, the front camera, central camera, or both communicate with and / or are connected to a monitor and / or a codec including memory and a processor, as will be discussed in more detail below. In step 504, an AI subject detection model is used to detect human heads in the images, as described above. For example, the AI ​​subject detection model is applied to the images captured by each camera in the camera array to identify a head bounding box with specified room and / or pixel coordinates for each detected human head. The AI ​​subject detection model also determines coordinates for each bounding box and may also extract an embedding associated with each detected human head. In some applications, step 504 may also include detecting body bounding boxes, as described above. In step 506, the centroid is determined based on the coordinates of the bounding boxes, or more specifically, based on the coordinates of the center of each bounding box. In step 508, a first participant in the image captured by the front camera is identified as the primary anchor point. The primary anchor point may be selected based on image quality and / or bounding box coordinates. In step 510, the Euclidean distance between embeddings in different images (i.e., images captured by the front camera and the central camera) is measured to match the primary anchor point identified in the image captured by the front camera with a secondary anchor point in the image captured by the central camera. In some aspects, the secondary anchor point is a participant with an embedding detected in the image captured by the central camera, wherein the embedding to the primary anchor point is at the minimum Euclidean distance relative to other subjects in the image captured by the central camera. In step 512, ID tags are aligned in a counter-clockwise order, for example, starting with the first ID tag associated with the bounding box having the minimum polar angle relative to the centroid. In some aspects, this ordering is stored in an inside-out identification system. In this way, the inside-out recognition system associates ID tags across images captured by the front and central cameras, which in turn allows the system to track participants across images. In some aspects, the inside-out recognition system repeats each step of method 500 during normal operation, meaning that centroid-based recognition continuously re-identifies participants as the first and second cameras capture images of the location. Generally, method 500 can be performed in real time or near real time. For example, in some aspects, steps 502, 504, 506, 508, 510, 512 of method 500 are repeated after a period of time has elapsed (such as, for example, at least every 30 seconds, or at least every 15 seconds, or at least every 10 seconds, or at least every 5 seconds, or at least every 3 seconds, or at least every 1 second, or at least every 0.5 seconds).

[0071] It should be noted that the above method 500 or any method or process described herein can be implemented as a set of instructions tangibly embodied on a non-transitory computer-readable medium, such that a processor device can execute the instructions based on reading them from the computer-readable medium.

[0072] Figure 16 The diagram shows what can be done with... Figure 1 The front camera 20A shown in the image or... Figure 7 An exemplary camera 620 similar to the front camera 320 shown in the figure, and similar to the one in the figure Figure 1 The first microphone array 22A shown is an exemplary microphone array 622, similar to this one. The camera 620 has a housing 624 with a centrally located lens 626 for operation with the imager 628. A series of microphone openings 630 (such as five openings 630) are provided as ports for the microphones in the microphone array 622. In some examples, the openings 630 form a horizontal line 632 to provide the desired angle determination for the SSL process, as discussed above. Figure 16 This is an exemplary illustration of camera 620, but numerous other configurations with different camera lens and microphone configurations are possible. Additionally, in some examples, aspects of this technology (including computerized implementations of the methods according to this technology) can be implemented as systems, methods, apparatus, or articles of art that use standard programming or engineering techniques to generate software, firmware, hardware, machine-readable instructions, or any combination thereof to control processor devices (e.g., serial or parallel general-purpose or special-purpose processor chips, single-core or multi-core chips, microprocessors, field-programmable gate arrays, control units, arithmetic logic units, and processor registers, and so on), computers (e.g., processor devices operatively coupled to memory), or other electronically operated controllers, thereby implementing the aspects detailed herein. Thus, for example, this technology can be implemented as a set of instructions tangibly embodied on a non-transitory computer-readable medium, such that a processor device can implement the instructions based on reading them from the computer-readable medium. Some examples of this technology may include (or utilize) control devices, such as, for example, automated devices, special-purpose or general-purpose computers including various computer hardware, software, and firmware, and so on, consistent with the following discussion. As a specific example, a control device may include a processor, microcontroller, field-programmable gate array, programmable logic controller, logic gate, etc., as well as other suitable components for implementing appropriate functions (e.g., memory, communication system, power supply, user interface, and other inputs).

[0073] As described above, some aspects of the method disclosed herein include using an AI subject detector model to detect the location of individual meeting participants. Reference now is made to... Figure 17 The illustration depicts an exemplary process 700 for determining the coordinates of a detected human head using such an AI subject detector process (and particularly an AI human head detector process). The AI ​​human head detector process uses a machine learning-based AI human head detector model 704 to analyze incoming room view video frame images 702 of a conference room scene to detect and display human heads with corresponding head bounding boxes 706, 708, and 710. In some examples, the width and height dimensions of the head bounding boxes 706, 708, and 710 can be manipulated, for example, to define an upper body bounding box 712. As depicted, each incoming room view video frame image 702 can be captured by a front camera 20A in a video conferencing system. Each incoming room view video frame image 702 can be processed by a device-side AI human head detector model 704, which may be located at the corresponding camera capturing the video frame image. However, in other examples, the AI ​​human head detector model 704 may be located at a remote or centralized location, or at a single camera. Regardless of location, the AI ​​human head detector model 704 may include multiple processing modules 714, 716, 718, 720 that implement machine learning models trained to detect or classify human heads from incoming video frame images and identify a head bounding box with specified image planar coordinates and size information for each detected human head.

[0074] In this example, the AI ​​human head detector model 704 may include a first preprocessing module 714, which applies image preprocessing (such as color conversion, image scaling, image enhancement, image resizing, etc.) to prepare the input video frame image for subsequent AI processing. Additionally, a second module 716 may include training data parameters and / or model architecture definitions, which may be predefined and used to train and define the human head detection model 704 to accurately detect or classify human heads from the incoming video frame image. In the selected example, the human head detection model module 718 may be implemented as: model inference software or a machine learning model, such as a convolutional neural network (CNN) model, specifically trained for video codec operations, to detect heads in the input image by generating pixel-by-pixel locations for each detected head and by generating corresponding head bounding boxes that map the detected head. Ultimately, the AI ​​human head detector model 704 may include a post-processing module 720, which applies image post-processing to the output from the AI ​​human head detector model module 718 to make the processed image suitable for human viewing and understanding. Additionally, the post-processing module 720 may reduce the size of the data output generated by the human head detection model module 718, for example, by merging or grouping multiple head bounding boxes or frames generated from a single meeting participant to specify a single head bounding box or frame.

[0075] Based on the results of processing modules 716, 718, and 720, the AI ​​human head detector model 704 can generate output video frame images 702, in which the detected human heads are constructed using corresponding head bounding boxes 706, 708, and 710, and the detected upper body is constructed using a corresponding upper body bounding box 712. As depicted, the first output video frame image 702a includes head bounding boxes 706a, 706b, and 706c and upper body bounding boxes 712a, 712b, and 712c, which are respectively superimposed around each detected human head and upper body. Additionally, the second output video frame image 702b includes head bounding boxes 708a, 708b, and 708c and upper body bounding boxes 712a, 712b, and 712c, which are respectively superimposed around each detected human head and upper body. Furthermore, the third output video frame image 702c includes head bounding boxes 710a, 710b and upper body bounding boxes 712a, 712b, which are respectively overlaid around each detected human head and upper body. The AI ​​human head detector model 704 can use any suitable pixel-based parameters to specify each head bounding box, such as combining the height and width dimensions of the head bounding box or frame to define the x and y pixel coordinates of the head bounding box or frame. Additionally, the AI ​​human head detector model 704 can use any suitable measurement technique to specify the distance metric between the camera location and the location of the detected human head. The AI ​​human head detector model 704 can also calculate a corresponding confidence metric or score for each head bounding box, which quantifies the model's confidence in detecting the human head.

[0076] In some examples of this disclosure, the AI ​​human head detector model 704 can specify all head detections in a data structure that stores the coordinates of each detected human head along with its detection confidence. More specifically, for a number of n human heads, the human head data structure can be generated as follows:

[0077]

[0078] In this example, x i and y i Let represent the image plane coordinates of the i-th detected head, where the width is . i and height i This refers to the width and height information of the head bounding box of the i-th detected head. Additionally, the score... iThe confidence level is reflected as a percentage within the range [0, 100] for the i-th detected head. This data structure can be used as input for various applications (such as mapping, tracking, composition, recording, switching, reporting, encoding, etc.). In this exemplary data structure, the first detected head is located in a head bounding box in the image frame, which is located at pixel location parameters x1, y1 and extends horizontally with a width of 1 and vertically downward with a height of 1. Additionally, the second detected head is located in a head bounding box in the image frame, which is located at pixel location parameters x2, y2 and extends horizontally with a width of 2 and vertically downward with a height of 2, and the n-th detected head is located in a head bounding box in the image frame, which is located at pixel location parameters x1, y1 and extends horizontally with a width of 1 and vertically downward with a height of 2. n y n and according to width n Extend laterally and according to height n Extending vertically downwards. In some respects, the center of each head bounding box is determined using the following equation:

[0079] This human head data structure can then be used as input to a distance estimation process that takes the {width, height} parameters of each head bounding box to calculate the distance from a lookup table (as described above) according to the meeting room coordinates {x}. 房间 y 房间 Selecting the best matching distance: First, use either the width or height parameter with a first lookup table, and then determine the coordinates of multiple meeting rooms {x} using the first parameter. 房间 y 房间 If the condition is not met, other parameters are used as tie-breaking conditions. The human head data structure itself can be subsequently modified to embed distance information with each head as well, resulting in a modified human head data structure that looks like this:

[0080]

[0081] Where {x 房间1 y 房间1}、{x 房间2 y 房间2}、...、{x 房间n y 房间n Specify head1, head2, ..., head according to two-dimensional coordinates. n Distance from the camera.

[0082] Figure 18Aspects of a codec 800 according to some examples of this disclosure are illustrated. As discussed above, codec 800 may be a standalone device in a video conferencing system or may be incorporated into one or more cameras (such as a main camera) within the video conferencing system. Generally, codec 800 includes machine-readable instructions for maintaining video calls with video conferencing endpoints, receiving streams from secondary cameras (and the main camera, if codec 800 is not integrated with the main camera), and encoding and synthesizing the streams according to the methods described herein for transmission to the endpoints.

[0083] As in Figure 18 As shown, codec 800 may include one or more speakers 802, but in many cases, speakers 802 are provided in monitor 804. Codec 800 may include one or more microphones 806 docked via bus 808. Microphones 806 are connected via analog-to-digital (AID) converter 810, and speakers 802 are connected via digital-to-analog (D / A) converter 812. Codec 800 also includes a processing unit 814, a network interface 816, flash or other non-transitory memory 818, RAM 820, and an input / output (I / O) universal interface 822, all of which are coupled via bus 808. Camera 824 is connected to I / O universal interface 822. One or more microphones 806 are connected to network interface 816. HDMI interface 826 is connected to bus 808 and is connected to an external display or monitor 804. Bus 808 is illustrative and can be used with any interconnect between components, such as Peripheral Component Fast Interconnect (PCIe) links and switches, Universal Serial Bus (USB) links and hubs, and combinations thereof. Camera 824 and microphone 806, 806 can be contained in a housing that includes other components, or they can be external and movable, connected by wired or wireless connections.

[0084] The processing unit 814 may include a digital signal processor (DSP), a central processing unit (CPU), a graphics processing unit (GPU), and dedicated hardware components such as a neural network accelerator and a hardware codec.

[0085] Flash memory 818 stores modules for controlling various functions of codec 800 in the form of software and firmware (generally, programs or machine-readable instructions). The illustrated modules include video codec 828, camera control 830, composition 832, other video processing 834, audio codec 836, audio processing 838, network operation 840, user interface 842, and operating system, as well as various other modules 844. In some examples, an AI subject detector module is included using modules incorporated in flash memory 818. Furthermore, in some examples, machine-readable instructions may be stored in flash memory 818, causing processing unit 814 to perform any of the methods described above. RAM 820 is used to store modules, video images of video streams, and audio samples of audio streams while any module in flash memory 818 is executing, and RAM 820 can be used for temporary storage operations of processing unit 814.

[0086] Network interface 816 enables communication between codec 800 and other devices, and can be wired, wireless, or a combination thereof. In one example, network interface 816 is connected to or coupled to the Internet 846 to communicate with a remote endpoint 848 in a video conference. In one example, general interface 822 provides data transmission with local devices (not shown), such as a keyboard, mouse, printer, projector, monitor, external speakers, additional camera and microphone pods, etc.

[0087] In one example, camera 824 and microphone 806 capture video and audio respectively in a video conferencing environment and generate video and audio streams or signals that are transmitted to processing unit 814 via bus 808. As discussed herein, a “view” or “image” of the captured location may include individual frames and / or frames within the captured video stream. For example, camera 824 may be instructed to continuously capture specific views of the location (e.g., images within the video stream) over the duration of the video conference. In one example of this disclosure, processing unit 814 processes the video and audio using processes in modules stored in flash memory 818. The processed audio and video streams may be sent to and received from remote devices coupled to network interface 816 and devices coupled to general interface 822.

[0088] The microphones in the microphone array used for SSL can be used as microphones to provide voice to remote sites, or separate microphones (such as microphone 806) can be used.

[0089] Certain operations of the methods according to this technology, or certain operations of the systems performing those methods, may be schematically illustrated in the accompanying drawings or otherwise discussed herein. Unless otherwise specified or limited, the representation of a particular operation in the accompanying drawings according to a particular spatial order may not necessarily require that those operations be performed in a particular sequence corresponding to said particular spatial order. Accordingly, certain operations illustrated in the accompanying drawings or otherwise disclosed herein may be performed in an order different from the order explicitly illustrated or described, depending on the specific example of this technology. Furthermore, in some examples, certain operations may be performed in parallel (including by dedicated parallel processing devices or independent computing devices interoperating as part of a large system).

[0090] The disclosed technology is not limited in its application to the details of the arrangement and construction of the components set forth in the description below or illustrated in the accompanying drawings. Other examples of the disclosed technology are possible, and the examples described and / or illustrated herein can be practiced or implemented in various ways.

[0091] Multiple hardware and software-based devices and multiple different structural components can be used to implement the disclosed techniques. Additionally, examples of the disclosed techniques may include hardware, software, and electronic components or modules, which, for the purposes of discussion, may be illustrated and described as appearing to be implemented primarily in hardware. However, in one example, the electronic-based aspect of the disclosed techniques may be implemented in software (e.g., stored on a non-transitory computer-readable medium) executed by a processor. Although some figures illustrate hardware and software residing within a particular device, these depictions are for illustrative purposes. In some examples, the illustrated components may be combined or divided into separate software, firmware, hardware, or combinations thereof. As an example, logic and processing may be distributed across multiple electronic processors rather than residing within and executed by a single electronic processor. Regardless of how they are combined or divided, hardware and software components may reside on the same computing device or may be distributed across different computing devices connected via a network or other suitable communication links.

[0092] Any suitable non-transitory computer-usable or computer-readable medium may be used. Computer-usable or computer-readable media can be, for example, but not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatuses, or devices. More specific examples (not an exhaustive list) of computer-readable media will include: portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), portable optical disc read-only memory (CD-ROM), optical storage devices, or magnetic storage devices. In the context of this disclosure, a computer-usable or computer-readable medium can be any medium that can contain, store, deliver, or transmit programs for use by or in connection with an instruction execution system, apparatus, or device.

[0093] As used herein in the context of computer implementation, unless otherwise specified or limited, the terms "component," "system," "module," "block," etc., are intended to cover part or all of a computer-related system, including hardware, software, a combination of hardware and software, or software in execution. For example, a component can be, but is not limited to: a processor device, a process being executed (or executable) by a processor device, an object, an executable file, an executing thread, a computer program, or a computer. As an illustration, both an application running on a computer and the computer itself can be components. A component (or system, module, etc.) may reside within an executing process or thread, may be located on a single computer, may be distributed across two or more computers or other processor devices, or may be included within another component (or system, module, etc.).

Claims

1. A method for identifying participants in a location using a front camera and a central camera, the method comprising: The front camera and the central camera are used to capture images of the location; A machine learning subject detector model is applied to the image to identify coordinates and extract embeddings for each participant detected in the image. Determine the centroid of the coordinates for each participant detected in the image; In each image, identification tags are applied to each participant; The first participant in the first image captured by the front camera will be identified as the main anchor point; The distance between the embeddings in the image is measured to match the primary anchor point with the secondary anchor point detected in the second image captured by the central camera; and Based on the matching, the identification label associated with each participant is aligned across each image in the images.

2. The method of claim 1, wherein the central camera is configured as a 360-degree camera.

3. The method of claim 1, wherein applying the machine learning subject detector model further comprises generating a head bounding box and an upper body bounding box for each participant detected in the image.

4. The method of claim 1, further comprising applying a background removal model to the image to isolate the upper body and head of each participant from the background in the image.

5. The method of claim 1, wherein identifying the first participant as the master anchor comprises one of: identifying the first participant having an IOU score of less than 10%; and identifying the first participant to the centroid.

6. The method of claim 1, wherein measuring the distance between the embeddings in the image comprises: Compare the Euclidean distance between the first embedding associated with the main anchor point and the second embedding associated with each participant detected in the second image; and The second participant detected in the second image, having the minimum Euclidean distance between the first embedding and the second embedding, is selected as the secondary anchor point.

7. The method of claim 6, wherein rearranging the identification tag further comprises: The identification tags in the second image are rearranged in a clockwise order starting from the secondary anchor point.

8. A system for identifying participants in a location, the system comprising: Front camera, used to capture the first image of the location; A central camera is used to capture a second image of the location; A processor, connected to the front camera, the central camera, or both, is configured to execute a program to perform video conferencing operations, including transmitting data to a remote video conferencing site. as well as A memory, coupled to the processor, stores instructions that, when executed by the processor, cause the processor to: For each participant detected in the first and second images, identify coordinates and extract embeddings; Determine the centroid of the coordinates for each participant detected in the first and second images; In the first and second images, identification tags are applied to each participant; The first participant in the first image will be identified as the main anchor point; The embeddings in the first image and the second image are compared to match the primary anchor point with the secondary anchor point detected in the second image; Furthermore, based on the matching, the identification tags associated with each participant are aligned across the first and second images.

9. The system of claim 8, wherein the central camera is configured as a 360-degree camera.

10. The system of claim 8, wherein a machine learning subject detector model is applied to the first image and the second image to generate head bounding boxes and upper body bounding boxes for each participant.

11. The system of claim 8, wherein a background removal model is applied to the first image and the second image to isolate the upper body and head of each participant from the background.

12. The system of claim 8, wherein the processor is configured to identify the primary anchor point as a participant with less than 10% IOU, or the participant closest to the centroid.

13. The system of claim 8, wherein the Euclidean distance between a first embedding associated with the primary anchor point and a second embedding associated with each participant detected in the second image is measured, and wherein the second participant detected in the second image with the minimum Euclidean distance between the first embedding and the second embedding is selected as the secondary anchor point.

14. A non-transitory computer-readable medium comprising instructions that, when executed, cause a processor to: The front camera is instructed to capture a first image of the location, and the central camera is instructed to capture a second image of the location. A machine learning subject detector model is applied to the first image and the second image to identify coordinates and extract embeddings for each participant detected in the first image and the second image; In the first and second images, identification tags are applied to each participant; The first participant in the first image will be identified as the main anchor point; The embedding in the first image and the embedding in the second image are compared to match the primary anchor point with the secondary anchor point detected in the second image; and Based on the matching, the identification labels associated with each participant are aligned across the first and second images.

15. The non-transitory computer-readable medium of claim 14, wherein the Euclidean distance between a first embedding associated with the primary anchor point and a second embedding associated with each participant detected in the second image is measured, and wherein the second participant detected in the second image with the smallest Euclidean distance between the first embedding and the second embedding is selected as the secondary anchor point.

Images