System and method for participant re-identification in multi-camera video conferencing

Abstract

Examples of the present disclosure provide a device, system, and method for identifying participants in a location using multiple cameras. The method includes capturing images of a location using a primary camera and a secondary camera, applying a machine learning human head detector model to the images to detect human heads in the images and identifying coordinates of each human head detected in the images, and determining a centroid of the coordinates of each human head detected in the images and a centroid angle of each human head detected in the images relative to the centroid. The method further includes ranking an identification label of each human head detected in the images based on the centroid angle of each human head, identifying a reference human head in a primary image captured by the primary camera, and aligning the identification labels across each of the images to re-identify each human head.

Description

Background Technology

[0001] Video conferencing systems typically connect people at one video conferencing endpoint (such as a video conferencing room) to people at other video conferencing endpoints. In some video conferencing modes, all participants detected in the video conferencing room are separated and placed in a gallery view to create equality with remote participants. Attached Figure Description

[0002] Figure 1 This is a top view of an example meeting room including multiple cameras, based on some aspects of this disclosure.

[0003] Figure 2 This is a schematic perspective view of another example meeting room, in which three individuals are positioned at different coordinates relative to the video conferencing camera.

[0004] Figure 3 yes Figure 2 A top view of an example meeting room.

[0005] Figure 4 This is yet another example front view of a meeting room based on some examples of this disclosure, in which three individuals are located at different coordinate positions.

[0006] Figure 5 It is a side view determined according to an example vertical dimension of a human head of a subject according to an example of this disclosure.

[0007] Figure 6 It is a top view determined according to an example horizontal dimension of the human head of a subject according to an example of this disclosure.

[0008] Figure 7 Based on Figure 5 and Figure 6 The example defines a perspective view of a subject in a meeting room with different vertical dimensions.

[0009] Figure 8 The room coordinates are determined according to an example of a camera and a schematic diagram of a two-dimensional image plane, with a head bounding box, based on an example of this disclosure.

[0010] Figure 9 This is a top view of an example video conferencing room drawn on an example two-dimensional world plane, with three cameras arranged in an outside-in configuration.

[0011] Figure 10 yes Figure 9 Another top view of the example video conferencing room, featuring two cameras arranged in an inside-out configuration.

[0012] Figure 11 This is a flowchart illustrating a method for implementing a multi-camera calibration system in a conference room, according to an example of this disclosure.

[0013] Figure 12A This is a rear view of yet another example meeting room with a first camera configuration, according to an example of this disclosure.

[0014] Figure 12B It has a second camera configuration. Figure 12A The example is a rear view of a meeting room.

[0015] Figure 13 yes Figure 12A The example meeting room is shown in pixel coordinates from the front view.

[0016] Figure 14 yes Figure 13 The example is a polar coordinate plot of the front view of a meeting room.

[0017] Figure 15 yes Figure 12A The pixel coordinates of the left view of the example meeting room.

[0018] Figure 16 yes Figure 15 The example is a polar coordinate plot of the left view of a meeting room.

[0019] Figure 17 yes Figure 12A The pixel coordinates of the right view of the example meeting room.

[0020] Figure 18 yes Figure 17 The polar coordinate plot of the right view of the example meeting room.

[0021] Figure 19 This is an example of a reference person that has been identified according to this disclosure. Figure 13 Pixel image.

[0022] Figure 20 This is an example of a reference person that has been identified according to this disclosure. Figure 15 Pixel image.

[0023] Figure 21 This is an example of a reference person that has been identified according to this disclosure. Figure 17 Pixel image.

[0024] Figure 22 The ID label is aligned with the ID labels in other views of the example meeting room. Figure 13 Pixel image.

[0025] Figure 23The ID labels are aligned with the ID labels in other views of the example meeting room. Figure 15 Pixel image.

[0026] Figure 24 The ID labels are aligned with the ID labels in other views of the example meeting room. Figure 17 Pixel image.

[0027] Figure 25 This is a flowchart illustrating a method for implementing a centroid-based identification system in a meeting room, according to an example of this disclosure.

[0028] Figure 26 This is a front view of a camera according to an example of this disclosure.

[0029] Figure 27 This is a flowchart of a method for determining the image plane coordinates of a detected subject, according to an example of this disclosure.

[0030] Figure 28 This is a schematic diagram of an example codec based on one example of this disclosure.

[0031] Figure 29 This is a flowchart of a method for re-identifying participants across multiple views in a video conference and selecting the optimal view for each participant, according to an example of this disclosure. Detailed Implementation

[0032] In a video conferencing room equipped with multiple cameras, the same participant may appear in the field of view of more than one camera. Therefore, a problem with video conferencing using multiple cameras is participant duplication, which involves transmitting more than one image of a particular participant to the remote end of the video conference. Correspondingly, the framing of individuals in a video conferencing room can be improved by determining the positions of individual participants in the room relative to each other or a specific reference point. For example, if person A is sitting 2.5 meters from the camera and person B is sitting 4 meters from the camera, the ability to detect this positional information can enable a variety of advanced framing and tracking experiences. For instance, participant positional information can be used to compare multiple camera views and re-identify the same participant in each camera view to determine the optimal camera view for that participant.

[0033] More specifically, when multiple cameras of a video conferencing system are used in a meeting room with two or more participants, multiple views of the participants are captured, such as multiple views from different angles. This is particularly true when the video conferencing system includes a front-side camera and at least one side camera for recording visual data of the video conferencing system. For example, the primary camera may capture a participant's first or front view, and one or more side cameras may capture one or more side views of the meeting participants. As a result, multiple images of the same one or more participants can be transmitted to the far end of the video conference, which can in turn confuse other participants in the video conference. Correspondingly, it is undesirable to transmit a particular view to the far end of the video conference if a participant's face is not fully visible in that particular view. Furthermore, sufficient industry standards or specifications have not yet been developed to track individuals across multiple camera views in a video conferencing system based on their relative distance from each other or their relative positioning in the meeting room.

[0034] For many applications, knowing the horizontal and vertical positions of participants in a room is useful to provide a more comprehensive and complete understanding of the video conferencing room environment. While it may be possible to determine two-dimensional room distance parameters for each participant and use such parameters to identify each participant by using external feature anchors or monocular depth estimation models based on computationally intensive machine learning, such methods impose significant hardware and / or processing costs without providing accuracy in identifying participants across multiple camera views. Furthermore, such methods are limited in their adaptability to a wide variety of different room geometries and locations due to their reliance on dedicated hardware.

[0035] For example, various conventional techniques attempt to perform so-called person re-identification (ReID) to match a participant captured by one camera with the same participant captured by another camera. In one variant, a pre-trained deep learning ReID model is used to identify the participant's embeddings in a specific camera view and match those embeddings with embeddings found in other camera views. However, this technique relies on resource-intensive computation requiring considerable memory to perform, and such computation still proves ineffective for distinguishing participants with similar clothing from one another (e.g., participants wearing uniforms). Furthermore, deep learning ReID models are primarily trained on upright pedestrians, so their performance is further compromised when such models are applied to seated participants.

[0036] In another variation, ReID-based methods involve mapping a three-dimensional (3D) world coordinate system to a two-dimensional (2D) camera plane by using external object or eigenvector matching to determine the transformation matrix. The reliance on external reference points within the room limits the applicability of such models to the physical rooms in which they are situated. As a result, ReID techniques that depend on external reference points to compute the transformation matrix lack adaptability and cannot be used in a wide variety of video conferencing room setups.

[0037] Therefore, in some examples, this disclosure provides methods and apparatus for re-identifying participants across multiple camera views in a video conference. Specifically, this disclosure provides a method for identifying each participant in a video conference room from multiple camera views, preventing duplicate participant images from being transmitted to the remote end of the video conference, and / or identifying the optimal view of the participant to be transmitted to the remote end, rather than another less desirable duplicate view. Thus, when the conference room is covered by multiple cameras, the remote end can be shown a single manipulated stream that does not have duplicate people and has the optimal view of each person in the video conference room. By utilizing the disclosed ReID method, communication between participants in a video conference can be clearer, and the overall video conference experience can be more enjoyable for the participants. Furthermore, the methods discussed herein are applicable to a wide variety of different locations and room designs, meaning that the disclosed methods can be easily assembled and applied to any particular conference room.

[0038] As an example, Figure 1 The illustration depicts a meeting room 38 used in a video conference. Meeting room 38 includes a conference table 40 and a series of chairs 42. Video conference participants 44 are seated in the chairs 42 surrounding the conference table 40. Figure 1 In the non-restrictive example illustrated, the first participant 44A, the second participant 44B, the third participant 44C, and the fourth participant 44D are seated around the conference table 40.

[0039] Still referencing Figure 1 In some aspects, the video conferencing system 46 includes a camera 48, a microphone array 50, and a monitor 52. More specifically, such as Figure 1 As shown in the example, video conferencing system 46 may include a primary camera or front-side camera 48A and (one or more) secondary cameras, such as a secondary camera or left-side camera 48B and a tertiary camera or right-side camera 48C. However, it is envisioned that video conferencing system 46 may include more than Figure 1The illustrations depict more or fewer cameras 48. Each of the cameras 48 has a horizontal and a vertical field of view (FOV), and an axis or center line (CL) extending in a direction corresponding to which the respective camera 48 is pointing (i.e., the line of sight of the respective camera 48 is a straight line at a 90-degree angle to its focal point). In some aspects, each of the cameras 48 includes a corresponding microphone array 50 (i.e., a primary microphone array 50A corresponding to the primary camera 48A, a secondary microphone array 50B corresponding to the secondary camera 48B, and a tertiary microphone array 50C corresponding to the tertiary camera 48C). The microphone arrays 50 can be used to record and transmit audio data in video conferencing using sound source localization (SSL). In some examples, SSL is used in a manner similar to International Application No. PCT / US2023 / 016764 and U.S. Patent Application Publication No. 2023 / 0053202, which are incorporated herein by reference in their entirety. In some examples, each of the microphone arrays 50 is housed on or within the housing of each of the cameras 48. Furthermore, the video conferencing system 46 may include a monitor 52 or television provided to display one or more remote meeting locations and generally providing speaker output. Monitor 52 may be coupled to the front-side camera 48A and the front-side microphone array 50A. However, it is contemplated that each of the left-side camera 48B and the right-side camera 48C may also be coupled to a separate monitor (not shown), and the video conferencing system 46 may include any number of other monitors (not shown) in addition to monitor 52.

[0040] like Figure 1 As illustrated, cameras 48 are positioned such that front camera 48A defines a front CL 54A centered on the length of conference table 40, and left camera 48B and right camera 48C are angled about the front CL 54A on either side of conference table 40. In some respects, a center microphone (not shown) is also provided on conference table 40 to capture the speaker (i.e., the participant speaking) for transmission to the remote end of the video conference. Angles between left camera 48B and right camera 48C on either side of conference table 40 provide additional views of conference room 38 to video conferencing system 46. This, in turn, provides a better opportunity to see the faces of video conferencing participants 44 seated on either side of conference table 40 when participants 44 are facing each other in conference room 38.

[0041] In some respects, participant 44 is located within more than one field of view (FOV) of camera 48, meaning that participant 44 may be duplicated when the view of camera 48 is transmitted to the remote end of the video conference. This could, in turn, cause confusion in the video conference and / or result in suboptimal views of participant 44 being transmitted to the remote end. Therefore, it is advantageous to reidentify participant 44 for each view across camera 48, thereby reducing confusion in the video conference and ensuring that only one optimal view of each participant 44 is shown in the video conference. Furthermore, although Figure 1 An example of a video conferencing room with four participants 44 is illustrated, but at any given time, more or fewer participants 44 may be seated around the conference table 40, and other examples of video conferencing rooms, the positions of participants in the video conferencing room, and the camera arrangement will be discussed in more detail below.

[0042] Correspondingly, when a particular participant 44 looks forward, i.e., towards the front of the meeting room 38, the front-facing camera 48A can provide a better view of their face. In some aspects, the camera 48 and microphone array 50 are used in combination to provide multiple views of the meeting room 38, and one or more of the processes described herein are applied to each of the views to re-identify the participants there and prevent duplicate images of participants from being transmitted to the remote end of the video conference. For example, the processes described herein allow for the identification and selection of the optimal view for each meeting participant for transmission, while less desirable views are not transmitted to the remote end of the video conference. This is achieved using artificial intelligence (AI) or machine learning human head detector models, as discussed below. As used herein, the “optimal” view can be the view that provides the best frontal view of the participant, i.e., the best view of the participant’s face. For example, the optimal view can be determined by applying facial recognition technology to an image, such as those discussed in U.S. Patent Application Publication No. 2023 / 0216988, which is incorporated herein by reference in its entirety. In particular, the optimal view can be the view in which the quality of facial features identified by facial recognition technology is higher than any other view.

[0043] In some respects, the AI ​​human head detector model is substantially similar to that described in International Application No. PCT / US2023 / 016764, which is incorporated herein by reference in its entirety. For example, now refer to Figure 2 and Figure 3The illustration shows a conference room 64 in which three video conference participants 66, 68, and 70 are located at different coordinate positions. In conference room 64, a front-side camera 48A has both horizontal and vertical fields of view (FOV), and the camera's position relative to room 64 is indicated by three-dimensional (3D) coordinates {0, 0, 0}. Further, the front-side camera 48A captures views of all three participants 66, 68, and 70, each having a position that can be characterized in terms of a pan angle ΦPAN relative to the centerline 56 of the front-side camera 48A and a distance metric between the front-side camera 48A and each participant 66, 68, and 70. Specifically, the first participant 66 has a position defined by a first pan angle 76 and a first distance 78. Furthermore, the second participant 68 has a position defined by a pan angle 80 and a second distance 82, and the third participant 70 has a position defined by a pan angle 84 and a third distance metric 86.

[0044] Now for specific reference Figure 3 The illustration shows Figure 2 A top view of the meeting room 64. In some examples, the positions of each participant 66, 68, 70 can be determined from x... 房间 Dimensions or axes 88 and y 房间 Characterized by the panning angles 76, 80, 84 and distances 78, 82, 86 derived from the dimension or axis 90, where the front camera 48A is located in {x 房间 y 房间 The coordinates are located at {0, 0}. Specifically, the first participant 66 has a position defined by a first panning angle 76 and a first distance metric 78, characterized by two-dimensional room distance parameters {-0.5, 1}, to indicate the participant's location along the y-axis from the front camera 48A. 房间 At a vertical distance of 1 meter measured along axis 90, and located along a line perpendicular to y 房间 x of axis 90 房间 The horizontal distance measured by axis 88 is -0.5 meters. Furthermore, the second participant 68 has a position defined by a second panning angle 80 and a second distance metric 82, characterized by two-dimensional room distance parameters {0, 3}, to indicate that the participant is located at a vertical distance of 3 meters (along the y-axis). 房间 (measured at 90 degrees on the axis) and at a horizontal distance of 0 meters (along the x-axis) 房间 The second person is located at a position along the centerline 56 of the front camera 48 (measured along axis 88). Finally, the third participant 70 has a position defined by a third panning angle 84 and a third distance metric 86, characterized by two-dimensional room distance parameters {1, 2.5}, to indicate that the participant is located at a vertical distance of 2.5 meters (along the y-axis). 房间 (measured at 90 degrees along the axis) and at a horizontal distance of 1 meter (along the x-axis). 房间(Axis 88 measurement).

[0045] Pan angle value (ΦPAN) and two-dimensional room distance parameter {x 房间 y 房间 The relationship between them can be determined using a reference coordinate table (not shown), where the coordinates are for the coordinates located at... Figure 2 and Figure 3 Different coordinate positions in example meeting room 64 {x 房间 y 房间 The meeting participants at location} calculate the panning angle ΦPAN value of the front-side camera 48A in the video conference. This can be used to locate the coordinates {-x} in the example meeting room 64. 房间 y 房间 The same table (not shown) is used to calculate the negative panning angle ΦPAN value (e.g., -ΦPAN). Therefore, it will be understood that, for the centerline 56 (e.g., x) along the front camera 48A... 房间 =0) at any depth metric (e.g., y 房间 For meeting participants with a pan angle of 0.5-8, the same pan angle ΦPAN value will be generated (e.g., ΦPAN = 0). Similarly, for participants located in x... 房间 =y 房间 For any meeting participant located at any coordinate, the same pan angle ΦPAN value will be generated (e.g., ΦPAN = 45). As illustrated, the pan angle ΦPAN alone may not be sufficient as a two-dimensional room distance parameter {x} for determining the participant's location. 房间 y 房间 Information such as the vanishing point perspective. For example, due to vanishing point perspective, the first participant 66 may appear larger than the second participant 68 to the front camera 48A. Therefore, as meeting participants move further away from the front camera 48A, their apparent height and width become smaller to the video conferencing system, and fewer pixels are used to represent each participant when projected onto the camera image sensor 92 compared to participants closer to the front camera 48A. Furthermore, if two heads appear to be the same size through the front camera 48A, they are not necessarily located at the same distance, and as... Figure 3 As illustrated in the figure, they are in two dimensions x 房间 -y 房间 The position in plane 94 may vary due to the panning angle ΦPAN and distortion in height and width.

[0046] To illustrate the challenges posed by perspective projection effects in determining the location of meeting participants, see now for an example. Figure 4 , Figure 4Image 96 illustrates another example meeting room 98. Three subjects or meeting participants 100, 102, and 104 are located at different coordinate positions within meeting room 98 and have corresponding head frames or bounding boxes 106, 108, and 110 identified with respect to the coordinate position of each of participants 100, 102, and 104. As depicted, the room width dimension x can be referenced. 房间 and room depth dimensions y 房间 To measure coordinates and determine location. Room width x 房间 From the front camera 48A (see) Figure 1 ) centerline 56 (see Figure 1 The width of the meeting room extends by 98, making x 房间 The negative value is located at the center line 56 (see Figure 1 to the left of ) and x 房间 The positive value is located at the center line 56 (see Figure 1 To the right of ). Additionally, the room depth dimension y 房间 Extending downwards along the length of room 98 from the center line 56 parallel to the front camera 48A (see...) Figure 2 By applying computer vision processing to image 96, the first meeting participant 100 was detected in the rear left corner of room 98, and the region of interest around the head of the first meeting participant 100 was framed using a first head bounding box 106, wherein the first meeting participant 100 is located at a distance parameter {x} in the two-dimensional room. 房间 =-3, y 房间 =21}. Similarly, the second meeting participant 102, seated at table 116, is detected, with the head of the second meeting participant 102 framed using a second head bounding box 108, where the second meeting participant 102 is located at the two-dimensional room distance parameter {x}. 房间 =-1, y 房间 =13}. Finally, the third meeting participant 104 standing on the right was detected, with the head of the third meeting participant 104 framed using the third head bounding box 110, where the third meeting participant 104 is located at the two-dimensional room distance parameter {x}. 房间 =5, y 房间 =14}.

[0047] Specifically, the statistical distribution of human head height and width measurements can be used to determine the minimum-median-maximum metric of head size in centimeters. Furthermore, by knowing the FOV resolution of the front-side camera 48A in both the horizontal and vertical directions, and the corresponding horizontal and vertical pixel counts, the angular range of each head measurement can be used to calculate the percentage of the total frame occupied by the head and the number of pixels for the head height and width metrics. Using this information to calculate a lookup table of minimum-median-maximum head sizes (height and width) at various distances, an artificial intelligence (AI) human head detector model can be applied to detect the position of each head in a two-dimensional viewing plane having specified image plane coordinates of the head frame or bounding box and associated width and height metrics (e.g., {x...}). 框 y 框 (width, height). By using a reverse lookup table operation, the distance between the front camera 48A and each head located on the centerline 56 of the front camera 48A can be determined.

[0048] In some examples, the subject detection process is similar to the AI ​​head detection process disclosed in U.S. Patent Application No. 17 / 971,564, filed October 22, 2022, which is incorporated herein by reference in its entirety. See specifically... Figure 5 The illustration shows a side view of an example vertical dimension determination 200 for a human head 202. In some examples, a front-side camera 48A is positioned to capture an image of the human head 202, such that the vertical head height metric V can be calculated based on the angle range angle θFRAME_V / 2 of the upper half of the vertical head height V / 2 and the distance d between the front-side camera 48A and the human head 202. As illustrated, the human head 202 has a head height V, which corresponds to the vertical dimension of the head bounding box (not shown). From the perspective of the front-side camera 48A, the vertical head height V forms an angle θFRAME_V extending from the bottom to the top of the head 202. When bisecting the angle θFRAME_V, the upper half of the vertical head height V / 2 forms an angle θFRAME_V / 2 with the camera's focal point (i.e., the camera's centerline 56). As a result, the vertical head height metric V can be calculated using the equation tan(θFRAME_V / 2)=(V / 2) / d, based on the angle range θFRAME_V / 2 and the distance d between the front camera 48A and the head 202. Solving for V, the vertical head height metric V can be calculated as V=2d x tan(θFRAME_V / 2).

[0049] Figure 6The illustration shows an example of horizontal dimension determination 206 for a human head 208. In some examples, a front-side camera 48A is positioned to capture an image of the human head 208, such that the horizontal head width metric H can be calculated based on the angular range angle θFRAME_H / 2, which is half of the horizontal head width H / 2, and the distance d between the front-side camera 48A and the human head 208. As depicted, the human head 208 has a head width H, which corresponds to the horizontal dimension of the head bounding box. From the perspective of the front-side camera 48A, the horizontal head width H forms an angle θFRAME_H extending from the side of the head 208. When bisecting the angle θFRAME_H, the upper half of the horizontal head width H / 2 forms an angle θFRAME_H / 2 with the camera's focal point (or a line of sight at a 90-degree angle to the camera's focal point). As a result, the horizontal head width metric H can be calculated using the equation tan(θFRAME_H / 2) = (H / 2) / d based on the angular range θFRAME_H / 2 and the distance d between the front-side camera 48A and the head 208. Solving for H, the horizontal head width metric H can be calculated as H = 2d x tan(FRAME_H / 2). When the human head moves laterally or sideways from the center line 56 of the camera focus by a panning angle ΦPAN, perspective projection makes the head appear smaller than its original vertical and horizontal head metrics V and H.

[0050] Figure 7 The illustration shows a front-side camera 48A and a two-dimensional image plane 210. In some examples, the front-side camera 48A is used to provide localization at a first center 212 and in the x-axis. 房间 Images of the conference participants at the second panning position 214, which is laterally shifted in the direction. In the first centering position, the conference participants are located at d0 = Y meters along the camera's centerline 56 (e.g., ΦPAN = 0), therefore the two-dimensional room distance parameter of the first centering position 212 is {x 房间 =0, y 房间 =Y}. At the second panning position, the conference participants are at x 房间 The panning angle ΦPAN is shifted laterally in the direction, and it is located at a position where d1 > d0 meters. Therefore, the two-dimensional room distance parameter of the second panning position 214 is {x 房间 =P,y 房间=Y}. Furthermore, the same vertical head height measurement V / 2 for the meeting participants' positions 212 and 214 will cause the angle range θFRAME_V1 / 2 of the first meeting participant's position 212 to be greater than the angle range θFRAME_V2 / 2 of the second meeting participant's position 214. In fact, the fact that the second panning position 214 is positioned further away from the front camera 48A than the first center position 212 (d1>d0) makes the angle range of the second panning position 214 appear smaller than the angle range of the first center position 212, such that θFRAME_V1 / 2>θFRAME_V2 / 2.

[0051] Based on the foregoing, the problem is to find the angular range θ across the entire head height. HH This is then expressed as a percentage of the full-frame vertical field of view (VFrame_Percentage), which is then converted into the number of pixels the head will occupy at a specific distance and panning angle ΦPAN (VHead_Pixel_Count). For this purpose, the angular range θ of the entire head height at the first participant's position 212 is... HH1 It can be expressed by the equation lan(θ) HH The calculation begins with (V / 2) / d0. The solution is to determine the angle range θ1, and the angle range θ for the entire head height. HH1 It can be calculated as θ HH1 =2arctan((V / 2) / d0). Similarly, the angular range θ of the entire head height of the second participant position 214 at the panning angle ΦPAN. HH2 It can be expressed by the equation tan(θ) HH2 The calculation begins with (V / 2) / d1, where... Solve for the angle range θ HH2 The angular range θ of the entire head height HH2 It can be calculated as Based on this calculation, the percentage of the frame occupied by the head height of the second meeting participant at position 214 can be calculated as VFrame_Percentage = θ HH2 / Vertical FOV. Furthermore, the number of pixels corresponding to the head height of the second meeting participant at position 214 can be calculated as VHead_Pixel_Count - VFrame_Percentage x Vertical FOV (in pixels). Based on the aforementioned calculation, the angular range θ of the entire head height... HH =θFRAME_V can be in x 房间 and y 房间 The distance is calculated at a discrete distance of 0.5 meters in each direction, which is equivalent to the various angular panning angles ΦPAN that can be listed in a lookup table (not shown).

[0052] Figure 8 The illustration shows a front-side camera 48A and a video conferencing room 300 including a two-dimensional image plane 310, to illustrate how distance metric X can be calculated from the distance metric X by calculating the direct distance metric HYP between the front-side camera 48A and the positions of the meeting participants. 房间 (meters) Calculate the vertical or depth room distance Y to the location of meeting participants. 房间 (meters). The two-dimensional image plane 310 includes multiple two-dimensional coordinate points 312, 314, and 316, wherein the two-dimensional coordinate points 312, 314, and 316 utilize the coordinates {x} of the image plane 310 as described above. i y i The header bounding box 318 is defined by reference to the starting coordinates {x1, y1} of the header bounding box 318, and the width dimension (along the x... i (axis measurement) and height dimension (along the y-axis) i (Axis measurement) is used to define the vertical or depth room distance Y from the front camera (48A). 房间 The vertical angular range (θ) of the head bounding box 318 is calculated as θ = height * V_FOV / V_PIXELS, where height is the height of the head bounding box in pixels, V_FOV is the vertical FOV in degrees, and V_PIXELS is the vertical FOV in pixels. Next, the vertical angular range (θ / 2) of the upper half of the head bounding box is calculated and used to derive the direct distance metric HYP between the front camera 48A and the participant's position, HYP = V_HEAD / (2 x tan(θ / 2)), where HYP is the direct distance metric from the panning angle ΦPAN to the participant's position. Finally, using the Pythagorean theorem, the direct distance metric HYP and the distance metric X are calculated. 房间 (meters) Export vertical or depth room distance Y 房间 (rice),

[0053] Based on this understanding of the AI ​​human head detector model, this disclosure provides methods, apparatus, systems, and computer-readable media for accurately detecting and re-identifying participants in a video conferencing system using multiple cameras. As discussed above, the position of each conference participant is determined by the AI ​​human head detector model using a room distance parameter. Specifically, coordinates, such as image and / or world coordinates, are determined for each participant in each camera view. In some aspects, the world coordinates identified by the AI ​​human head detector model are referred to as world coordinate points. Further, identification (ID) tags are assigned to each conference participant detected by the AI ​​human head detector model, and each ID tag in a first view captured by a first camera is grouped or paired with each ID tag in at least one second view captured by at least one second camera. In some aspects, ID tags in a second image are paired with ID tags in a first image based on the distance between the ID tags in the first and second images. However, in another example, the participant coordinate positioning is transformed into 2D world coordinates and projected back onto the coordinate system of one of the cameras (e.g., a first or primary camera). By mapping the positions of conference participants from each camera view to a single camera coordinate system, it becomes possible to identify the same participant in each camera view, thus preventing duplicate views of participants from being transmitted to the remote end of the video conference. In yet another example, the centroid of each camera view can be calculated using the image coordinates of detected human heads, and then the image coordinates can be transformed into polar coordinates. In some aspects, ID labels are assigned to each human head in each camera view based on its corresponding polar coordinates, such as the angle with respect to the corresponding centroid. Subsequently, a reference human head can be determined for each camera view, which is known to be the same participant in each camera view. By re-ranking the ID labels in counter-clockwise order with respect to each centroid, starting with the reference human head, ID labels can be aligned across all camera views. Therefore, it will be understood that multiple methods can be used to identify human heads across different camera views without departing from the scope of this disclosure.

[0054] According to some aspects of this disclosure, a multi-camera calibration system is provided that can identify and map participant coordinates across different camera views and coordinate systems. In some examples, the multi-camera calibration system includes multiple cameras, such as, for example, primary and secondary cameras, which are used to capture images of a location. In some aspects, the location is part of a meeting room, an enclosed room, an open-concept workspace, and / or an open-concept workspace. The primary and secondary cameras may each define a corresponding coordinate system. According to one aspect of this disclosure, points in one coordinate system (e.g., the secondary camera coordinate system) can be mapped to another coordinate system (e.g., the primary camera coordinate system) by considering or evaluating scaling, translation, and rotation factors between the two coordinate systems. For example, the following transformation equation governs the coordinate mapping between the primary coordinate system A of the primary camera and the secondary coordinate system B of the secondary camera: X a =O x +S·cos(α)*X b -S*sin(α)*Y b Y a =O y +S·sin(α)*X b +S*cos(α)*Y b In the equation above, (Xa, Ya) are points represented in the primary coordinate system A, (Xb, Yb) are points represented in the secondary coordinate system B, (Ox, Oy) is the origin of the secondary coordinate system B, α is the angle between the axes of the two coordinate systems A and B, and S is the scaling factor between the two coordinate systems A and B.

[0055] Furthermore, the multi-camera calibration system determines the coordinates (O) of each camera in the room relative to each other. x O y The equations above can be simplified to obtain the following simplified transformation equation: X a =a*X b -b*Y b +c Y a =b*X b +a*Y b +d To determine the parameters a, b, c, and d in the equation above, the cameras may include calibration settings to capture calibration points. In some aspects, a calibration point is defined as the world coordinates of a single participant (i.e., a single human head) visible to each camera in a video conferencing room. Further, an AI head detector model, as described above, can be used to determine the participant's world coordinates. During the calibration phase, each camera may capture an image of a single participant over a time period to record a predetermined number of calibration points. In some aspects, the calibration phase may require at least 1500 frames of the participant to be captured by each camera, or approximately 50 seconds of video at 30 frames per second (FPS). Therefore, it will be understood that each calibration point corresponds to one frame from the camera, meaning that the total number of calibration points captured by the cameras corresponds to the total number of frames captured during the calibration phase.

[0056] After the calibration points have been captured, the parameters a, b, c, and d in the simplified transformation equation above are solved using the least-squares solver AX = L as described below: Therefore, referring to the matrix equation above, in the least squares solver AX = L, L is defined as the vector of calibration points captured by the primary camera, X is defined as the vector [a, b, c, d], and A is defined as the matrix of calibration points captured by the secondary camera. Once the parameters a, b, c, and d have been determined, α, S, and (O) can be calculated using the following relations. x O y ): O x =c and O y =d.

[0057] After the calibration phase is complete, the positions of the primary and secondary cameras are known to each other. This means that the first set of equations above can be used to map any point in the secondary coordinate system to the primary coordinate system, and vice versa. Therefore, the world coordinates of every human head detected in the room using the AI ​​head detector method discussed above are all geometrically transformed, i.e., projected onto the primary coordinate system. Accordingly, each frame captured by the camera is projected onto the primary coordinate system, and a relabeling process is used to further analyze neighboring frame pairs. In the relabeling phase, the Euclidean distance between all points captured by neighboring frame pairs is measured, and points with the minimum Euclidean distance are then grouped together in pairs or tuples. In other words, points in neighboring frames are compared in the relabeling phase to determine which points are closest to each other. Any two points that are closer to each other than any other point in the frame, i.e., points with the minimum Euclidean distance, are identified as corresponding to the same participant. In this way, pairs of points with the minimum Euclidean distance are assigned a common ID, thereby identifying each participant in each camera view.

[0058] Now for reference Figure 9 A top view of the video conferencing room 400 is drawn on a two-dimensional world plane 402. Specifically, world plane 402 corresponds to the actual spatial location of objects and / or participants in the video conferencing room 400. World plane 402 defines world plane coordinates {x} as described above. i y i The coordinate system is 404, where xi denotes the x-axis of the world plane 402, and y i The y-axis is marked on world plane 402. Accordingly, it can be understood that the right side 406 of room 400 is defined by the negative y-axis. i The coordinates indicate that room 400 is to the left of 408 by positive x. i Coordinates are indicated. In other words, room 400 is x i The axis is bisected. In the first example, the video conferencing room 400 includes a first camera Cam0, a second camera Cam1, and a third camera Cam2. In some respects, the first camera Cam0 is located at the origin O of coordinate system 404, the second camera Cam1 is located to the right 406 of room 400, and the third camera Cam2 is located to the left 408 of room 400. Each of the cameras Cam0, Cam1, and Cam2 is configured to capture a view of room 400. Specifically, the first camera Cam0 defines a view along the x-axis. i The first FOV 410, oriented along the axis, and the second camera Cam1 define the x-axis from the right 406. i The second FOV 412 is angled to the axis, and the third camera Cam2 defines the x-axis from the left 408. iA third FOV 414 is angled to the axis. In the illustrated example, the first camera Cam0 and the third camera Cam2 are located within the second FOV 412, and the first camera Cam0 and the second camera Cam1 are located within the third FOV 414. In some aspects, FOVs 410, 412, and 414 overlap each other, and the arrangement of cameras Cam0, Cam1, and Cam2 is referred to as an outside-in arrangement. In some aspects, the area closed by each of FOVs 410, 412, and 414 is defined as an intersection area 416. As illustrated, participants A, B, C, D, E, F, and G are located within each of FOVs 410, 412, and 414, meaning that each of the participants is visible in every view captured by cameras Cam0, Cam1, and Cam2. In some aspects, the first camera Cam0 includes a codec with a processing unit (see...). Figure 28 The processing unit maintains video conference calls or events (e.g., streaming video to a remote endpoint) and further applies the AI ​​head detection model discussed above to the view captured by cameras Cam0, Cam1, and Cam2. Correspondingly, the AI ​​head detection model determines the 2D world coordinates of each participant relative to the coordinate systems of each of cameras Cam0, Cam1, and Cam2. For example, participant A has world coordinates (3, 2), participant B has world coordinates (5, 2), participant C has world coordinates (6, 6, 4), and so on.

[0059] In some aspects, according to the calibration phase, the outside-in calibration system first determines the positions of the second camera Cam1 and the third camera Cam2 relative to the first camera Cam0 in the first camera's coordinate system, i.e., world coordinates. Specifically, the transformation equations discussed above are applied to the images captured by each camera Cam0, Cam1, and Cam2, and a single participant, such as participant A, is used to create calibration points as discussed above. In this way, the coordinates of the second camera Cam1 and the third camera Cam2 can be mapped to the first camera Cam0 coordinate system using the following equation, which is derived from the previously discussed transformation equations: X 01 =O x1 -Y1 Y 01 =O y1 +X1 X 02 =O x2 +Y2 Y 02 =O y2 -X2 In the equation above, X 01 Y 01The coordinates have been mapped from the second camera's Cam1 coordinate system to the first camera's Cam0 coordinate system, and X... 02 Y 02 The coordinates have been mapped from the third camera's Cam2 coordinate system to the first camera's Cam0 coordinate system. Correspondingly, when viewed from the first camera Cam0, O x1 O y1 O x2 O y2 These are the coordinates of the second camera Cam1 and the third camera Cam2. After the calibration points have been mapped onto the coordinate system of the first camera Cam0, the least-squares solver discussed above is used to determine each parameter in the simplified transformation equation, which in turn is used to determine the angle α, scaling factor S, and origin coordinates O of the second camera Cam1 and the third camera Cam2. x1 O y1 O x2 O y2 Once these parameters are known, the calibration phase is complete.

[0060] Following the calibration phase, the cameras enter a reidentification phase, which remains active throughout the video conference. During the reidentification phase, an AI head detector model, as discussed above, is applied to images captured by cameras Cam0, Cam1, and Cam2 to determine the world coordinates of each participant detected in each image. Using the transformation equations discussed above, the world coordinates of participants in each view are mapped to the primary coordinate system of the first camera Cam0. This, in turn, results in clusters of points formed around the coordinates identified by the first camera Cam0. The Euclidean distance between each point in the cluster, i.e., the point corresponding to the participant's location in room 400, is measured, and the point pairs or tuples with the smallest Euclidean distance are assigned a common ID, thus identifying the point pair or tuple as a single conference participant. In this way, individual participants are reidentified across each captured camera view.

[0061] Now for reference Figure 10 The illustration shows another top view of a video conferencing room 400. In this example, the video conferencing room 400 includes a primary camera or a front-facing camera. f and secondary camera or center camera Cam c In some respects, the front-facing camera... f Located at the origin O of coordinate system 404, and the center camera Cam c Located along x i At the center of room 400 of the axis. For example, the center camera Cam. c It has world coordinates (6, 0). In some respects, the front-side camera... f and center camera Camc The arrangement is referred to as an inside-out arrangement. Front-side camera (Cam) f Defined along x i The first FOV is 410, oriented along the axis, and the center camera is Cam. c It can be a 360-degree camera, which means the center camera (Cam) c A fourth circular field of view (FOV) 418 is defined (as opposed to the sector-shaped FOVs 410-414), extending circumferentially around itself, for example, around the perimeter of the video conferencing room 400 and / or a portion of the video conferencing room 400. Therefore, Figure 10 The intersection area 416 in the diagram, arranged from the inside out, is entirely controlled by the front camera Cam. f The first FOV is 410. In some respects, multi-camera calibration systems first require that participants can be within the range of cameras (Cam). f Cam c Before being re-identified in the captured images, the center camera Cam c The coordinates for the front camera Cam f (That is, the primary camera) is known. Because the central camera Cam c Given a circular FOV of 418, the transformation equations discussed above can be used to convert world Cartesian coordinates to coordinates about the central camera Cam. c The world polar coordinates. For example, the transformation equation above can be converted into polar coordinate form, as shown below: X fc =O xc +r*cosθ fc =O yc +r*sinθe polar coordinate transformation equation, X fc Y fc The markings have been removed from the center camera. c Mapped to the front camera Cam f The coordinates of O, and O xc O yc Indicates when using the front-side camera (Cam) f Viewing Center Camera Cam c The coordinates. Similar to the outside-in arrangement discussed above, a least-squares solver and calibration points are used to solve for the parameters during the calibration phase, and the AI ​​head detector model, as discussed above, is applied to the camera. f Cam c The captured images were used to determine the world coordinates for each image. The Cartesian world coordinates were converted to polar world coordinates and input into the polar transformation equation above to map the position coordinates of all participants to the front-side camera. fOn the coordinate system, a point cluster is generated in this manner. During the reidentification phase, the Euclidean distance between each point in the cluster is measured, i.e., the point corresponding to the participant's location in room 400, and the point pair or tuple with the smallest Euclidean distance is assigned a common ID, thus identifying the point pair or tuple as a single meeting participant. Therefore, individual participants are reidentified across each captured camera view. Thus, as shown above, some aspects of a multi-camera calibration system can be applied to a wide variety of different camera settings, meaning that the multi-camera calibration system is scalable and can include more than... Figure 9 and Figure 10 The illustration shows fewer or additional cameras.

[0062] In view of the above, Figure 11 The illustration depicts a method 500 for using a multi-camera calibration system in a meeting room, as discussed above. At step 502, calibration frames are captured using a primary camera and one or more secondary cameras. As discussed above, the calibration frames contain calibration points corresponding to individual participants, and the individual participant can move around the meeting room while the cameras capture the calibration frames. At step 504, the world coordinates of one or more secondary cameras are determined relative to the primary camera, i.e., relative to the view captured by the primary camera. In some respects, the transformation equations discussed above are used to determine the world coordinates of one or more secondary cameras. Once the world coordinates of one or more secondary cameras are known, the multi-camera calibration system exits the calibration phase (which may include steps 502 and 504) and enters a normal operation phase, in which an AI head detection model is used to detect human heads, as shown at step 506. For example, an AI head detection model is applied to each image captured by the camera to design a head bounding box with specified room coordinates and dimensions for each detected human head. The specified room coordinates and dimensions are then used to calculate the horizontal panning distance and depth dimension distance measured from the camera from a top-down viewpoint within the room. In this way, the two-dimensional world coordinate position of each detected human head is determined.

[0063] At step 508, the multi-camera calibration system enters a re-identification phase, and the world coordinates of the human heads detected in each captured view are projected, i.e., mapped onto the coordinate system of the primary camera. This, in turn, generates clusters around the points corresponding to the heads detected in the images captured by the primary camera. At step 510, the Euclidean distance between all points is measured. At step 512, the points with the smallest Euclidean distance are clustered, i.e., sorted in pairs and / or tuples, and assigned a shared ID. Accordingly, points with the same ID are identified as corresponding to the same participant, and this information is relayed to the multi-camera calibration system. In some respects, method 500 then returns to step 506 and continues as discussed above. In other words, the multi-calibration system returns to the normal operation phase and repeats steps 506, 508, 510, and 512 for the remainder of the meeting. In this way, meeting participants are continuously re-identified by the multi-calibration system, which in turn eliminates the transmission of duplicate views to the remote end of the meeting. In other words, when all participants are displayed at the far end of the meeting, for example, in a gallery view, using method 500 above, each participant is transmitted only once, even though they appear in multiple FOVs. Generally, method 500 can be performed in real-time or near real-time. For example, in some aspects, the multiple calibration system enters a re-identification mode after a period of time has elapsed, such as, for example, at least every 30 seconds, or at least every 15 seconds, or at least every 10 seconds, or at least every 5 seconds, or at least every 3 seconds, or at least every second, or at least every 0.5 seconds.

[0064] It should be noted that any of the cameras used in the systems and methods described herein may have machine learning computational capabilities, enabling them to run machine learning models, or such computation may be offloaded to a primary camera or a separate centralized codec. Therefore, the steps of method 500 described above (or any of the other methods described herein) may be machine-readable instructions executed by a processor of one or more cameras and / or a separate codec coupled to the system.

[0065] According to another aspect of this disclosure, a centroid-based identification system is provided to identify and track participants across multiple camera views based on their relative positioning around a conference table. In some examples, the conference room includes multiple cameras, such as, for example, primary and secondary cameras. The primary and secondary cameras can capture images of the conference room, and more specifically, images of participants seated around a conference table in the conference room. The images captured by the cameras can define an image plane represented by a pixel coordinate system. In some aspects, an AI head detection model is applied to the centroid-based identification system to create bounding boxes around the participants' heads and determine the centroid of all bounding boxes in a single image plane, which are then displayed in the corresponding pixel coordinate system. In each image, the pixel coordinates of each bounding box are transformed to polar coordinates and ranked counterclockwise, starting with the bounding box having the lowest polar angle to the centroid relative to other polar angles in the image. In other words, the bounding boxes are ranked counterclockwise, starting with the bounding box having the smallest centroid angle. A reference person is selected in each image, and the ranking of each image is rearranged starting with the reference person in each image. The resulting rankings are stored in the system and / or appended to the original rankings. In this way, the rankings for each view are aligned with each other, and ID labels are assigned to each participant. Correspondingly, the ID labels are the same across all camera views, meaning that each participant is identified across all camera views.

[0066] Figures 12A-24 An example of a centroid-based labeling system as discussed above is illustrated. Specifically, Figure 12A and Figure 12B The illustration shows a meeting room 600 defined by a front wall 602, a left wall 604, and a right wall 606. A table 608 is located at the center of the meeting room 600, and six participants 610 are located within the meeting room 600. For example, first participant 610A, second participant 610B, third participant 610C, fourth participant 610D, fifth participant 610E, and sixth participant 610F are located in the meeting room 600. Specifically, first participant 610A and second participant 610B are seated along the left side 612 of the table 608, third participant 610C and fourth participant 610D are seated along the rear side 614 of the table 608, and fifth participant 610E and sixth participant 610F are seated along the right side 616 of the table 608. Furthermore, a centroid-based signage system includes a first camera 618, a second camera 620, and a third camera 622. In some aspects, the first camera 618 is coupled to the monitor 624, for example, fixed to the top of the monitor 624, and the monitor 624 is located in front of the meeting room 600, adjacent to the front side 626 of the table 608. In some aspects, the second camera 620 and the third camera 622 are also coupled to the monitor (not shown).

[0067] Now for specific reference Figure 12A A first example arrangement of cameras 618, 620, and 622 may include mounting the first camera 618 forward-facing, for example, directly in front of and / or on the front wall 602; mounting the second camera 620 to the left wall 604; and mounting the third camera 622 to the right wall 606. In the illustrated example, each of cameras 618, 620, and 622 is angled toward the table 608 to observe participant 610. In other words, each of participant 610 is within the field of view (FOV) of the first camera 618, the second camera 620, and / or the third camera 622. However, it is contemplated that cameras 618, 620, and 622 may be arranged in various other configurations without departing from the scope of this disclosure. Figure 12B The illustration shows a second example arrangement of cameras 618, 620, and 622, wherein the first camera 618 is mounted on top of monitor 624 and faces the rear wall (not shown) of meeting room 600. Furthermore, the second camera 620 can be mounted on the left side of monitor 624 and diagonally towards the rear wall (not shown) and right wall 604 (see [reference needed]). Figure 12A The camera is angled so that it is focused on the right side 616 of the table 608. Similarly, the third camera 622 can be mounted to the right side of the monitor 624, and diagonally towards the rear wall (not shown) and the left wall 606 (see table 608). Figure 12A The cameras are angled so that they are focused on the left side 614 of table 608. Therefore, it will be understood that cameras 618, 620, and 622 can be arranged in a variety of different positions, including... Figure 12A and Figure 12B The positions shown in the diagram. In some respects, cameras 618, 620, 622 are configured to rotate to focus on the active speaker, or the cameras are fixed in their respective positions and do not rotate.

[0068] Now for reference Figure 13 The front view image 628, i.e., the image of the meeting room 600 captured by the first camera 618, is overlaid on a first pixel coordinate system 630, which may be a Cartesian coordinate system. As illustrated, the table 608, the participants 610, the second camera 620, and the third camera 622 are all visible in the front view image 628, i.e., within the field of view (FOV) of the first camera 618. In some respects, the arrangement of cameras 618, 620, and 622 is substantially similar to... Figure 12AThe arrangement is illustrated in the diagram. After capturing the frontal image 628, as discussed above, an AI head detector model can be applied to this image. The AI ​​head detector model generates head bounding boxes 640A, 640B, 640C, 640D, 640E, and 640F for participants 610A, 610B, 610C, 610D, 610E, and 610F respectively, and draws bounding boxes 640 on the first pixel coordinate system 630. Once the pixel coordinates of each bounding box 640 are known, the center of each bounding box is calculated and stored in a centroid-based labeling system. In some respects, the center of each bounding box 640 is used to describe the position of each participant 610 using pixel coordinates, which will be discussed in more detail below. Using the center of the bounding box 640, the first centroid 642 is calculated using the following formula: and In the formula above, (x k y k ), k = 0, 1, ..., n corresponds to the pixel coordinates of the center of each bounding box 640, and (x c y c The pixel coordinates of the first centroid are marked as 642.

[0069] Once the first centroid 642 has been determined, ID labels ID_N, such as ID_0, ID_1, ID_2, ID_3, ID_4, and ID_5, are assigned to each of the participants 610. For example, ID labels can be arbitrarily assigned to each participant 610 using an AI head detector model, or they can be assigned based on a predetermined order. In one example, ID labels can be assigned to each participant 610 based on the order in which they are calculated and generated from the bounding boxes 640, such as with the one having the largest (x) k y k The ID labels are assigned to participants 610 in a clockwise order, starting with the sum of the pixel coordinate values ​​of the participants (e.g., the sixth participant 610F), or another method. Therefore, it will be understood that various methods can be used to assign ID labels to participants 610. Once the ID labels are assigned to participants 610, the centroid-based identification system measures the distance from the center of each bounding box 640 to the first centroid 642, and then transforms the first pixel coordinate system 630, which includes the coordinates of the Cartesian bounding box 640 center and the coordinates of the first centroid 642, into polar coordinates. Specifically, the pixel coordinates are transformed into polar coordinates using the following equation, such that the first centroid 642 is the polar coordinate origin O: x′ k =x k -x c y′ k =y c-y k

[0070] In the equation above, when using pixel coordinates (x... c y c When ) is the origin O, 〖(x〗′ k y′ k ) corresponds to the updated pixel coordinates of the center of each bounding box 640. Further, ρ corresponds to the distance between the center of each bounding box 610 and the origin O (i.e., the first centroid 642), and This corresponds to the angle between the center of each bounding box 640 and the origin O. Therefore, it will be understood that... The new polar coordinates of the center of each bounding box 610 relative to the first centroid 642 are indicated. In some respects, the angle between the center of each bounding box 610 and the origin O is also indicated. This can be referred to as the centroidal angle. In some examples, it is such that all centroidal angles are... Both are positive, which can be achieved using the following conditional equation: In other words, if the centroid angle If the value is negative, then add 360 degrees to the centroid angle.

[0071] Figure 14 The illustration shows a 2-D representation of the position of participant 610 relative to the first centroid 642 using polar coordinate system 644. As discussed above, the polar coordinates of each participant 610 relative to the first centroid 642 are determined. For example, the fourth participant 610E can have polar coordinates. like Figure 14 As illustrated in the diagram. Once the polar coordinates of the center of each bounding box 640 have been determined... The centroid-based identification system determines which bounding box 640° center has the smallest centroid angle. exist Figure 13 and Figure 14 In the non-restrictive example illustrated, the fifth participant 610E has the smallest centroid angle. And then it was labeled accordingly. Then, based on the label of the centroid, each of the participants 610 was ranked in counter-clockwise order with respect to the first centroid 642, to find the one with the smallest centroid angle. Starting with participant 610, for example, fifth participant 610E. For example, participants 610 are ranked in the following order: fifth participant 610E, fourth participant 610D, third participant 610C, second participant 610B, first participant 610A, and sixth participant 610F. This ranking is stored in a centroid-based labeling system and corresponds to the front image 628. However, it is envisioned that another system could be used to rank participants 610, such as, for example, clockwise with respect to the first centroid 642. Furthermore, it is envisioned that different participants 610 could be used instead of those with the smallest centroid angle. Participants, such as those with the largest centroid angle The participants. While a wide variety of different techniques can be used to rank participants, it will be understood that the techniques used to rank participants are consistent across all camera views (e.g., views captured by the first camera 618 as well as the second camera 620 and the third camera 622) to ensure accurate identification and re-identification of participants, which will be discussed in more detail below.

[0072] Now for reference Figure 15 The left-hand image 648, which is the image of the meeting room 600 captured by the second camera 620, is overlaid on the second pixel coordinate system 650. For the front-hand image 628 (see...), Figure 13 The centroid-based labeling system applies an AI head detector model to the left image 648, which in turn generates head bounding boxes 640 for participants 610, and calculates a second centroid 652 based on the center of the bounding boxes 640. Furthermore, as discussed above, ID tags ID_N are arbitrarily or using a specific regime applied to each of the participants 610. In some aspects, the ID tags are used in conjunction with... Figure 13 The different orders illustrated in the diagram are applied to participant 610. Then, using the second centroid 652 as the origin O, the second pixel coordinate system 650 is transformed into polar coordinates. System 654, such as Figure 16 As illustrated in the diagram. A centroid-based identification system determines which bounding box 640° center has the smallest centroid angle. exist Figure 15 and Figure 16 In the non-restrictive example illustrated, the sixth participant 610F has the smallest centroid angle. And then it was labeled accordingly. Then, based on the label of the centroid, each of the participants 610 was ranked in counter-clockwise order with respect to the second centroid 652, to have the smallest centroid angle. Starting with participant 610, for example, the sixth participant 610F. For example, participants 610 are ranked in the following order: sixth participant 610F, fifth participant 610E, fourth participant 610D, third participant 610C, second participant 610B, and first participant 610A. Therefore, it will be understood that the same system is used to rank participants 610 in both the front image 628 and the left image 648, but the ranking order of participants 610 differs between the front image 628 and the left image 648 due to the different camera angles of the meeting room 600.

[0073] Now for reference Figure 17 The right-side image 658, which is the image of the meeting room 600 captured by the third camera 622, is overlaid on the third pixel coordinate system 660. See also the front image 628 and the left-side image 648 (see...). Figure 13 and Figure 15 The centroid-based labeling system applied the AI ​​head detector model to the right-hand image 658, which in turn generated head bounding boxes 640 for participant 610, and calculated a third centroid 662 based on the center of the bounding boxes 640. Furthermore, ID tags ID_N were arbitrarily applied to each of the participants 610 using a specific regime as discussed above. In some aspects, the ID tags were used in conjunction with... Figure 13 The different orders illustrated are applied to participant 610. Then, using the third centroid 662 as the origin O, the third pixel coordinate system 660 is transformed into polar coordinates. System 664, such as Figure 18 As illustrated in the diagram. A centroid-based identification system determines which bounding box 640° center has the smallest centroid angle. exist Figure 17 and Figure 18 In the non-limiting example illustrated, the third participant 610C has the smallest centroid angle. And then it was labeled accordingly. Then, based on the label of the centroid, each of the participants 610 was ranked in counter-clockwise order with respect to the second centroid 652, to have the smallest centroid angle. Participant 610 is listed first, for example, third participant 610C. For instance, participants 610 are ranked in the following order: third participant 610C, second participant 610B, first participant 610A, sixth participant 610F, fifth participant 610E, and fourth participant 610D. In some respects, the same ranking system is used for participants 610 in the front image 628, left image 648, and right image 658, but due to the different camera angles of the meeting room 600, the ranking order of participants 610 differs between images 628, 648, and 658.

[0074] Therefore, it will be understood that the centroid-based identification system identifies and stores the ranking of participant 610 for each of images 628, 648, and 658. In some aspects, the centroid-based identification system identifies reference persons in images 628, 648, and 658 and rearranges the rankings based on them. To this end, Figure 19 The illustration shows a front view 628 of the meeting room 600 captured by the first camera 618. Figure 20 The illustration shows a left-side image 648 of the meeting room 600 captured by the second camera 620, and... Figure 21 The illustration shows a right-side image 658 of the meeting room 600 captured by the third camera 622. Now refer to... Figures 19-21 The centroid-based identification system uses the known pixel coordinates of participants (610) to identify reference persons. It is envisioned that the specific method for determining reference persons could depend on the geometry of the room, meaning the method for determining reference persons could be modified to best suit a particular meeting room. Figures 19-21 In the non-limiting example illustrated, the pixel coordinates {x} of cameras 618, 620, and 622 are... i y i The positions are known to each other. By manipulating the pixel coordinates {x} of participant 610 i y i The centroid-based identification system makes it possible to identify the same participant across each of images 628, 648, and 658. In some respects, the centroid-based identification system involves applying the equations in Table 1 to identify reference persons in images 628, 648, and 658: Table 1 Front view - bottom right <![CDATA[argmax(x i +and i )]]> Left view - top right <![CDATA[argmin(x i -and i )]]> Right view - bottom left <![CDATA[argmax(x i -and i )]]> .

[0075] In the table above, argmax(x) i +y i ) was used to identify the participant furthest from the bottom right corner 664 of the front-facing image 628, argmin(x i -y i ) is used to identify the participant furthest from the top right corner 668 of the left-hand image 648, and argmax(x) i -y iThe centroid-based identification system is used to identify the participant furthest from the bottom left corner 670 of the right image 658. As a result, the centroid-based identification system identifies the same participant (e.g., the sixth participant 610F) as the reference person. However, it will be readily understood that different participants can be identified by adjusting the equations in the table above. For example, participants in the bottom left corner 672 of the front image 628, the bottom right corner 674 of the left image 648, and the top left corner 676 of the right image 658, such as the first participant 610A, can be identified as the reference person 680.

[0076] Once reference person 680 has been identified, the centroid-based identification system rearranges the ID labels ID_N in each image, i.e., re-ranks participants 610. Specifically, the identification system re-ranks participants 610 starting with reference person 680, i.e., the sixth participant 610F, and continues counterclockwise about the centroids 642, 652, and 662 of each of images 628, 648, and 658. For example, the identification system appends the new ranking to the original ID label ranking, or the identification system stores the new ranking as a separate ranking, as depicted in Table 2 below: Table 2

[0077] Therefore, it will be understood that the ID tag ID_N is re-ranked clockwise starting with the common reference person 680, aligning the ID tag ID_N across images 628, 648, and 658. In other words, the step of re-ranking the ID tag ID_N based on reference person 680 allows the centroid-based identification system to identify each re-identifying participant 610 across images 628, 648, and 658. Thus, once the rankings are aligned, the centroid-based identification system will rank them in relation to each other. For example: it is known that the sixth participant 610F corresponds to ID_5 in the front image 628, ID_4 in the left image 648, and ID_1 in the right image 658; it is known that the fifth participant 610E corresponds to ID_4 in the front image 628, ID_5 in the left image 648, and ID_4 in the right image 658; it is known that the fourth participant 610D corresponds to ID_3 in the front image 628, ID_1 in the left image 648, and ID_2 in the right image 658; and so on.

[0078] In some aspects, the centroid-based labeling system reassigns ID tags ID_N from images 648 and 658 captured by the left camera 620 and the right camera 622 to match the ID tag ID_N in the front image 628 captured by the front camera 618 (e.g., the primary camera). In other examples, the centroid-based labeling system creates a relabeled ID tag ReID_N for the left image 648 and the right image 658 (e.g., the secondary image) that is identical to the original ID tag ID_N in the front image 628 (e.g., the primary image). Figures 22-24 Images 628, 648, and 658 are illustrated respectively, in which the re-identification label ReID_N is aligned with the original ID label ID_N in the preceding image 628. Figure 23 and Figure 24 In the left image 648 and the right image 658, respectively, the first participant 610A is assigned a re-identified ID label ReID_0, the second participant 610B is assigned a re-identified ID label ReID_1, the third participant 610C is assigned a re-identified ID label ReID_2, and so on. In this way, each of the participants 610 in each of the images 628, 648, and 658 is identified and re-identified by a centroid-based identification system.

[0079] Therefore, the centroid-based identification system disclosed herein can re-identify video conference participants across different camera views. Correspondingly, the centroid-based identification system prevents multiple views of the same participant from being transmitted to the remote end of the video conference, which in turn reduces clutter in video conferences. In some respects, the centroid-based identification system disclosed herein is particularly advantageous in crowded meeting rooms and / or when there is a small distance between participants in the meeting room. Furthermore, it is envisioned that… Figures 12A-24 The illustration shows a non-limiting example of a centroid-based signage system, which can be applied to a wide variety of different meeting rooms and is compatible with a wide variety of different camera arrangements.

[0080] Figure 25The illustration depicts a method 700 for implementing the centroid-based identification system discussed above. At step 702, images of the location are captured using a primary camera and one or more secondary cameras. As discussed above, the primary camera may be positioned at the front of the meeting room, and the secondary cameras (one or more) may be positioned on the left and right sides of the meeting room, respectively. In some respects, the primary camera is a camera that communicates with and / or is connected to a monitor and / or codec, including a memory and a processor, which will be discussed in more detail below. At step 704, human heads in the images are detected using an AI head detection model, as described above. For example, the AI ​​head detection model is applied to each image captured by the camera to identify a head bounding box with specified room and / or pixel coordinates for each detected human head. The AI ​​head detection model also determines the pixel coordinates of the center of each bounding box, as will be discussed in more detail below. At step 706, the centroid is determined based on the pixel coordinates of the bounding boxes, or more specifically, based on the pixel coordinates of the center of each bounding box. At step 708, the pixel coordinates are transformed to polar coordinates, and ID labels are assigned to each bounding box. Specifically, the pixel coordinates of each bounding box are transformed to polar coordinates using the centroid as the origin. At step 710, the ID labels are ranked in counter-clockwise order, starting with the ID label associated with the bounding box having the smallest polar angle with respect to the centroid. In some aspects, this ranking is stored in a centroid-based labeling system. At step 712, a reference human head is identified based on specific pixel coordinates of a reference human head (i.e., a reference bounding box). In some aspects, the reference human head corresponds to the same participant in each image captured by the primary and secondary cameras. At step 714, the ID labels are rearranged in counter-clockwise order, starting with the reference human head, thereby aligning the ID labels across the images captured by the primary and secondary cameras. In this way, the centroid-based labeling system associates ID labels across the images captured by the primary and secondary cameras, which in turn allows the system to track participants across images. In some aspects, the centroid-based identification system repeats each step of method 700 during normal operation, meaning that the centroid-based identification continuously re-identifies the participants as the primary and secondary cameras capture images of the location. Generally, method 500 can be performed in real-time or near real-time. For example, in some aspects, steps 702, 704, 706, 708, 710, 712, and 714 of method 700 are repeated after a time period has elapsed, such as, for example, at least every 30 seconds, or at least every 15 seconds, or at least every 10 seconds, or at least every 5 seconds, or at least every 3 seconds, or at least every second, or at least every 0.5 seconds.

[0081] Figure 26 The illustration shows an example camera 848, which can be similar to the front-side cameras 48A, Cam0, and Cam. f618 (see) Figure 1 , Figure 10 , Figure 11 (Figure 12); and an example microphone array 850, which is similar to microphone array 50 (see Figure 12). Figure 1 Camera 848 has a housing 852 with a centrally located lens 854 for operation with imager 856. A series of openings 858, such as five openings 858, are provided as ports for microphones in microphone array 850. In some examples, the microphone openings 858 form a horizontal line 860 to provide desired angle determination for the SSL process, as discussed above. Figure 26 This is an example illustration of camera 848, although many other configurations are possible, with varying lens and microphone configurations. Additionally, in some examples, aspects of this technology, including computerized implementations of methods according to this technology, can be implemented as systems, methods, apparatus, or articles of art that use standard programming or engineering techniques to generate software, firmware, hardware, machine-readable instructions, or any combination thereof to control processor devices (e.g., serial or parallel general-purpose or special-purpose processor chips, single-core or multi-core chips, microprocessors, field-programmable gate arrays, control units, arithmetic logic units, and processor registers, etc., any various combinations thereof), computers (e.g., processor devices operatively coupled to memory), or other electronic operating controllers to implement the aspects detailed herein. Thus, for example, the technology can be implemented as a set of instructions tangibly embodied on a non-transitory computer-readable medium, such that a processor device can implement said instructions based on instructions read from the computer-readable medium. Some examples of this technology may include (or utilize) control devices consistent with the discussion below, such as, for example, automated devices, special-purpose or general-purpose computers including various computer hardware, software, firmware, etc. As a specific example, a control device may include a processor, microcontroller, field-programmable gate array, programmable logic controller, logic gate, etc., as well as other suitable components for implementing appropriate functionality (e.g., memory, communication system, power source, user interface, and other inputs).

[0082] The above description assumes a front-side camera 48A and a microphone array 50 (see...). Figure 1 The axes are juxtaposed. If the axes are displaced, the displacement is used to transfer a defined sound angle from the microphone array 850 to the camera reference frame.

[0083] As described above, some approaches include using AI human head detector models to detect the location of individual meeting participants. Now refer to Figure 27The illustration depicts an example process 900 using such an AI human head detector process to determine the coordinates of detected human heads. The AI ​​human head detector process utilizes machine learning, an AI human head detector model 904, to analyze incoming room view video frame images 902 of a meeting room scene to detect and display human heads with corresponding head bounding boxes 906, 908, and 910. In some aspects, the AI ​​human head detector process further identifies and displays the center 912 of the head bounding boxes 906, 908, and 910. As depicted, each incoming room view video frame image 902 can be captured by a front-side camera 48A in a video conferencing system. For example, a first view of a meeting participant is captured by a first camera (not shown) in a first profile image or video frame 902a, a second camera (not shown) captures a second profile image or video frame 902b, and a third camera (not shown) captures a third profile image or video frame 902c. Each incoming room view video frame image 902 can be processed using an on-device AI human head detector model 904, which may be located at the corresponding camera capturing the video frame image. However, in other examples, the AI ​​human head detector model 904 may be located at a remote or centralized location, or at a single camera. Regardless of its location, the AI ​​human head detector model 904 may include multiple processing modules 914, 916, 918, 920 implementing a machine learning model trained to detect or classify human heads from the incoming video frame image and to have a head bounding box with specified image planar coordinates and dimensional information for each detected human head identifier.

[0084] In this example, the AI ​​human head detector model 904 may include a first preprocessing module 914, which applies image preprocessing (such as color conversion, image scaling, image enhancement, image resizing, etc.) to prepare the input video frame image for subsequent AI processing. Furthermore, a second module 916 may include training data parameters and / or model architecture definitions, which can be predefined and used to train and define the human head detection model 904 to accurately detect or classify human heads from the incoming video frame image. In the selected example, the human head detection model module 918 may be implemented as model inference software or a machine learning model, such as a convolutional neural network (CNN) model, specifically trained for video codec operations to detect heads in the input image by generating pixel-wise positions for each detected head and generating corresponding head bounding boxes for framing the detected head. Finally, the AI ​​human head detector model 904 may include a post-processing module 920 that applies image post-processing to the output from the AI ​​human head detector model module 918 to make the processed image suitable for human viewing and understanding. Furthermore, the post-processing module 920 may also reduce the size of the data output generated by the human head detection model module 918, such as by merging or grouping multiple head bounding boxes or frames generated from a single meeting participant, thereby specifying a single head bounding box or frame.

[0085] Based on the results of processing modules 914, 916, 918, and 920, the AI ​​human head detector model 904 can generate output video frame images 902, in which detected human heads are framed using corresponding head bounding boxes 906, 908, and 910. As depicted, the first output video frame image 902a includes head bounding boxes 906a-c superimposed around each detected human head. Furthermore, the second output video frame image 902b includes head bounding boxes 908a-c superimposed around each detected human head, and the third output video frame image 902c includes head bounding boxes 910a and 910b superimposed around each detected human head. The AI ​​human head detector model 904 can use any suitable pixel-based parameters to specify each head bounding box, such as defining the x and y pixel coordinates of the head bounding box or frame in combination with the height and width dimensions of the head bounding box or frame. Furthermore, the AI ​​human head detector model 904 can use any suitable measurement technique to specify the distance metric between the camera position and the position of the detected human head. The AI ​​human head detector model 904 can also calculate a corresponding confidence metric or score for each head bounding box, which quantifies the confidence that the model has detected a human head.

[0086] In some examples disclosed herein, the AI ​​human head detector model 904 can specify all head detections in a data structure that stores the coordinates of each detected human head along with its detection confidence. More specifically, a human head data structure can be generated for a number n human heads as follows: In this example, x i and y i Let Width refer to the image plane coordinates of the i-th detected head, where Width i and Height i This refers to the width and height information of the head bounding box of the i-th detected head. Additionally, the Score... i Within the range [0, 100], the confidence level is expressed as a percentage for the i-th detected head. This data structure can be used as input for various applications, such as framing, tracking, composition, recording, switching, reporting, encoding, etc. In this example data structure, the first detected head is located in an image frame within a head bounding box located at pixel position parameters x1, y1 and extending horizontally to Width1 and vertically downward to Height1. Furthermore, the second detected head is located in an image frame within a head bounding box located at pixel position parameters x2, y2 and extending horizontally to Width2 and vertically downward to Height2, and the n-th detected head is located within a head bounding box located at pixel position parameters x1, y1 and extending horizontally to Width1 and vertically downward to Height2, respectively. n y n Width n and vertically downward extension Height n The image frame within the head bounding box. In some respects, the center of each head bounding box is determined using the following equation:

[0087] This human head data structure can then be used as input to a distance estimation process that takes the {Width, Height} parameters of each head bounding box, by first using one of the Width or Height parameters with a first lookup table, and then if multiple meeting room coordinates {x 房间 y 房间 If this parameter is determined, then another parameter is used as the tiebreaker, and the coordinates of the meeting room {x} are selected from the lookup table. 房间 y 房间 The optimal matching distance is determined by the distance information. Then, the human head data structure itself can be modified to also embed distance information into each head, resulting in a modified human head data structure that looks like this: Where {x 房间1 y 房间1}、{x 房间2 y 房间2}、…、{x 房间n y 房间n Specify Head1, Head2, ..., Head in a two-dimensional coordinate system. n Distance from the camera.

[0088] Figure 28 The illustrations depict various aspects of a codec 1000 according to some examples of this disclosure. As discussed above, the codec 1000 may be a standalone device in a video conferencing system, or it may be incorporated into one or more cameras within the video conferencing system, such as a primary camera. Generally, the codec 100 includes machine-readable instructions to maintain a video call with a video conferencing endpoint according to the methods described herein, receive streams from secondary cameras (and primary cameras, if not integrated with primary cameras), and encode and synthesize the streams for transmission to the endpoint.

[0089] like Figure 28 As shown, codec 1000 may include one or more speakers 1002, although in many cases speakers 1002 are provided in monitor 1004. Codec 1000 may include one or more microphones 1006 connected via bus 1008. Microphones 1006 are connected via analog-to-digital (AID) converter 1010, and speakers 1002 are connected via digital-to-analog (D / A) converter 1012. Codec 1000 also includes processing unit 1014, network interface 1016, flash memory or other non-transitory memory 1018, RAM 1020, and input / output (I / O) universal interface 1022, all coupled via bus 1008. Camera 1024 is connected to I / O universal interface 1022. Microphones 1006 are connected to network interface 1016. HDMI interface 1026 is connected to bus 1008 and external display or monitor 1004. Bus 1008 is illustrative and can be used with any interconnect between components, such as Fast Peripheral Component Interconnect (PCIe) links and switches, Universal Serial Bus (USB) links and hubs, and combinations thereof. Camera 1024 and microphone 1006 can be contained in a housing that includes other components, or they can be external and removable, connected via wired or wireless connections.

[0090] The processing unit 1014 may include a digital signal processor (DSP), a central processing unit (CPU), a graphics processing unit (GPU), and dedicated hardware components such as a neural network accelerator and a hardware codec.

[0091] Flash memory 1018 stores modules with varying functionalities in the form of software and firmware, typically programs or machine-readable instructions for controlling codec 1000. The illustrated modules include video codec 1028, camera control 1030, viewfinder 1032, other video processing 1034, audio codec 1036, audio processing 1038, network operation 1040, user interface 1042, and operating system, as well as various other modules 1044. In some examples, an AI head detector module is included within the flash memory 1018. Furthermore, in some examples, machine-readable instructions may be stored in flash memory 1018 to cause processing unit 1014 to perform any of the methods described above. RAM 1020 is used to store any modules in flash memory 1018 while a module is executing, stores video images from the video stream and audio samples from the audio stream, and can be used for scratchpad operations of processing unit 1014.

[0092] Network interface 1016 enables communication between codec 1000 and other devices, and can be wired, wireless, or a combination thereof. In one example, network interface 1016 is connected to or coupled to the Internet 1046 to communicate with a remote endpoint 1048 in a video conference. In one example, general interface 1022 provides data transmission with a local device (not shown), such as a keyboard, mouse, printer, projector, monitor, external speaker, additional camera, and microphone box.

[0093] In one example, camera 1024 and microphone 1006 capture video and audio respectively in a video conferencing environment and generate video and audio streams or signals that are transmitted to processing unit 1014 via bus 1008. As discussed herein, a “view” or “image” of the captured location may include captured individual frames and / or frames within a video stream. For example, camera 1024 may be instructed to continuously capture specific views of a location, such as images within a video stream, over the duration of a video conference. In one example of this disclosure, processing unit 1014 processes the video and audio using processes stored in modules in flash memory 1018. The processed audio and video streams may be sent to, or received from, remote devices coupled to network interface 1016 and devices coupled to general interface 1022.

[0094] The microphones in the microphone array used for SSL can be used as microphones to provide voice to distant sites, or separate microphones, such as microphone 1006, can be used.

[0095] In view of the above, Figure 29 The illustration depicts a method 1100 according to an example of this disclosure, which re-identifies participants in a video conference and transmits a composite stream of optimal views of the participants to a remote end of the video conference. At step 1102, a first image and a second image of the location are captured using a primary camera and one or more secondary cameras. In some aspects, the primary and secondary cameras communicate and / or connect to a monitor and / or codec including memory and a processing unit, as discussed above. In some aspects, method 1100 is executable via machine-readable instructions stored on the codec and / or executed on the processing unit, meaning that the processor instructs the primary and secondary cameras to capture the first and second images of the location.

[0096] At step 1104, as described above, an AI head detection model is used to detect human heads in the images. For example, the AI ​​head detection model is applied to each image captured by the camera to identify a head bounding box with specified room and / or pixel coordinates for each detected human head. In some aspects, the AI ​​head detection model includes a facial feature recognition model configured to evaluate the detected head based on visible facial features in each image. At step 1106, ID labels are assigned to each bounding box using the AI ​​head detection model. At step 1108, the same detected human heads are found in the first and second images using the ID labels, for example, using one of the methods described above. In some aspects, finding the same detected human heads in step 1108 includes grouping the ID labels in the second image with the associated ID labels in the first image, thus re-identifying the participant across the first and second images. In some examples, the ID labels are grouped based on the distance between the identification labels in the second image and the associated ID labels in the first image. At step 1110, an image with the optimal view for each participant is selected. For example, an image with the best front view of each participant is selected, i.e., the image in which the participant's facial features are most visible, and step 1110 is repeated such that the best view for each participant is selected. At step 1112, the composite stream of the participants' best views is transmitted to the remote end of the video conference. In some respects, only the best view of each participant is visible in the composite stream, meaning that repeated views of the participants are not transmitted to the remote end of the video conference. As discussed above, it is envisioned that method 1100 as a whole (including any of the other methods described above) can be executed within the primary camera and / or the secondary camera, and / or method 1100 is executable via machine-readable instructions stored on the codec and / or executed on the processing unit. Therefore, it will be understood that the methods described herein can be computationally lightweight and can be executed entirely in the primary camera, thus reducing the need for resource-intensive GPUs and / or other dedicated computing machines.

[0097] Specific operations of the methods according to this technology or the systems performing those methods may be schematically represented in the figures or otherwise discussed herein. Unless otherwise specified or limited, representing specific operations in a particular spatial order in the figures may not necessarily require those operations to be performed in a specific order corresponding to the particular spatial order. Correspondingly, specific operations represented in the figures or otherwise disclosed herein may be performed in an order different from that explicitly illustrated or described, as appropriate for a particular example of this technology. Further, in some examples, specific operations may be performed in parallel, including by dedicated parallel processing devices or discrete computing devices that are partially interoperable with a large system.

[0098] The disclosed technology is not limited in its application to the construction details and component arrangements set forth in the following description or illustrated in the following figures. Other examples of the disclosed technology are possible, and the examples described and / or illustrated herein can be practiced or implemented in various ways.

[0099] Multiple hardware and software-based devices and multiple different structural components can be used to implement the disclosed technology. Furthermore, examples of the disclosed technology may include hardware, software, and electronic components or modules, which, for the purposes of discussion, may be illustrated and described as if most components were implemented solely in hardware. However, in one example, the electronic-based aspects of the disclosed technology may be implemented in processor-executable software (e.g., stored on a non-transitory computer-readable medium). Although specific figures illustrate hardware and software residing within a particular device, these depictions are for illustrative purposes. In some examples, the illustrated components may be combined or divided into separate software, firmware, hardware, or combinations thereof. As an example, instead of being located within and executed by a single electronic processor, logic and processing may be distributed across multiple electronic processors. Regardless of their combination or division, hardware and software components may reside on the same computing device or may be distributed across different computing devices connected via a network or other suitable communication links.

[0100] Any suitable non-transitory computer-usable or computer-readable medium may be used. Computer-usable or computer-readable media can be, for example, but not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatuses, or devices. More specific examples (a non-exhaustive list) of computer-readable media will include: portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), portable compact disc read-only memory (CD-ROM), optical storage devices, or magnetic storage devices. In the context of this disclosure, a computer-usable or computer-readable medium can be any medium capable of containing, storing, transmitting, or delivering a program for use by or in connection with an instruction execution system, apparatus, or device.

[0101] As used herein in the context of computer implementation, unless otherwise specified or limited, the terms “component,” “system,” “module,” “block,” and similar terms are intended to cover part or all of a computer-related system comprising hardware, software, combinations of hardware and software, or software in execution. For example, a component can be, but is not limited to, a processor device, a process executed (or executable) by a processor device, an object, an executable program, a thread of execution, a computer program, or a computer. By way of illustration, both an application running on a computer and the computer itself can be components. A component (or system, module, etc.) may reside within a process or thread of execution, may be localized on a single computer, may be distributed across two or more computers or other processor devices, or may be included within another component (or system, module, etc.).

Claims

1. A method for identifying participants in a location and using multiple cameras, the method comprising: Use primary and secondary cameras to capture images of the location; A machine learning human head detector model is applied to an image to identify the coordinates of each human head detected in the image; the centroid of the coordinates of each human head detected in the image and the centroid angle of each human head detected in the image relative to the centroid are determined. Based on the centroid angle of each human head detected in the image, the label of each human head is ranked for each image. Identify a reference human head in a primary image captured by a primary camera; and Align the labels across each image to re-identify each human head.

2. The method of claim 1, wherein ranking the identification labels based on the centroid angle of each human head detected in the image comprises ranking the identification labels of each human head in a counterclockwise order, starting with the human head having the lowest centroid angle.

3. The method of claim 1, wherein applying a machine learning human head detector model to identify the coordinates of each detected human head includes determining the Cartesian coordinates of each human head and subsequently transforming the Cartesian coordinates into polar coordinates.

4. The method according to claim 3, wherein the polar coordinates include the distance from the centroid and the centroid angle.

5. The method of claim 1, wherein the reference human head is positioned furthest from the lower right corner of the primary image, and wherein the reference human head is subsequently identified in a secondary image captured by the secondary camera.

6. The method of claim 5, wherein aligning the identification labels further comprises re-ranking the identification labels in the secondary image in a counter-clockwise order, starting with a reference to a human head.

7. The method of claim 6, further comprising: For each human head detected in the secondary image, a relabeling label is created, which is matched with the label in the primary image.

8. A method for identifying a participant in a location, the method comprising: Images of the location are captured using a primary camera and a secondary camera. The primary camera defines a primary coordinate system, and the secondary camera defines a secondary coordinate system. A machine learning human head detector model is applied to images to identify world coordinate points associated with each human head detected in the image, including world coordinate points in the secondary coordinate system associated with images captured by the secondary camera and world coordinate points in the primary coordinate system associated with images captured by the primary camera. Project the world coordinate points from the secondary coordinate system onto the primary coordinate system; Measure the distance between each point in the world coordinate system; and Identifiers are assigned to groups of world coordinate points that are the closest to each other, and the identifiers identify the first participant.

9. The method of claim 8, wherein projecting world coordinate points includes evaluating scaling, translation, and rotation factors between the primary and secondary coordinate systems.

10. The method of claim 8, wherein projecting the world coordinates further comprises capturing calibration points in the images using a primary camera and a secondary camera, and wherein the calibration points are world coordinates of a single human head detected by a machine learning human head detector model.

11. The method of claim 10, wherein capturing calibration points comprises capturing at least 1,500 frames using a primary camera and a secondary camera.

12. The method of claim 8, wherein the secondary camera comprises a plurality of secondary cameras.

13. The method of claim 8, wherein projecting the world coordinate point further includes determining the world coordinates of the secondary camera relative to the primary camera in the primary coordinate system.

14. A non-transitory computer-readable medium containing instructions that, when executed, cause a processor to: Instruct multiple cameras to capture the first and second images of the location; A machine learning human head detector model is applied to the first image and the second image to detect human heads in the first image and the second image, and the coordinates of each detected human head are identified. Identification labels are assigned to each human head detected in the first and second images; and Each identifier label in the second image is grouped together with its associated identifier label in the first image.

15. The non-transitory computer-readable medium of claim 14, wherein the identifiers in the second image are grouped with the identifiers in the first image based on the distance between each of the identifiers and each other.

Images