Face detection guided sound source localization horizontal angle post-processing for smart camera speaker tracking and framing

Abstract

This application discloses face detection guided sound source localization horizontal angle post-processing for intelligent camera speaker tracking and framing. A video conferencing system includes a camera that acquires image data and a microphone array that acquires audio data. The image data is used in conjunction with sound source localization (SSL) data to localize a speaker depicted in the image data. The audio data is processed by SSL and SSL horizontal angle values are determined that indicate an estimated direction of sound. Columns of pixels in an image are associated with slots. For each SSL horizontal angle value of the audio data that falls within a given slot, a slot count is incremented. A bounding box is determined in the image data that encompasses a face depicted in the image data. A pixel range of the bounding box is determined, such as extending from a leftmost column to a rightmost column. A slot having a highest slot count that also overlaps the pixel range of the bounding box is deemed to contain the speaker.

Description

[0001] priority

[0002] This application claims priority to U.S. Provisional Patent Application No. 63 / 203,435, filed July 22, 2021, entitled “Face Detection Guided SoundSource Localization Pan Angle Post Processing for Smart Camera TalkerTracking and Framing,” which is incorporated herein by reference in its entirety. Background Technology

[0003] In video conferencing, it's desirable for the camera to focus on the speaker, rather than maintaining a panoramic view of the room. To aid in this focus, acoustic source localization (SSL) is used to provide the direction or angle from which the camera can focus to capture the speaker. Various methods can be used to determine the SSL angle. However, acoustic effects in the conference room (such as echo) and the distance between the speaker and the microphone array can cause jitter in the determined SSL angle. Such jitter makes framing the speaker more challenging. Attached Figure Description

[0004] For illustration, certain examples described in this disclosure are shown in the accompanying drawings. In the drawings, the same numbers consistently denote the same elements. The full scope of the invention disclosed herein is not limited to the precise arrangements, dimensions, and devices shown. In the drawings:

[0005] Figure 1 This is a top-view illustration of a conference room, including cameras, microphone arrays, monitors, tables, and chairs, based on some implementation methods.

[0006] Figure 2 It depends on some implementation methods. Figure 1 A top-view illustration of a conference room, showing one of several individuals speaking.

[0007] Figure 3 It depends on some implementation methods. Figure 2 A camera view illustration of the conference room.

[0008] Figure 4 It depends on some implementation methods. Figure 3 The camera view diagram shows the pixel boundaries.

[0009] Figure 5 It is a diagram illustrating the division of operations between the codec and the camera with a microphone array, based on some implementation methods.

[0010] Figure 6 It is a flowchart for determining the final SSL angle using facial recognition data based on some implementation methods.

[0011] Figure 7 It is a block diagram of a codec based on some implementation methods.

[0012] Figure 8 It is a block diagram of a camera based on some implementation methods.

[0013] Figure 9 It is a block diagram of a processor based on some implementation methods.

[0014] Figure 10 It is an illustration of the front view of a camera based on some implementation methods. Detailed Implementation

[0015] Sound source localization (SSL) is the process of processing audio data samples acquired by a microphone array to determine the source location (e.g., SSL horizontal angle) of the sound detected by the microphone array. Conference room acoustics and the distance between the speaker and the microphone array can cause jitter in the SSL horizontal angle data. To provide more stable SSL horizontal angle data, it can be combined with image data. More stable SSL horizontal angle data can then be used to improve the framing of the speaker during video conferencing.

[0016] A video conferencing system may include one or more cameras that acquire image data and a microphone array that acquires audio data. The image data may be processed by a trained machine learning system to detect one or more features of individuals in the meeting room. These features may include the individual's face, head, body, etc. For example, in one embodiment, the trained machine learning system may output data describing the face of an individual in the meeting room, wherein the face is depicted within a bounding box.

[0017] The microphone array acquires audio samples at a predetermined rate. SSL technology is used to process the audio samples to determine estimated SSL horizontal angles corresponding to one or more of the horizontal or vertical positions associated with the audio samples. Sighting based on these raw SSL horizontal angles can be unreliable due to various factors such as echo, distance between the speaker and the microphone array, etc. Combining SSL horizontal angle data and image data improves the determination of the final direction associated with sound, such as the voice of one or more speakers tracked during a conference call.

[0018] To combine image data with SSL data, a relationship is established between the horizontal angle values ​​associated with the audio samples and the pixels in the image. Bins are generated to form this relationship. One or more columns or rows of pixels in the image are associated with a bin. The same bin is also associated with one or more SSL horizontal angle values. For example, the pixel width of the image can be divided by the range of SSL horizontal angle values ​​to determine the number of bins to use.

[0019] The SSL horizontal angles determined from the audio samples can be placed in corresponding slots, such that the slot count increases for each SSL horizontal angle value placed in a slot. The processed image data provides bounding boxes, which depict certain features within the image, such as the user's face. A pixel range associated with each bounding box is defined. For example, the pixel range of the first bounding box may include a range from a specific left pixel column to a specific right pixel column.

[0020] The count of slot entries is recorded for the pixel range associated with the bounding box. In one example, the total count or sum of slot entries associated with each bounding box is determined, and the bounding box with the highest slot entry count is considered the speaker or source of the sound detected by the microphone array. In another embodiment, other operations can be performed to determine the source of the sound using the bounding boxes, such as by considering the average number of slot entries for the bounding boxes, the bounding box with the highest total number of slot entries for a given pixel, etc.

[0021] Once the bounding box with the highest slot entry count is identified, output data is generated to reflect the final source location of the sound. In one embodiment, the output data may represent the bounding box, a face, or a user. In another embodiment, the output data may represent a reference pixel associated with the bounding box with the highest slot entry count, such as the center pixel of the bounding box. The output data is then used to determine the final source location of the sound, which may represent the final SSL horizontal angle value. This final source location data can be used to provide framing of the speaker during a video conference.

[0022] The final SSL horizontal angle value used for framing is improved by combining bounding box recognition data with SSL horizontal angle data to remove jitter caused by room acoustics and variations in distance from the speaker. Specifically, by narrowing down the options for the SSL horizontal angle value to those matching faces or bounding boxes within the meeting room, the final SSL angle value more accurately represents the actual speaker during the video conference and provides better overall framing.

[0023] Now for reference Figure 1The diagram illustrates a conference room C configured for use in a video conference. Conference room C includes a conference table 10 and a series of chairs 12. A camera 1116 is positioned in conference room C to observe individuals seated in the various chairs 12. A monitor or television 1120 is set up to display one or more remote meeting locations and typically provides speaker output. The camera 1116 has a field of view (FoV) and an axis or centerline (CL). The camera 1116 includes a microphone array 1214 for SSL (Security, Service, and Validation). In some examples, the microphone array 1214 is centered on the centerline of the camera 1116 for ease of operation. Figure 1 In the layout, camera 1116 is positioned such that the CL of camera 1116 is centered along the length of conference table 10.

[0024] Now go to Figure 2 Four individuals, 1, 2, 3, and 4, are sitting in chairs 12. Individual 3 is speaking, as indicated by the shadow of individual 3.

[0025] Figure 3 A camera view 300 of conference room C is shown. Bounding boxes are shown for each individual 1 through 4. Bounding box 410 surrounds the face of individual 1, bounding box 404 surrounds the face of individual 2, bounding box 406 surrounds the face of individual 3, and bounding box 408 surrounds the face of individual 4. In some examples, the bounding boxes are formed by analyzing the video signal captured by camera 1116 using a neural network. A trained machine learning system can analyze video signals or images to detect features of the users depicted therein. In other examples, a trained machine learning system can detect features of the users, such as the face, head, or body.

[0026] Figure 4 The diagram shows the area covered by dashed lines. Figure 3 View 400 of camera view 300, the dashed lines representing a selected column of pixels of a set of pixel count widths of a conference room image captured by camera 1116. For clarity, view 400 shows the selected column of pixels, where it should be understood that the image may consist of more columns of pixels, such as 1280 columns of pixels in a 720p image.

[0027] The SSL horizontal angle values ​​corresponding to the pixel values ​​shown are shown, referenced from the left side of the image and from the center of the image. For example, the SSL horizontal angle values ​​can be within a range of 120 degrees from the left or within a range of 60 degrees in either direction from the center.

[0028] Slot 450 is used to correlate pixels in an image with SSL horizontal angle values. Specifically, slot 450 is associated with a single pixel or a range of pixels in the image. As shown, an image can have 1280 pixels. Slot 450 can also be associated with a specific SSL horizontal angle or a range of SSL horizontal angle values. As shown, the SSL horizontal angle can range from 0 degrees to 120 degrees, as illustrated from left to right, or it can range from -60 degrees at the far left to 0 degrees at the center and 60 degrees at the far right. In other examples, the SSL horizontal angle values ​​can be represented using different axes, such as when the SSL horizontal angle values ​​represent vertical rows.

[0029] Once defined, slot 450 is used to correlate a specific pixel and a specific SSL horizontal angle value with a similar location in a meeting room. In some examples, the number of slots used can match the number of pixels used. In other examples, the number of slots can be determined by dividing the total number of pixels by the range of SSL horizontal angles. Once established, slot 450 can help correlate the position of a user's face within the bounding box with possible SSL horizontal angle values ​​that could represent the actual speaker.

[0030] In a preferred example, the processing of audio and video and the selection of the desired camera are separated between camera 1116 and codec 1100. (Reference) Figure 5 In step 1002, camera 1116 performs SSL based on the sound received at microphone array 1214 and provides raw SSL direction information, typically a horizontal angle value. In some examples, the determined SSL direction limit is the same as the FOV of camera 1116. In step 1003, the camera image or video is input to a face detection module for processing to detect faces. This is preferably done using a neural network to provide a series of bounding boxes, one for each face. There are many different neural networks to perform face detection and provide bounding box outputs. The SSL direction information from step 1002 is combined with the bounding boxes provided by step 1003 to provide the final SSL direction or horizontal angle value. Step 1004 provides the SSL direction information and bounding boxes to codec 1100. Figure 6 Step 1004 is described in detail. The video stream from camera 1116 is also provided to codec 1100.

[0031] It should be understood that SSL determination and face detection are performed only periodically, not for each video frame. In some examples, face detection is performed once every second to once every five seconds. This is satisfactory because changes in an individual's position are not much faster than those time intervals. In some examples, SSL determination is performed every 80 milliseconds (msec) to 200 msec, resulting in multiple SSL determinations being performed for each face detection.

[0032] In step 1006, codec 1100 receives final SSL orientation information and bounding box from camera 1116. The final SSL orientation information and bounding box are provided to framing step 1012. The framing video from camera 1116 and the video from microphones 1114A and 1114B connected to codec 1100 are then combined. Figure 7 The audio (as shown in the image) is provided to the remote end.

[0033] Figure 6 This is a flowchart illustrating the formation of final SSL direction information from raw SSL direction information and bounding boxes. At step 602, slots are created. For this, the pixel width of the camera image (e.g., 1280 for a 720p image) is divided by the SSL horizontal angle to correlate the received raw SSL angle value with the appropriate pixel value. A slot, such as SSLHistory[N]

[1280] , is created for each pixel, where N is the number of samples and 1280 is the pixel width. In other embodiments, slots can be determined alternatively, such as by setting the number of slots equal to the number of pixels or the number of horizontal angle values.

[0034] At step 604, a sample time period is started. Various sample time periods can be used, but an example is 2 seconds, which matches the face detection time period and provides 20 raw SSL angle sample values ​​obtained with SSL determination every 100 ms. At step 606, the raw SSL horizontal angle values ​​are received and correlated with the appropriate or corresponding slot. A slot entry is created in the appropriate slot SSLHistory[N]

[1280] to represent the raw SSL horizontal angle value. At step 608, it is determined whether the sample time period is complete. If the sample time period is not complete, the process returns to step 606 to process the next raw SSL angle value.

[0035] If the sampling time period is complete, a reference pixel value for each bounding box is determined at 610. In one embodiment, the reference pixel may represent a center pixel value, such as FaceCenterX[n], where n is zero based on the nth face. In other embodiments, the reference pixel may represent another pixel value, such as the top-left or top-right corner pixel of the bounding box, the bottom-left or bottom-right corner pixel of the bounding box, etc. The extent of each bounding box is then determined relative to the reference pixel of the bounding box. Figure 6 Three alternative methods for determining this extent by performing steps 612 to 616, 640, or 650 are shown. In some cases, the extent of each bounding box can be determined using the same set of steps (selected from 612 to 616, 640, or 650), or different steps can be used to determine different bounding boxes.

[0036] At step 612, for each bounding box, it is determined whether the bounding box width is less than the minimum pixel value. If so, the bounding box width value of that bounding box is set to the minimum pixel value at 614. After step 614, or if the bounding box width exceeds the minimum width in step 612, the width of the bounding box is multiplied by a multiplier or factor at 616. The multiplier or factor can be any integer predetermined to provide a range of pixel values, such as 4, to check the original SSL value count for the left and right pixel values, such as FaceWiderX[n][L, R].

[0037] An alternative method for determining the extent of the bounding boxes is step 640. At 640, the extent of each bounding box can be determined based on a predetermined range. For example, a range of 40 to 50 pixels can be predetermined and specified for the bounding boxes. In this case, the range of pixels is set to 40 to 50 pixels, where the center pixel of this range coincides with the reference pixel of each bounding box.

[0038] In another alternative, step 650 can be used to determine the extent of the bounding boxes. At 650, the extent of the bounding boxes is determined based on the distance between two or more bounding boxes. For example, in the case where two users represented by two bounding boxes are relatively close to each other, the distance between the two bounding boxes can be determined. In such cases, the extents of the two bounding boxes can be set to smaller extents to avoid overlap of the bounding box extents, which could complicate the determination of the source of the sound of one of the two users. In other cases, if three or more bounding boxes are detected and only two of these bounding boxes are close to each other, steps 612 to 616 or 640 can be used to determine the extent of the bounding boxes farther away from the other two, and the extent of the two bounding boxes close to each other can be determined by the distance between the two user bounding boxes, as in 650. Alternatively, the extent of each bounding box can be calculated in the same way for all detected bounding boxes. In yet another case, an initial determination can be made to see if any bounding boxes are close to each other, and if so, step 650 is used to determine the extent of all bounding boxes. If the bounding boxes are not close to each other, the range can be set to the default value using 612 to 614 or 640.

[0039] After the extent of the bounding box is determined (using steps 612 to 616, 640, or 650), the process continues to step 618. At step 618, slot entries for each pixel within the wider bounding box are counted to provide a total raw SSL position count for the expanded bounding box region, such as FaceSSLHits[n]. For example, a sum can be recorded for all slot entries within the extent of each bounding box. The sum of slot entries for each bounding box can be calculated and compared with each other. In other embodiments, an average number of slot entries for each bounding box can be calculated instead of a sum and compared. In yet another example, different operations can be performed using the slot entries, such as finding the median or modulus of the slot entries within the bounding box.

[0040] At step 620, a bounding box with the highest slot entry count is determined. The highest slot entry count can be stored in a location such as FaceSSLHitsMax. At step 622, this highest slot entry count is compared with a threshold. In some embodiments, the threshold can be set to 5 or other values.

[0041] By performing step 622, SSL level angle values ​​based on random noise or other factors that do not indicate the speaker can be filtered out. If the highest slot entry count is not greater than a threshold, the operation returns to step 604 to begin a new sampling period. Conversely, if the highest count is greater than the threshold, the process continues to step 624. It is worth noting that in some embodiments, step 622 may be omitted.

[0042] At step 624, output data indicating the bounding box with the highest slot entry count is determined. In one embodiment, the output data may specify the bounding box with the highest slot entry count and provide it to codec 1100. In other embodiments, the output data may specify the pixel position associated with the bounding box with the highest slot entry count. For example, a reference pixel value (such as a center pixel value) of the bounding box with the highest slot entry count may be provided as output data sent to codec 1100. This reference pixel value may be considered as the final SSL direction. This final SSL direction represents the original SSL horizontal angle value that coincides with the highest slot count, which is associated with the extent of the bounding box representing the actual user who is speaking. After step 624, the operation returns to step 604.

[0043] Figure 7Aspects of a codec 1100 according to an example of this disclosure are shown. Codec 1100 may include a speaker 1122 and a microphone 1114A, which are interfaced with a bus 1115 via an interface. However, in many cases, the speaker 1122 is located in a monitor 1120, the microphone 1114A is interfaced with the bus via an analog-to-digital (A / D) converter 1112, and the speaker 1122 is interfaced with the bus via a digital-to-analog (D / A) converter 1113. Codec 1100 also includes a processing unit 1102, a network interface 1108, flash memory 1104, RAM 1105, and a general-purpose input / output (I / O) interface 1110, all of which are connected via bus 1115. A camera 1116 is shown connected to the I / O interface 1110. Microphone 1114B is connected to the network interface 1108. An HDMI interface 1118 is connected to bus 1115 and to an external display or monitor 1120. Bus 1115 is illustrative and can be used with any interconnect between components, such as peripheral component interconnect high-speed (PCIe) links and switches, universal serial bus (USB) links and hubs, or combinations thereof. Camera 1116 and microphones 1114A and 1114B can be housed in a housing containing other components, or can be externally and removably connected via wired or wireless connections.

[0044] Processing unit 1102 may include any desired combination of digital signal processor (DSP), central processing unit (CPU), graphics processing unit (GPU), dedicated hardware components (such as neural network accelerators and hardware codecs), etc.

[0045] Flash memory 1104 stores various functional modules or general-purpose programs in software and firmware for controlling codec 1100. The modules shown include video codec 1150, camera control 1152, viewfinder 1154, other video processing 1156, audio codec 1158, audio processing 1160, network operation 1166, user interface 1168, and operating system and various other modules 1170. RAM 1105 is used to store any of the modules in flash memory 1104 when the modules are executed, storing video images from the video stream and audio samples from the audio stream, and can be used for temporary storage operations of processing unit 1102.

[0046] Network interface 1108 enables communication between codec 1100 and other devices, and can be wired, wireless, or a combination thereof. In one example, network interface 1108 connects to or is coupled to the Internet 1130 to communicate with a remote endpoint 1140 in a video conference. In one or more examples, general interface 1110 provides data transmission to local devices such as keyboards, mice, printers, projectors, monitors, external speakers, attached cameras, and microphone pods.

[0047] In one example, camera 1116 and microphone 1114 capture video and audio respectively in a video conferencing environment, and generate video and audio streams or signals that are transmitted to processing unit 1102 via bus 1115. In at least one example of this disclosure, processing unit 1102 processes the video and audio using algorithms stored in modules in flash memory 1104. The processed audio and video streams can be sent to and received from remote devices connected to network interface 1108 and devices connected to general interface 1110. This is an example of the configuration of codec 1100.

[0048] Figure 8 Aspects of a camera 1200 according to an example of this disclosure are shown, which may be a camera 1116. Camera 1200 includes an imager or sensor 1216 and a microphone array 1214, which are interfaced with a bus 1215 via an interface. The microphone array 1214 is interfaced with the bus via an analog-to-digital (A / D) converter 1212, and the imager 1216 is interfaced with the bus via an imager interface 1218. Camera 1200 also includes a processing unit 1202, flash memory 1204, RAM 1205, and a universal input / output interface 1210, all interconnected via bus 1215. Bus 1215 is illustrative and any interconnection between components can be used, such as peripheral component interconnect high-speed (PCIe) links and switches, universal serial bus (USB) links and hubs, and combinations thereof. Codec 1100 preferably uses a USB interface to connect to I / O interface 1210.

[0049] The processing unit 1202 may include any desired combination of digital signal processors (DSPs), central processing units (CPUs), graphics processing units (GPUs), and dedicated hardware components such as neural network accelerators and hardware codecs.

[0050] Flash memory 1204 stores various functional modules or general-purpose programs in software and firmware for controlling camera 1200. The modules shown include camera control 1252, face and body detection 1254, sound source localization 1260, neural network module 1262, and final SSL orientation 1264, as shown in... Figure 6 The RAM 1205 is used to store any of the modules in the flash memory 1204 when the module is executing, to store video images of video streams and audio samples of audio streams, and can be used for temporary storage operations of the processing unit 1202.

[0051] In the second configuration, the camera is simplified, and codec capabilities and processing are increased. In this second configuration, the neural network operates on the codec instead of the camera. In the first configuration, the camera feeds its raw video stream to the codec, which then uses the neural network to analyze the raw video stream to discover bounding boxes, pose estimates, and keypoints formed by the camera. In the second configuration, the camera still performs the raw SSL and feeds the raw SSL position information to the codec, but the final SSL angle determination is performed within the codec. In the third configuration, SSL is also performed by the codec, where the camera feeds audio streams from each microphone. This third configuration allows for the simplest camera, but at the cost of further increased processing requirements from the codec. The second configuration is a compromise between the first and third configurations, requiring less processing from the codec but more processing in the camera. The use of a particular configuration depends on the component cost of the required performance level.

[0052] It is well known that for video conferencing endpoints and other devices used in other ways, there are other configurations with different components and component arrangements.

[0053] Figure 9This is a block diagram of an exemplary system-on-chip (SoC) 1300 that can be used as processing units 1102 or 1202. A series of more powerful microprocessors 1302 (such as ARM A72 or A53 cores) form the main general-purpose processing block of the SoC 1300, while a more powerful digital signal processor (DSP) 1304 and several less powerful DSPs 1305 provide dedicated computing capabilities. A simpler processor 1306 (such as an ARM R5F core) provides general-purpose control capabilities within the SoC 1300. The more powerful microprocessor 1302, the more powerful DSP 1304, the less powerful DSP 1305, and the simpler processor 1306 each include various data and instruction caches (such as L1I, L1D, and L2D) to improve the speed of operation. A high-speed interconnect 1308 connects the microprocessor 1302, the more powerful DSP 1304, the simpler DSP 1305, and the processor 1306 to various other components within the SoC 1300. For example, a multi-core shared memory controller 1310, including onboard memory or SRAM 1312, is connected to high-speed interconnect 1308 to act as onboard SRAM for SoC 1300. A DDR (Double Data Rate) memory controller system 1314 is connected to high-speed interconnect 1308 and acts as an external interface to external DRAM memory. RAM 1105 or 1205 is formed by SRAM 1312 and external DRAM memory. Video acceleration module 1316 and Radar Processing Accelerator (PAC) module 1318 are similarly connected to high-speed interconnect 1308. Neural network acceleration module 1317 is provided with hardware acceleration for neural network operations. Visual Processing Accelerator (VPACC) module 1320 and Depth and Motion PAC (DMPAC) module 1322 are also connected to high-speed interconnect 1308.

[0054] Graphics acceleration module 1324 is connected to high-speed interconnect 1308. Display subsystem 1326 is connected to high-speed interconnect 1308 to allow operation and connection to various video monitors. System service block 1332 is provided for normal SoC 1300 operation; this system service block includes items such as a direct memory access (DMA) controller, memory management unit, general-purpose I / O, mailbox, etc. Serial connection module 1334 is connected to high-speed interconnect 1308 and includes modules as is normal in the SoC. Vehicle connectivity module 1336 provides interconnection for external communication interfaces such as PCIe block 1338, USB block 1340, and Ethernet switch 1342. Capture / MIPI module 1344 includes a four-channel CSI-2 compliant transmitter block and a four-channel CSI-2 receiver module and hub.

[0055] MCU Island 1360 is provided as an auxiliary subsystem and handles the operation of the integrated SoC 1300 when other components are powered down to conserve energy. MCU ARM processor 1362 (such as one or more ARM R5F cores) operates as a master device and is coupled to high-speed interconnect 1308 via isolation interface 1361. MCU general-purpose I / O (GPIO) block 1364 operates as a slave device. MCU RAM 1366 is provided to serve as local memory for MCU ARM processor 1362. CAN bus block 1368 (i.e., an additional external communication interface) is connected to allow operation with a standard CAN bus environment in a vehicle. Ethernet MAC (Media Access Control) block 1370 is provided for further connectivity. External memory (generally non-volatile memory (NVM), such as flash memory 1104) is connected to MCU ARM processor 1362 via external memory interface 1369 to store instructions loaded into various other memories for execution by various appropriate processors. The MCU ARM processor 1362 operates as a security processor, monitoring the operation of the SoC 1300 to ensure proper operation of the SoC 1300.

[0056] It should be understood that this is one example of a SoC provided for illustration, and many other SoC examples with different numbers of processors, DSPs, accelerators, etc., are possible.

[0057] Figure 10 A front view of a camera 1200 (such as camera 1116) is provided. Camera 1200 has a housing 1402 in which a lens 1404 is centrally located for operation with an imager 1216. A series of five openings 1406 are provided as ports for microphones in a microphone array 1214. It should be noted that the microphone openings 1406 form a horizontal line to provide the desired angle determination to the SSL algorithm. In other embodiments, the microphone openings 1406 may form a vertical line, or a combination of horizontal and vertical lines. This is an exemplary illustration of camera 1200, and many other configurations with different lens and microphone configurations are possible.

[0058] Although the above description uses a conference room as an exemplary environment, the environment can be any setup. It should be understood that slots can be provided for groups of horizontal pixels, rather than for each horizontal pixel as described above, depending on the number of pixels in the camera image and the desired resolution of the final SSL orientation value. While the bounding box is described above as representing the location of an individual's face, other parameters providing the face location can be used. Although the face is used to determine an individual's location, other body parameters, such as upper limbs, can be used instead or in combination with the face.

[0059] By incorporating a neural network and using raw SSL angle information, the facial positions of individuals within the camera's field of view are determined. The positions of speakers within a group of individuals are provided with an SSL position centered as the bounding box of the face. The use of the bounding box center eliminates jitter based on SSL position values ​​derived from room acoustics or speaker distance from the microphone array. This jitter removal allows for improved speaker framing.

[0060] The various examples described are provided by way of illustration and should not be construed as limiting the scope of this disclosure. Various modifications and alterations may be made to the principles and examples described herein without departing from the scope of this disclosure and without departing from the appended claims.

[0061] The following are further examples of this application:

[0062] 1. A method comprising:

[0063] Images are acquired using a camera (1116), wherein the images include pixels arranged in rows and columns;

[0064] Determine one or more bounding boxes (1003) corresponding to one or more faces depicted in the image;

[0065] Determine a plurality of slots (602) associated with the image, wherein each of the plurality of slots is associated with one or more of the columns;

[0066] A microphone array (1214) is used to acquire multiple audio samples (1002);

[0067] Determine the source localization (SSL) horizontal angle value (1002) associated with one or more of the plurality of audio samples;

[0068] For the corresponding SSL horizontal angle value (1002), determine one or more corresponding slots among the plurality of slots;

[0069] For each of the one or more corresponding slots associated with each corresponding SSL horizontal angle value, increment the slot entry count (606);

[0070] A first range (616, 640, 650) is determined relative to each of the one or more bounding boxes, the first range indicating pixels extending from a first column to a second column;

[0071] For each bounding box, determine the sum of the slot entry counts (618) associated with the first range (616, 640, 650) of each bounding box;

[0072] A first bounding box (620) with the largest sum is determined based on the sum of the slot entry counts (618) of the one or more bounding boxes; and

[0073] Determine the output data (624) indicating the first bounding box.

[0074] 2. The method according to Example 1, wherein determining the plurality of slots (602) associated with the image includes:

[0075] Determine the first pixel width of the image;

[0076] Determine a second range, wherein the second range includes a range of possible SSL horizontal angle values ​​associated with the output from the microphone array during operation; and

[0077] Divide the width of the first pixel by the second range.

[0078] 3. The method described in Example 1 further includes:

[0079] The SSL algorithm (1002) is used to process one or more of the plurality of audio samples (1002) to determine the SSL horizontal angle value indicating the direction of the sound.

[0080] 4. The method described in Example 1 further includes:

[0081] The image is processed using a trained machine learning system (1003) to determine the one or more bounding boxes depicted in the image.

[0082] 5. The method described in Example 1 further includes:

[0083] Determine the first width (610) of the first bounding box;

[0084] Determine that the first width (610) of the first bounding box is less than a first value (612, 614), wherein the first value (612) indicates a minimum pixel value (612); and

[0085] The first range (616) of the first bounding box is determined by multiplying the first value (614) by the second value (616), wherein the second value is a predetermined value (616).

[0086] 6. The method according to Example 1 further includes:

[0087] Determine the first width (610) of the first bounding box;

[0088] Determine that the first width (610) of the first bounding box is greater than a first value (612, 616), where the first value (612) indicates the minimum pixel value (612); and

[0089] The first range (616) of the first bounding box is determined by multiplying the first width (610) by a second value (616), wherein the second value is a predetermined value (616).

[0090] 7. The method described in Example 1 further includes:

[0091] Determine the reference pixel (610) of the first bounding box; and

[0092] The predetermined pixel range (640) is designated as the first range (640) of the first bounding box relative to the reference pixel.

[0093] 8. The method according to Example 1 further includes:

[0094] Determine the distance (650) between two or more bounding boxes; and

[0095] The first extent (650) of at least one of the two or more bounding boxes is determined based on the distance (650) between the two or more bounding boxes.

[0096] 9. The method according to Example 1 further includes:

[0097] Determine that the sum of the slot entry counts (618) of the first bounding box is greater than a threshold (622); and

[0098] The output data (624) is sent in response to the sum of the slot entry counts of the first bounding box being greater than the threshold (622, 624).

[0099] 10. The method according to Example 1 further includes:

[0100] Determine the reference pixel (610) associated with the first bounding box; and

[0101] The output data (624) is sent, wherein the output data (624) indicates the direction associated with the reference pixel (624).

[0102] 11. A system comprising:

[0103] Camera (1116);

[0104] Microphone array (1214);

[0105] Memory (1104, 1204), the memory storing computer-executable instructions; and

[0106] A hardware processor (1102, 1202), connected to the camera (1116) and the microphone array (1214), the hardware processor (1102, 1202) being configured to execute the computer-executable instructions to:

[0107] The camera (1116) is used to acquire an image (300), wherein the image (300) comprises pixels arranged in rows and columns (300, 400);

[0108] Determine one or more bounding boxes (404-410) corresponding to one or more users (1-4) depicted in the image (300);

[0109] Determine a plurality of slots (450) associated with the image (400), wherein each of the plurality of slots is associated with one or more of the columns (400);

[0110] The microphone array (1214) is used to acquire multiple audio samples (1002);

[0111] Determine the source localization (SSL) horizontal angle value (1002) associated with one or more of the plurality of audio samples (1002);

[0112] For the corresponding SSL horizontal angle value (1002), determine one or more corresponding slots (450) among the plurality of slots (450);

[0113] For each of the one or more corresponding slots (450) associated with each corresponding SSL horizontal angle value (606), increment the slot entry count (606);

[0114] Determine a first extent (616, 640, 650) for each of the one or more bounding boxes (404-410), the first extent (616, 640, 650) indicating pixels extending from a first column to a second column (400);

[0115] For each bounding box, determine the slot entry count (618) associated with the first range (616, 640, 650) of each bounding box (404-410);

[0116] Determine the first bounding box (404-410) with the maximum entry count (620); and

[0117] Determine the output data (620) indicating the first bounding box (404 to 410).

[0118] 12. The system according to Example 11, wherein the computer-executable instructions for determining the plurality of slots associated with the image further include instructions for performing the following operations:

[0119] Determine the first pixel width (400) of the image (300);

[0120] Determine a second range (400), wherein the second range (400) includes a range of possible SSL horizontal angle values ​​(400) associated with the output (1002) from the microphone array (1214) during operation; and

[0121] Divide the first pixel width (400) by (602) the second range (400).

[0122] 13. According to the system described in Example 11, the hardware processor is further configured to execute the computer-executable instructions to:

[0123] The image (300) is processed using a trained machine learning system (1003) to determine the one or more bounding boxes (404-410) depicted in the image (300); and

[0124] The SSL algorithm (1002) is used to process one or more of the plurality of audio samples (1002) to determine the SSL horizontal angle value (400) indicating the direction of the sound.

[0125] 14. According to the system described in Example 11, the hardware processor is further configured to execute the computer-executable instructions to:

[0126] The first extent (616, 640, 650) of the first bounding box (404-410) is determined by one of the following operations:

[0127] Multiply the first width (616) of the first bounding box (404-410) by the first value (616).

[0128] Use the predefined pixel range (640), or

[0129] Use the distance (650) between two or more of the bounding boxes (404 to -410).

[0130] 15. According to the system of Example 11, the hardware processor is further configured to execute the computer-executable instructions to:

[0131] The output data (624) is sent, wherein the output data (624) indicates the direction associated with the user who is speaking (624, 1006, 1012).

[0132] 16. A method comprising:

[0133] Images are acquired using a camera (1116), wherein the images include pixels arranged relative to a first axis and a second axis;

[0134] Determine one or more bounding boxes (1003) corresponding to one or more faces depicted in the image;

[0135] Determine a plurality of slots (602) associated with the image, wherein each of the plurality of slots is associated with one or more pixels along the first axis of the image;

[0136] A microphone array (1214) is used to acquire multiple audio samples (1002);

[0137] Determine the source localization (SSL) horizontal angle value associated with one or more of the plurality of audio samples (1002);

[0138] For the corresponding SSL horizontal angle value (1002), determine one or more corresponding slots among the plurality of slots;

[0139] For each of the one or more corresponding slots associated with each corresponding SSL horizontal angle value, increment the slot entry count (606);

[0140] A first range (616, 640, 650) is determined relative to each of the one or more bounding boxes, the first range (616, 640, 650) indicating a pixel extending from a first pixel along the first axis to a second pixel along the first axis;

[0141] For each bounding box, determine the slot entry count (618) associated with the first range (618) of each bounding box;

[0142] Determine the first bounding box with the maximum slot entry count (620); and

[0143] Send the output data (624) associated with the first bounding box.

[0144] 17. According to the method described in Example 16, wherein:

[0145] The first axis specifies column (1002), and

[0146] The second axis specifies row (1002).

[0147] 18. According to the method described in Example 16, wherein:

[0148] The first axis specifies row (1002), and

[0149] The second axis specifies column (1002).

[0150] 19. The method according to Example 16 further includes:

[0151] For each bounding box, a first value (618) is determined that is associated with the slot entry count, the slot entry count being associated with the first range (618) of each bounding box;

[0152] Based on the first value (618), it is determined that the first bounding box has a maximum first value (620); and

[0153] The determination of the first bounding box having the maximum slot entry count (620) is based on the first bounding box having the maximum first value (620).

[0154] 20. The method according to Example 16, wherein the output data (624) indicates the final direction (1006) associated with the user who is speaking at the time the image is acquired and the plurality of audio samples are acquired.

Claims

1. A method for post-processing of horizontal angle for sound source localization guided by face detection for speaker tracking and framing in smart cameras, comprising: Images are acquired using a camera (1116), wherein the images include pixels arranged relative to a first axis and a second axis; Determine one or more bounding boxes (1003) corresponding to one or more faces depicted in the image; Determine a plurality of slots (602) associated with the image, wherein each of the plurality of slots is associated with one or more pixels along the first axis of the image; A microphone array (1214) is used to acquire multiple audio samples (1002); Determine the SSL horizontal angle value (1002) for the sound source localization associated with one or more of the plurality of audio samples; For the corresponding SSL horizontal angle value (1002), determine one or more corresponding slots among the plurality of slots; For each of the one or more corresponding slots associated with each corresponding SSL horizontal angle value, increment the slot entry count (606); A first range (616, 640, 650) is determined relative to each of the one or more bounding boxes, the first range indicating the width or height of the bounding box and encompassing pixels extending from a first pixel along the first axis to a second pixel along the first axis; For each bounding box, determine the slot entry count (618) associated with the first range (616, 640, 650) of each bounding box; Determine the first bounding box with the maximum slot entry count (620); as well as Determine the output data (624) indicating the first bounding box, wherein the output data includes information that allows determination of the source location of the sound to be emitted.

2. The method of claim 1, wherein determining the plurality of slots (602) associated with the image comprises: Determine the first pixel width of the image; Determine a second range, wherein the second range includes a range of possible SSL horizontal angle values ​​associated with the output from the microphone array during operation; and Divide the width of the first pixel by the second range.

3. The method according to claim 1, further comprising: The SSL algorithm (1002) is used to process one or more of the plurality of audio samples (1002) to determine the SSL horizontal angle value indicating the direction of the sound.

4. The method according to claim 1, further comprising: The image is processed using a trained machine learning system (1003) to determine the one or more bounding boxes depicted in the image.

5. The method of claim 1, further comprising: Determine the first width (610) of the first bounding box; Determine that the first width (610) of the first bounding box is less than a first value (612, 614), wherein the first value (612) indicates a minimum pixel value (612); and The first range (616) of the first bounding box is determined by multiplying the first value (614) by the second value (616), wherein the second value is a predetermined value (616).

6. The method of claim 1, further comprising: Determine the first width (610) of the first bounding box; Determine that the first width (610) of the first bounding box is greater than a first value (612, 616), where the first value (612) indicates the minimum pixel value (612); and The first range (616) of the first bounding box is determined by multiplying the first width (610) by a second value (616), wherein the second value is a predetermined value (616).

7. The method of claim 1, further comprising: Determine the reference pixel (610) of the first bounding box; as well as The predetermined pixel range (640) is designated as the first range (640) of the first bounding box relative to the reference pixel.

8. The method of claim 1, further comprising: Determine the distance (650) between two or more bounding boxes; as well as The first extent (650) of at least one of the two or more bounding boxes is determined based on the distance (650) between the two or more bounding boxes.

9. The method of claim 1, further comprising: Determine that the slot entry count (618) of the first bounding box is greater than the threshold (622); as well as The output data (624) is sent in response to the slot entry count of the first bounding box being greater than the threshold (622, 624).

10. The method of claim 1, further comprising: Determine the reference pixel (610) associated with the first bounding box; and The output data (624) is sent, wherein the output data (624) indicates the direction associated with the reference pixel (624).

Images