An audio-video cooperative control method, device and conference equipment

Abstract

Embodiments of the present application disclose an audio-video cooperative control method and device and conference equipment. The method is applied to the conference equipment, and the conference equipment comprises a first camera module, a second camera module and a microphone module. The method comprises the following steps: acquiring an infrared dot matrix and a depth map collected by the first camera module and a sound source direction collected by the microphone module; determining three-dimensional position information of each participant based on the infrared dot matrix and the depth map; determining a speaker in a conference scene based on the three-dimensional position information of the participants and the sound source direction; and synchronously adjusting shooting information of the second camera module and a beam direction of the microphone module based on the three-dimensional position information of the speaker, so as to collect audio-video of the speaker through the adjusted second camera module and the microphone module, thereby realizing accurate positioning of the speaker, and effectively improving audio-video experience and communication efficiency of the conference.

Description

Technical Field

[0001] The embodiments of the present invention relate to computer technology, and more particularly to an audio and video collaborative control method, apparatus and conferencing equipment. Background Technology

[0002] In remote, multi-person, far-field conferences, participants are typically far from the conferencing equipment, spread out over large distances, and speakers frequently switch roles. Existing conferencing equipment usually employs traditional two-dimensional face detection for framing and tracking, relying solely on planar image features to identify people. In far-field environments, faces are small in proportion to the image, easily affected by indoor lighting, obstruction from sideways movement, and complex backgrounds, leading to frequent issues such as missed face detection and framing failures. Furthermore, the current independent acquisition processes for audio and video data result in problems like audio-visual misalignment, audio-visual asynchrony, and uneven sound reception at different distances, significantly reducing the accuracy of audio-visual coordination and communication efficiency during meetings. Summary of the Invention

[0003] This invention provides an audio-visual collaborative control method, device, and conference equipment that combines spatial perception to achieve precise positioning of the speaker and synchronously control the collection of the speaker's audio-visual data, thereby achieving a high degree of unity in the speaker's audio-visual collaboration and effectively improving the audio-visual experience and communication efficiency of the meeting.

[0004] In a first aspect, embodiments of the present invention provide an audio-visual collaborative control method applied to a conference device, the conference device including a first camera module, a second camera module, and a microphone module; the method includes: The infrared dot matrix image and depth map corresponding to the meeting scene are acquired through the first camera module, and the direction of the sound source in the meeting scene is acquired through the microphone module. Based on the infrared dot matrix and the depth map, the three-dimensional position information of each participant in the meeting scene is determined; Based on the three-dimensional position information of each participant and the direction of the sound source, the speaker in the meeting scene is determined from the participants; Based on the speaker's three-dimensional position information, the shooting information of the second camera module and the beam direction of the microphone module are adjusted synchronously to capture audio and video of the speaker through the adjusted second camera module and microphone module.

[0005] Secondly, embodiments of the present invention also provide an audio-visual collaborative control device integrated into a conferencing device, the conferencing device including a first camera module, a second camera module, and a microphone module; the device includes: The information acquisition module is used to acquire the infrared dot matrix image and depth map corresponding to the meeting scene captured by the first camera module, as well as the sound source direction in the meeting scene captured by the microphone module; A three-dimensional position determination module is used to determine the three-dimensional position information of each participant in the meeting scene based on the infrared dot matrix and the depth map; The speaker identification module is used to identify the speaker in the meeting scene from the participants based on the three-dimensional position information of each participant and the direction of the sound source. The information adjustment module is used to synchronously adjust the shooting information of the second camera module and the beam direction of the microphone module based on the three-dimensional position information of the speaker, so as to collect audio and video of the speaker based on the adjusted second camera module and microphone module.

[0006] Thirdly, embodiments of the present invention also provide a conference device, the conference device comprising: a first camera module, a second camera module, a microphone module, and a processor; wherein, The first camera module is used to capture infrared dot matrix images and depth maps corresponding to the meeting scene; The second camera module is used to capture video footage of the speaker in the meeting scene; The microphone module is used to collect the direction of the sound source and the speaker's audio information in the meeting scene; The processor is used to implement the audio and video collaborative control method as provided in any embodiment of the present invention.

[0007] Fourthly, embodiments of the present invention also provide a computer-readable storage medium storing computer instructions, which are used to cause a processor to execute and implement the audio-visual collaborative control method provided in any embodiment of the present invention.

[0008] Fifthly, embodiments of the present invention also provide a computer program product, including a computer program that, when executed by a processor, implements the audio-visual collaborative control method provided in any embodiment of the present invention.

[0009] One embodiment of the above invention has the following advantages or beneficial effects: The conference equipment uses a first camera module to capture infrared dot matrix images and depth maps of the conference scene, and a microphone module to capture the direction of sound sources in the conference scene. Based on the captured infrared dot matrix images and depth maps, the three-dimensional position information of each participant in the conference scene is determined, enabling the conference equipment to have the ability to perceive the conference space. Based on the three-dimensional position information of each participant and the captured sound source direction, the speaker in the conference scene can be more accurately identified from the participants. Based on the speaker's three-dimensional position information, the shooting information of the second camera module and the beam direction of the microphone module in the conference scene are adjusted synchronously. Through the adjusted second camera module and microphone module, the speaker's audio and video are captured synchronously, resulting in a high degree of unified audio-visual collaboration between the speaker and the audience, effectively improving the audio-visual experience and communication efficiency of the conference.

[0010] It should be understood that the description in this section is not intended to identify key or essential features of the embodiments of the present invention, nor is it intended to limit the scope of the invention. Other features of the invention will become readily apparent from the following description. Attached Figure Description

[0011] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0012] Figure 1 This is a flowchart of an audio-visual collaborative control method provided in one embodiment of the present invention; Figure 2 This is a flowchart of another audio-visual collaborative control method provided in one embodiment of the present invention; Figure 3 This is a flowchart of another audio-visual collaborative control method provided in one embodiment of the present invention; Figure 4 This is a schematic diagram of the structure of an audio-visual collaborative control device provided in one embodiment of the present invention; Figure 5 This is a schematic diagram of the structure of a conference device provided in one embodiment of the present invention. Detailed Implementation

[0013] To enable those skilled in the art to better understand the present invention, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of the present invention.

[0014] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this invention are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of the invention described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover a non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0015] Figure 1 This is a flowchart illustrating an audio-visual collaborative control method according to an embodiment of the present invention. This embodiment is applicable to situations requiring collaborative control of audio and video during meetings, particularly suitable for remote multi-person far-field conferences. A remote multi-person far-field conference refers to a cross-regional video conference scenario where multiple participants in a local meeting room are geographically dispersed and located far from the conference equipment, and conduct the conference via a network. This method can be executed by an audio-visual collaborative control device, which can be implemented in software and / or hardware and can be configured within the conference equipment. The conference equipment includes a first camera module, a second camera module, and a microphone module. Figure 1 As shown, the method specifically includes the following steps: S110: Obtain the infrared dot matrix image and depth map corresponding to the meeting scene captured by the first camera module, and the direction of the sound source in the meeting scene captured by the microphone module.

[0016] The first camera module can be a camera module in the conference equipment used to detect spatial depth information. This first camera module does not participate in video data acquisition; it is specifically used to acquire infrared dot matrix images and depth maps corresponding to the conference scene. For example, the first camera module can be an infrared camera equipped with an infrared dot matrix projector. The infrared camera can be a monochrome / near-infrared (850nm / 940nm) camera, which, in conjunction with an active infrared dot matrix projector, acquires infrared dot matrix images and depth maps. The infrared dot matrix image can refer to the infrared grayscale texture image generated after the infrared camera acquires the infrared speckle image projected onto the conference scene by the infrared dot matrix projector. The infrared dot matrix image only contains grayscale brightness information, which can directly present the distribution of infrared light spots and object outlines within the conference scene, and is unaffected by the brightness of the ambient light, thus enabling millimeter-level depth positioning in completely dark environments. The depth map can be calculated based on the infrared dot matrix image, used to record a quantized image of the actual distance between each pixel and the first camera module. The brighter the grayscale of each pixel in the depth map, the closer that location is to the first camera module, thus intuitively presenting the three-dimensional spatial distance relationships within the conference scene. The infrared dot matrix image and the depth map are synchronously aligned, meaning that each pixel in the infrared dot matrix image corresponds one-to-one with each pixel in the depth map.

[0017] For example, the first camera module is encapsulated in an independent shock-absorbing box and has no rigid connection to the speaker module in the conference equipment. The speaker module can be used to play audio data output by the conference equipment, such as audio data from remote participants, to enable two-way voice communication, or to play local audio. When a speaker module is present in the conference equipment, the first camera module can be encapsulated in an independent metal shock-absorbing box, filled with silicone damping pads. Since the first camera module and the speaker module have no rigid contact, the interference caused by the speaker module's vibration is significantly reduced. For example, under 100Hz low-frequency vibration conditions, this method can achieve a vibration suppression effect of more than 20 decibels. The first camera module can also be connected to the motherboard via a flexible ribbon cable.

[0018] The microphone module can be a module in the conference equipment used for audio acquisition and processing. For example, the microphone module can be a microphone array containing multiple microphones, supporting beamforming. The microphone module can calculate the direction of the sound source based on the sound reception time difference. The direction of the sound source can refer to the horizontal angle information of the sound source in the conference scene relative to the microphone module. It only represents the planar orientation of the sound source and cannot reflect the actual distance between the sound source and the equipment.

[0019] It should be understood that placing conference equipment in the conference setting, such as on a conference table, allows for real-time data acquisition and processing. After the equipment is activated, the first camera module can capture infrared dot matrix and depth maps of the conference scene in real time, giving the equipment spatial awareness. The microphone module can capture the direction of sound sources in the conference scene in real time. The first camera module and microphone module can operate on the same clock to achieve synchronized acquisition of depth maps and sound source directions, thus ensuring accurate speaker identification.

[0020] S120. Based on infrared dot matrix and depth map, determine the three-dimensional position information of each participant in the meeting scene.

[0021] In this context, "participant" refers to each person attending the meeting within the meeting setting. The three-dimensional position information of the participants can be used to characterize their spatial location within the meeting setting.

[0022] It should be understood that by combining infrared dot matrix images and depth maps, the three-dimensional position information of each participant in the meeting scene can be determined in real time, thereby enabling more accurate positioning of each participant.

[0023] For example, step S120 may include: determining the human body mask of each participant in the meeting scene based on the infrared dot matrix map and the depth map; performing human body key point detection based on the infrared dot matrix area corresponding to the human body mask to obtain the key point pixel coordinates of each participant; and determining the three-dimensional position information of each participant based on the key point pixel coordinates and the depth map.

[0024] The human body mask for attendees can be a binary mask image, where human body areas (such as the torso, limbs, and head) are white, and non-human body areas (such as walls, floors, furniture, and distant environments) are black. The human body mask can be used to retain only human body pixel data and remove irrelevant pixels, thereby accurately marking the human body contour area of ​​the attendees. Human body key points can include, but are not limited to, the head, mouth, shoulders, and torso.

[0025] Specifically, the human body region is filtered based on the depth corresponding to each pixel in the depth map to quickly remove distant background and retain only pixels within the effective range of the human body, thus obtaining the human body region. The degree of light spot deformation at each location within the human body region is determined based on the infrared dot matrix image. This deformation degree is used to distinguish between the human body and nearby objects; for example, locations with a deformation degree greater than a preset value are identified as human body locations, while those less than or equal to the preset value are identified as non-human body locations. This allows for the selection of a more refined target human body region. Connected component segmentation is then performed on the target human body region to separate the individual human body regions of multiple participants. A human body mask is generated for each participant based on their corresponding human body region. Based on each participant's human body mask, the corresponding infrared dot matrix region is obtained from the infrared dot matrix image. Human key points can be detected and located based on the light spot features within this infrared dot matrix region to obtain the key point pixel coordinates for each participant. This can be achieved through template matching based on the light spot feature templates of each human key point, or by using a deep learning model to detect human key points. The depth values ​​of each human keypoint are obtained from the depth map based on the keypoint pixel coordinates of each participant. A transformation from the image coordinate system to a 3D coordinate system is then performed based on the keypoint pixel coordinates and corresponding depth values ​​of each participant to obtain the 3D coordinates of each human keypoint of each participant in the 3D coordinate system. The 3D coordinates of each human keypoint of a participant can be used as the participant's 3D position information; alternatively, the 3D coordinates of the head corresponding to the head keypoint can be directly used as the participant's 3D position information to adapt to close-range face tracking framing.

[0026] S130. Based on the three-dimensional position information and sound source direction of each participant, determine the speaker in the meeting scene from among the participants.

[0027] In this context, "speaker" can refer to the participant currently speaking in the meeting setting. It should be understood that compared to considering only the 2D image position of the participant, comprehensively considering the 3D position information of each participant and the direction of the sound source allows for more accurate identification of the speaker in the meeting setting. This achieves precise human body positioning, effectively avoiding interference from changes in lighting, screen reflections, personnel obstruction, and complex backgrounds, significantly improving the stability and accuracy of speaker tracking.

[0028] S140. Based on the speaker's three-dimensional position information, simultaneously adjust the shooting information of the second camera module and the beam direction of the microphone module to collect audio and video of the speaker through the adjusted second camera module and microphone module.

[0029] The second camera module can be a camera module in the conferencing equipment used to capture video data. The second camera module is responsible for the video output of the video conference. For example, the second camera module can be an RGB color camera. The second camera module and the first camera module capture data synchronously, and the capture frame rates can be the same or different. For example, the first camera module captures infrared dot matrix images and depth maps at 60fps, while the second camera module captures RGB video frames at 30fps.

[0030] It should be understood that, based on the speaker's three-dimensional position information in the meeting scenario, the shooting information of the second camera module and the beam direction of the microphone module are adjusted synchronously. For example, the field of view of the second camera module and the sound pickup direction of the microphone module are adjusted so that the adjusted second camera module and microphone module are aimed at the speaker for synchronous audio and video acquisition. This achieves precise "what you see is what you hear" tracking, ensuring a high degree of audio-visual coordination in scenarios where multiple people take turns speaking, and effectively improving the audio-visual experience and communication efficiency of remote multi-person far-field meetings.

[0031] For example, based on the baseline between the first camera module and the second camera module, the speaker's three-dimensional position information in the three-dimensional coordinate system corresponding to the first camera module can be transformed to obtain the speaker's three-dimensional position information in the three-dimensional coordinate system corresponding to the second camera module. Based on the transformed three-dimensional position information, the shooting information of the second camera module and the beam direction of the microphone module can be controlled, thereby ensuring the accuracy of audio and video acquisition.

[0032] For example, the speaker's three-dimensional position information includes the three-dimensional coordinates of the speaker's head and the three-dimensional coordinates of the mouth; step S140 includes: adjusting the zoom level and / or gimbal angle of the second camera module based on the three-dimensional coordinates of the speaker's head, so that the adjusted speaker's head is located at the center of the image captured by the second camera module; and adjusting the beam direction of the microphone module based on the three-dimensional coordinates of the speaker's mouth, so that the adjusted microphone module is aligned with the position of the speaker's mouth.

[0033] Specifically, based on the speaker's head coordinates, the target zoom level and / or target gimbal angle for the second camera module when aimed at the speaker's head are determined. The zoom level of the second camera module is then adjusted to the target zoom level, and / or the gimbal angle is adjusted to the target gimbal angle, ensuring that the center of the video image captured by the adjusted second camera module is the speaker's head area. This ensures that the speaker's head is always centered in the video frame, improving the video visual effect. Simultaneously, based on the speaker's mouth coordinates, the target beam direction for the microphone module when aimed at the speaker's mouth is determined. The microphone module's beam direction is then adjusted to the target beam direction, ensuring that both the image and audio follow the speaker's movements, achieving a highly unified meeting effect of audiovisual coordination.

[0034] For example, the infrared dot matrix and depth map collected in real time by the first camera module can determine the speaker and the speaker's three-dimensional position information in the meeting scene in real time. If the speaker or the same speaker's three-dimensional position information changes, it is necessary to readjust the shooting information of the second camera module and the beam direction of the microphone module based on the currently determined speaker's three-dimensional position information, so as to re-collect the speaker's audio and video through the adjusted second camera module and microphone module, thereby ensuring the synchronous tracking of the meeting's audio and video with the speaker and improving the meeting's audio-visual experience.

[0035] The technical solution of this embodiment uses a first camera module in the conference equipment to collect infrared dot matrix images and depth maps corresponding to the conference scene, and a microphone module in the conference equipment to collect sound source directions in the conference scene. Based on the collected infrared dot matrix images and depth maps, the three-dimensional position information of each participant in the conference scene is determined, enabling the conference equipment to have the ability to perceive the conference space. Based on the three-dimensional position information of each participant and the collected sound source directions, the speaker in the conference scene can be more accurately identified from the participants. Based on the speaker's three-dimensional position information, the shooting information of the second camera module and the beam direction of the microphone module in the conference scene are simultaneously adjusted. Through the adjusted second camera module and microphone module, the speaker's audio and video are collected synchronously, resulting in a high degree of unified audio-visual collaboration between the speaker and the audience, effectively improving the audio-visual experience and communication efficiency of the meeting.

[0036] In some alternative implementations, the method also includes at least one of the following: When acquiring infrared dot matrix and depth maps corresponding to the conference scene through the first camera module, the low-frequency output power of the speaker module in the conference equipment is reduced, and the low-frequency output power of the speaker module is restored after the acquisition of infrared dot matrix and depth maps is completed. In response to a power-on or vibration event of the conference equipment, the baseline between the first camera module and the second camera module is automatically calibrated.

[0037] Specifically, to avoid the impact of speaker module vibration on the baseline accuracy between the two camera modules, at least one of the above-mentioned anti-interference methods can be adopted to ensure stable operation of the two camera modules under high volume conditions. The first anti-interference method is as follows: During the acquisition of infrared dot matrix and depth maps corresponding to the conference scene by the first camera module, for example, within 3ms of acquiring the infrared dot matrix and depth maps, the low-frequency output power of the speaker module can be actively reduced using a digital filter, for example, by 12dB. Immediately after acquiring the infrared dot matrix and depth maps, the low-frequency output power of the speaker module is restored to its original level. This suppression of the speaker module's low-frequency output power reduces interference generated by the speaker module, further improving the accuracy of the participant's 3D position determination. The second anti-interference method is as follows: Upon detecting a power-on event or vibration event of the conference equipment, the baseline between the first and second camera modules is automatically calibrated to achieve self-calibration of the baseline during power-on or vibration. Here, a power-on event can refer to the start-up event of the conference equipment. A vibration event can be a vibration signal amplitude in the environment where the conference equipment is located exceeding a preset threshold. The automatic calibration process can be as follows: when a power-on event or vibration event is detected on the conference equipment, the first camera module automatically takes a picture of the fixed target inside the machine (such as a built-in checkerboard), and calculates the current baseline offset through an image matching algorithm. Based on the current baseline offset, the initial baseline is offset to obtain the offset target baseline. Then, the three-dimensional position information is determined based on the target baseline. This avoids the impact of speaker module vibration on the baseline accuracy between the two camera modules and ensures that the two camera modules work stably under high volume conditions.

[0038] In some optional implementations, the method further includes: determining the target gesture action generated within a preset distance range based on the infrared dot matrix and depth map; determining the target operation command corresponding to the target gesture action; and controlling the conference equipment based on the target operation command.

[0039] It should be understood that, based on infrared dot matrix images and depth maps, the target's trajectory and corresponding hand gestures within a preset distance range (e.g., 0.3~1.0 meters) can be accurately determined. Based on a pre-configured correspondence between hand gestures and operation commands, the target operation command corresponding to the target hand gesture can be determined. By executing the target operation command, the conference equipment can be automatically controlled accordingly, thereby achieving precise gesture control of the conference equipment, reducing the false trigger rate, and further improving meeting efficiency and user experience. The target operation command can include mute commands, hang-up commands, or volume adjustment commands, etc.

[0040] In some optional implementations, the method further includes: determining the speaker's human body mask based on an infrared dot matrix and a depth map; and blurring or virtually replacing the background in the speaker's image captured by the adjusted second camera module based on the human body mask.

[0041] It should be understood that, based on infrared dot matrix images and depth maps, the speaker's human body mask can be determined in real time. The specific process can be referred to the description above and will not be repeated here. Based on the human body mask, the speaker's image captured by the adjusted second camera module is divided into human body areas and background areas. The divided background areas are then blurred or virtually replaced, such as masking or making the background transparent, or replacing the background with any virtual background, thereby protecting environmental privacy or increasing the diversity of meeting backgrounds.

[0042] In some optional implementations, the method further includes: in response to the fact that the distance between each participant and the conference device in the conference scenario is greater than a preset distance and continues for a preset duration, controlling the conference device to hang up the conference to automatically end the video conference.

[0043] It should be understood that the infrared dot matrix and depth maps of the meeting scene collected in real time by the first camera module can determine the three-dimensional position information of each participant. Based on the three-dimensional position information of each participant and the position of the meeting equipment, the distance between each participant and the meeting equipment can be determined. If all participants are detected to be more than the preset distance (e.g., 8 seconds) and no participants are detected to be less than or equal to the preset distance within the preset time, it indicates that the current meeting has ended. At this time, the meeting equipment can be controlled to automatically hang up, thereby realizing the automatic end of the video conference without manual operation, further improving the user experience of the meeting equipment.

[0044] Figure 2This is a flowchart of another audio-visual collaborative control method provided in one embodiment of the present invention. Based on the above embodiments, this embodiment optimizes the step of "determining the speaker in the meeting scene from the participants based on the three-dimensional position information and sound source direction of each participant." Explanations of terms that are the same as or corresponding to those in the above embodiments are not repeated here.

[0045] See Figure 2 The other audio-visual collaborative control method provided in this embodiment specifically includes the following steps: S210: Obtain the infrared dot matrix image and depth map corresponding to the meeting scene captured by the first camera module, and the direction of the sound source in the meeting scene captured by the microphone module.

[0046] S220: Based on infrared dot matrix and depth map, determine the three-dimensional position information of each participant in the meeting scene.

[0047] S230. Based on the three-dimensional position information of each participant and the direction of the sound source, determine at least one first participant located in the direction of the sound source.

[0048] Specifically, based on the three-dimensional position information of each participant, the horizontal angle information of each participant relative to the microphone module can be determined. Based on the horizontal angle information of each participant and the horizontal angle information corresponding to the sound source direction, the horizontal angle difference of each participant can be determined. Each participant whose horizontal angle difference is less than a preset difference is identified as the first participant located in the sound source direction. The first participant refers to the participant who may be the speaker.

[0049] S240. Based on the three-dimensional position information of each first participant, determine the distance between each first participant and the conference equipment.

[0050] Specifically, based on the three-dimensional position information of each first participant and the three-dimensional position information of the conference equipment, the straight-line distance between each first participant and the conference equipment is determined. For example, based on the three-dimensional coordinates of each first participant's head in the conference scene and the three-dimensional coordinates of the conference equipment in the conference scene, the straight-line distance between each first participant and the conference equipment is determined.

[0051] S250: Based on distance, identify the speaker in the meeting scenario from the first participants.

[0052] Specifically, the participant closest to the conference equipment in the same direction of the sound source is usually the speaker. Thus, the first participant who is closest can be identified as the speaker in the conference scene. The final speaker is determined based on the actual distance between each first participant and the conference equipment, rather than the size of the face, thereby solving the problem of image confusion caused by overlapping.

[0053] For example, step S250 may include: comparing the distances corresponding to each first participant to obtain a minimum distance; and in response to the minimum distance being less than a preset distance, determining the first participant corresponding to the minimum distance as the speaker in the meeting scenario.

[0054] Specifically, the minimum distance among all distances is obtained. If the minimum distance is less than a preset distance, it indicates that the first participant corresponding to the minimum distance is within the speaking range of the conference equipment and is relatively close to the equipment. In this case, the first participant corresponding to the minimum distance can be identified as the speaker in the conference scenario. If the minimum distance is greater than or equal to the preset distance, it indicates that the first participant corresponding to the minimum distance has exceeded the speaking range of the conference equipment, and the first participant corresponding to the minimum distance will not be the speaker. In this case, it can be determined that there is no speaker in the conference scenario.

[0055] S260. Based on the speaker's three-dimensional position information, the shooting information of the second camera module and the beam direction of the microphone module are adjusted synchronously to capture audio and video of the speaker through the adjusted second camera module and microphone module.

[0056] The technical solution of this embodiment determines all first participants located in the direction of the sound source based on the three-dimensional position information and sound source direction of each participant. Based on the three-dimensional position information of each first participant, the distance between each first participant and the conference equipment is determined. Based on the distance, the speaker in the conference scene can be more accurately identified from the first participants, further ensuring the audiovisual tracking of the speaker and improving the audiovisual effect of the conference.

[0057] Figure 3 This is a flowchart illustrating another audio-visual collaborative control method according to an embodiment of the present invention. Based on the above embodiments, this embodiment provides a detailed description of audio-visual acquisition in a meeting scenario where there is no speaker. Explanations of terms that are the same as or corresponding to those in the above embodiments are not repeated here.

[0058] See Figure 3 The audio-visual collaborative control method provided in this embodiment specifically includes the following steps: S310, acquire the infrared dot matrix image and depth map corresponding to the meeting scene captured by the first camera module, and the direction of the sound source in the meeting scene captured by the microphone module.

[0059] S320: Based on infrared dot matrix and depth map, determine the three-dimensional position information of each participant in the meeting scene.

[0060] S330: Based on the three-dimensional position information and sound source direction of each participant, determine the speaker in the meeting scene from among the participants.

[0061] S340, in response to the absence of a speaker in the meeting scenario, determines the distance between each participant and the meeting equipment based on the three-dimensional position information of each participant.

[0062] The absence of a speaker in a meeting scenario may include, but is not limited to: no sound source signal being collected in the meeting scenario, thus no sound source direction being collected; or, no participant being located in the direction of the sound source; or, the distance between each participant and the meeting equipment is greater than or equal to a preset distance.

[0063] Specifically, in the absence of a speaker in the meeting scenario, the straight-line distance between each participant and the meeting equipment is determined based on the three-dimensional position information of each participant and the three-dimensional position information of the meeting equipment. For example, the straight-line distance between each participant and the meeting equipment is determined based on the three-dimensional coordinates of each participant's head in the meeting scenario and the three-dimensional coordinates of the meeting equipment in the meeting scenario.

[0064] S350. Based on the distance corresponding to each participant, determine the second participant from all participants.

[0065] The second participant can refer to the object captured by the second camera module, that is, the object displayed in the video feed. Specifically, in the absence of a speaker in the meeting scenario, the second participant to be captured is determined from all participants based on the distance between each participant and the meeting equipment.

[0066] For example, step S350 may include: comparing the distances corresponding to each participant to obtain the minimum distance; in response to the minimum distance being less than a preset distance, identifying the participant corresponding to the minimum distance as the second participant; in response to the minimum distance being greater than or equal to the preset distance, identifying all participants or historical speakers as the second participant.

[0067] Specifically, a minimum distance is determined from all distances. If the minimum distance is less than a preset distance, the participant closest to the minimum distance can be directly designated as the second participant, allowing for video recording of the closest participant. If the minimum distance is greater than or equal to the preset distance, it indicates that all participants are relatively far from the conference equipment. In this case, each participant can be designated as a second participant, allowing for panoramic video recording of the conference scene. Alternatively, previous speakers can be designated as second participants, such as the most recent speaker, for video recording of that speaker.

[0068] S360: Based on the three-dimensional position information of the second participant, adjust the shooting information of the second camera module and the beam direction of the microphone module, and collect audio and video of the second participant based on the adjusted second camera module and microphone module.

[0069] Specifically, based on the second participant's 3D position information within the meeting scenario, the shooting information of the second camera module and the beam direction of the microphone module are simultaneously adjusted. For example, the field of view of the second camera module and the sound pickup direction of the microphone module are adjusted. This ensures that both the adjusted second camera module and microphone module are aimed at the second participant for synchronized audio and video acquisition, centered on the second participant in the second camera's image, further enhancing the meeting experience. For instance, when the second participant is the closest participant, the center of the video image captured by the second camera module is the closest participant. When the second participant is a previous speaker, the second camera module continues to capture images of previous speakers. When the second participant includes all participants, the center position information of all participants can be determined based on each participant's 3D position information. The shooting information of the second camera module is then adjusted based on this center position information, allowing the second camera module to capture a panoramic view of the meeting including all participants.

[0070] For example, when the second participant is the closest participant and the conference equipment includes two identical second camera modules, the second closest participant can be identified as the third participant. Based on the 3D position information of the second and third participants, the shooting information of the two second camera modules is adjusted so that one adjusted second camera module is aimed at the second participant and the other adjusted second camera module is aimed at the third participant. This allows for the simultaneous acquisition of two video feeds: one showing the closest participant and the other showing the second closest participant. These two feeds can be displayed in a preset manner, such as picture-in-picture or side-by-side. In picture-in-picture mode, the closest participant's feed can be displayed as the main feed, and the second closest participant's feed can be displayed as the secondary feed, further enhancing the visual presentation and conference experience.

[0071] The technical solution of this embodiment determines the distance between each participant and the conference equipment based on the three-dimensional position information of each participant when there is no speaker in the conference scenario. A suitable second participant can be identified from all participants. Based on the three-dimensional position information of the second participant, the shooting information of the second camera module and the beam direction of the microphone module are adjusted. Based on the adjusted second camera module and microphone module, audio and video are captured from the second participant, thereby realizing reasonable display of the picture when no one is speaking, and further improving the immersion and conference experience of the conference.

[0072] The following are embodiments of the audio-visual collaborative control device provided in this invention. This device and the audio-visual collaborative control methods in the above embodiments belong to the same inventive concept. For details not described in detail in the embodiments of the audio-visual collaborative control device, please refer to the embodiments of the above audio-visual collaborative control methods.

[0073] Figure 4 This is a schematic diagram of an audio-visual collaborative control device provided in an embodiment of the present invention. This embodiment is applicable to situations where audio and video are collaboratively controlled during a meeting, and is particularly suitable for remote multi-person far-field meetings. The device is integrated into a conference device, which includes a first camera module, a second camera module, and a microphone module. Figure 4 As shown, the device specifically includes: an information acquisition module 410, a three-dimensional position determination module 420, a speaker determination module 430, and an information adjustment module 440.

[0074] The information acquisition module 410 is used to acquire the infrared dot matrix image and depth map corresponding to the meeting scene captured by the first camera module, and the sound source direction in the meeting scene captured by the microphone module; the three-dimensional position determination module 420 is used to determine the three-dimensional position information of each participant in the meeting scene based on the infrared dot matrix image and the depth map; the speaker determination module 430 is used to determine the speaker in the meeting scene from the participants based on the three-dimensional position information of each participant and the sound source direction; the information adjustment module 440 is used to synchronously adjust the shooting information of the second camera module and the beam direction of the microphone module based on the three-dimensional position information of the speaker, so as to collect audio and video of the speaker based on the adjusted second camera module and microphone module.

[0075] The technical solution of this embodiment uses a first camera module in the conference equipment to collect infrared dot matrix images and depth maps corresponding to the conference scene, and a microphone module in the conference equipment to collect sound source directions in the conference scene. Based on the collected infrared dot matrix images and depth maps, the three-dimensional position information of each participant in the conference scene is determined, enabling the conference equipment to have the ability to perceive the conference space. Based on the three-dimensional position information of each participant and the collected sound source directions, the speaker in the conference scene can be more accurately identified from the participants. Based on the speaker's three-dimensional position information, the shooting information of the second camera module and the beam direction of the microphone module in the conference scene are simultaneously adjusted. Through the adjusted second camera module and microphone module, the speaker's audio and video are collected synchronously, resulting in a high degree of unified audio-visual collaboration between the speaker and the audience, effectively improving the audio-visual experience and communication efficiency of the meeting.

[0076] Optionally, the three-dimensional position determination module 420 is specifically used for: Based on the infrared dot matrix image and the depth map, a human body mask for each participant in the meeting scene is determined; based on the infrared dot matrix area corresponding to the human body mask, human key point detection is performed to obtain the key point pixel coordinates of each participant; based on the key point pixel coordinates of each participant and the depth map, the three-dimensional position information of each participant is determined.

[0077] Optionally, the speaker determination module 430 includes: The first participant determination unit is used to determine at least one first participant located in the direction of the sound source based on the three-dimensional position information of each participant and the direction of the sound source. The distance determination unit is used to determine the distance between each first participant and the conference equipment based on the three-dimensional position information of each first participant; A speaker determination unit is used to determine the speaker in the meeting scenario from the first participants based on the distance.

[0078] Optionally, the speaker determination unit is specifically used for: The distances corresponding to each first participant are compared to obtain the minimum distance; in response to the minimum distance being less than a preset distance, the first participant corresponding to the minimum distance is determined as the speaker in the meeting scenario.

[0079] Optionally, the speaker's three-dimensional position information includes the three-dimensional coordinates of the speaker's head and mouth. Information adjustment module 440 is specifically used for: Based on the three-dimensional coordinates of the speaker's head, the zoom level and / or gimbal angle of the second camera module are adjusted so that the speaker's head is positioned in the center of the image captured by the second camera module; based on the three-dimensional coordinates of the speaker's mouth, the beam direction of the microphone module is adjusted so that the adjusted microphone module is aligned with the speaker's mouth.

[0080] Optionally, the device further includes: The distance determination module is used to determine the distance between each participant and the conference device based on the three-dimensional position information of each participant in the conference scenario when there is no speaker. The second participant determination module is used to determine the second participant from all participants based on the distance corresponding to each participant. The audio and video acquisition and adjustment module is used to adjust the shooting information of the second camera module and the beam direction of the microphone module based on the three-dimensional position information of the second participant, and to acquire audio and video of the second participant based on the adjusted second camera module and microphone module.

[0081] Optionally, the second participant determination module is specifically used for: The distances corresponding to each participant are compared to obtain the minimum distance; in response to the minimum distance being less than a preset distance, the participant corresponding to the minimum distance is determined as the second participant; in response to the minimum distance being greater than or equal to the preset distance, all participants or previous speakers are determined as the second participant.

[0082] Optionally, the device further includes at least one of the following: The power adjustment module is used to reduce the low-frequency output power of the speaker module in the conference equipment when the infrared dot matrix image and depth map corresponding to the conference scene are acquired by the first camera module, and to restore the low-frequency output power of the speaker module after the acquisition of the infrared dot matrix image and the depth map is completed. An automatic calibration module is used to automatically calibrate the baseline between the first camera module and the second camera module in response to a power-on event or a vibration event of the conference equipment.

[0083] Optionally, the device further includes: The target gesture action determination module is used to determine the target gesture action generated within a preset distance range based on the infrared dot matrix and the depth map. The conference equipment control module is used to determine the target operation command corresponding to the target gesture action, and control the conference equipment based on the target operation command.

[0084] Optionally, the device further includes: A human body mask determination module is used to determine the human body mask of the speaker based on the infrared dot matrix and the depth map; The background processing module is used to blur or virtually replace the background in the speaker's image captured by the adjusted second camera module based on the human body mask.

[0085] Optionally, the first camera module is an infrared camera equipped with an infrared dot matrix projector; The second camera module is an RGB camera; The first camera module is encapsulated in an independent shock-absorbing box and is not rigidly connected to the speaker module in the conference equipment.

[0086] The audio-visual collaborative control device provided in the embodiments of the present invention can execute the audio-visual collaborative control method provided in any embodiment of the present invention, and has the corresponding functional modules and beneficial effects of executing the audio-visual collaborative control method.

[0087] It is worth noting that in the embodiments of the above-mentioned audio and video collaborative control device, the various units and modules included are only divided according to functional logic, but are not limited to the above division, as long as the corresponding functions can be realized; in addition, the specific names of each functional unit are only for easy differentiation and are not used to limit the scope of protection of the present invention.

[0088] Figure 5 A schematic diagram of a conference device provided in an embodiment of the present invention, such as... Figure 5 As shown, the conference equipment includes: a first camera module 510, a second camera module 520, a microphone module 530, and a processor 540.

[0089] The first camera module 510 is used to acquire infrared dot matrix and depth maps corresponding to the meeting scene; the second camera module 520 is used to acquire video image information of the speaker in the meeting scene; the microphone module 530 is used to acquire the direction of the sound source and the audio information of the speaker in the meeting scene; and the processor 540 is used to implement the audio and video collaborative control method as provided in any embodiment of the present invention.

[0090] The first camera module 510 can be an infrared camera equipped with an infrared dot matrix projector. The second camera module 520 is an RGB camera. The first camera module 510 is encapsulated in an independent shock-absorbing box and is not rigidly connected to the speaker module in the conference equipment. The microphone module 530 can be a microphone array containing multiple microphones, supporting beamforming. A shock-absorbing structure is provided between the second camera module 520 and the speaker module, and a baseline self-calibration algorithm runs within the processor 540 to compensate for baseline shift caused by vibration.

[0091] In this embodiment, the conference equipment acquires infrared dot matrix images and depth maps corresponding to the conference scene through a first camera module, and acquires sound source directions in the conference scene through a microphone module. Based on the acquired infrared dot matrix images and depth maps, the three-dimensional position information of each participant in the conference scene is determined, enabling the conference equipment to have the ability to perceive the conference space. Based on the three-dimensional position information of each participant and the acquired sound source directions, the speaker in the conference scene can be more accurately identified from the participants. Based on the speaker's three-dimensional position information, the shooting information of the second camera module and the beam direction of the microphone module in the conference scene are simultaneously adjusted. Through the adjusted second camera module and microphone module, the speaker's audio and video are acquired synchronously, resulting in a high degree of unified audio-visual collaboration between the speaker and the audience, effectively improving the audio-visual experience and communication efficiency of the conference.

[0092] This embodiment provides a computer-readable storage medium storing a computer program thereon. When executed by a processor, the program implements the audio-visual collaborative control method steps provided in any embodiment of the present invention. The method includes: The infrared dot matrix image and depth map corresponding to the meeting scene are acquired through the first camera module, and the direction of the sound source in the meeting scene is acquired through the microphone module. Based on the infrared dot matrix and the depth map, the three-dimensional position information of each participant in the meeting scene is determined; Based on the three-dimensional position information of each participant and the direction of the sound source, the speaker in the meeting scene is determined from the participants; Based on the speaker's three-dimensional position information, the shooting information of the second camera module and the beam direction of the microphone module are adjusted synchronously to capture audio and video of the speaker through the adjusted second camera module and microphone module.

[0093] The computer storage medium of this invention can be any combination of one or more computer-readable media. A computer-readable medium can be a computer-readable signal medium or a computer-readable storage medium. A computer-readable storage medium can be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of computer-readable storage media (a non-exhaustive list) include: an electrical connection having one or more wires, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination thereof. In this document, a computer-readable storage medium can be any tangible medium that contains or stores a program that can be used by or in conjunction with an instruction execution system, apparatus, or device.

[0094] Computer-readable signal media may include data signals propagated in baseband or as part of a carrier wave, carrying computer-readable program code. Such propagated data signals may take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. Computer-readable signal media may also be any computer-readable medium other than computer-readable storage media, capable of sending, propagating, or transmitting programs for use by or in connection with an instruction execution system, apparatus, or device.

[0095] Program code contained on a computer-readable medium may be transmitted using any suitable medium, including but not limited to: wireless, wire, optical fiber, RF, etc., or any suitable combination thereof.

[0096] This invention also provides a computer program product, including a computer program that, when executed by a processor, implements the audio-visual collaborative control method provided in any embodiment of this invention.

[0097] In the implementation of a computer program product, computer program code for performing the operations of this invention can be written in one or more programming languages ​​or a combination thereof. Programming languages ​​include object-oriented programming languages ​​as well as conventional procedural programming languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving remote computers, the remote computer can be connected to the user's computer via any type of network—including a local area network (LAN) or a wide area network (WAN)—or can be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0098] Those skilled in the art will understand that the modules or steps of the present invention described above can be implemented using general-purpose computing devices. They can be centralized on a single computing device or distributed across a network of multiple computing devices. Optionally, they can be implemented using computer-executable program code, thereby allowing them to be stored in a storage device for execution by a computing device, or they can be fabricated as separate integrated circuit modules, or multiple modules or steps can be fabricated as a single integrated circuit module. Thus, the present invention is not limited to any particular combination of hardware and software.

[0099] Note that the above description is merely a preferred embodiment of the present invention and the technical principles employed. Those skilled in the art will understand that the present invention is not limited to the specific embodiments described herein, and various obvious changes, readjustments, and substitutions can be made without departing from the scope of protection of the present invention. Therefore, although the present invention has been described in detail through the above embodiments, the present invention is not limited to the above embodiments, and may include many other equivalent embodiments without departing from the concept of the present invention, the scope of which is determined by the scope of the appended claims.

Claims

1. An audio-visual collaborative control method, characterized in that, The method is applied to a conferencing device, which includes a first camera module, a second camera module, and a microphone module; the method includes: The infrared dot matrix image and depth map corresponding to the meeting scene are acquired through the first camera module, and the direction of the sound source in the meeting scene is acquired through the microphone module. Based on the infrared dot matrix and the depth map, the three-dimensional position information of each participant in the meeting scene is determined; Based on the three-dimensional position information of each participant and the direction of the sound source, the speaker in the meeting scene is determined from the participants; Based on the speaker's three-dimensional position information, the shooting information of the second camera module and the beam direction of the microphone module are adjusted synchronously to capture audio and video of the speaker through the adjusted second camera module and microphone module.

2. The method according to claim 1, characterized in that, The determination of the three-dimensional position information of each participant in the meeting scene based on the infrared dot matrix and the depth map includes: Based on the infrared dot matrix and the depth map, the human body mask of each participant in the conference scene is determined; Human key points are detected based on the infrared dot matrix area corresponding to the human mask to obtain the key point pixel coordinates of each participant. Based on the keypoint pixel coordinates of each participant and the depth map, the three-dimensional position information of each participant is determined.

3. The method according to claim 1, characterized in that, The step of determining the speaker in the meeting scenario from the participants based on the three-dimensional position information of each participant and the direction of the sound source includes: Based on the three-dimensional position information of each participant and the direction of the sound source, at least one first participant located in the direction of the sound source is determined; Based on the three-dimensional location information of each first participant, the distance between each first participant and the conference equipment is determined; Based on the distance, the speaker in the meeting scenario is determined from the first participants.

4. The method according to claim 3, characterized in that, The step of determining the speaker in the meeting scenario from the first participants based on the distance includes: The distances corresponding to each first participant are compared to obtain the minimum distance; In response to the minimum distance being less than a preset distance, the first participant corresponding to the minimum distance is determined as the speaker in the meeting scenario.

5. The method according to claim 1, characterized in that, The speaker's three-dimensional position information includes the three-dimensional coordinates of the speaker's head and mouth. The step of synchronously adjusting the shooting information of the second camera module and the beam direction of the microphone module based on the speaker's three-dimensional position information includes: Based on the three-dimensional coordinates of the speaker's head, the zoom level and / or gimbal angle of the second camera module are adjusted so that the adjusted head of the speaker is located in the center of the image captured by the second camera module. Based on the three-dimensional coordinates of the speaker's mouth, the beam direction of the microphone module is adjusted so that the adjusted microphone module is aligned with the speaker's mouth.

6. The method according to claim 1, characterized in that, The method further includes: In response to the absence of a speaker in the meeting scenario, the distance between each participant and the meeting device is determined based on the three-dimensional position information of each participant; Based on the distance corresponding to each participant, the second participant is determined from all participants; Based on the three-dimensional position information of the second participant, the shooting information of the second camera module and the beam direction of the microphone module are adjusted, and audio and video are captured from the second participant based on the adjusted second camera module and microphone module.

7. The method according to claim 6, characterized in that, The process of determining the second participant from all participants based on the distance corresponding to each participant includes: The distances corresponding to each participant are compared to obtain the minimum distance; In response to the minimum distance being less than a preset distance, the participant corresponding to the minimum distance is determined as the second participant; In response to the minimum distance being greater than or equal to a preset distance, all participants or previous speakers are identified as the second participant.

8. The method according to claim 1, characterized in that, The method further includes at least one of the following: When acquiring infrared dot matrix and depth maps corresponding to the conference scene through the first camera module, the low-frequency output power of the speaker module in the conference equipment is reduced, and the low-frequency output power of the speaker module is restored after the acquisition of the infrared dot matrix and depth maps is completed. In response to a power-on event or a vibration event of the conference device, the baseline between the first camera module and the second camera module is automatically calibrated.

9. The method according to claim 1, characterized in that, The method further includes: Based on the infrared dot matrix and the depth map, the target hand gesture generated within a preset distance range is determined; The target operation command corresponding to the target gesture is determined, and the conference equipment is controlled based on the target operation command.

10. The method according to claim 1, characterized in that, The method further includes: Based on the infrared dot matrix and the depth map, the human body mask of the speaker is determined; Based on the human body mask, the background in the speaker's image captured by the adjusted second camera module is blurred or virtually replaced.

11. The method according to any one of claims 1-10, characterized in that, The first camera module is an infrared camera equipped with an infrared dot matrix projector; The second camera module is an RGB camera; The first camera module is encapsulated in an independent shock-absorbing box and is not rigidly connected to the speaker module in the conference equipment.

12. An audio-visual collaborative control device, characterized in that, Integrated into a conference device, the conference device including a first camera module, a second camera module, and a microphone module; the device includes: The information acquisition module is used to acquire the infrared dot matrix image and depth map corresponding to the meeting scene captured by the first camera module, as well as the sound source direction in the meeting scene captured by the microphone module; A three-dimensional position determination module is used to determine the three-dimensional position information of each participant in the meeting scene based on the infrared dot matrix and the depth map; The speaker identification module is used to identify the speaker in the meeting scene from the participants based on the three-dimensional position information of each participant and the direction of the sound source. The information adjustment module is used to synchronously adjust the shooting information of the second camera module and the beam direction of the microphone module based on the three-dimensional position information of the speaker, so as to collect audio and video of the speaker based on the adjusted second camera module and microphone module.

13. A conference device, characterized in that, The conference equipment includes: a first camera module, a second camera module, a microphone module, and a processor; wherein, The first camera module is used to capture infrared dot matrix images and depth maps corresponding to the meeting scene; The second camera module is used to capture video footage of the speaker in the meeting scene; The microphone module is used to collect the direction of the sound source and the speaker's audio information in the meeting scene; The processor is used to implement the audio and video collaborative control method according to any one of claims 1-11.

Images