Sound processing method, system and device based on audio and video fusion and storage medium
By combining microphone arrays and cameras, sound source localization and image recognition are performed, and the sound curtain angle range is set to achieve precise enhancement of the speaker's audio, thus solving the problem of noise interference in video conferencing and improving the audio processing effect.
Patent Information
- Application Number
- CN202410855898.4
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-06-28
- Publication Date
- 2024-11-08
AI Technical Summary
In fields such as video conferencing, how to effectively identify and block noise, improve audio enhancement, and especially accurately pick up and enhance the audio of speakers is crucial.
By using a microphone array to locate the sound source and combining it with camera image recognition, the system determines the sound source location angle and the angle of the person, sets the sound curtain angle range, and determines whether to enhance or suppress the sound source audio, thus achieving precise audio processing.
It improves audio capture in video conferencing, effectively identifies and enhances the speaker's audio, reduces noise interference, and improves meeting quality.
Smart Images

Figure CN118918914A_ABST
Abstract
Description
Background Art
[0002] Microphone arrays are often used in video conferencing, security, transportation, military or some industrial applications. For example, in a video conferencing system, the microphone array can be used to locate the speaker, control the camera to turn to the speaker for close-up image acquisition, and form a directional beam in the direction of the speaker for voice enhancement. During a video conference, various noises may occur. How to accurately identify and block noise is very important to improve the effect of video conferencing. Summary of the invention
[0003] In response to the problems in the prior art, the purpose of the present application is to provide a sound processing method, system, device and storage medium based on audio and video fusion, which can effectively identify the sound source audio that needs to be enhanced and perform enhanced processing.
[0004] The present invention provides a method for processing audio and video based on audio and video fusion, comprising the following steps:
[0005] Acquire a sound source localization result, and determine first sound source localization angle information based on the sound source localization result;
[0006] Acquire an image of a person to be processed, and obtain angle information to be matched of each person based on the pixel coordinates to be matched of each person in the image of the person to be processed;
[0007] Based on the angle information to be matched of each person, it is determined whether there is angle information of a person that matches the first sound source localization angle information, and both the first sound source localization angle information and the angle information of the matched person are within a preset sound curtain angle range;
[0008] If so, the sound source audio corresponding to the first sound source positioning angle information is enhanced.
[0009] By adopting the sound processing method of the present application, firstly, the first sound source positioning angle information determined based on the sound source positioning result and the angle information of each person obtained based on the person image are respectively obtained, and the first sound source positioning angle information, the angle information of the person and the preset sound curtain angle range are combined for fusion processing to determine whether it is necessary to enhance the first sound source positioning angle information, and when it is determined that enhancement is required, the corresponding sound source audio is enhanced, thereby improving the pickup effect of the corresponding sound source audio. The sound processing method can be applied to video conferencing scenarios and is implemented based on video conferencing equipment. The video conferencing equipment is provided with a camera and a microphone array. The camera is used to obtain the image of the person to be processed, and the microphone array is used to collect audio. By adopting this method, the audio of the speaker can be effectively identified and enhanced, effectively improving the video conferencing effect.
[0010] In some embodiments, the sound source localization result includes a plurality of candidate sound source localization angle information, and determining the first sound source localization angle information based on the sound source localization result includes the following steps:
[0011] According to a preset abnormal screening rule, the abnormal sound source localization angle information among the multiple candidate sound source localization angle information is excluded;
[0012] A cluster analysis is performed on the candidate sound source localization angle information, and a candidate sound source localization angle information is determined according to the clustering result as the first sound source localization angle information.
[0013] In some embodiments, obtaining the to-be-matched angle information of each person based on the pixel coordinates of each person in the image of the person to be processed includes the following steps:
[0014] Performing image recognition on the person image to be processed to obtain a face recognition result of the person image, and obtaining the pixel coordinates to be matched of each person in the person image based on the face recognition result;
[0015] Performing coordinate conversion on the to-be-matched pixel coordinates of each person to obtain a horizontal angle and a pitch angle in a Cartesian coordinate system;
[0016] The angle information to be matched of each person is calculated based on the horizontal angle and the pitch angle in the Cartesian coordinate system.
[0017] In some embodiments, the sound curtain angle range is pre-set by the following steps:
[0018] Acquire an initial person image, and obtain initial angle information of each person based on initial pixel coordinates of each person in the initial person image;
[0019] The sound curtain angle range is adaptively set based on the initial angle information of all persons, so that the sound curtain angle range includes the initial angle information of all persons.
[0020] In some embodiments, the following steps are also included to determine whether to update the sound curtain angle range:
[0021] Acquire a real-time person image at the current moment and perform image recognition to obtain a face recognition result at the current moment, and obtain the pixel coordinates of each person in the real-time person image at the current moment based on the face recognition result;
[0022] Match the face recognition result at the current moment with the face in the face recognition result at the previous moment to determine whether there is a change in the number of people;
[0023] Compare the pixel coordinates of each person at the current moment with the pixel coordinates at the previous moment to determine whether there is a change in the position of the person;
[0024] If the change in the number of people and / or the change in their positions meets the preset sound curtain angle update requirements, the current moment angle information of each person is obtained based on the current moment pixel coordinates of each person, and the sound curtain angle range is updated based on the current moment angle information, so that the updated sound curtain angle range includes the current angle information of all people.
[0025] In some embodiments, after obtaining the image of the person to be processed, the method further includes performing image recognition on the image of the person to be processed to obtain the confidence probability of each person;
[0026] Based on the angle information to be matched of each person, it is determined whether there is angle information of a person that matches the first sound source localization angle information, and both the first sound source localization angle information and the angle information of the matched person are within a preset sound curtain angle range, including the following steps:
[0027] Filtering the first sound source positioning angle and the angle information of the person to be matched within the angle range of the sound curtain as the angle information of the candidate first sound source positioning angle and the candidate person;
[0028] It is determined whether there is angle information of a candidate person that matches the candidate first sound source localization angle and whether the confidence probability corresponding to the angle information of the matching candidate person is greater than a preset confidence threshold.
[0029] In some embodiments, enhancing the sound source audio corresponding to the first sound source localization angle information comprises the following steps:
[0030] According to the angle information of the matched person, the first sound source localization angle information is adjusted to obtain adjusted second sound source localization angle information;
[0031] The corresponding sound source audio is enhanced according to the second sound source localization angle information.
[0032] The embodiment of the present application further provides a sound processing system based on audio and video fusion, which is applied to the sound processing method based on audio and video fusion, and the system includes:
[0033] A sound source localization module, used to obtain a sound source localization result, and determine first sound source localization angle information based on the sound source localization result;
[0034] An image processing module is used to obtain an image of a person to be processed, and obtain angle information to be matched of each person based on the pixel coordinates to be matched of each person in the image of the person to be processed;
[0035] An angle matching module, used to determine whether there is angle information of a person that matches the first sound source localization angle information based on the angle information to be matched of each person, and the first sound source localization angle information and the angle information of the matched person are both within a preset sound curtain angle range;
[0036] A sound source enhancement module is used to enhance the sound source audio corresponding to the first sound source localization angle information if there is angle information of a person matching the first sound source localization angle information, and the first sound source localization angle information and the angle information of the matching person are both within a preset sound curtain angle range.
[0037] By adopting the sound processing system, each step of the above-mentioned sound processing method can be implemented when the functions of each module are executed, so the technical effect of the above-mentioned sound processing method can be obtained.
[0038] The embodiment of the present application also provides a sound processing device based on audio and video fusion, including:
[0039] processor;
[0040] a memory storing executable instructions of the processor;
[0041] Wherein, the processor is configured to perform the steps of the sound processing method based on audio and video fusion by executing the executable instructions.
[0042] By adopting the sound processing device based on audio and video fusion provided in the present application, the processor executes the sound processing method based on audio and video fusion when executing the executable instructions, thereby obtaining the beneficial effects of the above-mentioned sound processing method based on audio and video fusion.
[0043] An embodiment of the present application also provides a computer-readable storage medium for storing a program, which, when executed by a processor, implements the steps of the sound processing method based on audio and video fusion.
[0044] By adopting the computer-readable storage medium provided in the present application, the program stored therein implements the steps of the sound processing method based on audio and video fusion when executed, thereby obtaining the beneficial effects of the sound processing method based on audio and video fusion. BRIEF DESCRIPTION OF THE DRAWINGS
[0045] Other features, objects and advantages of the present application will become more apparent by reading the detailed description of non-limiting embodiments made with reference to the following drawings.
[0046] Figure 1 is a flow chart of a sound processing method based on audio and video fusion according to an embodiment of the present application;
[0047] Figure 2 is a schematic diagram of the angle range of the sound curtain according to an embodiment of the present application;
[0048] Figure 3 is a schematic diagram of converting pixel coordinates into a Cartesian coordinate system according to an embodiment of the present application;
[0049] Figure 4 is a schematic diagram of obtaining angle information of a person according to Cartesian coordinates according to an embodiment of the present application;
[0050] Figure 5 is a schematic diagram of a sound processing system based on audio and video fusion according to an embodiment of the present application;
[0051] Figure 6 is a structural schematic diagram of a sound processing device based on audio and video fusion according to an embodiment of the present application;
[0052] Figure 7 It is a schematic diagram of the structure of a computer storage medium according to an embodiment of the present application. DETAILED DESCRIPTION
[0053] Example embodiments will now be described more fully with reference to the accompanying drawings. However, example embodiments can be implemented in a variety of forms and should not be construed as limited to the embodiments set forth herein; rather, these embodiments are provided so that this application will be comprehensive and complete and the concepts of the example embodiments will be fully conveyed to those skilled in the art. The same reference numerals in the figures represent the same or similar structures, and thus their repeated description will be omitted. Although "first" or "second" etc. are used in this specification to represent certain features, they are only used to represent the function and are not used as a limitation on the number and importance of specific features.
[0054] In order to solve the technical problems in the prior art, the present application provides a sound processing method based on audio and video fusion. The sound processing method can be applied to video conferencing scenarios and implemented based on video conferencing equipment. The video conferencing equipment is provided with a camera and a microphone array. The camera is used to obtain the image of the person to be processed, and the microphone array is used to collect audio. The camera is, for example, a fixed-focus panoramic lens, and the microphone array is, for example, a linear microphone array, but the present application is not limited to this. The use of other types of cameras or microphone arrays (such as a plane array) also falls within the scope of protection of the present application. The following takes the application of this method to a video conferencing scenario as an example to specifically introduce the detailed implementation of the sound processing method. However, the present application is not limited to this. In other embodiments, the method can also be applied to other scenarios where it is necessary to collect the voices and images of multiple people speaking, for example, a scene where multiple people shoot a video together, etc., all of which fall within the scope of protection of the present application.
[0055] like Figure 1As shown, in one embodiment, the present application provides a sound processing method based on audio and video fusion, comprising the following steps:
[0056] S100: Acquire a sound source localization result, and determine first sound source localization angle information based on the sound source localization result;
[0057] In this embodiment, the sound source audio is acquired through the microphone array, and after the sound source is localized, multiple sound source localization angle information can be obtained, which is used as the sound source localization result, and the first sound source localization angle information that is most likely to correspond to the sound source of the current speaker is obtained by screening;
[0058] S200: Acquire a person image to be processed, and obtain angle information to be matched of each person based on the pixel coordinates to be matched of each person in the person image to be processed;
[0059] The acquisition time of the image of the person to be processed is the same as the acquisition time of the sound source audio of the sound source localization result, or the time difference is very small, so that the image and the audio can correspond to each other;
[0060] In this embodiment, image recognition is performed on the image of the person to be processed to identify faces in the image, each face corresponds to a person, and a face recognition result is obtained. After obtaining the pixel coordinates to be matched of the person in the image of the person to be processed based on the face recognition result, the pixel coordinates are converted to obtain the angle information to be matched.
[0061] S300: Based on the angle information to be matched of each person, it is determined whether there is angle information of a person that matches the first sound source localization angle information, and the first sound source localization angle information and the angle information of the matched person are both within a preset sound curtain angle range;
[0062] The angle information of the person that matches the first sound source localization angle information is the angle information of the person that is close to the first sound source localization angle information. For example, the angle information of the matching person can be selected by judging whether the angle difference between the angle information to be matched of the person and the first sound source localization angle information meets the preset angle difference requirement;
[0063] Sound curtain technology refers to the technology of setting a certain sound curtain range, enhancing the audio within the sound curtain range, and shielding the audio outside the sound curtain range. The sound curtain angle range is the corresponding angle range within the preset sound curtain range. The audio within this sound curtain angle range can be enhanced, while the audio outside this sound curtain angle range is shielded;
[0064] If it exists, it means that the sound source corresponding to the first sound source localization angle information is the sound source of the current speaker, then S400: enhance the sound source audio corresponding to the first sound source localization angle information; further, at the same time, other sound source audios may be suppressed or shielded;
[0065] Here, enhancing the audio refers to increasing the volume of the audio or other enhancing means, suppressing the audio refers to reducing the volume of the audio or other weakening means, and shielding the audio refers to excluding the audio without processing;
[0066] If not, S500: do not enhance the sound source audio corresponding to the first sound source localization angle information. That is, if there is no angle information of a person matching the first sound source localization angle information, the angle information of a person matching the first sound source localization angle information is outside the angle range of the sound screen, or the first sound source localization angle information is outside the angle range of the sound screen, the sound source audio corresponding to the first sound source localization angle information will be suppressed or shielded.
[0067] By adopting the sound processing method of the present application, the first sound source positioning angle information determined based on the sound source positioning result and the angle information of each person obtained based on the person image are obtained through steps S100 and S200 respectively, and the first sound source positioning angle information, the angle information of the person and the preset sound curtain angle range are combined through step S300 for fusion processing to determine whether the first sound source positioning angle information needs to be enhanced, and when it is determined that enhancement is required, the corresponding sound source audio is enhanced through step S400, thereby improving the pickup effect of the corresponding sound source audio. When this method is applied to a video conferencing scenario, it can be deployed in the controller of the video conferencing equipment to effectively identify the audio of the speaker and enhance it, effectively improving the video conferencing effect.
[0068] In this embodiment, before step S300, a step of presetting the sound curtain angle range is also included. The sound curtain angle range is set as follows: Figure 2 As shown in the figure, J100 represents the video conferencing equipment, including a linear microphone array, with a camera at the center of the linear microphone array. J200 represents the set sound screen angle range, which can be set using two parameters: the minimum sound screen angle m1 and the maximum sound screen angle m2. The sound screen angle range is [m1, m2]. The sound screen angle range can cover the range of participants. Figure 2 Each circle in the figure represents each participant.
[0069] Specifically, in different implementations, there are two ways to set the sound curtain angle range: (1) manually set, for example, the staff can use their own terminal equipment to set the sound curtain angle range parameters m1 and m2, and then transmit them to the video conferencing device through the serial port protocol, or provide a sound curtain angle setting page on the video conferencing device, and the staff can directly input the sound curtain angle range parameters m1 and m2 on the page. (2) Based on the camera to collect images, and adaptively set the sound curtain angle range according to the collected images.
[0070] Specifically, the adaptive setting of the sound curtain angle range includes the following steps:
[0071] Acquire an initial person image, and obtain initial angle information of each person based on initial pixel coordinates of each person in the initial person image;
[0072] Specifically, the initial person image is subjected to image recognition to identify the face in the image, and an initial face recognition result is obtained. After the initial pixel coordinates of the person in the initial person image are obtained based on the initial recognition result, the initial angle information of each person is obtained by conversion based on the initial pixel coordinates. The method of converting the pixel coordinates into the angle information is the same as the method of converting the pixel coordinates to be matched into the angle information to be matched in the above step S200, which will be described in detail below.
[0073] Adaptively setting the sound curtain angle range based on the initial angle information of all personnel, so that the sound curtain angle range includes the initial angle information of all personnel;
[0074] Specifically, when performing image recognition, the confidence probability of each face recognition is also obtained as the confidence probability of the corresponding person. The confidence probability is, for example, the probability value of each target being a face output by the image recognition model based on machine learning. A first confidence probability threshold is set, and only persons whose confidence probability is greater than the first confidence probability threshold will be considered within the sound curtain angle range, while persons whose confidence probability is less than or equal to the first confidence probability threshold will not be considered.
[0075] In this embodiment, the sound source localization result includes a plurality of candidate sound source localization angle information. The step S100: determining the first sound source localization angle information based on the sound source localization result includes the following steps:
[0076] According to a preset abnormal screening rule, the abnormal sound source localization angle information among the multiple candidate sound source localization angle information is excluded;
[0077] Specifically, voice activity detection is continuously performed during the video conference. When the voice activity detection VAD value is 1, the SRP (steered-response power, a technology that uses beamforming to detect and enhance sounds in different directions of space) sound source localization results are obtained, including multiple candidate sound source localization angle information (each representing an angle value). Among these candidate sound sources, some may be the speaker sound source that needs to be enhanced, and some may be noise or noise sound sources other than the current speaker. First, preliminary abnormal screening needs to be performed;
[0078] For example, the srp sound source localization result is put into the cache. After the number of srp sound source localization results in the cache reaches a certain number threshold, a maximum angle abnormal_ANGLE of abnormal jitter is set (for example, it can be set to 10 to 20 degrees). First, the maximum value and the minimum value in the candidate sound source localization angle information in the cache are excluded, and the average value mean is calculated for the remaining angles. Then, abs(angle-mean) is calculated for each remaining candidate sound source localization information angle, and it is determined whether the calculation result of abs(angle-mean) is greater than abnormal_ANGLE. If so, it is considered that the jitter of the corresponding candidate sound source localization information is too large, and the corresponding candidate sound source localization information is excluded.
[0079] Performing cluster analysis on the candidate sound source localization angle information, and determining a candidate sound source localization angle information according to the clustering result as the first sound source localization angle information;
[0080] In this embodiment, K-means clustering analysis is performed on the candidate sound source localization angle information to find the cluster center of the largest cluster, and a candidate sound source localization angle information is selected from the candidate sound source localization angle information in the cache according to the cluster center as the first sound source localization angle information. K-means clustering analysis is an iterative clustering analysis algorithm, which comprises the following steps: pre-dividing the data into K groups, randomly selecting K objects as initial cluster centers, and then calculating the distance between each object and each seed cluster center, and assigning each object to the cluster center closest to it.
[0081] In this embodiment, the step S200: obtaining the to-be-matched angle information of each person based on the pixel coordinates of each person in the to-be-processed person image, comprises the following steps:
[0082] Performing image recognition on the person image to be processed to obtain a face recognition result of the person image, and obtaining the pixel coordinates to be matched of each person in the person image based on the face recognition result;
[0083] Performing coordinate conversion on the to-be-matched pixel coordinates of each person to obtain a horizontal angle and a pitch angle in a Cartesian coordinate system;
[0084] In this embodiment, the pixel coordinate system is a two-dimensional coordinate system, with the center point of the image as the origin, the horizontal right direction is the positive direction of the x-axis, and the horizontal upward direction is the positive direction of the y-axis. The Cartesian coordinate system is a three-dimensional coordinate system, with the center point of the video conferencing device as the origin, the length direction of the screen of the video conferencing device is the x-axis direction, the height direction is the z-axis direction, and the thickness direction is the y-axis direction;
[0085] Figure 3 exemplarily shows a schematic diagram of converting a pixel coordinate into a horizontal angle in a Cartesian coordinate system, wherein the horizontal angle α can be calculated based on the following formula:
[0086]
[0087]
[0088]
[0089] Among them, Φ Hfov is the horizontal field of view, which indicates the range that the lens can detect or shoot in the horizontal direction, W is the width of the image, x is the x-axis coordinate value of the pixel coordinate, d is the focal length, and W ′ is the length of the base of the triangle determined by the horizontal field of view and focal length, x ′ According to α ′ The length of the base of the triangle determined by the angle and focal length, α ′ Angle is the complementary angle of the horizontal angle α;
[0090] The calculation principle of the pitch angle β is similar to that of the horizontal angle. Specifically, the pitch angle β can be calculated based on the following formula:
[0091]
[0092]
[0093]
[0094] Among them, Φ Vfov is the vertical field of view, which indicates the range that the lens can detect or shoot in the vertical direction. H is the height of the image, z is the z-axis coordinate value of the pixel coordinate, d is the focal length, and H ′ is the length of the base of the triangle determined by the vertical field angle and focal length, z ′ According to β ′ The length of the base of the triangle determined by the angle and focal length, β ′ Angle is the complementary angle of the pitch angle β;
[0095] Calculating the angle information to be matched of each person based on the horizontal angle and the pitch angle in the Cartesian coordinate system;
[0096] In this embodiment, since the distance between the camera and the microphone array is small, the angle deviation between the camera and the microphone array can be ignored. This solution is applicable to single-angle scenes of linear arrays and is not affected by the pitch angle of the camera; Figure 4 As shown, the angle information of each person corresponds to the actual angle θ. After the horizontal angle and the pitch angle are calculated, the angle information of each person can be calculated using the following formula:
[0097]
[0098] In this embodiment, in the step S200, after obtaining the image of the person to be processed and performing image recognition on the image of the person, a step of obtaining the confidence probability of each person is also included. The confidence probability is, for example, the probability value of each target being a face output by the image recognition model based on machine learning. The confidence probability value can indicate the possibility that the detected target is a real person target. Only when the confidence probability value of a person target is greater than a preset confidence threshold, the matching relationship between it and the first sound source localization angle information will be considered.
[0099] The step S300: based on the angle information to be matched of each person, determining whether there is angle information of a person matching the first sound source localization angle information, and the first sound source localization angle information and the angle information of the matched person are both within a preset sound curtain angle range, comprises the following steps:
[0100] Filtering the first sound source positioning angle and the angle information of the person to be matched within the angle range of the sound curtain as the angle information of the candidate first sound source positioning angle and the candidate person;
[0101] Specifically, since the first sound source localization angle may be affected by the external environment during detection and deviate, when judging whether the first sound source localization angle is within the sound curtain angle range, a deviation range of positive and negative DeltaAngle degrees may be added to the sound curtain angle range, that is, judging whether the first sound source localization angle is within the range of [m1-DeltaAngle, m2+DeltaAngle]. If so, it is considered that the first sound source localization angle information is within the sound curtain angle range;
[0102] When screening the angle information of a person to be matched, since the angle information to be matched is calculated based on the pixel coordinates obtained by image recognition, it is relatively accurate. It is not necessary to set the deviation range of positive and negative DeltaAngle degrees. It can directly determine whether the angle of the person to be matched is within the range of [m1, m2], or it can also determine whether the angle of the person to be matched is within the range of [m1-DeltaAngle, m2+DeltaAngle].
[0103] It is determined whether there is angle information of a candidate person that matches the angle information of the candidate first sound source localization and the confidence probability corresponding to the angle information of the matched candidate person is greater than a preset confidence threshold. In this embodiment, a preset confidence threshold may be set, and the angle information of the candidate person whose confidence probability is greater than the preset confidence threshold is calculated with the angle information of the candidate first sound source localization, and the angle information of one or more candidate persons with the smallest angle difference is used as the angle information of the matched person.
[0104] In this embodiment, the step S400: enhancing the sound source audio corresponding to the first sound source localization angle information includes the following steps:
[0105] According to the angle information of the matched person, the first sound source localization angle information is adjusted to obtain adjusted second sound source localization angle information;
[0106] Further, a second confidence threshold may be set, and only when the confidence probability of the angle information of the matched person is greater than the second confidence threshold, the first sound source localization angle information is adjusted based on the angle information of the matched person, and when the confidence probability of the matched person is less than or equal to the second confidence threshold, the first sound source localization angle information is not adjusted;
[0107] Here, the first sound source localization angle information is adjusted according to the angle information of the matched person, for example, the first sound source localization angle information is replaced by the angle information of the matched person as the second sound source localization angle information, that is, a reverse correction effect of sound source localization is achieved;
[0108] enhancing the corresponding sound source audio according to the second sound source positioning angle information;
[0109] In this embodiment, the enhancement is based on the second sound source localization angle information after reverse correction, which is further corrected based on the image detection result. Compared with the first sound source localization angle information directly obtained by sound source localization, it is more accurate and less affected by the environment. Therefore, the sound source audio enhancement processing based on the second sound source localization angle information has a better sound source audio enhancement effect, and can more accurately restore the speaking audio of the enhanced target speaker.
[0110] The values of the first confidence threshold, the second confidence threshold and the preset confidence threshold can be selected and set as needed, and can be set to the same value or to different values, all of which fall within the protection scope of the present application.
[0111] In this embodiment, during the video conference, considering that personnel changes may occur, such as the entry of new personnel or the temporary departure of personnel, in the implementation method of adaptively setting the sound screen angle range, the sound processing method may also include the step of updating the sound screen angle range according to actual conditions. In one implementation, an update time interval may be set to update the sound screen angle range in real time at fixed intervals, and after obtaining the latest angle information of all personnel during the update, the updated sound screen angle range includes the latest angle information of all personnel.
[0112] In another embodiment, considering that personnel changes during a video conference may not be particularly frequent, the frequency of updating the sound screen angle range can be reduced, and the time to update the sound screen angle range can be adaptively determined according to actual conditions to reduce the workload of the video conferencing device. Specifically, the following steps are used to determine whether to update the sound screen angle range:
[0113] Acquire a real-time person image at the current moment and perform image recognition to obtain a face recognition result at the current moment, and obtain the pixel coordinates of each person in the real-time person image at the current moment based on the face recognition result; further, when obtaining the face recognition result, also obtain the confidence probability of each face, filter out the face recognition results with a confidence probability lower than a preset confidence threshold, and use the face recognition results with a confidence probability higher than the preset confidence threshold for subsequent judgment;
[0114] Match the face recognition result at the current moment with the face in the face recognition result at the previous moment to determine whether there is a change in the number of people; for example, by determining whether the faces in the face recognition result at the previous moment can all correspond to the faces in the face recognition result at the current moment, if a new face appears at the current moment, it means that a new person has joined; if a face at the previous moment does not appear at the current moment, it means that a person has left, and the addition of new people and the departure of people are recorded as personnel changes, and the change in the number of people is counted;
[0115] Compare the pixel coordinates of each person at the current moment with the pixel coordinates at the previous moment to determine whether there is a change in the position of the person; specifically, when determining the change in the number of people in the previous step, the correspondence between the face at the current moment and the face at the previous moment is established, and compare whether the difference between the pixel coordinates corresponding to one face of the person and the pixel coordinates corresponding to the face at the previous moment is within a preset allowable difference range. If yes, it means that the position of the person has not changed, and if not, it means that the position of the person has changed; count the number of people whose positions have changed; by setting the preset allowable difference range, some people's smaller movements such as shaking their heads and turning their heads can be excluded, and when the pixel coordinate difference is small, it is considered that the position of the person has not changed;
[0116] If the change in the number of personnel and / or the change in the position of personnel does not meet the preset sound curtain angle update requirement, the sound curtain angle range is not updated at the current moment;
[0117] If the change in the number of personnel and / or the change in the position of personnel meets the preset sound curtain angle update requirements, the current moment angle information of each person is obtained based on the current moment pixel coordinates of each person (i.e., the horizontal angle and pitch angle in the Cartesian coordinate system are converted based on the current moment pixel coordinates, and then the angle information is calculated), and the sound curtain angle range is updated based on the current moment angle information, so that the updated sound curtain angle range includes the current angle information of all personnel. The updated sound curtain angle range parameters include the updated minimum sound curtain angle m1 and the maximum sound curtain angle m2.
[0118] In this embodiment, for example, it can be set that when the number of people whose number has changed is greater than a preset number threshold, the preset sound screen angle update requirement is met, or when the number of people whose positions have changed is greater than a preset number threshold, the preset sound screen angle update requirement is met, or when at least one of the current number of people whose number has changed and the number of people whose positions have changed is greater than a preset number threshold, the preset sound screen angle update requirement is met, etc.
[0119] In another embodiment, it is also possible to detect whether the position of a person close to the boundary of the sound screen angle range has changed. For example, based on the angle information of all persons, screening is performed to select persons whose angle difference with the boundary angle value of the sound screen angle range is within a certain range, and to determine whether the difference between the pixel coordinates of such persons at the current moment and the pixel coordinates at the previous moment is within a preset allowable difference range.
[0120] In this embodiment, when the sound curtain angle range is initially set and when the sound curtain angle range is updated, an allowable deviation of delta can be set. After the minimum angle value m1 and the maximum angle value m2 are set, when the sound curtain angle range is applied for judgment in the subsequent step S300, the sound curtain angle range used is actually [m1-DeltaAngle, m2+DeltaAngle], thereby avoiding slight movements of people close to the boundary line of the sound curtain angle range (such as turning the head, shaking the head, etc.) which will exceed the boundary line range and fail to achieve sound source enhancement.
[0121] like Figure 5 As shown, the embodiment of the present application also provides a sound processing system based on audio and video fusion, which is applied to the sound processing method based on audio and video fusion, and the system includes:
[0122] The sound source localization module M100 is used to obtain a sound source localization result and determine first sound source localization angle information based on the sound source localization result;
[0123] The image processing module M200 is used to obtain the image of the person to be processed, and obtain the angle information to be matched of each person based on the pixel coordinates to be matched of each person in the image of the person to be processed;
[0124] An angle matching module M300, used to determine whether there is angle information of a person matching the first sound source localization angle information based on the angle information to be matched of each person, and the first sound source localization angle information and the angle information of the matching person are both within a preset sound curtain angle range;
[0125] The sound source enhancement module M400 is used to enhance the sound source audio corresponding to the first sound source localization angle information if there is angle information of a person matching the first sound source localization angle information, and the first sound source localization angle information and the angle information of the matching person are both within a preset sound curtain angle range.
[0126] By adopting the sound processing system of the present application, the first sound source positioning angle information determined based on the sound source positioning result and the angle information of each person obtained based on the person image are obtained through the sound source positioning module M100 and the image processing module M200 respectively, and the first sound source positioning angle information, the angle information of the person and the preset sound curtain angle range are combined by the angle matching module M300 for fusion processing to determine whether it is necessary to enhance the first sound source positioning angle information, and when it is determined that enhancement is required, the corresponding sound source audio is enhanced by the sound source enhancement module M400, thereby improving the pickup effect of the corresponding sound source audio. The sound processing method can be applied to video conferencing scenarios and is implemented based on video conferencing equipment. The video conferencing equipment is provided with a camera and a microphone array. The camera is used to obtain the image of the person to be processed, and the microphone array is used to collect audio. By adopting this method, the audio of the speaker can be effectively identified and enhanced, effectively improving the video conferencing effect.
[0127] An embodiment of the present application also provides a sound processing device based on audio and video fusion, comprising a processor; a memory, in which executable instructions of the processor are stored; wherein the processor is configured to perform the steps of the sound processing method based on audio and video fusion by executing the executable instructions.
[0128] Those skilled in the art will appreciate that various aspects of the present application may be implemented as a system, method or program product. Therefore, various aspects of the present application may be specifically implemented in the following forms, namely: a complete hardware implementation, a complete software implementation (including firmware, microcode, etc.), or a combination of hardware and software, which may be collectively referred to as "circuit", "module" or "system" herein.
[0129] Refer to the following Figure 5 The electronic device 600 according to this embodiment of the present application is described. Figure 6 The electronic device 600 shown is merely an example and should not bring any limitation to the functions and scope of use of the embodiments of the present application.
[0130] like Figure 6 As shown, the electronic device 600 is in the form of a general computing device. The components of the electronic device 600 may include, but are not limited to: at least one processing unit 610, at least one storage unit 620, a bus 630 connecting different system components (including the storage unit 620 and the processing unit 610), a display unit 640, etc.
[0131] The storage unit stores a program code, which can be executed by the processing unit 610, so that the processing unit 610 performs the steps of various exemplary embodiments of the present application described in the above-mentioned sound processing method based on audio and video fusion of this specification. For example, the processing unit 610 can perform the following steps: Figure 1 Follow the steps shown in .
[0132] The storage unit 620 may include a readable medium in the form of a volatile storage unit, such as a random access memory unit (RAM) 6201 and / or a cache memory unit 6202 , and may further include a read-only memory unit (ROM) 6203 .
[0133] The storage unit 620 may also include a program / utility 6204 having a set (at least one) of program modules 6205, such program modules 6205 including but not limited to: an operating system, one or more application programs, other program modules and program data, each of which or some combination may include the implementation of a network environment.
[0134] Bus 630 may represent one or more of several types of bus structures, including a memory unit bus or memory unit controller, a peripheral bus, an accelerated graphics port, a processing unit, or a local bus using any of a variety of bus architectures.
[0135] The electronic device 600 may also communicate with one or more external devices 700 (e.g., keyboards, pointing devices, Bluetooth devices, etc.), may also communicate with one or more devices that enable a user to interact with the electronic device 600, and / or communicate with any device that enables the electronic device 600 to communicate with one or more other computing devices (e.g., routers, modems, etc.). Such communication may be performed via an input / output (I / O) interface 650. Furthermore, the electronic device 600 may also communicate with one or more networks (e.g., a local area network (LAN), a wide area network (WAN), and / or a public network, such as the Internet) via a network adapter 660. The network adapter 660 may communicate with other modules of the electronic device 600 via a bus 630. It should be understood that, although not shown in the figure, other hardware and / or software modules may be used in conjunction with the electronic device 600, including but not limited to: microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, and data backup storage systems.
[0136] By adopting the sound processing device based on audio and video fusion provided in the present application, the processor executes the sound processing method based on audio and video fusion when executing the executable instructions, thereby obtaining the beneficial effects of the above-mentioned sound processing method based on audio and video fusion.
[0137] The embodiment of the present application also provides a computer-readable storage medium for storing a program, which, when executed by a processor, implements the steps of the sound processing method based on audio and video fusion. In some possible implementations, various aspects of the present application can also be implemented in the form of a program product, which includes a program code, and when the program product is run on a terminal device, the program code is used to enable the terminal device to execute the steps of various exemplary implementations of the present application described in the above-mentioned sound processing method based on audio and video fusion section of this specification.
[0138] refer to Figure 7 As shown, a program product 800 for implementing the above method according to an embodiment of the present application is described, which can adopt a portable compact disk read-only memory (CD-ROM) and include program code, and can be run on a terminal device, such as a personal computer. However, the program product of the present application is not limited thereto. In this document, a readable storage medium can be any tangible medium containing or storing a program, which can be used by or in combination with an instruction execution system, an apparatus or a device.
[0139] The program product may use any combination of one or more readable media. The readable medium may be a readable signal medium or a readable storage medium. The readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples (non-exhaustive list) of readable storage media include: an electrical connection with one or more wires, a portable disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above.
[0140] The computer readable storage medium may include a data signal propagated in a baseband or as part of a carrier wave, wherein a readable program code is carried. This propagated data signal may take a variety of forms, including but not limited to an electromagnetic signal, an optical signal, or any suitable combination of the above. The readable storage medium may also be any readable medium other than a readable storage medium, which may send, propagate, or transmit a program for use by an instruction execution system, an apparatus, or a device or used in combination with it. The program code contained on the readable storage medium may be transmitted with any appropriate medium, including but not limited to wireless, wired, optical cable, RF, etc., or any suitable combination of the above.
[0141] Program code for performing the operations of the present application may be written in any combination of one or more programming languages, including object-oriented programming languages such as Java, C++, etc., and conventional procedural programming languages such as "C" or similar programming languages. The program code may be executed entirely on the user computing device, partially on the user device, as a separate software package, partially on the user computing device and partially on a remote computing device, or entirely on a remote computing device or cluster. In the case of a remote computing device, the remote computing device may be connected to the user computing device through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computing device (e.g., using an Internet service provider to connect through the Internet).
[0142] By adopting the computer-readable storage medium provided in the present application, the program stored therein implements the steps of the sound processing method based on audio and video fusion when executed, thereby obtaining the beneficial effects of the sound processing method based on audio and video fusion.
[0143] The above content is a further detailed description of the present application in combination with specific preferred implementation methods, and it cannot be determined that the specific implementation of the present application is limited to these descriptions. For ordinary technicians in the technical field to which the present application belongs, several simple deductions or substitutions can be made without departing from the concept of the present application, which should be deemed to fall within the scope of protection of the present application.
Claims
1. A sound processing method based on audio and video fusion, characterized in that: The steps include: Acquire a sound source localization result, and determine first sound source localization angle information based on the sound source localization result; Acquire an image of a person to be processed, and obtain angle information to be matched of each person based on the pixel coordinates to be matched of each person in the image of the person to be processed; Based on the angle information to be matched of each person, it is determined whether there is angle information of a person that matches the first sound source localization angle information, and both the first sound source localization angle information and the angle information of the matched person are within a preset sound curtain angle range; If so, the sound source audio corresponding to the first sound source positioning angle information is enhanced.
2. The sound processing method based on audio and video fusion according to claim 1 is characterized in that: The sound source localization result includes a plurality of candidate sound source localization angle information, and determining the first sound source localization angle information based on the sound source localization result includes the following steps: According to a preset abnormal screening rule, the abnormal sound source localization angle information among the multiple candidate sound source localization angle information is excluded; A cluster analysis is performed on the candidate sound source localization angle information, and a candidate sound source localization angle information is determined according to the clustering result as the first sound source localization angle information.
3. The sound processing method based on audio and video fusion according to claim 1 is characterized in that: Obtaining the to-be-matched angle information of each person based on the pixel coordinates of each person in the image of the person to be processed includes the following steps: Performing image recognition on the person image to be processed to obtain a face recognition result of the person image, and obtaining the pixel coordinates to be matched of each person in the person image based on the face recognition result; Performing coordinate conversion on the to-be-matched pixel coordinates of each person to obtain a horizontal angle and a pitch angle in a Cartesian coordinate system; The angle information to be matched of each person is calculated based on the horizontal angle and the pitch angle in the Cartesian coordinate system.
4. The sound processing method based on audio and video fusion according to claim 1 is characterized in that: It also includes presetting the sound curtain angle range by following the steps below: Acquire an initial person image, and obtain initial angle information of each person based on initial pixel coordinates of each person in the initial person image; The sound curtain angle range is adaptively set based on the initial angle information of all persons, so that the sound curtain angle range includes the initial angle information of all persons.
5. The sound processing method based on audio and video fusion according to claim 4 is characterized in that: It also includes the following steps to determine whether to update the sound curtain angle range: Acquire a real-time person image at the current moment and perform image recognition to obtain a face recognition result at the current moment, and obtain the pixel coordinates of each person in the real-time person image at the current moment based on the face recognition result; Match the face recognition result at the current moment with the face in the face recognition result at the previous moment to determine whether there is a change in the number of people; Compare the pixel coordinates of each person at the current moment with the pixel coordinates at the previous moment to determine whether there is a change in the position of the person; If the change in the number of people and / or the change in their positions meets the preset sound curtain angle update requirements, the current moment angle information of each person is obtained based on the current moment pixel coordinates of each person, and the sound curtain angle range is updated based on the current moment angle information, so that the updated sound curtain angle range includes the current angle information of all people.
6. The sound processing method based on audio and video fusion according to claim 1 is characterized in that: After obtaining the image of the person to be processed, the method further includes performing image recognition on the image of the person to be processed to obtain the confidence probability of each person; Based on the angle information to be matched of each person, it is determined whether there is angle information of a person that matches the first sound source localization angle information, and both the first sound source localization angle information and the angle information of the matched person are within a preset sound curtain angle range, including the following steps: Filtering the first sound source positioning angle and the angle information of the person to be matched within the angle range of the sound curtain as the angle information of the candidate first sound source positioning angle and the candidate person; It is determined whether there is angle information of a candidate person that matches the candidate first sound source localization angle and whether the confidence probability corresponding to the angle information of the matching candidate person is greater than a preset confidence threshold.
7. The sound processing method based on audio and video fusion according to claim 1 is characterized in that: Enhancing the sound source audio corresponding to the first sound source localization angle information comprises the following steps: According to the angle information of the matched person, the first sound source localization angle information is adjusted to obtain adjusted second sound source localization angle information; The corresponding sound source audio is enhanced according to the second sound source localization angle information.
8. A sound processing system based on audio and video fusion, characterized in that: The sound processing method based on audio and video fusion applied to any one of claims 1 to 7, the system comprising: A sound source localization module, used to obtain a sound source localization result, and determine first sound source localization angle information based on the sound source localization result; An image processing module is used to obtain an image of a person to be processed, and obtain angle information to be matched of each person based on the pixel coordinates to be matched of each person in the image of the person to be processed; An angle matching module, used for judging whether there is angle information of a person matching the first sound source localization angle information based on the angle information to be matched of each person, and the first sound source localization angle information and the angle information of the matched person are both within a preset sound curtain angle range; A sound source enhancement module is used to enhance the sound source audio corresponding to the first sound source localization angle information if there is angle information of a person matching the first sound source localization angle information, and the first sound source localization angle information and the angle information of the matching person are both within a preset sound curtain angle range.
9. A sound processing device based on audio and video fusion, characterized in that: include: processor; a memory storing executable instructions of the processor; The processor is configured to execute the steps of the sound processing method based on audio and video fusion according to any one of claims 1 to 7 by executing the executable instructions.
10. A computer-readable storage medium for storing a program, characterized in that: When the program is executed by a processor, the steps of the sound processing method based on audio and video fusion described in any one of claims 1 to 7 are implemented.
Citation Information
Cited By
Multi-modal interaction method and device suitable for group chat
CN120412595A
Unattended platform monitoring alarm system and method based on Internet platform
CN120510699A
