Speaker positioning method and device based on audio data and image data

Abstract

The invention discloses a spokesman positioning method and device based on audio data and image data, and relates to the technical field of spokesman positioning. According to the method, the area where the target spokesman is located is obtained through the original audio data, and finally the position of the spokesman is obtained through positioning by collecting the image data of the area, calculating the speaking feature value of each person in the area and combining the face angle information and the audio features of each person in the area. By combining the audio signal with the image information, utilizing the omnibearing characteristics of the audio signal and the accuracy of the image information, and iteratively updating the audio data, the area where the spokesman is located can be accurately obtained, and the speaking state of the spokesman is accurately described in a digital form through the speaking state coefficient. Therefore, the audio signal and the image information can be complementary, errors caused by positioning through a single audio signal or image information are avoided, and the accuracy of spokesman positioning is improved.

Description

Technical Field

[0001] The present invention relates to the technical field of speaker positioning, and in particular to a speaker positioning method and device based on audio data and image data. Background Art

[0002] In meetings or speeches, it is often necessary to collect or record the speaker's speech, which requires obtaining the speaker's position; however, most of the time there is not only one fixed speaker, and the speaker may move randomly; therefore, it is crucial to accurately locate the speaker who is speaking in a meeting or speech.

[0003] Among the existing speaker positioning technologies, one is to locate the speaker based on the one-dimensional audio dimension signal of the microphone array, but this solution cannot accurately distinguish the actual speakers among multiple targets in the same direction; the other is to locate the speaker through multiple microphones and cameras, but this solution requires the simultaneous control of multiple microphones and cameras, but the layout of the environment and equipment is relatively strict, and is subject to greater environmental limitations. Once there is a deviation in the layout, it will also lead to inaccurate speaker positioning; another is to optimize the microphone array structure to achieve a two-dimensional spatial audio positioning effect, but the optimized multi-dimensional stereo microphone array requires a large device space, so the device form integration problem is difficult to solve. Therefore, there is an urgent need for a new speaker positioning method and device to solve the defects in the existing technology. Summary of the invention

[0004] The invention provides a speaker positioning method and device based on audio data and image data to solve the technical problem of how to accurately position the speaker.

[0005] In order to solve the above technical problems, an embodiment of the present invention provides a speaker positioning method based on audio data and image data, which may include:

[0006] Obtaining third audio data and an optimal audio angle according to the currently collected original audio data;

[0007] Determining the speech area and audio features of the third audio data according to the third audio data and the optimal audio angle;

[0008] Collect image data of the speaking area to obtain the speech feature value of each person in the speaking area;

[0009] The facial angle information of each person in the speaking area is obtained, and the position of the speaker is located by combining the audio features of the third audio data and the speech feature values ​​of each person in the speaking area.

[0010] It can be understood that, compared with the prior art, the present invention obtains the third audio data and the optimal audio angle through the original audio data, and obtains the area where the target speaker is located based on this, and finally obtains the speaker's position by collecting image data of the area, calculating the speech feature value of each person in the area, and combining the facial angle information and audio features of each person in the area. The present invention combines audio signals with image information, utilizes the omnidirectional characteristics of audio signals and the accuracy of image information, and iteratively updates audio data to accurately obtain the speaker's area, and accurately describes the person's speaking state in a digital form through the speaking state coefficient, so that the audio signal and image information can complement each other, avoiding errors caused by positioning with a single audio signal or image information, and improving the accuracy of speaker positioning.

[0011] As a preferred solution, obtaining the third audio data and the optimal audio angle according to the currently collected original audio data specifically includes:

[0012] Collect raw audio data;

[0013] Extracting first audio data and second audio data from original audio data;

[0014] Acquire a first maximum audio angle based on the first audio data, and acquire a second maximum audio angle based on the second audio data;

[0015] Obtaining a comparison result based on the first maximum audio angle and the second maximum audio angle;

[0016] Based on the comparison result, the original audio data is updated, and the first audio data and the second audio data are re-extracted to obtain the updated first maximum audio angle and the second maximum audio angle, and then the updated comparison result is obtained, until the first maximum audio angle and the second maximum audio angle are equal, to obtain the third audio data and the optimal audio angle.

[0017] By iteratively updating the original audio data and comparing the maximum audio angle, the accuracy of the audio data is confirmed, providing a basis for the subsequent calculation of the speaking area based on the audio data.

[0018] As a preferred solution, extracting the first audio data and the second audio data from the original audio data specifically includes:

[0019] Eliminate audio signals that are not within the preset camera viewing angle range in the original audio data;

[0020] The first audio data and the second audio data are extracted from the original audio data after the audio signal is removed.

[0021] By eliminating audio signals that are not within the camera's viewing angle, correct complementarity between audio data and image data is ensured, so that image data in the speech area determined according to the audio data can be correctly collected.

[0022] As a preferred solution, obtaining the first maximum audio angle based on the first audio data and obtaining the second maximum audio angle based on the second audio data specifically includes:

[0023] Obtain first audio angle statistical data based on the first audio data, and obtain second audio angle statistical data based on the second audio data;

[0024] Calculate the first maximum audio angle according to the proportion of each first audio angle in the time dimension in the first audio angle statistical data and the audio energy value of each first audio angle;

[0025] The second maximum audio angle is calculated based on the proportion of each second audio angle in the time dimension in the second audio angle statistical data and the audio energy value of each second audio angle.

[0026] By using the proportion of the audio angle in the time dimension and the corresponding audio energy value, the accuracy of the extracted maximum audio angle is guaranteed in terms of both the time dimension and the energy value, thereby ensuring the accuracy of the audio data determined based on the comparison result of the maximum audio angle.

[0027] As a preferred solution, the method of obtaining the third audio data and the optimal audio angle until the first maximum audio angle is equal to the second maximum audio angle specifically includes:

[0028] When the first maximum audio angle and the second maximum audio angle are equal, using the current original audio data as the third audio data;

[0029] The first maximum audio angle or the second maximum audio angle is used as the optimal audio angle.

[0030] By comparing the maximum audio angles of the two audio data, the correctness of the current original audio data and the optimal audio angle is ensured, so as to ensure the correctness of the speech area and audio features subsequently calculated based on the current original audio data and the optimal audio angle.

[0031] As a preferred solution, determining the speech area and the audio features of the third audio data according to the third audio data and the optimal audio angle specifically includes:

[0032] Obtaining third audio angle statistics and a third audio angle range according to the third audio data;

[0033] Obtaining an optimal statistical value of an optimal audio angle according to the third audio angle statistical data;

[0034] Determine the audio features of the third audio data based on the proportion of the optimal statistical value in the third audio angle statistical data in combination with a preset proportion threshold;

[0035] According to the preset first conversion algorithm, the speech area corresponding to the third audio data is determined in combination with the third audio angle range.

[0036] By calculating the audio features and speaking area, the mapping of audio data to image data is realized, the characteristics of audio data and image data are complemented, and the accuracy of subsequent speaker positioning is improved.

[0037] As a preferred solution, the collecting of image data of the speech area to obtain the speech feature value of each person in the speech area specifically includes:

[0038] Input the image data into a preset face detection model to obtain face region information of each person in the speaking area;

[0039] Inputting the facial region information into a preset key point detection model in sequence, and extracting the lip key point coordinates of each face;

[0040] Input the coordinates of the key points of the lips into the preset lip movement judgment model in sequence to obtain the lip movement value of each person;

[0041] The lip movement value of each person is converted to obtain the speech feature value of each person in the speaking area.

[0042] By using multiple image processing models to process the image data, it is possible to accurately extract the speech feature value of each person from the image data and digitize the speech features of each person.

[0043] As a preferred solution, the lip key point coordinates are sequentially input into a preset lip movement judgment model to obtain the lip movement value of each person, specifically including:

[0044] Input the coordinates of the key points of the lips into the preset lip movement judgment model in sequence to obtain a number of lip opening relationship values ​​for each person;

[0045] According to the time series of each lip switch coefficient value, combined with the preset lip movement coefficient, the lip movement value of each person is calculated.

[0046] Through the time series, the judgment process of lip movement in the face is optimized from the time dimension, and the comprehensive detection and recognition of lip movement is achieved.

[0047] As a preferred solution, the acquiring of facial angle information of each person in the speaking area and locating the position of the speaker in combination with the audio features of the third audio data and the speech feature values ​​of each person in the speaking area specifically includes:

[0048] According to the facial angle information of each person in the speaking area, obtain the stable state of each person's facial angle;

[0049] When the audio feature of the third audio data is a preset audio feature, the position of the speaker is located according to the speech feature value of each person and the stable state of the face angle of each person.

[0050] By combining the stable state of the facial angle and speech feature values ​​of each person in the speaking area with the audio features, the speaker can be accurately positioned from multiple angles.

[0051] Accordingly, an embodiment of the present invention provides a speaker positioning device based on audio data and image data, which may include: an audio information calculation module, an audio information conversion module, a speech feature value calculation module, and a speaker positioning module;

[0052] The audio information calculation module is used to obtain the third audio data and the optimal audio angle according to the currently collected original audio data;

[0053] The audio information conversion module is used to determine the speech area and the audio features of the third audio data according to the third audio data and the optimal audio angle;

[0054] The speech feature value calculation module is used to collect image data of the speech area and obtain the speech feature value of each person in the speech area;

[0055] The speaker positioning module is used to obtain the facial angle information of each person in the speaking area, and locate the position of the speaker in combination with the audio features corresponding to the third audio data and the speech feature values ​​of each person in the speaking area.

[0056] It can be understood that, compared with the prior art, the present device obtains the third audio data and the optimal audio angle through the original audio data, and obtains the area where the target speaker is located based on this, and finally obtains the speaker's position by collecting image data of the area, calculating the speech feature value of each person in the area, and combining the facial angle information and audio features of each person in the area. The present device combines audio signals with image information, utilizes the omnidirectional characteristics of audio signals and the accuracy of image information, and iteratively updates audio data to accurately obtain the speaker's area, and accurately describes the person's speaking state in a digital form through the speaking state coefficient, so that the audio signal and image information can complement each other, avoiding errors caused by positioning with a single audio signal or image information, and improving the accuracy of speaker positioning. BRIEF DESCRIPTION OF THE DRAWINGS

[0057] Figure 1 : A flow chart of the steps of a speaker positioning method based on audio data and image data provided by an embodiment of the present invention;

[0058] Figure 2 : A flow chart of determining a target range based on an audio signal provided by an embodiment of the present invention;

[0059] Figure 3 : A schematic diagram of a flow chart of judging a speaking state based on an image provided by an embodiment of the present invention;

[0060] Figure 4 : A schematic diagram of the structure of a speaker positioning device based on audio data and image data provided by an embodiment of the present invention;

[0061] Among them, 201 is an audio information calculation module; 202 is an audio information conversion module; 203 is a speech feature value calculation module; 204 is a speaker positioning module. DETAILED DESCRIPTION

[0062] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0063] In formal occasions such as meetings or speeches, it is often necessary to record or tape the speaker's speech, which involves determining the speaker's specific location. However, in reality, the speaker is not always fixed in one location, and the speaker may move around the conference room; the speaker is not always a fixed person, and people in different locations may take turns speaking. Therefore, how to efficiently and accurately locate the speaker's position is critical to ensuring the complete recording and subsequent arrangement of the meeting content.

[0064] Embodiment 1

[0065] Please refer to Figure 1 , is a flowchart of a method for speaker localization based on audio data and image data provided by an embodiment of the present invention, including steps S101-S104.

[0066] Step S101: Obtain third audio data and an optimal audio angle based on the currently collected original audio data.

[0067] In this embodiment, obtaining the third audio data and the optimal audio angle according to the currently collected original audio data specifically includes:

[0068] Collect raw audio data;

[0069] Extracting first audio data and second audio data from original audio data;

[0070] Acquire a first maximum audio angle based on the first audio data, and acquire a second maximum audio angle based on the second audio data;

[0071] Obtaining a comparison result based on the first maximum audio angle and the second maximum audio angle;

[0072] Based on the comparison result, the original audio data is updated, and the first audio data and the second audio data are re-extracted to obtain the updated first maximum audio angle and the second maximum audio angle, and then the updated comparison result is obtained, until the first maximum audio angle and the second maximum audio angle are equal, to obtain the third audio data and the optimal audio angle.

[0073] By iteratively updating the original audio data and comparing the maximum audio angle, the accuracy of the audio data is confirmed, providing a basis for the subsequent calculation of the speaking area based on the audio data.

[0074] In an optional embodiment, the original audio data refers to audio data with an audio data signal of 2.8s; the first audio data refers to the first audio data of 3s with data being transmitted 56 times when the audio signal is transmitted once every 55ms; the second audio data refers to the first audio data of 1.1s with data being transmitted 20 times when the audio signal is transmitted once every 55ms.

[0075] In this embodiment, extracting the first audio data and the second audio data from the original audio data specifically includes:

[0076] Eliminate audio signals that are not within the preset camera viewing angle range in the original audio data;

[0077] The first audio data and the second audio data are extracted from the original audio data after the audio signal is removed.

[0078] By eliminating audio signals that are not within the camera's viewing angle, correct complementarity between audio data and image data is ensured, so that image data in the speech area determined according to the audio data can be correctly collected.

[0079] In this embodiment, acquiring the first maximum audio angle based on the first audio data and acquiring the second maximum audio angle based on the second audio data specifically includes:

[0080] Obtain first audio angle statistical data based on the first audio data, and obtain second audio angle statistical data based on the second audio data;

[0081] Calculate the first maximum audio angle according to the proportion of each first audio angle in the time dimension in the first audio angle statistical data and the audio energy value of each first audio angle;

[0082] The second maximum audio angle is calculated based on the proportion of each second audio angle in the time dimension in the second audio angle statistical data and the audio energy value of each second audio angle.

[0083] By using the proportion of the audio angle in the time dimension and the corresponding audio energy value, the accuracy of the extracted maximum audio angle is guaranteed in terms of both the time dimension and the energy value, thereby ensuring the accuracy of the audio data determined based on the comparison result of the maximum audio angle.

[0084] In an optional embodiment, obtaining first audio angle statistical data based on the first audio data and obtaining second audio angle statistical data based on the second audio data specifically includes:

[0085] According to the order of the audio signals in the first audio data, a preset reference value is assigned to each first audio signal in the first audio data to obtain first audio angle data;

[0086] Performing statistics on the distribution of each first audio signal based on a histogram statistical method, and combining the first audio angle data to obtain first audio angle statistical data;

[0087] According to the order of the audio signals in the second audio data, a preset reference value is assigned to each second audio signal in the second audio data to obtain second audio angle data;

[0088] The distribution of each second audio signal is statistically analyzed based on a histogram statistical method, and the second audio angle data is combined to obtain second audio angle statistical data.

[0089] In an optional embodiment, the first maximum audio angle is calculated according to the proportion of each first audio angle in the time dimension in the first audio angle statistical data, combined with the audio energy value of each first audio angle; the second maximum audio angle is calculated according to the proportion of each second audio angle in the time dimension in the second audio angle statistical data, combined with the audio energy value of each second audio angle, specifically including:

[0090] Get the preset audio calculation formula:

[0091] ratio(doa)=energy / 30+1+time_ratio,

[0092] time_ratio=1 / (total_doa_num / 10)*(total_doa_num-id) / 10,

[0093] Threshold(doa)=total(doa) / total_doa_num,

[0094] Among them, ratio(doa) is the ratio of the audio angle value, energy is the audio energy value, time_ratio is the ratio of the audio angle in the time dimension, total_doa_num is the total number of audio signals in this time period, and Threshold(doa) is the number of audio angles with doa in this time period;

[0095] According to the audio calculation formula, obtain the proportion value and audio energy value of each first audio angle in the time dimension in the first audio angle statistical data, and then obtain the first maximum audio angle;

[0096] According to the audio calculation formula, the proportion value and audio energy value of each second audio angle in the time dimension in the second audio angle statistical data are obtained, and then the second maximum audio angle is obtained.

[0097] In this embodiment, the step of obtaining the third audio data and the optimal audio angle until the first maximum audio angle is equal to the second maximum audio angle specifically includes:

[0098] When the first maximum audio angle and the second maximum audio angle are equal, using the current original audio data as the third audio data;

[0099] The first maximum audio angle or the second maximum audio angle is used as the optimal audio angle.

[0100] By comparing the maximum audio angles of the two audio data, the correctness of the current original audio data and the optimal audio angle is ensured, so as to ensure the correctness of the speech area and audio features subsequently calculated based on the current original audio data and the optimal audio angle.

[0101] Step S102: Determine the speech area and audio features of the third audio data according to the third audio data and the optimal audio angle.

[0102] In this embodiment, determining the speech area and the audio features of the third audio data according to the third audio data and the optimal audio angle specifically includes:

[0103] Obtaining third audio angle statistics and a third audio angle range according to the third audio data;

[0104] Obtaining an optimal statistical value of an optimal audio angle according to the third audio angle statistical data;

[0105] Determine the audio features of the third audio data based on the proportion of the optimal statistical value in the third audio angle statistical data in combination with a preset proportion threshold;

[0106] According to the preset first conversion algorithm, the speech area corresponding to the third audio data is determined in combination with the third audio angle range.

[0107] By calculating the audio features and speaking area, the mapping of audio data to image data is realized, the characteristics of audio data and image data are complemented, and the accuracy of subsequent speaker positioning is improved.

[0108] In an optional embodiment, the determining the audio feature of the third audio data based on the proportion of the optimal statistical value in the third audio angle statistical data in combination with a preset proportion threshold specifically includes:

[0109] Get the preset percentage thresholds, where the threshold from silent state to speaking state is 0.6, and the threshold from speaking state to silent state is 0.05;

[0110] Based on the proportion of the optimal statistical value in the third audio angle statistical data and in combination with the proportion threshold, the audio features of the third audio data are determined, wherein the audio features include: a speaking state or a non-speaking state.

[0111] In an optional embodiment, determining the speech area corresponding to the third audio data according to the preset first conversion algorithm and in combination with the third audio angle range specifically includes:

[0112] Get the overall offset range of the sound source;

[0113] filtering the third audio angle range in combination with the overall offset range according to a preset first conversion algorithm;

[0114] Determine a corresponding speech area according to the filtered third audio angle range;

[0115] The overall offset range of the sound source may be determined by the sound generation condition of the sound source;

[0116] The preset first conversion algorithm refers to the conversion relationship between the audio angle and the image angle:

[0117] img_doa=90+dev+(center_pixel-center_img_x) / focus,

[0118] Among them, dev refers to the optical center offset value under different magnifications. When the focus is fixed, the optical center offset value is 0; center_pixel refers to the center pixel of the target in the picture; center_img_x refers to the actual pixel center of the picture, and img_doa refers to the image conversion angle of the portrait target.

[0119] In an alternative embodiment, please refer to Figure 2 , which is a flowchart of determining a target range based on an audio signal provided by an embodiment of the present invention, wherein an audio signal is added to an audio queue, and the audio data within the camera field of view is screened for the audio queue; then, the long-time audio signal is extracted for block statistics to obtain a first angle corresponding to the maximum statistical value, and the short-time audio signal is extracted for block statistics to obtain a second angle corresponding to the maximum statistical value; the first angle and the second angle are compared for consistency, and a threshold judgment is made according to the comparison result, and then the speaking state or the non-speaking state is switched; wherein the long-time audio signal and the short-time audio signal can be determined by a technician according to the actual operating environment.

[0120] Step S103: Collect image data of the speech area to obtain speech feature values ​​of each person in the speech area.

[0121] In this embodiment, the collecting of image data of the speech area to obtain the speech feature value of each person in the speech area specifically includes:

[0122] Input the image data into a preset face detection model to obtain face region information of each person in the speaking area;

[0123] Inputting the facial region information into a preset key point detection model in sequence, and extracting the lip key point coordinates of each face;

[0124] Input the coordinates of the key points of the lips into the preset lip movement judgment model in sequence to obtain the lip movement value of each person;

[0125] The lip movement value of each person is converted to obtain the speech feature value of each person in the speaking area.

[0126] By using multiple image processing models to process the image data, it is possible to accurately extract the speech feature value of each person from the image data and digitize the speech features of each person.

[0127] In this embodiment, the lip key point coordinates are sequentially input into a preset lip movement judgment model to obtain the lip movement value of each person, specifically including:

[0128] Input the coordinates of the key points of the lips into the preset lip movement judgment model in sequence to obtain a number of lip opening relationship values ​​for each person;

[0129] According to the time series of each lip switch coefficient value, combined with the preset lip movement coefficient, the lip movement value of each person is calculated.

[0130] Through the time series, the judgment process of lip movement in the face is optimized from the time dimension, and the comprehensive detection and recognition of lip movement is achieved.

[0131] In an alternative embodiment, please refer to Figure 3 , is a flow chart of judging the speech status based on an image provided by an embodiment of the present invention, wherein frame 0, frame step-1 and frame 2step-1 refer to different sequentially progressive time steps under the timing logic, and in the first time step, a face detection model is used to perform face detection on each face frame, and then a key point detection model is used to perform key point detection on each face frame. As the time step progresses, after the key point detection, the targets at different times are matched, the lip movement judgment model is used to calculate the lip closure coefficient, and the lip closure coefficient is post-processed.

[0132] In an optional embodiment, a SVM classification model is used to calculate the lip opening relationship value, wherein a coefficient value greater than 0.3 indicates an open mouth, and a coefficient value less than 0.1 indicates a closed mouth.

[0133] In an optional embodiment, the time series of each lip switch coefficient value, the mouth opening coefficient value and the mouth closing coefficient value are used as a sliding window, and the lip movement value of each person is calculated using the window sliding method;

[0134] Among them, the window sliding method is used to calculate the lip movement value of each person, including:

[0135] move_ratio=abs(a[1]-a[2]+abs(a[2]-a[3])+...+abs(a[n-1]-a[n]) / n-1;

[0136] When move_ratio>0.2, the person is in the speaking state; move_ratio is the lip movement value, a[n] is the lip opening relationship value calculated based on the lip key points at the nth moment, abs(a[n-1]-a[n]) is the difference between the lip states at the n-1th moment and the nth moment, and n is the time axis.

[0137] Step S104: acquiring the facial angle information of each person in the speaking area, and locating the position of the speaker by combining the audio features of the third audio data and the speech feature values ​​of each person in the speaking area.

[0138] In this embodiment, the facial angle information of each person in the speaking area is obtained, and the position of the speaker is located by combining the audio features of the third audio data and the speech feature values ​​of each person in the speaking area, specifically including:

[0139] According to the facial angle information of each person in the speaking area, obtain the stable state of each person's facial angle;

[0140] When the audio feature of the third audio data is a preset audio feature, the position of the speaker is located according to the speech feature value of each person and the stable state of the face angle of each person.

[0141] By combining the stable state of the facial angle and speech feature values ​​of each person in the speaking area with the audio features, the speaker can be accurately positioned from multiple angles.

[0142] In an optional embodiment, when the audio features of the third audio data are in a stable state, that is, in a speaking state, the audio features of the third audio data are superimposed with the stable state of each person's facial angle, and the position of the speaker is located based on each person's speaking feature value.

[0143] In an optional embodiment, when the audio feature of the third audio data is in an unstable state, that is, in a silent state, the preset percentage threshold is modified, the original audio data is collected again, and the third audio data is updated.

[0144] This embodiment obtains the third audio data and the optimal audio angle through the original audio data, and obtains the area where the target speaker is located based on this. Finally, by collecting image data of the area, the speech feature value of each person in the area is calculated, and the position of the speaker is located by combining the facial angle information and audio features of each person in the area. This embodiment combines audio signals with image information, utilizes the omnidirectional characteristics of audio signals and the accuracy of image information, and iteratively updates audio data to accurately obtain the area where the speaker is located. Through the speech state coefficient, the speech state of the person is accurately described in a digital form, so that the audio signal and image information can complement each other, avoiding errors caused by positioning with a single audio signal or image information, and improving the accuracy of speaker positioning.

[0145] Embodiment 2

[0146] Please refer to Figure 4 , which is a structural schematic diagram of a speaker positioning device based on audio data and image data provided by an embodiment of the present invention, including: an audio information calculation module 201, an audio information conversion module 202, a speech feature value calculation module 203, and a speaker positioning module 204.

[0147] The audio information calculation module 201 is used to obtain the third audio data and the optimal audio angle according to the currently collected original audio data.

[0148] In this embodiment, the audio information calculation module 201 includes: an audio information calculation unit;

[0149] The audio information calculation unit is used to collect original audio data;

[0150] Extracting first audio data and second audio data from original audio data;

[0151] Acquire a first maximum audio angle based on the first audio data, and acquire a second maximum audio angle based on the second audio data;

[0152] Obtaining a comparison result based on the first maximum audio angle and the second maximum audio angle;

[0153] Based on the comparison result, the original audio data is updated, and the first audio data and the second audio data are re-extracted to obtain the updated first maximum audio angle and the second maximum audio angle, and then the updated comparison result is obtained, until the first maximum audio angle and the second maximum audio angle are equal, to obtain the third audio data and the optimal audio angle.

[0154] In this embodiment, the audio information calculation unit includes: an audio data extraction subunit;

[0155] The audio data extraction subunit is used to remove audio signals that are not within a preset camera viewing angle range from the original audio data;

[0156] The first audio data and the second audio data are extracted from the original audio data after the audio signal is removed.

[0157] In this embodiment, the audio information calculation unit includes: a maximum audio angle calculation subunit;

[0158] The maximum audio angle calculation subunit is used to obtain first audio angle statistics based on the first audio data, and obtain second audio angle statistics based on the second audio data;

[0159] Calculate the first maximum audio angle according to the proportion of each first audio angle in the time dimension in the first audio angle statistical data and the audio energy value of each first audio angle;

[0160] The second maximum audio angle is calculated based on the proportion of each second audio angle in the time dimension in the second audio angle statistical data and the audio energy value of each second audio angle.

[0161] In this embodiment, the audio information calculation unit includes: a final audio information acquisition subunit;

[0162] The final audio information acquisition subunit is used for using the current original audio data as the third audio data when the first maximum audio angle and the second maximum audio angle are equal;

[0163] The first maximum audio angle or the second maximum audio angle is used as the optimal audio angle.

[0164] The audio information conversion module 202 is used to determine the speech area and the audio features of the third audio data according to the third audio data and the optimal audio angle.

[0165] In this embodiment, the audio information conversion module 202 includes: an audio information conversion unit;

[0166] The audio information conversion unit is used to obtain third audio angle statistics and a third audio angle range according to the third audio data;

[0167] Obtaining an optimal statistical value of an optimal audio angle according to the third audio angle statistical data;

[0168] Determine the audio features of the third audio data based on the proportion of the optimal statistical value in the third audio angle statistical data in combination with a preset proportion threshold;

[0169] According to the preset first conversion algorithm, the speech area corresponding to the third audio data is determined in combination with the third audio angle range.

[0170] The speech feature value calculation module 203 is used to collect image data of the speech area and obtain the speech feature value of each person in the speech area.

[0171] In this embodiment, the speech feature value calculation module 203 includes: a speech feature value calculation unit;

[0172] The speech feature value calculation unit is used to input the image data into a preset face detection model to obtain face area information of each person in the speech area;

[0173] Inputting the facial region information into a preset key point detection model in sequence, and extracting the lip key point coordinates of each face;

[0174] Input the coordinates of the key points of the lips into the preset lip movement judgment model in sequence to obtain the lip movement value of each person;

[0175] The lip movement value of each person is converted to obtain the speech feature value of each person in the speaking area.

[0176] In this embodiment, the speech feature value calculation unit includes: a lip movement value calculation subunit;

[0177] The lip movement value calculation subunit is used to input the coordinates of the lip key points into the preset lip movement judgment model in sequence to obtain a number of lip opening relationship values ​​of each person;

[0178] According to the time series of each lip switch coefficient value, combined with the preset lip movement coefficient, the lip movement value of each person is calculated.

[0179] The speaker positioning module 204 is used to obtain the facial angle information of each person in the speaking area, and locate the position of the speaker in combination with the audio features corresponding to the third audio data and the speech feature values ​​of each person in the speaking area.

[0180] In this embodiment, the speaker locating module 204 includes: a speaker locating unit;

[0181] The speaker positioning unit is used to obtain the stable state of each person's face angle according to the face angle information of each person in the speaking area;

[0182] When the audio feature of the third audio data is a preset audio feature, the position of the speaker is located according to the speech feature value of each person and the stable state of the face angle of each person.

[0183] This embodiment obtains the third audio data and the optimal audio angle through the original audio data, and obtains the area where the target speaker is located based on this. Finally, by collecting image data of the area, the speech feature value of each person in the area is calculated, and the position of the speaker is located by combining the facial angle information and audio features of each person in the area. This embodiment combines audio signals with image information, utilizes the omnidirectional characteristics of audio signals and the accuracy of image information, and iteratively updates audio data to accurately obtain the area where the speaker is located. Through the speech state coefficient, the speech state of the person is accurately described in a digital form, so that the audio signal and image information can complement each other, avoiding errors caused by positioning with a single audio signal or image information, and improving the accuracy of speaker positioning.

[0184] In summary, this embodiment obtains the third audio data and the optimal audio angle through the original audio data, and obtains the area where the target speaker is located based on this. Finally, by collecting image data of the area, the speech feature value of each person in the area is calculated, and the facial angle information and audio features of each person in the area are combined to locate the position of the speaker. This embodiment combines audio signals with image information, utilizes the omnidirectional characteristics of audio signals and the accuracy of image information, and iteratively updates audio data to accurately obtain the area where the speaker is located. Through the speech state coefficient, the speaking state of the person is accurately described in a digital form, so that the audio signal and image information can complement each other, avoiding errors caused by positioning with a single audio signal or image information, and improving the accuracy of speaker positioning.

[0185] The specific embodiments described above further illustrate the purpose, technical solutions and beneficial effects of the present invention. It should be understood that the above description is only a specific embodiment of the present invention and is not intended to limit the scope of protection of the present invention. It is particularly pointed out that for those skilled in the art, any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention should be included in the scope of protection of the present invention.

Claims

1. A speaker positioning method based on audio data and image data, characterized in that: include: Obtaining third audio data and an optimal audio angle according to the currently collected original audio data; Determining the speech area and audio features of the third audio data according to the third audio data and the optimal audio angle; Collect image data of the speaking area to obtain the speech feature value of each person in the speaking area; The facial angle information of each person in the speaking area is obtained, and the position of the speaker is located by combining the audio features of the third audio data and the speech feature values ​​of each person in the speaking area.

2. A speaker positioning method based on audio data and image data as claimed in claim 1, characterized in that: The obtaining of the third audio data and the optimal audio angle according to the currently collected original audio data specifically includes: Collect raw audio data; Extracting first audio data and second audio data from original audio data; Acquire a first maximum audio angle based on the first audio data, and acquire a second maximum audio angle based on the second audio data; Obtaining a comparison result based on the first maximum audio angle and the second maximum audio angle; Based on the comparison result, the original audio data is updated, and the first audio data and the second audio data are re-extracted to obtain the updated first maximum audio angle and the second maximum audio angle, and then the updated comparison result is obtained, until the first maximum audio angle and the second maximum audio angle are equal, to obtain the third audio data and the optimal audio angle.

3. A speaker positioning method based on audio data and image data as claimed in claim 2, characterized in that: The step of extracting the first audio data and the second audio data from the original audio data specifically includes: Eliminate audio signals that are not within the preset camera viewing angle range in the original audio data; The first audio data and the second audio data are extracted from the original audio data after the audio signal is removed.

4. A speaker positioning method based on audio data and image data as claimed in claim 2, characterized in that: The obtaining of a first maximum audio angle based on the first audio data and obtaining of a second maximum audio angle based on the second audio data specifically includes: Obtain first audio angle statistical data based on the first audio data, and obtain second audio angle statistical data based on the second audio data; Calculate the first maximum audio angle according to the proportion of each first audio angle in the time dimension in the first audio angle statistical data and the audio energy value of each first audio angle; The second maximum audio angle is calculated based on the proportion of each second audio angle in the time dimension in the second audio angle statistical data and the audio energy value of each second audio angle.

5. The speaker positioning method based on audio data and image data as claimed in claim 2, characterized in that: The method of obtaining the third audio data and the optimal audio angle until the first maximum audio angle is equal to the second maximum audio angle specifically includes: When the first maximum audio angle and the second maximum audio angle are equal, using the current original audio data as the third audio data; The first maximum audio angle or the second maximum audio angle is used as the optimal audio angle.

6. A speaker positioning method based on audio data and image data as claimed in claim 1, characterized in that: The determining, according to the third audio data and the optimal audio angle, the speech area and the audio features of the third audio data specifically includes: Obtaining third audio angle statistics and a third audio angle range according to the third audio data; Obtaining an optimal statistical value of an optimal audio angle according to the third audio angle statistical data; Determine the audio features of the third audio data based on the proportion of the optimal statistical value in the third audio angle statistical data in combination with a preset proportion threshold; According to the preset first conversion algorithm, the speech area corresponding to the third audio data is determined in combination with the third audio angle range.

7. A speaker positioning method based on audio data and image data as claimed in claim 1, characterized in that: The collecting of image data of the speech area to obtain the speech feature value of each person in the speech area specifically includes: Input the image data into a preset face detection model to obtain face region information of each person in the speaking area; Inputting the facial region information into a preset key point detection model in sequence, and extracting the lip key point coordinates of each face; Input the coordinates of the key points of the lips into the preset lip movement judgment model in sequence to obtain the lip movement value of each person; The lip movement value of each person is converted to obtain the speech feature value of each person in the speaking area.

8. A speaker location method based on audio data and image data as claimed in claim 7, characterized in that: The step of sequentially inputting the lip key point coordinates into a preset lip movement judgment model to obtain the lip movement value of each person specifically includes: Input the coordinates of the key points of the lips into the preset lip movement judgment model in sequence to obtain a number of lip opening relationship values ​​for each person; According to the time series of each lip switch coefficient value, combined with the preset lip movement coefficient, the lip movement value of each person is calculated.

9. The speaker positioning method based on audio data and image data as claimed in claim 6, characterized in that: The acquiring of the facial angle information of each person in the speaking area and locating the position of the speaker in combination with the audio features of the third audio data and the speech feature values ​​of each person in the speaking area specifically includes: According to the facial angle information of each person in the speaking area, obtain the stable state of each person's facial angle; When the audio feature of the third audio data is a preset audio feature, the position of the speaker is located according to the speech feature value of each person and the stable state of the face angle of each person.

10. A speaker positioning device based on audio data and image data, characterized in that: include: Audio information calculation module, audio information conversion module, speech feature value calculation module, speaker positioning module; The audio information calculation module is used to obtain the third audio data and the optimal audio angle according to the currently collected original audio data; The audio information conversion module is used to determine the speech area and the audio features of the third audio data according to the third audio data and the optimal audio angle; The speech feature value calculation module is used to collect image data of the speech area and obtain the speech feature value of each person in the speech area; The speaker positioning module is used to obtain the facial angle information of each person in the speaking area, and locate the position of the speaker in combination with the audio features of the third audio data and the speech feature values ​​of each person in the speaking area.

Images