Sound signal acquisition method, microphone array system and storage medium

Abstract

The present application discloses a sound signal acquisition method, a microphone array system and a storage medium, and relates to the field of microphone array technology, including: acquiring sound data through a microphone array, and acquiring environmental point cloud data through a millimeter-wave radar; jointly calibrating the microphone array and the millimeter-wave radar to determine the conversion matrix between the microphone array and the millimeter-wave radar; identifying and marking the sound data to obtain true value data; projecting the environmental point cloud data through the conversion matrix to obtain the corresponding relationship between the environmental point cloud data and the true value data; using a maximum weight matching algorithm to match the true value data and the environmental point cloud data with a corresponding relationship, and determining the sound source position based on the matching result, and controlling the microphone array to acquire the sound signal based on the sound source position, so as to improve the accuracy of the determined sound source position, and further improve the accuracy of the directional adjustment of the microphone array.

Description

Technical Field

[0001] The present application relates to the field of microphone array technology, and in particular to a sound signal acquisition method, a microphone array system and a storage medium. Background Art

[0002] In the current conference environment, in order to improve the clarity and efficiency of voice communication, conference microphone arrays with automatic adjustment of directionality have become an important technical solution. These microphone arrays are usually designed with intelligent control systems that can automatically adjust the direction of the microphone according to the position of the speaker to ensure the best sound collection effect.

[0003] However, in actual operation, due to the influence of various factors, such as ambient noise in the conference room, multipath reflection, physical limitations of the microphone array, and limitations of the algorithm, the microphone array often cannot accurately align with the sound source when adjusting its direction. This deviation not only affects the quality of sound collection, but may also lead to a decline in the experience of meeting participants, especially in large or multi-source conference scenarios.

[0004] In addition, the existing automatic directional microphone arrays often have insufficient speed and accuracy in adjusting the direction when dealing with fast-moving sound sources or multiple sound sources emitting sound at the same time, which further exacerbates the problem of deviation in the adjustment direction. Therefore, although the conference microphone array with automatic directional adjustment has significant advantages in improving the effect of conference communication, the deviation in the adjustment direction is still a key technical bottleneck that restricts its performance, and it is urgent to find an effective solution to improve the accuracy and reliability of its directional adjustment. Summary of the invention

[0005] The main purpose of the present application is to provide a sound signal acquisition method, a microphone array system and a storage medium, aiming to solve the technical problem of poor accuracy of directional adjustment of a microphone array.

[0006] To achieve the above purpose, the present application proposes a sound signal collection method, which includes:

[0007] Collect sound data through microphone arrays and environmental point cloud data through millimeter-wave radars;

[0008] Jointly calibrate the microphone array and millimeter-wave radar to determine the conversion matrix between the microphone array and the millimeter-wave radar;

[0009] Identify and mark the sound data to obtain true value data;

[0010] The environmental point cloud data is projected through the transformation matrix to obtain the corresponding relationship between the environmental point cloud data and the true value data;

[0011] The maximum weight matching algorithm is used to match the true value data and the environmental point cloud data with corresponding relationships, and the sound source position is determined based on the matching results. The microphone array is controlled to collect sound signals based on the sound source position.

[0012] In one embodiment, the steps of collecting sound data through a microphone array and collecting environmental point cloud data through a millimeter wave radar include:

[0013] Acquire the sound data of each microphone in the microphone array in real time, and acquire the environmental point cloud data detected by the millimeter wave radar in real time;

[0014] Among them, sound data and environmental point cloud data are stored according to timestamps.

[0015] In one embodiment, the step of jointly calibrating the microphone array and the millimeter wave radar to determine the conversion matrix between the microphone array and the millimeter wave radar includes:

[0016] Calibrate the internal parameters of the microphone array to obtain the internal parameter matrix of the microphone array;

[0017] The external parameters of the millimeter-wave radar and microphone array are calibrated, and the rotation matrix and translation matrix between the millimeter-wave radar and the microphone array are obtained by combining the internal parameter matrix;

[0018] Among them, the transformation matrix includes the intrinsic parameter matrix, the rotation matrix and the translation matrix.

[0019] In one embodiment, the step of jointly calibrating the microphone array and the millimeter wave radar to determine the conversion matrix between the microphone array and the millimeter wave radar includes:

[0020] Calibrate the internal parameters of the microphone array to obtain the internal parameter matrix of the microphone array;

[0021] The external parameters of the millimeter-wave radar and microphone array are calibrated, and the rotation matrix and translation matrix between the millimeter-wave radar and the microphone array are obtained by combining the internal parameter matrix;

[0022] Among them, the transformation matrix includes the intrinsic parameter matrix, the rotation matrix and the translation matrix.

[0023] In one embodiment, the step of projecting the environmental point cloud data through a transformation matrix to obtain a corresponding relationship between the environmental point cloud data and the true value data includes:

[0024] According to the timestamp corresponding to each sound data, the environmental point cloud data corresponding to the sound data is searched at the timestamp to obtain the target point cloud data;

[0025] Based on the intrinsic parameter matrix, rotation matrix and translation matrix, the target point cloud data is projected to obtain the point cloud coordinate position of the target point cloud data in the sound field coordinate system of the microphone array, and then the correspondence between the target point cloud data and the true value data is determined according to the point cloud coordinate position.

[0026] In one embodiment, the maximum weight matching algorithm includes a first maximum weight matching algorithm and a second maximum weight matching algorithm. The maximum weight matching algorithm is used to match the true value data and the environmental point cloud data having a corresponding relationship, including:

[0027] An undirected graph is created by using a first maximum weight matching algorithm, and a corresponding relationship between the target point cloud data and a first true value bounding box of the true value data is obtained by using the undirected graph;

[0028] Traversing each target point cloud data, and determining whether each target point cloud data is within the first true value bounding box corresponding to the target point cloud data;

[0029] If the target point cloud data is within the first true value bounding box corresponding to the target point cloud data, the target point cloud data within the first true value bounding box corresponding to the target point cloud data is determined as the first point cloud data, a first distance between the first point cloud data and the center of the bounding box of the first true value bounding box corresponding to the first point cloud data is calculated, and a matching relationship between the first point cloud data and the true value data having a corresponding relationship is determined based on the first distance;

[0030] If the target point cloud data is not within the first true value bounding box corresponding to the target point cloud data, determining whether the target point cloud data is within a second true value bounding box established based on the first true value bounding box corresponding to the target point cloud data;

[0031] If the target point cloud data is within the second true value bounding box corresponding to the target point cloud data, the target point cloud data within the second true value bounding box corresponding to the target point cloud data is determined as the second point cloud data, and a second distance between the second point cloud data and the center of the bounding box of the second true value bounding box corresponding to the second point cloud data is calculated, and a matching relationship between the second point cloud data and the true value data having a corresponding relationship is determined based on the second distance.

[0032] In one embodiment, the step of determining the sound source position according to the obtained matching result includes:

[0033] Determine multiple sound source preselected positions based on the obtained matching results;

[0034] Calculate the distance value between any two pre-selected positions of sound sources;

[0035] After obtaining the target distance value from each distance value and determining the two target sound source pre-selected positions corresponding to the target distance value, the midpoint of the line connecting the two target sound source pre-selected positions is determined as the sound source position.

[0036] In one embodiment, the step of controlling the microphone array to collect sound signals based on the sound source position includes:

[0037] After generating an adjustment instruction based on the sound source position, adjusting the sound pickup angle of the microphone array according to the adjustment instruction;

[0038] The adjusted microphone array is controlled to collect sound signals.

[0039] In addition, to achieve the above-mentioned purpose, the present application also proposes a microphone array system, which includes: a memory, a processor, and a computer program stored in the memory and executable on the processor, and the computer program is configured to implement the steps of the sound signal acquisition method described above.

[0040] In addition, to achieve the above-mentioned purpose, the present application also proposes a storage medium, which is a computer-readable storage medium, and a computer program is stored on the storage medium. When the computer program is executed by a processor, the steps of the sound signal acquisition method described above are implemented.

[0041] One or more technical solutions proposed in this application have at least the following technical effects:

[0042] The sound data is collected through a microphone array, and the environmental point cloud data is collected through a millimeter-wave radar; the microphone array and the millimeter-wave radar are jointly calibrated to determine the conversion matrix between the microphone array and the millimeter-wave radar; the sound data is identified and marked to obtain the true value data; the environmental point cloud data is projected through the conversion matrix to obtain the corresponding relationship between the environmental point cloud data and the true value data; the maximum weight matching algorithm is used to match the true value data and the environmental point cloud data with the corresponding relationship, and the sound source position is determined based on the matching result, and the microphone array is controlled to collect the sound signal based on the sound source position.

[0043] In the data matching process, the present application combines the joint calibration results of the microphone array and the millimeter-wave radar, that is, the transformation matrix, to project the environmental point cloud data into the sound field coordinate system, obtain the correspondence between the environmental point cloud data and the true value data, and achieve spatial alignment. Then, the maximum weight matching algorithm is used to match the true value data and the environmental point cloud data with the corresponding relationship, thereby achieving high-precision matching operations between the environmental point cloud data and the true value data, thereby improving the accuracy of the determined sound source position and further improving the accuracy of the directional adjustment of the microphone array. BRIEF DESCRIPTION OF THE DRAWINGS

[0044] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present application and, together with the description, serve to explain the principles of the present application.

[0045] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, for ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative labor.

[0046] Figure 1 A flow chart of an embodiment of a sound signal acquisition method provided in the present application;

[0047] Figure 2 Schematic diagram of the device structure of the hardware operating environment involved in the sound signal acquisition method in the embodiment of the present application.

[0048] The purpose, features and advantages of this application will be further described in conjunction with the embodiments and with reference to the accompanying drawings. DETAILED DESCRIPTION

[0049] It should be understood that the specific embodiments described herein are only used to explain the technical solutions of the present application and are not used to limit the present application.

[0050] In order to better understand the technical solution of the present application, a detailed description will be given below in conjunction with the accompanying drawings and specific implementation methods.

[0051] The main solution of the embodiment of the present application is: collecting sound data through a microphone array, and collecting environmental point cloud data through a millimeter-wave radar; jointly calibrating the microphone array and the millimeter-wave radar to determine the conversion matrix between the microphone array and the millimeter-wave radar; identifying and marking the sound data to obtain true value data; projecting the environmental point cloud data through the conversion matrix to obtain the corresponding relationship between the environmental point cloud data and the true value data; using the maximum weight matching algorithm to match the true value data and the environmental point cloud data with a corresponding relationship, and determining the sound source position based on the matching result, and controlling the microphone array to collect sound signals based on the sound source position.

[0052] In actual operation, due to the influence of various factors, such as ambient noise in the conference room, multipath reflection, physical limitations of the microphone array, and limitations of the algorithm, the microphone array is often unable to accurately align with the sound source when adjusting its direction. This deviation not only affects the quality of sound collection, but may also lead to a decline in the experience of conference participants, especially in large or multi-source conference scenarios. In addition, the existing automatically adjusted directional microphone arrays often have insufficient speed and accuracy in adjusting the direction when dealing with fast-moving sound sources or multiple sound sources that sound at the same time, which further aggravates the problem of deviation in the adjustment direction. Therefore, although the conference microphone array that automatically adjusts its direction has significant advantages in improving the effect of conference communications, the deviation problem in the adjustment direction is still a key technical bottleneck that restricts its performance, and it is urgent to find an effective solution to improve the accuracy and reliability of its directional adjustment.

[0053] The present application provides a solution. During the data matching process, the joint calibration results of the microphone array and the millimeter-wave radar, namely the transformation matrix, are combined to project the environmental point cloud data into the sound field coordinate system, and the corresponding relationship between the environmental point cloud data and the true value data is obtained to achieve spatial alignment. Then, the maximum weight matching algorithm is used to match the true value data and the environmental point cloud data with the corresponding relationship, so as to achieve high-precision matching operations between the environmental point cloud data and the true value data, thereby improving the accuracy of the determined sound source position and further improving the accuracy of the directional adjustment of the microphone array.

[0054] It should be noted that the execution subject of this embodiment can be a computing service device with data processing, network communication and program running functions, such as a tablet computer, a personal computer, a mobile phone, etc., or an electronic device capable of realizing the above functions, a microphone array system, etc. The following takes the microphone array system as an example to illustrate this embodiment and the following embodiments.

[0055] Based on this, the present application embodiment provides a method for collecting sound signals. Figure 1 , Figure 1 This is a flow chart of an embodiment of a sound signal acquisition method of the present application.

[0056] In this embodiment, the sound signal collection method includes steps S10 to S50:

[0057] Step S10, collecting sound data through a microphone array and collecting environmental point cloud data through a millimeter wave radar.

[0058] It should be noted that the microphone array system in this embodiment integrates a microphone array for collecting conference sound data and a millimeter-wave radar for collecting conference environment point cloud data. By establishing a connection relationship between the controller in the microphone array system and the microphone array and the millimeter-wave radar, the controller can effectively control the microphone array and the millimeter-wave radar to collect sound data and environmental point cloud data respectively after entering the sound signal collection process, as described in detail as follows:

[0059] In a feasible implementation, step S10 includes step S11:

[0060] Step S11, acquiring the sound data of each microphone in the microphone array in real time, and acquiring the environmental point cloud data detected by the millimeter wave radar in real time, wherein the sound data and the environmental point cloud data are stored according to timestamps.

[0061] When the controller establishes correct connections with the millimeter-wave radar and the microphone array respectively, and both the millimeter-wave radar and the microphone array are able to work normally and are turned on synchronously, the controller continuously collects sound data from different microphones in the microphone array, and synchronously controls the millimeter-wave radar to collect environmental point cloud data in the current conference environment, and stores all collected sound data and environmental point cloud data according to the timestamp of the bet.

[0062] Among them, the timestamp is used for time alignment of environmental point cloud data and sound data.

[0063] It should be noted that the environmental point cloud data includes point cloud data corresponding to all objects in the current conference environment, including but not limited to point cloud data corresponding to tables, chairs, human bodies, etc.

[0064] Step S20, jointly calibrate the microphone array and the millimeter-wave radar to determine a conversion matrix between the microphone array and the millimeter-wave radar.

[0065] After calibrating the microphone internal parameters in advance, the external parameters are calibrated in the self-developed millimeter-wave radar and microphone joint calibration tool to obtain the rotation matrix and translation matrix (i.e., conversion matrix), so as to achieve spatial alignment between the microphone and the millimeter-wave radar point cloud, and provide a reliable basis for matching the scene's real point cloud data, as described below:

[0066] In a feasible implementation manner, step S20 includes steps S21-S22:

[0067] Step S21, calibrating the internal parameters of the microphone array to obtain an internal parameter matrix of the microphone array;

[0068] Step S22 calibrates the external parameters of the millimeter-wave radar and the microphone array, and obtains the rotation matrix and translation matrix between the millimeter-wave radar and the microphone array in combination with the intrinsic parameter matrix, wherein the conversion matrix includes the intrinsic parameter matrix, the rotation matrix and the translation matrix.

[0069] Assume that the coordinates of the millimeter wave radar in the spatial coding coordinate system are , , It should be noted that The coordinate system (direction coordinate system) is The posture in the coordinate system (reference coordinate system) is ,in They are pitch angle pitch, yaw angle, and roll angle roll respectively. The rotation matrix of the axis is ,exist The rotation matrix of the axis is ,exist The rotation matrix of the axis is , we can get the right day Before the coordinate system turns right Coordinate system, the specific internal parameter matrix is ​​shown in Formula 1:

[0070] ————Formula 1

[0071] The specific rotation matrix is ​​shown in Formula 2:

[0072] ————Formula 2

[0073] The specific translation matrix is ​​shown in Formula 3:

[0074] ——Formula 3

[0075] in, , and is the point cloud position coordinate of the environmental point cloud data in the directional coordinate system, K is the internal parameter matrix of the microphone, u and v are the point cloud position coordinates of the environmental point cloud data in the spatial encoding coordinate system, , and It is the point cloud position coordinates of the environment point cloud data in the spatial mapping coordinate system.

[0076] Step S30, identifying and marking the sound data to obtain true value data.

[0077] It should be noted that this embodiment combines deep learning to identify and mark the sound feature information in the sound data, so that the generated true value data is closer to the actual value, thereby improving the accuracy of the determined sound source location.

[0078] In a feasible implementation manner, step S30 includes steps S31 to S34:

[0079] Step S31, extracting features from the sound data to obtain sound feature information from the sound data.

[0080] Step S32: Use a preset deep learning target detection model to detect each sound feature information and mark the target position corresponding to each sound feature information.

[0081] Step S33, using a preset algorithm to generate a bounding box containing each target position, and recording the category and confidence score of each target position to obtain detection data.

[0082] Step S34, associating the detection data with the timestamp to obtain true value data.

[0083] First, feature extraction is performed on the sound data to extract representative sound feature information from the sound data, ensuring that the selected sound feature information covers the sound features corresponding to all sounds in the conference environment. Subsequently, the pre-trained YOLOv8 deep learning target detection model (i.e., the preset deep learning target detection model) is used to detect and process the sound feature information to be analyzed. The model automatically identifies and marks the target position corresponding to the sound feature information by analyzing the characteristics of each sound feature information. During the detection process, the YOLOv8 algorithm in the YOLOv8 deep learning target detection model will generate a bounding box containing the target position, which is used to record the category and confidence score of each target position and use it as the detection result, which can ensure the accuracy and reliability of the detection result.

[0084] In addition, all detection results will be associated with timestamp information to form a structured data set. The detection results will be used as true value data to match and statistically analyze the environmental point cloud data in subsequent steps. This process not only improves the efficiency of scene analysis, but also lays the foundation for the precise alignment and statistical analysis of subsequent data.

[0085] Step S40, projecting the environmental point cloud data through the conversion matrix to obtain the corresponding relationship between the environmental point cloud data and the true value data.

[0086] In a feasible implementation manner, step S40 includes steps S41-S42:

[0087] Step S41 , according to the timestamp corresponding to each sound data, searching for the environmental point cloud data corresponding to the sound data at the timestamp to obtain the target point cloud data.

[0088] Step S42, projecting the target point cloud data based on the intrinsic parameter matrix, the rotation matrix and the translation matrix, obtaining the point cloud coordinate position of the target point cloud data in the sound field coordinate system of the microphone array, and determining the correspondence between the target point cloud data and the true value data according to the point cloud coordinate position.

[0089] First, according to the timestamp corresponding to each sound data, find the timestamp corresponding to or closest to the environmental point cloud data, ensure that the two timestamps are as consistent as possible in time, and determine the found environmental point cloud data as the target point cloud data.

[0090] The determined target point cloud data will be converted to the spatial encoding coordinate system of the microphone to achieve spatial alignment of the data. Therefore, it is necessary to project the target point cloud data based on the intrinsic matrix, rotation matrix and translation matrix, and perform projection calculation on each point cloud data to calculate the point cloud coordinate position of each target point cloud data in the spatial encoding coordinate system, so as to determine the correspondence between the target point cloud data and the true value data.

[0091] Step S50, using a maximum weight matching algorithm, matches the true value data and the environmental point cloud data having a corresponding relationship, determines the sound source position according to the obtained matching result, and controls the microphone array to collect sound signals based on the sound source position.

[0092] In this embodiment, a maximum weight matching algorithm is required to match the true value data calculated based on step S40 with the target point cloud data. The maximum weight matching algorithm can effectively reduce the mismatch between the true value data and the target point cloud data, improve the accuracy and precision of the matching, and thus improve the accuracy of the recognition of the sound source position.

[0093] In a feasible implementation manner, steps S51 to S60:

[0094] Step S51, creating an undirected graph through a first maximum weight matching algorithm, and obtaining a corresponding relationship between the target point cloud data and a first true value bounding box of the true value data through the undirected graph.

[0095] Step S52, traversing each target point cloud data, and determining whether each target point cloud data is within a first true value boundary box corresponding to the target point cloud data.

[0096] Step S53: If the target point cloud data is within the first true value bounding box corresponding to the target point cloud data, the target point cloud data within the first true value bounding box corresponding to the target point cloud data is determined as the first point cloud data, and a first distance between the first point cloud data and the center of the bounding box of the first true value bounding box corresponding to the first point cloud data is calculated, and a matching relationship between the first point cloud data and the true value data having a corresponding relationship is determined based on the first distance.

[0097] In the first maximum weight matching process, an undirected graph is first created to represent the relationship between the first truth bounding box of the truth data and the target point cloud data. Each first truth bounding box is traversed and a node is created for each truth bounding box. The node is named gt_x, where x is the index of the truth data.

[0098] All target point cloud data are traversed, and for each target point cloud data, its pixel point cloud data coordinates in the pixel coordinate system are checked to see whether they are within the first true value bounding box of a certain true value data.

[0099] If a certain target point cloud data is within a first true value bounding box of a certain true value data, the target point cloud data within the first true value bounding box is determined as the first point cloud data, a first distance between a point of the first point cloud data and the center of the bounding box of the first true value bounding box is calculated, and a first score is calculated according to the first distance. The calculation formula of the score is shown in Formula 4.

[0100] ————Formula 4

[0101] It should be noted that score1 is the first score calculated based on the first distance, and distance1 is the first distance.

[0102] If a certain target point cloud data is not within the first true value bounding box corresponding to any true value data, the first score of the target point cloud data is directly determined to be none.

[0103] After all target point cloud data are calculated, for target point cloud data whose first score is not none, an edge is added to the undirected graph from the point of the first point cloud data to the center of the bounding box of the first true value bounding box, and the weight of the edge is the first score corresponding to the first point cloud data.

[0104] Step S54: if the target point cloud data is not within the first true value bounding box corresponding to the target point cloud data, it is determined whether the target point cloud data is within a second true value bounding box established based on the first true value bounding box corresponding to the target point cloud data.

[0105] Step S55: If the target point cloud data is within the second true value bounding box corresponding to the target point cloud data, the target point cloud data within the second true value bounding box corresponding to the target point cloud data is determined as the second point cloud data, and a second distance between the second point cloud data and the center of the bounding box of the second true value bounding box corresponding to the second point cloud data is calculated, and a matching relationship between the second point cloud data and the true value data having a corresponding relationship is determined based on the second distance.

[0106] Obtain the target point cloud data whose first score is none in step S52, that is, the target point cloud data is not in the first true value bounding box corresponding to the target point cloud data, and perform a second maximum weight matching calculation on the target point cloud data, specifically: use three quarters of the diagonal length of the first true value bounding box of the true value data as a radius, and use the first true value bounding box as the center of the circle to establish a second true value bounding box for each true value data, and determine whether the target point cloud data whose first score is none exists in the second true value bounding box corresponding to any true value data.

[0107] If the target point cloud data exists in any second true value bounding box, the target point cloud data in the second true value bounding box is determined as the second point cloud data, a second distance between the point of the second point cloud data and the center of the bounding box of the second true value bounding box is calculated, and a second score is calculated according to the second distance. The calculation formula of the score is shown in Formula 5.

[0108] ————Formula 5

[0109] It should be noted that score2 is the second score calculated based on the second distance, and distance2 is the second distance.

[0110] If a certain target point cloud data is not within the second true value bounding box corresponding to any true value data, the second score of the target point cloud data is directly determined to be none.

[0111] After calculating all the target point cloud data, for the second point cloud data whose second score is not none, an edge is added to the undirected graph from the point of the second point cloud data to the center of the bounding box of the second true value bounding box, and the weight of the edge is the second score corresponding to the second point cloud data.

[0112] From the above, we can know that through the first maximum weight matching process and the second maximum weight matching process, the matching relationship between the true value data and the target point cloud data can be determined, ensuring high-quality pairing results in subsequent analysis and statistics. This process lays the foundation for the subsequent matching result statistics and analysis.

[0113] Step S56, determining multiple sound source pre-selected positions based on the obtained matching results.

[0114] Step S57, calculating the distance value between any two pre-selected sound source positions.

[0115] Step S58, obtaining a target distance value from each distance value, and after determining two target sound source pre-selected positions corresponding to the target distance value, determining the midpoint of the line connecting the two target sound source pre-selected positions as the sound source position.

[0116] If there is a certain true value data that can match the target point cloud data, that is, the target position corresponding to the true value data is a sound source preselected position, after determining the corresponding sound source preselected position of each target position corresponding to the true value data that matches the target power data, multiple sound source preselected positions can be obtained. At this time, the distance values ​​between each sound source preselected position are calculated, and the distance value with the largest distance is obtained as the target distance value. Then, the two sound source preselected positions corresponding to the target distance value are connected, and the midpoint of the connecting line is determined as the sound source position.

[0117] Because each sound source preselected position has a deviation compared with the sound source position corresponding to the sound signal, and each microphone in the microphone array is distributed at an arbitrary position around the sound signal, the sound source preselected positions with deviations must also be distributed at an arbitrary position around the sound source position corresponding to the sound signal. The deviation directions of the two farthest sound source preselected positions must be different. Taking the midpoint of the line connecting the two farthest sound source preselected positions as the sound source position corresponding to the sound signal will inevitably eliminate part of the deviation, thereby obtaining a more accurate sound source position.

[0118] Step S59: After an adjustment instruction is generated based on the sound source position, the sound pickup angle of the microphone array is adjusted according to the adjustment instruction.

[0119] Step S60, controlling the adjusted microphone array to collect sound signals.

[0120] After determining the location of the sound source, the controller can directly generate adjustment instructions for the microphone array based on the sound source location, where the adjustment instructions include angle adjustment instructions, direction adjustment instructions, etc. At this time, the adjusted microphone array can specifically instruct the sound source location to achieve high-precision sound signal collection effects.

[0121] In this embodiment, during the data matching process, the joint calibration results of the microphone array and the millimeter-wave radar, that is, the transformation matrix, are combined to project the environmental point cloud data into the sound field coordinate system, and the correspondence between the sound data and the true value data is obtained to achieve spatial alignment. Then, the maximum weight matching algorithm is used to match the true value data and the sound data with the corresponding relationship, thereby achieving high-precision matching operations between the sound data and the true value data, thereby improving the accuracy of the determined sound source position and further improving the accuracy of the directional adjustment of the microphone array.

[0122] The present application provides a microphone array system, which includes: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the sound signal collection method in the above-mentioned embodiment one.

[0123] Reference below Figure 2 , which shows a schematic diagram of the structure of a microphone array system suitable for implementing the embodiment of the present application. The microphone array system in the embodiment of the present application may include but is not limited to mobile terminals such as laptop computers, PADs (Portable Application Description: tablet computers), PMPs (Portable Media Player: portable multimedia players), etc., and fixed terminals such as digital TVs, desktop computers, etc. Figure 2 The microphone array system shown is merely an example and should not bring any limitation to the functions and scope of use of the embodiments of the present application.

[0124] like Figure 2 As shown, the microphone array system may include a processing device 1001 (e.g., a central processing unit, a graphics processor, etc.), which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM: Read Only Memory) 1002 or a program loaded from a storage device 1003 to a random access memory (RAM: Random Access Memory) 1004. In RAM1004, various programs and data required for the operation of the microphone array system are also stored. The processing device 1001, ROM1002, and RAM1004 are connected to each other through a bus 1005. An input / output (I / O) interface 1006 is also connected to the bus. Generally, the following systems can be connected to the I / O interface 1006: an input device 1007 including, for example, a touch screen, a touchpad, a keyboard, a mouse, an image sensor, a microphone, an accelerometer, a gyroscope, etc.; an output device 1008 including, for example, a liquid crystal display (LCD: Liquid Crystal Display), a speaker, a vibrator, etc.; a storage device 1003 including, for example, a magnetic tape, a hard disk, etc.; and a communication device 1009. The communication device 1009 can allow the microphone array system to communicate with other devices wirelessly or wired to exchange data. Although the figure shows a microphone array system with various systems, it should be understood that it is not required to implement or have all the systems shown. More or fewer systems can be implemented or provided instead.

[0125] In particular, according to the embodiments disclosed in the present application, the process described above with reference to the flowchart can be implemented as a computer software program. For example, the embodiments disclosed in the present application include a computer program product, which includes a computer program carried on a computer-readable medium, and the computer program includes a program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from a network through a communication device, or installed from a storage device 1003, or installed from a ROM 1002. When the computer program is executed by the processing device 1001, the above-mentioned functions defined in the method of the embodiment disclosed in the present application are executed.

[0126] The microphone array system provided by the present application adopts the sound signal collection method in the above embodiment, which can solve the technical problem of poor accuracy of the directional adjustment of the microphone array. Compared with the prior art, the beneficial effects of the microphone array system provided by the present application are the same as the beneficial effects of the sound signal collection method provided by the above embodiment, and other technical features of the microphone array system are the same as the features disclosed in the method of the previous embodiment, which will not be repeated here.

[0127] It should be understood that the various parts disclosed in this application can be implemented by hardware, software, firmware or a combination thereof. In the description of the above embodiments, specific features, structures, materials or characteristics can be combined in any one or more embodiments or examples in a suitable manner.

[0128] The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any person skilled in the art who is familiar with the present technical field can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

[0129] The present application provides a computer-readable storage medium having computer-readable program instructions (ie, computer programs) stored thereon, and the computer-readable program instructions are used to execute the sound signal acquisition method in the above-mentioned embodiment.

[0130] The computer-readable storage medium provided in the present application may be, for example, a USB flash drive, but is not limited to electrical, magnetic, optical, electromagnetic, infrared, or semiconductor systems, systems or devices, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM: Random Access Memory), a read-only memory (ROM: Read Only Memory), an erasable programmable read-only memory (EPROM: Erasable Programmable Read Only Memory or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM: CD-Read Only Memory), an optical storage device, a magnetic storage device, or any suitable combination of the above. In this embodiment, the computer-readable storage medium may be any tangible medium containing or storing a program, which may be used by or in combination with an instruction execution system, system or device. The program code contained on the computer-readable storage medium may be transmitted using any appropriate medium, including but not limited to: wires, optical cables, RF (Radio Frequency: Radio Frequency), etc., or any suitable combination of the above.

[0131] The computer-readable storage medium may be included in the microphone array system; or may exist independently without being assembled into the microphone array system.

[0132] The computer-readable storage medium carries one or more programs. When the one or more programs are executed by the microphone array system, the microphone array system: collects sound data through the microphone array, and collects environmental point cloud data through the millimeter-wave radar; jointly calibrates the microphone array and the millimeter-wave radar to determine the conversion matrix between the microphone array and the millimeter-wave radar; identifies and marks the sound data to obtain true value data; projects the environmental point cloud data through the conversion matrix to obtain the corresponding relationship between the environmental point cloud data and the true value data; uses the maximum weight matching algorithm to match the true value data and the environmental point cloud data with the corresponding relationship, and determines the sound source position based on the matching results, and controls the microphone array to collect sound signals based on the sound source position.

[0133] Computer program code for performing the operations of the present application may be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, C++, and conventional procedural programming languages ​​such as "C" or similar programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, as a separate software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0134] The flow chart and block diagram in the accompanying drawings illustrate the possible architecture, function and operation of the system, method and computer program product according to various embodiments of the present application. In this regard, each square box in the flow chart or block diagram can represent a module, a program segment or a part of a code, and the module, the program segment or a part of the code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the square box can also occur in a sequence different from that marked in the accompanying drawings. For example, two square boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each square box in the block diagram and / or flow chart, and the combination of the square boxes in the block diagram and / or flow chart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0135] The modules involved in the embodiments described in this application may be implemented by software or hardware, wherein the name of the module does not constitute a limitation on the unit itself in some cases.

[0136] The readable storage medium provided in the present application is a computer-readable storage medium, which stores computer-readable program instructions (i.e., computer programs) for executing the above-mentioned sound signal acquisition method, and can solve the technical problem of poor precision in directional adjustment of the microphone array. Compared with the prior art, the beneficial effects of the computer-readable storage medium provided in the present application are the same as the beneficial effects of the sound signal acquisition method provided in the above-mentioned embodiment, and will not be described in detail here.

[0137] The above descriptions are only some embodiments of the present application, and are not intended to limit the patent scope of the present application. All equivalent structural changes made using the contents of the present application specification and drawings under the technical concept of the present application, or direct / indirect applications in other related technical fields are included in the patent protection scope of the present application.

Claims

1. A sound signal collection method, characterized in that: The sound signal collection method is applied to a microphone array system, wherein the microphone array system includes a microphone array and a millimeter wave radar, and the sound signal collection method includes: Collecting sound data through the microphone array, and collecting environmental point cloud data through the millimeter wave radar; Performing joint calibration on the microphone array and the millimeter-wave radar to determine a conversion matrix between the microphone array and the millimeter-wave radar, wherein the conversion matrix includes an intrinsic parameter matrix, a rotation matrix, and a translation matrix; Performing identification marking on the sound data to obtain true value data; Projecting the environmental point cloud data through the transformation matrix to obtain a corresponding relationship between the environmental point cloud data and the true value data; Using a maximum weight matching algorithm, matching the true value data and the environmental point cloud data having a corresponding relationship, determining the sound source position according to the obtained matching result, and controlling the microphone array to collect sound signals based on the sound source position; The step of identifying and marking the sound data to obtain true value data includes: Extracting features from the sound data to obtain sound feature information from the sound data; Using a preset deep learning target detection model to identify each of the sound feature information, and marking the target position corresponding to each of the sound feature information; Generate a bounding box containing each of the target positions using a preset algorithm, and record the category and confidence score of each of the target positions to obtain detection data; Associating the detection data with a timestamp to obtain the true value data; The step of projecting the environmental point cloud data by means of the transformation matrix to obtain the corresponding relationship between the environmental point cloud data and the true value data comprises: According to the timestamp corresponding to each of the sound data, searching for the environment point cloud data corresponding to the sound data at the timestamp to obtain the target point cloud data; Based on the intrinsic parameter matrix, the rotation matrix and the translation matrix, the target point cloud data is projected to obtain the point cloud coordinate position of the target point cloud data in the sound field coordinate system of the microphone array, and then the corresponding relationship between the target point cloud data and the true value data is determined according to the point cloud coordinate position; The maximum weight matching algorithm includes a first maximum weight matching algorithm and a second maximum weight matching algorithm. The step of using the maximum weight matching algorithm to match the true value data and the environmental point cloud data having a corresponding relationship includes: Creating an undirected graph by using the first maximum weight matching algorithm, and obtaining a corresponding relationship between the target point cloud data and a first true value bounding box of the true value data by using the undirected graph; Traversing each of the target point cloud data, and determining whether each of the target point cloud data is within a first true value boundary box corresponding to the target point cloud data; If the target point cloud data is within a first true value bounding box corresponding to the target point cloud data, the target point cloud data within the first true value bounding box corresponding to the target point cloud data is determined as the first point cloud data, a first distance between the first point cloud data and a center of a bounding box of the first true value bounding box corresponding to the first point cloud data is calculated, and a matching relationship between the first point cloud data and the true value data having a corresponding relationship is determined based on the first distance; If the target point cloud data is not within the first true value bounding box corresponding to the target point cloud data, determining whether the target point cloud data is within a second true value bounding box established based on the first true value bounding box corresponding to the target point cloud data; If the target point cloud data is within a second true value bounding box corresponding to the target point cloud data, the target point cloud data within the second true value bounding box corresponding to the target point cloud data is determined as second point cloud data, and a second distance between the second point cloud data and the center of the bounding box of the second true value bounding box corresponding to the second point cloud data is calculated, and a matching relationship between the second point cloud data and the true value data having a corresponding relationship is determined based on the second distance.

2. The sound signal collection method according to claim 1, characterized in that: The steps of collecting sound data through the microphone array and collecting environmental point cloud data through the millimeter wave radar include: Acquire the sound data of each microphone in the microphone array in real time, and acquire the environmental point cloud data detected by the millimeter wave radar in real time; Wherein, the sound data and the environmental point cloud data are stored according to the timestamp.

3. The sound signal collection method according to claim 2, characterized in that: The step of jointly calibrating the microphone array and the millimeter-wave radar to determine a conversion matrix between the microphone array and the millimeter-wave radar comprises: Calibrate the internal parameters of the microphone array to obtain the internal parameter matrix of the microphone array; The external parameters of the millimeter-wave radar and the microphone array are calibrated, and the rotation matrix and the translation matrix between the millimeter-wave radar and the microphone array are obtained by combining the internal parameter matrix.

4. The sound signal collection method according to claim 3, characterized in that: The step of determining the sound source position according to the obtained matching result comprises: Determine multiple sound source preselected positions based on the obtained matching results; Calculating the distance between any two pre-selected positions of the sound sources; A target distance value is obtained from each of the distance values, and after two target sound source pre-selected positions corresponding to the target distance value are determined, a midpoint of a line connecting the two target sound source pre-selected positions is determined as the sound source position.

5. The sound signal collection method according to claim 4, characterized in that: The step of controlling the microphone array to collect sound signals based on the sound source position includes: After generating an adjustment instruction based on the sound source position, adjusting the sound pickup angle of the microphone array according to the adjustment instruction; The adjusted microphone array is controlled to collect the sound signal.

6. A microphone array system, characterized in that: The microphone array system comprises: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the computer program is configured to implement the steps of the sound signal collection method according to any one of claims 1 to 5.

7. A storage medium, characterized in that: The storage medium is a computer-readable storage medium, and a computer program is stored on the storage medium. When the computer program is executed by a processor, the steps of the sound signal collection method according to any one of claims 1 to 5 are implemented.

Images