Pickup method and device, electronic equipment and storage medium

Abstract

The embodiment of the invention discloses a pickup method and device, electronic equipment and a storage medium. The method comprises the following steps: acquiring at least one target pickup model corresponding to a target space; respectively inputting to-be-processed audios collected by the microphone array into the at least one target pickup model, and obtaining at least one pickup result which is output by the at least one target pickup model and respectively corresponds to the plurality of speakers; one pickup result corresponding to each of the plurality of speakers is a pickup result of a target pickup area corresponding to each of the plurality of speakers output by one target pickup model, and the target pickup area corresponding to each of the plurality of speakers is determined by a target resolution of the target pickup model; and based on the at least one pickup result corresponding to the plurality of speakers, determining target pickup results corresponding to the plurality of speakers. According to the method, the flexibility of directional pickup of the microphone array arranged in the target space is improved.

Description

Technical Field

[0001] The present application belongs to the technical field of audio processing, and specifically relates to a sound pickup method, device, electronic device and storage medium. Background Art

[0002] Smart conferences and other scenarios require the use of microphone arrays to achieve directional sound pickup. The directional sound pickup method behind it is usually implemented by a fixed traditional signal processing method or a trained neural network model. Regardless of which method is used, the parameters are fixed when it is put into use and can only be adapted to fixed usage scenarios, such as conference rooms of a specific size range or fixed sound pickup areas. This greatly limits the flexibility of microphone array directional sound pickup. Summary of the invention

[0003] In view of the above problems, the present application proposes a sound pickup method, device, electronic device and storage medium to improve the above problems.

[0004] In a first aspect, an embodiment of the present application provides a sound pickup method, the method comprising: obtaining at least one target sound pickup model corresponding to a target space, the target sound pickup model being a sound pickup model determined based on the size of the target space, a microphone array being provided in the target space and a plurality of speakers being included in the target space; inputting the audio to be processed collected by the microphone array into the at least one target sound pickup model respectively, and obtaining at least one sound pickup result corresponding to each of the plurality of speakers output by the at least one target sound pickup model, wherein one of the target sound pickup models outputs a sound pickup result corresponding to each of the plurality of speakers; one sound pickup result corresponding to each of the plurality of speakers is a sound pickup result of a target sound pickup area corresponding to each of the plurality of speakers output by one of the target sound pickup models, and the target sound pickup area corresponding to each of the plurality of speakers is determined by a target resolution of the target sound pickup model; and determining target sound pickup results corresponding to each of the plurality of speakers based on the at least one sound pickup result corresponding to each of the plurality of speakers.

[0005] In a second aspect, an embodiment of the present application provides a sound pickup device, the device comprising: a model acquisition unit, used to acquire at least one target sound pickup model corresponding to a target space, the target sound pickup model being a sound pickup model determined based on the size of the target space, a microphone array being arranged in the target space and a plurality of speakers being included in the target space; an output unit, used to input the to-be-processed audio collected by the microphone array into the at least one target sound pickup model, respectively, and acquire at least one sound pickup result corresponding to each of the plurality of speakers output by the at least one target sound pickup model, wherein one of the target sound pickup models outputs a sound pickup result corresponding to each of the plurality of speakers; one sound pickup result corresponding to each of the plurality of speakers is a sound pickup result of a target sound pickup area corresponding to each of the plurality of speakers output by one of the target sound pickup models, and the target sound pickup area corresponding to each of the plurality of speakers is determined by a target resolution of the target sound pickup model; a determination unit, used to determine the target sound pickup results corresponding to each of the plurality of speakers based on the at least one sound pickup result corresponding to each of the plurality of speakers.

[0006] In a third aspect, an embodiment of the present application provides an electronic device, comprising one or more processors and a memory; one or more programs, wherein the one or more programs are stored in the memory and configured to be executed by the one or more processors, and the one or more programs are configured to execute the above-mentioned method.

[0007] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium, in which a program code is stored, wherein the above method is executed when the program code is run.

[0008] In a fifth aspect, an embodiment of the present application provides a computer program product, including a computer program / instructions, which implement the steps of the above method when executed by a processor.

[0009] The embodiment of the present application provides a sound pickup method, device, electronic device and storage medium. First, at least one target sound pickup model corresponding to the target space is obtained, the target sound pickup model is a sound pickup model determined based on the size of the target space, a microphone array is provided in the target space and a plurality of speakers are included in the target space, and then the audio to be processed collected by the microphone array is respectively input into the at least one target sound pickup model, and at least one sound pickup result corresponding to each of the plurality of speakers output by the at least one target sound pickup model is obtained, wherein one of the target sound pickup models outputs a sound pickup result corresponding to each of the plurality of speakers; one sound pickup result corresponding to each of the plurality of speakers is a sound pickup result of a target sound pickup area corresponding to each of the plurality of speakers output by the target sound pickup model, and the target sound pickup area corresponding to each of the plurality of speakers is determined by the target resolution of the target sound pickup model, and finally, based on at least one sound pickup result corresponding to each of the plurality of speakers, the target sound pickup results corresponding to each of the plurality of speakers are determined. Through the above method, the target space can be divided into different numbers of pickup areas according to different target resolutions corresponding to the target space, so that the target pickup model can output pickup results corresponding to different numbers of pickup areas according to the audio to be processed, thereby improving the flexibility of the microphone array for directional pickup. BRIEF DESCRIPTION OF THE DRAWINGS

[0010] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For those skilled in the art, other drawings can be obtained based on these drawings without creative work.

[0011] Figure 1 A flow chart of a sound pickup method proposed in an embodiment of the present application is shown;

[0012] Figure 2 A flow chart of a sound pickup method proposed in another embodiment of the present application is shown;

[0013] Figure 3 A structural block diagram of a sound pickup device proposed in an embodiment of the present application is shown;

[0014] Figure 4 A structural block diagram of a sound pickup device proposed in an embodiment of the present application is shown;

[0015] Figure 5 A structural block diagram of an electronic device used to execute a sound pickup method according to an embodiment of the present application is shown in the present application in real time;

[0016] Figure 6The present invention shows a storage unit in real time for storing or carrying a program code for implementing a sound pickup method according to an embodiment of the present invention. DETAILED DESCRIPTION

[0017] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0018] Smart conferences and other scenarios require the use of microphone arrays to achieve directional sound pickup. The directional sound pickup method behind it is usually implemented by a fixed traditional signal processing method or a trained neural network model. Regardless of which method is used, the parameters are fixed when it is put into use and can only be adapted to fixed usage scenarios, such as conference rooms of a specific size range or fixed sound pickup areas. This greatly limits the flexibility of microphone array directional sound pickup.

[0019] Therefore, the inventors propose the sound pickup method, device, electronic device and storage medium in the present application. First, at least one target sound pickup model corresponding to the target space is obtained, the target sound pickup model is a sound pickup model determined based on the size of the target space, a microphone array is provided in the target space and a plurality of speakers are included in the target space, and then the audio to be processed collected by the microphone array is respectively input into the at least one target sound pickup model, and at least one sound pickup result corresponding to each of the plurality of speakers output by the at least one target sound pickup model is obtained, wherein one of the target sound pickup models outputs a sound pickup result corresponding to each of the plurality of speakers; one sound pickup result corresponding to each of the plurality of speakers is a sound pickup result of a target sound pickup area corresponding to each of the plurality of speakers output by the target sound pickup model, and the target sound pickup area corresponding to each of the plurality of speakers is determined by the target resolution of the target sound pickup model, and finally, based on at least one sound pickup result corresponding to each of the plurality of speakers, the target sound pickup results corresponding to each of the plurality of speakers are determined. Through the above method, the target space can be divided into different numbers of pickup areas according to different target resolutions corresponding to the target space, so that the target pickup model can output pickup results corresponding to different numbers of pickup areas according to the audio to be processed, thereby improving the flexibility of the microphone array for directional pickup.

[0020] The embodiments of the present application will be described in detail below with reference to the accompanying drawings.

[0021] See also Figure 1 , an embodiment of the present application provides a sound pickup method, the method comprising:

[0022] Step S110: obtaining at least one target sound pickup model corresponding to the target space, wherein the target sound pickup model is a sound pickup model determined based on the size of the target space, a microphone array is provided in the target space, and the target space includes a plurality of speakers.

[0023] In an embodiment of the present application, the target space is an environmental space in which a microphone array is provided, and the target space can be a space of any shape, such as a square conference room, a square classroom, a circular office, etc., which is not specifically limited here. The size of a target space of any shape can be represented by a length L and a width W. Therefore, after the microphone is placed in the target space, the target space can be scanned by a millimeter-wave radar to obtain the shape and information of the target space. Among them, the millimeter-wave radar can be set at the array center or other specified positions of the microphone array, which is not specifically limited here. Optionally, when a speaker enters the target space, the target space can be scanned by a millimeter-wave radar to obtain the number N of speakers included in the target space and the position (x i ,y i ), and the position of the array center itself (x c ,y c ), where i is used to represent the i-th speaker. Optionally, the position of the array center itself can be determined based on the position of each microphone in the microphone array. Specifically, a preset number of microphones are first arranged in the target space to form a microphone array, and then the positions of all microphones in the microphone array are averaged to obtain the position of the array center of the microphone array.

[0024] The target sound pickup model is a pre-trained directional sound pickup neural network model for outputting the sound pickup results corresponding to each of the multiple sound pickup areas in the target space. The sound pickup result refers to the sound within the sound pickup range corresponding to each sound pickup area. The target sound pickup model in the embodiment of the present application is a sound pickup model obtained from a preset directional sound pickup neural network model library based on the size of the target space. The preset directional sound pickup neural network model library includes multiple pre-trained directional sound pickup neural network models, each of which is adapted to a specific space size [L min ~L max ,W min ~W max ], and the spatial ranges between the models overlap to ensure that there is no size boundary. For example, for two adjacent models 1 and 2, the spatial length range of model 1 is L min1 ~L max1 , the spatial length range of model 2 is L min2 ~L max2 . Where L min1 <Lmin2 , L max1 <L max2 , then L max1 >L min2 , and similarly for W. Optionally, the preset directional sound pickup neural network model library can be pre-stored in a preset storage area. When it is necessary to obtain at least one target sound pickup model corresponding to the target space, the corresponding target sound pickup model can be obtained from the preset directional sound pickup neural network model library in the preset storage area according to the size of the target space. The preset storage area can be a storage area in a cloud server or a local server, which is not specifically limited here.

[0025] The at least one target sound pickup model corresponding to the target space can be understood as a target sound pickup model of the target space or multiple target sound pickup models (referring to two or more sound pickup models) of the target space that can be obtained from the preset directional sound pickup neural network model library.

[0026] Step S120: input the to-be-processed audio collected by the microphone array into the at least one target sound pickup model respectively, and obtain at least one sound pickup result corresponding to each of the multiple speakers output by the at least one target sound pickup model.

[0027] Among them, one of the target sound pickup models outputs a sound pickup result corresponding to each of the multiple speakers; the sound pickup result corresponding to each of the multiple speakers is a sound pickup result of a target sound pickup area corresponding to each of the multiple speakers output by the target sound pickup model, and the target sound pickup area corresponding to each of the multiple speakers is determined by the target resolution of the target sound pickup model.

[0028] In the embodiment of the present application, the audio to be processed refers to the audio collected by all microphones in the target space. The audio to be processed is multi-channel audio, and the audio collected by each microphone in the microphone array forms a channel. The resolution of the sound pickup model refers to the number of sound pickup areas into which the space is divided; the target resolution of the target sound pickup model refers to the target number of sound pickup areas corresponding to the target sound pickup model determined from the multiple resolutions corresponding to the target sound pickup model. In other words, each sound pickup model in the embodiment of the present application corresponds to multiple resolutions.

[0029] Furthermore, the sound pickup results of the target sound pickup areas corresponding to the multiple speakers output by a target sound pickup model refer to dividing the target space into a target number of sound pickup areas according to the target resolution of the target sound pickup model, and the target sound pickup model can output the target number of sound pickup results, that is, output the sound pickup results of each sound pickup area, and then the target sound pickup area corresponding to each speaker can be determined according to the position of each speaker, so as to determine the sound pickup results of the target sound pickup area corresponding to each speaker. The position of each speaker refers to the orientation of each speaker relative to the center of the array.

[0030] As a method, in response to an audio acquisition instruction, the audio to be processed is collected through a microphone array. The audio acquisition instruction may be an instruction sent by an electronic device that has established a communication connection with all microphones in the microphone array. After receiving the audio acquisition instruction, all microphones in the microphone array respond to the audio acquisition instruction and start collecting audio within a preset time period as the audio to be processed.

[0031] After obtaining the audio to be processed, if the target sound pickup model corresponding to the determined target space includes multiple, the audio to be processed is input into the multiple target sound pickup models respectively, so that each target sound pickup model can output the sound pickup results of the target number of sound pickup areas corresponding to each target sound pickup model, and then the sound pickup results of the target sound pickup areas corresponding to each speaker can be determined. Exemplarily, if the target sound pickup model corresponds to three sound pickup areas, namely, sound pickup area 1, sound pickup area 2, and sound pickup area 3, then the sound pickup results output by the target sound pickup model include the sound pickup results corresponding to sound pickup area 1, sound pickup area 2, and sound pickup area 3, that is, including sound pickup result 1 corresponding to sound pickup area 1, sound pickup result 2 corresponding to sound pickup area 2, and sound pickup result 3 corresponding to sound pickup area 3. At this time, if it is determined that the target sound pickup areas corresponding to speakers 1 and 2 are sound pickup area 2, and the target sound pickup area corresponding to speaker 3 is sound pickup area 1, then the sound pickup results corresponding to speakers 1 and 2 are sound pickup results 2, and the sound pickup results corresponding to speaker 2 are sound pickup results 1.

[0032] In the embodiment of the present application, the number of target sound pickup models corresponding to the determined target space determines the number of sound pickup results corresponding to each speaker outputted, that is, if there are 3 target sound pickup models corresponding to the determined target space, then there are 3 sound pickup results corresponding to each speaker outputted.

[0033] Step S130: determining target sound pickup results corresponding to each of the multiple speakers based on at least one sound pickup result corresponding to each of the multiple speakers.

[0034] In the embodiment of the present application, if multiple speakers each correspond to multiple sound pickup results, the multiple sound pickup results corresponding to each speaker can be merged based on a preset rule to obtain a target sound pickup result corresponding to each speaker. The preset rule is a pre-set rule for merging multiple sound pickup results.

[0035] The present application provides a sound pickup method, which enables a target space to be divided into different numbers of sound pickup areas according to different target resolutions corresponding to the target space, so that a target sound pickup model can output sound pickup results corresponding to different numbers of sound pickup areas according to the audio to be processed, thereby improving the flexibility of a microphone array for directional sound pickup.

[0036] See also Figure 4 , an embodiment of the present application provides a sound pickup method, the method comprising:

[0037] Step S210: obtaining at least one target sound pickup model corresponding to the target space, wherein the target sound pickup model is a sound pickup model determined based on the size of the target space, a microphone array is provided in the target space, and the target space includes a plurality of speakers.

[0038] As one approach, the dimensions include length and width; the obtaining of at least one target sound pickup model corresponding to the target space includes: obtaining the length and / or width of the target space; based on the length and / or width, obtaining at least one target sound pickup model corresponding to the target space from a preset sound pickup model library, wherein the preset sound pickup model library pre-stores a plurality of pre-trained sound pickup models, and different sound pickup models correspond to different length ranges and / or width ranges; the target sound pickup model is a sound pickup model including the length and / or width in the plurality of pre-trained sound pickup models.

[0039] In an embodiment of the present application, the preset sound pickup model library can be understood as the aforementioned preset directional sound pickup neural network model library, and different sound pickup models in the preset sound pickup model library correspond to specific spatial sizes, that is, different sound pickup models correspond to different length ranges and / or width ranges, and there is overlap in the spatial ranges between different sound pickup models to ensure that there is no size boundary.

[0040] After obtaining the length and / or width of the target space, the length and / or width of the target space can be compared with the length range and / or width range corresponding to each pickup model in the preset pickup model library to determine at least one target pickup model corresponding to the target space.

[0041] Specifically, the sound pickup model including the length and / or width of the target space in the corresponding length range and / or width range in the preset sound pickup model library is determined as the target sound pickup model corresponding to the target space. It can be known that, due to the overlap of intervals between models, there may be multiple target sound pickup models determined according to the length and / or width of the target space. If there are multiple target sound pickup models determined, it is necessary to calculate the weight coefficient of each target sound pickup model so as to fuse the output results of the target sound pickup models later.

[0042] Step S220: Based on the respective positions of the multiple speakers in the target space, determine the target resolution corresponding to each of the at least one target pickup models, wherein the target resolution is used to characterize the number of pickup areas into which the target space is divided with the array center of the microphone array as the origin.

[0043] In an embodiment of the present application, the positions of the multiple speakers in the target space can be obtained by a millimeter wave radar pre-set at a specified position. After the positions of the multiple speakers in the target space are obtained, the target resolution corresponding to each target pickup model can be determined based on the positions of the multiple speakers. The number of pickup areas into which the target space is divided with the array center of the microphone array as the origin can refer to the number of pickup areas into which the target space is evenly divided with the array center of the microphone array as the origin.

[0044] As one way, the determining of the target resolution corresponding to each of the at least one target sound pickup models based on the respective positions of the multiple speakers in the target space includes: dividing the multiple speakers into multiple speaker combinations, the speaker combinations including any two speakers of the multiple speakers, and at least one speaker included in different speaker combinations is different; determining the angle corresponding to each speaker combination based on the positions of the speakers included in each speaker combination and the position of the center of the array, and obtaining multiple angles corresponding to the multiple speaker combinations; determining the ideal resolution corresponding to each of the at least one target sound pickup models based on the minimum angle among the multiple angles; and determining the target resolution corresponding to each of the at least one target sound pickup models based on the ideal resolution corresponding to each of the at least one target sound pickup models.

[0045] The ideal resolution is used to represent the number of pickup areas into which the target space is evenly divided.

[0046] After the positions of the multiple speakers are obtained, for any two speakers among the multiple speakers, the angle between them is calculated using their position coordinates and the position coordinates of the center of the array. That is, the angle corresponding to each speaker combination is calculated using the position coordinates of each speaker included in each determined speaker combination and the position coordinates of the center of the array, and multiple angles corresponding to the multiple speaker combinations are obtained. The number of speaker combinations corresponding to the multiple speakers can be calculated by the formula N(N-1) / 2, where N represents the number of speakers.

[0047] For example, for two speakers A and B included in any speaker combination, their position coordinates (x A ,y A ),(x B ,y B ) and the position coordinates of the array center (x c ,y c ) calculate the angle θ between them. The specific calculation formula is as follows:

[0048]

[0049] After the multiple angles corresponding to the multiple speaker combinations are calculated by the above calculation formula, the ideal resolution of each target sound pickup model can be calculated based on the minimum angle among the multiple angles and the preset calculation formula. The preset calculation formula is: ideal resolution = 2π / θ min , where θ min Represents the minimum angle among the multiple angles selected above.

[0050] It should be noted that when calculating multiple angles corresponding to multiple speaker combinations, if the angle of a speaker combination is 0, then the two speakers are combined into one speaker for processing.

[0051] As one approach, the target resolution corresponding to each of the at least one target sound pickup models is determined based on the ideal resolution corresponding to each of the at least one target sound pickup models, including: obtaining at least one candidate resolution corresponding to each of the at least one target sound pickup models, the candidate resolution being a preset resolution greater than the corresponding ideal resolution among multiple preset resolutions corresponding to each of the target sound pickup models; taking the candidate resolution closest to the corresponding ideal resolution among the at least one candidate resolution corresponding to each of the target sound pickup models as the target resolution of each of the target sound pickup models, so as to obtain the target resolution corresponding to each of the at least one target sound pickup models.

[0052] Wherein, obtaining at least one candidate resolution corresponding to each target sound pickup model refers to determining a preset resolution greater than an ideal resolution among multiple preset resolutions corresponding to each target sound pickup model as a candidate resolution. Multiple preset resolutions refer to multiple resolution versions corresponding to each target sound pickup model that are pre-trained, and different resolution versions are used to divide the target space into different numbers of sound pickup areas.

[0053] After determining the candidate resolution of each target sound pickup model, the candidate resolution closest to the determined ideal resolution among at least one candidate resolution corresponding to each target sound pickup model is used to determine the target resolution corresponding to each target sound pickup model. min The closest one is selected as the target resolution of each target pickup model in the version, where F represents the resolution.

[0054] Optionally, if the positions of the multiple speakers in the target space change, the target resolution corresponding to each of the at least one target sound pickup models is re-determined based on the changed positions of each of the multiple speakers.

[0055] After determining the target pickup area corresponding to each speaker, if the speaker moves, the pickup area selection model will operate in real time. If the minimum angle does not change or increases, the target resolution corresponding to each target pickup model does not need to be switched. It is only necessary to change the target pickup area of ​​the corresponding speaker in real time according to the speaker's position relative to the center of the array. If the minimum angle decreases, it is necessary to reselect the target resolution corresponding to each target pickup model, and then reselect the target pickup area of ​​each speaker.

[0056] Step S230: Determine at least one target sound pickup area corresponding to each of the multiple speakers based on the respective orientations of the multiple speakers relative to the center of the array and the target resolution corresponding to each of the at least one target sound pickup models, wherein a target sound pickup area corresponding to each of the multiple speakers is determined based on the target resolution of one of the target sound pickup models.

[0057] In the embodiment of the present application, after the target resolution of each target sound pickup model is determined, for each speaker, the target sound pickup interval of each speaker can be determined according to the position of the speaker relative to the center of the array.

[0058] Step S240: input the to-be-processed audio collected by the microphone array into the at least one target sound pickup model respectively, and obtain at least one sound pickup result corresponding to each of the multiple speakers output by the at least one target sound pickup model.

[0059] Step S250: determining target sound pickup results corresponding to each of the multiple speakers based on at least one sound pickup result corresponding to each of the multiple speakers.

[0060] As a method, step S250 may specifically include: determining the weight coefficient of each of the at least one target sound pickup models based on the length range, width range and length and / or width of the target space corresponding to each of the at least one target sound pickup models; merging at least one sound pickup result corresponding to each of the multiple speakers based on the weight coefficient of each of the at least one target sound pickup models to determine the target sound pickup results corresponding to each of the multiple speakers.

[0061] In the embodiment of the present application, when at least one target sound pickup model corresponding to the target space is obtained, the specific space size adapted by each target sound pickup model is known, that is, the minimum length L corresponding to each target sound pickup model is known. min , maximum length L max , minimum width W min and the maximum width W max Then, the weight coefficient corresponding to each target sound pickup model can be calculated by a preset calculation formula. The specific preset calculation formula is as follows:

[0062]

[0063] Wherein, O represents the weight coefficient of each target sound pickup model.

[0064] After determining the weight coefficient of each target sound pickup model, multiple sound pickup results corresponding to each speaker can be merged by linear interpolation based on the weight coefficient of each target sound pickup model to obtain the target sound pickup result of each speaker. That is, multiple sound pickup results are merged into one sound pickup result.

[0065] Optionally, steps S210 to S250 in the embodiment of the present application can be performed by the electronic device or server alone, or by the electronic device and the server together, which is not specifically limited here. Among them, the electronic device or the server may include an automatic environment detection module, an automatic model selection module, a pickup area selection module, a pickup area dynamic adjustment module, and a pickup result merging module. Among them, the automatic environment detection module is used to scan the target space to obtain the length and width of the target space, the number of speakers and the position of each speaker, etc.; the model automatic selection module is used to determine at least one target pickup model according to the length and width of the target space; the pickup area selection module is used to determine the target resolution of each target pickup model and the target pickup area of ​​each speaker; the pickup area dynamic adjustment module is used to adjust the target resolution of each target pickup model and the target pickup area of ​​each speaker according to the change of the speaker's position, etc.; the pickup result merging module is used to merge at least one pickup result corresponding to each speaker output by at least one target pickup model.

[0066] The present application provides a sound pickup method, which enables a target space to be divided into different numbers of sound pickup areas according to different target resolutions corresponding to the target space, so that a target sound pickup model can output sound pickup results corresponding to different numbers of sound pickup areas according to the audio to be processed, thereby improving the flexibility of a microphone array for directional sound pickup.

[0067] See also Figure 3 , an embodiment of the present application provides a sound pickup device 500, the device 300 comprising:

[0068] The model acquisition unit 310 is used to acquire at least one target sound pickup model corresponding to the target space, where the target sound pickup model is a sound pickup model determined based on the size of the target space, a microphone array is arranged in the target space, and the target space includes a plurality of speakers.

[0069] As a method, the size includes length and width; the model acquisition unit 310 is specifically used to obtain the length and / or width of the target space; based on the length and / or width, at least one target sound pickup model corresponding to the target space is obtained from a preset sound pickup model library, wherein the preset sound pickup model library pre-stores a plurality of pre-trained sound pickup models, and different sound pickup models correspond to different length ranges and / or width ranges; the target sound pickup model is a sound pickup model including the length and / or width in the plurality of pre-trained sound pickup models.

[0070] The output unit 320 is used to input the to-be-processed audio collected by the microphone array into the at least one target sound pickup model, respectively, to obtain at least one sound pickup result corresponding to each of the multiple speakers output by the at least one target sound pickup model, wherein one of the target sound pickup models outputs a sound pickup result corresponding to each of the multiple speakers; and one of the sound pickup results corresponding to each of the multiple speakers is a sound pickup result of a target sound pickup area corresponding to each of the multiple speakers output by the target sound pickup model, and the target sound pickup area corresponding to each of the multiple speakers is determined by a target resolution of the target sound pickup model.

[0071] The determination unit 330 is configured to determine a target sound pickup result corresponding to each of the multiple speakers based on at least one sound pickup result corresponding to each of the multiple speakers.

[0072] As a method, the determination unit 330 is specifically used to determine the weight coefficient of each of the at least one target sound pickup models based on the length range, width range and length and / or width of the target space corresponding to each of the at least one target sound pickup models; based on the weight coefficient of each of the at least one target sound pickup models, at least one sound pickup result corresponding to each of the multiple speakers is merged to determine the target sound pickup results corresponding to each of the multiple speakers.

[0073] See also Figure 4 , the device 300 further includes:

[0074] The sound pickup area determination unit 340 is used to determine the target resolution corresponding to each of the at least one target sound pickup models based on the respective positions of the multiple speakers in the target space, wherein the target resolution is used to characterize the number of sound pickup areas into which the target space is divided with the array center of the microphone array as the origin; and to determine at least one target sound pickup area corresponding to each of the multiple speakers based on the respective orientations of the multiple speakers relative to the array center and the target resolution corresponding to each of the at least one target sound pickup models, wherein one target sound pickup area corresponding to each of the multiple speakers is determined based on the target resolution of one of the target sound pickup models.

[0075] As a method, the sound pickup area determination unit 340 is also used to divide the multiple speakers into multiple speaker combinations, where the speaker combination includes any two speakers from the multiple speakers, and at least one speaker included in different speaker combinations is different; based on the positions of the speakers included in each speaker combination and the position of the center of the array, the angle corresponding to each speaker combination is determined to obtain multiple angles corresponding to the multiple speaker combinations; based on the minimum angle among the multiple angles, the ideal resolution corresponding to each of the at least one target sound pickup models is determined; based on the ideal resolution corresponding to each of the at least one target sound pickup models, the target resolution corresponding to each of the at least one target sound pickup models is determined.

[0076] As another embodiment, the sound pickup area determination unit 340 is also used to obtain at least one candidate resolution corresponding to each of the at least one target sound pickup models, and the candidate resolution is a preset resolution that is greater than the corresponding ideal resolution among the multiple preset resolutions corresponding to each of the target sound pickup models; and the candidate resolution that is closest to the corresponding ideal resolution among the at least one candidate resolution corresponding to each of the target sound pickup models is used as the target resolution of each of the target sound pickup models to obtain the target resolution corresponding to each of the at least one target sound pickup models.

[0077] Optionally, the sound pickup area determination unit 340 is further configured to re-determine the target resolution corresponding to each of the at least one target sound pickup models based on the changed positions of the multiple speakers if the positions of the multiple speakers in the target space change.

[0078] It should be noted that the device embodiment in the present application corresponds to the aforementioned method embodiment. The specific principles in the device embodiment can be found in the contents of the aforementioned method embodiment and will not be repeated here.

[0079] The following will be combined Figure 5 An electronic device provided by the present application is described.

[0080] See also Figure 5 Based on the above-mentioned sound pickup method and device, the embodiment of the present application also provides another electronic device 800 that can execute the above-mentioned sound pickup method. The electronic device 800 includes one or more (only one is shown in the figure) processors 802, a memory 804 and a network module 806 that are coupled to each other. Among them, the memory 804 stores a program that can execute the content of the above-mentioned embodiment, and the processor 802 can execute the program stored in the memory 804.

[0081] Among them, the processor 802 may include one or more processing cores. The processor 802 uses various interfaces and lines to connect various parts of the entire electronic device 800, and executes various functions and processes data of the electronic device 800 by running or executing instructions, programs, code sets or instruction sets stored in the memory 804, and calling data stored in the memory 804. Optionally, the processor 802 can be implemented in at least one hardware form of digital signal processing (Digital Signal Processing, DSP), field programmable gate array (Field-Programmable Gate Array, FPGA), and programmable logic array (Programmable Logic Array, PLA). The processor 802 can integrate one or a combination of a central processing unit (Central Processing Unit, CPU), a graphics processing unit (Graphics Processing Unit, GPU) and a modem. Among them, the CPU mainly processes the operating system, user interface and application programs; the GPU is responsible for rendering and drawing display content; and the modem is used to process wireless communications. It can be understood that the above-mentioned modem may not be integrated into the processor 802, but may be implemented separately through a communication chip.

[0082] The memory 804 may include a random access memory (RAM) or a read-only memory (ROM). The memory 804 may be used to store instructions, programs, codes, code sets or instruction sets. The memory 804 may include a program storage area and a data storage area, wherein the program storage area may store instructions for implementing an operating system, instructions for implementing at least one function (such as a touch function, a sound playback function, an image playback function, etc.), instructions for implementing the following various method embodiments, etc. The data storage area may also store data (such as a phone book, audio and video data, chat record data) created by the terminal 800 during use.

[0083] The network module 806 is used to receive and send electromagnetic waves, realize the mutual conversion between electromagnetic waves and electrical signals, so as to communicate with a communication network or other devices, such as communicating with electronic devices. The network module 806 may include various existing circuit elements for performing these functions, such as antennas, radio frequency transceivers, digital signal processors, encryption / decryption chips, user identity modules (SIM) cards, memories, etc. The network module 806 may communicate with various networks such as the Internet, corporate intranets, wireless networks, or communicate with other devices via wireless networks. The above-mentioned wireless networks may include cellular telephone networks, wireless local area networks, or metropolitan area networks. For example, the network module 806 may exchange information with a base station.

[0084] Please refer to Figure 6 , which shows a structural block diagram of a computer-readable storage medium provided in an embodiment of the present application. The computer-readable medium 900 stores program codes, which can be called by a processor to execute the method described in the above method embodiment.

[0085] The computer-readable storage medium 900 may be an electronic memory such as a flash memory, an EEPROM (electrically erasable programmable read-only memory), an EPROM, a hard disk, or a ROM. Optionally, the computer-readable storage medium 900 includes a non-transitory computer-readable storage medium. The computer-readable storage medium 900 has storage space for program code 910 that performs any method step in the above method. These program codes can be read from or written to one or more computer program products. The program code 910 can be compressed, for example, in an appropriate form.

[0086] The present application provides a sound pickup method, device, electronic device and storage medium. First, at least one target sound pickup model corresponding to a target space is obtained, wherein the target sound pickup model is a sound pickup model determined based on the size of the target space, wherein a microphone array is provided in the target space and the target space includes a plurality of speakers. Then, the audio to be processed collected by the microphone array is respectively input into the at least one target sound pickup model, and at least one sound pickup result corresponding to each of the plurality of speakers output by the at least one target sound pickup model is obtained, wherein one of the target sound pickup models outputs a sound pickup result corresponding to each of the plurality of speakers; and one sound pickup result corresponding to each of the plurality of speakers is a sound pickup result of a target sound pickup area corresponding to each of the plurality of speakers output by the target sound pickup model, and the target sound pickup area corresponding to each of the plurality of speakers is determined by the target resolution of the target sound pickup model. Finally, based on the at least one sound pickup result corresponding to each of the plurality of speakers, the target sound pickup results corresponding to each of the plurality of speakers are determined. Through the above method, the target space can be divided into different numbers of pickup areas according to different target resolutions corresponding to the target space, so that the target pickup model can output pickup results corresponding to different numbers of pickup areas according to the audio to be processed, thereby improving the flexibility of the microphone array for directional pickup.

[0087] The embodiments of the present invention are described above in conjunction with the accompanying drawings, but the present invention is not limited to the above-mentioned specific implementation modes, which are merely illustrative rather than restrictive. Under the guidance of the present invention, ordinary technicians in this field can also make many forms without departing from the scope of protection of the present invention and the claims, all of which are within the protection of the present invention.

Claims

1. A sound pickup method, characterized in that: The method comprises: Acquire at least one target sound pickup model corresponding to a target space, wherein the target sound pickup model is a sound pickup model determined based on a size of the target space, a microphone array is provided in the target space, and a plurality of speakers are included in the target space; Input the to-be-processed audio collected by the microphone array into the at least one target sound pickup model respectively, and obtain at least one sound pickup result corresponding to each of the multiple speakers output by the at least one target sound pickup model, wherein one of the target sound pickup models outputs a sound pickup result corresponding to each of the multiple speakers; one of the sound pickup results corresponding to each of the multiple speakers is a sound pickup result of a target sound pickup area corresponding to each of the multiple speakers output by the target sound pickup model, and the target sound pickup area corresponding to each of the multiple speakers is determined by a target resolution of the target sound pickup model; Based on at least one sound pickup result corresponding to each of the multiple speakers, target sound pickup results corresponding to each of the multiple speakers are determined.

2. The method according to claim 1, characterized in that Before the audio to be processed collected by the microphone array is respectively input into the at least one target sound pickup model, and at least one sound pickup result corresponding to each of the multiple speakers output by the at least one target sound pickup model is obtained, the method further includes: Determine, based on the respective positions of the multiple speakers in the target space, a target resolution corresponding to each of the at least one target sound pickup models, wherein the target resolution is used to represent the number of sound pickup areas into which the target space is divided with the array center of the microphone array as the origin; Based on the respective orientations of the multiple speakers relative to the center of the array and the respective target resolutions corresponding to the at least one target sound pickup model, at least one target sound pickup area corresponding to each of the multiple speakers is determined, wherein a target sound pickup area corresponding to each of the multiple speakers is determined based on the target resolution of one of the target sound pickup models.

3. The method according to claim 2, characterized in that The determining, based on the respective positions of the multiple speakers in the target space, a target resolution corresponding to each of the at least one target sound pickup models comprises: dividing the multiple speakers into multiple speaker combinations, wherein the speaker combinations include any two speakers from the multiple speakers, and at least one speaker included in different speaker combinations is different; Based on the positions of the speakers included in each speaker combination and the position of the center of the array, determining the angle corresponding to each speaker combination, and obtaining multiple angles corresponding to the multiple speaker combinations; Determining an ideal resolution corresponding to each of the at least one target sound pickup model based on a minimum angle among the multiple angles; Based on the ideal resolution corresponding to each of the at least one target sound pickup models, the target resolution corresponding to each of the at least one target sound pickup models is determined.

4. The method according to claim 3, characterized in that The determining, based on the ideal resolution corresponding to each of the at least one target sound pickup models, a target resolution corresponding to each of the at least one target sound pickup models, comprises: Obtain at least one candidate resolution corresponding to each of the at least one target sound pickup models, wherein the candidate resolution is a preset resolution greater than the corresponding ideal resolution among a plurality of preset resolutions corresponding to each of the target sound pickup models; Among at least one candidate resolution corresponding to each target sound pickup model, the candidate resolution closest to the corresponding ideal resolution is used as the target resolution of each target sound pickup model to obtain the target resolution corresponding to each of the at least one target sound pickup model.

5. The method according to claim 2, characterized in that: The method further comprises: If the positions of the multiple speakers in the target space change, the target resolution corresponding to each of the at least one target sound pickup models is re-determined based on the changed positions of the multiple speakers.

6. The method according to claim 1, characterized in that The obtaining of at least one target sound pickup model corresponding to the target space includes: Said dimensions include length and width; Acquire the length and / or width of the target space; Based on the length and / or width, at least one target sound pickup model corresponding to the target space is obtained from a preset sound pickup model library, wherein the preset sound pickup model library pre-stores a plurality of pre-trained sound pickup models, and different sound pickup models correspond to different length ranges and / or width ranges; the target sound pickup model is a sound pickup model including the length and / or width among the plurality of pre-trained sound pickup models.

7. The method according to claim 6, characterized in that The determining, based on at least one sound pickup result corresponding to each of the multiple speakers, target sound pickup results corresponding to each of the multiple speakers comprises: Determining a weight coefficient of each of the at least one target sound pickup models based on a length range, a width range and a length and / or a width of the target space corresponding to each of the at least one target sound pickup models; Based on the weight coefficient of each of the at least one target sound pickup models, at least one sound pickup result corresponding to each of the multiple speakers is merged to determine the target sound pickup results corresponding to each of the multiple speakers.

8. A sound pickup device, characterized in that: The device comprises: A model acquisition unit, configured to acquire at least one target sound pickup model corresponding to a target space, wherein the target sound pickup model is a sound pickup model determined based on a size of the target space, wherein a microphone array is provided in the target space, and wherein a plurality of speakers are included in the target space; an output unit, configured to input the to-be-processed audio collected by the microphone array into the at least one target sound pickup model, respectively, and obtain at least one sound pickup result corresponding to each of the multiple speakers output by the at least one target sound pickup model, wherein one of the target sound pickup models outputs a sound pickup result corresponding to each of the multiple speakers; and one of the sound pickup results corresponding to each of the multiple speakers is a sound pickup result of a target sound pickup area corresponding to each of the multiple speakers output by the target sound pickup model, and the target sound pickup area corresponding to each of the multiple speakers is determined by a target resolution of the target sound pickup model; The determination unit is configured to determine a target sound pickup result corresponding to each of the multiple speakers based on at least one sound pickup result corresponding to each of the multiple speakers.

9. An electronic device, characterized in that: The method comprises one or more processors; one or more programs are stored in the memory and configured to execute the method according to any one of claims 1 to 7 by the one or more processors.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores program code, wherein when the program code is executed by a processor, the method according to any one of claims 1 to 7 is executed.

11. A computer program product comprising a computer program / instructions, characterized in that When the computer program / instructions are executed by a processor, the steps of the method described in any one of claims 1 to 7 are implemented.

Images