Sound source positioning method and device, medium and household appliance

Abstract

The invention discloses a sound source positioning method and apparatus, a medium and a household appliance. The method comprises the steps of obtaining target sound data emitted by a sound source and collected by a target device; acquiring a target relative position between the target equipment and an obstacle; and based on the target sound data and the target relative position, determining a target sound source position through a sound source positioning model. Through the technical scheme provided by the invention, the positioning accuracy of the target sound source position can be improved.

Description

Technical Field

[0001] The present application belongs to the technical field of sound source localization, and in particular, relates to a sound source localization method, device, medium and household appliance. Background Art

[0002] Sound source localization refers to obtaining the position of the sound source relative to the sound collection device by analyzing the sound signal collected by the sound collection device. Sound source localization technology is widely used in products and scenarios that require voice interaction, such as smart homes (such as smart home appliances). However, the existing technology has the problem of poor accuracy in sound source localization. Summary of the invention

[0003] The embodiments of the present application provide a sound source localization method, device, medium and household appliance, thereby improving the accuracy of localizing the target sound source at least to a certain extent.

[0004] Other features and advantages of the present application will become apparent from the following detailed description, or may be learned in part by the practice of the present application.

[0005] According to a first aspect of an embodiment of the present application, a sound source localization method is provided, which is used by a target device to identify a sound source position, comprising:

[0006] Acquiring target sound data emitted by a sound source and collected by the target device;

[0007] Acquiring a target relative position between the target device and the obstacle;

[0008] Based on the target sound data and the relative position of the target, the target sound source position is determined by a sound source localization model.

[0009] In some embodiments of the present application, based on the above solution, the target device is provided with a microphone array, and the acquiring of target sound data emitted by a sound source and collected by the target device includes:

[0010] Target sound data emitted by a sound source and collected by each microphone in the microphone array is obtained.

[0011] In some embodiments of the present application, based on the above solution, the method further includes:

[0012] Acquire first geometric data of the microphone array, and construct a first machine learning model according to the first geometric data and a first sound source localization algorithm;

[0013] Acquire multiple groups of first sample data, wherein each group of first sample data includes a first sample sound source position, a first sample relative position between the target device and the obstacle, and first sample sound data emitted by the sound source at the first sample sound source position and collected by the microphone array;

[0014] Based on the multiple groups of first sample data, the first machine learning model is trained to obtain the sound source localization model.

[0015] In some embodiments of the present application, based on the above solution, the sound source localization model includes a sound source localization basic model and a sound source localization correction model, and the method further includes:

[0016] Acquire second geometric data of the microphone array, and construct a sound source localization basic model according to the second geometric data and a second sound source localization algorithm;

[0017] Acquire multiple groups of second sample data, wherein each group of second sample data includes a second sample sound source position, a second sample relative position between the target device and the obstacle, and a sample initial sound source position determined by the sound source localization basic model according to the second sample sound data, wherein the second sample sound data is sound data emitted by the sound source at the second sample sound source position and collected by the microphone array;

[0018] Based on the multiple groups of second sample data, a second machine learning model is trained to obtain the sound source localization correction model.

[0019] In some embodiments of the present application, based on the aforementioned solution, determining the target sound source position through a sound source localization model based on the target sound data and the relative position of the target includes:

[0020] Based on the target sound data, determining the target initial sound source position through the sound source localization basic model;

[0021] Based on the target initial sound source position and the target relative position, the target sound source position is determined by the sound source localization correction model.

[0022] In some embodiments of the present application, based on the aforementioned solution, determining the target initial sound source position through the sound source localization basic model based on the target sound data includes:

[0023] Based on the target sound data, determining time difference data of the target sound emitted by the sound source reaching each microphone in the microphone array;

[0024] Based on the time difference data, the target initial sound source position is determined by the sound source localization basic model.

[0025] In some embodiments of the present application, based on the above solution, the step of obtaining the target relative position between the target device and the obstacle includes:

[0026] Obtaining an environment map of the environment where the target device is located;

[0027] Determining the locations of the target device and obstacles in the environment map;

[0028] Based on the positions of the target device and the obstacle in the environment map, a target relative position between the target device and the obstacle is determined.

[0029] In some embodiments of the present application, based on the aforementioned solution, the sound source localization model is stored in a server, and the target sound source position is determined by the sound source localization model based on the relative position of the target sound data and the target, including:

[0030] Sending the target sound data and the target relative position to the server, so that the server determines the target sound source position through a sound source localization model;

[0031] Acquire the target sound source position sent by the server.

[0032] According to a second aspect of an embodiment of the present application, a sound source localization apparatus is provided, which is disposed in a target device to identify a sound source position, and includes:

[0033] A first acquisition unit, configured to acquire target sound data emitted by a sound source and collected by the target device;

[0034] A second acquisition unit, used to acquire a target relative position between the target device and the obstacle;

[0035] The position determination unit is used to determine the target sound source position through a sound source localization model based on the target sound data and the relative position of the target.

[0036] According to a third aspect of an embodiment of the present application, a readable storage medium is provided, wherein the storage medium stores a computer program, and when the computer program is executed, the steps of the method described in any one of the first aspects can be implemented.

[0037] According to a fourth aspect of an embodiment of the present application, a household appliance is provided, comprising a processor, wherein the processor is used to execute the steps of the method described in any one of the first aspects.

[0038] It should be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present application.

[0039] One or more technical solutions provided by the embodiments of the present application achieve at least the following technical effects or advantages:

[0040] The embodiment of the present application obtains target sound data emitted by a sound source collected by a target device, and a target relative position between the target device and an obstacle, uses the target relative position as a variable for locating the sound source emitting the target sound data, and outputs the target sound source position through a sound source localization model, thereby taking into account the influence of obstacles on the accuracy of sound source localization and improving the accuracy of locating the target sound source position. BRIEF DESCRIPTION OF THE DRAWINGS

[0041] The drawings herein are incorporated into the specification and constitute a part of the specification, showing embodiments consistent with the present application, and together with the specification, are used to explain the principles of the present application. Obviously, the drawings described below are only some embodiments of the present application, and for ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work. In the drawings:

[0042] Figure 1 A flow chart of a sound source localization method in an embodiment of the present application is shown;

[0043] Figure 2 A schematic diagram showing an application scenario of the sound source localization method in an embodiment of the present application is shown;

[0044] Figure 3 A schematic diagram of the three-dimensional structure of the target device in the embodiment of the present application is shown;

[0045] Figure 4 A schematic diagram of the side structure of the target device in an embodiment of the present application is shown;

[0046] Figure 5 A schematic diagram of a top view of the target device in an embodiment of the present application is shown;

[0047] Figure 6 A schematic diagram showing the control principle of the target device in an embodiment of the present application is shown;

[0048] Figure 7 A schematic diagram of the structure of a sound source localization device in an embodiment of the present application is shown;

[0049] Figure 8 A schematic structural diagram of a household appliance according to an embodiment of the present application is shown. DETAILED DESCRIPTION

[0050] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0051] In addition, described feature, structure or characteristic can be combined in one or more embodiments in any suitable manner. In the following description, many specific details are provided to provide a full understanding of the embodiments of the present application. However, those skilled in the art will appreciate that the technical scheme of the present application can be put into practice without one or more of the specific details, or other methods, components, devices, steps, etc. can be adopted. In other cases, known methods, devices, realizations or operations are not shown or described in detail to avoid blurring the various aspects of the application.

[0052] The block diagrams shown in the accompanying drawings are merely functional entities and do not necessarily correspond to physically independent entities. That is, these functional entities may be implemented in software form, or in one or more hardware modules or integrated circuits, or in different networks and / or processor devices and / or microcontroller devices.

[0053] The flowcharts shown in the accompanying drawings are only exemplary and do not necessarily include all the contents and operations / steps, nor must they be executed in the order described. For example, some operations / steps can be decomposed, and some operations / steps can be combined or partially combined, so the actual execution order may change according to actual conditions.

[0054] It should also be noted that the terms "first", "second", etc. in the specification and claims of the present application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the objects used in this way can be interchanged where appropriate, so that the embodiments of the present application described herein can be implemented in an order other than those shown or described.

[0055] In order to make those skilled in the art better understand the present application, Figure 1 The application scenarios involved in this application are briefly described.

[0056] It should be noted that the target devices mentioned below in the embodiments of the present application include but are not limited to various types of household appliances, such as mobile air conditioners, smart refrigerators, smart TVs, smart speakers, etc., which are not limited here.

[0057] See also Figure 1 , Figure 1The flowchart of the sound source localization method in the embodiment of the present application is shown. The sound source localization method can be executed by a device with a computing and processing function, which can be a cloud server, a mobile client with a sound source localization model construction function, or a target device with a sound source localization model construction function. The embodiment of the present application does not limit the specific execution subject of the method.

[0058] like Figure 1 As shown, according to a first aspect of an embodiment of the present application, a sound source localization method is provided, which is used by a target device to identify a sound source position, including but not limited to being implemented by steps S110 to S130:

[0059] Step S110. Acquire target sound data emitted by a sound source and collected by the target device;

[0060] It should be noted that the target devices in the embodiments of the present application include but are not limited to various types of home appliances, such as smart air conditioners, sweeping robots, smart refrigerators, smart TVs, smart speakers, sweeping robots, etc. The sound source in the embodiments of the present application can be a signal source of various sound signals, such as a user of a home appliance, a voice robot, etc. The target sound data in the embodiments of the present application can be various sound signal data emitted by various sound sources, such as wake-up words and other data.

[0061] In some embodiments, the target device is provided with a voice module, through which the sound data emitted by the sound source can be collected, and the voice module includes but is not limited to a microphone array with a sound pickup function, and the microphone array can be an array formed by a group of omnidirectional microphones located at different positions on the target device arranged according to a certain shape rule, and is used to sample the sound signal emitted by the sound source in space. Among them, the certain shape rule can be linear, annular, spherical, etc. In the embodiment of the present application, a microphone array arranged in an annular shape is used.

[0062] Based on the above, in some embodiments of the present application, based on the aforementioned solution, the target device is provided with the microphone array, and the acquiring of the target sound data emitted by the sound source and collected by the target device includes:

[0063] Target sound data emitted by a sound source and collected by each microphone in the microphone array is obtained.

[0064] For example: the target device is set in a home environment, and the user wakes up the target device through voice. At this time, the user is the sound source in the embodiment of the present application, and the target sound data is the sound data emitted by the user, such as "turn on", "move to the living room" and other sound data. For example, if the microphone array on the target device has three microphones, the user emits target sound data, and each microphone emits target sound data, it can be understood that each microphone is located at a different position on the target device, so the time when the target sound data emitted by the user reaches each microphone may be different.

[0065] Step S120. Obtaining the target relative position between the target device and the obstacle;

[0066] It should be noted that when the target device is located in an environment, there may be obstacles in the environment that affect the propagation of the sound source data, such as obstacles that affect the propagation path of the sound data. For example, when the target device is located in a home environment, obstacles such as walls, cabinets, and screens in the home environment may interfere with the propagation of the sound source data. On this basis, the embodiment of the present application obtains the target relative position between the target device and the obstacle, and uses the target relative position as one of the variables for sound source positioning. Then, the sound source positioning result obtained is a result related to the target relative position, thereby making the sound source positioning more accurate.

[0067] The target relative position may be: in a first spatial coordinate system, the target device is at a first position in the first spatial coordinate system, the obstacle is at a second position in the first spatial coordinate system, and the vector distance between the first position and the second position is the target relative position. The first spatial coordinate system may be a spatial coordinate system established with one position in the environment where the target device is located as the origin.

[0068] The target relative position may also be: in a second spatial coordinate system, with the spatial position of the target device as the origin, the distance and orientation of the spatial position of the obstacle relative to the spatial position of the target device are calculated, that is, the target relative position. The second spatial coordinate system may be a spatial coordinate system established with the spatial position of the target device as the origin.

[0069] It should be noted that the spatial position of the target device and the spatial position of the obstacle can be three-dimensional point cloud data. In the embodiment of the present application, the three-dimensional point cloud data can be used to characterize the position distribution of the target device and the obstacle in the spatial coordinate system. It should be noted that in the second spatial coordinate system, if the target device is in a moving state, the spatial position of the second spatial coordinate system established with the target device as the origin in the environment where it is located also changes. It should be further noted that although the spatial position of the obstacle in the environment where the target device is located is usually fixed, as the target device moves, the three-dimensional point cloud data reflected by the spatial position of the obstacle in the second spatial coordinate system also changes accordingly.

[0070] It is understandable that there may be multiple obstacles in the environment where the target device is located. For example, the obstacles include walls and cabinets. In this case, obtaining the target relative position between the target device and the obstacle may include: obtaining a first target relative position between the target device and the wall, and obtaining a second target relative position between the target device and the cabinet.

[0071] In some embodiments of the present application, based on the above solution, the step of obtaining the target relative position between the target device and the obstacle includes:

[0072] Step S121. Obtain an environment map of the environment where the target device is located;

[0073] In some embodiments, the target device is a mobile device, and a laser module is provided on the target device. The laser module is provided with multiple position sensors. During the movement of the target device, the surrounding environment information is obtained through each position sensor to construct an environment map, and the position data of the target device in the environment map is obtained. Specifically, the position data of the target device in the environment map can be obtained through a SLAM (Simultaneous Localization And Mapping) algorithm.

[0074] In some embodiments, the environment map may be at least one of a grid map, a feature map, and a geometric semantic map, wherein the grid map and the feature map are commonly used map types for the SLAM algorithm, and may be applied, for example, to the target device in the embodiment of the present application for indoor navigation in a home environment to accurately describe the environment geometry information; the geometric semantic map may characterize the obstacle type, spatial form and position, and the position of the target device in the environment map.

[0075] Step S122. Determine the positions of the target device and obstacles in the environment map;

[0076] It should be noted that if the environmental map is an environmental map established based on the above-mentioned first spatial coordinate system, then determining the positions of the target device and the obstacle in the environmental map can be: taking one of the positions in the environment where the target device is located as the origin, determining the vector position of the target device from the origin, and determining the vector position of the obstacle from the origin.

[0077] It should be noted that if the environmental map is an environmental map established based on the above-mentioned second spatial coordinate system, then determining the positions of the target device and the obstacle in the environmental map can be: taking the spatial position of the target device as the origin, determining the vector position of the obstacle relative to the target device located at the origin.

[0078] Step S123: Determine a target relative position between the target device and the obstacle based on the positions of the target device and the obstacle in the environment map.

[0079] It should be noted that if the environmental map is an environmental map established based on the above-mentioned first spatial coordinate system, the target relative position between the target device and the obstacle is the vector distance between the vector position of the target device from the origin and the vector position of the obstacle from the origin.

[0080] It should be noted that, if the environment map is an environment map established based on the second spatial coordinate system, the target relative position between the target device and the obstacle is the vector distance between the obstacle and the origin.

[0081] Step S130: Based on the target sound data and the relative position of the target, determine the target sound source position through a sound source localization model.

[0082] In step S130, the sound source localization model may be a pre-trained sound source localization model. By training the sound source localization model, the trained sound source localization model can output the sound source position based on the relative position between the target device and obstacles in its environment, thereby enabling the target device to locate the sound source with higher accuracy.

[0083] In some implementations of step S130, the training process of the sound source localization model includes but is not limited to the implementation of steps S11 to S13:

[0084] Step S11. Acquire first geometric data of the microphone array, and construct a first machine learning model according to the first geometric data and a first sound source localization algorithm;

[0085] The first geometric data of the microphone array refers to: geometric relationship data of each microphone in the microphone array arranged according to a certain geometric shape rule, and the geometric relationship data includes but is not limited to spatial position data of each microphone in the environment, relative position data between each microphone, and relative angle data between each microphone, etc.

[0086] In the embodiment of the present application, the first sound source localization algorithm includes but is not limited to an algorithm based on beamforming, an algorithm based on high-resolution spectrum estimation, and an algorithm based on time difference of arrival. The principles of the above three algorithms are respectively described below.

[0087] Among them, the algorithm based on beamforming is a controllable beamforming technology based on maximum output power. The sound data collected by each microphone in the microphone array on the target device is weighted and summed to form a beam. The beam is guided by searching for the possible location of the sound source, and the output signal power of the microphone array is maximized by modifying the weights, thereby realizing sound source positioning. The algorithm based on beamforming can be used in both the time domain and the frequency domain, so it has strong adaptability to the environment.

[0088] Among them, algorithms based on high-resolution spectrum estimation include autoregressive AR (Augmented Reality) model, minimum variance spectrum estimation (MV) and eigenvalue decomposition algorithm (such as Music algorithm), etc. All of these algorithms calculate the correlation matrix of the spatial spectrum by acquiring the signal of the microphone array, thereby effectively estimating the direction of the sound source.

[0089] Among them, the algorithm based on time difference of sound arrival usually first estimates the time difference of sound arrival (that is, the time difference between the sound data emitted by the sound source and the arrival of each microphone in the microphone array), and obtains the sound signal delay between the array elements in the microphone array. The obtained time difference of sound arrival is then used in combination with the known spatial geometric position of the microphone array to further determine the position of the sound source.

[0090] In an embodiment of the present application, the first machine learning model refers to a model constructed using a machine learning method based on a traditional sound source localization model, and the traditional sound source localization model refers to a model constructed using the above-mentioned multiple first sound source localization algorithms, that is, the first machine learning model is a model constructed by fusing the traditional sound source localization model with the machine learning model. The first machine learning model can adopt at least one of a neural network model (e.g., a recursive neural network model), a decision tree model, a support vector machine model, or a linear regression model.

[0091] Step S12. Acquire multiple groups of first sample data, wherein each group of first sample data includes a first sample sound source position, a first sample relative position between the target device and the obstacle, and first sample sound data emitted by the sound source at the first sample sound source position and collected by the microphone array;

[0092] It should be noted that, in each group of first sample data, the first sample relative position, the first sample sound data and the first sample sound source position have a corresponding relationship, so that when training the first machine learning model, the first sample relative position and the first sample sound data can be used as model inputs, and the first sample sound source position can be used as model outputs, and the parameters of the model can be corrected by calculating the loss function to optimize the model training.

[0093] In some embodiments, when the target device is applied in an environment, for example, when the target device is applied in a home environment, the surrounding environment information can be collected multiple times by a laser module and a voice module provided on the target device, thereby obtaining multiple sets of the first sample data, and then constructing a training sample set for the first machine learning model.

[0094] Step S13. Based on the multiple groups of first sample data, train the first machine learning model to obtain the sound source localization model.

[0095] In an embodiment of the present application, the first machine learning model is configured to receive the first sample data and process the first sample data according to the machine learning model parameters to generate a predicted sound source position relative to the sample data, calculate a loss function through the predicted sound source position and the first sample sound source position, and use the loss function to modify the machine learning model parameters, and obtain the sound source localization model through multiple iterative optimization training. For example, if the first machine learning model is a neural network model, the machine learning model parameters may include weight values ​​of neuron connections in the neural network model.

[0096] In some embodiments, the server acquires first geometric data of the microphone array, and constructs a first machine learning model according to the first geometric data and a first sound source localization algorithm; the target device collects multiple groups of first sample data and sends the multiple groups of first sample data to the server; the server trains the first machine learning model using the multiple groups of first sample data to obtain the sound source localization model, and stores the sound source localization model; when used, the target device sends the collected target sound data and the target relative position to the server; the server outputs the target sound source position based on the sound source localization model according to the target sound data and the target relative position, and sends the target sound source position to the target device.

[0097] Based on the above implementation, by training the first machine learning model in the server, the server's computing resources are abundant, making the model training more efficient. At the same time, the server is used to store the sound source localization model, and the target sound source position is output through the sound source localization model, that is, the main computing tasks are deployed on the server for execution, thereby reducing the hardware requirements for the target device and further reducing the hardware cost of the target device.

[0098] In some embodiments, the server acquires first geometric data of the microphone array, and constructs a first machine learning model according to the first geometric data and a first sound source localization algorithm; the target device collects multiple groups of first sample data and sends the multiple groups of first sample data to the server; the server trains the first machine learning model using the multiple groups of first sample data to obtain the sound source localization model, and sends the sound source localization model to the target device; when used, the target device inputs the collected target sound data and the target relative position into its own sound source localization model, and outputs the target sound source position based on the sound source localization model.

[0099] Based on the above implementation, by training the first machine learning model in the server, the server's computing resources are abundant, making the model training more efficient, and the sound source localization model is stored in the target device, so that the target device can input the collected target sound data and the target relative position into its own sound source localization model, and quickly output the target sound source position based on the sound source localization model, thereby improving the efficiency of sound source localization.

[0100] In some embodiments, the target device acquires first geometric data of the microphone array, constructs a first machine learning model according to the first geometric data and a first sound source localization algorithm; the target device collects multiple groups of first sample data, and uses the multiple groups of first sample data to train the first machine learning model to obtain the sound source localization model; when used, the target device inputs the collected target sound data and the target relative position into its own sound source localization model, and outputs the target sound source position based on the sound source localization model.

[0101] Based on the above implementation, the first machine learning model is trained through the target device to obtain a sound source localization model, and the sound source localization model is used to output the target sound source position. When the target device is used in a home environment, the privacy of the user's home environment information can be ensured to a certain extent.

[0102] In some embodiments, the target device acquires first geometric data of the microphone array, and constructs a first machine learning model according to the first geometric data and a first sound source localization algorithm; the target device collects multiple groups of first sample data, trains the first machine learning model using the multiple groups of first sample data to obtain the sound source localization model, and sends the sound source localization model to the server; when used, the target device sends the collected target sound data and the target relative position to the server, so that the server outputs the target sound source position based on the sound source localization model.

[0103] Based on the above implementation mode, the target device trains the first machine learning model to obtain a sound source localization model, and stores the sound source localization model on the server. The server outputs the target sound source position based on the sound source localization model, which enables the target device to reduce the utilization of its own resources for sound source localization after completing the model training, thereby enabling the target device to allocate computing resources to other functional modules other than sound source localization, such as image recognition functional modules, indoor navigation functional modules, etc.

[0104] In some implementations of step S130, the sound source localization model includes a sound source localization basic model and a sound source localization correction model. The training process of the sound source localization model includes but is not limited to the implementation of steps S21 to S23:

[0105] Step S21. Acquire second geometric data of the microphone array, and construct a sound source localization basic model according to the second geometric data and a second sound source localization algorithm;

[0106] It can be understood that the construction principle of the sound source localization basic model in step S21 is similar to the construction principle of the traditional sound source localization model in the first machine learning model in the above embodiment, so the construction principle of the sound source localization basic model is not described here.

[0107] Step S22. Acquire multiple groups of second sample data, wherein each group of second sample data includes a second sample sound source position, a second sample relative position between the target device and the obstacle, and a sample initial sound source position determined by the sound source localization basic model according to the second sample sound data, wherein the second sample sound data is sound data emitted by the sound source at the second sample sound source position and collected by the microphone array;

[0108] It can be understood that the second sample sound source position, the second sample relative position and the second sample sound data of each group of second sample data in step S22 are similar to the method for obtaining each group of first sample data in the above embodiment, so the method for obtaining each of the above second sample data is not described here. It is worth noting that each group of second sample data also includes the sample initial sound source position determined by the sound source localization basic model according to the second sample sound data, that is, the sound source localization basic model of the embodiment of the present application obtains the sample initial sound source position according to the second geometric data and the second sound source localization algorithm.

[0109] Step S23. Based on the multiple groups of second sample data, train a second machine learning model to obtain the sound source localization correction model.

[0110] It can be understood that the second machine learning model refers to a model that is trained based on the initial sound source position using the second sample relative position between the target device and the obstacle to compensate for the initial sound source position. Based on the above implementation, only the position compensation part of the sound source position is trained, thereby reducing the data size and improving training efficiency.

[0111] In step S130, determining the target sound source position through a sound source localization model based on the target sound data and the relative position of the target includes:

[0112] Step S131. Based on the target sound data, determine the target initial sound source position through the sound source localization basic model;

[0113] In step S131, determining the target initial sound source position based on the target sound data by using the sound source localization basic model includes:

[0114] Step S1311. Based on the target sound data, determine the time difference data of the target sound emitted by the sound source reaching each microphone in the microphone array;

[0115] Step S1312: Based on the time difference data, determine the target initial sound source position through the sound source localization basic model.

[0116] In the embodiment of the present application, the sound source localization basic model adopts the above-mentioned algorithm based on the time difference of arrival to determine the initial sound source position of the target. The algorithm principle is as follows:

[0117] Assume that in the environment map, there is a sound source s(t), the position of the sound source in the environment map is S, and there are microphones m i , i∈n, n represents the number of microphones, the position of the microphone in the environment map is M, and the sound signal received by each microphone from the sound source s(t) is x i , the calculation formulas for each sound signal are as follows:

[0118] x i (t) = s(t-τ i )+k i (t)

[0119] Among them, τ i represents the time it takes for the sound source to reach each microphone, k i Represents the additive noise of each sound signal during the transmission process.

[0120] Then, the time difference r of the sound signal from the sound source arriving at each microphone is i for:

[0121]

[0122] Among them, r i It represents the time difference between the i-th microphone and the j-th microphone among the n microphones when receiving the sound signal emitted by the sound source, M i represents the position of the i-th microphone among n microphones in the environment map, M j represents the position of the jth microphone among n microphones in the environment map, and c represents the speed of sound.

[0123] In addition, in order to reduce the impact of noise and reverberation on the transmission of sound signals in actual environments, when the geometric data of the microphone array is known, a delay estimation function is usually used to estimate the time difference of the sound signal arriving at each microphone. For example, a generalized cross-correlation function is used to weight the sound signal in the cross-power spectrum domain using a weighting function, and then the peak estimated delay is found after an inverse Fourier transform (IFFT) operation.

[0124] After obtaining the time difference of arrival of the sound signal of the sound source at each microphone, the azimuth angle of the sound source relative to the target device can be calculated based on the geometric relationship between the microphones in the microphone array, such as relative position, relative angle, etc.

[0125] Step S132: Based on the target initial sound source position and the target relative position, determine the target sound source position through the sound source localization correction model.

[0126] It should be noted that after obtaining the target initial sound source position, the sound source localization correction model can obtain the compensation amount of the target initial sound source position according to the target relative position between the target device and the obstacle, and then determine the target sound source position.

[0127] In step S130, the sound source localization model is stored in the server, and the target sound source position is determined by the sound source localization model based on the target sound data and the relative position of the target, including:

[0128] Sending the target sound data and the target relative position to the server, so that the server determines the target sound source position through a sound source localization model;

[0129] Acquire the target sound source position sent by the server.

[0130] Based on the above implementation, by storing the sound source localization model on the server, the target device sends the collected target sound data and the target relative position to the server, and the server outputs the target sound source position based on the sound source localization model and sends the target sound source position to the target device. Thus, the main computing tasks of sound source localization are deployed on the server for execution, which can reduce the hardware requirements for the target device and further reduce the hardware cost of the target device.

[0131] Based on the above disclosed content, the embodiment of the present application obtains the target sound data emitted by the sound source collected by the target device, and the target relative position between the target device and the obstacle, and uses the target relative position as a variable for locating the sound source emitting the target sound data, and outputs the target sound source position through the sound source localization model, thereby taking into account the influence of obstacles on the accuracy of sound source localization, improving the positioning accuracy of the target sound source position, and further improving the adaptability of the target device to complex environments.

[0132] In order to make those skilled in the art better understand the present application, Figures 2 to 6 The application scenarios involved in the embodiments of the present application are briefly described.

[0133] See also Figures 2 to 6 , Figure 2 A schematic diagram showing an application scenario of the sound source localization method in an embodiment of the present application is shown; Figure 3 A schematic diagram of the three-dimensional structure of the target device in the embodiment of the present application is shown; Figure 4 A schematic diagram of the side structure of the target device in an embodiment of the present application is shown; Figure 5A schematic diagram of a top view of the target device in an embodiment of the present application is shown; Figure 6 A schematic diagram of the control principle of the target device in the embodiment of the present application is shown.

[0134] like Figures 2 to 6 As shown, in the embodiment of the present application, the target device involved may be a mobile air-conditioning robot, which is applied to a home environment with obstacles such as walls, and the user, as a sound source, sends a sound signal to the mobile air-conditioning robot. In the process of performing cooling or heating tasks in the home environment, the mobile air-conditioning robot may move according to the needs of cooling or heating in various areas of the home environment. When moving to different positions, the target relative position of the obstacle and the mobile air-conditioning robot may be different, so the impact on the sound data emitted by the sound source may also be different. Therefore, the mobile air-conditioning robot needs to sense the different target relative positions between the obstacle and the mobile air-conditioning robot each time it moves to a different position.

[0135] In the embodiment of the present application, the mobile air-conditioning robot is provided with an interactive main controller, a navigation controller, a chassis motion controller, a microphone array and a laser module. The microphone array includes a plurality of microphones 41, and the plurality of microphones 41 are arranged in a ring on the mobile air-conditioning robot. The laser module includes a camera 42, a single-point TOF (Time Of Fight, time of flight) sensor 43, a laser radar 44 and a line laser sensor 45 and other position sensors. Specifically, the interactive main controller controls the navigation controller to achieve indoor navigation; the interactive main controller controls the chassis motion controller to achieve movement in the home environment; the sound signal emitted by the user is received by each microphone so that the interactive main controller calculates the time difference data of the sound signal reaching each microphone; the various position information in the home environment is collected by each position sensor to establish an environment map, and the position information of the mobile air-conditioning robot in the environment map and the position information of obstacles in the environment map are collected, so that the target relative position information between the mobile air-conditioning robot and the obstacle is calculated by the interactive main controller.

[0136] In an embodiment of the present application, communication is established between the mobile air-conditioning robot and a cloud server. The cloud server stores a positioning sound source model. The mobile air-conditioning robot sends the time difference data of the sound signal calculated above reaching each microphone and the geometric data of each microphone to the cloud server. The cloud server outputs the corresponding sound source position through the sound source positioning model, thereby being able to locate the user who sends the sound signal.

[0137] See also Figure 7 , shows a schematic structural diagram of a sound source localization device according to an embodiment of the present application.

[0138] like Figure 7 As shown, according to a second aspect of an embodiment of the present application, a sound source localization device 200 is provided, which is arranged in a target device to identify a sound source position, and includes:

[0139] A first acquisition unit 201 is used to acquire target sound data emitted by a sound source and collected by the target device;

[0140] A second acquisition unit 202 is used to acquire a target relative position between the target device and the obstacle;

[0141] The position determination unit 203 is used to determine the target sound source position through a sound source localization model based on the target sound data and the relative position of the target.

[0142] According to a third aspect of an embodiment of the present application, a computer-readable storage medium is provided, in which at least one computer program instruction is stored. The at least one computer program instruction is loaded and executed by a processor to implement the operations performed by any method described in the first aspect.

[0143] The computer readable storage medium may be a portable compact disk read-only memory (CD-ROM) and include program code, and may be run on a terminal device, such as a personal computer. However, the computer readable storage medium of the present application is not limited thereto. In the present application, the readable storage medium may be any tangible medium containing or storing a program, which may be used by or in combination with an instruction execution system, device or device. The readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination thereof. More specific examples of readable storage media (a non-exhaustive list) include: an electrical connection with one or more wires, a portable disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination thereof.

[0144] Program code for performing the operations of the present application may be written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Java, C++, etc., and conventional procedural programming languages ​​such as "C" or similar programming languages. The program code may be executed entirely on the user computing device, partially on the user device, as a separate software package, partially on the user computing device and partially on a remote computing device, or entirely on a remote computing device or server. In the case of a remote computing device, the remote computing device may be connected to the user computing device through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computing device (e.g., using an Internet service provider to connect through the Internet).

[0145] See also Figure 8 , showing a schematic diagram of the structure of a computer system of a household appliance suitable for implementing an embodiment of the present application.

[0146] According to a fourth aspect of an embodiment of the present application, a household appliance is provided, characterized in that it includes one or more processors and one or more memories, wherein at least one program code is stored in the one or more memories, and the at least one program code is loaded and executed by the one or more processors to implement the operations performed by any of the methods described in the first aspect.

[0147] like Figure 8 As shown, the household appliance 400 is in the form of a general computing device. The components of the household appliance 400 may include but are not limited to: at least one processing unit 410, at least one storage unit 420, and a bus 430 connecting different system components (including the storage unit 420 and the processing unit 410).

[0148] The storage unit stores program codes, which can be executed by the processing unit 410, so that the processing unit 410 executes the steps described in the above “Example Method” section of this specification according to various exemplary implementations of the present application.

[0149] The storage unit 420 may include a readable medium in the form of a volatile storage unit, such as a random access memory unit (RAM) 421 and / or a cache memory unit 422 , and may further include a read-only memory unit (ROM) 423 .

[0150] The storage unit 420 may also include a program / utility 424 having a set (at least one) of program modules 425, such program modules 425 including but not limited to: an operating system, one or more application programs, other program modules, and program data, each of which or some combination may include an implementation of a network environment.

[0151] Bus 430 may represent one or more of several types of bus structures, including a memory unit bus or memory unit controller, a peripheral bus, an accelerated graphics port, a processing unit, or a local bus using any of a variety of bus architectures.

[0152] The home appliance 400 may also communicate with one or more external devices 500 (e.g., keyboards, pointing devices, Bluetooth devices, etc.), may also communicate with one or more devices that enable a user to interact with the home appliance 400, and / or communicate with any device that enables the home appliance 400 to communicate with one or more other computing devices (e.g., routers, modems, etc.). Such communication may be performed via an input / output (I / O) interface 450. In addition, the home appliance 400 may also communicate with one or more networks (e.g., a local area network (LAN), a wide area network (WAN), and / or a public network, such as the Internet) via a network adapter 460. As shown, the network adapter 460 communicates with other modules of the home appliance 400 via a bus 430. It should be understood that, although not shown in the figure, other hardware and / or software modules may be used in conjunction with the home appliance 400, including but not limited to: microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, and data backup storage systems.

[0153] The functions described herein may be implemented in hardware, software executed by a processor, firmware, or any combination thereof. If implemented in software executed by a processor, the functions may be stored as one or more instructions or codes on a computer-readable medium or transmitted via a computer-readable medium. Other examples and implementations are within the scope and spirit of the present invention and the appended claims. For example, due to the nature of software, the functions described above may be implemented using software executed by a processor, hardware, firmware, hard wiring, or a combination of any of these. In addition, each functional unit may be integrated into a processing unit, each unit may exist physically separately, or two or more units may be integrated into one unit.

[0154] In the several embodiments provided in this application, it should be understood that the disclosed technical content can be implemented in other ways. Among them, the device embodiments described above are only schematic. For example, the division of the units can be a logical function division. There may be other division methods in actual implementation. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of units or modules, which can be electrical or other forms.

[0155] The units described as separate components may or may not be physically separated, and the components of the control device may or may not be physical units, that is, they may be located in one place or distributed in multiple units. Some or all of the units may be selected according to actual needs to achieve the purpose of the present embodiment.

[0156] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions for a computer device (which can be a personal computer, a server or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), mobile hard disk, magnetic disk or optical disk and other media that can store program codes.

[0157] The above description is only an embodiment of the present application and is not intended to limit the present application. For those skilled in the art, the present application may have various modifications and variations. Any modification, equivalent substitution, improvement, etc. made within the spirit and principle of the present application shall be included in the scope of the claims of the present application.

Claims

1. A sound source localization method, used for a target device to identify a sound source position, characterized in that: include: Acquiring target sound data emitted by a sound source and collected by the target device; Acquiring a target relative position between the target device and the obstacle; Based on the target sound data and the relative position of the target, the target sound source position is determined by a sound source localization model.

2. The method according to claim 1, characterized in that The target device is provided with a microphone array, and the step of obtaining target sound data emitted by a sound source and collected by the target device includes: Target sound data emitted by a sound source and collected by each microphone in the microphone array is obtained.

3. The method according to claim 2, characterized in that The method further comprises: Acquire first geometric data of the microphone array, and construct a first machine learning model according to the first geometric data and a first sound source localization algorithm; Acquire multiple groups of first sample data, wherein each group of first sample data includes a first sample sound source position, a first sample relative position between the target device and the obstacle, and first sample sound data emitted by the sound source at the first sample sound source position and collected by the microphone array; Based on the multiple groups of first sample data, the first machine learning model is trained to obtain the sound source localization model.

4. The method according to claim 2, characterized in that: The sound source localization model includes a sound source localization basic model and a sound source localization correction model, and the method further includes: Acquire second geometric data of the microphone array, and construct a sound source localization basic model according to the second geometric data and a second sound source localization algorithm; Acquire multiple groups of second sample data, wherein each group of second sample data includes a second sample sound source position, a second sample relative position between the target device and the obstacle, and a sample initial sound source position determined by the sound source localization basic model according to the second sample sound data, wherein the second sample sound data is sound data emitted by the sound source at the second sample sound source position and collected by the microphone array; Based on the multiple groups of second sample data, a second machine learning model is trained to obtain the sound source localization correction model.

5. The method according to claim 4, characterized in that The determining the target sound source position by using a sound source localization model based on the target sound data and the relative position of the target includes: Based on the target sound data, determining the target initial sound source position through the sound source localization basic model; Based on the target initial sound source position and the target relative position, the target sound source position is determined by the sound source localization correction model.

6. The method according to claim 5, characterized in that The determining the target initial sound source position based on the target sound data by using the sound source localization basic model comprises: Based on the target sound data, determining time difference data of the target sound emitted by the sound source reaching each microphone in the microphone array; Based on the time difference data, the target initial sound source position is determined by the sound source localization basic model.

7. The method according to claim 1, characterized in that The obtaining of the target relative position between the target device and the obstacle includes: Obtaining an environment map of the environment where the target device is located; Determining the locations of the target device and obstacles in the environment map; Based on the positions of the target device and the obstacle in the environment map, a target relative position between the target device and the obstacle is determined.

8. The method according to claim 1, characterized in that The sound source localization model is stored in a server, and determining the target sound source position through the sound source localization model based on the target sound data and the relative position of the target includes: Sending the target sound data and the target relative position to the server, so that the server determines the target sound source position through a sound source localization model; Acquire the target sound source position sent by the server.

9. A sound source localization device, provided in a target device to identify the location of a sound source, characterized in that: include: A first acquisition unit, configured to acquire target sound data emitted by a sound source and collected by the target device; A second acquisition unit, used to acquire a target relative position between the target device and the obstacle; The position determination unit is used to determine the target sound source position through a sound source localization model based on the target sound data and the relative position of the target.

10. A readable storage medium, characterized in that: The storage medium stores a computer program, and when the computer program is executed, the steps of the method according to any one of claims 1 to 8 can be implemented.

11. A household appliance, characterized in that: The method comprises a processor configured to execute the steps of the method according to any one of claims 1 to 8.

Images