Robot sound source localization method, device and equipment based on graph signal processing and storage medium

Abstract

The invention discloses a robot sound source localization method, device and equipment based on graph signal processing and a storage medium, and relates to the technical field of sound source localization, and the method comprises the steps: collecting an original voice signal through a robot microphone array; constructing a space diagram topological structure and a time diagram topological structure, and combining the space diagram topological structure and the time diagram topological structure; mapping the space-time diagram signal of the original voice signal to a diagram frequency domain through joint diagram Fourier transform, and calculating the power spectral density of the diagram frequency domain voice signal in each microphone channel; and sound source localization is carried out based on the MPDR filter of the graph frequency domain voice signal. According to the method, the construction method of the graph signal topological structure of the voice signals in the time dimension is designed, the situation that the voice signals change along with time is fully and effectively analyzed, and the feature information of the multi-channel voice signals is comprehensively analyzed from two dimensions in combination with the space graph topological structure constructed through the microphone steering vector according to the space layout.

Description

Technical Field

[0001] The present invention relates to the technical field of sound source localization, and in particular to a robot sound source localization method, device, equipment and storage medium based on graph signal processing. Background Art

[0002] Humanoid robots, with their ability to mimic human appearance and movements, are considered a major breakthrough in the field of science and technology. They have broad application prospects in multiple industries. In order to achieve more natural and intelligent communication, accurate sound localization and speech recognition technologies are essential for robots. These technologies enable robots to better understand human actions and intentions, thereby assisting humans to complete tasks more efficiently and accurately in various application scenarios.

[0003] Patent CN118534413A provides a robot sound source localization method that integrates audio and image features. It obtains audio information and image information separately, constructs the observation value of the target model, and then determines the target position of the target object through the expectation maximization algorithm. However, it needs to extract features from audio and image separately, which is highly complex and consumes a lot of computing resources.

[0004] Patent CN115713946A provides a sound source localization method based on a neural network, which converts the acquired speech signal into a spectrogram, separates the non-human voice signal through a neural network, and finally determines the direction of the human voice signal to achieve sound source localization. However, the patent does not comprehensively consider time and space for feature extraction, and uses a neural network for nonlinear calculation, which is highly complex. It requires high computing resources and is difficult to adapt to complex practical application scenarios. Summary of the invention

[0005] The technical problem to be solved by the present invention is to overcome the shortcomings of the prior art and provide a robot sound source localization method, device, equipment and storage medium based on graph signal processing.

[0006] In order to solve the above technical problems, the technical solution of the present invention is as follows: A robot sound source localization method based on graph signal processing, comprising: Use the robot microphone array to collect the original speech signal; A space graph topology structure and a time graph topology structure are constructed based on the original speech signal, and the space graph topology structure and the time graph topology structure are combined by using the Kronecker product to form a joint space-time graph topology structure; The spatiotemporal graph signal of the original speech signal is mapped to the graph frequency domain by joint graph Fourier transform, and the power spectral density of the graph frequency domain speech signal in each microphone channel is calculated; Sound source localization based on MPDR filter of speech signal in frequency domain.

[0007] As a preferred solution of the robot sound source localization method based on graph signal processing of the present invention, wherein: the construction of the spatial graph topology structure and the time graph topology structure based on the original speech signal includes: The original speech signal is passed through the spatial graph topology Represented, where Vc represents the vertex set of the spatial graph signal, and the value on the vertex represents the speech signal of each microphone, Ec represents the set of all edges, and the edges between each vertex reflect the relationship between the signals of each microphone, Ac is the spatial graph adjacency matrix, which represents the correlation between each vertex, and is defined as: ,in, is the steering vector of the microphone array, and the time delay of different microphones in the microphone array receiving the sound source is defined as , is the distance between the two microphones, is the angle between the incident direction of the sound wave and the sensor normal reaching the microphone array, c is the speed of sound in the air, then the steering vector of the microphone array is ; The original speech signal is passed through the time graph topology Represented as follows, where each frame of speech signal in each channel is taken as a vertex, and the vertices formed by each frame of speech signal are connected by edges. The i-th frame of speech signal in each channel is defined as the i-th vertex in the time graph topology, i=1,2,…,m, m is the number of frames of speech signal, then the weight of the edge between vertices i and j is defined as: ,in, is the weight factor between speech frames, then the time graph adjacency matrix At is: .

[0008] As a preferred solution of the robot sound source localization method based on graph signal processing of the present invention, wherein: the use of Kronecker product to combine the spatial graph topology structure and the temporal graph topology structure to form a joint space-time graph topology structure includes: The Kronecker product is used to combine the constructed spatial graph topology Gc with the temporal graph topology Gt to form a joint spatiotemporal graph topology, and the adjacency matrix A of the joint spatiotemporal graph is used. J express: .

[0009] As a preferred solution of the robot sound source localization method based on graph signal processing of the present invention, the mapping of the spatiotemporal graph signal of the original speech signal to the graph frequency domain by joint graph Fourier transform comprises: Based on the singular value decomposition of the graph adjacency matrix, and using the obtained singular eigenvectors to define the eigenbasis of the original speech signal, the spatial graph adjacency matrix Ac is: , where Vc and Dc represent the unitary matrices composed of left and right singular vectors, respectively. is the conjugate transpose of Dc, The diagonal is a singular value, and the time graph adjacency matrix At is: ; use and As the graph Fourier transform basis of spatial dimension and time dimension respectively, the original speech signal collected by the microphone array is transformed into Mapped to the frequency domain, it is expressed as: ,in, , which represents the joint Fourier transform basis of the image signal, is obtained based on the Kronecker product. , then the joint graph Fourier transform is expressed as .

[0010] As a preferred solution of the robot sound source localization method based on graph signal processing of the present invention, the calculation of the power spectrum density of the graph frequency domain speech signal in each microphone channel includes: Use JGFT to collect M channels of noisy speech signals from the microphone array Mapped to the frequency domain, we get ; The power spectrum density of the speech signal in the frequency domain is calculated as: ,in, is the speech signal in the frequency domain The covariance matrix of the noisy speech signal and the power spectrum density of the noise is obtained as follows: .

[0011] As a preferred solution of the robot sound source localization method based on image signal processing of the present invention, the sound source localization based on the MPDR of the image frequency domain speech signal includes: The filter h is set to ensure that the output power of the speech signal after the filter h is as small as possible while ensuring that the speech signal in the direction of the sound source is not distorted, and the optimization problem is obtained: ,in, , solve the optimization problem and get the optimal MPDR filter as: , the power P is: ; Calculate the power P successively, search for the spectrum peak of P, and determine the corresponding direction angle , and based on the direction angle Locate the source of the sound.

[0012] As a preferred solution of the robot sound source localization method based on image signal processing of the present invention, after the sound source localization is performed based on the MPDR of the image frequency domain speech signal, the method further includes: The enhanced image-frequency domain speech signal calculated based on the optimal MPDR filter formula is: ; Performing inverse Fourier transform on the enhanced frequency domain speech signal to obtain an enhanced speech signal, and outputting it to the robot for recognition; The robot obtains the direction of the sound source and the enhanced voice signal, converts the voice signal into motion instructions using speech recognition technology, and performs corresponding robot operations based on the motion instructions.

[0013] The present invention also provides a robot sound source localization device based on graph signal processing, comprising: A voice signal acquisition module is used to collect original voice signals using a robot microphone array; A graph signal processing module is used to construct a spatial graph topology structure and a temporal graph topology structure based on the original speech signal, and use the Kronecker product to combine the spatial graph topology structure and the temporal graph topology structure to form a joint space-time graph topology structure, map the space-time graph signal of the original speech signal to the graph frequency domain through a joint graph Fourier transform, and calculate the power spectral density of the speech signal in the graph frequency domain in each microphone channel; The sound source localization module is used to localize the sound source based on the MPDR filter of the image-frequency domain speech signal, calculate the enhanced speech signal, and output it to the robot for recognition; The speech recognition processing module is used to obtain the direction of the sound source and the enhanced speech signal, convert the speech signal into motion instructions using speech recognition technology, and perform corresponding robot operations based on the motion instructions.

[0014] The present invention also discloses a computer device, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the program, the method described in any one of the above-mentioned methods for robot sound source localization based on graph signal processing is implemented.

[0015] The present invention also discloses a computer-readable storage medium on which a computer program is stored. When the program is executed by a processor, the method described in any of the above-mentioned robot sound source localization methods based on graph signal processing is implemented.

[0016] The beneficial effects of the present invention are: (1) The present invention designs a method for constructing a graph signal topological structure of a speech signal in the time dimension. The speech time graph topological structure is constructed based on the correlation between the speech frames after the speech signal is framed, so as to fully and effectively analyze the changes of the speech signal over time. Combined with the spatial graph topological structure constructed by using microphone steering vectors according to the spatial layout, the characteristic information of the multi-channel speech signal is comprehensively analyzed from two dimensions.

[0017] (2) The present invention designs a sound source localization method based on graph signals, uses joint graph Fourier transform for linear calculation, maps the graph signal from the vertex domain to the graph frequency domain, and extracts features from two dimensions, intersecting with nonlinear calculation based on neural network, reducing the consumption of computing resources and enhancing scalability. Specifically, the multi-channel speech signal collected by the microphone array is mapped to the graph frequency domain through joint graph Fourier transform using graph signal processing technology, and an MPDR filter is constructed, and then the angles are traversed to find the maximum signal power, which is the direction of the sound source.

[0018] (3) The present invention further designs a speech enhancement scheme based on MPDR. By using the designed graph MPDR filter, the original signal can be filtered to obtain a speech graph signal with maximum noise suppression and maximum speech power gain. Then, the inverse joint graph Fourier transform can be used to obtain an enhanced speech signal, which provides a guarantee for robot speech recognition. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other accompanying drawings can be obtained based on these accompanying drawings without paying creative labor.

[0020] Figure 1 A schematic diagram of the flow of a robot sound source localization method based on graph signal processing provided by the present invention; Figure 2 It is a schematic diagram of the joint spatiotemporal graph topology of the speech signal of the microphone array in different time frames; Figure 3 A schematic diagram of a robot sound source localization device based on graph signal processing provided by the present invention; Figure 4 A schematic diagram of a computer device provided by the present invention. DETAILED DESCRIPTION

[0021] In order to make the content of the present invention more clearly understood, the present invention is further described in detail below according to specific implementation modes and in combination with the accompanying drawings.

[0022] Figure 1 A schematic flow chart of a robot sound source localization method based on graph signal processing provided in an embodiment of the present application. The method specifically comprises the following steps: Step S101: using the robot microphone array to collect original speech signals.

[0023] Specifically, in speech signal processing, traditional digital signal processing methods usually deal with time-varying and non-stationary problems through short-term analysis, but this method often loses some valuable information. The speech signal collected by the microphone array is a multi-channel signal with spatial and temporal characteristics. By constructing an effective graph topology, the relevant information between signal samples can be better captured, thereby improving the processing effect.

[0024] The original speech data is collected using the microphone array on the robot. Assume that the original speech signal collected by microphone n (n=1,2,…,N) at time t is , the voice signals collected by each microphone are integrated and framed, and the original voice signal is obtained as a tensor , where M represents the number of channels, T represents the number of frames, and F represents the frame length.

[0025] Step S102: constructing a space graph topology structure and a time graph topology structure based on the original speech signal, and using the Kronecker product to unite the space graph topology structure and the time graph topology structure to form a joint space-time graph topology structure.

[0026] Specifically, in this embodiment, the correlation between the time and channels of the speech signal is comprehensively considered, and the speech signal is analyzed in two dimensions. Compared with only considering the spatial dimension or only considering the temporal dimension, the accuracy is higher.

[0027] The collected speech signal is divided into frames. First, the microphones in the microphone array are taken as vertices in space, and a graph topology structure in the spatial dimension is constructed in the vertex domain. At the same time, because the speech signals in the same channel have a high correlation between adjacent frames, the speech signal of each frame is taken as a vertex to construct a graph topology structure in the time dimension. On this basis, the joint time-space graph topology structure is constructed through the Kronecker product.

[0028] Figure 2The figure is a schematic diagram of the topological structure of the joint space-time graph of the speech signal of the microphone array in different time frames. As shown in the figure, in the spatial dimension, each microphone represents a vertex, and the value on each vertex represents the speech signal collected in a certain time frame. The weight between two vertices represents the correlation between the two vertices. The larger the weight, the stronger the correlation between the two vertices, that is, the lower the time delay of the speech signals received by the two microphones. In the time dimension, each speech frame represents a vertex, and the value on each vertex represents each speech frame signal collected in a microphone channel. The value on each vertex represents a speech signal of a certain time frame collected by a certain microphone. The larger the weight between two vertices, the higher the similarity of the speech signals of adjacent frames.

[0029] Spatial graph topology: For multi-channel speech signals, when only analyzing from the spatial dimension, that is, within a fixed time frame, facing the same sound source, the signal can be represented as: , passing the signal through the time graph topology Represents. Each microphone in the microphone array is regarded as a vertex V n ,So It is represented as a vertex set of spatial graph signals. The values ​​on the vertices represent the speech signals of each microphone. The edges between each vertex reflect the relationship between the signals of each microphone. All the edges form a set E c The steering vector of the microphone array is used to represent the weight of each edge, that is, the spatial graph adjacency matrix To represent the correlation between vertices, it is defined as: ,in, is the steering vector of the microphone array. Since the relative positions of the microphones in the microphone array are different, there is a delay in receiving the sound source, which is defined as , is the distance between the two microphones, is the angle between the incident direction of the sound wave and the sensor normal reaching the microphone array, and c is the speed of sound in the air. Then the steering vector of the microphone array is Through this composition method, multi-channel speech signals are represented by graph topology, which effectively reveals the correlation between speech signals in various channels.

[0030] Time graph topology: For the speech signal X, only looking at its time dimension, in the same microphone channel, it can be represented as a signal , passing the signal through the time graph topology Represented by. Consider each frame of the speech signal in each channel as a vertex, assuming that the vertices formed by each frame of the speech signal are connected by edges. Define the i-th (i=1,2,…,m) frame of the speech signal of each channel as the i-th vertex in the time graph topology, where m is the number of frames of the speech signal. We know that the closer the distance in time, the greater the correlation between speech frames, so the weight of the edge between vertices i and j is defined as: ,in, is the weight factor between speech frames. The lower the correlation between two speech frames, The larger the value, the lower the value. Then the time graph adjacency matrix At is: .

[0031] Joint space-time graph topology: A spatial graph topology G constructed on the spatial and temporal dimensions of multi-channel speech signals c and the temporal graph topology G t , these two topological structures are combined by using the Kronecker product to form a joint space-time graph topological structure, using the joint space-time graph adjacency matrix A J express: .

[0032] In order to analyze the sound source direction of the speech signal X obtained from the microphone array, we need to transform the speech signal X and the joint space-time graph adjacency matrix A J Pass the processing layer to the next step.

[0033] Step S103: Map the spatiotemporal graph signal of the original speech signal to the graph-frequency domain by joint graph Fourier transform, and calculate the power spectral density of the graph-frequency domain speech signal in each microphone channel.

[0034] Specifically, the speech signal X collected from the microphone array needs to be calculated by the robot's processor using graph signal processing through graph MPDR. First, the speech signal X needs to be jointly Fourier transformed to map it to the graph frequency domain, and then the graph frequency domain speech signal power spectral density in each microphone channel is calculated, and finally the graph MPDR is used to estimate the sound source direction.

[0035] It should be noted that graph signals are often used to process signals in irregular domains. Graph signals are often represented by graph topology, represented as G above, and the adjacency matrix A or Laplacian matrix in G is used for calculation.

[0036] The specific operations of this step are as follows: Joint graph Fourier transform: Use singular value decomposition based on the graph adjacency matrix and use the obtained singular eigenvectors to define the eigenbasis of the multi-channel speech graph signal, because the eigenbasis based on singular decomposition can ensure that the graph frequency of the speech signal in the graph frequency domain is non-negative, so that the speech signal will not be lost or moved. Then the spatial graph adjacency matrix Ac can be expressed as: , where Vc and Dc represent the unitary matrices composed of left and right singular vectors, respectively. is the conjugate transpose of Dc, and the diagonal of Ac is the singular value. Similarly, the time graph adjacency matrix At is: Then the Fourier transform of the graph is used for processing in both spatial and temporal dimensions, that is, and As the graph Fourier transform basis of spatial dimension and time dimension respectively, the original speech signal collected by the microphone array is transformed into Mapped to the frequency domain, it is expressed as: .in, represents the joint Fourier transform basis of the image signal, then using the properties of the Kronecker product, we can get , then the joint graph Fourier transform is expressed as .

[0037] Figure Calculation of the power spectral density of speech signals in the frequency domain: For the M channels of noisy speech signals collected by the microphone array , use JGFT to map the signal to the frequency domain , assuming that the noise is irrelevant to the speech signal, the power spectrum density of the speech signal in the frequency domain can be obtained as: ,in, is the speech signal in the frequency domain The covariance matrix of . Then we can get the relationship between the power spectrum density of the noisy speech signal and the noise: .

[0038] Step S104: Perform sound source localization based on the MPDR filter of the image-frequency domain speech signal.

[0039] Specifically, if the speech signal X is the speech direction we need, then the average power of the signal in this direction should be the largest, that is, the signals and noise in other directions are suppressed to the greatest extent. While using MPDR to locate the sound source, we can obtain the filter h in the direction with the maximum power. By filtering the signal in the direction of the sound source through this h, we can also obtain the enhanced signal with the maximum power. Then, we design a filter h that meets the following conditions: Under the premise of ensuring that the speech signal in the direction of the sound source is not distorted, the output power of the signal passing through h, that is, the speech signal weighted by the h vector, is as small as possible, and the corresponding optimization problem can be obtained: ,in, , ensuring that the direction The speech signal can get the maximum output.

[0040] Solving this optimization problem, we can get the optimal MPDR filter as: , so the power P can be obtained as: Then, scan at a certain observation angle according to a certain step, calculate the power P one by one, and search for the spectrum peak of P to determine the corresponding direction angle. , and based on the direction angle Locate the source of the sound.

[0041] Step S105: The enhanced frequency domain speech signal is calculated based on the optimal MPDR filter formula: ; Perform inverse Fourier transform on the enhanced frequency domain speech signal to obtain the enhanced speech signal, and output it to the robot for recognition; the robot obtains the direction of the sound source and the enhanced speech signal, uses speech recognition technology to convert the speech signal into motion instructions, and performs corresponding robot operations based on the motion instructions.

[0042] Specifically, using the solution formula The optimal MPDR filter can be used to obtain the enhanced speech signal. Finally, the enhanced frequency domain speech signal is inversely transformed and overlapped and added to obtain the enhanced speech signal, which is then output to the robot for recognition and corresponding actions. After obtaining the speech direction and enhanced speech signal, the robot uses speech recognition technology to understand the words in the speaker's speech as movement instructions, and then performs corresponding robot operations on the parsed movement instructions.

[0043] Specifically, the speech graph spectrum features are extracted from the noise-reduced and enhanced speech signal, and the speech features are input into the speech recognition model trained using the graph spectrum to obtain the speech recognition results. The recognition results are then matched with the robot's command library to identify the corresponding commands, and the robot is controlled to perform corresponding operations according to the commands.

[0044] Therefore, the above technical solution takes into account the time-varying multi-channel speech signals and the correlation between channels, constructs the graph topology of the multi-channel speech signals from these two dimensions, and comprehensively analyzes the graph signals in the two dimensions of time and space to more effectively extract the feature information of the multi-channel speech signals and accurately characterize the audio features of the speaker. At the same time, the use of graph signal processing technology and linear transformation to capture structural information from high-dimensional data for feature extraction can effectively save computing resources compared to the extraction of latent space information by nonlinear feature transformation based on neural networks. In addition, the MPDR filter based on graph signals is further designed to locate the direction of the sound source using the frequency domain information of the graph, and to suppress noise and enhance the signal of the speech signal in the direction of the sound source, providing a guarantee for subsequent speech recognition.

[0045] Figure 3 The schematic diagram of the robot sound source localization device based on graph signal processing provided in the embodiment of the present application includes: a speech signal acquisition module 201, a graph signal processing module 202, a sound source localization module 203 and a speech recognition processing module 204.

[0046] Specifically, the voice signal acquisition module is used to collect original voice signals using the robot microphone array.

[0047] The graph signal processing module is used to construct a spatial graph topology structure and a temporal graph topology structure based on the original speech signal, and use the Kronecker product to unite the spatial graph topology structure and the temporal graph topology structure to form a joint space-time graph topology structure. The space-time graph signal of the original speech signal is mapped to the graph frequency domain through a joint graph Fourier transform, and the power spectral density of the graph-frequency domain speech signal in each microphone channel is calculated.

[0048] The sound source localization module is used to localize the sound source based on the MPDR filter of the frequency domain speech signal, calculate the enhanced speech signal, and output it to the robot for recognition.

[0049] The speech recognition processing module is used to obtain the direction of the sound source and the enhanced speech signal, convert the speech signal into motion instructions using speech recognition technology, and perform corresponding robot operations based on the motion instructions.

[0050] By integrating voice acquisition modules, image signal processing modules, sound source localization modules, and voice recognition modules, the robot can efficiently capture, accurately analyze, and intelligently respond to voice signals in the surrounding environment. The collaborative work of these modules enables the robot to accurately identify the speaker's location, obtain the content of their voice, and analyze the user's intentions through in-depth semantic understanding. Based on these comprehensive analyses, the robot generates corresponding movement or behavior instructions, such as moving to the user's side, adjusting posture for a more natural conversation, and even performing specific tasks according to voice instructions. Through this series of steps, the robot not only has the ability to perceive, understand, and respond, but also makes human-computer interaction more natural and smooth, bringing users intelligent services close to human interaction experience. This capability can be widely used in many fields such as home assistants, public services, medical rehabilitation, etc., further promoting the popularization and application of robot technology.

[0051] See also Figure 4 This embodiment also provides a computer device, and the components of the computer device may include but are not limited to: one or more processors or processing units, a system memory, and a bus connecting different system components (including the system memory and the processing unit).

[0052] The term "bus" refers to one or more of several types of bus structures, including a memory bus or memory controller, a peripheral bus, an accelerated graphics port, a processor or a local bus using any of a variety of bus architectures. For example, these architectures include, but are not limited to, the Industry Standard Architecture (ISA) bus, the Micro Channel Architecture (MAC) bus, the Enhanced ISA bus, the Video Electronics Standards Association (VESA) local bus, and the Peripheral Component Interconnect (PCI) bus.

[0053] The computer system / server typically includes a variety of computer system readable media. These media can be any available media that can be accessed by the computer system / server, including volatile and non-volatile media, removable and non-removable media.

[0054] The system memory may include computer system readable media in the form of volatile memory, such as random access memory (RAM) and / or cache memory. The computer device may further include other removable / non-removable, volatile / non-volatile computer system storage media. By way of example only, the storage system may be used to read and write non-removable, non-volatile magnetic media. A disk drive for reading and writing removable non-volatile disks (such as "floppy disks") and an optical disk drive for reading and writing removable non-volatile optical disks (such as CD-ROMs, DVD-ROMs or other optical media) may be provided. In these cases, each drive may be connected to the bus via one or more data medium interfaces. The memory may include at least one program product having a set (e.g., at least one) of program modules that are configured to perform the functions of the various embodiments of the present invention.

[0055] A program / utility having a set (at least one) of program modules may be stored, for example, in a memory, such program modules including, but not limited to, an operating system, one or more application programs, other program modules, and program data, each of which or some combination may include an implementation of a network environment. The program modules generally perform the functions and / or methods of the embodiments described herein.

[0056] The computer device may also communicate with one or more external devices such as a keyboard, a pointing device, a display, etc. Such communication may be performed through an input / output (I / O) interface. Furthermore, the computer device may also communicate with one or more networks (such as a local area network (LAN), a wide area network (WAN) and / or a public network, such as the Internet) through a network adapter.

[0057] The processing unit executes the functions and / or methods described in the embodiments of the present invention by running the programs stored in the system memory.

[0058] The above-mentioned computer program can be set in a computer storage medium, that is, the computer storage medium is encoded with a computer program, and when the program is executed by one or more computers, it enables one or more computers to execute the method flow and / or device operation shown in the above-mentioned embodiments of the present invention.

[0059] With the development of time and technology, the meaning of medium is becoming more and more extensive, and the propagation path of computer programs is no longer limited to tangible media, but can also be downloaded directly from the network, etc. Any combination of one or more computer-readable media can be used. Computer-readable media can be computer-readable signal media or computer-readable storage media. Computer-readable storage media can be, for example, - but not limited to - electrical, magnetic, optical, electromagnetic, infrared, or semiconductor systems, devices or devices, or any combination of the above. More specific examples of computer-readable storage media (non-exhaustive list) include: electrical connections with one or more wires, portable computer disks, hard disks, random access memories (RAM), read-only memories (ROM), erasable programmable read-only memories (EPROM or flash memory), optical fibers, portable compact disk read-only memories (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the above. In this document, computer-readable storage media can be any tangible medium containing or storing programs, which can be used by or in combination with instruction execution systems, devices or devices.

[0060] Computer-readable signal media may include a data signal propagated in baseband or as part of a carrier wave, which carries a computer-readable program code. Such propagated data signals may take a variety of forms, including, but not limited to, electromagnetic signals, optical signals, or any suitable combination of the above. Computer-readable signal media may also be any computer-readable medium other than a computer-readable storage medium, which may send, propagate, or transmit a program for use by or in conjunction with an instruction execution system, apparatus, or device.

[0061] Program code embodied on a computer readable medium may be transmitted using any appropriate medium, including but not limited to wireless, wireline, optical fiber cable, RF, etc., or any suitable combination of the foregoing.

[0062] Computer program code for performing the operations of the present invention may be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages, such as Java, Smalltalk, C++, and conventional procedural programming languages, such as "C" or similar programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, as a separate software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0063] In addition to the above embodiments, the present invention may also have other implementation modes; any technical solutions formed by equivalent replacement or equivalent transformation shall fall within the protection scope required by the present invention.

Claims

1. A robot sound source localization method based on graph signal processing, characterized in that: include: Use the robot microphone array to collect the original speech signal; A space graph topology structure and a time graph topology structure are constructed based on the original speech signal, and the space graph topology structure and the time graph topology structure are combined by using the Kronecker product to form a joint space-time graph topology structure; The spatiotemporal graph signal of the original speech signal is mapped to the graph frequency domain by joint graph Fourier transform, and the power spectral density of the graph frequency domain speech signal in each microphone channel is calculated; Sound source localization based on MPDR filter of speech signal in frequency domain.

2. The robot sound source localization method based on graph signal processing according to claim 1, characterized in that: The construction of the spatial graph topology structure and the temporal graph topology structure based on the original speech signal comprises: The original speech signal is passed through the spatial graph topology Represented, where Vc represents the vertex set of the spatial graph signal, and the value on the vertex represents the speech signal of each microphone, Ec represents the set of all edges, and the edges between each vertex reflect the relationship between the signals of each microphone, Ac is the spatial graph adjacency matrix, which represents the correlation between each vertex, and is defined as: ,in, is the steering vector of the microphone array, and the time delay of different microphones in the microphone array receiving the sound source is defined as , is the distance between the two microphones, is the angle between the incident direction of the sound wave and the sensor normal reaching the microphone array, c is the speed of sound in the air, then the steering vector of the microphone array is ; The original speech signal is passed through the time graph topology Represented as follows, where each frame of speech signal in each channel is taken as a vertex, and the vertices formed by each frame of speech signal are connected by edges. The i-th frame of speech signal in each channel is defined as the i-th vertex in the time graph topology, i=1,2,…,m, m is the number of frames of speech signal, then the weight of the edge between vertices i and j is defined as: ,in, is the weight factor between speech frames, then the time graph adjacency matrix At is: .

3. The robot sound source localization method based on graph signal processing according to claim 2, characterized in that: The method of using the Kronecker product to combine the spatial graph topology structure and the temporal graph topology structure to form a combined spatial-temporal graph topology structure includes: The Kronecker product is used to combine the constructed spatial graph topology Gc with the temporal graph topology Gt to form a joint spatiotemporal graph topology, and the adjacency matrix A of the joint spatiotemporal graph is used. J express .

4. The robot sound source localization method based on graph signal processing according to claim 3 is characterized in that: Mapping the space-time graph signal of the original speech signal to the graph frequency domain by joint graph Fourier transform includes: Based on the singular value decomposition of the graph adjacency matrix, and using the obtained singular eigenvectors to define the eigenbasis of the original speech signal, the spatial graph adjacency matrix Ac is: , where Vc and Dc represent the unitary matrices composed of left and right singular vectors, respectively. is the conjugate transpose of Dc, The diagonal is a singular value, and the time graph adjacency matrix At is: ;use and As the graph Fourier transform basis of spatial dimension and time dimension respectively, the original speech signal collected by the microphone array is transformed into Mapped to the frequency domain, it is expressed as: ,in, , which represents the joint Fourier transform basis of the image signal, is obtained based on the Kronecker product. , then the joint graph Fourier transform is expressed as .

5. The robot sound source localization method based on graph signal processing according to claim 4, characterized in that: The calculating of the power spectral density of the frequency domain speech signal in each microphone channel comprises: Use JGFT to collect M channels of noisy speech signals from the microphone array Mapped to the frequency domain, we get ; The power spectrum density of the speech signal in the frequency domain is calculated as: ,in, is the speech signal in the frequency domain The covariance matrix of the noisy speech signal and the power spectrum density of the noise is obtained as follows: .

6. The robot sound source localization method based on graph signal processing according to claim 5, characterized in that: The MPDR filter based on the image frequency domain speech signal performs sound source localization, including: The filter h is set to ensure that the output power of the speech signal after the filter h is as small as possible while ensuring that the speech signal in the direction of the sound source is not distorted, and the optimization problem is obtained: ,in, , solve the optimization problem and get the optimal MPDR filter as: , the power P is: ; Calculate the power P successively, search for the spectrum peak of P, and determine the corresponding direction angle , and based on the direction angle Locate the source of the sound.

7. The robot sound source localization method based on graph signal processing according to claim 6, characterized in that: After the MPDR based on the image frequency domain speech signal performs sound source localization, the method further includes: The enhanced image-frequency domain speech signal calculated based on the optimal MPDR filter formula is: ; Performing inverse Fourier transform on the enhanced frequency domain speech signal to obtain an enhanced speech signal, and outputting it to the robot for recognition; The robot obtains the direction of the sound source and the enhanced voice signal, converts the voice signal into motion instructions using speech recognition technology, and performs corresponding robot operations based on the motion instructions.

8. A robot sound source localization device based on graph signal processing, characterized in that: include: A voice signal acquisition module is used to collect original voice signals using a robot microphone array; A graph signal processing module is used to construct a spatial graph topology structure and a temporal graph topology structure based on the original speech signal, and use the Kronecker product to combine the spatial graph topology structure and the temporal graph topology structure to form a joint space-time graph topology structure, map the space-time graph signal of the original speech signal to the graph frequency domain through a joint graph Fourier transform, and calculate the power spectral density of the speech signal in the graph frequency domain in each microphone channel; The sound source localization module is used to localize the sound source based on the MPDR filter of the image-frequency domain speech signal, calculate the enhanced speech signal, and output it to the robot for recognition; The speech recognition processing module is used to obtain the direction of the sound source and the enhanced speech signal, convert the speech signal into motion instructions using speech recognition technology, and perform corresponding robot operations based on the motion instructions.

9. A computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the program, the method according to any one of claims 1 to 7 is implemented.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the method according to any one of claims 1 to 7 is implemented.

Images