Mobile robot sound source tracking method, target mobile robot, equipment and medium

Abstract

The embodiment of the invention provides a mobile robot sound source tracking method, a target mobile robot, equipment and a medium, and belongs to the technical field of robot intelligent control, the method is applied to the target mobile robot, and the method comprises the following steps: collecting and positioning a multi-channel audio signal to obtain an estimated navigation point; performing weighted average processing on the estimated navigation point and the historical navigation point to obtain a target navigation point, performing global and local path fusion planning on the environment states of the target navigation point and the target mobile robot, outputting a fusion control instruction, and performing navigation control according to the fusion control instruction. According to the embodiment of the invention, the estimated navigation point is generated by collecting and positioning the audio signal, and weighted average processing is performed by combining the historical navigation point, so that the sound source can be positioned only according to the sound source signal, and real-time path planning and control instruction generation are performed by further combining the environment state and the target navigation point; operation of the mobile robot can be ensured in a graph-free environment, and the tracking effect of the mobile robot is improved.

Description

Technical Field

[0001] This application relates to the field of robot intelligent control technology, and in particular to a method for tracking the sound source of a mobile robot, a target mobile robot, equipment, and medium. Background Technology

[0002] Currently, traditional mobile robot tracking methods typically rely on onboard optical sensors, vision sensors, or lidar for visual tracking of targets. For example, in safety production and disaster prevention, mitigation, and relief operations, mobile robots first use onboard optical sensors, vision sensors, or lidar to collect visual information about the target. Based on this visual information, they locate the target's position and plan a navigation path, which the robot then follows to track the target. However, this method is ineffective in dark room detection scenarios or congested pipeline systems (such as underground natural gas pipelines, gasoline pipelines, and localized cable integration systems). These spaces are characterized by light blocking, dense equipment deployment leading to limited operational space, and the inability of optical sensors like depth cameras to function effectively. General lidar also struggles to provide localization, making accurate target location difficult for mobile robots. Therefore, solving the problem of accurate target location for mobile robots has become a challenging research issue.

[0003] To address this issue, existing technologies equip mobile robots with acoustic sensing elements and combine them with static prior maps or known environmental models to perform autonomous navigation tasks. However, in practical applications, these methods suffer from complex acoustic reflections and multipath effects introduced by obstacles such as urban environments, buildings, and vehicles. Environmental information is not always available or consistently perceived, leading to uncertainty in the estimation of sound source locations and potentially inaccurate tracking. Furthermore, in crowded dynamic environments, existing mobile robot navigation methods can only achieve simple path planning, resulting in unreasonable navigation path planning and an inability to effectively achieve dynamic obstacle avoidance capabilities. Consequently, the tracking performance of mobile robots is poor. Therefore, improving the tracking performance of mobile robots has become an urgent problem to be solved. Summary of the Invention

[0004] The main objective of this application is to propose a method for tracking a sound source in a mobile robot, a target mobile robot, a device, and a medium, with the aim of improving the tracking performance of the mobile robot.

[0005] To achieve the above objectives, a first aspect of this application proposes a method for tracking a sound source in a mobile robot, applied to a target mobile robot, the method comprising:

[0006] Signal acquisition is performed using a three-dimensional spherical microphone array module to obtain multi-channel audio signals;

[0007] The sound source localization module is used to locate the sound source of the multi-channel audio signal to obtain the estimated navigation point of the target time step;

[0008] The target navigation point is obtained by dynamically averaging the estimated navigation point and the historical navigation points of the historical time step through the navigation point calculation module.

[0009] The autonomous navigation fusion module performs global and local path fusion planning on the target navigation point and the environmental state of the target mobile robot, and outputs fusion control commands.

[0010] The real-time control and execution module drives the motion actuator to complete navigation control according to the fused control commands.

[0011] In some embodiments, the step of performing a weighted average dynamic processing on the estimated navigation point and historical navigation points at historical time steps through the navigation point calculation module to obtain the target navigation point includes:

[0012] Construct a sliding time window based on the historical navigation points of the historical time steps;

[0013] The historical navigation points at the historical time steps are subjected to median filtering to obtain filtered navigation points;

[0014] Within the sliding time window and in the preset direction, the weights of the filtered navigation points are calculated to obtain the target weights.

[0015] Within the sliding time window, a weighted average is performed on the filtered navigation point and the target navigation point to obtain the target navigation point.

[0016] In some embodiments, the autonomous navigation fusion module performs global and local path fusion planning on the target navigation point and the environmental state of the target mobile robot, and outputs fusion control commands, including:

[0017] Global path navigation planning is performed on the target navigation point and the environmental state of the target mobile robot to obtain global control commands;

[0018] The target navigation point is discretely sampled according to the global control command to obtain discrete path points;

[0019] Obtain the timestamps corresponding to the discrete points of the path, and construct a preliminary local path navigation function based on the discrete points of the path and the timestamps to obtain the preliminary local path navigation function;

[0020] By adding velocity and acceleration constraints, obstacle avoidance constraints, and time interval constraints to the initial local path navigation function, an optimized local path navigation function is obtained.

[0021] The target local path navigation function is obtained by weighted summation of the optimized local path navigation function, velocity, and acceleration.

[0022] The target local path navigation function is solved to obtain the fused control command.

[0023] In some embodiments, the step of constructing a preliminary local path navigation function based on the path discrete points and the timestamps to obtain the preliminary local path navigation function includes:

[0024] The first function term is constructed based on two adjacent discrete points of the path;

[0025] Construct a second function term based on three adjacent discrete points along the path;

[0026] Construct a third function term based on two adjacent timestamps;

[0027] The first function term, the second function term, and the third function term are summed to obtain the preliminary local path navigation function.

[0028] In some embodiments, after obtaining the target local path navigation function by weighted summation based on the optimized local path navigation function, velocity, and acceleration, the method further includes:

[0029] The maximum linear velocity and maximum angular velocity of the target mobile robot are obtained, and the target local path navigation function is modified according to the maximum linear velocity and the maximum angular velocity to obtain the modified velocity constraint.

[0030] The target local path navigation function is modified with acceleration constraints according to the modified velocity constraints to obtain the modified acceleration constraints.

[0031] Based on the modified acceleration constraint, the target local path navigation function is modified by a two-dimensional velocity increment space constraint to obtain the modified velocity vector constraint.

[0032] The initial feedback speed of the target mobile robot is obtained. If the initial feedback speed exceeds the preset speed range, the initial feedback speed is corrected to obtain the corrected feedback speed constraint.

[0033] The target local path navigation function is modified according to the preset speed constraints, the modified speed constraints, the modified acceleration constraints, the modified speed vector constraints, and the modified feedback speed constraints to obtain the modified local path navigation function, and the modified local path navigation function is determined as the target local path navigation function.

[0034] In some embodiments, the step of locating the sound source of the multi-channel audio signal using a sound source localization module to obtain the estimated navigation point for the target time step includes:

[0035] The time difference phase of the multi-channel audio signal is calculated in a predetermined direction to obtain the target time difference phase correlation value of the target time step;

[0036] The beam energy of the multi-channel audio signal in a predetermined direction is calculated based on the target time difference phase correlation value to obtain the target beam energy.

[0037] The location corresponding to the largest beam energy from the target beam energy is selected as the estimated navigation point for the target time step.

[0038] In some embodiments, the step of calculating the time difference phase of the multi-channel audio signal in a predetermined direction to obtain the target time difference phase correlation value for the target time step includes:

[0039] Obtain the delay time between the multi-channel audio signals;

[0040] The multi-channel audio signal is subjected to signal spectrum processing to obtain the target signal spectrum;

[0041] Obtain the spectral conjugate complex number of the target signal spectrum, and the target exponent term;

[0042] The target time difference phase correlation value of the target time step is obtained by calculating the delay time phase based on the delay time, the target signal spectrum, the spectrum conjugate complex number, and the target exponent term.

[0043] To achieve the above objectives, a second aspect of this application provides a target mobile robot, comprising:

[0044] A three-dimensional spherical microphone array module is used for signal acquisition to obtain multi-channel audio signals;

[0045] The sound source localization module is used to locate the sound source of the multi-channel audio signal and obtain the estimated navigation point of the target time step;

[0046] The navigation point calculation module is used to perform a weighted average dynamic processing on the estimated navigation point and the historical navigation points of the historical time step to obtain the target navigation point;

[0047] The autonomous navigation fusion module is used to perform global and local path fusion planning on the target navigation point and the environmental state of the target mobile robot, and output fusion control commands.

[0048] The real-time control and execution module is used to drive the motion actuator to complete navigation control according to the fused control instructions.

[0049] To achieve the above objectives, a third aspect of this application provides an electronic device, which includes a memory and a processor. The memory stores a computer program, and the processor executes the computer program to implement the method described in the first aspect.

[0050] To achieve the above objectives, a fourth aspect of the present application provides a storage medium storing a computer program that, when executed by a processor, implements the method of the first aspect described above.

[0051] The mobile robot sound source tracking method, target mobile robot, device, and medium proposed in this application first acquire signals through a three-dimensional spherical microphone array module to obtain multi-channel audio signals. Then, a sound source localization module performs sound source localization on the multi-channel audio signals to obtain the estimated navigation point of the target time step. This method can locate the sound source in real time using only the sound source signal and convert the sound source signal into a navigation point, effectively eliminating the dependence on static prior maps. Second, a navigation point calculation module performs weighted averaging dynamic processing on the estimated navigation point and the historical navigation points of historical time steps to obtain the target navigation point. This method can smooth the estimated navigation point, reduce the impact of instantaneous noise, and generate a more reliable target navigation point. Finally, the autonomous navigation fusion module performs global and local path fusion planning on the target navigation point and the environmental state of the target mobile robot, and outputs fused control commands. It can generate control commands that adapt to environmental changes in a dynamic environment without a map, and realize real-time planning of feasible, smooth and obstacle-avoiding navigation paths. This effectively improves the robot's autonomous navigation capability in unknown environments. Furthermore, the real-time control and execution module drives the motion actuator to complete navigation control according to the fused control commands. This can ultimately ensure the stable and smooth operation of the mobile robot in complex environments without a map, and achieve adaptive tracking of dynamic sound source targets, significantly improving the tracking effect of the mobile robot. Attached Figure Description

[0052] Figure 1 This is a flowchart of the mobile robot sound source tracking method provided in the embodiments of this application;

[0053] Figure 2 yes Figure 1 The flowchart of step S102 in the document;

[0054] Figure 3 yes Figure 2 The flowchart of step S201 in the text;

[0055] Figure 4 yes Figure 1 The flowchart of step S103 in the process;

[0056] Figure 5 yes Figure 1 The flowchart of step S104 in the process;

[0057] Figure 6 yes Figure 5 The flowchart of step S503 in the process;

[0058] Figure 7 This is another flowchart of the mobile robot sound source tracking method provided in the embodiments of this application;

[0059] Figure 8 This application provides a schematic diagram of a three-dimensional spherical microphone array and a root mean square error curve of the microphone array.

[0060] Figure 9 This is a schematic diagram of the target mobile robot provided in an embodiment of this application;

[0061] Figure 10 This is a schematic diagram of the hardware structure of the electronic device provided in the embodiments of this application. Detailed Implementation

[0062] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0063] It should be noted that although functional modules are divided in the device schematic diagram and a logical order is shown in the flowchart, in some cases, the steps shown or described may be performed in a different order than the module division in the device or the order in the flowchart. The terms "first," "second," etc., in the specification, claims, and the aforementioned drawings are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence.

[0064] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs. The terminology used herein is for the purpose of describing embodiments of this application only and is not intended to limit this application.

[0065] This application provides a method for tracking a sound source in a mobile robot, a target mobile robot, a device, and a medium, aiming to improve the tracking effect of the mobile robot.

[0066] The mobile robot sound source tracking method, target mobile robot, device and medium provided in the embodiments of this application are specifically described through the following embodiments. First, the mobile robot sound source tracking method in the embodiments of this application is described.

[0067] The mobile robot sound source tracking method provided in this application relates to the field of robot intelligent control technology. The mobile robot sound source tracking method provided in this application can be applied to a terminal, a server, or software running on either a terminal or a server. In some embodiments, the terminal can be a smartphone, tablet, laptop, desktop computer, etc.; the server can be configured as an independent physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server providing basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, CDN, and big data and artificial intelligence platforms; the software can be an application implementing the mobile robot sound source tracking method, but is not limited to the above forms.

[0068] This application can be used in a wide variety of general-purpose or special-purpose computer system environments or configurations. Examples include: personal computers, server computers, handheld or portable devices, tablet devices, multiprocessor systems, microprocessor-based systems, set-top boxes, programmable consumer electronics, network PCs, minicomputers, mainframe computers, and distributed computing environments including any of the above systems or devices. This application can be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc., that perform specific tasks or implement specific abstract data types. This application can also be practiced in distributed computing environments where tasks are performed by remote processing devices connected via a communication network. In distributed computing environments, program modules can reside in local and remote computer storage media, including storage devices.

[0069] Figure 1 This is an optional flowchart of a mobile robot sound source tracking method provided in this application embodiment, applied to a target mobile robot. Figure 1 The method may include, but is not limited to, steps S101 to S105.

[0070] Step S101: Signal acquisition is performed using a three-dimensional spherical microphone array module to obtain multi-channel audio signals.

[0071] Step S102: The sound source localization module performs sound source localization on the multi-channel audio signal to obtain the estimated navigation point of the target time step.

[0072] Step S103: The estimated navigation point and the historical navigation points of the historical time step are dynamically processed by the navigation point calculation module to obtain the target navigation point.

[0073] Step S104: The autonomous navigation fusion module performs global and local path fusion planning on the target navigation point and the environmental state of the target mobile robot, and outputs fusion control commands.

[0074] In step S105, the real-time control and execution module drives the motion actuator to complete navigation control according to the fusion control command.

[0075] Steps S101 to S105 as shown in the embodiments of this application firstly acquire signals through a three-dimensional spherical microphone array module to obtain multi-channel audio signals, and then use a sound source localization module to locate the sound source of the multi-channel audio signals to obtain an estimated navigation point for the target time step. This allows for real-time location of the sound source using only the sound source signal and converts the sound source signal into a navigation point, effectively eliminating the dependence on static prior maps. Secondly, the navigation point calculation module performs weighted averaging dynamic processing on the estimated navigation point and the historical navigation points of the historical time steps to obtain the target navigation point. This smooths the estimated navigation point, reduces the impact of instantaneous noise, and generates a more reliable target navigation point. Finally, the autonomous navigation fusion module performs global and local path fusion planning on the target navigation point and the environmental state of the target mobile robot, and outputs fused control commands. It can generate control commands that adapt to environmental changes in a dynamic environment without a map, and realize real-time planning of feasible, smooth and obstacle-avoiding navigation paths. This effectively improves the robot's autonomous navigation capability in unknown environments. Furthermore, the real-time control and execution module drives the motion actuator to complete navigation control according to the fused control commands. This can ultimately ensure the stable and smooth operation of the mobile robot in complex environments without a map, and achieve adaptive tracking of dynamic sound source targets, significantly improving the tracking effect of the mobile robot.

[0076] In step S101 of some embodiments, specifically, the three-dimensional spherical microphone array module is used to acquire signals and obtain multi-channel audio signals.

[0077] Specifically, multi-channel audio signals refer to original sound source signals that are synchronously acquired by a three-dimensional spherical microphone array at the same time step, recorded in the form of independent channels, and carry spatial orientation information.

[0078] For example, in scenarios involving darkroom detection or congested transmission pipeline systems (such as underground natural gas pipelines, gasoline transmission pipelines, and local cable integration systems), multi-channel audio signals can be used to detect sound sources from pipeline leaks, cable leaks, and people affected by disasters.

[0079] Please see Figure 2 In some embodiments, step S102 includes, but is not limited to, steps S201 to S203:

[0080] Step S201: Calculate the delay time phase of the multi-channel audio signal in a predetermined direction to obtain the target time difference phase correlation value of the target time step.

[0081] Step S202: Calculate the beam energy of the multi-channel audio signal in a predetermined direction based on the target time difference phase correlation value to obtain the target beam energy.

[0082] Step S203: Select the azimuth corresponding to the largest beam energy from the target beam energy and determine it as the estimated navigation point for the target time step.

[0083] Please see Figure 3 In some embodiments, step S201 includes, but is not limited to, steps S301 to S303:

[0084] Step S301: Obtain the delay time between the multi-channel audio signals.

[0085] Step S302: Perform signal spectrum processing on the multi-channel audio signal to obtain the target signal spectrum.

[0086] Step S303: Obtain the spectral conjugate complex number of the target signal spectrum and the target exponent term.

[0087] Step S304: Calculate the phase of the delay time based on the delay time, the target signal spectrum, the complex conjugate of the spectrum, and the target exponent term to obtain the target time difference phase correlation value for the target time step.

[0088] In step S301 of some embodiments, specifically, the delay time refers to the time difference between the arrival of the same sound source at different microphones.

[0089] In step S302 of some embodiments, specifically, the target signal spectrum refers to the amplitude and phase information of the audio signal converted from the time domain to the frequency domain.

[0090] Specifically, the Fast Fourier Transform (FFT) can be used to convert the time domain of the multi-channel audio signal at different frequencies into amplitude and phase information in the frequency domain, in order to determine the spectrum of the target signal.

[0091] In step S303 of some embodiments, specifically, the spectral conjugate complex number refers to the conjugate operation on the spectrum of the target signal.

[0092] Specifically, the target exponent term refers to a complex exponent term determined based on the delay time and signal frequency.

[0093] In step S304 of some embodiments, specifically, the target time difference phase correlation value refers to the difference value between the time difference and phase relationship between the multi-channel audio signals, which is used to reflect the signal correlation of the multi-channel audio signals in a predetermined direction.

[0094] Specifically, PHAT (Phase Transform) technology can be used to transform the spectrum of multi-channel audio signals to obtain the spectral representation of the sound signal of each sensor (i.e. each microphone) at different frequencies. For each frequency, PHAT technology can adjust the signal spectrum of each sensor to make phase adjustment over the delay time τ.

[0095] Specifically, the target time difference phase correlation value can be determined using the following formula:

[0096]

[0097] Where PHAT(τ,f) represents the target time difference phase correlation value over the delay time τ, N represents the signal length, and X n (f) represents the target signal spectrum of sensor n at frequency f. X represents n The spectral conjugate complex of (f) is chosen for each frequency f, with the delay time τ having the largest time difference phase correlation value, to represent the most likely arrival time of the sound source at that frequency.

[0098] Specifically, the maximum time difference phase correlation value across all frequencies is considered holistically, and the average of the maximum time difference phase correlation values ​​across all frequencies is taken as the final time difference phase correlation value.

[0099] Through steps S301 to S304, by processing signal time difference, spectrum, and phase, the phase difference that may be introduced when multi-channel audio signals propagate in a three-dimensional spherical microphone array can be effectively eliminated, so as to accurately reflect the direction of the sound source and help improve the accuracy of beamforming in the future.

[0100] In step S202 of some embodiments, specifically, the target beam energy refers to the energy obtained by beamforming the multi-channel audio signal in a predetermined direction, which reflects the intensity of the sound source signal in that direction and is used to determine the location of the sound source.

[0101] Specifically, the spatial location of a sound source can be determined by SRP (Steered Response Power) sound source localization technology, which uses multi-channel audio signals collected by a sensor array (i.e., a three-dimensional spherical microphone array) to calculate the beam energy in different directions.

[0102] Specifically, the target beam energy can be determined using the following formula:

[0103]

[0104] Where SRP(θ) represents the target beam energy in direction θ, i.e., the sum of squares of the signal spectra of all sensors; X m The signal spectrum of sensor m at frequency f is represented by f; M represents the number of sensors.

[0105] In this embodiment, the beam energy of the multi-channel audio signal in a predetermined direction is calculated based on the target time difference phase correlation value, which can effectively enhance the sound source signal energy in the target direction and suppress interference from other directions.

[0106] In step S203 of some embodiments, specifically, the estimated navigation point is usually represented in the form of azimuth and distance, which is used to reflect the position information of the sound source in the robot coordinate system, so as to guide the robot to move towards the sound source.

[0107] For example, in a 360° direction, if the beam energy is greatest in the 120° direction, then the 120° direction is determined as the estimated navigation point for the target time step.

[0108] Through steps S201 to S203, the phase alignment of multi-channel audio signals can be achieved, effectively suppressing multipath effects and noise interference. Even in complex acoustic environments, background noise and other interference signals can be effectively distinguished, thereby improving the accuracy of sound source localization.

[0109] Please see Figure 4 In some embodiments, step S103 includes, but is not limited to, steps S401 to S404:

[0110] Step S401: Construct a sliding time window based on the historical navigation points of the historical time step.

[0111] Step S402: Perform median filtering on the historical navigation points of the historical time step to obtain filtered navigation points.

[0112] Step S403: Calculate the weights of the filtered navigation points within the sliding time window and in the preset direction to obtain the target weights.

[0113] Step S404: Perform a weighted average of the filtered navigation point and the target navigation point within the sliding time window to obtain the target navigation point.

[0114] In step S401 of some embodiments, specifically, the sliding time window refers to a fixed-length time interval used for weighted average dynamic processing of historical navigation points, and the sliding time window slides forward as the current time step is updated, always maintaining a fixed length.

[0115] For example, a sliding time window can contain historical navigation points from the past 10 time steps. If each time step is 0.1 seconds, then the sliding time window covers 10 historical navigation points from the past 1 second.

[0116] In step S402 of some embodiments, specifically, the filtered navigation point refers to the orientation data after outlier removal by median filtering.

[0117] Specifically, the historical navigation points of all historical time steps within the sliding time window are sorted according to their azimuth angles, and the median of the azimuth angles is determined. Azimuth angles exceeding the median are removed to determine the filtered navigation points after removing anomalies.

[0118] In this embodiment, by performing median filtering on the historical navigation points of the historical time steps, it is possible to effectively suppress outlier values ​​of navigation points caused by sudden noise or measurement errors, thereby ensuring the reliability of navigation points.

[0119] In step S403 of some embodiments, specifically, the target weight is a value dynamically allocated within a sliding time window based on the proximity of the filtered navigation point to a preset direction, used to determine the contribution of historical navigation points in the weighted average.

[0120] Specifically, the weighted average dynamic window sound source localization (WADW-SSL) method can be used to perform median filtering on historical navigation points in a predetermined direction at historical time steps, and select a point of a certain length in that direction to establish a dynamic sliding time window at continuous time intervals. Then, a weighted moving average of the azimuth angle is performed within the sliding time window, and the weight value of historical navigation points in later time steps is larger.

[0121] Specifically, the target weight can be determined using the following formula:

[0122]

[0123] Among them, w iλ represents the target weight of the current estimated navigation point i; λ represents the decay factor, which controls the distribution of weights (λ can be 0.8). When λ is large, the WADW-SSL method tends to favor the latest estimate and is suitable for rapidly changing environments. When λ is small, the WADW-SSL method gives higher weight to historical navigation points and is suitable for stable environments; H represents the sliding time window size; j represents the historical navigation point sequence.

[0124] In step S404 of some embodiments, specifically, the target navigation point is the final orientation data obtained by weighting the current navigation point and historical navigation points according to the target weight.

[0125] Specifically, the target navigation point can be determined using the following formula:

[0126]

[0127] in, The target azimuth (i.e., target navigation point) represents the target time step t; w i H represents the target weight currently estimated for navigation point i; H represents the sliding time window size; θ t-H+1+i This represents the estimated azimuth angle of the target time step t within the H sliding time window.

[0128] Through steps S401 to S404, by using a sliding time window, median filtering, weight calculation, and weighted averaging, the target navigation point can be calculated by combining historical navigation points from multiple historical time steps. By using weight calculation, the weighted values ​​of historical navigation points from later time steps are increased to enhance real-time performance. Furthermore, the navigation points can be dynamically smoothed to reduce the impact of instantaneous noise. At the same time, the navigation points in the target direction are highlighted, which helps to enhance the adaptability of the mobile robot in complex dynamic environments and enables it to track the target sound source more smoothly.

[0129] Please see Figure 5 In some embodiments, step S104 includes, but is not limited to, steps S501 to S506:

[0130] Step S501: Perform global path navigation planning on the target navigation point and the environmental state of the target mobile robot to obtain global control commands.

[0131] Step S502: Discretely sample the target navigation points according to the global control command to obtain discrete path points.

[0132] Step S503: Obtain the timestamps corresponding to the discrete points of the path, and construct a preliminary local path navigation function based on the discrete points of the path and the timestamps to obtain the preliminary local path navigation function.

[0133] Step S504: Add velocity and acceleration constraints, obstacle avoidance constraints, and time interval constraints to the initial local path navigation function to obtain the optimized local path navigation function.

[0134] Step S505: The target local path navigation function is obtained by weighted summation of the optimized local path navigation function, velocity, and acceleration.

[0135] Step S506: Solve the path navigation function of the target local path navigation function to obtain the fused control command.

[0136] In step S501 of some embodiments, specifically, the global control command refers to a global path navigation command planned for the target mobile robot from the current position to the target position in an unknown environment. The global control command includes the linear velocity and angular velocity of the global navigation, and the global control command is used to reflect the path navigation of the target mobile robot from the current position to the target position.

[0137] Specifically, the autonomous navigation fusion module integrates deep reinforcement learning and a time-elastic band algorithm to generate path navigation commands based on the target navigation point and the environmental state of the target mobile robot. The deep reinforcement learning algorithm continuously interacts with the environmental state of the target mobile robot to determine global control commands and models the state and motion spaces of the target mobile robot, enabling the learning of global path navigation strategies under different dynamic scenarios.

[0138] Specifically, the TD3 (Twin Delayed Deep Deterministic Policy Gradient) algorithm can be used to model the robot navigation environment process driven by the DRL (Deep Reinforcement Learning) model into the following MDP (Markov Decision Process), i.e., MDP. ours =(S ours A ours ,P ours ,R ours ,γ ours This algorithm employs reinforcement learning to perform global path navigation on the target navigation point and the environmental state of the target mobile robot, ultimately outputting global control commands. Specifically, the TD3 algorithm is used for reinforcement learning in the continuous action space of the target mobile robot; DRL uses a deep neural network to approximate the optimal global path navigation strategy or value function; and MDP is used to model the dynamic system in the decision-making process, where the probability of state transitions depends only on the target mobile robot's current perceived state and executed actions. oursA represents the state space, used to describe the environmental state in which the target mobile robot exists; ours The action space represents the action space used to describe the actions performed by the target mobile robot; P ours R represents the state transition probability. ours Let γ represent the reward function. ours This represents the discount factor.

[0139] Furthermore, the state space can be represented as S ours :S ours ={S laser ,||·||2,θ,a ours}. Among them, S laser This indicates the ranging return values ​​of the lidar in the simulation environment (e.g., returning 20 measurement values). ||·||2 represents the distance from the target navigation point to the mobile robot's reference coordinate system, i.e. θ represents the azimuth angle of the mobile robot at the current time step; a ours This represents the motion space, specifically the linear velocity 'a' around the positive x-axis. ours [0] and the angular velocity a around the positive Z-axis. ours [1], that is

[0140] Furthermore, the state transition probability can be expressed as P(s) ours′ |s ours ,a ours Specifically, this means that if the target mobile robot executes all control commands it receives, i.e., P(s) ours′ ) = 1.

[0141] Furthermore, the reward function can be expressed as R(s) ours ,a ours The specific reward function is designed as follows:

[0142]

[0143] Wherein, R(s) ours ,a ours The reward R at the target time step t depends on three conditions: if the current time step is ||·||², that is, the distance to the target is less than the target threshold δ, then the positive target reward R is applied. g If a collision is detected, a negative collision reward R is applied. c If neither of these two conditions exists, a reward is immediately granted based on the current linear velocity v and angular velocity w to guide the global navigation strategy. Simultaneously, for a given target, a delayed attribute reward method is employed, i.e. Here, n represents the first few steps of updating the reward, meaning that in the last n steps before reaching the goal, the positive goal reward will gradually decrease.

[0144] Furthermore, the discount factor can be γ. ours =0.99.

[0145] In this embodiment, global path navigation planning is performed on the target navigation point and the environmental state of the target mobile robot to obtain global control commands. By combining the sound source location information and the current environmental state, the acoustic heading can be planned as a global navigation waypoint, thereby improving the autonomous navigation effect of the target mobile robot in the absence of a prior map.

[0146] In step S502 of some embodiments, specifically, the path discrete points refer to a series of discrete spatiotemporal points obtained by sampling the planned path trajectory at certain time intervals (e.g., 0.1 seconds) according to global control commands. These path spatiotemporal points can be represented as (x i ,y i ,t i ), where (x i ,y i ) represents the position of the target mobile robot in a two-dimensional plane, t i This indicates the timestamp corresponding to that location.

[0147] Specifically, the Time-Elastic Band (TEB) algorithm can transform the global navigation strategy of global control commands into a local path plan for the mobile robot. For details, please refer to [link to relevant documentation]. Figure 6 In some embodiments, step 503 includes, but is not limited to, steps S601 to S604:

[0148] Step S601: Construct the first function term based on two adjacent path discrete points.

[0149] Step S602: Construct a second function term based on three adjacent path discrete points.

[0150] Step S603: Construct a third function term based on two adjacent timestamps.

[0151] Step S604: Summate the first function term, the second function term, and the third function term to obtain the preliminary local path navigation function.

[0152] In step S601 of some embodiments, specifically, the first function term is used to describe the path trajectory of the target mobile robot between two adjacent path discrete points.

[0153] Specifically, the first function term can be represented by the following formula:

[0154]

[0155] Where J1 represents the first function term, used to describe the smoothness of the path trajectory; p i represents the i-th discrete point on the path; w1 represents the weight of the path trajectory between two adjacent times; t i This represents the timestamp corresponding to the i-th discrete point on the path.

[0156] In this embodiment, a first function term is constructed based on two adjacent path discrete points, which can minimize the displacement change in the trajectory. By minimizing the sum of squared distances between two adjacent path discrete points, unnecessary sharp turns in the path are reduced, thereby ensuring the smoothness of the navigation path.

[0157] In step S602 of some embodiments, specifically, the second function term is used to describe the path trajectory of the target mobile robot between three adjacent path discrete points.

[0158] Specifically, the second function term can be represented by the following formula:

[0159]

[0160] Where J2 represents the second function term, used to describe the smoothness of the path trajectory; p i p represents the i-th discrete point on the path; i+1 p represents the next discrete point on the path adjacent to i; i+2 w1 represents the next discrete point on the path adjacent to i+1; w2 represents the weight of the path trajectory at three adjacent times.

[0161] In this embodiment, a second function term is constructed based on three adjacent discrete points on the path. This can calculate and minimize the curvature between every three adjacent points, ensuring smooth transitions in each part of the path trajectory and avoiding overly sharp turns in the path, which helps improve the stability of the target mobile robot's motion.

[0162] In step S603 of some embodiments, specifically, the third function term is used to control the target mobile robot to reach a predetermined position within a predetermined time.

[0163] Specifically, the third function term can be represented by the following formula:

[0164]

[0165] Where J3 represents the third function term, used to optimize the time allocation of discrete points along the path trajectory; t i+1 t represents the timestamp corresponding to the (i+1)th discrete point on the path; iw3 represents the timestamp corresponding to the i-th discrete point on the path; w3 represents the weight assigned to two adjacent times.

[0166] In this embodiment, a third function term is constructed based on two adjacent timestamps, which can minimize the difference between adjacent timestamps to optimize the time allocation of the path trajectory. This helps the target mobile robot move more evenly, avoids excessively concentrated acceleration or deceleration, and ensures smooth movement.

[0167] In step S604 of some embodiments, specifically, the preliminary local path navigation function is a function that integrates the first function term, the second function term, and the third function term, and is used to describe the local path trajectory navigation and motion time of the target mobile robot.

[0168] Specifically, the preliminary local path navigation function can be represented by the following formula:

[0169] J = J1 + J2 + J3,

[0170] Where J represents the initial local path navigation function, J1 represents the first function term, J2 represents the second function term, and J3 represents the third function term.

[0171] Through steps S601 to S604, the smoothness, fluidity, and travel time of the path trajectory can be considered to effectively avoid overly sharp turns in the planned path, ensure the smoothness and transition of each part of the path trajectory, and make reasonable time allocation, thereby improving the stability of the target mobile robot's movement.

[0172] In step S504 of some embodiments, specifically, the local path navigation function is optimized to describe the path trajectory that comprehensively considers acceleration and acceleration constraints, obstacle avoidance constraints, and time interval constraints.

[0173] Specifically, velocity and acceleration constraints can be represented by the following formulas:

[0174]

[0175] Among them, v i v represents the velocity corresponding to i discrete points along the path; max Indicates the maximum speed constraint, a i Let a represent the acceleration corresponding to the i-th discrete point on the path. max This represents the maximum acceleration constraint, and N represents the number of discrete points on the path.

[0176] Furthermore, obstacle avoidance constraints can be expressed using the following formula:

[0177]

[0178] Wherein d(pi O j ) represents the i-th discrete point p on the path. i With the j-th obstacle O j The distance between them, d safe This represents the safe distance between the mobile robot and the obstacle, and N represents the number of discrete points along the path.

[0179] Furthermore, the time interval constraint can be expressed by the following formula:

[0180]

[0181] Where, Δt i t represents the time interval between the i-th discrete point and the (i+1)-th discrete point on the path. min t represents the minimum time interval. max This represents the maximum time interval, and N represents the number of discrete points along the path.

[0182] In step S505 of some embodiments, specifically, the target local path navigation function is used to describe the final path navigation plan that takes into account the discrete points of adjacent paths, adjacent time allocation, acceleration and acceleration constraints, obstacle avoidance constraints and time interval constraints.

[0183] Specifically, the target local path navigation function can be represented by the following formula:

[0184]

[0185] Among them, J * Let J(x,y,t) represent the target local path navigation function, J(x,y,t) represent the optimized local path navigation function corresponding to the discrete points on the path, λ1 represent the velocity constraint adjustment weight, and v i Let λ represent the velocity corresponding to the i-th discrete point on the path, λ2 represent the acceleration constraint adjustment weight, and a i This represents the acceleration corresponding to the i-th discrete point on the path.

[0186] Specifically, to further improve tracking performance in completely unknown environments, a dynamic smooth fusion model is used. This model, while satisfying kinematic and dynamic constraints, achieves a smooth transition in the target mobile robot's speed control through feedback correction and failure protection mechanisms. The dynamic smooth fusion model primarily corrects constraints in four aspects: velocity, acceleration, vector normalization, and feedback velocity. For details, please refer to [link to relevant documentation]. Figure 8 In some embodiments, the mobile robot sound source tracking method further includes, but is not limited to, steps S701 to S705:

[0187] Step S701: Obtain the maximum linear velocity and maximum angular velocity of the target mobile robot, and perform velocity constraint correction on the target local path navigation function based on the maximum linear velocity and maximum angular velocity to obtain the corrected velocity constraint.

[0188] Step S702: Correct the acceleration constraint of the target local path navigation function according to the corrected velocity constraint to obtain the corrected acceleration constraint.

[0189] Step S703: Based on the modified acceleration constraint, the target local path navigation function is modified with a two-dimensional velocity increment space constraint to obtain the modified velocity vector constraint.

[0190] Step S704: Obtain the initial feedback speed of the target mobile robot. If the initial feedback speed exceeds the preset speed range, perform speed correction on the initial feedback speed to obtain the corrected feedback speed constraint.

[0191] Step S705: The target local path navigation function is modified according to the preset velocity constraints, modified velocity constraints, modified acceleration constraints, modified velocity vector constraints and modified feedback velocity constraints to obtain the modified local trajectory function, and the modified local path navigation function is determined as the target local path navigation function.

[0192] In step S701 of some embodiments, specifically, the maximum linear velocity represents the maximum speed that the target mobile robot can achieve in linear motion.

[0193] Specifically, maximum angular velocity represents the maximum speed that the target mobile robot can achieve during rotational motion.

[0194] Specifically, the modified velocity constraint is used to describe the maximum linear velocity and maximum angular limit of the target mobile robot's movement.

[0195] Specifically, if the target velocity vector is v t =[v t ,ω t ] T The velocity amplitude constraint, i.e., the modified velocity constraint, can be expressed by the following formula:

[0196]

[0197] Among them, v t′ ω represents the corrected linear velocity. t′ Indicates the corrected angular velocity, v t ω represents the linear velocity at the current time step t. t v represents the angular velocity at the current time step t. max ω represents the maximum linear velocity. max -v represents the maximum angular velocity. max-ω represents the minimum linear velocity. max This represents the minimum angular velocity.

[0198] In step S702 of some embodiments, specifically, the acceleration constraint is modified to describe the maximum acceleration limit when the target mobile robot moves.

[0199] Specifically, the acceleration constraint can be modified using the following formula:

[0200] Δv=[Δv,Δω] T =v t′ -v k-1

[0201] Where Δv represents the velocity increment, i.e., the difference between the velocity at the current moment and the velocity at the previous moment; Δv represents the linear velocity increment; Δω represents the angular velocity increment; v t′ Indicates the corrected linear velocity; v k-1 It indicates the velocity at the previous moment.

[0202] In step S703 of some embodiments, specifically, the velocity vector constraint is modified to describe the two-dimensional velocity increment space constraint for the movement of the target mobile robot.

[0203] Specifically, the two-dimensional velocity increment space constraint correction can be achieved by applying vector normalization constraints and component constraints to the target local path navigation function.

[0204] Specifically, vector normalization constraints can be expressed by the following formula:

[0205]

[0206] Where, ||Δv||² represents the Euclidean norm of the velocity increment Δv, i.e., the magnitude of the resultant velocity change from the linear velocity change to the angular velocity change; a lim The acceleration limit vector represents the maximum permissible values ​​of linear and angular acceleration; Δt represents the time interval; θ represents the velocity increment Δv and the acceleration limit vector a. lim The difference in azimuth between them; a ω This represents the limiting value of angular acceleration; a v Δv represents the limit value of linear acceleration; Δv represents the increment of linear velocity; Δω represents the increment of angular velocity.

[0207] Furthermore, component constraints are implemented simultaneously based on the included angle:

[0208]

[0209] Where Δv′ represents the velocity increment after vector normalization and component constraints, i.e., the corrected velocity vector constraint; a ω This represents the limiting value of angular acceleration; av Δv represents the limit value of linear acceleration; Δω represents the increment of linear velocity; Δt represents the increment of angular velocity; and Δt represents the time interval.

[0210] Specifically, in the vector normalization constraint, the L2 norm of the velocity increment vector is first calculated, ensuring it does not exceed the L2 norm of the product of the acceleration limit vector and the time increment. Then, the azimuth difference between the velocity increment vector and the acceleration limit vector is calculated to ensure the direction of velocity change aligns with the direction of acceleration. Simultaneously, in the component constraint, the linear velocity and angular velocity components of the velocity increment are adjusted according to the sign of the azimuth difference to ensure they do not exceed their respective limits.

[0211] In this embodiment, the two-dimensional velocity increment spatial constraint correction of the target local path navigation function based on the modified acceleration constraint can ensure the smoothness of the velocity change of the target mobile robot during the movement process, as well as its safety.

[0212] In step S704 of some embodiments, specifically, the correction feedback speed is used to describe that the speed at which the target mobile robot moves does not exceed a safe range.

[0213] Specifically, the correction of the feedback speed constraint can be expressed by the following formula:

[0214]

[0215] in, v represents the minimum corrected feedback speed. k-1 Let a represent the velocity at the previous moment. dec Δt represents the deceleration limit of the target mobile robot within the time interval. Indicates the maximum corrected feedback speed, a. acc Δt represents the acceleration limit of the target mobile robot within the time interval.

[0216] Furthermore, when the feedback speed v f When the threshold range is exceeded, the following correction is performed:

[0217]

[0218] Among them, v k Indicates the corrected feedback speed, v f Indicates feedback speed. Indicates the minimum feedback linear velocity. ω represents the maximum feedback linear velocity. f Indicates the feedback angular velocity. Indicates the minimum angular velocity of the feedback. This indicates the maximum angular velocity of the feedback.

[0219] Furthermore, to prevent anomalies caused by the failure of control commands to the target moving robot, the following input timeliness judgment conditions are established:

[0220] When t current -t last >t inactive At that time, perform a zero-speed reset:

[0221] Among them, t inactive This indicates the timeliness judgment condition for the input. Represents the time threshold, t current t represents the current time step. last Indicates the last update time step, v t′ Indicates the correction speed.

[0222] In step S705 of some embodiments, specifically, the modified local trajectory function is used to describe the modified local path trajectory after comprehensively considering the velocity constraint, the modified velocity constraint, the modified acceleration constraint, the modified velocity vector constraint, and the modified feedback velocity constraint, and the modified local path navigation function is determined as the final target local path navigation function.

[0223] Specifically, the motion continuity of the target mobile robot described by the velocity constraint is guaranteed by the following two conditions:

[0224] (1) Speed ​​smoothness satisfies the condition

[0225]

[0226] Among them, a acc a represents the acceleration limit of the target mobile robot. ang This represents the limit value of the angular acceleration of the target mobile robot.

[0227] Specifically, a stable speed ensures that the acceleration of the target mobile robot is within a reasonable range, avoiding sudden speed changes, so as to guarantee the feasibility and controllability of the path trajectory.

[0228] (2) Dynamic security meets the conditions

[0229]

[0230] Among them, v f Indicates feedback speed, v k Indicates the corrected feedback speed, a dec The value of deceleration is represented by Δt, where Δt represents the time interval, and a represents the maximum deceleration of the target mobile robot. ang This represents the limit value of the angular acceleration of the target mobile robot.

[0231] Specifically, dynamic safety meets the requirements to ensure that the target mobile robot can stably adjust its speed within a limited time during deceleration or turning, thereby ensuring the safety of obstacle avoidance and path tracking.

[0232] Furthermore, the stability of the target mobile robot control can also be guaranteed by the following convergence conditions:

[0233]

[0234] Where v represents the target velocity, v t This represents the velocity at the target time step t.

[0235] Specifically, the convergence condition can guarantee that the target mobile robot will converge to zero if and only if there is no speed command input during long-term operation, and stable control can also be achieved at other time steps.

[0236] Through steps S701 to S705, by correcting the constraints on velocity and acceleration, an optimized trajectory function that meets the physical limitations of the target mobile robot and satisfies the requirements for robot motion stability can be obtained, effectively improving the robot's navigation accuracy and efficiency.

[0237] In step S506 of some embodiments, specifically, the fusion control instruction refers to the final path navigation instruction that comprehensively considers global path planning and local path planning. The fusion control instruction includes the linear velocity and angular velocity that are fused from global and local path navigation.

[0238] Specifically, a quadratic trajectory planning problem for the target local path navigation function can be constructed using the Sequential Quadratic Programming (SQP) method, and the quadratic programming subproblem can be solved iteratively to determine the final path navigation instructions.

[0239] Furthermore, a quadratic trajectory planning problem is constructed for the target local path navigation function, resulting in a quadratic trajectory planning problem. This includes: linearizing the target local path navigation function to obtain a linear local path navigation function; obtaining linear approximation results of preset equality constraints and preset inequality constraints; constructing the target Hessian matrix based on the target local path navigation function; and constructing a quadratic trajectory planning problem based on the target Hessian matrix, the linear local trajectory function, and the linear approximation results.

[0240] Specifically, the first-order Taylor expansion of the target local path navigation function, i.e., the linear local path navigation function, can be determined by the following expression:

[0241]

[0242] Where J(x,y,t) represents the optimized local path navigation function corresponding to the discrete points of the path; J(x k ) represents the discrete point x in the current iteration path. k The objective function value at x; k =(x k ,y k ,t k ) represents the discrete point of the current iteration path, including (x k ,y k The position of the target mobile robot in a two-dimensional plane, t k This represents the timestamp corresponding to that location; x represents the variable vector.

[0243] Furthermore, regarding the equality constraint g... i (x,y,t)=0 and inequality constraint h j The linear approximations of (x,y,t)≤0 are as follows:

[0244]

[0245] Among them, g i (x) represents the equality constraint of the variable vector, g i (x k ) represents the equality constraint of the discrete points of the current iteration path, x k =(x k ,y k ,t k ) represents the discrete point of the current iteration path, x represents the variable vector, and h j (x) represents an inequality constraint representing a vector of variables, h j (x k ) represents the inequality constraints of the discrete points of the current iteration path.

[0246] Specifically, the Hessian matrix can be determined using the following expression:

[0247]

[0248] In the formula, H represents the target Hessian matrix. J represents the second-order partial derivative of the variable vector x. * This represents the target local path navigation function. H represents the Hessian matrix of the target local path navigation function. v =2λ1I represents the Hessian matrix of the velocity constraint, λ1 represents the velocity constraint adjustment weight, 2λ1I is the Dr. velocity regularization term, and H a =2λ2I represents the Hessian matrix of the acceleration constraint, 2λ2I represents the acceleration regularization term, λ2 represents the acceleration constraint adjustment weight, and I represents the identity matrix.

[0249] Furthermore, based on the linearization process described above, the quadratic programming subproblem can be determined using the following formula:

[0250]

[0251] Where p represents the parameter update direction, used to update the discrete points of the current iteration path; H k J(x) represents an approximation of the Hessian matrix; k ) represents the discrete point x in the current iteration path. k The objective function value at that point.

[0252] Specifically, after linearization, equality constraints and inequality constraints are transformed into:

[0253]

[0254] Among them, g i (x k ) represents the equality constraint of the discrete points of the current iteration path, p represents the parameter update direction, and -g i (x k ) represents the negative value of the equality constraint at the discrete point of the current iteration path, h j (x k ) represents the inequality constraints on the discrete points of the current iteration path, -h j (x k ) represents the negative value of the inequality constraint at the discrete point of the current iteration path.

[0255] Furthermore, the update direction p can be obtained by iteratively solving the quadratic programming subproblem, with the specific formula as follows:

[0256] x k+1 =x k +αp

[0257] Where, x k+1 Let x represent the discrete point of the next iteration path, α represent the iteration step size adjustment factor, and x represent the discrete point of the next iteration path. k denoted by , where represents the discrete point of the current iteration path, and p represents the parameter update direction.

[0258] Furthermore, parameter updates can be implemented using the modified Newton's method as shown below:

[0259]

[0260] Where, x k+1 Let x represent the discrete point of the next iteration path. k Let H represent the discrete points of the current iteration path, and let J represent the target Hessian matrix. * This represents the target local path navigation function.

[0261] Specifically, the convergence of the function solution is determined based on three criteria: the change in the target local path navigation function ΔJ. * <ε1, constraint violation degree The parameter update direction ||p|| < ε3 is also considered. When all three criteria are met, the optimization process is considered converged to determine the navigation path. Here, ε1 represents the convergence threshold for the change in the target local path navigation function; ε2 represents the convergence threshold for the constraint violation degree; and ε3 represents the convergence threshold for the parameter update direction.

[0262] In this embodiment, the path navigation solution is obtained by using the target local path navigation function to obtain fused control commands. This can transform path planning into a nonlinear optimization problem with spatiotemporal constraints. By adjusting the spatiotemporal distribution of local trajectory points through nonlinear optimization, the smoothness, dynamic feasibility, and obstacle avoidance capability of the planned path are synergistically optimized, thereby generating feasible, smooth, and obstacle-avoiding local paths in real time, further improving the robot's autonomous navigation performance.

[0263] Through steps S501 to S506, global navigation planning can be performed by combining sound source localization and the robot's current environment. This eliminates the need for prior maps and provides a global navigation planning strategy for the target mobile robot, enabling preliminary autonomous navigation. Furthermore, through discrete sampling and the construction of a preliminary local path navigation function, precise path navigation planning can be performed for the robot's local movement areas. Adding constraints on speed, acceleration, obstacle avoidance, and time intervals further optimizes the path trajectory, ensuring the smoothness and dynamic feasibility of the robot's motion. By solving the target local path navigation function, feasible, smooth, and obstacle-avoiding local navigation paths can be generated in real-time in dynamic environments, further improving the robot's autonomous navigation capability. This effectively solves the problems of unreasonable navigation path planning and the inability to effectively achieve dynamic obstacle avoidance in existing technologies, thereby improving the subsequent tracking effect of the mobile robot.

[0264] In step S105 of some embodiments, specifically, the real-time control and execution module includes a path tracking controller and a chassis actuator.

[0265] Specifically, the pose error of the target mobile robot can be calculated based on the difference between the planned navigation path and the path trajectory points. Then, by using the final linear velocity and angular velocity control commands of the target mobile robot, a proportional-integral-derivative control strategy is adopted to correct the control output of the target mobile robot to ensure the smoothness of the motion response. Finally, the fused control commands are sent to the motor drive system of the chassis actuator, and the target mobile robot is driven to perform navigation and tracking through the path tracking controller.

[0266] In this embodiment, the real-time control and execution module drives the motion execution mechanism to complete navigation control according to the fused control instructions. This ensures the stable and smooth operation of the mobile robot in complex environments without a map, and enables adaptive tracking of dynamic sound source targets, significantly improving the tracking effect of the mobile robot.

[0267] The application embodiment first acquires signals using a three-dimensional spherical microphone array module to obtain multi-channel audio signals. Then, it uses a sound source localization module to locate the sound source of the multi-channel audio signals and obtain an estimated navigation point for the target time step. This allows for real-time location of the sound source using only the sound source signal and converts the sound source signal into a navigation point, effectively eliminating the dependence on static prior maps. Secondly, the navigation point calculation module performs a weighted average dynamic processing on the estimated navigation point and the historical navigation points of the historical time steps to obtain the target navigation point. This smooths the estimated navigation point and reduces the impact of instantaneous noise, thereby generating a more reliable target navigation point. Finally, the autonomous navigation fusion module performs global and local path fusion planning on the target navigation point and the environmental state of the target mobile robot, and outputs fused control commands. It can generate control commands that adapt to environmental changes in a dynamic environment without a map, and realize real-time planning of feasible, smooth and obstacle-avoiding navigation paths. This effectively improves the robot's autonomous navigation capability in unknown environments. Furthermore, the real-time control and execution module drives the motion actuator to complete navigation control according to the fused control commands. This can ultimately ensure the stable and smooth operation of the mobile robot in complex environments without a map, and achieve adaptive tracking of dynamic sound source targets, significantly improving the tracking effect of the mobile robot.

[0268] This application also provides a target mobile robot that can implement the above-described mobile robot sound source tracking method, including:

[0269] A three-dimensional spherical microphone array module is used for signal acquisition to obtain multi-channel audio signals;

[0270] The sound source localization module is used to locate the sound source of multi-channel audio signals and obtain the estimated navigation point of the target time step;

[0271] The navigation point calculation module is used to perform a weighted average dynamic processing on the estimated navigation point and the historical navigation points of the historical time step to obtain the target navigation point;

[0272] The autonomous navigation fusion module is used to perform global and local path fusion planning on the target navigation point and the environmental state of the target mobile robot, and output fusion control commands;

[0273] The real-time control and execution module is used to drive the motion actuator to complete navigation control according to the fusion control commands.

[0274] The specific implementation method of the target mobile robot is basically the same as the specific implementation method of the mobile robot sound source tracking method described above, and will not be repeated here.

[0275] See Figure 8 As shown, a three-dimensional spherical microphone array can be placed at the center of a circle with a radius of 2.0m, establishing a coordinate system as shown in the schematic diagram of the three-dimensional spherical microphone array. The X-axis of this coordinate system points from 0° to 360°, and the azimuth angle is the angle between the sound source position and the positive X-axis direction clockwise. Thus, the range from 0° to 360° is divided into 12 equal angles, which are 0 / 360°, 30°, 60°, 90°, 120°, 150°, 180°, 210°, 240°, 270°, 300°, and 330° clockwise from the positive X-axis direction of the robot coordinate system. During the experiment, microphone arrays mic_1, mic_2, mic_3, mic_4, mic_5, and mic_6 were used at different positions to form microphone arrays Array_1 (diameter 14.4cm), Array_2 (diameter 24.9cm), and Array_3 (top diameter 14.1cm, bottom diameter 24.9cm, height 11.2cm). The microphones in Array_1 and Array_2 were all located on a single plane, while all microphones in Array_3 were located on a sphere. A total of 36 measurements were taken using these microphone arrays, collecting 100 azimuth angle data points each time, resulting in 3600 usable data points.

[0276] Furthermore, referring to the RMSE (Root Mean Square Error) curves of all microphone arrays, we can see that the experimental results of the signal acquisition effects of the three different microphone arrays are as follows: the error of the 3D spherical microphone array Array_3 is relatively small, with an average RMSE of 34.0; the error of the 3D spherical microphone array Array_2 is the largest, with an average RMSE of 43.9; and the average RMSE of the 3D spherical microphone array Array_1 is 37.4. Therefore, the signal acquisition effect of the 3D spherical microphone array Array_3 is the best. By acquiring signals through the 3D spherical microphone array Array_3, multi-channel audio signals containing azimuth information can be collected, thereby capturing the location of the sound source in real time and providing reliable spatial cues for the robot's mapless navigation.

[0277] Please see Figure 9Base_Link represents the robot chassis, used to control the robot's movement. The coordinate system built based on the robot chassis is the global coordinate system. All other local coordinate systems in the diagram are transformed to the global coordinate system of Base_Link by rotation matrices. This global coordinate system serves as the reference for the other local coordinate systems to ensure the consistency of the robot control parameters represented by the other local coordinate systems and the global coordinate system. Mic represents a three-dimensional spherical microphone array, used for acquiring sound source signals, and a Mic local coordinate system is built based on Mic. A two-dimensional LiDAR Laser is used for obstacle avoidance, and the signals acquired by the Laser are used as input to the DRL-TEB fusion model. A Laser local coordinate system is built based on the Laser. An IMU (Inertial Measurement Unit) is used to acquire the pose of the target mobile robot, and an IMU local coordinate system is built based on the IMU. Among them, the X-axis is the forward direction axis, used to represent the direction of movement of the target mobile robot; the Y-axis is the lateral axis, used to represent the displacement or rotation of the target mobile robot in the left and right directions; and the Z-axis is the vertical axis, used to represent the pitch angle, roll angle, or vertical displacement of the target mobile robot.

[0278] This application also provides an electronic device, which includes a memory and a processor. The memory stores a computer program, and the processor executes the computer program to implement the aforementioned mobile robot sound source tracking method. This electronic device can be any smart terminal, including tablet computers, in-vehicle computers, etc.

[0279] Please see Figure 10 , Figure 10 The hardware structure of an electronic device according to another embodiment is illustrated. The electronic device includes:

[0280] The processor 1001 can be implemented using a general-purpose CPU (Central Processing Unit), microprocessor, application-specific integrated circuit (ASIC), or one or more integrated circuits, and is used to execute relevant programs to implement the technical solutions provided in the embodiments of this application.

[0281] The memory 1002 can be implemented as a read-only memory (ROM), static storage device, dynamic storage device, or random access memory (RAM). The memory 1002 can store the processing system and other application programs. When the technical solutions provided in the embodiments of this specification are implemented through software or firmware, the relevant program code is stored in the memory 1002 and is called and executed by the processor 1001 to execute the mobile robot sound source tracking method of the embodiments of this application.

[0282] Input / output interface 1003 is used to implement information input and output;

[0283] The communication interface 1004 is used to enable communication and interaction between this device and other devices. Communication can be achieved through wired means (such as USB, network cable, etc.) or wireless means (such as mobile network, WIFI, Bluetooth, etc.).

[0284] Bus 1005 transmits information between various components of the device (e.g., processor 1001, memory 1002, input / output interface 1003, and communication interface 1004);

[0285] The processor 1001, memory 1002, input / output interface 1003 and communication interface 1004 are connected to each other within the device via bus 1005.

[0286] This application embodiment also provides a storage medium storing a computer program that, when executed by a processor, implements the above-described mobile robot sound source tracking method.

[0287] Memory, as a non-transitory storage medium, can be used to store non-transitory software programs and non-transitory computer-executable programs. Furthermore, memory may include high-speed random access memory, and may also include non-transitory memory, such as at least one disk storage device, flash memory device, or other non-transitory solid-state storage device. In some embodiments, memory may optionally include memory remotely located relative to the processor, and these remote memories can be connected to the processor via a network. Examples of such networks include, but are not limited to, the Internet, intranets, local area networks, mobile communication networks, and combinations thereof.

[0288] The embodiments described in this application are for the purpose of more clearly illustrating the technical solutions of the embodiments of this application, and do not constitute a limitation on the technical solutions provided by the embodiments of this application. As those skilled in the art will know, with the evolution of technology and the emergence of new application scenarios, the technical solutions provided by the embodiments of this application are also applicable to similar technical problems.

[0289] Those skilled in the art will understand that the technical solutions shown in the figures do not constitute a limitation on the embodiments of this application, and may include more or fewer steps than shown, or combine certain steps, or different steps.

[0290] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs.

[0291] Those skilled in the art will understand that all or some of the steps in the methods disclosed above, as well as the functional modules / units in the systems and devices, can be implemented as software, firmware, hardware, or suitable combinations thereof.

[0292] The terms "first," "second," "third," "fourth," etc. (if present) in the specification and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data used are interchangeable where appropriate so that the embodiments of this application described herein can describe an implementation sequence other than those illustrated. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0293] It should be understood that in this application, "at least one (item)" means one or more, and "more than" means two or more. "And / or" is used to describe the relationship between related objects, indicating that three relationships can exist. For example, "A and / or B" can represent three cases: only A exists, only B exists, and both A and B exist simultaneously, where A and B can be singular or plural. The character " / " generally indicates that the preceding and following related objects are in an "or" relationship. "At least one (item) of the following" or similar expressions refer to any combination of these items, including any combination of single or plural items. For example, at least one (item) of a, b, or c can represent: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, and c can be single or multiple.

[0294] In the several embodiments provided in this application, it should be understood that the disclosed apparatus and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of the units described above is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. The coupling or direct coupling or communication connection between the shown or discussed units may be through some interfaces, or indirect coupling or communication connection between the apparatus or units, and may be electrical, mechanical, or other forms.

[0295] The units described above as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0296] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0297] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes multiple instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of the various embodiments of this application. The aforementioned storage medium includes various media capable of storing programs, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0298] The preferred embodiments of the present application have been described above with reference to the accompanying drawings, but this does not limit the scope of the claims of the present application. Any modifications, equivalent substitutions, and improvements made by those skilled in the art without departing from the scope and substance of the embodiments of the present application shall be within the scope of the claims of the present application.

Claims

1. A method for tracking sound sources in a mobile robot, characterized in that, Applied to a target mobile robot, the method includes: Signal acquisition is performed using a three-dimensional spherical microphone array module to obtain multi-channel audio signals; The sound source localization module is used to locate the sound source of the multi-channel audio signal to obtain the estimated navigation point of the target time step; The target navigation point is obtained by dynamically averaging the estimated navigation point and the historical navigation points of the historical time step through the navigation point calculation module. The autonomous navigation fusion module performs global and local path fusion planning on the target navigation point and the environmental state of the target mobile robot, and outputs fusion control commands. The real-time control and execution module drives the motion actuator to complete navigation control according to the fused control commands.

2. The method according to claim 1, characterized in that, The step of dynamically processing the estimated navigation point and the historical navigation points of historical time steps through the navigation point calculation module to obtain the target navigation point includes: Construct a sliding time window based on the historical navigation points of the historical time steps; The historical navigation points at the historical time steps are subjected to median filtering to obtain filtered navigation points; Within the sliding time window and in the preset direction, the weights of the filtered navigation points are calculated to obtain the target weights. Within the sliding time window, a weighted average is performed on the filtered navigation point and the target navigation point to obtain the target navigation point.

3. The method according to claim 1, characterized in that, The autonomous navigation fusion module performs global and local path fusion planning on the target navigation point and the environmental state of the target mobile robot, and outputs fusion control commands, including: Global path navigation planning is performed on the target navigation point and the environmental state of the target mobile robot to obtain global control commands; The target navigation point is discretely sampled according to the global control command to obtain discrete path points; Obtain the timestamps corresponding to the discrete points of the path, and construct a preliminary local path navigation function based on the discrete points of the path and the timestamps to obtain the preliminary local path navigation function; By adding velocity and acceleration constraints, obstacle avoidance constraints, and time interval constraints to the initial local path navigation function, an optimized local path navigation function is obtained. The target local path navigation function is obtained by weighted summation of the optimized local path navigation function, velocity, and acceleration. The target local path navigation function is solved to obtain the fused control command.

4. The method according to claim 3, characterized in that, The preliminary local path navigation function is constructed based on the path discrete points and the timestamp, resulting in the preliminary local path navigation function, including: The first function term is constructed based on two adjacent discrete points of the path; Construct a second function term based on three adjacent discrete points along the path; Construct a third function term based on two adjacent timestamps; The first function term, the second function term, and the third function term are summed to obtain the preliminary local path navigation function.

5. The method according to claim 3, characterized in that, After obtaining the target local path navigation function by weighted summation of the optimized local path navigation function, velocity, and acceleration, the method further includes: The maximum linear velocity and maximum angular velocity of the target mobile robot are obtained, and the target local path navigation function is modified according to the maximum linear velocity and the maximum angular velocity to obtain the modified velocity constraint. The target local path navigation function is modified with acceleration constraints according to the modified velocity constraints to obtain the modified acceleration constraints. Based on the modified acceleration constraint, the target local path navigation function is modified by a two-dimensional velocity increment space constraint to obtain the modified velocity vector constraint. The initial feedback speed of the target mobile robot is obtained. If the initial feedback speed exceeds the preset speed range, the initial feedback speed is corrected to obtain the corrected feedback speed constraint. The target local path navigation function is modified according to the preset speed constraints, the modified speed constraints, the modified acceleration constraints, the modified speed vector constraints, and the modified feedback speed constraints to obtain the modified local path navigation function, and the modified local path navigation function is determined as the target local path navigation function.

6. The method according to claim 1, characterized in that, The step of locating the sound source of the multi-channel audio signal using the sound source localization module to obtain the estimated navigation point for the target time step includes: The time difference phase of the multi-channel audio signal is calculated in a predetermined direction to obtain the target time difference phase correlation value of the target time step; The beam energy of the multi-channel audio signal in a predetermined direction is calculated based on the target time difference phase correlation value to obtain the target beam energy. The location corresponding to the largest beam energy from the target beam energy is selected as the estimated navigation point for the target time step.

7. The method according to claim 6, characterized in that, The step of calculating the time difference phase of the multi-channel audio signal in a predetermined direction to obtain the target time difference phase correlation value for the target time step includes: Obtain the delay time between the multi-channel audio signals; The multi-channel audio signal is subjected to signal spectrum processing to obtain the target signal spectrum; Obtain the spectral conjugate complex number of the target signal spectrum, and the target exponent term; The target time difference phase correlation value of the target time step is obtained by calculating the delay time phase based on the delay time, the target signal spectrum, the spectrum conjugate complex number, and the target exponent term.

8. A target mobile robot, characterized in that, include: A three-dimensional spherical microphone array module is used for signal acquisition to obtain multi-channel audio signals; The sound source localization module is used to locate the sound source of the multi-channel audio signal and obtain the estimated navigation point of the target time step; The navigation point calculation module is used to perform a weighted average dynamic processing on the estimated navigation point and the historical navigation points of the historical time step to obtain the target navigation point; The autonomous navigation fusion module is used to perform global and local path fusion planning on the target navigation point and the environmental state of the target mobile robot, and output fusion control commands. The real-time control and execution module is used to drive the motion actuator to complete navigation control according to the fused control instructions.

9. An electronic device, characterized in that, The electronic device includes a memory and a processor, the memory storing a computer program, and the processor executing the computer program to implement the mobile robot sound source tracking method according to any one of claims 1 to 7.

10. A storage medium storing a computer program, characterized in that, When the computer program is executed by the processor, it implements the mobile robot sound source tracking method according to any one of claims 1 to 7.

Images