Robotic dog monitoring system for bird sound source positioning and tracking

Abstract

The invention discloses a robot dog monitoring system for bird sound source positioning and tracking, and the system comprises a sound collection module which is used for collecting a multi-channel sound signal, and carrying out the preprocessing of the multi-channel sound signal; the edge calculation module is used for processing based on the preprocessed multi-channel sound signals to obtain a sound source positioning result; the robot dog motion control module is used for receiving the sound source positioning result, performing path planning in combination with a control algorithm, driving a robot dog to autonomously move to approach a sound source, and integrating a sound source array sensor to realize environment perception and navigation; the image recognition module is used for constructing a bird recognition model and deploying the bird recognition model in an image recognition sensor to obtain a bird recognition result; and the remote monitoring module is used for receiving and displaying the sound source positioning result, the motion trail of the robot dog and the bird recognition result in real time. According to the invention, an efficient and intelligent solution is provided for bird monitoring.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of bird ecology and the field of conservation biology technology, and in particular to a robot dog monitoring system for bird sound source positioning and tracking. BACKGROUND

[0002] With the increasingly serious problem of global biodiversity loss, the number of global wildlife has decreased by 73% in the past half century. This biodiversity crisis not only poses a serious threat to the balance of the ecosystem, but also directly affects human survival and long-term development.

[0003] At present, the global natural ecosystem is deteriorating, which seriously threatens the normal operation of the ecosystem, so wildlife protection has become a major issue that needs to be addressed worldwide. China attaches great importance to the protection of the ecosystem. The State Forestry and Grassland Administration and the Ministry of Agriculture and Rural Affairs organize the study of the standards and scope of important wildlife habitats, and draft control measures. The government continues to gradually increase funding to support the monitoring of various species and their habitats. Therefore, accurate assessment of the number of wildlife populations and their trends is crucial for developing reasonable and effective protection strategies. As an important indicator species of the ecosystem, changes in the number, distribution, and behavior of birds can directly reflect the health of the ecological environment. Through monitoring and research on birds, we can understand the dynamic changes of the ecosystem in a timely manner and provide scientific basis for ecological protection and environmental management.

[0004] Traditional bird monitoring methods, such as direct counting, line transect method, and point method, have obtained some information about birds to some extent, but they are time-consuming and labor-intensive, have low efficiency, and are difficult to monitor continuously. In addition, it is difficult to implement in remote or inaccessible areas. With the progress of technology, various intelligent devices have been gradually applied in the field of bird monitoring, including fixed devices such as infrared cameras and mobile monitoring tools such as drones. However, these devices still have obvious limitations in practical application: infrared cameras are limited by fixed viewing angles and cannot move autonomously, making it difficult to effectively track birds with a wide range of activities and difficult to meet the needs of comprehensive investigation; voice recorders have limited positioning capabilities, are limited by the environment, and cannot move; drones are widely used in large bird surveys, but have many shortcomings in monitoring small birds (such as songbirds), such as noise interference, bird disturbance, and difficulty in capturing and accurately identifying bird characteristics. Most importantly, for endangered birds that build nests on the ground, existing investigation methods and devices often fail to find their traces in the wild environment.

[0005] With the continuous development of computer technology and artificial intelligence, the application scenarios of robot dogs have become increasingly widespread. Their high passability and intelligence have been fully demonstrated in many fields: In the industrial field, robot dogs equipped with multiple sensors can accurately identify safety hazards and are widely used in scenarios such as power inspection and oil pipeline inspection; In fire rescue, they can transmit high-definition images in real time, detect combustible and toxic gases, track heat sources, and adapt to complex on-site environments with their flexible mobility.

[0006] Against this backdrop, there is an urgent need for a robotic dog monitoring system for bird sound source localization and tracking, which can make up for the shortcomings of existing technologies in an intelligent, high-throughput, low-interference, and real-time monitoring manner, thereby achieving effective monitoring of endangered birds with strong elusiveness. Summary of the Invention

[0007] To address the technical problems existing in the prior art, this invention proposes a robotic dog monitoring system for bird sound source localization and tracking. It utilizes a high-resolution camera and sound source array sensor to collect bird image and sound data in real time, and performs localized processing through embedded devices to achieve automatic identification and information transmission of target bird species. This solves the problems of poor real-time transmission and significant interference to birds in the prior art, providing an efficient and intelligent solution for bird monitoring.

[0008] On the one hand, to achieve the above objectives, the present invention provides a robotic dog monitoring system for bird sound source localization and tracking, comprising:

[0009] Sound acquisition module: used to acquire multi-channel sound signals and preprocess the multi-channel sound signals;

[0010] Edge computing module: used to process pre-processed multi-channel audio signals to obtain sound source localization results;

[0011] Robot dog motion control module: used to receive the sound source localization results, combine with the control algorithm to perform path planning, drive the robot dog to move autonomously closer to the sound source, and integrate a sound source array sensor to realize environmental perception and navigation;

[0012] Image recognition module: used to build a bird recognition model and deploy the bird recognition model in an image recognition sensor to obtain bird recognition results;

[0013] Remote monitoring module: used to receive and display sound source localization results, robot dog movement trajectory and bird recognition results in real time.

[0014] Preferably, the sound acquisition module synchronously acquires multi-channel sound signals through a microphone array, and performs filtering, noise reduction, gain compensation, and synchronization calibration to obtain the pre-processed multi-channel sound signals.

[0015] Preferably, the edge computing module includes a sound source localization unit, which is used to enhance the target signal by combining the optimal beamforming algorithm and to calculate the sound wave propagation time difference of each receiving channel of the microphone array using the generalized cross-correlation (GCC) method to obtain the sound source localization result.

[0016] Preferably, the time difference of sound wave propagation in each receiving channel of the microphone array is calculated as follows:

[0017] r1(t)=s(t)+n1(t)(1);

[0018] r2(t)=s(tD)+n2(t)(2);

[0019] In the formula, s(t) is the sound signal, n1(t) and n2(t) are the detection noise of the two sound sensors, D is the arrival time difference TDOA of the sound signal on the two microphone sensors, r1(t) is the signal detected by the first sound sensor, and r2(t) is the signal detected by the second sound sensor.

[0020] Preferably, obtaining the sound source localization result includes:

[0021] Let one microphone in the microphone array be the origin R0(0,0), and the coordinates of the other microphones be R... i (x i ,y i (i = 1, 2, ..., n-1), the sound source is located at S(a, b), and the speed of sound is v. sound The sound source signal reaches R0 and R i The time difference is Δt i ;

[0022] The sound source signal reaches R0 and R i The distance difference simplifies to |R0P|, where P is R i The projection onto the direction of the sound source, according to the sound velocity formula:

[0023] |R0R i |cosθ=|R0P|=v sound Δt i (3);

[0024] In the formula, θ is the sound source relative to R0R. i Angle of direction;

[0025] Vectorize and represent formula (3) using coordinates:

[0026] Using unit vectors to represent the direction of the sound source We can obtain:

[0027]

[0028] The direction of the sound source is determined by combining information from several microphones using the least squares method. Matrix M is set as the coefficient matrix composed of microphone coordinates. Let be the vector product of time difference and speed of sound, then:

[0029]

[0030] In the formula, The coordinate components representing the direction of the sound source are ultimately obtained using the arctangent function.

[0031] θ=arctan 2(x[1],x[0])(6).

[0032] Preferably, the robot dog motion control module adopts a real-time control framework based on C++, achieves communication through ROS, combines control algorithms to realize path planning, and relies on the Ubuntu system for embedded development. At the same time, it integrates sound source array sensor data to realize environmental perception and navigation.

[0033] Preferably, the image recognition module includes a model building unit, which is used to build a bird recognition model based on an existing bird image database and using the Keras-YOLOv3 algorithm; wherein, before using Keras-YOLOv3 for model training, the bird images are labeled using the labelImg tool to identify the species of birds in the images.

[0034] Preferably, in the training stage of the bird recognition model, Darknet-53 is used as the backbone network for feature extraction. The Darknet-53 network consists of 53 layers of convolution and residual blocks. Feature fusion adopts the multi-scale feature pyramid (FPN) strategy, which achieves the fusion of features at different scales through upsampling and feature concatenation.

[0035] After training, the network is input with the image to be detected to obtain multi-scale prediction results. Then, non-maximum suppression (NMS) is used to filter low-confidence prediction boxes and remove redundant detection results. Finally, the bird detection bounding boxes, their categories, and confidence information are output to complete the target detection and obtain the trained bird recognition model.

[0036] Preferably, the remote monitoring module transmits the images recognized by the FPGA development board to the computer through a video acquisition device and a wireless transmitter. The computer monitoring interface is built based on HTML+JavaScript+CSS and displays the sound source localization results, the robot dog's movement trajectory, and the bird recognition results in real time.

[0037] On the other hand, to achieve the above objectives, the present invention also provides a method for a robot dog monitoring system for bird sound source localization and tracking, comprising:

[0038] The sound acquisition module collects bird sound signals, calculates the sound source localization results, and transmits them to the robot dog.

[0039] Drive the robot dog to move to the sound source area along the direction of the sound source localization result;

[0040] Collect bird images and identify bird species using bird recognition models;

[0041] The location of the sound source, bird identification results, and movement trajectory are transmitted to the remote monitoring platform in real time.

[0042] Compared with the prior art, the present invention has the following advantages and technical effects:

[0043] This invention provides a robotic dog monitoring system for bird sound source localization and tracking. Based on a sound source array sensor, it calculates the spatial location of the bird's sound source by measuring the time difference or phase difference of sound signals received by multiple microphones. During the robotic dog's movement, the direction of movement can be adjusted in real time based on the sound sources received by the sound source array sensor. A bird recognition model is constructed based on existing bird image databases. Bird image and sound data are collected in real time using a high-resolution camera and sound source array sensor, and processed locally through an embedded device. This enables automatic identification and information transmission of target bird species, effectively solving problems such as the difficulty in detecting birds in hidden nests on the ground, poor real-time transmission in existing technologies, and significant interference with birds. This provides an efficient and intelligent solution for bird monitoring. Attached Figure Description

[0044] The accompanying drawings, which form part of this application, are used to provide a further understanding of this application. The illustrative embodiments and descriptions of this application are used to explain this application and do not constitute an undue limitation of this application. In the drawings:

[0045] Figure 1 This is a schematic diagram illustrating the workflow of a robotic dog monitoring system for bird sound source localization and tracking, according to an embodiment of the present invention. Detailed Implementation

[0046] It should be noted that, unless otherwise specified, the embodiments and features described in this application can be combined with each other. This application will now be described in detail with reference to the accompanying drawings and embodiments.

[0047] It should be noted that the steps shown in the flowchart in the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions, and although a logical order is shown in the flowchart, in some cases the steps shown or described may be executed in a different order than that shown here.

[0048] This embodiment proposes a robotic dog monitoring system for bird sound source localization and tracking, such as... Figure 1 ,include:

[0049] Sound acquisition module: used to acquire multi-channel sound signals and preprocess the multi-channel sound signals;

[0050] Edge computing module: used to process pre-processed multi-channel audio signals to obtain sound source localization results;

[0051] Robot dog motion control module: used to receive the sound source localization results, combine with the control algorithm to perform path planning, drive the robot dog to move autonomously closer to the sound source, and integrate a sound source array sensor to realize environmental perception and navigation;

[0052] Image recognition module: used to build a bird recognition model and deploy the bird recognition model in an image recognition sensor to obtain bird recognition results;

[0053] Remote monitoring module: used to receive and display sound source localization results, robot dog movement trajectory and bird recognition results in real time.

[0054] Specifically, this embodiment uses a robotic dog as a carrier and, based on a sound source array sensor, calculates the azimuth of the bird's sound source by measuring the time difference or phase difference of sound signals received by multiple microphones in the array. A deep learning-based bird recognition model is constructed based on an existing bird image database. This system utilizes a high-resolution camera and sound source array sensor to acquire bird image and sound data in real time, and performs localized processing through an embedded device to achieve automatic identification and information transmission of the target bird species.

[0055] Furthermore, the sound acquisition module synchronously acquires multi-channel sound signals through a microphone array, and performs filtering, noise reduction, gain compensation, and synchronization calibration to obtain the pre-processed multi-channel sound signals.

[0056] Specifically, multi-channel audio signals are acquired synchronously through a microphone array, and signal quality is optimized and the impact of noise interference and hardware differences is reduced through preprocessing steps such as filtering and noise reduction, gain compensation and synchronization calibration.

[0057] Signal enhancement: Combining the optimal beamforming algorithm, the target signal is enhanced, effectively suppressing background noise and acoustic reverberation, and optimizing the characteristics of the sound source signal.

[0058] Furthermore, the edge computing module includes a sound source localization unit, which is used to enhance the target signal by combining the optimal beamforming algorithm and to calculate the sound wave propagation time difference of each receiving channel of the microphone array using the generalized cross-correlation (GCC) method to obtain the sound source localization result.

[0059] Specifically, in the edge computing module, the generalized cross-correlation (GCC) algorithm is used to calculate the sound wave propagation time difference of each receiving channel of the microphone array, and then the time delay estimation method is used to derive the sound source location information.

[0060] The specific process is as follows:

[0061] First, a microphone array is used to determine the location of the sound source, using the following formula:

[0062] r1(t)=s(t)+n1(t)(1);

[0063] r2(t)=s(tD)+n2(t)(2);

[0064] In the formula, r1(t) is the signal detected by the first sound sensor, r2(t) is the signal detected by the second sound sensor, s(t) is the sound signal, n1(t) and n2(t) are the noise detected by the two sound sensors, and D is the time difference of arrival (TDOA) of the sound signal on the two microphone sensors, or time delay, which represents the delay of the signal reaching the other sensor with one sensor as a reference.

[0065] Let one microphone in the microphone array be the origin R0(0,0), and the coordinates of the other microphones be R... i (x i ,y i (i = 1, 2, ..., n-1), the sound source is located at S(a, b), and the speed of sound is v. sound The sound source signal reaches R0 and R i The time difference is Δt i .

[0066] Since the far field is approximately a plane wave, the sound source to R0 and R i The distance difference can be simplified to |R0P| (P is R i (Projection in the direction of the sound source), according to the sound velocity formula:

[0067] |R0R i |cosθ=|R0P|=v sound Δt i (3);

[0068] In the formula, θ is the sound source relative to R0R. i Angle of direction;

[0069] Vectorization and coordinate representation of equation (3): Equation (3) is converted into coordinate form, and the direction of the sound source is represented by a unit vector. We can obtain:

[0070]

[0071] Using the least squares method to solve for the direction: Due to the presence of noise and errors in actual measurements, the least squares method is usually used to combine information from multiple microphones to solve for the direction of the sound source.

[0072] Let matrix M be the coefficient matrix consisting of microphone coordinates. Let be the vector product of time difference and speed of sound, then:

[0073]

[0074] In the formula, The coordinate components representing the direction of the sound source are ultimately obtained using the arctangent function.

[0075] θ=arctan 2(x[1],x[0])(6).

[0076] Furthermore, the robot dog motion control module adopts a real-time control framework based on C++, achieves communication through ROS, combines control algorithms to realize path planning, and relies on the Ubuntu system for embedded development. At the same time, it integrates sound source array sensor data to realize environmental perception and navigation.

[0077] Specifically, the robot dog's motion control employs a C++-based real-time control framework, communicates via ROS, utilizes control algorithms for path planning, and is developed using an embedded system based on Ubuntu. It also integrates sensor data, such as a sound source array, to achieve environmental perception and navigation. The sound source localization result (azimuth angle) is transmitted to the robot dog via an edge computing module, driving it to autonomously move closer to the sound source. Simultaneously, the azimuth angle is transmitted in real-time to a remote monitoring platform, enabling real-time monitoring of the system status.

[0078] Furthermore, the image recognition module includes a model building unit, which is used to build a bird recognition model based on an existing bird image database and using the Keras-YOLOv3 algorithm. Before training the model using Keras-YOLOv3, the bird images are labeled using the labelImg tool to identify the species of birds in the images.

[0079] Specifically, in this embodiment, the AXU2CGB-E FPGA development board is used as the carrier to conduct research on bird classification and recognition algorithms. Based on the existing bird image database, the Keras-YOLOv3 algorithm is adopted, enabling the YOLOv3 bird detection algorithm to be quickly implemented under the Keras and TensorFlow framework.

[0080] Before training the model using Keras-YOLOv3, the labelImg tool was used to label the bird images and identify the bird species. During model training, Darknet-53 was used as the backbone network for feature extraction. This network consists of 53 convolutional layers and residual blocks, and the residual connections effectively alleviated the gradient vanishing problem in deep networks. Feature fusion employed a multi-scale feature pyramid (FPN) strategy, which, through upsampling and feature concatenation, achieved the fusion of features at different scales, thereby improving the model's ability to detect birds of different sizes.

[0081] In terms of model parameter optimization, the model parameters are continuously adjusted by optimizing the loss function, which includes coordinate error, confidence error, and classification error, so that the prediction results match the labeled data as closely as possible. After training, the network is input with the image to be detected to obtain multi-scale prediction results. Then, non-maximum suppression (NMS) is used to filter low-confidence prediction boxes and remove redundant detection results, finally outputting reliable bird detection bounding boxes and their category and confidence information, thus completing the target detection task.

[0082] Furthermore, the remote monitoring module transmits the images recognized by the FPGA development board to the computer via video acquisition equipment and wireless transmitter. The computer monitoring interface is built based on HTML+JavaScript+CSS and displays the sound source localization results, robot dog movement trajectory, and bird recognition results in real time.

[0083] This embodiment also provides a method for the robot dog monitoring system for bird sound source localization and tracking, including:

[0084] The sound acquisition module collects bird sound signals, calculates the sound source localization results, and transmits them to the robot dog.

[0085] Drive the robot dog to move to the sound source area along the direction of the sound source localization result;

[0086] Collect bird images and identify bird species using bird recognition models;

[0087] The location of the sound source, bird identification results, and movement trajectory are transmitted to the remote monitoring platform in real time.

[0088] The above are merely preferred embodiments of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. A robotic dog monitoring system for bird sound source localization and tracking, characterized in that, include: Sound acquisition module: used to acquire multi-channel sound signals and preprocess the multi-channel sound signals; Edge computing module: used to process pre-processed multi-channel audio signals to obtain sound source localization results; Robot dog motion control module: used to receive the sound source localization results, combine with the control algorithm to perform path planning, drive the robot dog to move autonomously closer to the sound source, and integrate a sound source array sensor to realize environmental perception and navigation; Image recognition module: used to build a bird recognition model and deploy the bird recognition model in an image recognition sensor to obtain bird recognition results; Remote monitoring module: used to receive and display sound source localization results, robot dog movement trajectory and bird recognition results in real time.

2. The robot dog monitoring system for bird sound source localization and tracking according to claim 1, characterized in that, The sound acquisition module synchronously acquires multi-channel sound signals through a microphone array, and performs filtering, noise reduction, gain compensation, and synchronization calibration to obtain the pre-processed multi-channel sound signals.

3. The robot dog monitoring system for bird sound source localization and tracking according to claim 2, characterized in that, The edge computing module includes a sound source localization unit, which is used to enhance the target signal by combining the optimal beamforming algorithm and to calculate the sound wave propagation time difference of each receiving channel of the microphone array using the generalized cross-correlation (GCC) method to obtain the sound source localization result.

4. The robot dog monitoring system for bird sound source localization and tracking according to claim 3, characterized in that, The time difference of sound wave propagation in each receiving channel of the microphone array is calculated as follows: r1(t)=s(t)+n1(t)(1); r2(t)=s(tD)+n2(t)(2); In the formula, s(t) is the sound signal, n1(t) and n2(t) are the detection noise of the two sound sensors, D is the arrival time difference TDOA of the sound signal on the two microphone sensors, r1(t) is the signal detected by the first sound sensor, and r2(t) is the signal detected by the second sound sensor.

5. The robot dog monitoring system for bird sound source localization and tracking according to claim 4, characterized in that, Obtaining the sound source localization result includes: Let one microphone in the microphone array be the origin R0(0,0), and the coordinates of the other microphones be R... i (x i ,y i (i = 1, 2, ..., n-1), the sound source is located at S(a, b), and the speed of sound is v. sound The sound source signal reaches R0 and R i The time difference is Δt i ; The sound source signal reaches R0 and R i The distance difference simplifies to |R0P|, where P is R i The projection onto the direction of the sound source, according to the sound velocity formula: |R0R i |cosθ=|R0P|=v sound Δt i (3); In the formula, θ is the sound source relative to R0R. i Angle of direction; Vectorize and represent formula (3) using coordinates: Using unit vectors to represent the direction of the sound source We can obtain: The direction of the sound source is determined by combining information from several microphones using the least squares method. Matrix M is set as the coefficient matrix composed of microphone coordinates. Let be the vector product of time difference and speed of sound, then: In the formula, The coordinate components representing the direction of the sound source are ultimately obtained using the arctangent function. θ=arctan 2(x[1],x[0])(6).

6. The robot dog monitoring system for bird sound source localization and tracking according to claim 1, characterized in that, The robot dog motion control module adopts a real-time control framework based on C++, communicates through ROS, implements path planning by combining control algorithms, and is developed using the Ubuntu system. It also integrates sound source array sensor data to achieve environmental perception and navigation.

7. The robot dog monitoring system for bird sound source localization and tracking according to claim 1, characterized in that, The image recognition module includes a model building unit, which is used to build a bird recognition model based on an existing bird image database and using the Keras-YOLOv3 algorithm. Before training the model using Keras-YOLOv3, the bird images are labeled using the labelImg tool to identify the species of birds in the images.

8. The robot dog monitoring system for bird sound source localization and tracking according to claim 7, characterized in that, During the training phase of the bird recognition model, Darknet-53 was used as the backbone network for feature extraction. The Darknet-53 network consists of 53 layers of convolution and residual blocks. Feature fusion adopts the multi-scale feature pyramid (FPN) strategy, which achieves the fusion of features at different scales through upsampling and feature concatenation. After training, the network is input with the image to be detected to obtain multi-scale prediction results. Then, non-maximum suppression (NMS) is used to filter low-confidence prediction boxes and remove redundant detection results. Finally, the bird detection bounding boxes, their categories, and confidence information are output to complete the target detection and obtain the trained bird recognition model.

9. The robot dog monitoring system for bird sound source localization and tracking according to claim 1, characterized in that, The remote monitoring module transmits the images recognized by the FPGA development board to the computer through video acquisition equipment and wireless transmitter. The computer monitoring interface is built based on HTML+JavaScript+CSS and displays the sound source localization results, robot dog movement trajectory and bird recognition results in real time.

10. A method for implementing the robot dog monitoring system for bird sound source localization and tracking as described in any one of claims 1-9, characterized in that, include: The sound acquisition module collects bird sound signals, calculates the sound source localization results, and transmits them to the robot dog. Drive the robot dog to move to the sound source area along the direction of the sound source localization result; Collect bird images and identify bird species using bird recognition models; The location of the sound source, bird identification results, and movement trajectory are transmitted to the remote monitoring platform in real time.

Images