Voiceprint array and visible light image fusion registration calibration method and system and inspection robot

Abstract

The invention provides a registration calibration method and system for fusion of a voiceprint array and a visible light image and an inspection robot, and the method comprises the steps: obtaining a visible light image of a calibration board, and collecting a voiceprint array signal from the calibration board; performing sound wave preprocessing on the voiceprint array signal to obtain each sound source point, and performing image preprocessing on the visible light image to obtain image feature points; target image feature points associated with all the sound source points are positioned from all the image feature points through a machine learning algorithm, so that a plurality of target matching pairs are obtained, and each target matching pair comprises the coordinate of one sound source point and the coordinate of one target image feature point; and constructing a registration model by using a perspective projection transformation model, and determining parameters of the registration model by using the target matching pair, thereby obtaining a target registration model for fusing the voiceprint array signal to the visible light image.

Description

Technical Field

[0001] The present invention relates to the technical field of multimodal data processing, and in particular to a registration and calibration method, system and inspection robot for fusing a voiceprint array with a visible light image. Background Art

[0002] With the development of multi-sensor fusion technology, the combination of voiceprint array and visible light camera can significantly improve the environmental perception ability of robot system. However, due to the difference in physical properties between acoustic and optical signals, traditional registration methods often face accuracy and real-time problems when processing such multimodal data. Therefore, developing a novel and efficient fusion registration and calibration method has important scientific research and application value. Summary of the invention

[0003] The present invention aims to solve one of the technical problems in the related art at least to a certain extent.

[0004] To this end, the first purpose of the present invention is to propose a registration and calibration method for the fusion of voiceprint array and visible light image, so as to achieve efficient and accurate fusion of voiceprint data and visible light image data under complex environmental conditions.

[0005] The second objective of the present invention is to provide a registration and calibration system for the fusion of voiceprint array and visible light image.

[0006] The third objective of this application is to provide an inspection robot.

[0007] The fourth objective of the present application is to provide an electronic device.

[0008] A fifth objective of the present application is to provide a computer-readable storage medium.

[0009] To achieve the above-mentioned purpose, the first aspect of the present invention proposes a registration and calibration method for the fusion of a voiceprint array and a visible light image, wherein a calibration plate is provided and a plurality of sound generators are arranged on the calibration plate. The method comprises the following steps:

[0010] Acquiring a visible light image of the calibration plate, and collecting a voiceprint array signal from the calibration plate;

[0011] Performing acoustic wave preprocessing on the voiceprint array signal to obtain various sound source points, and performing image preprocessing on the visible light image to obtain image feature points;

[0012] Using a machine learning algorithm to locate target image feature points associated with each sound source point from all image feature points to obtain a plurality of target matching pairs, each target matching pair including the coordinates of a sound source point and the coordinates of a target image feature point;

[0013] A registration model is constructed using a perspective projection transformation model, and the parameters of the registration model are determined using the target matching pair, so as to obtain a target registration model that fuses the voiceprint array signal into the visible light image.

[0014] In the method of the first aspect of the present invention, the method of locating target image feature points associated with each sound source point from all image feature points using a machine learning algorithm to obtain a plurality of target matching pairs includes: using geometric constraints to screen out initial image feature points associated with each sound source point from all image feature points to obtain an initial matching pair, wherein the initial matching pair includes the coordinates of a sound source point and the coordinates of an initial image feature point; obtaining a trained machine learning algorithm, wherein the input of the machine learning algorithm is the coordinates of an experimental sound source point, and the output is the coordinates of an image feature point associated with the experimental sound source point; inputting the coordinates of each sound source point into the trained machine learning algorithm to obtain predicted coordinates of the image feature points associated with each sound source point; and adjusting the coordinates of the initial image feature points in the initial matching pair based on the predicted coordinates to obtain target image feature points associated with each sound source point, thereby obtaining a plurality of target matching pairs.

[0015] In the method of the first aspect of the present invention, after obtaining the initial matching pairs, the initial matching pairs need to be screened based on similarity before using the trained machine learning algorithm.

[0016] In the method of the first aspect of the present invention, after determining the parameters of the registration model using the target matching pair, it also includes optimizing the parameters of the registration model using a nonlinear optimization algorithm.

[0017] In the method of the first aspect of the present invention, the sound wave preprocessing of the voiceprint array signal to obtain each sound source point includes: converting the voiceprint array signal in the time domain into voiceprint frequency domain characteristics; and using beamforming technology based on the voiceprint frequency domain characteristics to obtain the coordinates of each sound source point.

[0018] In the method of the first aspect of the present invention, the image preprocessing of the visible light image to obtain image feature points includes: after performing histogram equalization on the visible light image, using an edge detection algorithm and a feature extraction algorithm to extract characteristic points of the equalized visible light image, thereby obtaining image feature points.

[0019] To achieve the above-mentioned purpose, the second aspect of the present invention proposes a registration and calibration system for the fusion of a voiceprint array and a visible light image, wherein a calibration plate is provided and a plurality of sound generators are arranged on the calibration plate, and the system comprises:

[0020] An acquisition module, used to acquire a visible light image of the calibration plate and to acquire a voiceprint array signal from the calibration plate;

[0021] A preprocessing module, used to perform acoustic wave preprocessing on the voiceprint array signal to obtain various sound source points, and perform image preprocessing on the visible light image to obtain image feature points;

[0022] A matching module, used to locate target image feature points associated with each sound source point from all image feature points using a machine learning algorithm to obtain a plurality of target matching pairs, each target matching pair including the coordinates of a sound source point and the coordinates of a target image feature point;

[0023] The registration and calibration module is used to construct a registration model using a perspective projection transformation model, and to determine the parameters of the registration model using the target matching pair, so as to obtain a target registration model that fuses the voiceprint array signal into the visible light image.

[0024] To achieve the above-mentioned object, the third aspect of the present invention provides an inspection robot, comprising:

[0025] A vehicle body, a visible light image acquisition device arranged on the vehicle body, a voiceprint array sensor, and a processing unit;

[0026] The visible light image acquisition device is used to acquire a target visible light image at a target position, and the voiceprint array sensor is used to acquire a target sound wave array signal at a target position;

[0027] The processing unit includes a target registration model obtained by the method proposed in the first aspect of the present invention, and the processing unit is used to obtain a target visible light image and a target acoustic wave array signal, and input the target acoustic wave array signal into the target registration model to fuse the target voiceprint array signal into the target visible light image.

[0028] To achieve the above-mentioned purpose, the fourth aspect of the present invention proposes an electronic device, comprising: a processor, and a memory communicatively connected to the processor; the memory stores computer-executable instructions; the processor executes the computer-executable instructions stored in the memory to implement the method proposed in the first aspect of the present invention.

[0029] To achieve the above-mentioned purpose, the fifth aspect of the present invention proposes a computer-readable storage medium, in which computer-executable instructions are stored. When the computer-executable instructions are executed by a processor, they are used to implement the method proposed in the first aspect of the present invention.

[0030] In the registration and calibration method, system, electronic device and storage medium for the fusion of voiceprint array and visible light image provided by the present invention, a calibration plate is set and a plurality of sound generators are arranged on the calibration plate, a visible light image of the calibration plate is obtained, and the voiceprint array signal from the calibration plate is collected; the voiceprint array signal is subjected to acoustic wave preprocessing to obtain each sound source point, and the visible light image is subjected to image preprocessing to obtain image feature points; a machine learning algorithm is used to locate the target image feature points associated with each sound source point from all the image feature points to obtain a plurality of target matching pairs, each target matching pair including the coordinates of a sound source point and the coordinates of a target image feature point; a perspective projection transformation model is used to construct a registration model, and the parameters of the registration model are determined using the target matching pairs, so as to obtain a target registration model for fusing the voiceprint array signal to the visible light image. In this case, with respect to the calibration plate and the multiple sound emitters arranged on the calibration plate, visible light images and voiceprint array signals are collected, and the sound source points and image feature points are obtained using the voiceprint array signals and the visible light images. A plurality of target matching pairs are obtained by integrating the machine learning algorithm, thereby making the matching pairs more accurate. Then, the target matching pairs are used to determine the parameters of the constructed registration model, thereby making the parameters of the model more accurate. The registration model with determined parameters can project the voiceprint array signals into the visible light image, thereby achieving efficient and accurate fusion of voiceprint data and visible light image data under complex environmental conditions.

[0031] Additional aspects and advantages of the present invention will be given in part in the following description and in part will be obvious from the following description, or will be learned through practice of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0032] The above and / or additional aspects and advantages of the present invention will become apparent and easily understood from the following description of the embodiments in conjunction with the accompanying drawings, in which:

[0033] Figure 1 A schematic flow chart of a registration and calibration method for fusion of a voiceprint array and a visible light image provided by an embodiment of the present invention;

[0034] Figure 2 A block diagram of a registration and calibration system for fusing a voiceprint array with a visible light image provided by an embodiment of the present invention;

[0035] Figure 3 A schematic diagram of an inspection robot provided by an embodiment of the present invention;

[0036] Figure 4 A schematic diagram of the planar structure of the inspection robot provided by an embodiment of the present invention;

[0037] Figure 5 A schematic diagram of a three-dimensional structure of an inspection robot provided by an embodiment of the present invention;

[0038] Figure 6 A schematic diagram of the three-dimensional structure of the inspection robot provided by an embodiment of the present invention from another angle;

[0039] Figure 7 A schematic diagram of the structure of a shielding device of an inspection robot provided in an embodiment of the present invention;

[0040] Figure 8 This is a schematic diagram of the structure of the anti-collision bar of the inspection robot provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0041] Embodiments of the present invention are described in detail below, examples of which are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements having the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are intended to be used to explain the present invention, and should not be construed as limiting the present invention.

[0042] The following describes the registration and calibration method and system for the fusion of a voiceprint array and a visible light image according to an embodiment of the present invention with reference to the accompanying drawings.

[0043] Based on this, an embodiment of the present invention provides a registration and calibration method for the fusion of voiceprint array and visible light image, so as to realize efficient and accurate fusion of voiceprint data and visible light image data under complex environmental conditions.

[0044] In the present invention, a calibration board needs to be pre-set and multiple sound generators are arranged on the calibration board. Each sound generator is a sound source. Then, a visible light image acquisition device (such as a camera) and an acoustic wave sensor (such as a soundprint array sensor) are used to acquire visible light images and soundprint array signals to perform the registration and calibration method and system of the present invention.

[0045] Figure 1 A schematic flow chart of a registration and calibration method for the fusion of a voiceprint array and a visible light image provided in an embodiment of the present invention.

[0046] like Figure 1 As shown, the registration and calibration method for the fusion of the voiceprint array and the visible light image includes the following steps:

[0047] Step S101, obtaining a visible light image of a calibration plate, and collecting a voiceprint array signal from the calibration plate.

[0048] In step S101, a visible light image at the calibration plate may be captured by a visible light image acquisition device (eg, a visible light camera). The visible light camera includes parameters such as resolution, focal length, and field of view.

[0049] In step S101, a voiceprint array sensor may be used to collect a voiceprint array signal from a calibration board. The voiceprint array signal includes the sound source information corresponding to each sound generator arranged on the calibration board. The voiceprint array sensor needs to consider the array shape (such as linear or annular), sensor spacing, frequency response characteristics, etc.

[0050] In step S101, synchronous data collection is performed using the voiceprint array sensor and the visible light camera. When the voiceprint array sensor and the visible light camera are installed, it is necessary to ensure optimal coverage of the calibration plate.

[0051] In step S101, when collecting data, it is necessary to ensure that the data between the voiceprint array sensor and the visible light camera are aligned through precise timestamps.

[0052] Step S102, performing sound wave preprocessing on the voiceprint array signal to obtain each sound source point, and performing image preprocessing on the visible light image to obtain image feature points.

[0053] In step S102, the voiceprint array signal is subjected to acoustic wave preprocessing to obtain each sound source point, including: converting the voiceprint array signal in the time domain into voiceprint frequency domain features; and obtaining the coordinates of each sound source point using beamforming technology based on the voiceprint frequency domain features.

[0054] Specifically, the sound wave preprocessing includes: denoising the voiceprint array signal, wherein the denoising process can use an adaptive filter to eliminate the environmental noise in the voiceprint array signal; using the fast Fourier transform (FFT) to convert the time domain signal into frequency domain features of the denoised voiceprint array signal to obtain the voiceprint frequency domain features; applying beamforming technology (such as adaptive beamformer MVDR or fixed beamformer Delay-and-Sum Beamforming) based on the voiceprint frequency domain features to estimate the sound source point to obtain the coordinates of each sound source point. The sound source point includes information such as the direction of the sound source and its three-dimensional spatial position. Among them, filter parameters such as filter order, cutoff frequency, etc. need to be preset during denoising.

[0055] In step S102, image preprocessing is performed on the visible light image to obtain image feature points, including: after performing histogram equalization on the visible light image, characteristic points are extracted from the equalized visible light image using an edge detection algorithm and a feature extraction algorithm, thereby obtaining image feature points.

[0056] Specifically, image preprocessing includes: performing histogram equalization on the visible light image to enhance the image contrast; then using edge detection algorithms (such as the Canny operator) and feature extraction algorithms (such as scale-invariant feature transform SIFT or accelerated robust feature SURF) to extract image feature points from the visible light image. When using the edge detection algorithm, it is necessary to preset specific edge detection operator parameters, such as the high and low thresholds in the Canny algorithm.

[0057] Step S103, using a machine learning algorithm to locate target image feature points associated with each sound source point from all image feature points to obtain multiple target matching pairs, each target matching pair including the coordinates of a sound source point and the coordinates of a target image feature point.

[0058] In step S103, a machine learning algorithm is used to locate target image feature points associated with each sound source point from all image feature points to obtain a plurality of target matching pairs, including: using geometric constraints to screen out initial image feature points associated with each sound source point from all image feature points to obtain an initial matching pair, the initial matching pair including the coordinates of a sound source point and the coordinates of an initial image feature point; obtaining a trained machine learning algorithm, the input of the machine learning algorithm being the coordinates of the experimental sound source point, and the output being the coordinates of the image feature points associated with the experimental sound source point; inputting the coordinates of each sound source point into the trained machine learning algorithm to obtain the predicted coordinates of the image feature points associated with each sound source point; and adjusting the coordinates of the initial image feature points in the initial matching pair based on the predicted coordinates to obtain the target image feature points associated with each sound source point, thereby obtaining a plurality of target matching pairs.

[0059] Among them, after obtaining the initial matching pairs, the initial matching pairs need to be screened based on similarity before using the trained machine learning algorithm.

[0060] Specifically, the geometric constraint method is used to filter out the initial image feature points associated with each sound source point from all image feature points, thereby achieving preliminary matching. The geometric constraint method determines which feature points may be associated with the sound source based on the positional relationship of the feature points in space. The preliminary matching obtains multiple initial matching pairs. Each initial matching pair includes the coordinates of a sound source point and the coordinates of an initial image feature point.

[0061] The obtained initial matching pairs can also be used to eliminate abnormal matching pairs by using the RANSAC (Random Sample Consensus) algorithm. The RANSAC algorithm can effectively identify and eliminate erroneous matching points by repeatedly randomly selecting samples and evaluating adaptability, thereby improving the accuracy of registration.

[0062] After removing abnormal matching pairs, the initial matching pairs from which the abnormal matching pairs are removed can also be screened based on similarity. Specifically, for each initial matching pair from which the abnormal matching pairs are removed, the accuracy of the matching needs to be verified by calculating the similarity of each initial matching pair (such as using Euclidean distance or cosine similarity to calculate the similarity of the coordinates of the sound source point in each initial matching pair and the coordinates of the corresponding initial image feature point). Only matching pairs with a similarity higher than a set threshold are considered valid. Thus, valid initial matching pairs are screened out.

[0063] For the initial matching pairs after screening, in order to further improve the accuracy and robustness of the matching, a machine learning algorithm, such as a support vector machine (SVM) or a convolutional neural network (CNN), can also be introduced. These algorithms can automatically adjust the matching strategy by learning the features and matching relationships in the training set. Further enhance the accuracy and robustness of the matching. Specifically, a training set is obtained, the training set includes the known coordinates of the experimental sound source point and the coordinates of the image feature points associated with the experimental sound source point coordinates, the coordinates of the experimental sound source point are used as the input of the machine learning algorithm, and the coordinates of the image feature points associated with the experimental sound source point are used as labels, and the machine learning algorithm is trained so that the machine learning algorithm learns the matching relationship between the coordinates of the sound source point and the coordinates of the image feature points, thereby obtaining a trained machine learning algorithm. The coordinates of each sound source point obtained in step S102 are input into the trained machine learning algorithm to obtain the predicted coordinates of the image feature points associated with each sound source point; based on the predicted coordinates, the coordinates of the initial image feature points in the initial matching pair are adjusted to obtain the target image feature points associated with each sound source point, wherein the adjustment rule is that if the predicted coordinates are consistent with the coordinates of the initial image feature points in the initial matching pair, the coordinates of the initial image feature points are retained, and the coordinates of the initial image feature points are used as the coordinates of the required target image feature points; if they are inconsistent, the coordinates of the initial image feature points are retained when the distance between the coordinates of the initial image feature points and the predicted coordinates is within a set range, and the coordinates of the initial image feature points are used as the coordinates of the required target image feature points; if they are inconsistent and when the distance exceeds the set range, the coordinates of the initial image feature points are replaced with the predicted coordinates. After the adjustment is completed, multiple target matching pairs can be obtained. It should be noted that when training the machine learning algorithm, the image feature points associated with the experimental sound source points can be described by feature descriptors (such as SIFT, SURF, etc.) to achieve more accurate matching.

[0064] Step S104, using the perspective projection transformation model to construct a registration model, and using the target matching pair to determine the parameters of the registration model, thereby obtaining a target registration model that fuses the voiceprint array signal into the visible light image.

[0065] In step S104, it is easy to understand that perspective projection is a commonly used method for mapping three-dimensional points to a two-dimensional image plane. The basic principle is to project points in three-dimensional space to the image plane through the internal and external parameters of the camera (such as focal length, principal point position, rotation and translation). This step uses the perspective projection transformation model to construct a registration model, which is used to realize the projection of the coordinates of the sound source point onto the visible light image, thereby realizing the conversion of the sound source coordinate system to the image coordinate system. Among them, the sound source coordinate system is usually based on the center of the voiceprint array signal or a specific sound source position as the origin. The image coordinate system is based on the viewing angle of the camera, usually two-dimensional, and represents the pixel position in the image.

[0066] In step S104, the registration model is trained using the target matching pair to determine the parameters of the registration model, wherein the input of the registration model is the coordinates of the sound source point, and the output is the coordinates of the target image feature point.

[0067] In step S104, after the parameters of the registration model are determined by using the target matching pair, the parameters of the registration model are optimized by using a nonlinear optimization algorithm. The nonlinear optimization algorithm may be a Levenberg-Marquardt method, so as to minimize the error between the coordinates of the sound source point and the coordinates of the image feature point.

[0068] It is easy to understand that the Levenberg-Marquardt algorithm is a commonly used nonlinear least squares optimization algorithm suitable for solving parameter optimization problems. In the registration process, the goal is to minimize the error between the coordinates of the sound source point and the coordinates of the image feature points. The error E can usually be expressed as:

[0069]

[0070] Among them, P i is the coordinate of the i-th image feature point, f(X i , θ) is the predicted position calculated by the registration model, θ is the parameter of the registration model to be optimized (such as rotation and translation parameters). n is the number of image feature points. i is the coordinate of the i-th sound source point.

[0071] Iterative update process: The Levenberg-Marquardt algorithm is iteratively updated through the following steps:

[0072] Calculate Jacobian Matrix: Calculate the Jacobian matrix of the error function with respect to the parameters to facilitate the evaluation of the error change under the current parameters.

[0073] Update parameters: Calculate the update amount of the parameters based on the current error and Jacobian matrix. The update formula is usually:

[0074] θ new =θ old -(J T J+ρI) -1 J T γ

[0075] Where J is the Jacobian matrix, J T is the transpose of the Jacobian matrix, γ is the current residual, ρ is the damping factor, and I is the identity matrix. old is the parameter of the registration model before updating, θ new are the parameters of the updated registration model.

[0076] Error minimization: By continuously iterating the above steps, until the error converges to a preset threshold or reaches the maximum number of iterations.

[0077] In this embodiment, the target registration model obtained in step S104 can fuse the voiceprint array signal into the visible light image.

[0078] In step S104, after obtaining the target registration model, the target registration model can be calibrated and fine-tuned using the verification matching pair of μ-known coordinates. The verification matching pair of μ-known coordinates includes the coordinates of the sound source point of μ-known coordinates and the coordinates of the corresponding μ-known image feature points. When selecting the verification matching pair, the image feature points are generally selected to be stable in position and easy to identify.

[0079] In step S104, when the target registration model is calibrated and fine-tuned using the verification matching pairs of μ-known coordinates, cross-validation and leave-one-out methods can also be used to verify the performance of the target registration model and analyze the error distribution and accuracy.

[0080] In order to implement the above embodiment, the present invention further proposes a registration and calibration system for fusing a voiceprint array with a visible light image, wherein a calibration plate is provided and a plurality of sound generators are arranged on the calibration plate.

[0081] Figure 2 A block diagram of a registration and calibration system for fusing a voiceprint array with a visible light image provided by an embodiment of the present invention.

[0082] like Figure 2 As shown, the registration and calibration system for the fusion of the voiceprint array and the visible light image includes an acquisition module, a preprocessing module, a matching module and a registration and calibration module, wherein:

[0083] An acquisition module, used to acquire a visible light image of the calibration plate and to acquire a voiceprint array signal from the calibration plate;

[0084] A preprocessing module is used to perform acoustic wave preprocessing on the voiceprint array signal to obtain each sound source point, and to perform image preprocessing on the visible light image to obtain image feature points;

[0085] A matching module, used to locate target image feature points associated with each sound source point from all image feature points using a machine learning algorithm to obtain a plurality of target matching pairs, each target matching pair including the coordinates of a sound source point and the coordinates of a target image feature point;

[0086] The registration and calibration module is used to construct a registration model using a perspective projection transformation model, and to determine the parameters of the registration model using a target matching pair, thereby obtaining a target registration model that fuses the voiceprint array signal into a visible light image.

[0087] Furthermore, in a possible implementation of an embodiment of the present invention, in a preprocessing module, acoustic wave preprocessing is performed on the voiceprint array signal to obtain each sound source point, including: converting the voiceprint array signal in the time domain into voiceprint frequency domain characteristics; and using beamforming technology based on the voiceprint frequency domain characteristics to obtain the coordinates of each sound source point.

[0088] Furthermore, in a possible implementation of an embodiment of the present invention, in a preprocessing module, image preprocessing is performed on the visible light image to obtain image feature points, including: after histogram equalization of the visible light image, characteristic points of the equalized visible light image are extracted using an edge detection algorithm and a feature extraction algorithm, thereby obtaining image feature points.

[0089] Furthermore, in a possible implementation manner of an embodiment of the present invention, the matching module is specifically used to: use geometric constraints to filter out initial image feature points associated with each sound source point from all image feature points to obtain an initial matching pair, the initial matching pair including the coordinates of a sound source point and the coordinates of an initial image feature point; obtain a trained machine learning algorithm, the input of the machine learning algorithm is the coordinates of the experimental sound source point, and the output is the coordinates of the image feature points associated with the experimental sound source point; input the coordinates of each sound source point into the trained machine learning algorithm to obtain the predicted coordinates of the image feature points associated with each sound source point; adjust the coordinates of the initial image feature points in the initial matching pair based on the predicted coordinates to obtain target image feature points associated with each sound source point, and thereby obtain multiple target matching pairs.

[0090] Furthermore, in a possible implementation of the embodiment of the present invention, in the matching module, after obtaining the initial matching pairs, the initial matching pairs need to be screened based on similarity before using the trained machine learning algorithm.

[0091] Furthermore, in a possible implementation manner of the embodiment of the present invention, in the registration calibration module, after determining the parameters of the registration model by using the target matching pair, the parameters of the registration model are optimized by using a nonlinear optimization algorithm.

[0092] It should be noted that the above explanation of the embodiment of the registration and calibration method for the fusion of voiceprint array and visible light image is also applicable to the registration and calibration system for the fusion of voiceprint array and visible light image of this embodiment, which will not be repeated here.

[0093] In an embodiment of the present invention, a calibration board is provided and a plurality of sound generators are arranged on the calibration board. A visible light image of the calibration board is obtained, and a voiceprint array signal from the calibration board is collected. The voiceprint array signal is subjected to acoustic wave preprocessing to obtain each sound source point, and the visible light image is subjected to image preprocessing to obtain image feature points. A machine learning algorithm is used to locate target image feature points associated with each sound source point from all image feature points to obtain a plurality of target matching pairs, each target matching pair including the coordinates of a sound source point and the coordinates of a target image feature point. A perspective projection transformation model is used to construct a registration model, and the parameters of the registration model are determined using the target matching pairs, thereby obtaining a target registration model that fuses the voiceprint array signal into the visible light image. In this case, with respect to the calibration plate and the multiple sound emitters arranged on the calibration plate, visible light images and voiceprint array signals are collected, and the sound source points and image feature points are obtained using the voiceprint array signals and the visible light images. A plurality of target matching pairs are obtained by integrating the machine learning algorithm, thereby making the matching pairs more accurate. Then, the target matching pairs are used to determine the parameters of the constructed registration model, thereby making the parameters of the model more accurate. The registration model with determined parameters can project the voiceprint array signals into the visible light image, thereby achieving efficient and accurate fusion of voiceprint data and visible light image data under complex environmental conditions.

[0094] The method and system of the present invention are related to a registration and calibration method and system for integrating acoustic data with visible light image data. The method and system improve the registration accuracy of voiceprint data (i.e., voiceprint array signal) and image data (i.e., visible light image), and can be quickly applied in real-time systems. It is suitable for applications in a variety of complex environments, such as noisy environments, scenes with changing lighting, etc. It can be applied to applications such as robot navigation, augmented reality, and autonomous driving systems. It aims to achieve efficient and accurate registration and calibration of voiceprint data and visible light image data under complex environmental conditions, improve the spatial perception accuracy and processing efficiency of multimodal systems, and has good scalability and adaptability.

[0095] To this end, the present invention also proposes an inspection robot, comprising: a vehicle body, a visible light image acquisition device arranged on the vehicle body, a voiceprint array sensor, and a processing unit. Figure 3 Schematic diagram of the inspection robot provided by the embodiment of the present invention. Figure 3As shown, a visible light image acquisition device (such as a PTZ camera) and a voiceprint array sensor are arranged on the vehicle body (also called the robot body). The relative position of the visible light image acquisition device and the voiceprint array sensor is consistent with the relative position of the visible light image acquisition device and the voiceprint array sensor in the above-mentioned method embodiment and system embodiment. Among them, the visible light image acquisition device is used to acquire a target visible light image of the target position, and the voiceprint array sensor is used to acquire a target acoustic wave array signal of the target position. When installing the PTZ camera and the voiceprint array sensor, it is necessary to ensure the best coverage in a specific scene (such as indoors or outdoors). The processing unit includes a target registration model obtained by the method provided in the above-mentioned embodiment, and the processing unit is used to obtain a target visible light image and a target acoustic wave array signal, and input the target acoustic wave array signal into the target registration model to fuse the target voiceprint array signal into the target visible light image.

[0096] Figure 4 This is a schematic diagram of the planar structure of the inspection robot provided by an embodiment of the present invention. Figure 5 A schematic diagram of the three-dimensional structure of the inspection robot provided in an embodiment of the present invention. Figure 6 This is a schematic diagram of the three-dimensional structure of the inspection robot provided by an embodiment of the present invention from another angle. Figure 7 A schematic diagram of the structure of a shielding device for an inspection robot provided in an embodiment of the present invention. Figure 8 This is a schematic diagram of the structure of the anti-collision bar of the inspection robot provided by an embodiment of the present invention.

[0097] Specifically, Figure 4 As shown, the inspection robot 100 of an embodiment of the present invention includes a robot body 1 and a shielding device 2, the robot body 1 includes an upper body 11 and a lower body 12, a control system is provided in the lower body 12, and the shielding device 2 is attached to the outer side surface of the robot body 1 and is provided on the upper side of the lower body 12.

[0098] When the inspection robot 100 of the embodiment of the present invention performs inspection work in rainy weather, the shielding device 2 can shield rainwater to prevent rainwater from flowing along the side of the robot body 1 and seeping into the lower body 12, thereby protecting the control system in the lower body 12, improving the self-protection capability of the inspection robot 100, and reducing the risk of malfunction of the inspection robot 100 in rainy weather.

[0099] like Figures 4 to 6 As shown, the shielding device 2 is arranged at the height where the upper body 11 and the lower body 12 meet, and can collect rainwater dripping on the upper body 11 to prevent the rainwater from flowing along the outer side of the lower body 12 and penetrating into the lower body 12, thereby improving the self-protection capability of the inspection robot 100.

[0100] The inspection robot 100 further includes a walking device 3 , which includes four walking wheels. The four walking wheels are arranged on the lower side of the robot body 1 . The walking device 3 is driven by four-wheel drive and has a strong ability to climb slopes and overcome obstacles.

[0101] A support platform 13 is provided on the upper side of the upper body 11 , and a voice broadcast system 131 is provided on the support platform 13 . The voice broadcast system 131 can be used for two-way voice communication with the inspection robot 100 , and commands and feedback information can be given to the inspection robot 100 .

[0102] A pan-tilt platform 14 is provided on the upper side of the support platform 13, and visible light cameras 15 and infrared cameras 16 are provided on both sides of the pan-tilt platform 14; thus, the temperature of a designated area in the substation can be collected by infrared camera 16, and the designated area in the substation can be monitored by visible light camera 15. The visible light camera 15 and the infrared camera 16 constitute a pan-tilt camera.

[0103] A voiceprint monitoring device 17 is provided on the upper side of the pan / tilt platform 14. Voiceprint information can be collected through the voiceprint monitoring device 17, so that a more accurate inspection of the substation can be performed. The voiceprint monitoring device 17 is, for example, a voiceprint array sensor.

[0104] In some embodiments, the shielding device 2 is detachably connected to the robot body 1 .

[0105] By detachably connecting the shielding device 2 to the robot body 1 , when the shielding device 2 is damaged, it is convenient to disassemble, assemble and replace the shielding device 2 , thereby improving the maintenance convenience of the shielding device 2 .

[0106] As an example, Figure 6 and Figure 7 As shown, the shielding device 2 has an oblong through hole, and the robot body 1 is provided with a threaded hole. The through hole is aligned with the threaded hole, and a detachable connection of the shielding device 2 can be achieved by bolts.

[0107] In some embodiments, Figure 6 and Figure 7 As shown, the shielding device 2 has a drainage groove 21, and the drainage groove 21 has an opening arranged upward.

[0108] In rainy weather, rainwater dripping on the robot body 1 flows outward and flows into the drainage groove 21 through the opening of the drainage groove, and is discharged outward through the drainage groove 21, preventing rainwater from flowing along the side of the robot body 1 and seeping into the lower body 12, thereby protecting the control system in the lower body 12 and reducing the risk of malfunction of the inspection robot 100 in rainy weather.

[0109] In some embodiments, Figure 6 and Figure 7As shown, the shielding device 2 includes a first section 23, a second section 24 and a third section 25 arranged in sequence along the circumference of the robot body 1, and at least one of the end of the first section 23 away from the second section 24 and the end of the third section 25 away from the second section 24 has a drainage port 231.

[0110] After rainwater is collected in the drainage groove 21 , it can be discharged to the outside through the drainage port 231 .

[0111] As an example, Figure 6 and Figure 7 As shown, the control system is arranged in the lower body 12 and close to the rear side of the lower body 12. The shielding device 2 is in the shape of a long strip, and its cross section is in the shape of a groove to form a drainage groove 21. The shielding device 2 is bent along the robot body 1 into a U shape, thereby forming the above-mentioned first section 23, second section 24 and third section 25; the second section 24 is arranged at the rear side of the robot body 1, and the first section 23 and the third section 25 extend from both ends of the second section 24 to the front side of the robot body 1, that is, the first section 23 and the third section 25 are respectively arranged on both sides of the robot body 1, and the ends of the first section 23 and the third section 25 away from the second section 24 have drainage ports 231.

[0112] As a result, rainwater dripping on the robot body 1 flows to its edge and gathers in the drainage groove 21, and is then discharged from the drainage port 231, which can prevent the rainwater from flowing downward along the outer side of the robot body 1 and penetrating into the lower body 12, thereby reducing the risk of damage to the control system of the inspection robot 100.

[0113] In other embodiments, when the control system is disposed on the front side of the lower body 12, the shielding device 2 may also be disposed around the front side of the robot body 1, or the shielding device 2 may surround the robot body 1 for one circle, thereby achieving a better rain-shielding effect.

[0114] Alternatively, if Figure 5 As shown, the drainage port 231 is arranged on the upper side of the front running wheel, and a baffle 31 is provided on the upper side of the front running wheel.

[0115] Thus, rainwater is discharged from the drain port 231 onto the baffle plate 31 and then flows downward along the baffle plate 31. By providing the baffle plate 31, rainwater discharged from the drain port 231 can be prevented from impacting the front running wheel and causing damage to the front running wheel.

[0116] In some embodiments, Figure 5 and Figure 7 As shown, a drainage plate 26 is provided at the drainage outlet 231 , and the drainage plate 26 is arranged obliquely downward away from the drainage outlet 231 .

[0117] Through the above arrangement, the drainage board 26 can guide the rainwater, so that the rainwater flows smoothly along the bottom of the drainage groove 21, thereby improving the drainage efficiency.

[0118] Optionally, the included angle between the drain plate 26 and the bottom wall of the drain groove 21 is 10° to 30°, preferably 15°.

[0119] In some embodiments, Figure 4 and Figure 5 As shown, an anti-collision device 4 is provided on the robot body 1.

[0120] By providing the anti-collision device 4, the probability of the inspection robot 100 colliding with other obstacles during operation can be reduced, thereby further improving the self-protection capability of the inspection robot 100 and reducing the risk of damage to the inspection robot 100.

[0121] In some embodiments, Figure 5 and Figure 6 As shown, the anti-collision device 4 includes an obstacle detection device 41 for detecting obstacles in front of the robot body 1 , and the obstacle detection device 41 is connected to the robot body 1 .

[0122] While the inspection robot 100 is moving forward, the obstacle detection device 41 can detect obstacles on the front side, and then control the inspection robot 100 to stop and wait for the obstacle to be removed before continuing to move forward. If the obstacle still exists, the inspection robot 100 can report the information to the base station and wait for manual handling of the obstacle; thereby, the probability of the inspection robot 100 colliding with obstacles can be reduced, thereby improving its own protection capability.

[0123] Optionally, the obstacle detection device 41 is a laser radar.

[0124] Laser radar has the advantages of high precision and high reliability, and can realize accurate obstacle avoidance of the inspection robot 100.

[0125] like Figure 5 As shown, the inspection robot 100 has a central axis extending in the front-to-back direction thereof, and the obstacle detection device is arranged on the central axis and arranged near the front side of the robot body 1; the obstacle detection device 41 is connected to the upper body 11 and protrudes upward from the upper body 11. The obstacle detection device 41 can detect obstacles in front of the inspection robot 100, and when the distance between the inspection robot 100 and the obstacle reaches a preset value, the inspection robot 100 can be controlled to stop.

[0126] In some embodiments, the anti-collision device 4 includes an anti-collision bar 42 and an anti-collision sensor. The anti-collision bar 42 is connected to the robot body 1 , and the anti-collision sensor is connected to the anti-collision bar 42 .

[0127] When the inspection robot 100 collides with other objects during walking, the anti-collision bar 42 first contacts the colliding object, and then the anti-collision sensor detects the collision of the inspection robot 100, causing the inspection robot 100 to stop immediately and change its walking direction, thereby further improving the self-protection capability of the inspection robot 100 and reducing the risk of damage due to collision.

[0128] Optionally, the anti-collision strip 42 is made of a flexible material.

[0129] Optionally, the anti-collision sensor is a pressure sensitive switch.

[0130] Therefore, when the anti-collision strip 42 collides with an object, the anti-collision strip 42 undergoes elastic deformation, squeezing the pressure-sensitive switch. The pressure-sensitive switch senses that the anti-collision strip 42 collides with the other object, and then controls the inspection robot 100 to stop immediately.

[0131] As an example, Figure 8 As shown, the anti-collision strip 42 is in the shape of a long strip, and its length direction is perpendicular to the front-to-back direction of the robot body 1. The anti-collision strip 42 includes a connecting portion 421 and a protruding portion 422. The connecting portion 421 is connected to the lower body 12 by bolts. The cross-section of the protruding portion 422 is conical. The protruding portion 422 and the connecting portion 421 are fixed by bonding. There is a cavity between the protruding portion 422 and the connecting portion 421, and the anti-collision sensor is arranged in the cavity.

[0132] like Figure 4 As shown, there are two anti-collision strips 42, which are respectively arranged on the front and rear sides of the robot body 1; the edge of the raised portion 422 arranged on the front side is the front end of the inspection robot 100, and the edge of the raised portion 422 arranged on the rear side is the rear end of the inspection robot 100.

[0133] Therefore, when the inspection robot 100 collides with other objects during the forward or backward process, the anti-collision sensor can sense it in the first time, and then control the inspection robot 100 to immediately stop walking and adjust the walking direction.

[0134] In some embodiments, an anti-fall device 5 is provided on the front side of the robot body 1 .

[0135] When the robot is moving forward, the anti-fall device 5 can reduce the risk of the inspection robot 100 falling to high platforms, pits, stairs and other areas, thereby further improving the self-protection capability of the inspection robot 100 and reducing the risk of damage due to falling.

[0136] In some embodiments, the anti-fall device 5 includes an anti-fall detection device 52 and a mounting shell 51 , the mounting shell 51 is connected to the robot body 1 , and the anti-fall detection device 52 is connected to the mounting shell 51 .

[0137] The anti-fall detection device 52 can detect the environment in front of the inspection robot 100 as it moves forward, and promptly detect dangerous areas such as high platforms, pits, stairs, etc., thereby reducing the risk of the inspection robot 100 falling from a height.

[0138] Optionally, the anti-fall detection device 52 is a laser radar.

[0139] As an example, Figure 5 As shown, the anti-fall device 5 is arranged on the front side of the robot body 1 and installed on the upper side of the anti-collision strip 42; the mounting shell 51 is connected to the lower body 12 by bolts, and the mounting shell 51 has a mounting cavity, and the anti-fall detection device 52 is arranged in the mounting cavity, and the mounting cavity has an opening. The laser emitting end of the anti-fall detection device 52 emits laser downward at an angle through the opening to detect the environment on the front side of the inspection robot 100 to prevent the inspection robot 100 from falling.

[0140] There are two anti-fall devices 5 , both of which are arranged at the front side of the robot body 1 and spaced apart, thereby improving the accuracy of detecting dangerous areas such as high platforms, pits, stairs, etc.

[0141] In an embodiment of the present invention, the control system includes a registration and calibration system for the fusion of the voiceprint array and the visible light image. The inspection robot collects data from the monitored equipment (e.g., a calibration plate with a sound generator) to obtain visible light images and acoustic wave array signals, and performs registration and calibration through the registration and calibration system for the fusion of the voiceprint array and the visible light image to obtain a target registration model.

[0142] In an embodiment of the present invention, the control system includes a processing unit. The processing unit includes a target registration model. The processing unit is connected to a visible light image acquisition device and a soundprint array sensor. The processing unit is used to obtain a target visible light image and a target sound wave array signal, and input the target sound wave array signal into the target registration model to fuse the target soundprint array signal into the target visible light image. This provides more comprehensive and accurate environmental data for the positioning and navigation of the inspection robot, especially data for the monitored equipment, to improve the positioning and navigation accuracy of the robot.

[0143] In order to implement the above embodiments, the present invention also proposes an electronic device, comprising: a processor, and a memory communicatively connected to the processor; the memory stores computer-executable instructions; the processor executes the computer-executable instructions stored in the memory to implement the method provided by the above embodiments.

[0144] In order to implement the above embodiments, the present invention further proposes a computer-readable storage medium, in which computer-executable instructions are stored. When the computer-executable instructions are executed by a processor, they are used to implement the methods provided by the above embodiments.

[0145] In order to implement the above embodiments, the present invention further provides a computer program product, including a computer program, which implements the methods provided by the above embodiments when executed by a processor.

[0146] In the description of the aforementioned embodiments, the description with reference to the terms "one embodiment", "some embodiments", "example", "specific example", or "some examples" etc. means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present invention. In this specification, the schematic representations of the above terms do not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described may be combined in any one or more embodiments or examples in a suitable manner. In addition, those skilled in the art may combine and combine the different embodiments or examples described in this specification and the features of the different embodiments or examples, unless they are contradictory.

[0147] In addition, the terms "first" and "second" are used for descriptive purposes only and should not be understood as indicating or implying relative importance or implicitly indicating the number of the indicated technical features. Therefore, the features defined as "first" and "second" may explicitly or implicitly include at least one of the features. In the description of the present invention, the meaning of "plurality" is at least two, such as two, three, etc., unless otherwise clearly and specifically defined.

[0148] Any process or method description in a flowchart or otherwise described herein may be understood to represent a module, segment or portion of code comprising one or more executable instructions for implementing the steps of a custom logical function or process, and the scope of the preferred embodiments of the present invention includes alternative implementations in which functions may not be performed in the order shown or discussed, including performing functions in a substantially simultaneous manner or in reverse order depending on the functions involved, which should be understood by technicians in the technical field to which the embodiments of the present invention belong.

[0149] The logic and / or steps represented in the flowchart or otherwise described herein, for example, can be considered as an ordered list of executable instructions for implementing logical functions, and can be embodied in any computer-readable medium for use by an instruction execution system, device or apparatus (such as a computer-based system, a system including a processor, or other system that can fetch instructions from an instruction execution system, device or apparatus and execute the instructions), or in combination with these instruction execution systems, devices or apparatuses. For the purpose of this specification, "computer-readable medium" can be any device that can contain, store, communicate, propagate or transmit a program for use by an instruction execution system, device or apparatus, or in combination with these instruction execution systems, devices or apparatuses. More specific examples of computer-readable media (a non-exhaustive list) include the following: an electrical connection with one or more wires (electronic device), a portable computer disk box (magnetic device), a random access memory (RAM), a read-only memory (ROM), an erasable and programmable read-only memory (EPROM or flash memory), a fiber optic device, and a portable compact disk read-only memory (CDROM). In addition, the computer-readable medium may even be paper or other suitable medium on which the program is printed, since the program may be obtained electronically, for example, by optically scanning the paper or other medium and then editing, interpreting or processing in other suitable ways if necessary, and then stored in a computer memory.

[0150] It should be understood that the various parts of the present invention can be implemented by hardware, software, firmware or a combination thereof. In the above-mentioned embodiments, a plurality of steps or methods can be implemented by software or firmware stored in a memory and executed by a suitable instruction execution system. For example, if implemented by hardware, as in another embodiment, it can be implemented by any one of the following technologies known in the art or their combination: a discrete logic circuit having a logic gate circuit for implementing a logic function for a data signal, a dedicated integrated circuit having a suitable combination of logic gate circuits, a programmable gate array (PGA), a field programmable gate array (FPGA), etc.

[0151] A person skilled in the art may understand that all or part of the steps in the method for implementing the above-mentioned embodiment may be completed by instructing related hardware through a program, and the program may be stored in a computer-readable storage medium, which, when executed, includes one or a combination of the steps of the method embodiment.

[0152] In addition, each functional unit in each embodiment of the present invention may be integrated into a processing module, or each unit may exist physically separately, or two or more units may be integrated into one module. The above-mentioned integrated module may be implemented in the form of hardware or in the form of a software functional module. If the integrated module is implemented in the form of a software functional module and sold or used as an independent product, it may also be stored in a computer-readable storage medium.

[0153] The storage medium mentioned above may be a read-only memory, a magnetic disk or an optical disk, etc. Although the embodiments of the present invention have been shown and described above, it can be understood that the above embodiments are exemplary and cannot be understood as limiting the present invention. A person of ordinary skill in the art may change, modify, replace and modify the above embodiments within the scope of the present invention.

Claims

1. A registration and calibration method for the fusion of voiceprint array and visible light image, characterized in that: A calibration board is provided and a plurality of sound generators are arranged on the calibration board. The method comprises the following steps: Acquiring a visible light image of the calibration plate, and collecting a voiceprint array signal from the calibration plate; Performing acoustic wave preprocessing on the voiceprint array signal to obtain various sound source points, and performing image preprocessing on the visible light image to obtain image feature points; Using a machine learning algorithm to locate target image feature points associated with each sound source point from all image feature points to obtain a plurality of target matching pairs, each target matching pair including the coordinates of a sound source point and the coordinates of a target image feature point; A registration model is constructed using a perspective projection transformation model, and the parameters of the registration model are determined using the target matching pair, so as to obtain a target registration model that fuses the voiceprint array signal into the visible light image.

2. The registration and calibration method for the fusion of voiceprint array and visible light image according to claim 1 is characterized in that: The method of using a machine learning algorithm to locate target image feature points associated with each sound source point from all image feature points to obtain a plurality of target matching pairs includes: Using geometric constraints, initial image feature points associated with each sound source point are selected from all image feature points to obtain an initial matching pair, wherein the initial matching pair includes a coordinate of a sound source point and a coordinate of an initial image feature point; Obtaining a trained machine learning algorithm, wherein the input of the machine learning algorithm is the coordinates of the experimental sound source point, and the output is the coordinates of the image feature points associated with the experimental sound source point; Inputting the coordinates of each sound source point into the trained machine learning algorithm to obtain predicted coordinates of image feature points associated with each sound source point; The coordinates of the initial image feature points in the initial matching pairs are adjusted based on the predicted coordinates to obtain target image feature points associated with each sound source point, thereby obtaining a plurality of target matching pairs.

3. The registration and calibration method for the fusion of voiceprint array and visible light image according to claim 2 is characterized in that: After obtaining the initial matching pairs, they need to be screened based on similarity before using the trained machine learning algorithm.

4. The registration and calibration method for the fusion of voiceprint array and visible light image according to claim 1, characterized in that: After the parameters of the registration model are determined by using the target matching pair, the method further includes optimizing the parameters of the registration model by using a nonlinear optimization algorithm.

5. The registration and calibration method for the fusion of voiceprint array and visible light image according to claim 1, characterized in that: The performing acoustic wave preprocessing on the voiceprint array signal to obtain each sound source point includes: The voiceprint array signal in the time domain is converted into voiceprint frequency domain characteristics; based on the voiceprint frequency domain characteristics, the beamforming technology is used to obtain the coordinates of each sound source point.

6. The registration and calibration method for the fusion of voiceprint array and visible light image according to claim 1, characterized in that: The performing image preprocessing on the visible light image to obtain image feature points includes: After performing histogram equalization on the visible light image, characteristic points are extracted from the equalized visible light image using an edge detection algorithm and a feature extraction algorithm, thereby obtaining image feature points.

7. A registration and calibration system for the fusion of voiceprint array and visible light image, characterized in that: A calibration board is provided and a plurality of sound generators are arranged on the calibration board, the system comprising: An acquisition module, used to acquire a visible light image of the calibration plate and to acquire a voiceprint array signal from the calibration plate; A preprocessing module, used to perform acoustic wave preprocessing on the voiceprint array signal to obtain various sound source points, and perform image preprocessing on the visible light image to obtain image feature points; A matching module, used to locate target image feature points associated with each sound source point from all image feature points using a machine learning algorithm to obtain a plurality of target matching pairs, each target matching pair including the coordinates of a sound source point and the coordinates of a target image feature point; The registration and calibration module is used to construct a registration model using a perspective projection transformation model, and to determine the parameters of the registration model using the target matching pair, so as to obtain a target registration model that fuses the voiceprint array signal into the visible light image.

8. A patrol robot, characterized in that: include: A vehicle body, a visible light image acquisition device arranged on the vehicle body, a voiceprint array sensor, and a processing unit; The visible light image acquisition device is used to acquire a target visible light image at a target position, and the voiceprint array sensor is used to acquire a target sound wave array signal at a target position; The processing unit includes a target registration model obtained by using the method described in any one of claims 1-6, and the processing unit is used to obtain a target visible light image and a target acoustic wave array signal, and input the target acoustic wave array signal into the target registration model to fuse the target voiceprint array signal into the target visible light image.

9. An electronic device, characterized in that: include: A processor, and a memory communicatively connected to the processor; The memory stores computer-executable instructions; The processor executes the computer-executable instructions stored in the memory to implement the method according to any one of claims 1 to 6.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer-executable instructions, which are used to implement the method according to any one of claims 1 to 6 when executed by a processor.

Images