A Robot Indoor Positioning Method and Intelligent Terminal Based on Acoustic-Optical Data Fusion

Abstract

This invention discloses a robot indoor localization method based on acoustic-optical data fusion. The method includes: acquiring a predicted coordinate sequence; obtaining a training dataset and a test dataset based on the predicted coordinate sequence and the true coordinate sequence; constructing a neural network model; training the neural network model using the training dataset to obtain a trained neural network model; inputting newly acquired light source localization data and sound source localization data into the trained neural network model to obtain coordinate prediction values ​​after acoustic-optical data fusion; and performing Kalman filtering on the coordinate prediction values ​​after acoustic-optical data fusion to obtain the final coordinate prediction values. This invention considers the situation where indoor lighting is obstructed, using a sound source localization method based on arrival delay difference as compensation, and utilizing a BP neural network model to achieve the fusion of acoustic and optical methods, thus achieving high-precision indoor power inspection robot localization while ensuring real-time positioning.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of sound source and light source positioning, and particularly relates to a robot indoor positioning method based on sound-light data fusion and an intelligent terminal. BACKGROUND

[0002] The indoor positioning technology of the inspection robot refers to realizing the position positioning of the inspection robot in an indoor environment, mainly adopting wireless communication, base station positioning, inertial navigation positioning and other technologies to form an indoor position positioning system, so that the position of the inspection robot in the indoor space is monitored. The traditional positioning method of the inspection robot is usually to use the Bluetooth positioning and UWB positioning method based on radio frequency technology. However, in the indoor inspection scene of the substation, such as the information room, the power distribution room, the main control room and the high-voltage opening and closing room, there is a large electromagnetic interference due to the existence of power frequency noise and the operation of high-power power equipment, and the electromagnetic signal itself has a multipath effect, which leads to a large error of the positioning method based on radio frequency technology. The positioning method based on laser point cloud and computer vision has a large amount of calculation, and the indoor environment of the substation is complex, so that the inspection robot using the positioning method based on laser point cloud and computer vision cannot realize accurate indoor positioning.

[0003] Therefore, the prior art still needs to be improved and developed. SUMMARY

[0004] The technical problem to be solved by the present application is to provide a robot indoor positioning method based on sound-light data fusion and an intelligent terminal aiming at the above-mentioned defects of the prior art, so as to solve the problem of large positioning error and inability to realize accurate indoor positioning caused by complex environment in the prior art.

[0005] The technical solution adopted by the present application to solve the technical problem is as follows:

[0006] In a first aspect, the present application provides a robot indoor positioning method based on sound-light data fusion, wherein the method comprises:

[0007] obtaining a predicted coordinate sequence; the predicted coordinate sequence comprises a light source positioning predicted coordinate sequence and a sound source positioning predicted coordinate sequence;

[0008] obtaining a training data set and a test data set according to the predicted coordinate sequence and a real coordinate sequence;

[0009] constructing a neural network model;

[0010] training the neural network model through the training data set to obtain a trained neural network model;

[0011] input the newly collected light source positioning data and the sound source positioning data into the trained neural network model to obtain a coordinate prediction value after sound-light data fusion;

[0012] perform Kalman filtering on the coordinate prediction value after sound-light data fusion to obtain a final coordinate prediction value.

[0013] In an implementation manner, before the light source positioning data is acquired, the method further includes:

[0014] establishing a sound-light data fusion positioning model; wherein the sound-light data fusion positioning model comprises a sound source detection device, a white light LED light source, a fixed frequency sound source, a patrol robot, and a photodetector arranged at the top center position of the patrol robot; a microphone array is arranged on the sound source detection device; the white light LED light source is used to emit white light signals in turn; the photodetector is used to acquire an output voltage when the white light LED light source emits light; the fixed frequency sound source is used to emit an audio signal of a fixed frequency; and the microphone array is used to acquire the audio signal;

[0015] dividing an indoor space into a plurality of small areas, collecting a center position coordinate of each small area as the real coordinate sequence, and aligning the center position of the patrol robot with the center of each small area in turn to collect a prediction coordinate sequence.

[0016] In an implementation manner, the training data set and the test data set are obtained according to the prediction coordinate sequence and the real coordinate sequence, including:

[0017] performing a normalization operation on the prediction coordinate sequence and the real coordinate sequence to obtain a normalized prediction coordinate sequence and a normalized real coordinate sequence;

[0018] performing bilinear interpolation on the normalized prediction coordinate sequence and the normalized real coordinate sequence to obtain a plurality of groups of training data;

[0019] grouping the training data according to a preset proportion to obtain the training data set and the test data set.

[0020] In an implementation manner, the prediction coordinate sequence is acquired, including:

[0021] in each small area, acquiring an output voltage when a single white light LED light source emits light;

[0022] obtaining a light power of each white light LED light source when the white light LED light source emits light according to the output voltage when the single white light LED light source emits light;

[0023] obtaining a light power ratio between LED light sources according to the light power of each white light LED light source when the white light LED light source emits light;

[0024] According to the light power ratio, a receiving plane coordinate curve equation is obtained;

[0025] Solving the receiving plane coordinate curve equation, a predicted center position coordinate of each small area is obtained;

[0026] According to the predicted center position coordinate of each small area, a light source positioning prediction coordinate sequence is obtained.

[0027] In an implementation manner, the obtaining the prediction coordinate sequence comprises:

[0028] In each small area, an acoustic intensity signal emitted by a fixed frequency sound source is obtained, and a received acoustic intensity signal is obtained;

[0029] The received acoustic intensity signal is calculated by using a GCC-PATH algorithm, and a time delay estimation is obtained;

[0030] The time delay estimation is calculated by using a sound source positioning far field model, and a plurality of sound source azimuths are obtained; the sound source azimuth is a plane azimuth of a sound source relative to each sound source detection device;

[0031] According to the sound source azimuth, a position coordinate of the sound source detection device, and a vertical distance between the sound source detection device and the ground, a sound source coordinate value of the plurality of sound source detection devices is obtained;

[0032] An average value of the sound source coordinate value of the sound source detection device is calculated, and sound source positioning data in each small area is obtained;

[0033] According to the sound source positioning data in each small area, the light source positioning prediction coordinate sequence is obtained.

[0034] In an implementation manner, the neural network model comprises an input layer, an output layer and an intermediate layer; wherein the neural network model is a fully connected network model, the input layer has four components, the output layer has two components, the intermediate layer has seven neurons, the activation function is a Sigmoid activation function, and the error function is a mean square error function.

[0035] In an implementation manner, the training of the neural network model by using the training data set to obtain a trained neural network model comprises:

[0036] Initializing model parameters; wherein the initializing model parameters comprise random assignment of neurons of the input layer, the intermediate layer and the output layer of the neural network model to obtain initial model parameters;

[0037] input each set of training data in the training data set into an input layer of the neural network model, calculate and output layer by layer using the initial model parameters to obtain a predicted coordinate corresponding to each set of training data;

[0038] According to the predicted coordinate and the real coordinate corresponding to each set of training data, using the mean square error function to calculate, the error corresponding to each set of training data is obtained, and the error loss function gradient information is propagated in reverse using the error back propagation algorithm, and the neuron parameters in the neural network model are updated;

[0039] Select the model parameter with the minimum error in the error corresponding to each set of training data as the final model parameter, obtain the trained neural network model, and evaluate the training effect of the trained neural network model using the test data set.

[0040] In a second aspect, the embodiments of the present application also provide a robot indoor positioning device based on acoustic-optical data fusion, wherein the device comprises:

[0041] A predicted coordinate sequence acquisition module is configured to acquire a predicted coordinate sequence; the predicted coordinate sequence comprises a light source positioning predicted coordinate sequence and a sound source positioning predicted coordinate sequence;

[0042] A data set acquisition module is configured to obtain a training data set and a test data set according to the predicted coordinate sequence and a real coordinate sequence;

[0043] A model construction module is configured to construct a neural network model;

[0044] A training module is configured to train the neural network model through the training data set to obtain a trained neural network model;

[0045] A prediction module is configured to input newly collected light source positioning data and sound source positioning data into the trained neural network model to obtain a coordinate prediction value after acoustic-optical data fusion;

[0046] A final coordinate prediction value acquisition module is configured to perform Kalman filtering on the coordinate prediction value after acoustic-optical data fusion to obtain a final coordinate prediction value.

[0047] In a third aspect, the embodiments of the present application also provide an intelligent terminal, wherein the intelligent terminal comprises a memory, a processor, and a robot indoor positioning program based on acoustic-optical data fusion stored in the memory and executable on the processor; when the processor executes the robot indoor positioning program based on acoustic-optical data fusion, the steps of the robot indoor positioning method based on acoustic-optical data fusion are implemented.

[0048] In a fourth aspect, the embodiments of the present application further provide a computer readable storage medium, wherein the computer readable storage medium stores a robot indoor positioning program based on sound-light data fusion, and the robot indoor positioning program based on sound-light data fusion, when executed by a processor, implements the steps of the robot indoor positioning method based on sound-light data fusion as described in any one of the preceding aspects.

[0049] Beneficial effects: Compared with the prior art, the present application provides a robot indoor positioning method based on sound-light data fusion. The method first obtains a light source positioning prediction coordinate sequence and a sound source positioning prediction coordinate sequence, and obtains a training data set and a test data set according to the prediction coordinate sequence and a real coordinate sequence, and trains a neural network model. The combination of white light LED positioning technology and sound source positioning is achieved, and in the case that indoor lighting is blocked, a sound source positioning method based on time delay difference is used as compensation. Then, the newly collected light source positioning data and sound source positioning data are input into the trained neural network model to obtain a sound-light data fused coordinate prediction value. The fusion of the two methods is realized through the BP neural network model, and in the case of ensuring the real-time positioning, the high-precision indoor power inspection robot positioning is realized. Finally, the sound-light data fused coordinate prediction value is subjected to Kalman filtering to obtain a final coordinate prediction value, so as to eliminate the interference value and make the prediction result more accurate. BRIEF DESCRIPTION OF DRAWINGS

[0050] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the following will briefly introduce the drawings needed to be used in the embodiments or prior art description. Obviously, the drawings in the following description are only some embodiments of the present application, and for those skilled in the art, other drawings can also be obtained without creative labor.

[0051] Figure 1 is a flowchart of the robot indoor positioning method based on sound-light data fusion provided by the embodiments of the present application.

[0052] Figure 2 is a schematic diagram of a sound-light data fusion positioning model construction provided by the embodiments of the present application.

[0053] Figure 3 is a schematic diagram of microphone deployment provided by the embodiments of the present application.

[0054] Figure 4 is a schematic diagram of Lambertian transmission model provided by the embodiments of the present application.

[0055] Figure 5 is a schematic diagram of sound source signal coordinate system provided by the embodiments of the present application.

[0056] Figure 6is a far-field model schematic diagram of sound source positioning provided by an embodiment of the present application.

[0057] Figure 7 is a far-field model schematic diagram of sound source positioning of a cross microphone array provided by an embodiment of the present application.

[0058] Figure 8 is a neural network model structure schematic diagram provided by an embodiment of the present application.

[0059] Figure 9 is an experimental error schematic diagram provided by an embodiment of the present application.

[0060] Figure 10 is a physical schematic diagram of a sound source positioning device provided by an embodiment of the present application.

[0061] Figure 11 is a physical schematic diagram of a sound source positioning device provided by an embodiment of the present application.

[0062] Figure 12 is a principle block diagram of a robot indoor positioning device based on sound and light data fusion provided by an embodiment of the present application.

[0063] Figure 13 is a principle block diagram of an internal structure of an intelligent terminal provided by an embodiment of the present application. DETAILED DESCRIPTION

[0064] In order to make the purpose, technical scheme and effect of the present application more clear and explicit, the present application is further described in detail below with reference to the drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and do not limit the present application.

[0065] Those skilled in the art can understand that, unless specifically stated otherwise, the singular forms "a", "an" and "the" used herein also include the plural forms. It should be further understood that the use of the term "include" in the specification of the present application means that the features, integers, steps, operations, elements and / or components exist, but do not exclude the existence or addition of one or more other features, integers, steps, operations, elements, components and / or their combinations. It should be understood that when we say that an element is "connected" or "coupled" to another element, it can be directly connected or coupled to the other element, or there can be intermediate elements. In addition, the "connection" or "coupling" used herein can include wireless connection or wireless coupling. The phrase "and / or" used herein includes all or any single unit and all combinations of the associated listed items.

[0066] It will be understood by those skilled in the art that, unless otherwise defined, all terms used herein (including technical and scientific terms) have the same meaning as commonly understood by one of ordinary skill in the art to which this invention pertains. It should also be understood that terms such as those defined in general dictionaries should be understood to have the same meaning as in the context of the prior art, and should not be interpreted in an idealized or overly formal sense unless specifically defined as herein.

[0067] Traditional inspection robot positioning methods often use Bluetooth and UWB positioning based on radio frequency (RF) technology. However, in indoor inspection scenarios in substations, such as information rooms, power distribution rooms, main control rooms, and high-voltage switching rooms, there is significant electromagnetic interference due to power frequency noise and the operation of high-power electrical equipment. In addition, electromagnetic signals themselves have multipath effects, resulting in large errors for RF-based positioning methods. On the other hand, positioning methods based on laser point clouds and computer vision require a large amount of computation, and due to the complex indoor environment of substations, inspection robots using positioning methods based on laser point clouds and computer vision often cannot achieve accurate indoor positioning.

[0068] To address the aforementioned technical problems, this invention proposes a method combining white LED positioning technology and sound source positioning. The white LED positioning technology calculates the ratio of white LED light intensities received at different locations by a photodetector mounted on the power inspection robot to achieve indoor positioning. Considering the possibility of indoor lighting obstruction, a sound source positioning method based on arrival delay difference is used as compensation. A BP neural network model is used to fuse the two methods, achieving high-precision indoor power inspection robot positioning while ensuring real-time positioning.

[0069] Exemplary methods

[0070] This embodiment provides a robot indoor positioning method based on acoustic-optical data fusion. For example... Figure 1 As shown, the method includes the following steps:

[0071] Step S100: Obtain the predicted coordinate sequence; the predicted coordinate sequence includes the light source positioning predicted coordinate sequence and the sound source positioning predicted coordinate sequence.

[0072] Specifically, this embodiment uses white LED positioning technology to achieve indoor positioning of the robot mounted on the power inspection system, so as to obtain the predicted coordinate sequence of the light source positioning. At the same time, considering the situation that indoor lighting is blocked, a sound source positioning method based on arrival time delay difference is used as compensation to collect the predicted coordinate sequence of the sound source positioning.

[0073] For example, in this embodiment, 100 sets of predicted coordinate sequences (X, Y, X) of the light source positioning of the inspection robot in the indoor space are obtained.Li ,Y Li ) and sound source localization prediction coordinate sequence (X si ,Y si ), i = 0, 1, ..., 99.

[0074] In one implementation, the following steps are included before step S100 in this embodiment:

[0075] Step S10: Establish an acoustic-optical data fusion positioning model; wherein, the acoustic-optical data fusion positioning model includes a sound source detection device, a white LED light source, a fixed-frequency sound source, an inspection robot, and a photodetector deployed at the center of the top of the inspection robot; the sound source detection device is equipped with a microphone array; the white LED light source is used to emit white light signals in turn; the photodetector is used to acquire the output voltage of the white LED light source when it emits light; the fixed-frequency sound source is used to emit audio signals of a fixed frequency; the microphone array is used to acquire the audio signals;

[0076] For example, in one implementation, such as Figure 2 As shown, LEDs 1 to 4 are white LED light sources, numbered 1, 2, 3, and 4 respectively. In this embodiment, all position calculations use the center position of LED 1 as the coordinate origin. Four sound source positioning devices are installed above the center positions of each white LED. The photodetector and the speaker are installed near the center of the inspection robot's centerline. The microphone array uses a four-microphone cross-shaped array, and the microphones are silicon-based omnidirectional microphones. The same 12V DC voltage signal is applied to the four white LEDs numbered 1, 2, 3, and 4 in turn during different time slices (each time slice is 200ms). A rectangular wave signal with a frequency of 5000Hz and a duty cycle of 50% is applied to 100 parallel 12085 passive buzzers, making their sound frequency 5000Hz. The center-to-center distance between the sound source positioning devices is 945cm, and the center-to-center distance between the white LEDs is 945cm.

[0077] The component models used in this embodiment are shown in Table 1.

[0078] Table 1. Component Models

[0079] Name Type White LED LXHL-LW3C LED Photodetector PIN photodiode BPW34S, OSRAM Omnidirectional silicon-based microphone INMP441 omnidirectional microphone module Sound-emitting source 12085 passive buzzer

[0080] Step S20: Divide the indoor space into several small areas, collect the center coordinates of each small area as the true coordinate sequence, and align the center position of the inspection robot with the center of each small area in turn to collect the predicted coordinate sequence.

[0081] Specifically, in this embodiment, the indoor space floor is divided into 100 small areas, each with an area of ​​1 square meter. Since the starting coordinates of the indoor space and the side lengths of the small areas are known, the center coordinates of each small area can be obtained. During the movement of the inspection robot, its center position is sequentially aligned with the center of each small area. In this way, a set of predicted coordinates can be obtained for each small area traversed by the inspection robot, and the predicted coordinates of all small areas can form a predicted coordinate sequence.

[0082] In one implementation, step S100 of this embodiment includes the following steps:

[0083] Step S101: In each small area, obtain the output voltage of a single white LED light source when it emits light;

[0084] Step S102: Obtain the light power of each white LED light source when it emits light based on the output voltage of the individual white LED light source when it emits light;

[0085] Step S103: Obtain the light power ratio between LED light sources based on the light power of each white LED light source when it emits light;

[0086] Step S104: Obtain the equation of the receiving plane coordinate curve based on the optical power ratio;

[0087] Step S105: Solve the equation of the receiving plane coordinate curve to obtain the predicted center position coordinates of each small region;

[0088] Step S106: Obtain the light source positioning prediction coordinate sequence based on the predicted center position coordinates of each small region.

[0089] Specifically, such as Figure 4 As shown, a Lambertian model for the transmission of a single white LED is first established. The emission of a single LED can be described as a Lambertian volume. It is assumed that ψ is the receiving angle of the photodetector, FOV is the field of view of the photodetector, and φ... 1 / 2 Let φ be the half-power angle of the LED, d be the straight-line distance between the LED and the detector, and h be the perpendicular distance between the LED and the plane of the detector. Assuming the receiving angle ψ of the photodetector is always less than the detector's field of view (FOV), and the white LED emits light at a constant power, then the optical power received by the detector surface is:

[0090]

[0091] Among them, P r This is the LED's emission power, where m is the Lambert exponent, and A is the LED's emission power. r The effective area of ​​the detector is m, which is determined by φ. 1 / 2 The decision is:

[0092] m = -ln2 / ln(cosφ 1 / 2 ) (2)

[0093] In this embodiment, the transmission Lambert model of the four white light LED light sources is obtained according to the transmission Lambert model of the single white light LED.

[0094] Specifically, in this embodiment, as shown in Figure 5 , the four white light LEDs are fixed in the roof ceiling, the inspection robot moves in the bottom plane, and the photodetector is carried on the top center of the body. A constant direct current voltage signal is applied to the four LEDs in different time slices in turn, and the light power values received by the detector when the LEDs emit light alone in the four time slices are P r1 , P r2 , P r3 , and P r4 , respectively, in the order of LED1-LED4.

[0095] Then, according to formulas (1) and (2), the relationship formula about the received signal intensity ratio of the photodetector can be derived as follows:

[0096]

[0097]

[0098]

[0099] Through formulas (4), (5), and (6) in the above step, three curves about the received plane coordinates can be obtained, and the center positions (X L , Y L ) of the inspection robot in the four time slices can be obtained by solving the curves.

[0100] In each small area, the above steps are repeatedly run, and the continuous center coordinate sequence (X Li , Y Li ) of the inspection robot in the 100 small areas can be obtained, i = 0, 1, …, 99.

[0101] In step S107, the sound intensity signal emitted by the fixed frequency sound source is obtained in each small area to obtain the received sound intensity signal.

[0102] Specifically, for any two microphone arrays, a sound source positioning far field model is established.

[0103] Establish a spatial coordinate system, where s is the position of the sound source (x0, y0, z0); the positions of the two microphones m1 and m2 are (M1, 0, 0) and (M2, 0, 0), respectively. Given that the audio signal (sound intensity signal) emitted by the sound source is s(t), the sound intensity signals received by microphones m1 and m2 are respectively (ignoring slight losses in the air):

[0104] x1(t)=s(t-τ1)+n1(t) (7)

[0105] x²(t) = s(t-τ²) + n²(t) (8)

[0106] Where τ1 and τ2 are the delay times of the sound source signal arriving at the two microphones, respectively, and n1(t) and n2(t) are additive random noise, then the time difference of arrival of the sound source signal at the two microphones, i.e., TDOA (Time Difference of Arrival), is τ = τ1 + τ2.

[0107] Step S108: Calculate the received sound intensity signal using the GCC-PATH algorithm to obtain a time delay estimate;

[0108] Specifically, the time delays of x1(t) and x2(t) are estimated using the GCC-PATH algorithm on the two audio signals:

[0109] τ=argmaxR 12 (τ) (9)

[0110]

[0111] Where x1(w) and x2(w) are the Fourier transforms of x1(t) and x2(t), respectively; (·)* denotes conjugate; R 12 (τ) is the generalized cross-correlation function of x1(t) and x2(t), j is the imaginary unit, and w is the angular frequency.

[0112] Step S109: Calculate the time delay estimate using the far-field model of sound source localization to obtain several sound source azimuth angles; the sound source azimuth angles are the plane azimuth angles of the sound source relative to each sound source detection device.

[0113] Specifically, such as Figure 6 As shown, several azimuth angles of the sound source are obtained using a far-field model for sound source localization. Assuming the speed of sound and wavelength are λ, the distance between microphones is d = |M1 - M2|, and the distance between the sound source and the microphone is r, in the far-field model... The waveform reaching the microphone array from the sound source can be considered a plane wave, and according to the microphone's geometry, we have...

[0114]

[0115] Where θ is the angle between the sound source and the line connecting the two microphones, and θ is an acute angle, and c is the speed of sound.

[0116] Step S110: Based on the azimuth angle of the sound source, the position coordinates of the sound source detection device, and the vertical distance between the sound source detection device and the ground, obtain the sound source coordinate values ​​of several sound source detection devices;

[0117] Step S111: Calculate the average value of the sound source coordinates of the sound source detection device to obtain the sound source localization data in each small area;

[0118] Step S112: Based on the sound source localization data in each small area, obtain the sound source localization predicted coordinate sequence.

[0119] Specifically, in this embodiment, as Figure 7 As shown, the cross-shaped microphone array positioning is continued when four microphones are acquired. M1, M2, M3, and M4 represent the four microphones in the array, with coordinates (0, a, 0), (a, 0, 0), (-a, 0, 0), and (0, -a, 0) respectively. The sound source is at point S, with rectangular coordinates (x, y, z). The angle between the projection of OS onto the XOZ plane and the Z-axis is θ, with a value range of (-90°, 90°); the angle between the projection of OS onto the YXZ plane and the Z-axis is... The value is (-90°, +90°). Here, it is assumed that the speed of sound in air is v; from formulas (7) and (8), we can obtain:

[0120] The time delay between microphone ④ and microphone ① is τ. 41 =τ4-τ1,

[0121] The time delay between microphone ③ and microphone ② is τ. 32 =τ3-τ2,

[0122] via τ 41 τ 32 Formulas (9), (10), and (11) in Part II can be used to calculate the azimuth angle θ of the sound source. Furthermore, it can be determined that the projections of the sound source S onto the XOZ and YOZ planes lie on the asymptotes of two hyperbolas, where:

[0123]

[0124]

[0125] The coordinates of the sound source (X′) can be obtained from the above formulas (12) and (13), the location of the sound source locator, and the vertical distance between the sound source locator and the plane. s ,Y′ s), the coordinate values obtained by the four sound source positioning devices are summed and averaged to obtain the final coordinate output (X s , Y s ).

[0126] The final coordinate output (X s , Y s ) is obtained in the 100 small areas of the embodiment, and a sound source positioning prediction coordinate sequence (X si , Y si ) is obtained, i = 0, 1, …, 99.

[0127] Step S200, obtaining a training data set and a test data set according to the prediction coordinate sequence and the real coordinate sequence;

[0128] In an implementation manner, the step S200 of the embodiment comprises the following steps:

[0129] Step S201, performing a normalization operation on the prediction coordinate sequence and the real coordinate sequence to obtain a normalized prediction coordinate sequence and a normalized real coordinate sequence;

[0130] Step S202, performing a bilinear interpolation on the normalized prediction coordinate sequence and the normalized real coordinate sequence to obtain a plurality of groups of training data;

[0131] Step S203, grouping the training data according to a preset proportion to obtain the training data set and the test data set.

[0132] Specifically, the embodiment obtains 100 groups of (X Li , Y Li ) and (X si , Y si ) from the light source positioning prediction coordinate sequence and the sound source positioning prediction coordinate sequence in each small area, i = 0, 1, …, 99, that is, the sound and light predicted center position coordinates of the inspection robot, and performs a normalization operation thereon, and performs a bilinear interpolation on the normalized prediction coordinate sequence and the real coordinate sequence (taking the center position of each small area as the real coordinate sequence) to expand the data set, and finally obtains 181 groups of training data, which are divided into 21 groups of test set and 160 groups of training set.

[0133] Step S300, constructing a neural network model;

[0134] In an implementation manner, as shown in FIG. Figure 8 , the neural network model of the embodiment comprises an input layer, an output layer and an intermediate layer; wherein the neural network model is a fully connected network model, the input layer has four components, the output layer has two components, the intermediate layer has seven neurons, the activation function is a Sigmoid activation function, and the error function is a mean square error function.

[0135] Step S400, training the neural network model through the training data set to obtain a trained neural network model;

[0136] In an implementation manner, the step S400 includes the following steps:

[0137] Step S401, initializing model parameters; wherein the initializing model parameters includes randomly assigning values to neurons of input layers, intermediate layers and output layers of the neural network model to obtain initial model parameters;

[0138] Step S402, inputting each set of training data in the training data set into the input layer of the neural network model, calculating and outputting layer by layer using the initial model parameters to obtain a predicted coordinate corresponding to each set of training data;

[0139] Step S403, calculating an error corresponding to each set of training data according to the predicted coordinate and the real coordinate corresponding to each set of training data using a mean square error function, and updating neuron parameters in the neural network model using an error back propagation algorithm to perform back propagation on gradient information of an error loss function;

[0140] Step S404, selecting model parameters corresponding to the minimum error in the error corresponding to each set of training data as final model parameters to obtain a trained neural network model, and evaluating training effect of the trained neural network model using the test data set.

[0141] Specifically, in the embodiment, the neurons of the input layers, the intermediate layers and the output layers are randomly assigned values, the model parameters are initialized, each set of predicted central position values (X Li ,Y Li ) and (X si ,Y si ) of the inspection robot (i=0, 1, …, 99) are input into the input layer of the model, and the model parameters are calculated and output layer by layer using the initial values of the model parameters. Then, the error between the predicted coordinate and the real coordinate of the output layer is calculated using a mean square error function. The gradient information of the error loss function is back propagated using an error back propagation algorithm, and the neuron parameters in the entire model are updated. Finally, the above steps are repeated until 160 sets of training data are input, the predicted values are calculated forwardly, the error is calculated, the error is back propagated, and the model parameters corresponding to the minimum error are selected as the final model parameters.

[0142] Step S500, inputting newly collected light source positioning data and sound source positioning data into the trained neural network model to obtain a coordinate prediction value after sound-light data fusion;

[0143] Specifically, the sound source positioning predicted coordinate value and the white light LED positioning predicted coordinate value of the newly input are data normalized and input into the trained neural network model to obtain a coordinate prediction value (X p ,Y p ) after sound and light data fusion.

[0144] Step S600, Kalman filtering is performed on the coordinate prediction value after sound and light data fusion to obtain a final coordinate prediction value.

[0145] For example, it is assumed that two inspection robots move in a two-dimensional space and are labeled as target 1-2. The initial position of target 1 is (x, y) = (0 m, 0 m), the velocity vx = 2 m / s, and the vy = -2 m / s; the initial position of target 2 is (x, y) = (0 m, 0 m), the velocity vx = 2.5 m / s, and the vy = 2.5 m / s. It is assumed that the observation station 1 (i.e., the coordinate result obtained by white light LED positioning) has an observation noise mean value of 0 and a variance of 0.1; the observation station 2 (i.e., the coordinate result obtained by the cumulative average of the four sound source positioning devices) has an observation noise mean value of 0 and a variance of 0.2. The observation stations observe the targets independently and do not interfere with each other. The noises are independent and unrelated, and the multi-observation station data fusion error refers to the result obtained by inputting the sound source positioning coordinate prediction value and the white light LED positioning coordinate prediction value into the BP neural network and Kalman filtering. Here, the sampling period (i.e., the output rate of positioning) T = 1 s, 50 positioning experiments are performed, and the experimental error result is shown in Table 2. It can be seen that the multi-observation station information fusion error is much smaller than the single-observation station error. The horizontal axis is the test number, and the vertical axis is the positioning distance error, which is in cm. Table 2 is the test error under different coordinate results. Figure 9

[0146] Table 2, test error

[0147]

[0148]

[0149] Exemplary device

[0150] As shown in Figure 12 , the embodiment also provides a robot indoor positioning device based on sound and light data fusion, which comprises:

[0151] A predicted coordinate sequence acquisition module 10 is configured to acquire a predicted coordinate sequence; the predicted coordinate sequence comprises a light source positioning predicted coordinate sequence and a sound source positioning predicted coordinate sequence;

[0152] A data set acquisition module 20 is configured to obtain a training data set and a test data set according to the predicted coordinate sequence and a true coordinate sequence.​

[0153] The model construction module 30 is configured to construct a neural network model; the neural network model comprises an input layer, an output layer and an intermediate layer; wherein the neural network model is a fully connected network model, the input layer has four components, the output layer has two components, the intermediate layer has seven neurons, the activation function is a Sigmoid activation function, and the error function is a mean square error function.

[0154] The training module 40 is configured to train the neural network model through the training data set to obtain a trained neural network model.

[0155] The prediction module 50 is configured to input newly collected light source positioning data and sound source positioning data into the trained neural network model to obtain a coordinate prediction value after sound-light data fusion.

[0156] The final coordinate prediction value acquisition module 60 is configured to perform Kalman filtering on the coordinate prediction value after sound-light data fusion to obtain a final coordinate prediction value.

[0157] In an implementation manner, the prediction coordinate sequence acquisition module 10 comprises:

[0158] The output voltage acquisition unit is configured to acquire an output voltage of a single white light LED light source when the single white light LED light source emits light in each small area.

[0159] The optical power acquisition unit is configured to obtain an optical power of each white light LED light source when the white light LED light source emits light according to the output voltage of the single white light LED light source when the single white light LED light source emits light.

[0160] The optical power ratio acquisition unit is configured to obtain an optical power ratio between LED light sources according to the optical power of each white light LED light source when the white light LED light source emits light.

[0161] The receiving plane coordinate curve equation acquisition unit is configured to obtain a receiving plane coordinate curve equation according to the optical power ratio.

[0162] The prediction center position coordinate acquisition unit is configured to solve the receiving plane coordinate curve equation to obtain a prediction center position coordinate of each small area.

[0163] The light source positioning prediction coordinate sequence acquisition unit is configured to obtain the light source positioning prediction coordinate sequence according to the prediction center position coordinate of each small area.

[0164] The sound intensity signal acquisition unit is configured to acquire a sound intensity signal emitted by a fixed frequency sound source in each small area to obtain a received sound intensity signal.

[0165] a time delay estimation obtaining unit, configured to obtain time delay estimation by calculating the received sound intensity signals using a GCC-PATH algorithm;

[0166] a sound source azimuth angle obtaining unit, configured to obtain a plurality of sound source azimuth angles by calculating the time delay estimation using a sound source positioning far-field model; the sound source azimuth angle is a plane azimuth angle of a sound source relative to each sound source detection device;

[0167] a sound source coordinate value obtaining unit, configured to obtain a sound source coordinate value of each sound source detection device according to the sound source azimuth angle, the position coordinates of the sound source detection device, and the vertical distance between the sound source detection device and the ground;

[0168] a sound source positioning data obtaining unit, configured to obtain sound source positioning data in each small area by calculating an average value of the sound source coordinate values of the sound source detection device;

[0169] a sound source positioning predicted coordinate sequence obtaining unit, configured to obtain the sound source positioning predicted coordinate sequence according to the sound source positioning data in each small area.

[0170] In an implementation manner, the device further includes:

[0171] a modeling unit, configured to establish an acoustic-optical data fusion positioning model; wherein the acoustic-optical data fusion positioning model includes a sound source detection device, a white light LED light source, a fixed frequency sound source, a patrol robot, and a photodetector arranged at a central position on the top of the patrol robot; a microphone array is arranged on the sound source detection device; the white light LED light source is used to emit white light signals in turn; the photodetector is used to obtain an output voltage when the white light LED light source emits light; the fixed frequency sound source is used to emit an audio signal of a fixed frequency; and the microphone array is used to obtain the audio signal;

[0172] a predicted coordinate sequence collecting unit, configured to divide an indoor space into a plurality of small areas, collect a central position coordinate of each small area as the real coordinate sequence, and align a central position of the patrol robot with a central position of each small area in turn to collect a predicted coordinate sequence.

[0173] In an implementation manner, the data set obtaining module 20 further includes:

[0174] a normalization unit, configured to perform a normalization operation on the predicted coordinate sequence and the real coordinate sequence to obtain a normalized predicted coordinate sequence and a normalized real coordinate sequence;

[0175] a training data obtaining unit, configured to perform bilinear interpolation on the normalized predicted coordinate sequence and the normalized real coordinate sequence to obtain a plurality of groups of training data;

[0176] A grouping unit is configured to group the training data according to a preset proportion to obtain the training data set and a test data set.

[0177] In an implementation, the training module 40 comprises:

[0178] An initialization unit is configured to initialize model parameters, wherein the initialization of the model parameters comprises random assignment of neurons of an input layer, an intermediate layer and an output layer of the neural network model to obtain initial model parameters.

[0179] A predicted coordinate acquisition unit is configured to input each group of training data in the training data set into an input layer of the neural network model, calculate and output layer by layer using the initial model parameters to obtain predicted coordinates corresponding to each group of training data.

[0180] A training unit is configured to calculate errors corresponding to each group of training data according to the predicted coordinates and real coordinates corresponding to each group of training data using a mean square error function, and update neuron parameters in the neural network model by backward propagation of gradient information of an error loss function using an error back propagation algorithm.

[0181] A verification unit is configured to select model parameters corresponding to the minimum error in the errors corresponding to each group of training data as final model parameters to obtain a trained neural network model, and evaluate the training effect of the trained neural network model using the test data set.

[0182] Based on the above embodiments, the application further provides an intelligent terminal, a principle block diagram of which can be as shown in Figure 13 The intelligent terminal comprises a processor, a memory, a network interface, a display screen and a temperature sensor connected through a system bus. The processor of the intelligent terminal is configured to provide calculation and control capabilities. The memory of the intelligent terminal comprises a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operating system and the computer program in the non-volatile storage medium to run. The network interface of the intelligent terminal is configured to communicate with external terminals through network connection. The computer program is executed by the processor to implement a robot indoor positioning method based on sound-light data fusion. The display screen of the intelligent terminal can be a liquid crystal display screen or an electronic ink display screen. The temperature sensor of the intelligent terminal is pre-set in the intelligent terminal and is configured to detect the running temperature of internal devices.

[0183] Those skilled in the art can understand that, Figure 13The principle block diagram shown in the figure is only a block diagram of part of the structure related to the scheme of the present application, and does not constitute a limitation on the intelligent terminal to which the scheme of the present application is applied. The specific intelligent terminal can include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0184] In one embodiment, an intelligent terminal is provided, which comprises a memory, a processor, and a robot indoor positioning program based on sound-light data fusion stored in the memory and executable on the processor. When the processor executes the robot indoor positioning program based on sound-light data fusion, the following operation instructions are implemented:

[0185] obtaining a predicted coordinate sequence; the predicted coordinate sequence comprises a light source positioning predicted coordinate sequence and a sound source positioning predicted coordinate sequence;

[0186] obtaining a training data set and a test data set according to the predicted coordinate sequence and a real coordinate sequence;

[0187] constructing a neural network model;

[0188] training the neural network model through the training data set to obtain a trained neural network model;

[0189] inputting newly collected light source positioning data and sound source positioning data into the trained neural network model to obtain a coordinate prediction value after sound-light data fusion;

[0190] performing Kalman filtering on the coordinate prediction value after sound-light data fusion to obtain a final coordinate prediction value.

[0191] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program. The computer program can be stored in a non-volatile computer readable storage medium, and when the computer program is executed, the processes of the above-mentioned embodiments of the methods can be included. Any reference to memory, storage, operating database or other medium used in the embodiments provided by the present application can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. As an illustration but not limitation, RAM is available in various forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDR SDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM), etc.

[0192] In summary, the present application discloses a robot indoor positioning method based on acoustic-optical data fusion, which comprises the following steps: obtaining a predicted coordinate sequence; obtaining a training data set and a test data set according to the predicted coordinate sequence and a real coordinate sequence; constructing a neural network model; training the neural network model through the training data set to obtain a trained neural network model; inputting newly collected light source positioning data and sound source positioning data into the trained neural network model to obtain a coordinate prediction value after acoustic-optical data fusion; and performing Kalman filtering on the coordinate prediction value after acoustic-optical data fusion to obtain a final coordinate prediction value. The present application takes into account the case that indoor light is blocked, uses a sound source positioning method based on time delay difference as compensation, and realizes the fusion of acoustic and optical methods by using a BP neural network model, thereby realizing high-precision indoor power inspection robot positioning while ensuring real-time positioning.

[0193] Finally, it should be noted that: the above embodiments are only used to illustrate the technical solutions of the present application, and not to limit them; although the present application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that: it can still modify the technical solutions recorded in the foregoing embodiments, or make equivalent replacement for part of the technical features; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present application.

Claims

1. A robot indoor positioning method based on acoustic-optical data fusion, characterized in that, The method comprises: obtaining a predicted coordinate sequence; the predicted coordinate sequence comprises a light source positioning predicted coordinate sequence and a sound source positioning predicted coordinate sequence; obtaining a training data set and a test data set according to the predicted coordinate sequence and a real coordinate sequence; constructing a neural network model; training the neural network model through the training data set to obtain a trained neural network model; inputting collected light source positioning data and sound source positioning data into the trained neural network model to obtain a coordinate prediction value after sound and light data fusion; performing Kalman filtering on the coordinate prediction value after sound and light data fusion to obtain a final coordinate prediction value; Before collecting the light source positioning data, the method further comprises: establishing a sound and light data fusion positioning model; wherein the sound and light data fusion positioning model comprises a sound source detection device, a white light LED light source, a fixed frequency sound source, a patrol robot, and a photodetector arranged at the top center position of the patrol robot; a microphone array is arranged on the upper part of the sound source detection device; the white light LED light source is used to emit white light signals in turn; the photodetector is used to obtain the output voltage when the white light LED light source emits light; the fixed frequency sound source is used to emit audio signals of a fixed frequency; and the microphone array is used to obtain the audio signals; dividing an indoor space into a plurality of small areas, collecting the center position coordinates of each small area as a real coordinate sequence, and aligning the center position of the patrol robot with the center of each small area in turn to collect a predicted coordinate sequence; collecting the predicted coordinate sequence comprises: in each small area, obtaining the output voltage when a single white light LED light source emits light; obtaining the light power of each white light LED light source according to the output voltage when the single white light LED light source emits light; obtaining the light power ratio between LED light sources according to the light power of each white light LED light source when emitting light; obtaining a receiving plane coordinate curve equation according to the light power ratio; solving the receiving plane coordinate curve equation to obtain the predicted center position coordinates of each small area; obtaining the light source positioning predicted coordinate sequence according to the predicted center position coordinates of each small area. 2.The robot indoor positioning method based on acoustic-optical data fusion according to claim 1, characterized in that, The method comprises: performing a normalization operation on the predicted coordinate sequence and the real coordinate sequence to obtain a normalized predicted coordinate sequence and a normalized real coordinate sequence; performing bilinear interpolation on the normalized predicted coordinate sequence and the normalized real coordinate sequence to obtain a plurality of training data sets; grouping the training data according to a predetermined proportion to obtain the training data set and the test data set. 3.The robot indoor positioning method based on acoustic-optical data fusion according to claim 1, wherein, The method comprises: in each small area, obtaining the sound intensity signal emitted by the fixed frequency sound source to obtain a received sound intensity signal; calculating the received sound intensity signal using the GCC-PATH algorithm to obtain a time delay estimate; calculating the time delay estimate using a sound source positioning far field model to obtain a plurality of sound source azimuth angles; the sound source azimuth angle is the plane azimuth angle of the sound source relative to each sound source detection device. According to the sound source azimuth, the position coordinates of the sound source detection device and the vertical distance between the sound source detection device and the ground, the sound source coordinate values of the sound source detection devices are obtained; The average value of the sound source coordinate values of the sound source detection devices is calculated to obtain the sound source positioning data in each small area; According to the sound source positioning data in each small area, the sound source positioning prediction coordinate sequence is obtained. 4.The robot indoor positioning method based on acoustic-optical data fusion according to claim 1, wherein, The neural network model includes an input layer, an output layer and an intermediate layer; wherein the neural network model is a fully connected network model, the input layer has four components, the output layer has two components, the intermediate layer has seven neurons, the activation function is a Sigmoid activation function, and the error function is a mean square error function. 5.The robot indoor positioning method based on the acoustic-optical data fusion according to claim 1, wherein, The training of the neural network model through the training data set includes: Initializing model parameters; wherein the initialization of model parameters includes random assignment of neurons of the input layer, the intermediate layer and the output layer of the neural network model to obtain initial model parameters; Each set of training data in the training data set is input into the input layer of the neural network model, and the initial model parameters are used to calculate and output layer by layer to obtain the prediction coordinates corresponding to each set of training data; According to the prediction coordinates and the real coordinates corresponding to each set of training data, the error corresponding to each set of training data is calculated using the mean square error function, and the gradient information of the error loss function is propagated in reverse using the error back propagation algorithm to update the neuron parameters in the neural network model; The model parameters with the minimum error in the error corresponding to each set of training data are selected as the final model parameters to obtain the trained neural network model, and the training effect of the trained neural network model is evaluated using the test data set.

6. A robot indoor positioning device based on acoustic-optical data fusion, characterized by, The device includes: A prediction coordinate sequence acquisition module is configured to acquire a prediction coordinate sequence; the prediction coordinate sequence includes a light source positioning prediction coordinate sequence and a sound source positioning prediction coordinate sequence; A data set acquisition module is configured to obtain a training data set and a test data set according to the prediction coordinate sequence and a real coordinate sequence; A model construction module is configured to construct a neural network model; A training module is configured to train the neural network model through the training data set to obtain a trained neural network model; A prediction module is configured to input the collected light source positioning data and sound source positioning data into the trained neural network model to obtain a coordinate prediction value after sound-light data fusion; Before collecting the light source positioning data, the method further includes: The sound-light data fusion positioning model comprises a sound source detection device, a white light LED light source, a fixed frequency sound source, a patrol robot, and a photoelectric detector arranged at the top center of the patrol robot; a microphone array is arranged on the sound source detection device; the white light LED light source is used to emit white light signals in turn; the photoelectric detector is used to acquire output voltages when the white light LED light source emits light; the fixed frequency sound source is used to emit audio signals of a fixed frequency; and the microphone array is used to acquire the audio signals. The indoor space is divided into a plurality of small areas, and the center position coordinates of each small area are collected as a real coordinate sequence, and the center position of the patrol robot is sequentially aligned with the center of each small area to collect a predicted coordinate sequence. The predicted coordinate sequence is collected, comprising: In each small area, the output voltage of a single white light LED light source when emitting light is acquired; According to the output voltage of the single white light LED light source when emitting light, the light power of each white light LED light source when emitting light is obtained; According to the light power of each white light LED light source when emitting light, the light power ratio between the LED light sources is obtained; According to the light power ratio, a receiving plane coordinate curve equation is obtained; The receiving plane coordinate curve equation is solved to obtain the predicted center position coordinates of each small area; According to the predicted center position coordinates of each small area, the light source positioning predicted coordinate sequence is obtained; The final coordinate prediction value acquisition module is used to perform Kalman filtering on the coordinate prediction value after sound-light data fusion to obtain a final coordinate prediction value.

7. A smart terminal, characterized by The intelligent terminal comprises a memory, a processor, and a robot indoor positioning program based on sound-light data fusion stored in the memory and executable on the processor; when the processor executes the robot indoor positioning program based on sound-light data fusion, the steps of the robot indoor positioning method based on sound-light data fusion of any one of claims 1-5 are implemented.

8. A computer-readable storage medium, characterized in that, The computer readable storage medium stores a robot indoor positioning program based on sound-light data fusion; when the processor executes the robot indoor positioning program based on sound-light data fusion, the steps of the robot indoor positioning method based on sound-light data fusion of any one of claims 1-5 are implemented.

Images