Visible light positioning method and system based on ambient light sensor and image sensor

Abstract

The invention provides a visible light positioning method and system based on an ambient light sensor and an image sensor, and belongs to the technical field of communication networks. The method comprises the following steps: acquiring an LED lamp image received by a receiving end to be positioned, and receiving light intensity information of an LED lamp by using an ambient light sensor; because the sampling rate of an environment sensor is low, according to the method, the characteristic of frequency aliasing is utilized, a transmitting signal of each LED lamp is recognized through a dynamic undersampling principle to obtain sampling data, meanwhile, the problem that the sampling data are missing is solved along with the low sampling rate, the missing data is supplemented through nearest neighbor interpolation, and the accuracy of the sampling data is improved. Taking the corrected sampling data as sampling data of an ambient light sensor end; corresponding frequency and distance information is obtained by decoding the obtained LED lamp image and the corrected light intensity information, the distance information of the image sensor is used for correcting the distance information of the ambient light sensor, and an optimal solution is iterated by adopting a trilateral positioning algorithm and a nonlinear least square method to serve as a final positioning result.

Description

Technical Field

[0001] The present invention belongs to the technical field of communication networks, and in particular relates to a visible light positioning method and system based on an ambient light sensor and an image sensor. Background Art

[0002] With the popularity of mobile devices and smart devices, the application of positioning technology in daily life has become more and more extensive. Traditional positioning methods mainly rely on the Global Positioning System (GPS), but in indoor environments, the positioning accuracy of GPS is seriously affected due to signal blocking and multipath effects. To solve this problem, academia and industry have developed a variety of indoor positioning technologies, including Wi-Fi positioning, Bluetooth positioning, and ultra-wideband (UWB) positioning. However, these technologies still face some challenges in practical applications, such as high deployment cost, limited positioning accuracy, and susceptibility to environmental interference.

[0003] In recent years, visible light positioning (VLP) technology has become a new research hotspot due to its advantages of high bandwidth, high security and immunity to electromagnetic interference. Visible light-based positioning technology uses LED light sources as positioning beacons and obtains location information through optical sensors or image sensors at the receiving end. Compared with traditional radio wave positioning technology, the VLP positioning system has the advantages of low deployment cost, high precision and high communication security.

[0004] VLP systems are mainly divided into three categories according to the different receiving ends: the first category is a positioning system based on photodiodes. This positioning system is easy to implement, but the light intensity information obtained by the photodiode is easily affected by environmental noise, which in turn affects the positioning accuracy; the second category is a positioning system based on image sensors. It has high positioning accuracy and is not easily affected by environmental noise. However, as a receiving end, the image sensor not only consumes a lot of power but also has high requirements for the field of view and resolution, which increases the cost of the positioning system and affects the practical application of the visible light positioning system; the third category is a positioning system that integrates photodiodes and image sensors. Currently, the system is positioned in a combination of modules and is difficult to put into practical application. These have caused a certain degree of negative impact on the actual application of positioning and the accuracy of positioning. Summary of the invention

[0005] The technical problem to be solved by the present invention is to provide a visible light positioning method and system based on ambient light sensor and image sensor in view of the shortcomings of the above-mentioned prior art. The method mainly uses the distance information of the ambient light sensor and the distance information of the image sensor as an auxiliary correction. At the same time, the nonlinear least squares method is used to further filter the noise, optimize the positioning results, and thus improve the positioning accuracy.

[0006] In order to solve the above technical problems, the technical solution adopted by the present invention is:

[0007] In one aspect, the present invention provides a visible light positioning method based on an ambient light sensor and an image sensor, comprising the following steps:

[0008] Step 1: Obtain the light intensity information of N LED lights received by the receiving end to be located through the ambient light sensor, that is, the frequency information required for decoding the LED lights received by the receiving end to be located and the distance information between the receiving end to be located and the LED lights received by the receiving end to be located, wherein N ≥ 4, including the following steps:

[0009] Step 1.1: Obtain the frequency information required for decoding the LED light received by the receiver to be located through the ambient light sensor;

[0010] The ambient light sensor uses the property of frequency aliasing to decode the emission signal of the LED lamp and identify the emission signal frequency. When the modulation frequency is sampled at a frequency lower than the Nyquist sampling frequency, the aliasing frequency of the LED lamp is calculated by discrete Fourier transform to obtain the emission signal frequency of the LED lamp. The specific formula for calculating the aliasing frequency is:

[0011] f a =min(f0-(A-1)f s ,Af s -f0)

[0012] Among them, f a is the aliasing frequency, f0 is the emission signal frequency of the LED lamp, f s is the sampling data frequency, A satisfies Af s -The smallest natural number of f0;

[0013] The aliased frequencies associated with a single LED lamp are provided to the LED lamp position coordinate database to identify the transmission signal frequencies of various LED lamps;

[0014] Step 1.2: Obtain the distance information between the receiving end to be located and the LED light received by the receiving end to be located through the ambient light sensor;

[0015] The missing data in the sampled data are supplemented and corrected by using the nearest neighbor interpolation. The corrected sampled data is used as the sampled data of the ambient light sensor. The sampled data of the ambient light sensor is decoded into the aliasing frequency of the corresponding LED lamp, so as to obtain the modulation frequency of the LED lamp. The coordinates of the specific LED lamp in the world coordinate system are obtained by comparing the obtained modulation frequency of the LED lamp with the LED lamp position coordinate database. The world coordinate system of the LED lamp is known, so the signal sent by the LED lamp follows the Lambert model.

[0016] The specific formula for the ambient light sensor to obtain the distance information between the receiving end to be located and the LED light received by the receiving end to be located is:

[0017]

[0018] Where Pr is the relative saturation of the spectrum amplitude value after discrete Fourier transform of the sampling data of the ambient light sensor, C is the maximum emission power of the LED light, sin(cπ) is the duty cycle of the LED light, H(f k ) is the frequency response of the ambient light sensor transmission signal, cos l θ is the irradiance angle response of the LED lamp, is the incident angle response of the ambient light sensor, d is the distance between the LED light received by the receiving end to be located and the receiving end to be located, l and m are the Lambertian coefficients;

[0019] Step 2: Obtaining the frequency information required for decoding the LED light image received by the receiving end to be located and the distance information between the receiving end to be located and the LED light received by the receiving end to be located through the image sensor, specifically including the following steps:

[0020] Step 2.1: Obtain the frequency information required for decoding the LED light image received by the receiving end to be located through the image sensor;

[0021] An image sensor is used to capture an LED light image with light and dark stripes. The captured LED light image with light and dark stripes is extracted and grayscaled and binarized. That is, the LED light image is decoded to obtain the frequency information required for image decoding. The specific formula is:

[0022] [(1 / Y)*(1 / T)+x]*M=1 / 2f

[0023] Wherein, Y is the number of pixel rows of the LED light image, T is the exposure time of the camera used by the image sensor, M is the number of rows exposed by the camera used by the image sensor, x is the time required for the camera used by the image sensor to read data, and f is the transmission frequency of the LED light emission signal;

[0024] Step 2.2: Using a three-light positioning algorithm, the distance information between the receiving end to be positioned and the LED light received by the receiving end to be positioned is obtained through an image sensor;

[0025] Establish two three-dimensional coordinate systems, namely the world coordinate system and the camera coordinate system, and a two-dimensional coordinate system, namely the pixel coordinate system; perform image fitting on the LED light image captured by the image sensor to obtain the camera coordinates (a, b) of the origin of the LED light in the image sensor, and then obtain the coordinates (a1, b1) of the origin of the camera coordinate system in the pixel coordinate system according to the pixel size of the image. In the pixel coordinate system, the two-dimensional distance information between the LED light and the receiving end to be located is as follows:

[0026]

[0027] Among them, d u is the distance between the LED light and the receiving end to be located in the pixel coordinate system;

[0028] In the camera coordinate system, the optical center (a, b) of the camera lens is taken as the coordinate origin. Then, in the camera coordinate system, the coordinates of the light points I1, I2, and I3 in the LED light image captured by the image sensor are (i1 c ,p1 c ,-r),(i2 c ,p2 c ,-r),(i3 c ,p3 c ,-r),i i c ,p i c are the X-axis and Y-axis coordinate points of the LED light captured by the image sensor in the camera coordinate system, where i = 1, 2, 3, and r is the focal length of the camera used by the image sensor;

[0029] The vertical distance h between the LED light and the receiving end to be located is known. The specific formula for the three-dimensional distance between the two in the world coordinate system is:

[0030]

[0031] Among them, d w is the three-dimensional distance between the LED light and the receiving end to be located, h is the vertical distance between the receiving end and the LED light, d u is the two-dimensional distance between the LED light and the receiving end to be located, k i is the scale factor, which is a positive number;

[0032] Step 3: After obtaining the frequency information and distance information required for decoding the LED light image received by the receiving end to be located, the distance correction algorithm is used to correct the distance information between the receiving end to be located and the LED light received by the receiving end to be located obtained by the ambient light sensor, and the corrected distance information is used to perform three-sided positioning of the receiving end to be located, and the nonlinear least squares method is used for iterative optimization, and the final optimized position result is used as the final combined positioning result, including the following steps:

[0033] Step 3.1: Use the distance information obtained by the image sensor to correct the noisy distance information obtained by the ambient light sensor. After obtaining the corrected distance information, use the nonlinear least squares method to further reduce the noise of the corrected distance information. Finally, use the three-sided positioning algorithm to complete the positioning of the receiving end to be positioned.

[0034] When correcting the distance information, an image sensor at a certain location can only capture an image with W LED lights due to field of view limitations, where W ≥ 1. The distance d between the LED light and the image sensor is obtained through the image sensor distance algorithm. c,i , and then the coordinates of the corresponding LED light are obtained according to the emission frequency of the LED light; at the same time, the ambient light sensor receives the light intensity information of the LED light, and the ambient light sensor obtains the noisy distance information between the LED light and the ambient light sensor through the Lambertian model, and also obtains the three-dimensional distance d between the LED light and the ambient light sensor corresponding to the image sensor according to the emission frequency of the LED light. p,i , and then the distance information error caused by environmental noise is obtained, as shown in the following formula:

[0035] Δd i =d c,i -d p,i

[0036] Where, Δd i is the distance information error caused by environmental noise, i≥4, d c,i is the distance information between the i-th LED light and the image sensor at the receiving end to be located, d p,i is the distance information between the i-th LED light and the ambient light sensor at the receiving end to be located;

[0037] However, due to the limitation of the field of view, the image sensor can only capture one LED light, so it can only correct the noise effect of one LED light. When correcting the noise effect of the remaining LED lights, the same LED light is measured at different positions to obtain multiple noisy distance information d of the ambient light sensor. nSince the distance between the position of each receiver to be located and the LED light is known, there is a difference Δd between the noise-containing distance and the actual distance caused by noise at different positions of the same LED light. By fitting Δd and d n get Among them, a and b are unknown coefficients;

[0038] According to the distance information corrected by the image sensor and the multiple noisy distance information obtained by the ambient light sensor, the distance information error between the remaining uncorrected LED lights and the receiving end to be located is obtained. The specific formula is:

[0039]

[0040] Dividing the two equations, we get:

[0041]

[0042] Where, Δd j is the distance information error caused by environmental noise of the uncorrected LED light, j≥3, Δd e,i The distance error caused by the ambient noise of the LED light corrected for the image sensor, d n,j is the noisy distance information of the uncorrected LED light, d n,i is the noisy distance information of the LED light corrected by the image sensor;

[0043] Therefore, the specific formula for the corrected distance of the remaining LED lights is:

[0044]

[0045] Among them, d P,j is the corrected distance information between the ambient light sensor and the other LED lights, d c,i is the distance information between the i-th LED light and the image sensor at the receiving end to be located;

[0046] Step 3.2: Obtain the distance information d of one of the LED lights from the image sensor cam , after correction in step 3.1, we can get the distance information d of the remaining N-1 LED lights PD,2 , d PD,3 ……d PD,N Then the location information of the receiving end to be located is obtained. The specific formula is:

[0047]

[0048] Among them, (x1,y1,z1), (x2,y2,z2), (x3,y3,z3)......(x N ,y N ,zN ) is the coordinate of N LED lights in the world coordinate system, (x, y, z) is the position of the receiving end to be located, d cam To obtain the distance information of one of the LED lights by the image sensor, d PD,2 , d PD ,3……d PD,N The distance information of the remaining N-1 LED lights is obtained after correction in step 3.1;

[0049] Then, the nonlinear least square method is used to iteratively optimize the distance information corrected by the ambient light sensor to find the optimal solution for the position of the receiving end to be located;

[0050] On the other hand, the present invention also provides a visible light positioning system based on an ambient light sensor and an image sensor, comprising an ambient light sensor module, an image sensor module, a combined positioning module, an electronic device, a computer-readable storage medium, and a computer program product;

[0051] The ambient light sensor module is used to obtain the LED light intensity information of the receiving end to be located;

[0052] The image sensor module is used to obtain the LED light image information of the receiving end to be located;

[0053] The combined positioning module is used to decode and obtain the required distance information after the receiving end to be positioned obtains the light intensity information and image information of the LED lamp, and adopts the combined positioning algorithm and the nonlinear least squares optimization algorithm to obtain the optimal position result as the final positioning result;

[0054] The electronic device includes a processor and a memory, wherein the memory is used to store instructions for a visible light positioning method based on an ambient light sensor and an image sensor, and when the instructions are executed by the processor, the processor executes a visible light positioning method based on a combination of an embedded ambient light sensor and an image sensor;

[0055] The readable storage medium stores executable instructions for a visible light positioning method based on an ambient light sensor and an image sensor, and when the instructions are executed, the processor executes the visible light positioning method based on the ambient light sensor and the image sensor;

[0056] The computer program product includes a computer program or instructions, which, when executed by a processor, implements a visible light positioning method based on an ambient light sensor and an image sensor.

[0057] The beneficial effect of adopting the above technical scheme is that: the present invention proposes a visible light positioning method and system based on ambient light sensor and image sensor, which uses the distance information of image sensor and ambient light sensor for combined positioning, uses the distance information of image sensor to correct the distance information of ambient light sensor, and uses nonlinear least squares method for iterative optimization to filter the noise in the positioning process. The positioning method can achieve sub-meter positioning error; at the same time, the nearest neighbor interpolation method is adopted, and the piecewise constant function method is used to supplement the missing sampling value with the average value of the light intensity information before and after the missing, as the missing light intensity information supplement, and the overall update is used as the final light intensity information to decode the required light intensity information. BRIEF DESCRIPTION OF THE DRAWINGS

[0058] Figure 1 A flowchart of a visible light positioning method based on an ambient light sensor and an image sensor provided in an embodiment of the present invention;

[0059] Figure 2 A positioning principle diagram of an indoor visible light positioning system based on an ambient light sensor and an image sensor provided by an embodiment of the present invention;

[0060] Figure 3 The positioning error result diagram of the test under different environments provided by the embodiment of the present invention;

[0061] Figure 4 A cumulative distribution function curve diagram of positioning error under different environments provided by an embodiment of the present invention;

[0062] Figure 5 A schematic diagram of a principle block diagram of a visible light positioning system based on an ambient light sensor and an image sensor provided in an embodiment of the present invention;

[0063] Figure 6 A diagram showing the effect of nearest neighbor interpolation decoding provided by an embodiment of the present invention;

[0064] Figure 7 A diagram showing the effect of cubic spline interpolation decoding provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0065] The specific implementation of the present invention is further described in detail below in conjunction with the accompanying drawings and examples. The following examples are used to illustrate the present invention, but are not intended to limit the scope of the present invention.

[0066] In this embodiment, a visible light positioning method based on an ambient light sensor and an image sensor is provided. Figure 1As shown, an Android smartphone is used as the receiving end to be positioned, and the positive directions of the x-axis, y-axis and z-axis of the world coordinate system are defined according to the information provided by the built-in ambient light sensor and image sensor of the mobile phone, and the world coordinate position of the LED light received by the receiving end to be positioned is determined. In this embodiment, the mobile phone is placed horizontally on a bracket to collect information for indoor positioning, including the following steps:

[0067] Step 1: Obtain light intensity information of N LED lights received by the receiving end to be located through the ambient light sensor, that is, the frequency information required for decoding the LED lights received by the receiving end to be located and the distance information between the receiving end to be located and the LED lights received by the receiving end to be located, where N ≥ 4;

[0068] Due to the low sampling rate of the ambient light sensor, the ambient light sensor uses the nature of frequency aliasing to identify the emission signal of each LED lamp through the dynamic undersampling principle to obtain sampling data. At the same time, due to the low sampling rate of the ambient light sensor, the problem of missing sampling data is also accompanied. The nearest neighbor interpolation is used to supplement and correct the missing data. The corrected sampling data is used as the sampling data of the ambient light sensor. Discrete Fourier Transform (DFT) is used for the sampling data of the ambient light sensor to obtain the light intensity information of the LED lamp. Finally, according to the Lambert model, the distance information between the receiving end to be located and the LED lamp received by the receiving end to be located is obtained, including the following steps:

[0069] Step 1.1: Obtain the frequency information required for decoding the LED light received by the receiver to be located through the ambient light sensor;

[0070] Due to the inherent system limitations of smartphones, the sampling frequency of ambient light sensors is extremely low. In order to prevent people from feeling light flickering, the modulation frequency of LED lights is usually higher than 70HZ, which cannot meet the Nyquist sampling frequency. The ambient light sensor uses the nature of frequency aliasing to decode the emission signal of the LED light and identify the emission signal frequency. When the modulation frequency is lower than the Nyquist sampling frequency, the aliasing frequency of the LED light is calculated by discrete Fourier transform, thereby obtaining the emission signal frequency of the LED light. The calculation formula for the aliasing frequency is as follows:

[0071] f a =min(f0-(A-1)f s ,Af s -f0)

[0072] Among them, f a is the aliasing frequency, f0 is the emission signal frequency of the LED lamp, and f s is the sampling data frequency, A satisfies Af s-The smallest natural number of f0;

[0073] The aliasing frequency associated with a single LED light is provided to the LED light position coordinate database to identify the transmission signal frequencies of various LED lights; the relationship between the transmission signal frequency and the aliasing frequency can be established through a formula, and the aliasing frequency associated with the LED light is uploaded to the visible light positioning system based on the ambient light sensor and the image sensor in advance, so that the signal frequency sent by the LED light can be identified;

[0074] Step 1.2: Obtain the distance information between the receiving end to be located and the LED light received by the receiving end to be located through the ambient light sensor;

[0075] The missing data in the sampled data is supplemented and corrected by using the nearest neighbor interpolation. The corrected sampled data is used as the ambient light sensor sampled data. The ambient light sensor sampled data is obtained by decoding the obtained modulation frequency of the LED light into the aliasing frequency of the corresponding LED light, so as to obtain the modulation frequency of the LED light. The specific coordinates of the LED light in the world coordinate system are obtained by comparing the obtained modulation frequency of the LED light with the LED light position coordinate database. When the visible light positioning system based on the ambient light sensor and the image sensor starts to run, the ambient light sensor senses the change in light intensity and obtains the light intensity information. However, sometimes the ambient light sensor cannot sense the change in light intensity, resulting in incomplete collected data. Therefore, the missing data is supplemented by using the nearest neighbor interpolation. The corrected sampled data is used as the ambient light sensor sampled data. By obtaining the corrected light intensity information, the aliasing frequency of the corresponding LED light can be decoded. The specific coordinates of the LED light in the world coordinate system are obtained by comparing with the database. The world coordinate system of the LED light is known, so the signal sent by the LED light follows the Lambert model.

[0076] The specific formula for the ambient light sensor to obtain the distance information between the receiving end to be located and the LED light received by the receiving end to be located is:

[0077]

[0078] Where Pr is the relative saturation of the spectrum amplitude value after discrete Fourier transform of the sampling data of the ambient light sensor, C is the maximum emission power of the LED light, sin(cπ) is the duty cycle of the LED light, H(f k ) is the frequency response of the ambient light sensor transmission signal, cos l θ is the irradiance angle response of the LED lamp, is the incident angle response of the ambient light sensor, d is the distance between the corresponding LED light received and the receiving end to be located, l and m are the Lambertian coefficients;

[0079] Step 2: Obtain the frequency information required for decoding the LED light image received by the receiving end to be located and the distance information between the receiving end to be located and the LED light received by the receiving end to be located through the image sensor. In this embodiment, it is assumed that all LED lights are facing downwards and the mobile phone is parallel to the LED lights. Then, the coordinates of the i-th LED light and the coordinates of the smart phone are respectively (x i ,y i ,z i ) and (x,y,z), where and Then, according to the above formula, the distance information between the smartphone and the LED light can be obtained, which specifically includes the following steps:

[0080] Step 2.1: Obtain the frequency information required for decoding the LED light image received by the receiving end to be located through the image sensor;

[0081] Considering the influence of environmental noise, in this embodiment, the exposure value of the image sensor is set to 1 / 4000, and the sensitivity is set to 400. The image sensor obtains the frequency information required for decoding the LED light image received by the receiving end to be located, that is, the image sensor is used to capture the LED light image with light and dark stripes, and the captured LED light image with light and dark stripes is extracted and gray-scaled and binarized, that is, the LED light image is decoded to obtain the frequency information required for decoding. The specific formula is:

[0082] [(1 / Y)*(1 / T)+x]*M=1 / 2f

[0083] Wherein, Y is the number of pixel rows of the LED light image, T is the exposure time of the camera used by the image sensor, M is the number of rows exposed by the camera used by the image sensor, x is the time required for the camera used by the image sensor to read data, and f is the transmission frequency of the LED light emission signal;

[0084] Step 2.2: Using a three-light positioning algorithm, the distance information between the receiving end to be positioned and the LED light received by the receiving end to be positioned is obtained through an image sensor;

[0085] In this embodiment, the LED light transmits its three-dimensional information according to the rolling shutter effect of the camera. The camera is used to take a picture with the LED light, thereby capturing the coordinates of the image points of the LED light in the image. Because the LED light has a lampshade, the coordinates of the LED light cannot be directly obtained. In this embodiment, the center of the lampshade is used as the pixel coordinates of the LED light; two three-dimensional coordinate systems are established, namely the world coordinate system (WCS) and the camera coordinate system (CCS), and a two-dimensional coordinate system, namely the pixel coordinate system (PCS), and the image of the LED light in the image sensor is fitted to obtain the camera coordinates (a, b) of the origin of the LED light in the image sensor, and then the coordinates (a1, b1) of the origin of the image coordinate system in the pixel coordinate system are obtained according to the pixel size of the image. In the pixel coordinate system, the two-dimensional distance information between the LED light and the receiving end to be located is as follows:

[0086]

[0087] Among them, d u is the distance between the LED light and the receiving end to be located in the pixel coordinate system;

[0088] Set the coordinates of LED lights L1, L2, and L3 in the world coordinate system to be (x w 1,y w 1,z w 1), (x w 2,y w 2,z w 2), (x w 3,y w 3,z w 3), in the camera coordinate system, the optical center (a, b) of the camera lens is taken as the coordinate origin, m c =(0,0,0), then in the camera coordinate system, the coordinates of light points I1, I2, and I3 in the LED image captured by the image sensor are (i1 c ,p1 c ,-r),(i2 c ,p2 c ,-r),(i3 c ,p3 c ,-r),i i c ,p i c are the X-axis and Y-axis coordinate points of the LED light captured by the image sensor in the camera coordinate system, where i = 1, 2, 3, and r is the focal length of the camera used by the image sensor;

[0089] The vertical distance h between the LED light and the receiving end to be located is known. The specific formula for the three-dimensional distance between the two in the world coordinate system is:

[0090]

[0091] Among them, d w is the three-dimensional distance between the LED light and the receiving end to be located, h is the vertical distance between the receiving end and the LED light, d u is the two-dimensional distance between the LED light and the receiving end to be located, k i is a positive proportional factor;

[0092] Finally, in the traditional camera-based positioning system, the positioning can be completed according to the above relationship combined with the coordinates of the three LED lights and the direction angles. However, due to the field of view limitation of the camera and the need for three LED lights to complete the positioning, it is difficult to meet the requirements. Therefore, this embodiment uses the distance information between the LED light and the receiving end to be positioned provided by the camera as an auxiliary, combined with the distance information of the ambient light sensor for combined positioning;

[0093] In this embodiment, the coordinates of the LED lights are known in the world coordinate system, so the distance between any two LED lights can be obtained, and the k value can be obtained by combining the distance relationship in the camera coordinate system. In this embodiment, the receiving end to be positioned is in a horizontal state;

[0094] Step 3: After obtaining the frequency information and distance information required for decoding the LED light image received by the receiving end to be located, the distance correction algorithm is used to correct the distance information between the receiving end to be located and the LED light received by the receiving end to be located obtained by the ambient light sensor, and the corrected distance information is used to perform three-sided positioning of the receiving end to be located, and the nonlinear least squares method is used for iterative optimization, and the final optimized position result is used as the final combined positioning result, including the following steps:

[0095] Step 3.1: This embodiment uses the ambient light sensor and image sensor camera embedded in the smart phone as the receiving end to be positioned for positioning. Due to the interference of environmental noise, it is believed that the distance information provided by the image sensor camera is the distance information with strong anti-interference ability, and the ambient light sensor is easily interfered by environmental noise, so the distance information it provides is non-standard noisy distance information. Therefore, this application adopts the average height of a normal adult holding a mobile phone, so the image sensor camera can only capture one LED light due to the interference of the field of view. Therefore, only one standard distance information can be provided at the image sensor camera end, and the ambient light sensor can provide four noisy distance information. The distance information obtained by the image sensor is used to correct the noisy distance information obtained by the ambient light sensor. After obtaining the corrected distance information, the nonlinear least squares method is used to further reduce the noise of the corrected distance information. Finally, the three-sided positioning algorithm is used to complete the positioning of the receiving end to be positioned;

[0096] In this embodiment, when the distance information is corrected, the ambient light sensor at a certain location receives the light intensity information of the LED light, and the image sensor captures an image with W LED lights, where W ≥ 1, and the distance d between the LED light and the image sensor is obtained by the image sensor distance algorithm. c,i Then, the coordinates of the corresponding LED light are obtained according to the emission frequency of the LED light. At the same time, the ambient light sensor obtains the noisy distance information between the LED light and the ambient light sensor through the Lambertian model, and also obtains the distance d between the LED light and the ambient light sensor corresponding to the image sensor according to the emission frequency of the LED light. p,i , and then the distance information error caused by environmental noise is obtained, as shown in the following formula:

[0097] Δd i =d c,i -d p,i

[0098] Where, Δd i is the distance information error caused by environmental noise, i≥4, d c,i is the distance information between the i-th LED light and the image sensor at the receiving end to be located, d p,i is the distance information between the i-th LED light and the ambient light sensor at the receiving end to be located;

[0099] Then the camera can only capture one LED light at a time, so it can only correct the noise impact of one LED light. After correcting the noise impact of the remaining LED lights, this embodiment measures the same LED light at different positions to obtain multiple noise distance information d of the ambient light sensor. n Since the distance between the position of each receiver to be located and the LED light is known, the difference Δd between the noise-induced distance and the actual distance at different positions of the same LED light is calculated by fitting Δd and d n get Where a and b are unknown coefficients, Δd is the difference between the noise-containing distance extracted by the ambient light sensor and the actual distance, and d n is the noisy distance obtained by the ambient light sensor;

[0100] According to the distance information corrected by the image sensor and the multiple noisy distance information obtained by the ambient light sensor, the distance information error between the remaining uncorrected LED lights and the receiving end to be located is obtained. The specific formula is:

[0101]

[0102] Dividing the two equations, we get:

[0103]

[0104] Where, Δd j is the distance information error caused by environmental noise of the uncorrected LED light, j≥4, Δd e,i The distance error caused by the ambient noise of the LED light corrected for the image sensor, d n,j is the noisy distance information of the uncorrected LED light, d n,i is the noisy distance information of the LED light corrected by the image sensor;

[0105] In this embodiment, when b is equal to 3.2, the average positioning error is the smallest through fitting, so the specific formula for the corrected distance of the remaining LED lights is:

[0106]

[0107] Among them, d P,j is the corrected distance information between the ambient light sensor and the other LED lights, d c,i is the distance information between the i-th LED light and the image sensor at the receiving end to be located;

[0108] Step 3.2: Obtain the distance information d of one of the LED lights from the image sensor cam , after correction in step 3.1, we can get the distance information d of the remaining N-1 LED lights PD,2 , d PD,3 ……d PD,N Then the location information of the receiving end to be located is obtained. The specific formula is:

[0109]

[0110] Among them, (x1,y1,z1), (x2,y2,z2), (x3,y3,z3)......(x N ,y N ,z N ) is the coordinate of N LED lights in the world coordinate system, (x, y, z) is the position of the receiving end to be located, d cam To obtain the distance information of one of the LED lights by the image sensor, d PD,2 , d PD,3 ……d PD,N The distance information of the remaining N-1 LED lights is obtained after correction in step 3.1;

[0111] Then, due to the influence of noise, the nonlinear least square method is used to iteratively optimize the distance information corrected by the ambient light sensor to solve the optimal solution for the position of the receiving end to be located;

[0112] The schematic diagram of the indoor visible light positioning system based on the ambient light sensor and the image sensor in this embodiment is as follows: Figure 2As shown; a 2m×2m×2m positioning frame is built, and the LED lights can be flexibly arranged using the frame. In this embodiment, four LED lights are arranged in a 2×2 form on the top of the frame as the transmitting end. The system uses a Sony xperiaXZ1 smartphone as the receiving end;

[0113] In this embodiment, the transmitting end is divided into a signal transmitting part and a signal generating part; the signal transmitting part uses a 30W CXA1820 LED lamp as a light source, which can meet daily lighting needs, and the diameter of each LED lampshade is 19 cm; the signal generating part uses a GPS-430 power supply as a system power supply device, and uses an STM32F407 development board to modulate the transmitted signal, and allocates different frequencies to different LED lights;

[0114] The receiving end of this embodiment uses a Sony Xperia XZ1 smartphone with an Android operating system and captures images through a rear camera with a pixel of 1920x1080. In order to capture clear light and dark stripe images, it is necessary to set reasonable camera imaging parameters, set the image sensor exposure value to 1 / 4000, and set the sensitivity to 400. At the same time, the ambient light sensor in the mobile phone is called through the self-written Android software to obtain the light intensity information of the LED light at the position to be located;

[0115] This embodiment tests the positioning error results in different environments, such as Figure 3 As shown; in the fixed-point test, this embodiment selects 30 positions as test points to fix the receiving end, and introduces environmental noise in a dark room, and tests are performed in three scenarios after distance correction; a laser rangefinder is used to measure the real-world coordinates of the smartphone, and the measured coordinates are compared with the positioning coordinates calculated by the positioning system of this embodiment to obtain the system positioning error (accuracy). The average positioning error of the positioning system of this embodiment under dark room conditions is 11.15 cm, and when the environmental noise is introduced, the average positioning error of this embodiment becomes 23.48 cm. It can be found that the positioning error is significantly increased, which proves that the environmental noise has a great influence on the visible light positioning system. After correction, the average positioning error is 16.04 cm;

[0116] The cumulative distribution function curve of the positioning error in this embodiment is as follows: Figure 4 As shown, this embodiment draws the CDF (Cumulative Distribution Function) curve of system accuracy under different environments based on the errors of the above multiple groups of experimental data. Figure 4 It can be seen that there is obvious optimization;

[0117] In this embodiment, a visible light positioning system based on an ambient light sensor and an image sensor is provided. Figure 5 As shown, it includes an ambient light sensor module, an image sensor module, a combined positioning module, an electronic device, a computer-readable storage medium, and a computer program product;

[0118] The ambient light sensor module is used to obtain the LED light intensity information of the receiving end to be located;

[0119] The image sensor module is used to obtain the LED light image information of the receiving end to be located;

[0120] The combined positioning module is used to decode and obtain the required distance information after the receiving end to be positioned obtains the light intensity information and image information of the LED lamp, and adopts the combined positioning algorithm and the nonlinear least squares optimization algorithm to obtain the optimal position result as the final positioning result;

[0121] The nearest neighbor interpolation decoding effect diagram of this embodiment is as follows: Figure 6 As shown; the cubic spline interpolation decoding effect diagram of this embodiment, as shown Figure 7 As shown; since the change in light intensity received by the ambient light sensor is smaller than the resolution of the ambient light sensor, the ambient light sensor cannot detect the change in light intensity, resulting in missing data. Therefore, it is necessary to reasonably supplement the missing sample values ​​to improve the positioning accuracy. Therefore, this embodiment provides two methods for supplementing missing sample values:

[0122] (1) Nearest neighbor interpolation: Use the piecewise constant function method to supplement the missing sample value with the average value of the light intensity before and after the missing value;

[0123] (2) Cubic spline interpolation: Cubic spline interpolation can be used to fit the measured scattered data to a smooth continuous function curve to supplement the missing sample values. The basic idea is to use a low-order polynomial to approximate the data of a small interval, and decode the corrected data to obtain the correct aliasing frequency.

[0124] This embodiment receives a signal with a transmission frequency of 100 Hz and performs two signal correction experiments. It is found that both signal correction methods can correctly decode the aliasing frequency, but the cubic spline interpolation correction is unstable, causing the quantization error to become larger, and the nearest neighbor interpolation can more accurately correct the missing sample data. Therefore, this embodiment decides to use the nearest neighbor interpolation correction method;

[0125] The electronic device includes a processor and a memory, wherein the memory is used to store instructions for a visible light positioning method based on an ambient light sensor and an image sensor, and when the instructions are executed by the processor, the processor executes a visible light positioning method based on a combination of an embedded ambient light sensor and an image sensor;

[0126] The electronic device in this embodiment may be a mobile phone, a computer or a tablet computer, etc., including a memory and a processor;

[0127] The memory stores a computer program, which, when executed by the processor, implements the visible light positioning method based on the combination of the embedded ambient light sensor and the image sensor as described in the embodiment. It can be understood that the electronic device may also include an input / output (I / O) interface and a communication component;

[0128] The processor is used to execute all or part of the steps in the visible light positioning method based on the combination of the embedded ambient light sensor and the image sensor as described in the above embodiment, and the memory is used to store various types of data, which may include, for example, instructions of any application or method in the electronic device, and data related to the application; the processor may be an application specific integrated circuit (Application Specific Integrated Cricuit, referred to as ASIC), a digital signal processor (Digital Signal Processor, referred to as DSP), a programmable logic device (Programmable Logic Device, referred to as PLD), a field programmable gate array (Field Programmable Gate Array, referred to as FPGA), a controller, a microcontroller, a microprocessor or other electronic components, and is used to execute the visible light positioning method based on the combination of the embedded ambient light sensor and the image sensor described in the above embodiment;

[0129] The readable storage medium stores executable instructions for a visible light positioning method based on an ambient light sensor and an image sensor, and when the instructions are executed, the processor executes the visible light positioning method based on the ambient light sensor and the image sensor;

[0130] This embodiment provides a computer-readable storage medium storing executable instructions;

[0131] When the instructions are executed, if they are implemented in the form of software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium; the computer software product is stored in a storage medium, including several instructions for enabling a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the visible light positioning method based on the ambient light sensor and the image sensor described in each embodiment of the present application; and the aforementioned storage medium includes: flash memory, hard disk, multimedia card, card-type memory (for example, SD (Secure Digital Memory Card) or DX (Memory Data Register, MDR abbreviation, memory data register) memory, etc.), random access memory (RAM), static random access memory (SRAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), programmable read-only memory (PROM), magnetic memory, disk, optical disk, server, APP (Application, abbreviation of application software) application mall and other media that can store program check codes, on which a computer program is stored, and when the computer program is executed by the processor, the various steps of the visible light positioning method based on the self-correction of sensor dynamic error described above can be implemented;

[0132] The computer program product includes a computer program or instructions, which, when executed by a processor, implements a visible light positioning method based on an ambient light sensor and an image sensor.

[0133] This embodiment provides a computer program product, including a computer program or instructions, which, when executed by a processor, implements the visible light positioning method based on a combination of an embedded ambient light sensor and an image sensor.

[0134] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some or all of the technical features therein. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the scope defined by the claims of the present invention.

Claims

1. A visible light positioning method based on an ambient light sensor and an image sensor, characterized in that: The following steps are involved: Step 1: Obtain light intensity information of N LED lights received by the receiving end to be located through the ambient light sensor, that is, the frequency information required for decoding the LED lights received by the receiving end to be located and the distance information between the receiving end to be located and the LED lights received by the receiving end to be located, where N ≥ 4; Step 2: Obtain the frequency information required for decoding the LED light image received by the receiving end to be located and the distance information between the receiving end to be located and the LED light received by the receiving end to be located through the image sensor; Step 3: After obtaining the frequency information and distance information required for decoding the LED light image received by the receiving end to be located, use the distance correction algorithm to correct the distance information between the receiving end to be located and the LED light received by the receiving end to be located obtained by the ambient light sensor, use the corrected distance information to perform three-sided positioning of the receiving end to be located, and use the nonlinear least squares method for iterative optimization, and use the final optimized position result as the final combined positioning result.

2. The visible light positioning method based on an ambient light sensor and an image sensor according to claim 1, characterized in that: The specific method of step 1 is: Step 1.1: Obtain the frequency information required for decoding the LED light received by the receiver to be located through the ambient light sensor; The ambient light sensor uses the property of frequency aliasing to decode the emission signal of the LED lamp and identify the emission signal frequency. When the modulation frequency is sampled at a frequency lower than the Nyquist sampling frequency, the aliasing frequency of the LED lamp is calculated by discrete Fourier transform to obtain the emission signal frequency of the LED lamp. The specific formula for calculating the aliasing frequency is: f a =min(f0-(A-1)f s ,Of s -f0) Among them, f a is the aliasing frequency, f0 is the emission signal frequency of the LED lamp, f s is the sampling data frequency, A satisfies Af s -The smallest natural number of f0; The aliased frequencies associated with individual LED lights are provided to the LED light position coordinate database to identify the frequencies of transmitted signals from various LED lights; Step 1.2: Obtain the distance information between the receiving end to be located and the LED light received by the receiving end to be located through the ambient light sensor; The missing data in the sampled data are supplemented and corrected by using the nearest neighbor interpolation. The corrected sampled data is used as the sampled data of the ambient light sensor. The sampled data of the ambient light sensor is decoded into the aliasing frequency of the corresponding LED lamp, so as to obtain the modulation frequency of the LED lamp. The coordinates of the specific LED lamp in the world coordinate system are obtained by comparing the obtained modulation frequency of the LED lamp with the LED lamp position coordinate database. The world coordinate system of the LED lamp is known, so the signal sent by the LED lamp follows the Lambert model. The specific formula for the ambient light sensor to obtain the distance information between the receiving end to be located and the LED light received by the receiving end to be located is: Where Pr is the relative saturation of the spectrum amplitude value after discrete Fourier transform of the sampling data of the ambient light sensor, C is the maximum emission power of the LED light, sin(cπ) is the duty cycle of the LED light, H(f k ) is the frequency response of the ambient light sensor transmission signal, cos l θ is the irradiance angle response of the LED lamp, is the incident angle response of the ambient light sensor, d is the distance between the LED light received by the receiving end to be located and the receiving end to be located, and l and m are the Lambert coefficients.

3. The visible light positioning method based on an ambient light sensor and an image sensor according to claim 2, characterized in that: The specific method of step 2 is: Step 2.1: Obtain the frequency information required for decoding the LED light image received by the receiving end to be located through the image sensor; An image sensor is used to capture an LED light image with light and dark stripes. The captured LED light image with light and dark stripes is extracted and grayscaled and binarized. That is, the LED light image is decoded to obtain the frequency information required for image decoding. The specific formula is: [(1 / Y)*(1 / T)+x]*M=1 / 2f Wherein, Y is the number of pixel rows of the LED light image, T is the exposure time of the camera used by the image sensor, M is the number of rows exposed by the camera used by the image sensor, x is the time required for the camera used by the image sensor to read data, and f is the transmission frequency of the LED light emission signal; Step 2.2: Using a three-light positioning algorithm, the distance information between the receiving end to be positioned and the LED light received by the receiving end to be positioned is obtained through an image sensor; Establish two three-dimensional coordinate systems, namely the world coordinate system and the camera coordinate system, and a two-dimensional coordinate system, namely the pixel coordinate system; perform image fitting on the LED light image captured by the image sensor to obtain the camera coordinates (a, b) of the origin of the LED light in the image sensor, and then obtain the coordinates (a1, b1) of the origin of the camera coordinate system in the pixel coordinate system according to the pixel size of the image. In the pixel coordinate system, the two-dimensional distance information between the LED light and the receiving end to be located is as follows: Among them, d u is the distance between the LED light and the receiving end to be located in the pixel coordinate system; In the camera coordinate system, the optical center (a, b) of the camera lens is taken as the coordinate origin. Then, in the camera coordinate system, the coordinates of light points I1, I2, and I3 in the LED light image captured by the image sensor are are the X-axis and Y-axis coordinate points of the LED light captured by the image sensor in the camera coordinate system, where i = 1, 2, 3, and r is the focal length of the camera used by the image sensor; The vertical distance h between the LED light and the receiving end to be located is known. The specific formula for the three-dimensional distance between the two in the world coordinate system is: Among them, d w is the three-dimensional distance between the LED light and the receiving end to be located, h is the vertical distance between the receiving end and the LED light, d u is the two-dimensional distance between the LED light and the receiving end to be located, k i is the scaling factor, a positive number.

4. The visible light positioning method based on an ambient light sensor and an image sensor according to claim 3, characterized in that: The specific method of step 3 is: Step 3.1: Use the distance information obtained by the image sensor to correct the noisy distance information obtained by the ambient light sensor. After obtaining the corrected distance information, use the nonlinear least squares method to further reduce the noise of the corrected distance information. Finally, use the three-sided positioning algorithm to complete the positioning of the receiving end to be positioned. When correcting the distance information, an image sensor at a certain location can only capture an image with W LED lights due to field of view limitations, where W ≥ 1. The distance d between the LED light and the image sensor is obtained through the image sensor distance algorithm. c,i , and then the coordinates of the corresponding LED light are obtained according to the emission frequency of the LED light; at the same time, the ambient light sensor receives the light intensity information of the LED light, and the ambient light sensor obtains the noisy distance information between the LED light and the ambient light sensor through the Lambertian model, and also obtains the three-dimensional distance d between the LED light and the ambient light sensor corresponding to the image sensor according to the emission frequency of the LED light. p,i , and then the distance information error caused by environmental noise is obtained, as shown in the following formula: Δd i =d c,i -d p,i Where, Δd i is the distance information error caused by environmental noise, i≥4, d c,i is the distance information between the i-th LED light and the image sensor at the receiving end to be located, d p,i is the distance information between the i-th LED light and the ambient light sensor at the receiving end to be located; However, due to the limitation of the field of view, the image sensor can only capture one LED light, so it can only correct the noise effect of one LED light. When correcting the noise effect of the remaining LED lights, the same LED light is measured at different positions to obtain multiple noisy distance information d of the ambient light sensor. n Since the distance between the position of each receiver to be located and the LED light is known, there is a difference Δd between the noise-containing distance and the actual distance caused by noise at different positions of the same LED light. By fitting Δd and d n get Among them, a and b are unknown coefficients; According to the distance information corrected by the image sensor and the multiple noisy distance information obtained by the ambient light sensor, the distance information error between the remaining uncorrected LED lights and the receiving end to be located is obtained. The specific formula is: Dividing the two equations, we get: Where, Δd j is the distance information error caused by environmental noise of the uncorrected LED light, j≥3, Δd e,i The distance error caused by the ambient noise of the LED light corrected for the image sensor, d n,j is the noisy distance information of the uncorrected LED light, d n,i is the noisy distance information of the LED light corrected by the image sensor; Therefore, the specific formula for the corrected distance of the remaining LED lights is: Among them, d P,j is the corrected distance information between the ambient light sensor and the other LED lights, d c,i is the distance information between the i-th LED light and the image sensor at the receiving end to be located; Step 3.2: Obtain the distance information d of one of the LED lights from the image sensor cam , after correction in step 3.1, we can get the distance information d of the remaining N-1 LED lights PD,2 , d PD,3 ……d PD,N Then the location information of the receiving end to be located is obtained. The specific formula is: Among them, (x1,y1,z1), (x2,y2,z2), (x3,y3,z3)......(x N ,y N ,z N ) is the coordinate of N LED lights in the world coordinate system, (x, y, z) is the position of the receiving end to be located, d cam To obtain the distance information of one of the LED lights by the image sensor, d PD,2 , d PD,3 ……d PD,N The distance information of the remaining N-1 LED lights is obtained after correction in step 3.1; Then, the nonlinear least squares method is used to iteratively optimize the distance information corrected by the ambient light sensor to obtain the optimal solution for the position of the receiving end to be located.

5. A visible light positioning system based on an ambient light sensor and an image sensor, implemented based on the visible light positioning method based on an ambient light sensor and an image sensor according to claim 1, characterized in that: Including ambient light sensor module, image sensor module, combined positioning module; The ambient light sensor module is used to obtain the LED light intensity information of the receiving end to be located; The image sensor module is used to obtain the LED light image information of the receiving end to be located; The combined positioning module is used to decode the required distance information after the receiving end to be positioned obtains the light intensity information and image information of the LED lamp, and obtains the optimal position result as the final positioning result by using the combined positioning algorithm and the nonlinear least squares optimization algorithm.

6. The visible light positioning system based on an ambient light sensor and an image sensor according to claim 5, characterized in that: The system also includes electronic equipment; The electronic device includes a processor and a memory, wherein the memory is used to store instructions of a visible light positioning method based on an ambient light sensor and an image sensor. When the instructions are executed by the processor, the processor executes a visible light positioning method based on a combination of an embedded ambient light sensor and an image sensor.

7. The visible light positioning system based on an ambient light sensor and an image sensor according to claim 5, characterized in that: The system also includes a readable storage medium; The readable storage medium stores executable instructions for a visible light positioning method based on an ambient light sensor and an image sensor, and when the instructions are executed, a processor is enabled to execute the visible light positioning method based on an ambient light sensor and an image sensor.

8. The visible light positioning system based on an ambient light sensor and an image sensor according to claim 5, characterized in that: The system also includes a computer program product; The computer program product includes a computer program or instructions, which, when executed by a processor, implements a visible light positioning method based on an ambient light sensor and an image sensor.

Images