An indoor positioning system based on infrared camera and high-precision IMU

By adopting an indoor positioning system with infrared cameras and high-precision IMU in the film and television industry, the problems of low indoor positioning accuracy and virtual scene rendering delay are solved, high-precision real-time positioning and virtual scene rendering are achieved, and the workload of the film and television industry is reduced. Labor costs.

CN114777772BActive Publication Date: 2024-12-03GUANGZHOU ZHILI INFORMATION TECH CO LTD
View PDF 2 Cites 0 Cited by

Patent Information

Application Number
CN202210411143.6
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2022-04-19
Publication Date
2024-12-03
Estimated Expiration
2042-04-19

AI Technical Summary

Technical Problem

The existing indoor positioning system is low in precision and is susceptible to interference in complex environments, and cannot realize real-time virtual scene rendering, which increases the workload and labor costs of the film and television industry.

Method used

Using an indoor positioning system based on infrared cameras and high-precision IMU, real-time positioning and data processing are achieved through physical modules and algorithm units, including beacon units, sensor units, data transmission units and virtual photography units. 3D coordinates and scene rendering.

Benefits of technology

It realizes high-precision real-time positioning of indoor positioning, reduces post-processing needs, reduces labor costs in the film and television industry, and improves the technical standards of virtual photography.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN114777772B_ABST
    Figure CN114777772B_ABST
Patent Text Reader

Abstract

The invention discloses an indoor positioning system based on an infrared camera and a high-precision IMU, including a physical module and an algorithm unit, wherein the physical module includes a beacon unit, a sensor unit, a data transmission unit, a data processing unit and a virtual photography unit, and the invention relates to the fields of computer and film and television technology. The indoor positioning system based on an infrared camera and a high-precision IMU integrates the technologies in the field of indoor positioning and the field of film and television, and the indoor positioning system can obtain scene coordinates, character coordinates, depth of field data, and lens focal length in real time, which are used as the basis for the later virtual photography unit to position characters and scenes, which means that only a few CG rendering engine engineers are needed to replace a large later team, which greatly reduces the labor cost while ensuring that the photography quality will not be reduced or even improved, thereby enabling the film and television industry to improve the corresponding virtual photography technology standards.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the fields of computer and film and television technology, and in particular to an indoor positioning system based on an infrared camera and a high-precision IMU. Background Art

[0002] Indoor positioning refers to the realization of position positioning in indoor environments. It mainly uses wireless communication, base station positioning, inertial navigation positioning and other technologies to integrate into an indoor positioning system, so as to realize the position monitoring of people, objects, etc. in indoor spaces. In addition to the cellular positioning technology of the communication network, common indoor wireless positioning technologies include: WiFi, Bluetooth, infrared, ultra-wideband, RFID, ZigBee and ultrasound, etc. The indoor environment occupies 80% of human daily life time. Indoor location services have a wide range of application fields and potential needs. However, in actual applications, due to the complex indoor positioning environment and many interference sources, the accuracy of indoor positioning is generally at the regional or meter level, and is easily affected by environmental interference. The current indoor positioning system often uses visible light to analyze the indoor environment. The amount of information obtained by visible light is large, which leads to a corresponding increase in the degree of interference.

[0003] As an emerging industry in recent years, the virtual live broadcast industry generally uses web live broadcast and face tracking for live broadcast. The operation method is generally based on 2D flat characters. The interactivity between users and audiences is relatively low, and it is not possible to achieve universal use in all directions. The current film and television industry cannot perform real-time rendering. If you want to add a virtual scene, you need to arrange a green screen for shooting in the early stage, and the post-production staff will shoot the materials in the early stage for keying, set up the post-production scenes and render them. If problems are encountered during playback, they cannot be modified in real time, and the materials need to be further modified and rendered, which greatly increases the workload of film and television practitioners, and also increases the human resources and time costs of film and television practitioners. Summary of the invention

[0004] 1. Technical issues to be solved

[0005] In view of the shortcomings of the prior art, the present invention provides an indoor positioning system based on an infrared camera and a high-precision IMU, which solves the problem that the indoor positioning environment is complex and there are many interference sources in practical applications, and the use of visible light to analyze the indoor environment causes the degree of interference to increase accordingly due to the large amount of information obtained by visible light.

[0006] (II) Technical solution

[0007] To achieve the above objectives, the present invention is implemented through the following technical solutions: an indoor positioning system based on an infrared camera and a high-precision IMU, including a physical module and an algorithm unit, wherein the physical module includes a beacon unit, a sensor unit, a data transmission unit, a data processing unit and a virtual photography unit.

[0008] Preferably, the data acquisition and transmission algorithm of the algorithm unit uses optical fiber for long-distance transmission, uses FPGA for high-precision synchronization, and uses microcontroller 1 and FPGA1 to collect and synchronize infrared camera, IMU, and encoder data.

[0009] Preferably, the beacon unit includes an infrared reflective sticker, and the sensor unit includes an infrared camera, an infrared filter, infrared light LED lamp beads, 8 IMUs and 2 encoders.

[0010] Preferably, the data transmission unit includes 2 single-chip microcomputers, 2 FPGAs and optical fibers, and the data processing unit includes a high-performance X86 host.

[0011] Preferably, the virtual photography unit includes a jib, a stabilizer, a track, a drone and a flying cat system.

[0012] Preferably, the algorithm unit specifically includes the following steps:

[0013] 1. Mapping stage:

[0014] S1, collect infrared light images through infrared camera;

[0015] S2, selecting a threshold value for the image taken by the infrared camera and performing binarization processing to obtain a binary image;

[0016] S3, using the connected domain search algorithm, setting the connected domain threshold, and obtaining the centroid of the connected domain, i.e., the coordinates (u, v) of the infrared beacon in the image;

[0017] S4, using data association and tracking algorithms such as Kalman filtering or GNN (global nearest neighbor algorithm), to associate the infrared beacons obtained from consecutive frames and determine the unique beacon ID;

[0018] S5, using triangulation method, performing calculations on the same beacon between different frames to obtain the three-dimensional coordinates of the beacon;

[0019] S6. The operation in S5 involves converting the beacon coordinates from pixel coordinates (u, v) to three-dimensional coordinates (x, y, z) using the camera intrinsic and extrinsic matrix;

[0020] S7, solving the camera pose by PNP (N-point perspective pose solution) method, and then using the pose to reproject the beacon's three-dimensional coordinates onto the image to obtain the reprojected coordinates (u, v);

[0021] S8. The mean square error of the reprojected coordinates and the measured coordinates of all points is calculated to obtain a reprojection error value w, where w reflects the quality of the map currently constructed based on the beacon points.

[0022] S9. When the reprojection error is less than a certain value, the current map is considered to be available, and the three-dimensional coordinate sets of all beacon points are stored in a storage medium as a map;

[0023] 2. Tracking stage:

[0024] T1, obtain the infrared beacon coordinates (u, v) through the same algorithm as steps S1, S2 and S3 in the mapping stage;

[0025] T2. Match all currently obtained beacon coordinates with the map stored in the storage medium, determine the current map scene, and read the map into the program memory;

[0026] T3. According to the three-dimensional coordinates of the beacon points stored in the map, the camera pose is solved by the PNP (N-point perspective pose solution) method;

[0027] T4. Use algorithms such as EKFSLAM to fuse the camera pose and IMU data to obtain the final pose data;

[0028] T5. Send the final data to the virtual photography unit.

[0029] (III) Beneficial effects

[0030] The present invention provides an indoor positioning system based on an infrared camera and a high-precision IMU. It has the following beneficial effects: the indoor positioning system based on an infrared camera and a high-precision IMU includes a physical module and an algorithm unit. The physical module includes a beacon unit, a sensor unit, a data transmission unit, a data processing unit, and a virtual photography unit. By integrating the technologies in the field of indoor positioning and the field of film and television, the indoor positioning system can obtain scene coordinates, character coordinates, depth of field data (zoom), and lens focal length (focus) in real time, which are used as the basis for the later virtual photography unit to locate characters and scenes. The innovation is that it meets the real-time requirements that the current photography industry has not been able to achieve. It does not need to be processed in the later stage, and can output pictures with virtual scenes and virtual special effects in real time, which provides possibilities for the virtual live broadcast industry. By simply installing this positioning system in the virtual photography unit, you can obtain character positioning data, scene depth of field data, and focal length data that are difficult to obtain in post-processing. This type of data can determine the relative distance between the task and the specific object in the post-processing scene, and distinguish between people and objects, thereby achieving real-time virtual photography in the true sense. In addition, since the scene can be directly rendered through the CG rendering engine, it means that only a few CG rendering engine engineers are needed to replace the large post-production team, which greatly reduces labor costs while ensuring that the photography quality will not be reduced or even improved. With the application of this technology, the audience can better experience the feeling of being immersive, thereby enabling the film and television industry to improve the corresponding virtual photography technical standards. BRIEF DESCRIPTION OF THE DRAWINGS

[0031] Figure 1 is a block diagram of a physical module of the present invention;

[0032] Figure 2 This is a flow chart of the mapping phase of the present invention;

[0033] Figure 3 is a flow chart of the tracking phase of the present invention;

[0034] Figure 4 A reference map is set for the orientation data of the infrared camera of the present invention. DETAILED DESCRIPTION

[0035] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0036] See also Figure 1-4 An embodiment of the present invention provides a technical solution: an indoor positioning system based on an infrared camera and a high-precision IMU, including a physical module and an algorithm unit. The physical module includes a beacon unit, a sensor unit, a data transmission unit, a data processing unit and a virtual photography unit.

[0037] In the embodiment of the present invention, the data acquisition and transmission algorithm of the algorithm unit uses optical fiber for long-distance transmission and FPGA for high-precision synchronization, and uses microcontroller 1 and FPGA1 to collect and synchronize infrared camera, IMU, and encoder data.

[0038] In the embodiment of the present invention, the beacon unit includes infrared reflective stickers, and the sensor unit includes an infrared camera, an infrared filter (reflecting short infrared wavelengths), infrared LED lamp beads, 8 IMUs and 2 encoders (corresponding to zoom and focal length).

[0039] In the embodiment of the present invention, the data transmission unit includes 2 single-chip microcomputers, 2 FPGAs and optical fibers, and the data processing unit includes a high-performance X86 host (mainboard, serial port, network, pwm fan, memory, CPU, SSD and radiator).

[0040] In the embodiment of the present invention, the virtual photography unit includes a rocker arm, a stabilizer, a track, a drone and a flying cat system.

[0041] In the embodiment of the present invention, the algorithm unit specifically includes the following steps:

[0042] 1. Mapping stage:

[0043] S1, collect infrared light images through infrared camera;

[0044] S2, selecting a threshold value for the image taken by the infrared camera and performing binarization processing to obtain a binary image;

[0045] S3, using the connected domain search algorithm, setting the connected domain threshold, and obtaining the centroid of the connected domain, i.e., the coordinates (u, v) of the infrared beacon in the image;

[0046] S4, using data association and tracking algorithms such as Kalman filtering or GNN (global nearest neighbor algorithm), to associate the infrared beacons obtained from consecutive frames and determine the unique beacon ID;

[0047] S5, using triangulation method, performing calculations on the same beacon between different frames to obtain the three-dimensional coordinates of the beacon;

[0048] S6. The operation in S5 involves converting the beacon coordinates from pixel coordinates (u, v) to three-dimensional coordinates (x, y, z) using the camera intrinsic and extrinsic matrix;

[0049] S7, solving the camera pose by PNP (N-point perspective pose solution) method, and then using the pose to reproject the beacon's three-dimensional coordinates onto the image to obtain the reprojected coordinates (u, v);

[0050] S8. The mean square error of the reprojected coordinates and the measured coordinates of all points is calculated to obtain a reprojection error value w, where w reflects the quality of the map currently constructed based on the beacon points.

[0051] S9. When the reprojection error is less than a certain value, the current map is considered to be available, and the three-dimensional coordinate sets of all beacon points are stored in a storage medium as a map;

[0052] 2. Tracking stage:

[0053] T1, obtain the infrared beacon coordinates (u, v) through the same algorithm as steps S1, S2 and S3 in the mapping stage;

[0054] T2. Match all currently obtained beacon coordinates with the map stored in the storage medium, determine the current map scene, and read the map into the program memory;

[0055] T3. According to the three-dimensional coordinates of the beacon points stored in the map, the camera pose is solved by the PNP (N-point perspective pose solution) method;

[0056] T4. Use algorithms such as EKFSLAM to fuse the camera pose and IMU data to obtain the final pose data;

[0057] T5. Send the final data to the virtual photography unit.

[0058] Meanwhile, the contents not described in detail in this specification belong to the prior art known to those skilled in the art.

[0059] Actual use process:

[0060] 1. Install the infrared camera on the camera, determine the direction of the infrared camera, and measure the offset data between the camera's CCD and the infrared camera's CCD. Figure 4 .

[0061] 2. After connecting the optical fiber data cable and the network, turn on the host and wait for the infrared camera to start. It should be placed on a stationary plane to allow the infrared camera's gyroscope to automatically calibrate.

[0062] 3. Start scanning the reflective point map. You need to hold the infrared camera steadily and move the camera within the visible range of the map. Adjust the pan value of the infrared camera at any time. After the map is recognized, the signal is displayed as high quality and the map can be saved.

[0063] 4. Start to calibrate the map positioning:

[0064] (1) Plane calibration. This step requires data collection from four or more positioning points. Move the infrared camera to different points and ensure that the Roll and Till values ​​have a deviation less than 0.1. All points cover the entire reflective point map as much as possible. The height of the infrared camera can be set to any value in this step. You only need to ensure that the height of all points is consistent. For example, if it is placed on a flat floor, it can be set to 0.

[0065] (2) Zoom correction. You need to set the zoom mode. When the infrared camera is on the ground, choose to move the infrared camera to two points A and B. Make sure that the Pan\Till\Roll values ​​of the infrared camera at the two points A and B are consistent. Then measure the straight-line distance between the two points A and B. When the infrared camera is in the air and faces a direction other than the ground, select the reflective map point of the infrared camera as the zoom parameter. Select two marked points on the map. The distance between the two points should be as large as possible. Then fix the infrared camera, select the two points on the host, and measure the straight-line distance between the two points for input.

[0066] (3) Reset the pan value. Install the infrared camera back on the camera in the same way as before, adjust the camera's viewing angle, and reset the pan value after confirming the camera's direction.

[0067] (4) Reset the zero point to ensure that the attitude of the infrared camera and the difference in the Pan\Till\Roll values ​​on the camera are as small as possible. Then measure the vertical data of the camera CCD center point from the ground as the zero point height.

[0068] 5. Weight adjustment: here you can adjust the infrared camera's signal transmission smoothness, gyroscope weight, infrared light emission intensity, reflective point recognition threshold intensity, and infrared camera refresh rate (when using frame rate synchronization lock, select the general mode). In addition, you can choose half the frame rate and double the frame rate. The delay and delay ratio are also adjusted here and need to be adjusted according to the actual deviation.

[0069] 6. Visual preview of tracking data. Select different tracking protocols to obtain current tracking signal data.

[0070] 7. Lens file acquisition and lens correction, read the corresponding lens file, test its offset, adjust the lens center offset and FOV and other lens distortion data to ensure that the lens data sent to the virtual engine matches the real lens data.

[0071] 8. Set the UDP sending IP address (need component LAN). The tracking signal transmitter can send broadcast IP and specified IP, select the server IP address of the received tracking signal, and set the port. Tracking signals of different protocols can be sent according to different needs.

[0072] 9. The virtual engine server or any receiving end selects the same protocol as that sent by the tracking host and sets it to the same port number to receive the tracking signal.

[0073] 10. Test the tracking signal accuracy and delay. You can appropriately increase or decrease the delay frame rate and ratio in the weight adjustment interface and adjust according to actual feedback.

[0074] It should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device.

[0075] Although embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that various changes, modifications, substitutions and variations may be made to the embodiments without departing from the principles and spirit of the present invention, and that the scope of the present invention is defined by the appended claims and their equivalents.

Claims

1. An indoor positioning system based on an infrared camera and a high-precision IMU, characterized in that: It includes a physical module and an algorithm unit, wherein the physical module includes a beacon unit, a sensor unit, a data transmission unit, a data processing unit and a virtual photography unit; The data acquisition and transmission algorithm of the algorithm unit uses optical fiber for long-distance transmission and FPGA for high-precision synchronization. The single-chip microcomputer 1 and FPGA1 are used to collect and synchronize the infrared camera, IMU, and encoder data. The beacon unit includes infrared reflective stickers, the sensor unit includes an infrared camera, an infrared filter, infrared LED lamp beads, 8 IMUs, and 2 encoders. The data transmission unit includes 2 single-chip microcomputers, 2 FPGAs, and optical fibers. The data processing unit includes a high-performance X86 host. The virtual photography unit includes a rocker arm, a stabilizer, a track, a drone, and a flying cat system. An indoor positioning system based on an infrared camera and a high-precision IMU, characterized in that the algorithm unit specifically includes the following steps:

1. Mapping stage: S1, collect infrared light images through infrared camera; S2, selecting a threshold value for the image taken by the infrared camera and performing binarization processing to obtain a binary image; S3, using a connected domain search algorithm, setting a connected domain threshold, and obtaining the centroid of the connected domain, i.e., the coordinates of the infrared beacon in the image; S4. Use Kalman filter or GNN (global nearest neighbor algorithm) data association and tracking algorithm to associate the infrared beacons obtained from consecutive frames and determine the unique beacon ID; S5. Using triangulation, calculate the same beacon between different frames to obtain the three-dimensional coordinates of the beacon; S6. The operation in S5 involves converting the beacon coordinates from pixel coordinates to three-dimensional coordinates (x, y, z) using the camera intrinsic and extrinsic matrix; S7, solving the camera pose by PNP (N-point perspective pose solution) method, and then using the pose to reproject the beacon's three-dimensional coordinates onto the image to obtain the reprojected coordinates; S8. The mean square error of the reprojected coordinates and the measured coordinates of all points is calculated to obtain a reprojection error value w, where w reflects the quality of the map currently constructed based on the beacon points. S9. When the reprojection error is less than a certain value, the current map is considered to be available, and the three-dimensional coordinate sets of all beacon points are stored in a storage medium as a map; 2. Tracking stage: T1, obtain the infrared beacon coordinates through the same algorithm as steps S1, S2 and S3 in the mapping stage; T2. Match all currently obtained beacon coordinates with the map stored in the storage medium, determine the current map scene, and read the map into the program memory; T3. According to the three-dimensional coordinates of the beacon points stored in the map, the camera pose is solved by the PNP (N-point perspective pose solution) method; T4. Use the EKFSLAM algorithm to fuse the camera pose and IMU data to obtain the final pose data; T5. Send the final data to the virtual photography unit.

Citation Information

Patent Citations

  • Virtual photographing system and its instant communication method

    CN101272471A

  • Optical inertial fusion space positioning method, positioning equipment and positioning system

    CN109785381A