Robot positioning system, method, device and medium

Abstract

The present invention discloses a robot positioning system, method, device and medium, which combines photometric consistency and temperature consistency to perform posture tracking, so that the present invention has stable performance in different environments. In addition, the photometric information and temperature information are integrated into the same framework, achieving close coupling of the photometric information and temperature information, and then performing posture tracking to obtain a better positioning effect. In addition, the error accumulation of multiple postures of the inspection robot is optimized by the least squares method, and the optimized posture is output to locate the inspection process of the inspection robot, which can further improve the accuracy of positioning.

Description

Technical Field

[0001] The present invention relates to the field of robot inspection technology, and in particular to a robot positioning system, method, equipment and medium. Background Art

[0002] Unmanned inspection can make up for many defects of manual inspection and has been widely used in power grid systems. During the unmanned inspection process, the inspection robot needs to work between different devices, perceive and diagnose the environment, which puts great demands on the positioning technology of the inspection robot.

[0003] If only ordinary visible light cameras are used for positioning, the inspection robot can only be relatively sensitive to light, and the effect is poor during nighttime inspections. As a type of visual sensor, thermal infrared cameras can map the temperature value radiated by an object to the pixel intensity of an image, and are less affected by changing illumination or background clutter, making them a good supplement to traditional visible light cameras. However, the use of RGB and temperature information in the prior art is a loosely coupled process, that is, using visible light cameras and infrared cameras separately, obtaining positioning results and then optimizing them, but the positioning effect obtained in this way is still not accurate enough and needs further optimization. Summary of the invention

[0004] Based on this, it is necessary to provide a robot positioning system, method, device and medium to address the above-mentioned problems so as to solve the problem that the positioning effect is still not accurate enough.

[0005] A robot positioning system is applied to an inspection robot provided with a visible light camera and an infrared camera, wherein the visible light camera is used to capture visible light images, and the infrared camera is used to capture infrared images. The system comprises: a photometric calibration module, a temperature calibration module, a posture initialization module, a posture estimation module, and a back-end optimization module;

[0006] The photometric calibration module is used to calibrate the photometric parameters of the visible light camera during the imaging process, so as to adjust the photometric information of the visible light image based on the calibrated photometric parameters until the acquired visible light image satisfies photometric consistency; wherein the photometric consistency indicates that the photometric information of the same pixel in different visible light images is the same;

[0007] The temperature calibration module is used to calibrate the drift during the imaging process of the infrared camera to eliminate the temperature noise generated during the imaging process of the infrared camera until the acquired infrared image meets the temperature consistency; wherein the temperature consistency indicates that the temperature information of the same pixel in different infrared images is the same;

[0008] The posture initialization module is used to extract sparse feature points in the visible light image, and calculate the initial relative posture of the inspection robot using feature point pairs between two adjacent frames of visible light images; wherein the feature point pairs are composed of sparse feature points;

[0009] The posture estimation module is used to establish an error function based on the luminosity difference and temperature difference under the affine transformation of the sparse feature points, and minimize the error function to track the current posture of the inspection robot.

[0010] The back-end optimization module is used to optimize the error accumulation of multiple postures of the inspection robot through the least squares method, and output the optimized posture to locate the inspection process of the inspection robot.

[0011] In one embodiment, the photometric parameter is a response function of the optical camera, and the response function is used to indicate the relationship between the input exposure and the photometric information in the output visible light image. The photometric calibration module is specifically used to:

[0012] Whenever the visible light camera takes a picture, the exposure time of the visible light camera is adjusted to obtain a plurality of visible light images taken, and the luminosity information of each pixel in each visible light image is determined; wherein the plurality of visible light images are visible light images taken at different angles by the visible light camera for the same static scene;

[0013] The irradiance image of the visible light camera and the influencing parameters of the optical module in the optical camera are obtained, and based on the photometric information in different visible light images, the response function and the product of the irradiance image and the influencing parameters are minimized alternately through maximum likelihood estimation to calibrate the response function until the obtained visible light image meets the photometric consistency.

[0014] In one embodiment, the temperature calibration module is specifically used for:

[0015] Acquire a plurality of infrared images taken at the same time as the plurality of visible light images, and determine temperature information of each pixel in each infrared image;

[0016] Determine the moving speed of any pixel based on the same photometric information in different visible light images, and determine the same pixel corresponding to the different infrared images based on the moving speed and the time interval between the infrared images;

[0017] Based on the infrared radiation of the infrared camera, the gain and drift of the temperature information of the same pixel point in different infrared images are adjusted to eliminate the heat interference generated by the infrared camera during the imaging process and the fixed noise of the infrared image until the infrared image meets the temperature consistency.

[0018] In one embodiment, the posture initialization module is specifically used to:

[0019] If the grayscale difference between the target pixel point and the pixel points in the preset neighborhood is greater than a preset value, the target pixel point is determined as a sparse feature point; wherein the target pixel point is any pixel point in the visible light image;

[0020] Taking a visible light image with a number of sparse feature points greater than a first preset number as a key frame, taking the first key frame as an initial key frame, determining a feature point pair between the initial key frame and a frame next to the initial key frame, and performing triangulation initialization based on the feature point pair to obtain a map containing map points; wherein the map points of the map are used to describe the actual spatial positions of the sparse feature points;

[0021] A homography matrix and a basic matrix are constructed by using the map points and the feature point pairs, and an initial relative position and posture of the inspection robot is calculated according to the homography matrix and the basic matrix.

[0022] In one embodiment, the posture estimation module is specifically used to:

[0023] Taking the visible light image acquired at the current moment as the current frame, for the map points included in the current frame, determining a first projection point on the initial key frame and a second projection point on the current frame;

[0024] Calculate the grayscale difference and temperature difference between the first projection point and the second projection point under affine transformation, construct a corresponding two-norm error function according to the grayscale difference and the temperature difference, and minimize the two-norm error function to estimate the current posture of the inspection robot.

[0025] In one embodiment, the posture estimation module is further specifically used for:

[0026] Taking the first projection point as the center, determining the pixel point at the preset position in the initial key frame as the first projection image block, and taking the second projection point as the center, determining the pixel point at the same preset position in the current frame as the second projection image block;

[0027] An eight-dimensional residual function is constructed according to the grayscale difference and the temperature difference between the first projection image block and the second projection image block, and the eight-dimensional residual function is minimized to optimize the current posture of the inspection robot.

[0028] In one embodiment, the backend optimization module is specifically used to:

[0029] Creating a sliding window including a second preset number of key frames, and when a new key frame is determined to be added to the sliding window, marginalizing the map points corresponding to all key frames and the pixels not observed in the last two key frames;

[0030] The key frame to be removed from the sliding window is determined by accumulating the errors of multiple postures using the least square method, and the key frame to be removed is removed from the sliding window.

[0031] A robot positioning method is applied to an inspection robot provided with a visible light camera and an infrared camera, wherein the visible light camera is used to capture visible light images, and the infrared camera is used to capture infrared images. The method comprises:

[0032] Calibrate the photometric parameters of the visible light camera during the imaging process to adjust the photometric information of the visible light image based on the calibrated photometric parameters until the acquired visible light image satisfies photometric consistency; wherein the photometric consistency indicates that the photometric information of the same pixel in different visible light images is the same;

[0033] Calibrate the drift during the imaging process of the infrared camera to eliminate the temperature noise generated during the imaging process of the infrared camera until the acquired infrared image meets the temperature consistency; wherein the photometric consistency indicates that the temperature information of the same pixel in different infrared images is the same;

[0034] Extracting sparse feature points in the visible light image, and using feature point pairs between two adjacent frames of visible light images to calculate the initial relative posture of the inspection robot; wherein the feature point pairs are composed of sparse feature points;

[0035] An error function is established based on the luminosity difference and temperature difference under the affine transformation of the sparse feature points, and the error function is minimized to track the current posture of the inspection robot.

[0036] The error accumulation of multiple postures of the inspection robot is optimized by the least square method, and the optimized posture is output to locate the inspection process of the inspection robot.

[0037] A computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the processor executes the steps of the robot positioning method.

[0038] A robot positioning device comprises a memory and a processor, wherein the memory stores a computer program, and when the computer program is executed by the processor, the processor executes the steps of the robot positioning method.

[0039] The present invention provides a robot positioning system, method, device and medium, which calibrates the photometric parameters of a visible light camera during imaging, so as to adjust the photometric information of a visible light image based on the calibrated photometric parameters until the acquired visible light image satisfies photometric consistency. And calibrates the drift during the imaging process of an infrared camera, so as to eliminate the temperature noise generated during the imaging process of the infrared camera, until the infrared image satisfies temperature consistency. It can be seen that the posture tracking is performed by combining the photometric consistency and the temperature consistency, so that the present invention has stable performance in different environments. Further, sparse feature points in the visible light image are extracted, and the initial relative posture of the inspection robot is calculated by using the feature point pairs between two adjacent frames of visible light images; and an error function is established based on the photometric difference and temperature difference under the affine transformation of the sparse feature points, and the error function is minimized to track the current posture of the inspection robot. It can be seen that the present invention integrates the photometric information and the temperature information into the same framework, realizes the close coupling of the photometric information and the temperature information, and then performs posture tracking, which can obtain a better positioning effect. Finally, the error accumulation of multiple postures of the inspection robot is optimized by the least square method, and the optimized posture is output to locate the inspection process of the inspection robot, which can further improve the positioning accuracy. BRIEF DESCRIPTION OF THE DRAWINGS

[0040] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0041] in:

[0042] Figure 1 A schematic diagram of a robot positioning system in one embodiment;

[0043] Figure 2 A schematic diagram of determining sparse feature points in one embodiment;

[0044] Figure 3 A schematic diagram of selecting a projection image block in one embodiment;

[0045] Figure 4 A schematic diagram of a flow chart of a robot positioning method in one embodiment;

[0046] Figure 5 4 is a structural block diagram of a robot positioning device in one embodiment. DETAILED DESCRIPTION

[0047] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0048] like Figure 1 As shown, Figure 1 1 is a schematic diagram of the framework of a robot positioning system in an embodiment, and the robot positioning system is applied to an inspection robot equipped with a visible light camera and an infrared camera. During the inspection process of the inspection robot, the visible light camera is used to capture visible light images, and the infrared camera is used to capture infrared images. The robot positioning system includes: a photometric calibration module 102, a temperature calibration module 104, a posture initialization module 106, a posture estimation module 108, and a backend optimization module 110. The following describes the functions implemented by each module:

[0049] The photometric calibration module 102 is used to calibrate the photometric parameters of the visible light camera during the imaging process, so as to adjust the photometric information of the visible light image based on the calibrated photometric parameters until the acquired visible light image meets the photometric consistency.

[0050] Specifically, photometric consistency indicates that the photometric information of the same pixel in different visible light images is the same. The following photometric model is used in the photometric calibration process:

[0051] I i (x) = G(t i V(x)B i (x)

[0052] Among them, I i is the i-th visible light image obtained; V represents the influencing parameters of the camera optical module. The parameters of the optical module such as the lens, aperture, shutter, etc. of the visible light camera will affect the photometric information of the visible light image; t i represents the exposure time when acquiring the i-th visible light image; B i is the camera irradiance image when acquiring the i-th visible light image; G represents the response function, which represents the relationship between the camera input exposure and output brightness value.

[0053] First, the visible light camera in this embodiment is fixed on the inspection robot. During the inspection process of the inspection robot, the visible light camera takes visible light images at different angles for the same static scene. In this process, the irradiance image B of the camera iAnd each time the visible light camera takes a picture, the photometric calibration module adjusts the exposure time of the visible light camera, so that multiple visible light images with different angles and exposure times can be obtained, and then the photometric information of each pixel in each visible light image can be determined.

[0054] Since V is an inherent property of the camera, when calibrating the camera response function, the camera's photometric model is converted into:

[0055] I i (x) = G(t i B′(x)

[0056] Here B′(x)=V(x)B(x). When finding the response function, for convenience, we find the inverse function of the response function: U=G -1 , for an image I i , the corresponding exposure time is t i , assuming there is a zero-mean Gaussian noise on U(I(x)), the model has the following form:

[0057]

[0058] Through maximum likelihood estimation, we approximate the least squares form as follows:

[0059]

[0060] Among them, the first sum is for all images (with different exposure times), and the second sum is for all pixels on the image plane. Minimize the above formula and solve U and B' alternately to get:

[0061]

[0062] Then, based on the solved U, the calibrated response function G can be obtained. Whenever a new visible light image is obtained, the above photometric calibration process is repeated until the obtained visible light image meets the photometric consistency.

[0063] The temperature calibration module 104 is used to calibrate the drift during the imaging process of the infrared camera to eliminate the temperature noise generated during the imaging process of the infrared camera until the acquired infrared image meets the temperature consistency.

[0064] Specifically, temperature consistency indicates that the temperature information of the same pixel in different infrared images is the same. In this embodiment, the temperature calibration process is:

[0065] First, each time the visible light camera takes a visible light image, the infrared camera obtains an infrared image at the same time and determines the temperature information of each pixel in each infrared image.

[0066] Secondly, determine the corresponding pixel points in different infrared images. Considering the pixel at (x, y) at time t, its grayscale information and temperature information can be written as I(x, y, t) and T(x, y, t), respectively. In order to estimate the position of this pixel in the image at other times, based on the consistency of luminosity and temperature, at time t+dt, the pixel moves to (x+dx, y+dy). Since the grayscale and luminosity remain unchanged, we can get:

[0067] I(x+dx,y+dy,t+dt)=I(x,y,t),

[0068] T(x+dx,y+dy,t+dt)=T(x,y,t)

[0069] Taylor expansion is performed on the two equations respectively, the first-order terms are retained, and the change of the image over time and the change of the temperature over time are recorded as I t ,T t , expressed in matrix form, we can get:

[0070]

[0071]

[0072] Where u and v represent the speed of the pixel (x, y) in the horizontal and vertical directions of the visible light image and the infrared image, respectively. In this way, if the time interval between infrared images is known, the corresponding pixel point in different infrared images can be determined.

[0073] Finally, based on the principle of infrared radiation of infrared cameras, the gain and drift of the temperature information of the same pixel in different infrared images are adjusted to eliminate the thermal interference generated by the infrared camera during the imaging process and the fixed noise of the infrared image until the infrared image meets the temperature consistency.

[0074] The posture initialization module 106 is used to extract sparse feature points in the visible light image and calculate the initial relative posture of the inspection robot using the feature point pairs between two adjacent frames of visible light images.

[0075] Specifically, first, the sparse feature points are determined in the following way: for a certain visible light image I i For example, the target pixel point (visible light image I i Any pixel in the Figure 2 As shown, generally, the grayscale difference between the pixels within a circle with a radius of 3.4 pixels and an outer 16-pixel circle as the preset area) is greater than the preset value, then the target pixel is determined as a sparse feature point, and its formula is expressed as:

[0076]

[0077] Where I(x) represents the pixel value of the peripheral pixel, I(p) represents the pixel value of the central pixel, and ε d The default value for the setting.

[0078] Next, the visible light image with the number of sparse feature points greater than the first preset number (e.g., 50) is used as a key frame, and the first key frame is used as the initial key frame. The feature point pairs between the initial key frame and the next frame of the initial key frame are determined based on photometric consistency, and triangulation initialization is performed. The result of the triangulation initialization is a map containing map points, which are used to describe the actual spatial position of the sparse feature points, that is, to determine the depth information of each sparse feature point, and transform the sparse feature points from a two-dimensional angle description to a three-dimensional angle description.

[0079] Finally, the homography matrix and the basic matrix are constructed by the map points and feature point pairs. The homography matrix represents the homogeneous transformation between the two planes, which contains the camera pose information and the intrinsic parameter information. The pose extraction method based on the homography matrix and the basic matrix mainly includes the identification of the homography matrix, the decomposition of the homography matrix, and the determination of the unique solution. The existing technical means in this field can be used, so it will not be repeated here, and finally the initial relative pose of the inspection robot is obtained.

[0080] The posture estimation module 108 is used to establish an error function based on the luminosity difference and temperature difference under the affine transformation of the sparse feature points, and minimize the error function to track the current posture of the inspection robot.

[0081] Specifically, the visible light image acquired at the current moment is taken as the current frame, and for the map point p contained in the current frame, the first projection point u on the initial key frame is determined. i and the second projection point u′ on the current frame i If the camera's pose is not good enough, u i and u′ i The appearance and temperature of u′ will be obviously different, and we can find a more similar u′ by optimizing the camera’s position. i , the objective function minimized in the optimization process is the grayscale difference and temperature difference under affine transformation:

[0082] e1=I1(u′ i )-I2(u i )

[0083] e2=T1(u′ i )-T2(u i )

[0084] Assume that the luminance information of a map point is constant under different viewing angles, and the temperature information of the image is also constant. In practice, we can obtain many map points, and the pose estimation problem of the entire inspection robot can be represented by the error of vision and temperature. The optimization goal is the two-norm error function constructed based on the grayscale difference and temperature difference:

[0085]

[0086] Among them, I r (u i ) represents the grayscale information of the first projection point in the initial key frame, I k (u′ i ) represents the grayscale information of the second projection point in the current frame, T r (u i ) represents the temperature information of the first projection point in the initial key frame, T k (u′ i ) represents the temperature information of the second projection point in the current frame. α and β are two constants, which respectively represent the proportion of grayscale information and temperature information in the optimization function. The value can be set adaptively for different brightness environments and environments with different temperature differences. i and B i is the preset coefficient. To minimize this two-norm error function, we can use the Lie algebra perturbation model method to multiply the small perturbation on the left to estimate the current position of the inspection robot.

[0087] It can be seen that the pose estimation module simultaneously performs grayscale consistency constraints and temperature consistency constraints. Assuming that the pixels in a certain window have the same movement, the grayscale and temperature of the pixels are fixed in each image. Compared with the traditional single use of optical flow method, the present invention effectively integrates the photometric information and temperature information of the image pixels. For each pixel in the visible light image, its corresponding position and temperature value can be obtained in the infrared image, which enables the present invention to have better robustness in different environments. When the camera moves to a place where visual features are missing but there is a good temperature difference, although there is a lack of obvious texture information in the environment, there is a temperature gradient change, which can increase the proportion of temperature information, thereby more accurately calculating the pose of the inspection robot.

[0088] Further, such as Figure 3 As shown in , in order to make better use of the projection point information, the pose estimation module considers not only the projection point itself, but also the image block composed of the projection point and several surrounding points for each projection point. Taking the first projection point as the center, the initial key frame is Figure 3 The pixel point at the preset position is determined as the first projection image block, and the second projection point is taken as the center, and the pixel point at the preset position is determined as the first projection image block. Figure 3The pixel point at the preset position is determined as the second projection image block. Each surrounding projection point is equivalent to the central projection point. Since a total of 8 projection points are selected as projection image blocks, an eight-dimensional residual function can be constructed based on the above-mentioned two-norm error function, and the eight-dimensional residual function can be minimized to optimize the current posture of the inspection robot.

[0089] The back-end optimization module 110 is used to optimize the error accumulation of multiple postures of the inspection robot by the least square method, and output the optimized posture to locate the inspection process of the inspection robot.

[0090] In a specific embodiment, a sliding window containing a second preset number (for example, 7) of key frames is first created, and the points in the sliding window are divided into two types, one is an activation point and the other is an immature point. Usually, the points that have just been extracted in the image or whose depth has not converged are called immature points, which have a very large depth range. Through the depth filter, the depth of the point is continuously converged between frames, and the point finally obtained will be activated and added to the sliding window. This point is called an activation point. By combining all the key frame poses and activation points in the sliding window to optimize a total error accumulation, compared with the constraints between two frames, the sliding window provides more constraints, which can optimize the relative pose initial value estimated by the front end. When it is determined that a new key frame is added to the sliding window, the size of the sliding window will continue to increase, and eventually lose its flexible and efficient functions. In order to maintain a fixed size window, the map points corresponding to all key frames and the pixels that are not observed in the last two key frames are marginalized. The key frames removed from the sliding window are determined by the least squares method. The error accumulation of multiple poses is used to remove the key frames from the sliding window, and the key frames that are determined to be removed are removed from the sliding window, and the depth is restored at the same time.

[0091] The robot positioning system calibrates the photometric parameters of the visible light camera during the imaging process, so as to adjust the photometric information of the visible light image based on the calibrated photometric parameters until the acquired visible light image meets the photometric consistency. And the drift of the infrared camera during the imaging process is calibrated to eliminate the temperature noise generated during the imaging process of the infrared camera until the infrared image meets the temperature consistency. It can be seen that the pose tracking is carried out by combining the photometric consistency and the temperature consistency, so that the present invention has stable performance in different environments. Further, the sparse feature points in the visible light image are extracted, and the initial relative pose of the inspection robot is calculated by using the feature point pairs between two adjacent frames of visible light images; and the error function is established based on the photometric difference and temperature difference under the affine transformation of the sparse feature points, and the error function is minimized to track the current pose of the inspection robot. It can be seen that the present invention integrates the photometric information and the temperature information into the same framework, realizes the close coupling of the photometric information and the temperature information, and then performs pose tracking, which can obtain a better positioning effect. Finally, the error accumulation of multiple postures of the inspection robot is optimized by the least squares method, and the optimized posture is output to locate the inspection process of the inspection robot, which can further improve the positioning accuracy.

[0092] In one embodiment, Figure 4 As shown, a robot positioning method is proposed, the steps of the method include:

[0093] Step 402 , calibrating the photometric parameters of the visible light camera during the imaging process, so as to adjust the photometric information of the visible light image based on the calibrated photometric parameters, until the acquired visible light image meets the photometric consistency.

[0094] Step 404, calibrating the drift during the infrared camera imaging process to eliminate the temperature noise generated during the infrared camera imaging process until the acquired infrared image meets the temperature consistency.

[0095] Step 406 , extracting sparse feature points from the visible light image, and using feature point pairs between two adjacent frames of visible light images to calculate the initial relative position and posture of the inspection robot.

[0096] Step 408: Establish an error function based on the luminosity difference and temperature difference under the affine transformation of the sparse feature points, and minimize the error function to track the current position and posture of the inspection robot.

[0097] Step 410, optimizing the error accumulation of multiple postures of the inspection robot by the least square method, and outputting the optimized posture to locate the inspection process of the inspection robot.

[0098] Figure 5 FIG. 2 shows an internal structure diagram of a robot positioning device in one embodiment. Figure 5As shown, the robot positioning device includes a processor, a memory and a network interface connected via a system bus. The memory includes a non-volatile storage medium and an internal memory. The non-volatile storage medium of the robot positioning device stores an operating system and may also store a computer program. When the computer program is executed by the processor, the processor may implement the robot positioning method. The internal memory may also store a computer program. When the computer program is executed by the processor, the processor may implement the robot positioning method. Those skilled in the art will understand that Figure 5 The structure shown in the figure is merely a block diagram of a portion of the structure related to the present application scheme, and does not constitute a limitation on the robot positioning device to which the present application scheme is applied. The specific robot positioning device may include more or fewer components than those shown in the figure, or combine certain components, or have a different component arrangement.

[0099] A robot positioning device includes a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, the following steps are implemented: calibrating the photometric parameters of a visible light camera during imaging, so as to adjust the photometric information of a visible light image based on the calibrated photometric parameters until the acquired visible light image meets photometric consistency. Calibrating the drift during the imaging process of an infrared camera, so as to eliminate the temperature noise generated during the imaging process of the infrared camera, until the acquired infrared image meets temperature consistency. Extracting sparse feature points in the visible light image, using feature point pairs between two adjacent frames of visible light images, calculating the initial relative posture of the inspection robot; wherein the feature point pairs are composed of sparse feature points; establishing an error function based on the photometric difference and temperature difference under the affine transformation of the sparse feature points, minimizing the error function, so as to track the current posture of the inspection robot. Optimizing the error accumulation of multiple postures of the inspection robot by the least squares method, outputting the optimized posture, so as to locate the inspection process of the inspection robot.

[0100] A computer-readable storage medium stores a computer program, which implements the following steps when executed by a processor: calibrating the photometric parameters of a visible light camera during imaging, so as to adjust the photometric information of a visible light image based on the calibrated photometric parameters until the acquired visible light image satisfies photometric consistency. Calibrating the drift during the imaging process of an infrared camera, so as to eliminate the temperature noise generated during the imaging process of the infrared camera, until the acquired infrared image satisfies temperature consistency. Extracting sparse feature points from the visible light image, using feature point pairs between two adjacent frames of visible light images, calculating the initial relative posture of the inspection robot; wherein the feature point pairs are composed of sparse feature points; establishing an error function based on the photometric difference and temperature difference under the affine transformation of the sparse feature points, minimizing the error function, so as to track the current posture of the inspection robot. Optimizing the error accumulation of multiple postures of the inspection robot by the least squares method, outputting the optimized posture, so as to locate the inspection process of the inspection robot.

[0101] It should be noted that the above-mentioned robot positioning method, device, equipment and computer-readable storage medium belong to a general inventive concept, and the contents in the embodiments of the robot positioning method, device, equipment and computer-readable storage medium are applicable to each other.

[0102] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the program can be stored in a non-volatile computer-readable storage medium. When the program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, storage, database or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory. Non-volatile memory may include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory may include random access memory (RAM) or external cache memory. As an illustration and not limitation, RAM is available in many forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM).

[0103] The technical features of the above embodiments may be arbitrarily combined. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0104] The above embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the present application. It should be pointed out that, for a person of ordinary skill in the art, several modifications and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the present application shall be subject to the attached claims.

Claims

1. A robot positioning system, characterized in that: Applied to an inspection robot provided with a visible light camera and an infrared camera, the visible light camera is used to capture visible light images, and the infrared camera is used to capture infrared images. The system includes: a photometric calibration module, a temperature calibration module, a posture initialization module, a posture estimation module, and a backend optimization module; The photometric calibration module is used to calibrate the photometric parameters of the visible light camera during the imaging process, so as to adjust the photometric information of the visible light image based on the calibrated photometric parameters until the acquired visible light image satisfies photometric consistency; wherein the photometric consistency indicates that the photometric information of the same pixel in different visible light images is the same; The temperature calibration module is used to calibrate the drift during the imaging process of the infrared camera to eliminate the temperature noise generated during the imaging process of the infrared camera until the acquired infrared image meets the temperature consistency; wherein the temperature consistency indicates that the temperature information of the same pixel in different infrared images is the same; The posture initialization module is used to extract sparse feature points in the visible light image, and calculate the initial relative posture of the inspection robot using feature point pairs between two adjacent frames of visible light images; wherein the feature point pairs are composed of sparse feature points; The posture estimation module is used to establish an error function based on the luminosity difference and temperature difference under the affine transformation of the sparse feature points, and minimize the error function to track the current posture of the inspection robot; The back-end optimization module is used to optimize the error accumulation of multiple postures of the inspection robot by the least square method, and output the optimized posture to locate the inspection process of the inspection robot; The photometric parameter is a response function of the optical camera, and the response function is used to indicate the relationship between the input exposure and the photometric information in the output visible light image. The photometric calibration module is specifically used to: adjust the exposure time of the visible light camera every time the visible light camera shoots, obtain multiple visible light images taken, and determine the photometric information of each pixel in each visible light image; wherein the multiple visible light images are visible light images taken at different angles of the same static scene by the visible light camera; obtain the irradiance image of the visible light camera and the influencing parameters of the optical module in the optical camera, and based on the photometric information in different visible light images, alternately minimize the response function and the product of the irradiance image and the influencing parameters through maximum likelihood estimation to calibrate the response function until the obtained visible light image meets photometric consistency; The temperature calibration module is specifically used to: obtain multiple infrared images taken at the same time as the multiple visible light images, and determine the temperature information of each pixel in each infrared image; determine the moving speed of any pixel based on the same photometric information in different visible light images, and determine the same pixel corresponding to different infrared images according to the moving speed and the time interval between the infrared images; based on the infrared radiation of the infrared camera, adjust the gain and drift of the temperature information of the same pixel in different infrared images to eliminate the heat interference generated by the infrared camera during the imaging process and the fixed noise of the infrared image until the infrared image meets the temperature consistency; The posture initialization module is specifically used to: if the grayscale difference between the target pixel point and the pixel points in a preset neighborhood is greater than a preset value, determine the target pixel point as a sparse feature point; wherein the target pixel point is any pixel point in the visible light image; use the visible light image with a number of sparse feature points greater than a first preset number as a key frame, use the first key frame as an initial key frame, determine the feature point pairs between the initial key frame and the next frame of the initial key frame, and perform triangulation initialization based on the feature point pairs to obtain a map containing map points; wherein the map points of the map are used to describe the actual spatial positions of the sparse feature points; construct a homography matrix and a basic matrix through the map points and the feature point pairs, and calculate the initial relative posture of the inspection robot according to the homography matrix and the basic matrix; The posture estimation module is specifically used to: take the visible light image acquired at the current moment as the current frame, and for the map points contained in the current frame, determine the first projection point on the initial key frame and the second projection point on the current frame; calculate the grayscale difference and temperature difference between the first projection point and the second projection point under affine transformation, construct a corresponding two-norm error function based on the grayscale difference and the temperature difference, and minimize the two-norm error function to estimate the current posture of the inspection robot.

2. The robot positioning system according to claim 1, characterized in that: The posture estimation module is further specifically used for: Taking the first projection point as the center, determining the pixel point at the preset position in the initial key frame as the first projection image block, and taking the second projection point as the center, determining the pixel point at the same preset position in the current frame as the second projection image block; An eight-dimensional residual function is constructed according to the grayscale difference and the temperature difference between the first projection image block and the second projection image block, and the eight-dimensional residual function is minimized to optimize the current posture of the inspection robot.

3. The robot positioning system according to claim 2, characterized in that: The backend optimization module is specifically used for: Creating a sliding window including a second preset number of key frames, and when a new key frame is determined to be added to the sliding window, marginalizing the map points corresponding to all key frames and the pixels not observed in the last two key frames; The key frame to be removed from the sliding window is determined by accumulating the errors of multiple postures using the least square method, and the key frame to be removed is removed from the sliding window.

4. A robot positioning method, characterized in that: Applied to an inspection robot provided with a visible light camera and an infrared camera, the visible light camera is used to capture visible light images, and the infrared camera is used to capture infrared images, the method comprises: Calibrate the photometric parameters of the visible light camera during the imaging process to adjust the photometric information of the visible light image based on the calibrated photometric parameters until the acquired visible light image satisfies photometric consistency; wherein the photometric consistency indicates that the photometric information of the same pixel in different visible light images is the same; Calibrate the drift during the imaging process of the infrared camera to eliminate the temperature noise generated during the imaging process of the infrared camera until the acquired infrared image meets the temperature consistency; wherein the photometric consistency indicates that the temperature information of the same pixel in different infrared images is the same; Extracting sparse feature points in the visible light image, and using feature point pairs between two adjacent frames of visible light images to calculate the initial relative posture of the inspection robot; wherein the feature point pairs are composed of sparse feature points; An error function is established based on the luminosity difference and the temperature difference under the affine transformation of the sparse feature points, and the error function is minimized to track the current posture of the inspection robot; Optimizing the error accumulation of multiple postures of the inspection robot by the least square method, and outputting the optimized posture to locate the inspection process of the inspection robot; The photometric parameter is a response function of the optical camera, and the response function is used to indicate the relationship between the input exposure and the photometric information in the output visible light image. The photometric parameter of the visible light camera during the imaging process is calibrated to adjust the photometric information of the visible light image based on the calibrated photometric parameter until the acquired visible light image satisfies the photometric consistency, including: adjusting the exposure time of the visible light camera each time the visible light camera shoots, acquiring multiple visible light images obtained by shooting, and determining the photometric information of each pixel in each visible light image; wherein the multiple visible light images are visible light images taken at different angles by the visible light camera for the same static scene; acquiring the irradiance image of the visible light camera and the influencing parameters of the optical module in the optical camera, and based on the photometric information in different visible light images, alternately minimizing the response function and the product of the irradiance image and the influencing parameters through maximum likelihood estimation to calibrate the response function until the acquired visible light image satisfies the photometric consistency; The calibrating of the drift in the imaging process of the infrared camera to eliminate the temperature noise generated in the imaging process of the infrared camera until the acquired infrared image meets the temperature consistency includes: acquiring multiple infrared images taken at the same time as the multiple visible light images, and determining the temperature information of each pixel in each infrared image; determining the moving speed of any pixel based on the same photometric information in different visible light images, and determining the same pixel corresponding to different infrared images according to the moving speed and the time interval between the infrared images; adjusting the gain and drift of the temperature information of the same pixel in different infrared images based on the infrared radiation of the infrared camera to eliminate the heat interference generated in the imaging process of the infrared camera and the fixed noise of the infrared image until the infrared image meets the temperature consistency; The extracting of sparse feature points in the visible light image and calculating the initial relative pose of the inspection robot using the feature point pairs between two adjacent frames of visible light images include: if the grayscale difference between the target pixel point and the pixel points in a preset neighborhood is greater than a preset value, determining the target pixel point as a sparse feature point; wherein the target pixel point is any pixel point in the visible light image; using a visible light image in which the number of sparse feature points is greater than a first preset number as a key frame, using the first key frame as an initial key frame, determining the feature point pairs between the initial key frame and the next frame of the initial key frame, and performing triangulation initialization based on the feature point pairs to obtain a map containing map points; wherein the map points of the map are used to describe the actual spatial positions of the sparse feature points; constructing a homography matrix and a basic matrix through the map points and the feature point pairs, and calculating the initial relative pose of the inspection robot based on the homography matrix and the basic matrix; The method of establishing an error function based on the luminosity difference and temperature difference under the affine transformation of sparse feature points and minimizing the error function to track the current posture of the inspection robot includes: taking the visible light image acquired at the current moment as the current frame, and determining the first projection point on the initial key frame and the second projection point on the current frame for the map points contained in the current frame; calculating the grayscale difference and temperature difference between the first projection point and the second projection point under the affine transformation, constructing a corresponding two-norm error function according to the grayscale difference and the temperature difference, and minimizing the two-norm error function to estimate the current posture of the inspection robot.

5. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the processor is caused to perform the steps of the method as claimed in claim 4.

6. A robot positioning device, comprising a memory and a processor, characterized in that: The memory stores a computer program, and when the computer program is executed by the processor, the processor is caused to perform the steps of the method as claimed in claim 4.

Images