Camera posture compensation calculation method and device, storage medium, and electronic device

Abstract

The present invention provides a method and device, storage medium, and electronic device for calculating camera pose compensation, belonging to the field of image processing. The method comprises: acquiring a first image frame from a reference camera and a second image frame from a target camera; calculating a projection information matrix between the first and second image frames, calculating a deformation tensor between the first and second image frames, and calculating an environmental distortion information matrix based on the deformation tensor; constructing an information matrix using the projection information matrix and the environmental distortion information matrix; and using the information matrix to compensate and calculate the pose of the target camera. The embodiments of the present invention solve the technical problem of low camera pose accuracy in related technologies, improve measurement accuracy in the field of image measurement, and enhance the inference and positioning accuracy of vehicles.

Description

Technical Field

[0001] The present invention relates to the field of image processing, and in particular to a camera posture compensation calculation method and device, a storage medium, and an electronic device. Background Art

[0002] With the rapid development of computer science theory and hardware, numerous artificial intelligence technologies and devices, such as AlphaGo and ChatGPT, have gradually become part of our daily lives. Intelligent robots and self-driving cars, as among the smart devices most closely connected to our lives, are attracting widespread attention in today's society. Visual information, a crucial means for these intelligent devices to perceive their environment, forms the data foundation for control operations and intelligent analysis, particularly in the positioning, planning, and control of autonomous vehicles.

[0003] In related technologies, visual information processing primarily relies on semantic features and feature point features. The former utilizes deep learning techniques to obtain semantically perceived classification of images; the latter typically utilizes SFM techniques, combined with filtering theory, wheel speed measurement equipment, and GPS devices, to calculate camera pose and infer vehicle positioning. However, while current methods can achieve vehicle positioning accuracy sufficient for control decisions, the lack of environmental information and the failure to account for perspective distortion caused by camera angle changes often make it difficult to obtain highly accurate camera pose and positioning results. This, to a certain extent, hinders the development of high-precision image measurement technology and higher-level autonomous driving technologies.

[0004] For the above-mentioned problems existing in related technologies, no efficient and accurate solutions have been found yet. Summary of the Invention

[0005] The present invention provides a camera posture compensation calculation method and device, a storage medium, and an electronic device to solve technical problems in related technologies.

[0006] According to one embodiment of the present invention, a method for compensating and calculating a camera pose is provided, comprising: acquiring a first image frame of a reference camera and a second image frame of a target camera; calculating a projection information matrix between the first image frame and the second image frame, and calculating a deformation tensor between the first image frame and the second image frame, and calculating an environmental distortion information matrix based on the deformation tensor; constructing an information matrix using the projection information matrix and the environmental distortion information matrix; and compensating and calculating the camera pose of the target camera using the information matrix.

[0007] Furthermore, calculating the projection information matrix between the first image frame and the second image frame includes: projecting the first image frame to an Euler coordinate system and projecting the second image frame to a Lagrangian coordinate system; calculating the baseline and parallax between the reference camera and the target camera; and using the baseline and the parallax to calculate the projection information matrix between the first image frame and the second image frame.

[0008] Furthermore, calculating the deformation tensor between the first image frame and the second image frame includes: extracting a first feature set of the first image frame, and extracting a second feature set of the second image frame; determining a predefined derivative axis system, wherein the derivative axis system includes: a world coordinate system defined in the image, a dedistorted sensor coordinate system, and a normalized coordinate system under the image; and calculating the deformation tensor between the first feature set and the second feature set based on the derivative axis system.

[0009] Furthermore, calculating the deformation tensor between the first feature set and the second feature set based on the derivative axis system includes: using the second feature set to differentiate the variables in the image normalized coordinate system according to the chain rule to obtain a first intermediate value; using the first intermediate value to differentiate the variables in the image dedistortion sensing coordinate system according to the chain rule to obtain a second intermediate value; using the second intermediate value to differentiate the variables in the image normalized coordinate system according to the chain rule to obtain a third intermediate value; using the third intermediate value to differentiate the variables in the image world coordinate system according to the chain rule to obtain a first derivative value; using the first feature set to differentiate the variables in the image world coordinate system, the image normalized coordinate system, the image dedistortion sensing coordinate system, and the image normalized coordinate system in sequence according to the chain rule to obtain a second derivative value; performing derivative combination on the first derivative value and the second derivative value to obtain a composite result; performing polar decomposition on the composite result to obtain a deformation tensor.

[0010] Furthermore, calculating the environmental distortion information matrix based on the deformation tensor includes: determining whether the deformation tensor is equal to the unit matrix; if the deformation tensor is not equal to the unit matrix, analyzing the characteristic directions of the characteristic points in the second image frame; constructing the strain tensor based on the characteristic directions; performing principal strain analysis on the strain tensor, eliminating the tangential strain, and obtaining a principal strain tensor matrix, wherein the positive values ​​in the principal strain tensor matrix represent tensile deformation and the negative values ​​represent compressive deformation; selecting an environmental deformation correction coefficient, and using the environmental deformation correction coefficient as the Poisson's ratio; calculating the proportional coefficient of the principal strain tensor moment based on the Poisson's ratio and the deformation tensor, and counting the position fluctuation values ​​of the center point of the second image frame under multiple response conditions, and using multiple position fluctuation values ​​to construct a variance; calculating the environmental distortion information matrix according to the variance, the proportional coefficient, and the principal strain tensor matrix.

[0011] Further, calculating the environmental distortion information matrix according to the variance, the proportional coefficient, and the principal strain tensor matrix includes: calculating the environmental distortion information matrix using the following formula: Wherein, V is the proportional coefficient, is the variance, ε 2 is the principal strain tensor matrix, and T is the transpose symbol.

[0012] Furthermore, selecting the environmental deformation correction coefficient includes: obtaining meteorological information of the environment where the target camera is located; and selecting an environmental deformation correction coefficient that matches the meteorological information.

[0013] Furthermore, after constructing an information matrix using the projection information matrix and the environmental distortion information matrix, the method also includes: obtaining an imaging quality parameter of the target camera; determining whether the imaging quality parameter is less than a preset threshold; if the imaging quality parameter is less than the preset threshold, generating a sub-pixel interpolation information matrix of the target camera; compensating the sub-pixel interpolation information matrix in the information matrix, and updating the information matrix.

[0014] Further, using the information matrix to compensate for calculating the camera pose of the target camera includes: in a current iteration cycle of the target camera during the Gauss-Newton iteration, using the following formula to calculate the camera pose Δx of the target camera in the current iteration cycle: Δx=-(J T (σ 2 ) -1 J) -1 J(σ 2 ) -1 u; among them, σ 2 is the information matrix, J is the Jacobian matrix, u is the nonlinear function, and T is the transpose symbol.

[0015] According to another embodiment of the present invention, a compensation calculation device for camera pose is provided, including: an acquisition module for acquiring a first image frame of a reference camera and a second image frame of a target camera; a first calculation module for calculating a projection information matrix between the first image frame and the second image frame, and calculating a deformation tensor between the first image frame and the second image frame, and calculating an environmental distortion information matrix based on the deformation tensor; a construction module for constructing an information matrix using the projection information matrix and the environmental distortion information matrix; and a second calculation module for using the information matrix to compensate and calculate the camera pose of the target camera.

[0016] Furthermore, the first calculation module includes: a projection unit, used to project the first image frame into an Euler coordinate system and project the second image frame into a Lagrangian coordinate system; a first calculation unit, used to calculate the baseline and parallax between the reference camera and the target camera; and a second calculation unit, used to calculate the projection information matrix between the first image frame and the second image frame using the baseline and the parallax.

[0017] Furthermore, the first computing module includes: an extraction unit for extracting a first feature set of the first image frame and a second feature set of the second image frame; a determination unit for determining a predefined derivative axis system, wherein the derivative axis system includes: a world coordinate system defined in the image, a dedistorted sensor coordinate system, and a normalized coordinate system under the image; and a third computing unit for calculating the deformation tensor between the first feature set and the second feature set based on the derivative axis system.

[0018] Furthermore, the third calculation unit is also used to: use the second feature set to differentiate the variables in the image normalized coordinate system according to the chain rule to obtain a first intermediate value; use the first intermediate value to differentiate the variables in the image dedistortion sensing coordinate system according to the chain rule to obtain a second intermediate value; use the second intermediate value to differentiate the variables in the image normalized coordinate system according to the chain rule to obtain a third intermediate value; use the third intermediate value to differentiate the variables in the image world coordinate system according to the chain rule to obtain a first derivative value; use the first feature set to differentiate the variables in the image world coordinate system, the image normalized coordinate system, the image dedistortion sensing coordinate system, and the image normalized coordinate system in sequence according to the chain rule to obtain a second derivative value; perform derivative combination on the first derivative value and the second derivative value to obtain a composite result; perform polar decomposition on the composite result to obtain a deformation tensor.

[0019] Furthermore, the first calculation module includes: a judgment unit for judging whether the deformation tensor is equal to the unit matrix; an analysis unit for analyzing the characteristic direction of the feature point in the second image frame if the deformation tensor is not equal to the unit matrix; a construction unit for constructing a strain tensor based on the characteristic direction; an elimination unit for performing principal strain analysis on the strain tensor, eliminating the tangential strain, and obtaining a principal strain tensor matrix, wherein the positive values ​​in the principal strain tensor matrix represent tensile deformation and the negative values ​​represent compressive deformation; a selection unit for selecting an environmental deformation correction coefficient and using the environmental deformation correction coefficient as the Poisson's ratio; a fourth calculation unit for calculating the proportional coefficient of the principal strain tensor moment based on the Poisson's ratio and the deformation tensor, and counting the position fluctuation values ​​of the center point of the second image frame under multiple response conditions, and using multiple position fluctuation values ​​to construct a variance; a fifth calculation unit for calculating the environmental distortion information matrix based on the variance, the proportional coefficient, and the principal strain tensor matrix.

[0020] Furthermore, the fifth calculation unit includes: a calculation subunit, configured to calculate the environmental distortion information matrix using the following formula: Wherein, V is the proportional coefficient, is the variance, ε 2 is the principal strain tensor matrix, and T is the transpose symbol.

[0021] Furthermore, the selection unit includes: an acquisition subunit for acquiring meteorological information of the environment where the target camera is located; and a selection subunit for selecting an environmental deformation correction coefficient that matches the meteorological information.

[0022] Furthermore, the method also includes: an acquisition module, used to obtain the imaging quality parameters of the target camera after the construction module constructs the information matrix using the projection information matrix and the environmental distortion information matrix; a judgment module, used to judge whether the imaging quality parameters are less than a preset threshold; a generation module, used to generate a sub-pixel interpolation information matrix of the target camera if the imaging quality parameters are less than the preset threshold; and an update module, used to compensate the sub-pixel interpolation information matrix in the information matrix and update the information matrix.

[0023] Furthermore, the second calculation module includes: a calculation subunit, configured to calculate the camera pose Δx of the target camera in the current iteration cycle during the Gauss-Newton iteration of the target camera using the following formula: Δx=-(J T (σ 2 ) -1 J) -1 J(σ 2 ) -1 u; among them, σ 2is the information matrix, J is the Jacobian matrix, u is the nonlinear function, and T is the transpose symbol.

[0024] According to another aspect of an embodiment of the present application, a storage medium is further provided, which includes a stored program, and the above steps are executed when the program is run.

[0025] According to another aspect of an embodiment of the present application, an electronic device is also provided, including a processor, a communication interface, a memory and a communication bus, wherein the processor, the communication interface and the memory communicate with each other through the communication bus; wherein: the memory is used to store computer programs; the processor is used to execute the steps in the above method by running the program stored in the memory.

[0026] An embodiment of the present application also provides a computer program product comprising instructions, which, when executed on a computer, enables the computer to execute the steps in the above method.

[0027] Beneficial effects of the present invention:

[0028] 1. This invention calculates the projection information matrix and the environmental distortion information matrix between two camera frames, and uses this to compensate for the camera pose of the target camera. This compensates for the perspective distortion caused by camera angle changes and the distortion caused by environmental information, thereby improving the accuracy of the camera pose. This solves the technical problem of low camera pose accuracy in related technologies, improves the measurement accuracy in the field of image measurement, and improves the inference positioning accuracy of vehicles.

[0029] 2. The present invention compensates for the harsh test environment by modifying the environmental deformation correction coefficient and adding shear deformation correction, thereby correcting the distortion information caused by weather conditions to prevent overfitting correction;

[0030] 3. The present invention further compensates the sub-pixel interpolation information matrix by considering the information matrix difference caused by sub-pixel interpolation to form the final information matrix and compensate for the influence of low camera pixels. BRIEF DESCRIPTION OF THE DRAWINGS

[0031] The drawings described herein are used to provide a further understanding of the present invention and constitute a part of this application. The exemplary embodiments of the present invention and their descriptions are used to explain the present invention and do not constitute an improper limitation of the present invention. In the drawings:

[0032] Figure 1 This is a hardware structure block diagram of a vehicle according to an embodiment of the present invention;

[0033] Figure 2 is a flowchart of a camera pose compensation calculation method according to an embodiment of the present invention;

[0034] Figure 3is a schematic diagram of the principle of derivation of a coordinate system in an embodiment of the present invention;

[0035] Figure 4 Schematic diagram of the principle of solving the principal strain tensor matrix according to an embodiment of the present invention;

[0036] Figure 5 is an implementation flow chart of an embodiment of the present invention;

[0037] Figure 6 4 is a structural block diagram of a camera pose compensation calculation device according to an embodiment of the present invention. DETAILED DESCRIPTION

[0038] In order to enable those skilled in the art to better understand the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only embodiments of a part of the present application, not all embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without making creative work should fall within the scope of protection of this application. It should be noted that, in the absence of conflict, the embodiments in the present application and the features in the embodiments can be combined with each other.

[0039] It should be noted that the terms "first", "second", etc. in the specification and claims of the present application and the above-mentioned drawings are used to distinguish similar objects and are not necessarily used to describe a specific order or sequential order. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present application described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, for example, a process, method, product or device comprising a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0040] Example 1

[0041] The method embodiment provided in the first embodiment of the present application can be executed in a vehicle, a vehicle controller, a camera, or a similar processing device. Taking the operation on a vehicle as an example, Figure 1 FIG. 1 is a hardware structure diagram of a vehicle according to an embodiment of the present invention. Figure 1 As shown, the vehicle may include one or more ( Figure 1Only one is shown in the figure) a processor 102 (the processor 102 may include but is not limited to a microprocessor MCU or a programmable logic device FPGA and other processing devices) and a memory 104 for storing data. Optionally, the above vehicle may also include a transmission device 106 and an input and output device 108 for communication functions. It will be understood by those skilled in the art that Figure 1 The structure shown is only for illustration and does not limit the structure of the above vehicle. Figure 1 More or fewer components than shown, or with Figure 1 Different configurations shown.

[0042] The memory 104 can be used to store vehicle programs, for example, software programs and modules of application software, such as a vehicle program corresponding to a camera pose compensation calculation method in an embodiment of the present invention. The processor 102 executes various functional applications and data processing by running the vehicle program stored in the memory 104, that is, implementing the above-mentioned method. The memory 104 may include a high-speed random access memory and may also include a non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some examples, the memory 104 may further include a memory remotely located relative to the processor 102, and these remote memories may be connected to the vehicle via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0043] Transmission device 106 is used to receive or transmit data via a network. A specific example of such a network may include a wireless network provided by the vehicle's communications provider. In one embodiment, transmission device 106 includes a network interface controller (NIC), which can connect to other network devices via a base station to enable communication with the Internet. In another embodiment, transmission device 106 may be a radio frequency (RF) module for wireless communication with the Internet.

[0044] In this embodiment, a compensation calculation method for camera posture is provided. Figure 2 is a flowchart of a camera pose compensation calculation method according to an embodiment of the present invention. Figure 2 As shown, the process includes the following steps:

[0045] Step S202 , capturing a first image frame from a reference camera and a second image frame from a target camera;

[0046] In this embodiment, the reference camera and the target camera may be the same camera, such as a camera installed on a vehicle. The first image frame and the second image frame may be time-continuous image frames captured by the camera, or may be discontinuous image frames.

[0047] Step S204, calculating a projection information matrix between the first image frame and the second image frame, calculating a deformation tensor between the first image frame and the second image frame, and calculating an environmental distortion information matrix based on the deformation tensor;

[0048] The physical deformation described in this embodiment refers to the deformation of the same feature set between two different frames. This deformation refers to the difference between camera calibration parameters and the standard projection model. In this embodiment, all corresponding mechanical deformation parameters correspond to the relevant image deformation. The description of the mechanical parameters in this embodiment actually reflects the description of the image features.

[0049] Step S206, constructing an information matrix using the projection information matrix and the environmental distortion information matrix;

[0050] Optionally, the information matrix σ 2 The distortion information matrix can be obtained based on the deformation tensor and the elastic deformation environment and the projection information matrix Composition, can be composed of σ with a weight value of 1:1 according to the linear combination strategy 2 .

[0051] Step S208: Calculate the camera pose of the target camera using information matrix compensation.

[0052] Through the above steps, the first image frame of the reference camera and the second image frame of the target camera are collected, the projection information matrix between the first image frame and the second image frame is calculated, and the deformation tensor between the first image frame and the second image frame is calculated. The environmental distortion information matrix is ​​calculated based on the deformation tensor, and the projection information matrix and the environmental distortion information matrix are used to construct an information matrix. The camera pose of the target camera is calculated using the information matrix compensation. By calculating the projection information matrix and the environmental distortion information matrix between the two frames of camera images and compensating for the camera pose of the target camera, the perspective deformation caused by the change in camera angle and the distortion caused by environmental information are compensated, the accuracy of the camera pose is improved, and the technical problem of low camera pose accuracy in related technologies is solved. The measurement accuracy in the image measurement field can be improved, and the reasoning positioning accuracy of the vehicle is improved.

[0053] In the example of this embodiment, calculating the projection information matrix between the first image frame and the second image frame includes: projecting the first image frame to the Euler coordinate system and projecting the second image frame to the Lagrangian coordinate system; calculating the baseline and parallax between the reference camera and the target camera; and using the baseline and parallax to calculate the projection information matrix between the first image frame and the second image frame.

[0054] The Lagrangian coordinate system of this embodiment is a following coordinate system or a following coordinate system. The reference camera frame image A and the arbitrary target camera frame image B are determined. The Euler coordinate system is defined on the reference camera image A, and the Lagrangian body coordinate system is defined on the arbitrary image B. The projection information matrix is ​​calculated based on the baseline and parallax of the traditional projection relationship.

[0055] In one implementation of this embodiment, calculating the deformation tensor between the first image frame and the second image frame includes: extracting a first feature set of the first image frame, and extracting a second feature set of the second image frame; determining a predefined derivative axis system, wherein the derivative axis system includes: a world coordinate system defined in the image, a dedistorted sensor coordinate system, and a normalized coordinate system under the image; and calculating the deformation tensor between the first feature set and the second feature set based on the derivative axis system.

[0056] In one example, SURF (Speeded Up Robust Features) or other feature extraction methods are used to obtain feature point information in each image.

[0057] Based on the defined relationship between the image projection axes (derivative axis system), the deformation tensor of the feature points between the two frames is calculated. Furthermore, since the deformation assumption assumes that the same feature point between multiple frames is rigid and should not deform, the calculated non-unit deformation tensor matrix is ​​a comprehensive reflection of the environmental information and perspective deformation information that are not considered in this case.

[0058] Optionally, calculating the deformation tensor between the first feature set and the second feature set based on the derivative axis system includes: using the second feature set to differentiate the variables in the image normalized coordinate system according to the chain rule to obtain a first intermediate value; using the first intermediate value to differentiate the variables in the image dedistortion sensing coordinate system according to the chain rule to obtain a second intermediate value; using the second intermediate value to differentiate the variables in the image normalized coordinate system according to the chain rule to obtain a third intermediate value; using the third intermediate value to differentiate the variables in the image world coordinate system according to the chain rule to obtain a first derivative value; using the first feature set to differentiate the variables in the image world coordinate system, the image normalized coordinate system, the image dedistortion sensing coordinate system, and the image normalized coordinate system in sequence according to the chain rule to obtain a second derivative value; performing derivative combination on the first derivative value and the second derivative value to obtain a composite result; performing polar decomposition on the composite result to obtain a deformation tensor.

[0059] In this embodiment, the solution to the deformation tensor is a conversion solution between the Euler coordinate system and the Lagrangian coordinate system. Since the projection relationship between multiple frames of the same object or feature is a process of forward projection and back projection, and involves the definition axes of multiple imaging models, the solution process is actually a process of repeated differentiation and combination of multiple axes. The derivative axis system includes: the world coordinate system defined in the image, the sensor coordinate system after dedistortion, and the normalized coordinate system under the image. The order of the combination process is: first, the image B is derived, and then the image A is derived.

[0060] The deformation tensor of this embodiment describes the deformation of the material itself after the rigid body motion is proposed. In the image, it represents the difference between the same feature set of the image (such as a person or a car in the image) in different projection frames under the same parameters. Figure 3 This is a schematic diagram of the principle of coordinate system differentiation in an embodiment of the present invention. It defines the coordinate system for differentiation. The solution process steps are as follows: first, the feature set u on the B-frame image is differentiated with respect to the normalized image coordinate system, the resulting value is differentiated with respect to the image distortion sensor coordinate system, the resulting value is differentiated with respect to the normalized image coordinate system, and finally the resulting value is differentiated with respect to the image world coordinate system. After completing the derivative for the B-frame, the same set for the A-frame is then differentiated and recombined using the reverse rule according to the compounding principle. Finally, the recombined result is subjected to polar decomposition to obtain the image deformation tensor.

[0061] In one implementation of this embodiment, calculating the environmental distortion information matrix based on the deformation tensor includes: determining whether the deformation tensor is equal to the unit matrix; if the deformation tensor is not equal to the unit matrix, analyzing the characteristic directions of the characteristic points in the second image frame; constructing the strain tensor based on the characteristic directions; performing principal strain analysis on the strain tensor, eliminating the tangential strain, and obtaining the principal strain tensor matrix, wherein the positive values ​​in the principal strain tensor matrix represent tensile deformation and the negative values ​​represent compressive deformation; selecting an environmental deformation correction coefficient, and using the environmental deformation correction coefficient as the Poisson's ratio; calculating the proportional coefficient of the principal strain tensor moment based on the Poisson's ratio and the deformation tensor, and counting the position fluctuation values ​​of the center point of the second image frame under multiple response conditions, and using the multiple position fluctuation values ​​to construct the variance; calculating the environmental distortion information matrix based on the variance, the proportional coefficient, and the principal strain tensor matrix.

[0062] When analyzing the feature directions of feature points in the second image frame in the above embodiment, the number of directions for each feature is determined by the attributes of the feature itself. Whether a feature is composed of one-dimensional feature directions or two-dimensional feature directions is determined. For example, a corner feature is a feature composed of two-dimensional directions. The actual number of directions is determined by the features used.

[0063] Considering the orientation of each feature i in the image, a corresponding "strain tensor" is constructed. Principal strain analysis is performed on the obtained "strain tensor." Using the U transform, the principal strain tensor matrix is ​​obtained. The positive and negative values ​​represent different forms of deformation. Positive values ​​indicate tensile deformation, while negative values ​​indicate compressive deformation. Similar to mechanical analysis, to identify the principal directions, tensor analysis is performed on the "strain matrix" tensor. Using U diagonalization, the principal influencing directions are determined.

[0064] In this embodiment, the constructed "strain tensor" is physically the squared term of strain. The principal strain analysis performed on it eliminates the tangential strain and transforms it into a simple elastic deformation of "unidirectional" tension and compression. Since strain is actually a ratio, it can be used as the corresponding proportional coefficient term in the information matrix.

[0065] Figure 4 : is a diagram showing the principle of solving the principal strain tensor matrix according to an embodiment of the present invention, dx represents the vector connecting point P and point Q, dx' represents the vector connecting point P' and point Q'; n represents the unit vector of vector PQ, n' represents the unit vector of vector P'Q'; F is the deformation gradient; ε 2 is the quadratic term corresponding to the deformation tensor, demonstrating the basic principle of solving the "strain matrix". It can be seen that since the strain is the ratio of the calculated length to the original length, it can be used as a contribution coefficient to distribute the information matrix.

[0066] Considering the elastic deformation assumption, an appropriate deformation correction value is selected as a similar Poisson's ratio μ value, and its proportional coefficient in the information matrix is ​​calculated, which is also called the environmental deformation correction coefficient. It is mainly used to correct the distortion information caused by weather reasons to prevent overfitting correction.

[0067] Optionally, calculating the environmental distortion information matrix based on the variance, the proportional coefficient, and the principal strain tensor matrix includes: calculating the environmental distortion information matrix using the following formula: Where V is the proportionality coefficient, is the variance, ε 2 is the principal strain tensor matrix, and T is the transpose symbol.

[0068] Statistically calculate the response position fluctuation of the center point and construct the variance And substitute the proportional coefficient to obtain the environmental distortion information matrix considering the environment and distortion factors

[0069] In an example based on the above embodiment, selecting the environmental deformation correction coefficient includes: obtaining meteorological information of the environment where the target camera is located; and selecting the environmental deformation correction coefficient that matches the meteorological information.

[0070] Corresponding to the elastic assumption, the environmental deformation correction coefficient is selected. Considering that the deformation of the image feature points should be relatively simple, μ = 0.99 can be selected. If the image situation is more complex, such as considering whether the weather conditions are rainy or foggy, the appropriate environmental deformation correction coefficient can be selected according to the situation, and shear deformation correction can be added to compensate for some of the more severe test environment conditions. The environmental deformation correction coefficient value determines the quality of the image by controlling the deformation form, thereby forming another descriptive value of the environmental impact. A deep learning network based on a deep residual network (ResNet) can be used to select the environmental deformation correction coefficient. Specifically, images of sunny, rainy, and foggy days are collected for the same scene, and the offset of the feature position is taken as the object, and placed in a deep learning network based on ResNet as the skeleton to learn the environmental deformation correction coefficient. It is also possible to construct a second-order deformation function through the SSSIG method and obtain the environmental deformation correction coefficient by matching calculation.

[0071] In one implementation scenario of this embodiment, after constructing the information matrix using the projection information matrix and the environmental distortion information matrix, it also includes: obtaining the imaging quality parameters of the target camera; determining whether the imaging quality parameters are less than a preset threshold; if the imaging quality parameters are less than the preset threshold, generating a sub-pixel interpolation information matrix of the target camera; compensating the sub-pixel interpolation information matrix in the information matrix, and updating the information matrix.

[0072] If we also need to consider the information matrix difference caused by sub-pixel interpolation, we can compensate the sub-pixel interpolation information matrix in the original information matrix to form the final information matrix σ 2 .

[0073] The information matrix can be composed of the distortion information matrix based on the deformation tensor and the elastic deformation environment and the projected information matrix Where possible, the sub-pixel interpolation information matrix can be compensated In general, the preferred linear combination strategy is a 1:1:1 weight value consisting of σ 2 Of course, depending on the quality of the camera and lens used, linear combinations of different proportions can be selected. If the camera has higher pixels, the influence of the interpolation term can even be ignored.

[0074] In this embodiment, the camera pose calculation of the target camera using information matrix compensation includes: in the current iteration cycle of the target camera during the Gauss-Newton iteration process, the camera pose Δx of the target camera in the current iteration cycle is calculated using the following formula: Δx=-(J T (σ 2 ) -1 J) -1 J(σ 2 ) -1 u; among them, σ 2 is the information matrix, J is the Jacobian matrix, u is the nonlinear function, and T is the transpose symbol.

[0075] Following the Gauss-Newton iteration process, the information matrix σ2 is compensated in the iterative update value Δx and the normalization matrix, respectively. This continuous iteration allows for a more accurate camera pose and inference of high-precision vehicle positioning. By supplementing the iterative equation with the information matrix term and solving the camera pose increments as described above, a highly accurate camera pose is obtained. Based on the results from other sensors, the vehicle position or other measurements can be inferred.

[0076] This embodiment provides a high-precision method for estimating the pose of a single camera across multiple frames. This method can improve measurement accuracy in image measurement and improve vehicle inference positioning accuracy in autonomous driving, laying the technical foundation for higher-level autonomous driving. This embodiment uses a deformation tensor and an elastic deformation assumption to provide a high-precision method for estimating the pose of a single camera across multiple frames. This method constructs a deformation tensor, establishes a relationship between it and image deformation, and determines scaling coefficients based on the elastic deformation assumption, ultimately forming an information matrix. This compensates for information matrix terms in the iterative equation for calculating the camera pose. Specifically, the information matrix σ2 is compensated in both the iterative update value Δx and the normalization matrix. This continuous iteration allows for a more accurate camera pose. Furthermore, the coefficients determined based on the elastic deformation assumption can, to a certain extent, represent the environmental conditions at the time of image acquisition. Generally, assuming a good environment, a defined environmental deformation correction coefficient ratio of 0.99 is selected. In situations such as rain, this environmental deformation correction coefficient can be modified, and a shear deformation correction can be added to compensate for harsher test conditions.

[0077] Figure 5 The overall process of a high-precision single-camera pose estimation method between multiple frames based on deformation tensor and elastic deformation assumption is shown. The process includes: determining the A and B frame images, defining the Euler coordinate system; extracting features, and solving the deformation tensor C of the features, Grad(χ)Dx=F, C=F T F, Grad means finding the gradient; determine whether C is equal to I (unit matrix), if so, If not, determine the direction of the feature point and construct the "strain matrix" ε 2 , ε 2 =m T Cm, solve for the proportional coefficient V, ε 2 =VCV T ∈diag(R 2 ), V∈U(R 2*2 ), diag represents a diagonal matrix, U represents a unitary change, R represents a real number set, and m represents a unit vector of the same feature point between two frames; according to the coefficient calculation Whether to consider interpolation, if so, If no,

[0078] This method, based on deformation tensors and elastic deformation assumptions, combines changes in image features with the actual physical deformation. By analyzing deformation and strain relationships, it simultaneously transforms feature information across multiple frames and constructs an information matrix that is missing in traditional methods. This information matrix compensation decisively influences and optimizes the final Gauss-Newton iterative solution, effectively improving camera pose estimation.

[0079] Through the description of the above embodiments, those skilled in the art can clearly understand that the method according to the above embodiment can be implemented by means of software plus the necessary general hardware platform, and of course it can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of the present invention is essentially or the part that contributes to the prior art can be embodied in the form of a software product, which is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk), and includes a number of instructions for enabling a terminal device (which can be a mobile phone, computer, server, or network device, etc.) to execute the methods described in each embodiment of the present invention.

[0080] Example 2

[0081] This embodiment also provides a camera pose compensation calculation device, which is used to implement the above-mentioned embodiments and preferred embodiments. Details that have already been described will not be repeated. As used below, the term "module" may refer to a combination of software and / or hardware that implements a predetermined function. Although the devices described in the following embodiments are preferably implemented in software, implementation in hardware, or a combination of software and hardware, is also possible and contemplated.

[0082] Figure 6 is a structural block diagram of a camera pose compensation calculation device according to an embodiment of the present invention, such as Figure 6 As shown, the device includes:

[0083] An acquisition module 60 is configured to acquire a first image frame from a reference camera and a second image frame from a target camera;

[0084] a first calculation module 62, configured to calculate a projection information matrix between the first image frame and the second image frame, calculate a deformation tensor between the first image frame and the second image frame, and calculate an environmental distortion information matrix based on the deformation tensor;

[0085] A construction module 64 is configured to construct an information matrix using the projection information matrix and the environmental distortion information matrix;

[0086] The second calculation module 66 is configured to calculate the camera pose of the target camera using the information matrix compensation.

[0087] Optionally, the first calculation module includes: a projection unit, used to project the first image frame into an Euler coordinate system, and project the second image frame into a Lagrangian coordinate system; a first calculation unit, used to calculate the baseline and parallax between the reference camera and the target camera; and a second calculation unit, used to calculate the projection information matrix between the first image frame and the second image frame using the baseline and the parallax.

[0088] Optionally, the first computing module includes: an extraction unit for extracting a first feature set of the first image frame, and extracting a second feature set of the second image frame; a determination unit for determining a predefined derivative axis system, wherein the derivative axis system includes: a world coordinate system defined in the image, a dedistorted sensor coordinate system, and a normalized coordinate system under the image; and a third computing unit for calculating the deformation tensor between the first feature set and the second feature set based on the derivative axis system.

[0089] Optionally, the third calculation unit is also used to: use the second feature set to differentiate the variables in the image normalized coordinate system according to the chain rule to obtain a first intermediate value; use the first intermediate value to differentiate the variables in the image dedistortion sensing coordinate system according to the chain rule to obtain a second intermediate value; use the second intermediate value to differentiate the variables in the image normalized coordinate system according to the chain rule to obtain a third intermediate value; use the third intermediate value to differentiate the variables in the image world coordinate system according to the chain rule to obtain a first derivative value; use the first feature set to differentiate the variables in the image world coordinate system, the image normalized coordinate system, the image dedistortion sensing coordinate system, and the image normalized coordinate system in sequence according to the chain rule to obtain a second derivative value; perform derivative and composite on the first derivative value and the second derivative value to obtain a composite result; perform polar decomposition on the composite result to obtain a deformation tensor.

[0090] Optionally, the first calculation module includes: a judgment unit for judging whether the deformation tensor is equal to the unit matrix; an analysis unit for analyzing the characteristic directions of the feature points in the second image frame if the deformation tensor is not equal to the unit matrix; a construction unit for constructing a strain tensor based on the characteristic directions; an elimination unit for performing principal strain analysis on the strain tensor, eliminating tangential strain, and obtaining a principal strain tensor matrix, wherein positive values ​​in the principal strain tensor matrix represent tensile deformation and negative values ​​represent compressive deformation; a selection unit for selecting an environmental deformation correction coefficient and using the environmental deformation correction coefficient as the Poisson's ratio; a fourth calculation unit for calculating the proportional coefficient of the principal strain tensor moment based on the Poisson's ratio and the deformation tensor, and counting the position fluctuation values ​​of the center point of the second image frame under multiple response conditions, and using multiple position fluctuation values ​​to construct a variance; a fifth calculation unit for calculating the environmental distortion information matrix based on the variance, the proportional coefficient, and the principal strain tensor matrix.

[0091] Optionally, the fifth calculation unit includes: a calculation subunit, configured to calculate the environmental distortion information matrix using the following formula: Wherein, V is the proportional coefficient, is the variance, ε2 is the principal strain tensor matrix, and T is the transpose symbol.

[0092] Optionally, the selection unit includes: an acquisition subunit, used to acquire meteorological information of the environment where the target camera is located; and a selection subunit, used to select an environmental deformation correction coefficient that matches the meteorological information.

[0093] Optionally, the method also includes: an acquisition module, used to obtain the imaging quality parameters of the target camera after the construction module constructs the information matrix using the projection information matrix and the environmental distortion information matrix; a judgment module, used to judge whether the imaging quality parameters are less than a preset threshold; a generation module, used to generate the sub-pixel interpolation information matrix of the target camera if the imaging quality parameters are less than the preset threshold; and an update module, used to compensate the sub-pixel interpolation information matrix in the information matrix and update the information matrix.

[0094] Optionally, the second calculation module includes: a calculation subunit, configured to calculate the camera pose Δx of the target camera in the current iteration cycle during the Gauss-Newton iteration of the target camera using the following formula: Δx=-(J T (σ 2 ) -1 J) -1 J(σ 2 ) -1 u; among them, σ 2 is the information matrix, J is the Jacobian matrix, u is the nonlinear function, and T is the transpose symbol.

[0095] It should be noted that the above modules can be implemented through software or hardware. For the latter, it can be implemented in the following ways, but not limited to: the above modules are all located in the same processor; or the above modules are located in different processors in any combination.

[0096] Example 3

[0097] An embodiment of the present invention further provides a storage medium storing a computer program, wherein the computer program is configured to execute the steps of any of the above method embodiments when running.

[0098] Optionally, in this embodiment, the storage medium may be configured to store a computer program for performing the following steps:

[0099] S1, collecting a first image frame of a reference camera and a second image frame of a target camera;

[0100] S2, calculating a projection information matrix between the first image frame and the second image frame, calculating a deformation tensor between the first image frame and the second image frame, and calculating an environment distortion information matrix based on the deformation tensor;

[0101] S3, constructing an information matrix using the projection information matrix and the environmental distortion information matrix;

[0102] S4, using the information matrix to compensate and calculate the camera pose of the target camera.

[0103] Optionally, in this embodiment, the above-mentioned storage medium may include but is not limited to: a USB flash drive, a read-only memory (ROM), a random access memory (RAM), a mobile hard disk, a magnetic disk or an optical disk, and other media that can store computer programs.

[0104] An embodiment of the present invention further provides an electronic device, comprising a memory and a processor, wherein the memory stores a computer program, and the processor is configured to run the computer program to execute the steps in any one of the above method embodiments.

[0105] Optionally, the electronic device may further include a transmission device and an input / output device, wherein the transmission device is connected to the processor, and the input / output device is connected to the processor.

[0106] Optionally, in this embodiment, the processor may be configured to execute the following steps through a computer program:

[0107] S1, collecting a first image frame of a reference camera and a second image frame of a target camera;

[0108] S2, calculating a projection information matrix between the first image frame and the second image frame, calculating a deformation tensor between the first image frame and the second image frame, and calculating an environment distortion information matrix based on the deformation tensor;

[0109] S3, constructing an information matrix using the projection information matrix and the environmental distortion information matrix;

[0110] S4, using the information matrix to compensate and calculate the camera pose of the target camera.

[0111] Optionally, specific examples in this embodiment may refer to the examples described in the above embodiments and optional implementation modes, and this embodiment will not be described in detail here.

[0112] The serial numbers of the above embodiments of the present application are for description only and do not represent the advantages or disadvantages of the embodiments.

[0113] In the above embodiments of the present application, the description of each embodiment has its own focus. For parts that are not described in detail in a certain embodiment, please refer to the relevant description of other embodiments.

[0114] In the several embodiments provided in this application, it should be understood that the disclosed technical content can be implemented in other ways. Among them, the device embodiments described above are only schematic. For example, the division of the units is only a logical function division. In actual implementation, there may be other division methods, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of units or modules, which can be electrical or other forms.

[0115] The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of these units may be selected to achieve the purpose of this embodiment according to actual needs.

[0116] In addition, the functional units in the various embodiments of the present application may be integrated into a single processing unit, or each unit may exist physically separately, or two or more units may be integrated into a single unit. The aforementioned integrated units may be implemented in the form of hardware or software functional units.

[0117] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application is essentially or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions for enabling a computer device (which can be a personal computer, a server or a network device, etc.) to execute all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes: various media that can store program codes, such as a USB flash drive, a read-only memory (ROM), a random access memory (RAM), a mobile hard disk, a magnetic disk or an optical disk.

[0118] The above is only a preferred embodiment of the present application. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principles of the present application. These improvements and modifications should also be regarded as the scope of protection of the present application.

Claims

1. A camera pose compensation calculation method, characterized in that: include: Acquire a first image frame from a reference camera and a second image frame from a target camera; Calculating a projection information matrix between the first image frame and the second image frame, calculating a deformation tensor between the first image frame and the second image frame, and calculating an environmental distortion information matrix based on the deformation tensor; constructing an information matrix using the projection information matrix and the environmental distortion information matrix; The information matrix is ​​used to compensate and calculate the camera pose of the target camera.

2. The method according to claim 1, characterized in that Calculating a projection information matrix between the first image frame and the second image frame includes: Projecting the first image frame into an Euler coordinate system and projecting the second image frame into a Lagrangian coordinate system; Calculating a baseline and a disparity between the reference camera and the target camera; A projection information matrix between the first image frame and the second image frame is calculated using the baseline and the disparity.

3. The method according to claim 1, characterized in that Calculating a deformation tensor between the first image frame and the second image frame includes: extracting a first feature set from the first image frame, and extracting a second feature set from the second image frame; Determining a predefined derivative axis system, wherein the derivative axis system includes: a world coordinate system defined in the image, a sensor coordinate system after dedistortion, and a normalized coordinate system under the image; A deformation tensor between the first feature set and the second feature set is calculated based on the derivative axis system.

4. The method according to claim 3, characterized in that Calculating the deformation tensor between the first feature set and the second feature set based on the derivative axis system includes: Using the second feature set to derive a variable in a normalized image coordinate system according to the chain rule to obtain a first intermediate value; Using the first intermediate value to derive the variable in the image dedistortion sensing coordinate system according to the chain rule to obtain a second intermediate value; Using the second intermediate value to derive the variable in the normalized coordinate system of the image according to the chain rule to obtain a third intermediate value; Using the third intermediate value to derive the variable in the image world coordinate system according to the chain rule to obtain a first derivative value; Using the first feature set, sequentially deriving variables in the image world coordinate system, the image normalized coordinate system, the image dedistortion sensor coordinate system, and the image normalized coordinate system according to the chain rule to obtain a second derivative value; Performing derivative combination on the first derivative value and the second derivative value to obtain a composite result; The composite result is subjected to polar decomposition to obtain a deformation tensor.

5. The method according to claim 1, wherein Calculating the environmental distortion information matrix based on the deformation tensor includes: Determining whether the deformation tensor is equal to the identity matrix; If the deformation tensor is not equal to the identity matrix, analyzing the feature directions of the feature points in the second image frame; constructing a strain tensor based on the characteristic directions; Performing principal strain analysis on the strain tensor, eliminating tangential strain, and obtaining a principal strain tensor matrix, wherein positive values ​​in the principal strain tensor matrix represent tensile deformation, and negative values ​​represent compressive deformation; Selecting an environmental deformation correction coefficient, and using the environmental deformation correction coefficient as the Poisson's ratio; calculating a proportional coefficient of the principal strain tensor moment based on the Poisson's ratio and the deformation tensor, and counting position fluctuation values ​​of the center point of the second image frame under multiple response conditions, and constructing a variance using the multiple position fluctuation values; An environmental distortion information matrix is ​​calculated according to the variance, the proportional coefficient, and the principal strain tensor matrix.

6. The method according to claim 5, characterized in that Select the environmental deformation correction factor including: Obtaining meteorological information of the environment where the target camera is located; An environmental deformation correction coefficient that matches the meteorological information is selected.

7. The method according to claim 1, characterized in that After constructing an information matrix using the projection information matrix and the environmental distortion information matrix, the method further includes: Obtaining imaging quality parameters of the target camera; Determining whether the imaging quality parameter is less than a preset threshold; If the imaging quality parameter is less than a preset threshold, generating a sub-pixel interpolation information matrix of the target camera; The sub-pixel interpolation information matrix is ​​compensated in the information matrix, and the information matrix is ​​updated.

8. The method according to claim 1, characterized in that Calculating the camera pose of the target camera using the information matrix compensation includes: In the current iteration cycle of the target camera during the Gauss-Newton iteration process, the following formula is used to calculate the camera pose of the target camera in the current iteration cycle: Δx : ； in, is the information matrix, J is the Jacobian matrix, u is the nonlinear function, and T is the transpose symbol.

9. A camera pose compensation calculation device, characterized in that: include: An acquisition module, configured to acquire a first image frame from a reference camera and a second image frame from a target camera; a first calculation module, configured to calculate a projection information matrix between the first image frame and the second image frame, calculate a deformation tensor between the first image frame and the second image frame, and calculate an environmental distortion information matrix based on the deformation tensor; A construction module, configured to construct an information matrix using the projection information matrix and the environmental distortion information matrix; The second calculation module is used to calculate the camera pose of the target camera using the information matrix compensation.

10. A storage medium, characterized in that: The storage medium stores a computer program, wherein the computer program is configured to execute the method according to any one of claims 1 to 8 when executed.

11. An electronic device comprising a memory and a processor, characterized in that: A computer program is stored in the memory, and the processor is configured to run the computer program to perform the method according to any one of claims 1 to 8.

Images