A correction method, a terminal device, and a storage medium

Abstract

Embodiments of the present application disclose a correction method, a terminal device and a storage medium. The correction method comprises: searching for a feature point on a first image, and determining a first position of the feature point on the first image and a first center position of the first image; determining a second position of the feature point on a second image and a second center position of the second image; determining a rotation amount of a light field camera along an x-axis, a y-axis and a z-axis of an extended reality device according to the first position and the second position; determining a first translation amount of the light field camera along the z-axis of the extended reality device according to the first position, the second position, the first center position and the second center position; taking four regional images of different orientations from the second image respectively; determining four sharpness values corresponding to the four regional images respectively; determining a second translation amount of the light field camera along the x-axis and the y-axis of the extended reality device according to the four sharpness values; and performing pose correction on the light field camera according to the first translation amount, the second translation amount and the rotation amount.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of camera pose correction, in particular to a correction method, a terminal device and a storage medium. BACKGROUND

[0002] With the vigorous development of the metaverse industry, the detection demand of device manufacturers for extended reality device products is also becoming more and more urgent. As a unique imaging technology in the field of three-dimensional detection, light field cameras can quickly realize high-precision three-dimensional reconstruction through single-frame shooting, and are playing an irreplaceable role in the metaverse industry. When using light field cameras to detect the virtual imaging surface distance of extended reality devices and imaging defects, the light field camera is usually aligned with the lens of the extended reality device for shooting. However, due to the imaging characteristics of the extended reality device, even if the pose of the light field camera only changes a little, the shooting result may be very different. In order to ensure the accuracy and consistency of the detection results when the light field camera detects the extended reality device, the pose of the light field camera should be corrected first when detecting the extended reality device.

[0003] In the prior art, when correcting the pose of the light field camera, the attitude of the light field camera is usually adjusted by observing and judging the image captured by the light field camera with the human eye. However, the subjective judgment result of the human eye deviates from the true shooting result, and does not have high consistency, which is easy to cause large deviation. Moreover, the correction efficiency is low when the camera is calibrated by the subjective judgment of the human eye. SUMMARY

[0004] Therefore, the embodiments of the present application aim to provide a correction method, a terminal device and a storage medium, which can improve the accuracy and correction efficiency of light field camera correction.

[0005] To achieve the above-mentioned purpose, the technical solution of the present application is as follows:

[0006] In a first aspect, the embodiments of the present application provide a correction method, which comprises:

[0007] finding a feature point on a first image, and determining a first position of the feature point on the first image and a first center position of the first image; the first image is an image displayed on a virtual imaging surface of an extended reality device;

[0008] determining a second position of the feature point on a second image and a second center position of the second image; the second image is an image obtained by the light field camera shooting the first image;

[0009] determining the rotation amount of the light field camera along the x-axis, y-axis and z-axis of the extended reality device according to the first position and the second position;

[0010] determine a first translation amount of the light field camera along a z-axis of the extended reality device according to the first position, the second position, the first center position and the second center position;

[0011] obtain four regional images of four orientations respectively from the second image, and determine four sharpness values corresponding to the four regional images of the four orientations respectively;

[0012] determine a second translation amount of the light field camera along an x-axis and a y-axis of the extended reality device according to the four sharpness values;

[0013] correct the pose of the light field camera according to the first translation amount, the second translation amount and the rotation amount.

[0014] In a second aspect, an embodiment of the present application provides a terminal device, which comprises:

[0015] a searching unit, configured to search for a feature point on a first image;

[0016] a determining unit, configured to determine a first position of the feature point on the first image and a first center position of the first image, the first image being an image displayed on a virtual imaging surface of an extended reality device; determine a second position of the feature point on a second image and a second center position of the second image, the second image being an image obtained by capturing the first image by a light field camera; determine a rotation amount of the light field camera along an x-axis, a y-axis and a z-axis of the extended reality device according to the first position and the second position; determine a first translation amount of the light field camera along the z-axis of the extended reality device according to the first position, the second position, the first center position and the second center position; obtain four regional images of four orientations respectively from the second image, and determine four sharpness values corresponding to the four regional images of the four orientations respectively; and determine a second translation amount of the light field camera along the x-axis and the y-axis of the extended reality device according to the four sharpness values;

[0017] a correcting unit, configured to correct the pose of the light field camera according to the first translation amount, the second translation amount and the rotation amount.

[0018] In a third aspect, an embodiment of the present application provides a terminal device, which comprises a processor, a memory and a communication bus; the processor implements the above-mentioned correction method when executing a running program stored in the memory.

[0019] In a fourth aspect, an embodiment of the present application provides a storage medium having a computer program stored thereon, the computer program being executed by a processor to implement the above-mentioned correction method.

[0020] The embodiment of the present application provides a correction method, a terminal device and a storage medium, the method comprises the following steps: searching for a feature point on a first image; determining a first position of the feature point on the first image and a first center position of the first image; the first image is an image displayed on a virtual imaging surface of an extended reality device; determining a second position of the feature point on a second image and a second center position of the second image; the second image is an image obtained by shooting the first image according to a light field camera; determining a rotation amount of the light field camera along an x-axis, a y-axis and a z-axis of the extended reality device according to the first position and the second position; determining a first translation amount of the light field camera along the z-axis of the extended reality device according to the first position, the second position, the first center position and the second center position; taking four regional images of different directions from the second image respectively; and determining four sharpness values corresponding to the four regional images respectively; determining a second translation amount of the light field camera along the x-axis and the y-axis of the extended reality device according to the four sharpness values; and correcting the pose of the light field camera according to the first translation amount, the second translation amount and the rotation amount. In the process of correcting the pose of the light field camera, the first image on the extended reality device is shot by the light field camera to obtain the second image, and the relationship between the first position and the second position determined according to the corresponding feature points extracted from the first image and the second image and the center position on the second image can be used to quickly calculate the rotation amount of the light field camera along the x-axis, the y-axis and the z-axis of the extended reality device and the translation amount of the light field camera along the z-axis of the extended reality device, and the sharpness values of the corresponding four regional images in the second image can be used to quickly determine the translation amount of the light field camera along the x-axis and the y-axis of the extended reality device. The pose of the light field camera is calibrated by using the determined rotation amount or offset amount of the light field camera along the corresponding six degrees of freedom of the extended reality device. In this way, the pose of the light field camera can be automatically calibrated, the inaccuracy of manual visual inspection is avoided, and in the calibration process, the light field camera only needs to shoot one image, so that the accuracy and efficiency of the pose calibration of the light field camera can be greatly improved. BRIEF DESCRIPTION OF DRAWINGS

[0021] Figure 1 A flow of a correction method provided by the embodiment of the present application Figure 1 ;

[0022] Figure 2 An exemplary first image schematic diagram provided by the embodiment of the present application;

[0023] Figure 3 An exemplary second image schematic diagram provided by the embodiment of the present application;

[0024] Figure 4 An exemplary schematic diagram of extracting a feature point on a second image provided by the embodiment of the present application;

[0025] Figure 5 A schematic diagram of calculating a rotation amount when an example light field camera rotates along a y-axis of an extended reality device is provided for an embodiment of the present application;

[0026] Figure 6 A schematic diagram of calculating a translation amount when an example light field camera translates along an x-axis of an extended reality device is provided for an embodiment of the present application;

[0027] Figure 7 A relationship curve diagram of a Sobel definition difference corresponding to a left region image and a right region image and an offset amount of a light field camera along an x-axis direction of an extended reality device is provided for an embodiment of the present application;

[0028] Figure 8 A schematic diagram of a second image captured when a light field camera translates along an x-axis of an extended reality device is provided for an embodiment of the present application;

[0029] Figure 9 A flow of a correction method is provided for an embodiment of the present application Figure 2 ;

[0030] Figure 10 A structure schematic of a terminal device 1 is provided for an embodiment of the present application Figure 1 ;

[0031] Figure 11 A structure schematic of a terminal device 1 is provided for an embodiment of the present application Figure 2 . DETAILED DESCRIPTION

[0032] In order to enable a person skilled in the art to better understand the features and technical contents of the embodiments of the present application, the technical solutions of the present application will be further described in detail below with reference to the accompanying drawings and specific embodiments of the present application. The accompanying drawings are only used for reference and are not intended to limit the embodiments of the present application.

[0033] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which the present application belongs. The terms used herein are only for the purpose of describing the embodiments of the present application and are not intended to limit the present application.

[0034] In the following description, "some embodiments" are referred to, which describe a subset of all possible embodiments, but it can be understood that "some embodiments" can be the same subset or different subsets of all possible embodiments, and can be combined with each other without conflict. It should also be noted that the terms "first / second / third" referred to in the embodiments of the present application are only used to distinguish similar objects, and do not represent a specific order of the objects. Understandably, "first / second / third" can be interchanged with a specific order or sequence as allowed, so that the embodiments of the present application described herein can be implemented in an order other than that illustrated or described herein.

[0035] The correction method provided by the embodiments of the present application can include: Figure 1

[0036] S101, find a feature point on a first image; and determine a first position of the feature point on the first image and a first center position of the first image; the first image is an image displayed on a virtual imaging surface of an extended reality device.

[0037] In the embodiments of the present application, in order to overcome the inaccuracy of manual visual inspection and the low efficiency of calibration when using a light field camera to capture a virtual imaging surface on an extended reality device for light field camera pose calibration in the prior art, a correction method is provided in the embodiments of the present application. Specifically, the pose of the light field camera is calibrated by calculating the rotation amount of the light field camera along the x, y, and z axes of the extended reality device and the translation amount of the light field camera along the x, y, and z axes of the extended reality device.

[0038] It should be noted that the z-axis refers to the optical axis of the extended reality device imaging, the x-axis refers to the horizontal direction of the extended reality device and is perpendicular to the z-axis, and the y-axis refers to the vertical direction of the extended reality device and is perpendicular to the z-axis.

[0039] In the embodiments of the present application, the first image is an image displayed on the virtual imaging surface of the extended reality device, and the displayed image is an image with obvious features in each region of the image, such as Figure 2 the checkerboard image shown in the figure.

[0040] It should be noted that the more obvious features in the checkerboard can be the corner points of the black and white checkerboard and Figure 3 the circle points 1, 2, and 3 shown in the figure, specifically, the actual situation of the first image can be selected, which is not specifically limited in the present application.

[0041] ​In this embodiment of the application, feature points on the first image displayed on the extended reality device are searched, and the positions of the searched feature points on the first image are determined. Specifically, feature point extraction algorithms can be used, such as the network scale-invariant feature transform (SIFT) algorithm, the fast feature point extraction algorithm (ORB), and the feature point extraction algorithm (AKAZE), or a custom matching template.

[0042] It should be noted that the method of feature point search is not limited to the method used in this application. Specifically, it can be selected according to the actual situation, and no specific limitation is made in this application.

[0043] In this embodiment of the application, after extracting feature points in the first image using a feature point extraction algorithm and determining the first position corresponding to the feature points in the first image, it is also necessary to determine the center position point in the first image and obtain its corresponding first center position.

[0044] It should be noted that the first center position of the first image can be obtained from the position corresponding to the searched feature point. That is, the first center position can also be one of the searched feature point positions. The first center position point can also be calculated from the first image. Specifically, the method of obtaining the first center position can be selected according to the actual situation, and no specific limitation is made in this application.

[0045] It should be noted that the feature points found on the first image will all have corresponding locations in the first image.

[0046] S102, Determine the second position of the feature point on the second image and the second center position of the second image; the second image is the image obtained by taking the first image with a light field camera.

[0047] In this embodiment, the second image is obtained by capturing the first image displayed on the virtual imaging surface of the extended reality device using a light field camera, such as... Figure 4 The second image shown is obtained by taking a picture of the first image of the example using a light field camera.

[0048] It should be noted that the second image can specifically be a second image obtained by further processing the first image. The so-called second image is the center view image decoded from the original light field image.

[0049] In the embodiments of this application, such as Figure 4 As shown, the feature points in the second image can be Figure 5 Points 4, 5, 6, and 7 in the circle, namely the midpoint of the hypotenuse of the right triangle inside the circle and the three right-angle vertices, can all be characteristic points.

[0050] In the embodiment of the present application, after the second image is obtained by using the light field camera to capture the first image, the feature points in the second image are extracted by using a feature point extraction algorithm, and the feature points in the first image and the feature points in the second image are matched to determine the feature points in the second image corresponding to the matching feature points in the first image, and the second positions of the feature points in the second image corresponding to the matching feature points in the first image are determined.

[0051] It should be noted that the feature points searched in the first image and the feature points in the second image are one-to-one correspondence, and the first positions of the feature points in the first image corresponding to the second positions of the feature points in the second image are also one-to-one correspondence.

[0052] It should be noted that the feature point extraction algorithm can be the feature point extraction algorithm mentioned in step S101, or can be the feature point extraction algorithm mentioned in step S101, and specifically, it can be selected according to actual conditions, which is not limited in the present application.

[0053] In the embodiment of the present application, after the second position corresponding to the first position of the first image in the second image is determined, the second center position in the second image also needs to be determined.

[0054] It should be noted that the determination of the second center position can be obtained by calculating the second center point of the second image from the second image, and determining the position corresponding to the second center point.

[0055] It should be noted that the second center position can be one of the second positions.

[0056] S103, determining the rotation amount of the light field camera along the x-axis, y-axis and z-axis of the extended reality device according to the first position and the second position.

[0057] In the embodiment of the present application, the rotation amount includes a first rotation amount of the light field camera along the x-axis and y-axis of the extended reality device and a second rotation amount of the light field camera along the z-axis of the extended reality device.

[0058] In the embodiment of the present application, when calculating the first rotation amount of the light field camera along the x-axis and y-axis of the extended reality device and the second rotation amount of the light field camera along the z-axis of the extended reality device, the distance of the connecting line between the first position and the second position and the first angle between the connecting line and the z-axis of the light field camera can be determined; the first rotation amount is determined according to the distance of the connecting line; and the second rotation amount is determined according to the first angle.

[0059] In the embodiment of the present application, taking the rotation of the light field camera along the y-axis of the extended reality device as an example, the first rotation amount of the light field camera along the y-axis of the extended reality device is calculated.

[0060] In the embodiment of the present application, after the light field camera rotates along the y-axis of the extended reality device, as shown in Figure 4 , when the rotation angle θ is small, the actual offset amount O1O3 of the image center can be approximately represented as the product of the working distance OO1 of the camera and the tangent value of the rotation angle θ, that is, as shown in the following formula 1- formula 3:

[0061] O1O3≈O1O2 (1)

[0062] O1O2=OO1×tanθ (2)

[0063] O1O3≈OO1×tanθ (3)

[0064] It should be noted that O1O2 can be quantitatively represented, and the value of O1O2 can be used to approximately represent O1O3.

[0065] It should be noted that O1 and O3 here refer to feature points, which can be the center points of the first image and the second image, that is, the midpoints of the oblique sides of the isosceles right triangle formed by the three positioning circles in Figure 6 .

[0066] It should be noted that the camera working distance OO1 can be calibrated when the scale of the light field camera is calibrated, or it can be obtained by measurement, or it can be calculated. Specifically, it can be selected according to the actual situation, and the present application does not make specific limitations.

[0067] In the embodiment of the present application, from formula (3), the solving method of the rotation amount θ can be derived, as shown in formula (4):

[0068]

[0069] In the embodiment of the present application, when calculating the first rotation amount θ value, the camera working distance of the light field camera can be obtained first; the first rotation amount is determined by using the camera working distance and the distance of the connecting line.

[0070] In the embodiment of the present application, by calculating the value of the actual offset amount O1O3 of the feature point in the second image corresponding to the image center of the feature point of the first image, that is, the distance of the O1O3 connecting line, and obtaining the value of the working distance OO1 of the camera, the rotation amount θ of the light field camera rotating along the y-axis of the extended reality device can be calculated by substituting it into formula (4).

[0071] In the embodiment of the present application, the value of the actual offset amount O1O3 of the image center can be calculated by dividing the pixel offset amount by the lens magnification.

[0072] It should be noted that the pixel offset is the distance between the coordinates of the image center points obtained from the second image and the first image, and the lens magnification is inherent to the light field camera system and can be determined in advance based on the parameters of the light field camera.

[0073] It should be noted that the calculation method for the rotation amount θ of the light field camera along the x-axis of the extended reality device is the same as the calculation method for the rotation amount θ of the light field camera along the y-axis of the extended reality device in this application. Specifically, you can refer to the calculation method for the rotation amount θ of the light field camera along the y-axis of the extended reality device, which will not be repeated here.

[0074] In this embodiment, the calculation of the rotation amount of the light field camera along the z-axis of the extended reality device can be as follows: after determining the feature points of the first image and the second image, determine the angle θ between the line connecting the corresponding feature points in the two images and the z-axis of the extended reality device, and determine the obtained angle θ as the second rotation amount of the light field camera along the z-axis of the extended reality device.

[0075] S104. Based on the first position, the second position, the first center position, and the second center position, determine the first translation amount of the light field camera along the z-axis of the extended reality device.

[0076] In this embodiment of the application, determining the first translation amount of the light field camera along the z-axis of the extended reality device based on the first position, the second position, the first center position, and the second center position may involve determining the first distance between the first position and the first center position; determining the second distance between the second position and the second center position; determining the first scaling amount based on the first distance and the second distance; and determining the first translation amount corresponding to the first scaling amount based on the functional relationship between the scaling amount and the translation amount.

[0077] In this embodiment of the application, the first position is the position of the feature point on the first image, the second position is the position of the feature point on the second image, the first center position is the position of the center point of the first image, and the second center position is the position of the center point of the second image.

[0078] In this embodiment of the application, when calculating the first translation amount of the light field camera along the z-axis of the extended reality device, there can be multiple first and second positions. That is, multiple pairs of corresponding feature points are taken in the first image and the second image, and the first and second positions corresponding to the multiple pairs of feature points are determined respectively. After determining the first center position of the first image and the second center position of the second image, the Euclidean distance between each feature point in the first image and the first center position is calculated, and the Euclidean distance between each feature point in the second image and the second center position is calculated.

[0079] In the embodiments of the present application, after the Euclidean distances between each feature point in the first image and the first center position and the Euclidean distances between each feature point in the second image and the second center position are calculated respectively, the relative scaling amount between the first image and the second image can be obtained by dividing the plurality of Euclidean distances between the plurality of feature points of the first image and the first center position by the plurality of Euclidean distances between the plurality of feature points of the corresponding second image and the second center position respectively and performing an average value operation.

[0080] It should be noted that when dividing, the corresponding Euclidean distances of the corresponding feature points are divided. For example, there are three feature points A, B and C in the first image, and the corresponding feature points of the first image in the second image are D, E and F. The three distances a, b and c between the three feature points of the first image and the first center position in the first image are calculated respectively, and the three distances d, e and f between the three feature points in the second image and the second center position in the second image are calculated. The three values obtained by dividing a by d, b by e and c by f are obtained respectively, and the average value of the three values is calculated, which is the relative scaling amount between the first image and the second image.

[0081] It should be noted that the method of calculating the distance between the feature points of the first image and the first center position and the distance between the feature points of the second image and the second center position is not limited to the Euclidean distance, and can be selected according to the actual situation, which is not limited in the present application.

[0082] In the embodiments of the present application, according to the imaging model of the light field camera, the size of the object photographed by the light field camera will change when the distance of the object is different. Therefore, the relative scaling amount between the first image and the second image has a high correlation with the translation of the light field camera in the z-axis direction of the extended reality device. Therefore, the light field camera can be fixed on a moving device with a known moving distance to perform multiple shooting and calculation, and a functional relationship between the distance of the light field camera translation in the z-axis direction of the extended reality device and the relative scaling amount between the first image and the second image is established, so that in the actual shooting process of the light field camera, the first translation amount of the light field camera in the z-axis direction of the extended reality device can be calculated in real time according to the known translation distance of the light field camera.

[0083] S105, respectively taking four orientation region images from the second image; and determining four sharpness values corresponding to the four orientation region images respectively.

[0084] In the embodiments of the present application, the second image is obtained by the light field camera shooting the first image, wherein when the four orientation region images of the second image are obtained, the four orientations can include the upper, lower, left and right four orientations.

[0085] It should be noted that the up and down and the left and right are corresponding.

[0086] In the embodiment of the present application, when the four clarity values corresponding to the four region images in the four directions are determined, the upper region image, the lower region image, the left region image and the right region image can be taken from the second image respectively; the first clarity value corresponding to the upper region image, the second clarity value corresponding to the lower region image, the third clarity value corresponding to the left region image and the fourth clarity value corresponding to the right region image are determined respectively.

[0087] In the embodiment of the present application, as shown in Figure 7 The movement amount OO' of the light field camera along the x-axis of the extended reality device is equal to the corresponding image center offset amount O1O2, wherein O1O2 is the feature point on the first image and the second image. When the image center offset amount O1O2 is less than the object side resolution capability of the test system, that is, the pixel movement amount of the feature point in the first image to the second image is less than 1, the offset amount of the light field camera cannot be obtained through simple image processing.

[0088] In the embodiment of the present application, the clear imaging region of the extended reality device is extremely small, usually only the region with a width of a few millimeters in front of the lens. Therefore, when the light field camera has displacement along the x-axis and y-axis of the extended reality device, since the corresponding part of the light field camera lens is not in the best imaging area of the lens of the extended reality device, the edge region of the image captured by the light field camera may become blurred, and the blur degree is positively correlated with the degree of offset. Therefore, the offset amount of the light field camera along the x-axis and y-axis of the extended reality device can be calculated by quantifying the blur degree of the edge region of the image.

[0089] In the embodiment of the present application, taking the translation of the light field camera along the x-axis of the extended display device as an example, in the second image, the left region image and the right region image in the second image are taken, wherein the left region image and the right region image are equal in size, symmetric in position, and similar in image.

[0090] It should be noted that the image similarity is to avoid the influence of the pattern when calculating the clarity difference value of the left and right region images.

[0091] In the embodiment of the present application, after the left region image and the right region image in the second image are obtained, the third clarity value of the image in the left region image and the fourth clarity value of the image in the right region image are determined respectively.

[0092] In the embodiments of the present application, the calculation method of the definition value can be any algorithm reflecting the definition of the image, for example, the Sobel gradient algorithm of the whole image region, and the definition value of the image can be obtained by averaging the gradient, or the Laplacian gradient algorithm of the whole image region, and the definition value of the image can be obtained by averaging the gradient.

[0093] It should be noted that the method for calculating the definition value of the image is not limited to the calculation method in the present application, and can be selected according to the actual situation, and the present application is not specifically limited.

[0094] It should be noted that when the light field camera is translated along the y-axis direction of the extended display device, the calculation method of the first definition value and the second definition value corresponding to the upper region image and the lower region image in the second image region can refer to the calculation method of the third definition value and the fourth definition value, which will not be described here.

[0095] In the embodiments of the present application, the third definition value and the fourth definition value corresponding to the left region image and the right region image taken from the second image when the light field camera is translated along the x-axis of the extended reality device, and the first definition value and the second definition value corresponding to the upper region image and the lower region image taken from the second image when the light field camera is translated along the y-axis of the extended reality device can be calculated by the above calculation method.

[0096] S106, determining a second translation amount of the light field camera along the x-axis and the y-axis of the extended reality device according to the four definition values.

[0097] In the embodiments of the present application, the second translation amount includes a first sub-translation amount of the light field camera along the y-axis of the extended reality device and a second sub-translation amount of the light field camera along the x-axis of the extended reality device.

[0098] In the embodiments of the present application, the determination of the first sub-translation amount of the light field camera along the y-axis of the extended reality device can be to determine the first sub-translation amount according to the first definition value and the second definition value.

[0099] In the embodiments of the present application, the specific implementation of determining the first sub-translation amount can be to determine a first difference value between the first definition value and the second definition value, and to determine the first sub-translation amount corresponding to the first difference value from the definition difference value and the translation amount corresponding relationship.

[0100] In the embodiments of the present application, the determination of the second sub-translation amount of the light field camera along the x-axis of the extended reality device can be to determine the second sub-translation amount according to the third definition value and the fourth definition value.

[0101] In the embodiments of the present application, the specific implementation of determining the second sub-translation amount can be determining a second difference value between the third sharpness value and the fourth sharpness value; and determining the second sub-translation amount corresponding to the second difference value from the sharpness difference value-translation amount correspondence.

[0102] In the embodiments of the present application, each displacement amount corresponds to a sharpness difference value, the translation amount and the sharpness difference value are modeled first to obtain a relationship curve between the sharpness difference value and the translation amount, and when the first difference value or the second difference value is obtained in actual operation, the translation amount of the light field camera along the x axis or the y axis of the extended reality device can be determined according to the relationship curve between the sharpness difference value and the offset.

[0103] For example, the relationship between the Sobel sharpness difference value corresponding to the left region image and the right region image and the offset of the light field camera along the x axis direction of the extended reality device is as shown in Figure 7 Figure 7 In the figure, the horizontal axis is the offset of the light field camera along the x axis direction of the extended reality device, which can be controlled by an electric displacement table, and the vertical axis is the difference value of the sharpness of the left and right regions of the image. It can be seen that there is an approximate linear relationship between the two.

[0104] It should be noted that Figure 8 the negative value in the figure represents that the light field camera moves to the left along the x axis of the extended reality device, and the positive value represents that the light field camera moves to the right along the x axis of the extended reality device.

[0105] It should be noted that for the offset of the light field camera along the y axis direction of the extended reality device, the algorithm is similar to that of the light field camera along the x axis direction of the extended reality device, and only the direction of calculating the sharpness needs to be changed from left and right to up and down.

[0106] It should be noted that when the first difference value or the second difference value is close to 0, it means that the light field camera has moved to the best imaging region.

[0107] It should be noted that, in particular, for cameras or camera systems that can provide multiple viewing angles such as light field cameras, left and right viewing angle pictures can be used to replace the left and right regions of the same picture for sharpness calculation.

[0108] For example, taking the translation of the light field camera along the x axis of the extended reality device as an example, as shown in Figure 9 It can be seen that the blurring degree on the right side is obviously higher than that on the left side, so the sharpness calculated on the right side is lower than that on the left side. It can be judged that the camera currently exists a rightward offset, that is, moves along the x axis of the extended reality device, resulting in blurring on the right side.

[0109] S107, pose correction is performed on the light field camera according to the first translation amount, the second translation amount and the rotation amount.

[0110] ​In the embodiment of the present application, the first translation amount is the translation amount of the light field camera along the z-axis of the extended reality device, the second translation amount is the translation amount of the light field camera along the x-axis and y-axis of the extended reality device, and the rotation amount is the rotation amount of the light field camera along the x-axis, y-axis and z-axis of the extended reality device, so that the transformation amount of the pose of the light field camera along the six degrees of freedom of the extended reality device when the extended reality device is photographed can be determined by one shooting of the light field camera, and the light field camera is corrected in pose according to the first translation amount, the second translation amount and the rotation amount corresponding to the transformation amount, so as to detect the three-dimensional device by using the light field camera after the pose correction.

[0111] It can be understood that in the correction method provided in the embodiment of the present application, in the process of correcting the pose of the light field camera, the first image on the extended reality device is photographed by using the light field camera to obtain the second image, and the relationship between the first position and the second position determined according to the corresponding feature points extracted from the first image and the second image and the center position on the second image can be used to quickly calculate the rotation amount of the light field camera along the x-axis, y-axis and z-axis of the extended reality device respectively and the translation amount of the light field camera along the z-axis of the extended reality device, and the clarity values of the region images corresponding to the four directions in the second image can be used to quickly determine the translation amount of the light field camera along the x-axis and y-axis of the extended reality device respectively, and the pose of the light field camera is calibrated by using the determined rotation amount or offset amount corresponding to the six degrees of freedom of the extended reality device respectively, so that the pose of the light field camera can be automatically calibrated, the inaccuracy of manual visual inspection is avoided, and in the calibration process, the light field camera only needs to shoot one image, so that the accuracy and efficiency of the pose calibration of the light field camera can be greatly improved.

[0112] Based on the above embodiment, the correction method provided in the present application specifically includes the following steps as shown in the following table: Figure 10

[0113] Step 1, finding feature points on the first image; and determining the first position of the feature points on the first image and the first center position of the first image;

[0114] Step 2, determining the second position of the feature points on the second image and the second center position of the second image;

[0115] Step 3, determining the distance of the connecting line between the first position and the second position and the first included angle between the connecting line and the z-axis of the light field camera;

[0116] Step 4, determining the first rotation amount according to the distance of the connecting line;

[0117] Step 5, determining the second rotation amount according to the first included angle;

[0118] ​Step 6, determining a first distance between the first position and the first center position; determining a second distance between the second position and the second center position;

[0119] Step 7, determining a first scaling amount according to the first distance and the second distance;

[0120] Step 8, determining a first translation amount corresponding to the first scaling amount according to a functional relationship between the scaling amount and the translation amount;

[0121] Step 9, taking an upper region image, a lower region image, a left region image and a right region image from the second image respectively;

[0122] Step 10, determining a first sharpness value corresponding to the upper region image, a second sharpness value corresponding to the lower region image, a third sharpness value corresponding to the left region image and a fourth sharpness value corresponding to the right region image respectively;

[0123] Step 11, determining a first difference value between the first sharpness value and the second sharpness value; determining a first sub-translation amount corresponding to the first difference value from a correspondence relationship between the sharpness difference value and the translation amount;

[0124] Step 12, determining a second difference value between the third sharpness value and the fourth sharpness value; determining a second sub-translation amount corresponding to the second difference value from the correspondence relationship between the sharpness difference value and the translation amount;

[0125] Step 13, performing pose correction on the light field camera according to the first translation amount, the second translation amount and the rotation amount.

[0126] It should be noted that in the specific calculation process, the execution order of the above steps is not limited to the execution order in the present application, and specifically, the execution step order can be adjusted according to the actual situation, and the present application does not make specific limitations.

[0127] Based on the above embodiment, in another embodiment of the present application, a terminal device 1 is provided, as shown in Figure 11 The terminal device 1 comprises:

[0128] A searching unit 10 is configured to search for a feature point on the first image.

[0129] The determining unit 11 is configured to determine a first position of a feature point on a first image and a first center position of the first image, the first image being an image displayed on a virtual imaging surface of an extended reality device; determine a second position of the feature point on a second image and a second center position of the second image, the second image being an image obtained by capturing the first image by using a light field camera; determine a rotation amount of the light field camera along an x-axis, a y-axis and a z-axis of the extended reality device according to the first position and the second position; determine a first translation amount of the light field camera along the z-axis of the extended reality device according to the first position, the second position, the first center position and the second center position; obtain four region images in four directions from the second image respectively; and determine four sharpness values corresponding to the four region images respectively; and determine a second translation amount of the light field camera along the x-axis and the y-axis of the extended reality device according to the four sharpness values.

[0130] The correcting unit 12 is configured to perform pose correction on the light field camera according to the first translation amount, the second translation amount and the rotation amount.

[0131] Optionally, the rotation amount includes a first rotation amount of the light field camera along the x-axis and the y-axis of the extended reality device and a second rotation amount of the light field camera along the z-axis of the extended reality device.

[0132] Optionally, the determining unit 11 is further configured to determine a distance of a line connecting the first position and the second position and a first included angle between the line and the z-axis of the light field camera; determine the first rotation amount according to the distance of the line; and determine the second rotation amount according to the first included angle.

[0133] Optionally, the terminal device can further include an obtaining unit,

[0134] The obtaining unit is configured to obtain a camera working distance of the light field camera.

[0135] Optionally, the determining unit 11 is further configured to determine the first rotation amount by using the camera working distance and the distance of the line.

[0136] Optionally, the determining unit 11 is further configured to determine a first distance between the first position and the first center position; determine a second distance between the second position and the second center position; determine a first scaling amount according to the first distance and the second distance; and determine the first translation amount corresponding to the first scaling amount according to a functional relationship between the scaling amount and the translation amount.

[0137] Optionally, the determining unit 11 is further configured to obtain an upper region image, a lower region image, a left region image and a right region image from the second image respectively; determine a first sharpness value corresponding to the upper region image, a second sharpness value corresponding to the lower region image, a third sharpness value corresponding to the left region image and a fourth sharpness value corresponding to the right region image respectively.

[0138] Optionally, the second translation amount includes: a first sub-translation amount of the light field camera along a y-axis of the extended reality device, and a second sub-translation amount of the light field camera along an x-axis of the extended reality device.

[0139] Optionally, the determining unit 11 is further configured to determine the first sub-translation amount according to the first sharpness value and the second sharpness value, and determine the second sub-translation amount according to the third sharpness value and the fourth sharpness value.

[0140] Optionally, the determining unit 11 is further configured to determine a first difference value between the first sharpness value and the second sharpness value, and determine the first sub-translation amount corresponding to the first difference value from the correspondence between the sharpness difference value and the translation amount.

[0141] Optionally, the determining unit 11 is further configured to determine a second difference value between the third sharpness value and the fourth sharpness value, and determine the second sub-translation amount corresponding to the second difference value from the correspondence between the sharpness difference value and the translation amount.

[0142] The embodiment of the application provides a terminal device, which finds a feature point on a first image, and determines a first position of the feature point on the first image and a first center position of the first image; the first image is an image displayed on a virtual imaging surface of an extended reality device; a second position of the feature point on a second image and a second center position of the second image are determined; the second image is an image obtained by capturing the first image according to a light field camera; a rotation amount of the light field camera along an x-axis, a y-axis and a z-axis of the extended reality device is determined according to the first position and the second position; a first translation amount of the light field camera along the z-axis of the extended reality device is determined according to the first position, the second position, the first center position and the second center position; four regional images of four directions are taken from the second image respectively; and four sharpness values corresponding to the four regional images of the four directions are determined respectively; a second translation amount of the light field camera along the x-axis and the y-axis of the extended reality device is determined according to the four sharpness values; and the pose of the light field camera is corrected according to the first translation amount, the second translation amount and the rotation amount. As can be seen, the terminal device provided in the embodiment of the application can quickly calculate the rotation amount of the light field camera along the x-axis, the y-axis and the z-axis of the extended reality device and the translation amount of the light field camera along the z-axis of the extended reality device by using the first image captured by the light field camera on the extended reality device to obtain the second image, and the relationship between the first position and the second position of the corresponding feature points extracted from the first image and the second image and the center position on the second image, and can quickly determine the translation amount of the light field camera along the x-axis and the y-axis of the extended reality device by using the sharpness values of the corresponding four regional images of the four directions in the second image, and calibrate the pose of the light field camera by using the determined rotation amount or offset amount of the light field camera along the corresponding six degrees of freedom of the extended reality device. In this way, the pose of the light field camera can be automatically calibrated, the inaccuracy of manual visual inspection is avoided, and in the calibration process, the light field camera only needs to capture one image, so that the accuracy and calibration efficiency of the pose calibration of the light field camera can be greatly improved.

[0143] Figure 11 A schematic diagram of the composition structure of the terminal device 1 provided in the embodiment of the application is shown in FIG. 1. In actual application, based on the same disclosure concept of the above-mentioned embodiment, as shown in FIG. 1, the terminal device 1 of the embodiment comprises a processor 13, a memory 14 and a communication bus 15. ​

[0144] ​In the specific embodiment, the search unit 10, the determination unit 11, the correction unit 12, and the acquisition unit can be implemented by a processor 13 on the terminal device 1. The processor 13 can be at least one of an application specific integrated circuit (ASIC), a digital signal processor (DSP), a digital signal processing device (DSPD), a programmable logic device (PLD), a field programmable gate array (FPGA), a CPU, a controller, a microcontroller, or a microprocessor. It can be understood that, for different devices, the electronic device used to implement the processor can also be other devices, which is not limited in the embodiment.

[0145] In the embodiment of the present application, the communication bus 15 is used to realize the connection and communication between the processor 13 and the memory 14. When the processor 13 executes the running program stored in the memory 14, the following correction method is realized:

[0146] searching for a feature point on a first image, and determining a first position of the feature point on the first image and a first center position of the first image, the first image being an image displayed on a virtual imaging surface of an extended reality device; determining a second position of the feature point on a second image and a second center position of the second image, the second image being an image obtained by capturing the first image by using a light field camera; determining a rotation amount of the light field camera along an x-axis, a y-axis, and a z-axis of the extended reality device according to the first position and the second position; determining a first translation amount of the light field camera along the z-axis of the extended reality device according to the first position, the second position, the first center position, and the second center position; obtaining four regional images in four directions from the second image respectively, and determining four sharpness values corresponding to the four regional images respectively; determining a second translation amount of the light field camera along the x-axis and the y-axis of the extended reality device according to the four sharpness values; and correcting a pose of the light field camera according to the first translation amount, the second translation amount, and the rotation amount.

[0147] Further, the rotation amount includes a first rotation amount of the light field camera along the x-axis and the y-axis of the extended reality device, and a second rotation amount of the light field camera along the z-axis of the extended reality device.

[0148] Further, the processor 13 is further configured to determine a distance of a line connecting the first position and the second position, and a first included angle between the line and the z-axis of the light field camera; determine the first rotation amount according to the distance of the line; and determine the second rotation amount according to the first included angle.

[0149] Further, the processor 13 is further configured to obtain a camera working distance of the light field camera; and determine the first rotation amount based on the camera working distance and the distance of the line.

[0150] Further, the processor 13 is further configured to determine a first distance between the first position and the first center position; determine a second distance between the second position and the second center position; determine the first scaling amount based on the first distance and the second distance; and determine the first translation amount corresponding to the first scaling amount based on a functional relationship between the scaling amount and the translation amount.

[0151] Further, the processor 13 is further configured to obtain an upper region image, a lower region image, a left region image and a right region image from the second image; determine a first sharpness value corresponding to the upper region image, a second sharpness value corresponding to the lower region image, a third sharpness value corresponding to the left region image and a fourth sharpness value corresponding to the right region image.

[0152] Further, the second translation amount includes a first sub-translation amount of the light field camera along a y-axis of the extended reality device and a second sub-translation amount of the light field camera along an x-axis of the extended reality device.

[0153] Further, the processor 13 is further configured to determine the first sub-translation amount based on the first sharpness value and the second sharpness value; and determine the second sub-translation amount based on the third sharpness value and the fourth sharpness value.

[0154] Further, the processor 13 is further configured to determine a first difference value between the first sharpness value and the second sharpness value; and determine the first sub-translation amount corresponding to the first difference value from a correspondence relationship between a sharpness difference value and a translation amount.

[0155] Further, the processor 13 is further configured to determine a second difference value between the third sharpness value and the fourth sharpness value; and determine the second sub-translation amount corresponding to the second difference value from the correspondence relationship between the sharpness difference value and the translation amount.

[0156] Based on the above embodiments, the embodiments of the present application provide a storage medium having a computer program stored thereon, the computer readable storage medium stores one or more programs, the one or more programs can be executed by one or more processors, and the one or more programs are applied to a terminal device, and the computer program implements the correction method as described above.

[0157] It should be noted that, in the present document, the terms "comprising", "including", or any other variant thereof are intended to cover a non-exclusive inclusion, such that processes, methods, articles, or apparatuses that comprise a list of elements are not limited to those elements, but can also include other elements not expressly listed, or can also include elements inherent in such processes, methods, articles, or apparatuses. Without more limitations, an element defined by the phrase "comprising a" does not exclude the presence of additional identical elements in the process, method, article, or apparatus that includes the element.

[0158] From the above description of the embodiments, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be realized by means of software and necessary general hardware platforms, and of course, they can also be realized by hardware, but in many cases, the former is a better embodiment. Based on such understanding, the technical solutions of the present disclosure can be embodied in the form of a software product in essence or in the part that contributes to the related art, and the computer software product is stored in a storage medium (such as a ROM / RAM, a magnetic disk, or an optical disk), and includes a plurality of instructions for causing an image display device (which can be a mobile phone, a computer, a server, an air conditioner, or a network device, etc.) to execute the methods described in various embodiments of the present disclosure.

[0159] The above description is merely specific embodiments of the present application, but the protection scope of the present application is not limited thereto. Any person skilled in the art can easily think of changes or replacements within the technical scope disclosed in the present application, which should be covered within the protection scope of the present application. Therefore, the protection scope of the present application should be subject to the protection scope of the claims.

Claims

1. A correction method characterized by, The method comprises: finding a feature point on a first image; and determining a first position of the feature point on the first image and a first center position of the first image; the first image is an image displayed on a virtual imaging surface of an extended reality device; determining a second position of the feature point on a second image and a second center position of the second image; the second image is an image obtained by shooting the first image according to a light field camera; determining a rotation amount of the light field camera along the x-axis, y-axis and z-axis of the extended reality device according to the first position and the second position; determining a first translation amount of the light field camera along the z-axis of the extended reality device according to the first position, the second position, the first center position and the second center position; taking four area images of different directions from the second image respectively; and determining four sharpness values corresponding to the four area images respectively; determining a second translation amount of the light field camera along the x-axis and y-axis of the extended reality device according to the four sharpness values; wherein the second translation amount is determined based on a first difference value and a second difference value of the four sharpness values and a sharpness difference value and translation amount corresponding relationship, and the first difference value and the second difference value are respectively the difference between the sharpness values corresponding to each pair of opposite directions of the four area images; positioning and correcting the light field camera according to the first translation amount, the second translation amount and the rotation amount.

2. The method of claim 1, wherein, The rotation amount comprises a first rotation amount of the light field camera along the x-axis and y-axis of the extended reality device and a second rotation amount of the light field camera along the z-axis of the extended reality device; and the determining the rotation amount of the light field camera along the x-axis, y-axis and z-axis of the extended reality device according to the first position and the second position comprises: determining the distance of the connecting line between the first position and the second position and the first included angle between the connecting line and the z-axis of the light field camera; determining the first rotation amount according to the distance of the connecting line; determining the second rotation amount according to the first included angle.

3. The method of claim 2, wherein, The determining the first rotation amount according to the distance of the connecting line comprises: obtaining the camera working distance of the light field camera; determining the first rotation amount by using the camera working distance and the distance of the connecting line.

4. The method of claim 1, wherein, The determining the first translation amount of the light field camera along the z-axis of the extended reality device according to the first position, the second position, the first center position and the second center position comprises: determining a first distance between the first position and the first center position; determining a second distance between the second position and the second center position; determining a first scaling amount according to the first distance and the second distance; determining the first translation amount corresponding to the first scaling amount according to the functional relationship between scaling amount and translation amount.

5. The method of claim 1, wherein, The four directions include up, down, left and right; and the taking four area images of different directions from the second image respectively comprises: taking an upper area image, a lower area image, a left area image and a right area image from the second image respectively; ​ The first sharpness value corresponding to the upper region image, the second sharpness value corresponding to the lower region image, the third sharpness value corresponding to the left region, and the fourth sharpness value corresponding to the right region are determined respectively.

6. The method of claim 5, wherein, The second translation amount includes: a first sub-translation of the light field camera along the y-axis of the extended reality device and a second sub-translation of the light field camera along the x-axis of the extended reality device; determining the second translation amount of the light field camera along the x-axis and y-axis of the extended reality device based on the four sharpness values ​​includes: The first sub-translation amount is determined based on the first resolution value and the second resolution value; The second sub-translation amount is determined based on the third and fourth resolution values.

7. The method of claim 6, wherein, Determining the first sub-translation amount based on the first sharpness value and the second sharpness value includes: Determine a first difference between the first sharpness value and the second sharpness value; From the correspondence between the sharpness difference and the translation amount, determine the first sub-translation amount corresponding to the first difference; Accordingly, determining the second sub-translation amount based on the third and fourth resolution values ​​includes: Determine a second difference between the third sharpness value and the fourth sharpness value; The second sub-translation amount corresponding to the second difference is determined from the correspondence between the sharpness difference and the translation amount.

8. A terminal device, comprising: The terminal device includes: A search unit is used to find feature points on the first image; A determining unit is configured to determine the first position of the feature point on the first image and the first center position of the first image; the first image is an image displayed on the virtual imaging surface of the extended reality device; determine the second position of the feature point on a second image and the second center position of the second image; the second image is an image obtained by capturing the first image using a light field camera; determine the rotation amount of the light field camera along the x-axis, y-axis, and z-axis of the extended reality device based on the first position and the second position; determine the first translation amount of the light field camera along the z-axis of the extended reality device based on the first position, the second position, the first center position, and the second center position; extract four regional images from the second image; and determine four sharpness values ​​corresponding to the four regional images; determine the second translation amount of the light field camera along the x-axis and y-axis of the extended reality device based on the four sharpness values; wherein the second translation amount is determined based on a first difference and a second difference determined by the four sharpness values, and the correspondence between the sharpness difference and the translation amount, and the first difference and the second difference are the difference in sharpness values ​​corresponding to each pair of opposite directions in the four regional images; The correction unit is used to perform pose correction on the light field camera based on the first translation amount, the second translation amount, and the rotation amount.

9. A terminal device, comprising: The terminal device includes: a processor, a memory, and a communication bus; when the processor executes the running program stored in the memory, it implements the method as described in any one of claims 1-7.

10. A storage medium having stored thereon a computer program, characterized in that The computer program, which when executed by the processor, implements the method as claimed in any of claims 1-7.

Images