Picture processing method and device, storage medium and electronic equipment
By calculating the weight matrix of the picture data in the augmented reality device and performing interpolation processing, the distortion function is obtained, which solves the problems of complex picture distortion processing steps and insufficient versatility in the existing technology, and achieves efficient picture de-distortion processing.
Patent Information
- Application Number
- CN202310880618.0
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2023-07-17
- Publication Date
- 2025-01-21
AI Technical Summary
The prior art has complex steps and insufficient general use when reducing picture distortion in augmented reality devices.
By obtaining the distorted picture data and the calibrated picture data, the weight matrix is calculated, and interpolation processing is performed based on this to obtain the distortion function, thereby realizing the de-distortion processing of the picture data.
It improves the efficiency and versatility of image data de-distortion processing and provides users with undistorted image data.
Smart Images
Figure CN119338725A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of computer technology, and in particular to a picture processing method, device, storage medium and electronic device. Background Art
[0002] Nowadays, with the continuous development of computer technology and augmented reality technology, augmented reality devices have gradually become products with practical application value in reality. In the process of using augmented reality devices, picture distortion will inevitably occur, affecting the user experience. Summary of the invention
[0003] The embodiments of this specification provide a picture processing method, device, storage medium and electronic device, which can perform dedistortion processing on the acquired picture data, provide the user with undistorted picture data, and then obtain the distortion function of the terminal device according to calculation, thereby improving the efficiency and versatility of the picture data dedistortion processing.
[0004] In a first aspect, an embodiment of the present specification provides a method for processing a picture, the method comprising:
[0005] Acquire the collected distorted image data, and acquire the calibration image data corresponding to the distorted image data;
[0006] Acquire calibration coordinates corresponding to the calibration picture data and distortion coordinates corresponding to the distortion picture data;
[0007] Calculating a weight matrix based on the calibration coordinates and the distortion coordinates;
[0008] The distorted coordinates are interpolated based on the weight matrix to obtain a distortion function corresponding to the distorted picture data, and the distortion function is used to obtain target picture data corresponding to the distorted picture data.
[0009] In a second aspect, an embodiment of the present specification provides a picture processing device, the device comprising:
[0010] A data acquisition unit, used to acquire the collected distorted image data and acquire the calibration image data corresponding to the distorted image data;
[0011] A coordinate acquisition unit, used to acquire calibration coordinates corresponding to the calibration picture data and distortion coordinates corresponding to the distortion picture data;
[0012] A matrix acquisition unit, used for calculating a weight matrix based on the calibration coordinates and the distortion coordinates;
[0013] A function acquisition unit is used to perform interpolation processing on the distorted coordinates based on the weight matrix to obtain a distortion function corresponding to the distorted picture data, and the distortion function is used to obtain target picture data corresponding to the distorted picture data.
[0014] In a third aspect, an embodiment of the present specification provides a computer storage medium, wherein the computer storage medium stores a plurality of instructions, wherein the instructions are suitable for being loaded by a processor and executing the steps of the above method.
[0015] In a fourth aspect, an embodiment of the present specification provides an electronic device, comprising: a processor and a memory; wherein the memory stores a computer program, and the computer program is suitable for being loaded by the processor and executing the steps of the above method.
[0016] In the embodiments of the present specification, by acquiring distorted picture data and calibrated picture data, calibrated coordinates and distorted coordinates corresponding to the picture data are acquired for calculation, a weight matrix corresponding to the distorted picture data is obtained, and then interpolation processing is performed based on the weight matrix to obtain a distortion function corresponding to the distorted picture data, so as to obtain target picture data according to the distortion function, thereby achieving de-distortion processing of the acquired picture data, providing users with undistorted picture data, and then obtaining the distortion function of the terminal device according to the calculation, thereby improving the efficiency and versatility of de-distortion processing of the picture data. BRIEF DESCRIPTION OF THE DRAWINGS
[0017] In order to more clearly illustrate the technical solutions in this specification or the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this specification. For those skilled in the art, other drawings can be obtained based on these drawings without paying any creative work.
[0018] Figure 1 A system architecture diagram of a picture processing method provided in an embodiment of this specification;
[0019] Figure 2 A flowchart of a method for processing a picture provided by an embodiment of this specification;
[0020] Figure 3 This is a schematic diagram of an example of distorted image data provided in an embodiment of this specification;
[0021] Figure 4 This is an example schematic diagram of a calibration screen data provided by an embodiment of this specification;
[0022] Figure 5 An example schematic diagram of an interpolation process provided in an embodiment of this specification;
[0023] Figure 6 This is a schematic diagram of an example of obtaining target screen data provided by an embodiment of this specification;
[0024] Figure 7 A flowchart of a method for processing a picture provided by an embodiment of this specification;
[0025] Figure 8 This is a schematic diagram of an example of distorted image data provided in an embodiment of this specification;
[0026] Fig. 9 An example schematic diagram of an inverse distortion processing provided in an embodiment of this specification;
[0027] Fig.10 A schematic diagram of the structure of a picture processing device provided in an embodiment of this specification;
[0028] Fig.11 A schematic diagram of the structure of a matrix acquisition unit provided in an embodiment of this specification;
[0029] Fig.12 A schematic diagram of the structure of a picture processing device provided in an embodiment of this specification;
[0030] Fig.13 A schematic diagram of the structure of an inverse distortion unit provided in an embodiment of this specification;
[0031] Fig.14 A schematic diagram of the structure of an electronic device provided in an embodiment of this specification. DETAILED DESCRIPTION
[0032] In order to make the features and advantages of this specification more obvious and easy to understand, the technical solutions in this specification are described clearly and completely in conjunction with the drawings in this specification. Obviously, the described embodiments are only part of the embodiments of this specification, not all of them. Based on the embodiments in this specification, all other embodiments obtained by those skilled in the art without creative work are within the scope of protection of this specification.
[0033] In the prior art, when reducing the distortion of the image, the method used requires understanding the component structure of the device in order to perform the de-distortion process, which results in more complicated steps and insufficient versatility.
[0034] Based on this, an embodiment of the present specification provides a picture processing method. Using the embodiment of the present specification, by acquiring distorted picture data and calibrated picture data, the calibrated coordinates and the distorted coordinates corresponding to the picture data are acquired for calculation, and a weight matrix corresponding to the distorted picture data is obtained. Then, interpolation processing is performed based on the weight matrix to obtain a distortion function corresponding to the distorted picture data, so as to obtain target picture data according to the distortion function, thereby achieving de-distortion processing of the acquired picture data, providing users with undistorted picture data, and then obtaining the distortion function of the terminal device according to the calculation, thereby improving the efficiency and versatility of de-distortion processing of the picture data.
[0035] See also Figure 1 , which is a system structure diagram of a picture processing according to the embodiment of this specification. Figure 1 As shown, the picture processing method provided in the embodiment of this specification can be applied to a terminal device to implement the picture processing process. The system structure provided in the embodiment of this specification mainly includes a terminal device 10, a camera device 20 and a computing terminal 30. Among them, the terminal device 10 can be an enhanced display device, etc.; the camera device 20 can be a device with a shooting function, such as a camera, a smart phone, a smart tablet, etc.; the computing terminal 30 can be a terminal with a computing function, such as a server, or a terminal such as a laptop computer and a desktop computer.
[0036] In the embodiment of the present specification, the camera device 20 obtains the distorted image data displayed by the terminal device 10, the computing terminal 30 obtains the distorted image data collected by the camera device 20, obtains the calibration image data corresponding to the distorted image data, obtains the calibration coordinates of the calibration image data and the distorted coordinates of the distorted image data, calculates a weight matrix based on the calibration coordinates and the distorted coordinates, and interpolates the distorted coordinates based on the weight matrix to obtain a distortion function corresponding to the distorted image data, and the distortion function is the distortion function corresponding to the terminal device 10.
[0037] In the embodiments of the present specification, by acquiring distorted picture data and calibrated picture data, calibrated coordinates and distorted coordinates corresponding to the picture data are acquired for calculation, a weight matrix corresponding to the distorted picture data is obtained, and then interpolation processing is performed based on the weight matrix to obtain a distortion function corresponding to the distorted picture data, so as to obtain target picture data according to the distortion function, thereby achieving de-distortion processing of the acquired picture data, providing users with undistorted picture data, and then obtaining the distortion function of the terminal device according to the calculation, thereby improving the efficiency and versatility of de-distortion processing of the picture data.
[0038] based on Figure 1 The system architecture shown below will be combined with Figure 2 , the picture processing method provided in the embodiments of this specification is introduced in detail.
[0039] See also Figure 2 , is a flowchart of a method for processing a picture provided by an embodiment of this specification. Figure 2 As shown, the method may include the following steps S101-S104.
[0040] S101, acquiring collected distorted image data, and acquiring calibration image data corresponding to the distorted image data;
[0041] In one embodiment, a calibrated camera device is used to shoot a terminal device, collect distorted image data, and obtain calibrated image data corresponding to the distorted image data.
[0042] The distorted image data may be the image data generated by the terminal device, which is distorted when the image data passes through the optical element in the terminal device. For example, it may be: Figure 3 As shown, it can be seen Figure 3 The frame of the distorted image data in the image is distorted. It is understandable that the image is inevitably distorted due to the structure and assembly of the optical element itself.
[0043] It should be noted that the calibrated camera device may have undergone distortion correction, and the image data captured by the camera device is undistorted. Therefore, when calculations are performed based on the image data obtained after shooting with the calibrated camera device, it is possible to avoid the accuracy of subsequent calculations being affected by errors caused by the image data obtained by shooting.
[0044] It can be understood that in order to reduce the difficulty of data collection of distorted image data caused by interference from external factors, a feasible method is to cover the terminal device to reduce the real object light that can be collected by the terminal device and retain the virtual object light. The virtual object light can generate image data for the terminal device, thereby improving the accuracy of the calculated results.
[0045] The calibration image data may be image data with the same resolution as the distorted image data, and the calibration image data may be image data equivalent to the distorted image data without distortion. Figure 4 As shown, it can be seen Figure 4 The calibration screen data in Figure 3 Compared with the distorted picture data in the image, the picture content is the same and no distortion occurs. A feasible method for obtaining calibration picture data may be to directly read the picture data generated by the terminal device and use the picture data as calibration picture data. It is understandable that the picture data directly read does not pass through the optical element, resulting in distortion.
[0046] S102, obtaining calibration coordinates corresponding to the calibration image data and distortion coordinates corresponding to the distortion image data;
[0047] In one embodiment, after the distorted picture data and the calibrated picture data are acquired, a dot detection method is used to detect the calibrated picture data and the distorted picture data to obtain calibration coordinates corresponding to the calibrated picture data and distortion coordinates corresponding to the distorted picture data.
[0048] For example, Figure 3 and Figure 4 As shown, Figure 3 and Figure 4 The image data is obtained after the dot detection method is used, and the two-dimensional pixel coordinates of the centers of all circles in the image data are used as calibration coordinates and distortion coordinates. The expression of the calibration coordinates can be The expression of the distorted coordinates can be Wherein, N is the number of dots in the picture data.
[0049] S103, calculating a weight matrix based on the calibration coordinates and the distortion coordinates;
[0050] In one embodiment, in order to obtain the distorted coordinates of any position in the distorted image data, the distorted coordinates may be interpolated to obtain a distortion function, and then the coordinates of any position in the distorted image data may be determined according to the distortion function.
[0051] Exemplarily, a feasible difference processing method may be to use a basis function interpolation method. In order to calculate the distortion function corresponding to the distorted image data, it is necessary to obtain a weight matrix corresponding to the distorted image data. Specifically, it may be to calculate based on the calibration coordinates to obtain a parameter matrix corresponding to the calibration image data; obtain a distortion point matrix corresponding to the distorted image data based on the calibration coordinates and the distortion coordinates; and obtain a weight matrix based on the parameter matrix and the distortion point matrix.
[0052] Exemplarily, the parameter matrix may be obtained by setting a kernel function corresponding to the calibration picture data, and performing calculations based on the kernel function and the calibration coordinates to obtain the parameter matrix corresponding to the calibration picture data.
[0053] The kernel function can be expressed as formula (1):
[0054]
[0055] Among them, α may be a control variable used to improve the smoothness of the curve formed by interpolation, for example, α=5; r may be the minimum distance between two adjacent calibration coordinates in the calibration picture data.
[0056] Substitute the calibration coordinates into formula (1) to obtain the parameter matrix A corresponding to the calibration screen data. The expression of the parameter matrix A can be:
[0057]
[0058] Furthermore, the method for obtaining the distortion point matrix can be to calculate the isographic matrix based on the calibration coordinates and the distortion coordinates, obtain the projection coordinates corresponding to the distorted picture data based on the isographic matrix, and then obtain the distortion point matrix corresponding to the distorted picture data based on the projection coordinates.
[0059] It should be noted that when the image data is not distorted, there exists a monotropic matrix H that makes formula (3) valid. However, since the monotropic matrix is a theoretical value, the distortion coordinates are displaced due to the distortion in the actual calculation process, making formula (3) invalid. Therefore, a feasible method is to use the least squares method to calculate the optimal monotropic matrix H' between the calibration coordinates and the distortion coordinates, and use the optimal monotropic matrix as the monotropic matrix for approximate calculation to minimize the projection error L, thereby reducing the error of the distortion point matrix and improving the accuracy of subsequent calculations.
[0060] For example, the expression of the isometric matrix can be as shown in formula (3), and the expression of the projection error L can be as shown in formula (4),
[0061]
[0062]
[0063] Where s is a scale parameter, which can be any value; h 11 etc. can be elements in the monotropy matrix, h 21 ' can be an element in the optimal isotropic matrix.
[0064] Furthermore, after the least square method is used to calculate the optimal isometric matrix, the distortion coordinates are projected onto the calibration image data to obtain the projection coordinates. The expression of the projection coordinates can be: c x′ i =[ c u′ i , c v′ i ] T , and then we get expression (5),
[0065]
[0066] Further, according to the distortion point matrix B corresponding to the projection coordinate distortion picture data, the expression of the distortion point matrix B can be:
[0067]
[0068] After obtaining the parameter matrix A and the distortion point matrix B, the weight matrix W is obtained according to the relationship between the matrices "AW = B". The expression for calculating the weight matrix W can be:
[0069] W=(A T A) -1 A T B Formula (7)
[0070] It should be noted that the expression of the weight matrix W can be,
[0071]
[0072] S104, performing interpolation processing on the distorted coordinates based on the weight matrix to obtain a distortion function corresponding to the distorted picture data;
[0073] In one embodiment, after obtaining the weight matrix, the distorted coordinates of the distorted image data are interpolated based on the weight matrix to obtain a distortion function corresponding to the distorted image data, so that the terminal device can process the collected image data according to the distortion function to obtain target image data that is not distorted.
[0074] Exemplarily, the expression of the distortion function can be:
[0075]
[0076] For example, the interpolation effect obtained after interpolation of the projection coordinates can be as follows: Figure 5 As shown, Figure 5 The projection coordinates are connected by a curve to obtain the coordinate values of each point in the distorted image data. It can be understood that the projection coordinates are the centers of each circle obtained by using the dot detection method to distort the image data, and then each point can be other points in the distorted image data except the projection coordinates. Figure 5 In the figure, the points are other than the intersection points of the curves. It should be noted that when interpolating, the projection coordinates can be connected by curves or by other methods, which are set according to the actual situation.
[0077] Furthermore, the distortion function is used to obtain target picture data corresponding to the distorted picture data. A feasible method for obtaining the target picture data may be that the terminal device obtains the upper screen picture data, performs inverse distortion processing on the upper screen picture data based on the distortion function to obtain inverse distorted picture data, and distorts the inverse distorted picture data based on the optical element of the terminal device to obtain the target picture data corresponding to the inverse distorted picture data.
[0078] It can be understood that the shape of the target image data is the same as the shape of the upper screen image data, so that the image data viewed by the user through the terminal device is the image data without distortion. Figure 6 As shown, Figure 6 The upper screen image data in the image is inversely distorted by a distortion function to obtain inversely distorted image data, and then the inversely distorted image data is distorted according to the optical elements in the terminal device to obtain the target image data.
[0079] In the embodiments of the present specification, by acquiring distorted picture data and calibrated picture data, calibrated coordinates and distorted coordinates corresponding to the picture data are acquired for calculation, a weight matrix corresponding to the distorted picture data is obtained, and then interpolation processing is performed based on the weight matrix to obtain a distortion function corresponding to the distorted picture data, so as to obtain target picture data according to the distortion function, thereby achieving de-distortion processing of the acquired picture data, providing users with undistorted picture data, and then obtaining the distortion function of the terminal device according to the calculation, thereby improving the efficiency and versatility of de-distortion processing of the picture data.
[0080] See also Figure 7 , is a flowchart of a method for processing a picture provided by an embodiment of this specification. Figure 7 As shown, the method may include the following steps S201-S208.
[0081] S201, acquiring collected distorted image data, and acquiring calibration image data corresponding to the distorted image data;
[0082] In one embodiment, a calibrated camera device is used to shoot a terminal device, collect distorted image data, and obtain calibrated image data corresponding to the distorted image data.
[0083] The distorted image data may be the image data generated by the terminal device, which is distorted when the image data passes through the optical element in the terminal device. For example, it may be: Figure 3 As shown, it can be seen Figure 3 The frame of the distorted image data in the image is distorted. It is understandable that the image is inevitably distorted due to the structure and assembly of the optical element itself.
[0084] It should be noted that the calibrated camera device may have undergone distortion correction, and the image data captured by the camera device is undistorted. Therefore, when calculations are performed based on the image data obtained after shooting with the calibrated camera device, it is possible to avoid the accuracy of subsequent calculations being affected by errors caused by the image data obtained by shooting.
[0085] It can be understood that in order to reduce the difficulty of data collection of distorted image data caused by interference from external factors, a feasible method is to cover the terminal device to reduce the real object light that can be collected by the terminal device and retain the virtual object light. The virtual object light can generate image data for the terminal device, thereby improving the accuracy of the calculated results.
[0086] The calibration image data may be image data with the same resolution as the distorted image data, and the calibration image data may be image data equivalent to the distorted image data without distortion. Figure 4 As shown, it can be seen Figure 4 The calibration screen data in Figure 3 Compared with the distorted picture data in the image, the picture content is the same and no distortion occurs. A feasible method for obtaining calibration picture data may be to directly read the picture data generated by the terminal device and use the picture data as calibration picture data. It is understandable that the picture data directly read does not pass through the optical element, resulting in distortion.
[0087] S202, obtaining calibration coordinates corresponding to the calibration image data and distortion coordinates corresponding to the distortion image data;
[0088] In one embodiment, after the distorted picture data and the calibrated picture data are acquired, a dot detection method is used to detect the calibrated picture data and the distorted picture data to obtain calibration coordinates corresponding to the calibrated picture data and distortion coordinates corresponding to the distorted picture data.
[0089] For example, Figure 3 and Figure 4 As shown, Figure 3 and Figure 4 The image data is obtained after the dot detection method is used, and the two-dimensional pixel coordinates of the centers of all circles in the image data are used as calibration coordinates and distortion coordinates. The expression of the calibration coordinates can be The expression of the distorted coordinates can be Wherein, N is the number of dots in the picture data.
[0090] S203, calculating and obtaining a parameter matrix of the calibration picture data based on the calibration coordinates;
[0091] In one embodiment, in order to obtain the distorted coordinates of any position in the distorted image data, the distorted coordinates may be interpolated to obtain a distortion function, and then the coordinates of any position in the distorted image data may be determined according to the distortion function.
[0092] Exemplarily, the parameter matrix may be obtained by setting a kernel function corresponding to the calibration picture data, and performing calculations based on the kernel function and the calibration coordinates to obtain the parameter matrix corresponding to the calibration picture data.
[0093] The kernel function can be expressed as formula (1):
[0094]
[0095] Among them, α may be a control variable used to improve the smoothness of the curve formed by interpolation, for example, α=5; r may be the minimum distance between two adjacent calibration coordinates in the calibration picture data.
[0096] Substitute the calibration coordinates into formula (1) to obtain the parameter matrix A corresponding to the calibration screen data. The expression of the parameter matrix A can be:
[0097]
[0098] S204, acquiring a distortion point matrix corresponding to the distorted image data based on the calibration coordinates and the distortion coordinates;
[0099] In one embodiment, the method for obtaining the distortion point matrix can be to calculate the isographic matrix based on the calibration coordinates and the distortion coordinates, obtain the projection coordinates corresponding to the distorted picture data based on the isographic matrix, and then obtain the distortion point matrix corresponding to the distorted picture data based on the projection coordinates.
[0100] It should be noted that when the image data is not distorted, there exists a monotropic matrix H that makes formula (3) valid. However, since the monotropic matrix is a theoretical value, the distortion coordinates are displaced due to the distortion in the actual calculation process, making formula (3) invalid. Therefore, a feasible method is to use the least squares method to calculate the optimal monotropic matrix H' between the calibration coordinates and the distortion coordinates, and use the optimal monotropic matrix as the monotropic matrix for approximate calculation to minimize the projection error L, thereby reducing the error of the distortion point matrix and improving the accuracy of subsequent calculations.
[0101] For example, the expression of the isometric matrix can be as shown in formula (3), and the expression of the projection error L can be as shown in formula (4),
[0102]
[0103]
[0104] Where s is a scale parameter, which can be any value; h 11 etc. can be elements in the monotropy matrix, h 21 ' can be an element in the optimal monotropy matrix.
[0105] Furthermore, after the least square method is used to calculate the optimal isometric matrix, the distortion coordinates are projected onto the calibration image data to obtain the projection coordinates. The expression of the projection coordinates can be: c x′ i =[ c μ′ i ,cv′ i ] T , and then we get expression (5),
[0106]
[0107] Further, according to the distortion point matrix B corresponding to the projection coordinate distortion picture data, the expression of the distortion point matrix B can be:
[0108]
[0109] S205, obtaining a weight matrix based on the parameter matrix and the distortion point matrix;
[0110] In one embodiment, after the parameter matrix and the distortion point matrix are obtained, a calculation is performed based on the matrix relationship among the parameter matrix, the distortion point matrix and the weight matrix to obtain a weight matrix corresponding to the distorted picture data.
[0111] Furthermore, the matrix relationship between the parameter matrix, the distortion point matrix and the weight matrix can be "AW = B", and then the calculation expression of the weight matrix W can be obtained,
[0112] W=(A T A) -1 A T B Formula (7)
[0113] It should be noted that the expression of the weight matrix W can be,
[0114]
[0115] S206, performing interpolation processing on the distorted coordinates based on the weight matrix to obtain a distortion function corresponding to the distorted picture data;
[0116] In one embodiment, after obtaining the weight matrix, the distorted coordinates of the distorted image data are interpolated based on the weight matrix to obtain a distortion function corresponding to the distorted image data, so that the terminal device can process the collected image data according to the distortion function to obtain target image data that is not distorted.
[0117] Exemplarily, the expression of the distortion function can be:
[0118]
[0119] It should be noted that formula (9) can be used to calculate any distortion coordinate in the distorted picture data.
[0120] For example, the interpolation effect obtained after interpolation of the projection coordinates can be as follows: Figure 5 As shown, Figure 5 The projection coordinates are connected by a curve to obtain the coordinate values of each point in the distorted image data. It can be understood that the projection coordinates are the centers of each circle obtained by using the dot detection method to distort the image data, and then each point can be other points in the distorted image data except the projection coordinates. Figure 5 In the figure, the points are other than the intersection points of the curves. It should be noted that when interpolating, the projection coordinates can be connected by curves or by other methods, which are set according to the actual situation.
[0121] The distortion function can be used to obtain the target image data corresponding to the distorted image data, and the degree of distortion of the optical element of the terminal device can be determined according to the distortion function. Since the material, shape and other factors of the optical element in the terminal device, as well as the distortion caused by the assembly of the optical element, are determined when the terminal device is produced, the distortion function of the terminal device is unique and fixed, and the terminal device can be directly calculated based on the distortion function when it is actually used.
[0122] It is understandable that since the optical components of the terminal device are worn out during use, in order to ensure that the image data provided by the terminal device can achieve the desired effect, a feasible method is to calculate and correct the distortion function once in a period of time. The specific interval time can be set according to actual needs.
[0123] S207, acquiring upper screen picture data, and performing inverse distortion processing on the upper screen picture data based on the distortion function to obtain inverse distortion picture data;
[0124] In one embodiment, after the distortion function of the distorted picture data is calculated, the distortion function may be the distortion function corresponding to the terminal device. During actual use, the terminal device obtains the upper screen picture data, obtains the upper screen coordinates corresponding to the upper screen picture data, performs inverse distortion calculation on the upper screen coordinates based on the distortion function, obtains the inverse distortion coordinates corresponding to the upper screen coordinates, and obtains the inverse distortion picture data corresponding to the upper screen picture data based on the inverse distortion coordinates.
[0125] For example, a feasible method for obtaining the inverse distortion coordinates based on the distortion function may be to calculate the pre-distortion coordinates corresponding to the upper screen coordinates based on the distortion function, and obtain the inverse distortion coordinates corresponding to the upper screen coordinates based on the difference between the pre-distortion coordinates and the product coordinates. For example, the expression of the upper screen image data is: c ( c x), the expression of distorted image data is d ( c x′), then the relationship expression between the upper screen image data and the distorted image data can be obtained according to the distortion function:
[0126] I c ( c x)=I d (f( c x))=I d ( c x′) Formula (10)
[0127] The expression for inverse distortion processing can be:
[0128] I r ( c x)=I c (f( c x))=I c ( c x′) Formula (11)
[0129] Among them, I r ( c x) can be an expression for the inverse distortion picture data.
[0130] Exemplarily, the distorted image data corresponding to the pre-distorted coordinates obtained by the upper screen image data according to the distortion function can be as follows: Figure 8 As shown, it can be seen Figure 8 The pre-distorted picture data in is distorted compared to the upper screen picture data, and then based on the difference between the upper screen coordinates of the upper screen picture data and the pre-distorted coordinates of the pre-distorted picture data, the inverse distorted picture data corresponding to the upper screen picture data is obtained. The inverse distorted picture data can be as follows: Fig. 9 As shown, it can be seen Fig. 9 The inverse distortion image data in is based on Figure 8 The image data is obtained by reversely distorting the graph between the middle and upper screen image data and the pre-distorted image data.
[0131] S208, distorting the inversely distorted picture data based on an optical element of the terminal device to obtain target picture data corresponding to the inversely distorted picture data;
[0132] In one embodiment, after the upper screen image data is inversely distorted according to the distortion function to obtain inverse distorted image data, the inverse distorted image data is projected to the user's eyes through the optical elements of the terminal device, and the inverse distorted image data is distorted based on the optical elements of the terminal device to obtain target image data corresponding to the distorted image data.
[0133] It can be understood that the shape of the target image data is the same as the shape of the upper screen image data, so that the image data viewed by the user through the terminal device is the image data without distortion. Figure 6 As shown, Figure 6 The upper screen image data in the image is inversely distorted by a distortion function to obtain inversely distorted image data, and then the inversely distorted image data is distorted according to the optical elements in the terminal device to obtain the target image data.
[0134] It should be noted that since the distortion function of the terminal device is unique, the upper screen picture data is inversely distorted to obtain inversely distorted picture data, and then the inversely distorted picture data is distorted according to the optical element to obtain the target picture data. In theory, the target picture data is obtained by repeatedly changing the upper screen picture data, and can be considered to be the same picture data as the upper screen picture data, thereby providing users with undistorted picture data and improving the user experience.
[0135] In the embodiment of the present specification, by acquiring distorted picture data and calibrated picture data, calibrated coordinates and distorted coordinates corresponding to the picture data are acquired for calculation, a weight matrix corresponding to the distorted picture data is obtained, and then interpolation processing is performed based on the weight matrix to obtain a distortion function corresponding to the distorted picture data, so that the upper screen picture data is inversely distorted according to the distortion function to obtain inversely distorted picture data, and the inversely distorted picture data is distorted according to optical elements to obtain target picture data, thereby achieving de-distortion processing of the acquired picture data, providing users with undistorted picture data, and then obtaining the distortion function of the terminal device according to the calculation, thereby improving the efficiency and versatility of de-distortion processing of the picture data.
[0136] based on Figure 1 The system architecture shown below will be combined with Figure 10-13 , the picture processing device provided in the embodiment of this specification is introduced in detail. It should be noted that, Figure 10-13 The picture processing device in the embodiment of this specification is used to execute Figures 2 to 9 For the convenience of explanation, only the part related to the embodiment of this specification is shown. For the specific technical details not disclosed, please refer to the embodiment of this specification. Figures 2 to 9 The embodiment shown.
[0137] See also Fig.10, is a schematic diagram of the structure of a picture processing device provided in the embodiment of this specification. Fig.10 As shown, the picture processing device 1 of the embodiment of this specification may include: a data acquisition unit 11, a coordinate acquisition unit 12, a matrix acquisition unit 13 and a function acquisition unit 14.
[0138] The data acquisition unit 11 is used to acquire the collected distorted image data and acquire the calibration image data corresponding to the distorted image data;
[0139] A coordinate acquisition unit 12, used to acquire the calibration coordinates corresponding to the calibration picture data and the distortion coordinates corresponding to the distortion picture data;
[0140] A matrix acquisition unit 13, configured to calculate a weight matrix based on the calibration coordinates and the distortion coordinates;
[0141] The function acquisition unit 14 is used to perform interpolation processing on the distorted coordinates based on the weight matrix to obtain a distortion function corresponding to the distorted picture data, and the distortion function is used to obtain target picture data corresponding to the distorted picture data.
[0142] Optional, such as Fig.11 As shown, the matrix acquisition unit 13 includes:
[0143] A parameter matrix acquisition subunit 131 is used to calculate and obtain a parameter matrix of the calibration picture data based on the calibration coordinates;
[0144] A distortion point matrix acquisition subunit 132, configured to acquire a distortion point matrix corresponding to the distorted picture data based on the calibration coordinates and the distortion coordinates;
[0145] The weight matrix acquisition subunit 133 is used to obtain a weight matrix based on the parameter matrix and the distortion point matrix.
[0146] Optionally, the parameter matrix acquisition subunit 131 is further used for:
[0147] Setting a kernel function corresponding to the calibration picture data;
[0148] Calculation is performed based on the kernel function and the calibration coordinates to obtain a parameter matrix corresponding to the calibration picture data.
[0149] Optionally, the distortion point matrix acquisition subunit 132 is further used for:
[0150] Calculating a isometric matrix based on the calibration coordinates and the distortion coordinates;
[0151] Obtaining projection coordinates corresponding to the distorted image data based on the isographic matrix;
[0152] A distortion point matrix corresponding to the distorted picture data is obtained based on the projection coordinates.
[0153] Optional, such as Fig.12 As shown, the picture processing device 1 also includes:
[0154] The inverse distortion unit 15 is used to obtain upper screen picture data, and perform inverse distortion processing on the upper screen picture data based on the distortion function to obtain inverse distortion picture data;
[0155] The target data acquisition unit 16 is used to distort the inverse distortion picture data based on the optical element of the terminal device to obtain target picture data corresponding to the inverse distortion picture data, and the shape of the target picture data is the same as the shape of the upper screen picture data.
[0156] Optional, such as Fig.13 As shown, the inverse distortion unit 15 includes:
[0157] The upper screen coordinate acquisition subunit 151 is used to acquire upper screen picture data and acquire the upper screen coordinates corresponding to the upper screen picture data;
[0158] An inverse distortion coordinate acquisition subunit 152, configured to perform inverse distortion calculation on the upper screen coordinate based on the distortion function to obtain an inverse distortion coordinate corresponding to the upper screen coordinate;
[0159] The inverse distortion data acquisition subunit 153 is used to obtain the inverse distortion picture data corresponding to the upper screen picture data based on the inverse distortion coordinates.
[0160] Optionally, the inverse distortion data acquisition subunit 153 is further used for:
[0161] Calculating the pre-distortion coordinates corresponding to the upper screen coordinates based on the distortion function;
[0162] Based on the difference between the pre-distortion coordinates and the upper-screen coordinates, an inverse-distortion coordinate corresponding to the upper-screen coordinates is obtained.
[0163] In the embodiment of the present specification, by acquiring distorted picture data and calibrated picture data, calibrated coordinates and distorted coordinates corresponding to the picture data are acquired for calculation, a weight matrix corresponding to the distorted picture data is obtained, and then interpolation processing is performed based on the weight matrix to obtain a distortion function corresponding to the distorted picture data, so that the upper screen picture data is inversely distorted according to the distortion function to obtain inversely distorted picture data, and the inversely distorted picture data is distorted according to optical elements to obtain target picture data, thereby achieving de-distortion processing of the acquired picture data, providing users with undistorted picture data, and then obtaining the distortion function of the terminal device according to the calculation, thereby improving the efficiency and versatility of de-distortion processing of the picture data.
[0164] The present specification also provides a computer storage medium, which can store a plurality of program instructions, which are suitable for being loaded and executed by a processor as described above. Figure 1-Figure 9 The method steps of the embodiment shown in the figure can be found in the specific implementation process. Figure 1-Figure 9 The specific description of the illustrated embodiment will not be repeated here.
[0165] See also Fig.14 , is a schematic diagram of the structure of an electronic device provided in the embodiment of this specification. Fig.14 As shown, the electronic device 1000 may include: at least one processor 1001, such as a CPU, at least one network interface 1004, an input / output interface 1003, a memory 1005, and at least one communication bus 1002. The communication bus 1002 is used to realize the connection and communication between these components. The network interface 1004 may optionally include a standard wired interface, a wireless interface (such as a WI-FI interface). The memory 1005 may be a high-speed RAM memory, or a non-volatile memory (non-volatile memory), such as at least one disk storage. The memory 1005 may optionally also be at least one storage device located away from the aforementioned processor 1001. As shown in FIG. Fig.14 As shown, the memory 1005 as a computer storage medium may include an operating system, a network communication module, an input / output interface module, and a picture processing application.
[0166] exist Fig.14 In the electronic device 1000 shown, the input-output interface 1003 is mainly used to provide an input interface for the user and obtain data input by the user.
[0167] In one embodiment, the processor 1001 may be used to call a picture processing application stored in the memory 1005, and specifically perform the following operations:
[0168] Acquire the collected distorted image data, and acquire the calibration image data corresponding to the distorted image data;
[0169] Acquire calibration coordinates corresponding to the calibration picture data and distortion coordinates corresponding to the distortion picture data;
[0170] Calculating a weight matrix based on the calibration coordinates and the distortion coordinates;
[0171] The distorted coordinates are interpolated based on the weight matrix to obtain a distortion function corresponding to the distorted picture data, and the distortion function is used to obtain target picture data corresponding to the distorted picture data.
[0172] Optionally, when the processor 1001 calculates a weight matrix based on the calibration coordinates and the distortion coordinates, the processor 1001 specifically performs the following operations:
[0173] Calculate a parameter matrix of the calibration picture data based on the calibration coordinates;
[0174] Acquire a distortion point matrix corresponding to the distorted picture data based on the calibration coordinates and the distorted coordinates;
[0175] A weight matrix is obtained based on the parameter matrix and the distortion point matrix.
[0176] Optionally, when the processor 1001 calculates the parameter matrix of the calibration picture data based on the calibration coordinates, the processor 1001 specifically performs the following operations:
[0177] Setting a kernel function corresponding to the calibration picture data;
[0178] Calculation is performed based on the kernel function and the calibration coordinates to obtain a parameter matrix corresponding to the calibration picture data.
[0179] Optionally, when the processor 1001 acquires a distorted point matrix corresponding to the distorted image data based on the calibration coordinates and the distorted coordinates, the processor 1001 specifically performs the following operations:
[0180] Calculating a isometric matrix based on the calibration coordinates and the distortion coordinates;
[0181] Obtaining projection coordinates corresponding to the distorted image data based on the isographic matrix;
[0182] A distortion point matrix corresponding to the distorted picture data is obtained based on the projection coordinates.
[0183] Optionally, after executing to obtain the distortion function corresponding to the distorted picture data, the processor 1001 further performs the following operations:
[0184] Acquire upper screen picture data, and perform inverse distortion processing on the upper screen picture data based on the distortion function to obtain inverse distortion picture data;
[0185] Based on the optical element of the terminal device, the inverse distortion picture data is distorted to obtain target picture data corresponding to the inverse distortion picture data, and the shape of the target picture data is the same as the shape of the upper screen picture data.
[0186] Optionally, when the processor 1001 acquires the upper screen picture data and performs inverse distortion processing on the upper screen picture data based on the distortion function to obtain the inverse distortion picture data, the following operations are specifically performed:
[0187] Acquire upper screen image data, and acquire upper screen coordinates corresponding to the upper screen image data;
[0188] Performing inverse distortion calculation on the upper screen coordinates based on the distortion function to obtain inverse distortion coordinates corresponding to the upper screen coordinates;
[0189] The inverse distortion picture data corresponding to the upper screen picture data is obtained based on the inverse distortion coordinates.
[0190] Optionally, when the processor 1001 performs inverse distortion calculation on the upper screen coordinates based on the distortion function to obtain inverse distortion coordinates corresponding to the upper screen coordinates, the following operations are specifically performed:
[0191] Calculating the pre-distortion coordinates corresponding to the upper screen coordinates based on the distortion function;
[0192] Based on the difference between the pre-distortion coordinates and the upper-screen coordinates, an inverse-distortion coordinate corresponding to the upper-screen coordinates is obtained.
[0193] In the embodiment of the present specification, by acquiring distorted picture data and calibrated picture data, calibrated coordinates and distorted coordinates corresponding to the picture data are acquired for calculation, a weight matrix corresponding to the distorted picture data is obtained, and then interpolation processing is performed based on the weight matrix to obtain a distortion function corresponding to the distorted picture data, so that the upper screen picture data is inversely distorted according to the distortion function to obtain inversely distorted picture data, and the inversely distorted picture data is distorted according to optical elements to obtain target picture data, thereby achieving de-distortion processing of the acquired picture data, providing users with undistorted picture data, and then obtaining the distortion function of the terminal device according to the calculation, thereby improving the efficiency and versatility of de-distortion processing of the picture data.
[0194] A person skilled in the art can understand that all or part of the processes in the above-mentioned embodiments can be implemented by instructing the relevant hardware through a computer program, and the program can be stored in a computer-readable storage medium, and when the program is executed, it can include the processes of the embodiments of the above-mentioned methods. The storage medium can be a disk, an optical disk, a read-only memory (ROM) or a random access memory (RAM), etc.
[0195] The above disclosure is only the preferred embodiment of this specification, which certainly cannot be used to limit the scope of rights of this specification. Therefore, equivalent changes made according to the claims of this specification are still within the scope covered by this specification.
Claims
1. A picture processing method, characterized in that: The method comprises: Acquire the collected distorted image data, and acquire the calibration image data corresponding to the distorted image data; Acquire calibration coordinates corresponding to the calibration picture data and distortion coordinates corresponding to the distortion picture data; Calculating a weight matrix based on the calibration coordinates and the distortion coordinates; The distorted coordinates are interpolated based on the weight matrix to obtain a distortion function corresponding to the distorted picture data, and the distortion function is used to obtain target picture data corresponding to the distorted picture data.
2. The method according to claim 1, characterized in that The calculating a weight matrix based on the calibration coordinates and the distortion coordinates includes: Calculate a parameter matrix of the calibration picture data based on the calibration coordinates; Acquire a distortion point matrix corresponding to the distorted picture data based on the calibration coordinates and the distorted coordinates; A weight matrix is obtained based on the parameter matrix and the distortion point matrix.
3. The method according to claim 2, characterized in that The step of calculating a parameter matrix of the calibration picture data based on the calibration coordinates includes: Setting a kernel function corresponding to the calibration picture data; Calculation is performed based on the kernel function and the calibration coordinates to obtain a parameter matrix corresponding to the calibration picture data.
4. The method according to claim 2, characterized in that: The acquiring a distortion point matrix corresponding to the distorted picture data based on the calibration coordinates and the distortion coordinates includes: Calculating a isometric matrix based on the calibration coordinates and the distortion coordinates; Obtaining projection coordinates corresponding to the distorted image data based on the isographic matrix; A distortion point matrix corresponding to the distorted picture data is obtained based on the projection coordinates.
5. The method according to claim 1, characterized in that After obtaining the distortion function corresponding to the distorted picture data, the method further includes: Acquire upper screen picture data, and perform inverse distortion processing on the upper screen picture data based on the distortion function to obtain inverse distortion picture data; Based on the optical element of the terminal device, the inverse distortion picture data is distorted to obtain target picture data corresponding to the inverse distortion picture data, and the shape of the target picture data is the same as the shape of the upper screen picture data.
6. The method according to claim 5, characterized in that The acquiring of the upper screen picture data and performing inverse distortion processing on the upper screen picture data based on the distortion function to obtain inverse distortion picture data includes: Acquire upper screen image data, and acquire upper screen coordinates corresponding to the upper screen image data; Performing inverse distortion calculation on the upper screen coordinates based on the distortion function to obtain inverse distortion coordinates corresponding to the upper screen coordinates; The inverse distortion picture data corresponding to the upper screen picture data is obtained based on the inverse distortion coordinates.
7. The method according to claim 6, characterized in that The performing inverse distortion calculation on the upper screen coordinates based on the distortion function to obtain inverse distortion coordinates corresponding to the upper screen coordinates includes: Calculating the pre-distortion coordinates corresponding to the upper screen coordinates based on the distortion function; Based on the difference between the pre-distortion coordinates and the upper-screen coordinates, an inverse-distortion coordinate corresponding to the upper-screen coordinates is obtained.
8. A picture processing device, characterized in that: The device comprises: A data acquisition unit, used to acquire the collected distorted image data and acquire the calibration image data corresponding to the distorted image data; A coordinate acquisition unit, used to acquire calibration coordinates corresponding to the calibration picture data and distortion coordinates corresponding to the distortion picture data; A matrix acquisition unit, used for calculating a weight matrix based on the calibration coordinates and the distortion coordinates; A function acquisition unit is used to perform interpolation processing on the distorted coordinates based on the weight matrix to obtain a distortion function corresponding to the distorted picture data, and the distortion function is used to obtain target picture data corresponding to the distorted picture data.
9. A computer storage medium storing a plurality of instructions, wherein the instructions are suitable for being loaded by a processor and executing the steps of the method according to any one of claims 1 to 7.
10. An electronic device comprising: A processor and a memory; wherein the memory stores a computer program, and the computer program is suitable for being loaded by the processor and executing the steps of the method according to any one of claims 1 to 7.