A method for generating an accurate 3D pattern in a 2D image

Abstract

The present invention belongs to the technical field of image processing, and particularly relates to a method for generating an accurate 3D pattern in a 2D image. The present invention uses a projection matrix to represent the shape of a line, and adopts a back-projection method in the algorithm to calculate the shortest distance from a 2D point in each image matrix to a 3D line. Substantially, the distance between a ray and a target line segment is obtained. After screening through a threshold value, the most realistic shape of the line can be obtained. Compared with the prior art, the algorithm of the present invention is simple to implement, has high efficiency, and high accuracy.

Description

Background Art

[0002] People often need to overlay some three-dimensional patterns on camera images, and the position and size of this three-dimensional pattern are as close as possible to those in reality. For example, the reference lines when reversing, and in AR (Augmented Reality).

[0003] Most current curves are represented by multiple straight lines, and there are the following defects:

[0004] 1. Non-uniformity: The accuracy of the curve is controlled by the accuracy of segmentation. For example, if the accuracy is 1, it is a straight line, and if the accuracy is 100, the curve will be represented by 100 straight line segments (which is obviously very smooth). However, since the segmentation is performed in 3D, its distribution in 2D will be non-uniform;

[0005] 2. Low accuracy: Although the line thickness can be configured, it is not automatically configured but requires manual setting. Moreover, if different line widths are required at different positions on a line, manual setting or configuration will not only be inefficient but also lead to a decrease in accuracy.

[0006] Patent No. CN2007800236737A discloses a method for converting a 2D movie or any other arbitrary 2D image sequence into stereoscopic 3D image data for 3D display. In one embodiment, various types of image data clues can be collected from 2D source images through various methods, and these image data clues can be used to generate two clear stereoscopic 3D views. The architecture models of some embodiments of the system can also be applied to conversions, reproductions, and visual enhancements in a wide range of movies and other image sequences, including converting a 2D movie or 2D image sequence into 3D, re-recording a movie or video sequence at a different frame rate, improving the quality of a movie or other image sequence, or further promoting the improvement of visual image quality in a projector to generate enhanced images for other conversions.

[0007] The method for converting 2D images into 3D images is achieved based on collecting image data clues, and it is applicable to fields with high requirements for image generation quality, such as the film and television field. Therefore, the algorithm complexity is relatively high, and the requirements for equipment are also very high. Summary of the Invention

[0008] In view of this, the main object of the present invention is to provide a method for generating an accurate 3D pattern in a 2D image. Compared with the prior art, the algorithm of the present invention is simple to implement, has high efficiency, and high accuracy.

[0009] To achieve the above object, the technical solution of the present invention is realized as follows:

[0010] A method for generating an accurate 3D pattern in a 2D image, the method performing the following steps:

[0011] Step 1: Obtain an image matrix of the original image, and generate a projection matrix corresponding to the image matrix; the projection matrix has the same size as the image matrix and has 1 channel;

[0012] Step 2: Traverse the points in the entire image matrix, represented by p(i,j), which is a 2D point. During the traversal, the 2D points are converted to 3D points. The converted 3D points are represented by p ij (x, y, z) represents the origin of the 3D point with p0(0,0,0);

[0013] Step 3: The 3D pattern to be generated is represented by multiple line segments, and each line segment is represented by two 3D points, one of which is the starting point of the line segment and the other is the end point of the line segment;

[0014] Step 4: Connect the origin of the 3D point p0(0,0,0) and the point p(i,j) in the image matrix as a reference line segment;

[0015] Step 5: Calculate the shortest distance between each line segment and the reference line, and compare the calculated shortest distance with a set threshold. If the shortest distance is less than the set threshold, convert the point in the projection matrix to a set value N; the set value N is the color that each line segment is preset to represent;

[0016] Step 6: After the image matrix traversal is completed, the image in the projection matrix is ​​converted into the pattern to be generated, and the generated image is superimposed on the original image to obtain the 3D pattern to be generated.

[0017] Furthermore, the method for calculating the shortest distance between each line segment and the reference line in step 5 includes: taking any two points on the line segment, representing them as point A and point B respectively; taking any two points on the reference line, representing them as point C and point D respectively; obtaining the direction vector of the line segment and representing it as AB=(x1, y1, z1), and the direction vector of the reference line and representing it as CD=(x2, y2, z2); obtaining the direction vector N of the common perpendicular line of the two line segments as the cross product of AB and CD; and then calculating the dot product of AB and N to obtain the shortest distance

[0018] The present invention provides a method for generating an accurate 3D pattern in a 2D image, which has the following beneficial effects: the present invention uses a projection matrix to represent the shape of a line, and uses a reverse projection method in the algorithm to calculate the shortest distance from a 2D point in each image matrix to a 3D line, and essentially obtains the distance between a ray and a target line segment, and after screening through a threshold, the most realistic line shape can be obtained. Compared with the prior art, the algorithm of the present invention is simple to implement, highly efficient, and highly accurate. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] Figure 1 A schematic flow chart of a method for generating an accurate 3D pattern in a 2D image provided by an embodiment of the present invention;

[0020] Figure 2 A schematic diagram of converting points in a world coordinate system into points in a collinear coordinate system in a method for generating an accurate 3D pattern in a 2D image provided by an embodiment of the present invention;

[0021] Figure 3 A schematic diagram of the principle of conversion of points between an image plane and a virtual image plane in a method for generating an accurate 3D pattern in a 2D image provided by an embodiment of the present invention;

[0022] Figure 4 A schematic diagram of the offset principle of a point after refraction by a lens in a method for generating an accurate 3D pattern in a 2D image provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0023] The method of the present invention is further described in detail below in conjunction with the accompanying drawings and embodiments of the present invention.

[0024] Example 1

[0025] like Figure 1 A method for generating an accurate 3D pattern in a 2D image is shown, the method performing the following steps:

[0026] Step 1: Obtain an image matrix of the original image, and generate a projection matrix corresponding to the image matrix; the projection matrix has the same size as the image matrix and has 1 channel;

[0027] Step 2: Traverse the points in the entire image matrix, represented by p(i,j), which is a 2D point. During the traversal, the 2D points are converted to 3D points. The converted 3D points are represented by p ij (x, y, z) represents the origin of the 3D point with p0(0,0,0);

[0028] Step 3: The 3D pattern to be generated is represented by multiple line segments, and each line segment is represented by two 3D points, one of which is the starting point of the line segment and the other is the end point of the line segment;

[0029] Step 4: Connect the origin of the 3D point p0(0,0,0) and the point p(i,j) in the image matrix as a reference line segment;

[0030] Step 5: Calculate the shortest distance between each line segment and the reference line, and compare the calculated shortest distance with a set threshold. If the shortest distance is less than the set threshold, convert the point in the projection matrix to a set value N; the set value N is the color that each line segment is preset to represent;

[0031] Step 6: When the traversal of the image matrix is completed, the image in the projection matrix is converted into the pattern to be generated. The generated image is superimposed on the original image, and thus the 3D pattern to be generated is obtained.

[0032] Reference Figure 2 , Figure 3 and Figure 4 ,

[0033] The coordinate systems in the prior art generally fall into the following categories:

[0034] 1. Image coordinate system, with the lower left corner as the origin. Note that the image is inverted, and this is our position in the image.

[0035] 2. Image plane coordinate system, which has a different origin from the image coordinate system. The image plane takes the center point as the origin.

[0036] 3. World coordinate system;

[0037] If there is a point p(x, y) in the image coordinate system, it is first converted into a point p' on the image plane. Let k and l represent the x and y directions respectively. The length of one is pixel / m. (cx, cy) represents the optical center coordinates (here defined as the unit in the image coordinate system being pixel), which is the position where a photon enters the camera from the exact middle and should land on the sensor. In an ideal situation, it should be the exact center, that is, (W / 2, H / 2), where W is the width of the image and H is the height of the image.

[0038] However, in reality, there are often some deviations, denoted as (x×k - c x , y×l - c y ). If the center of the lens is taken as the origin, then a 3D coordinate P((x - c x )×k, (y - c y )×l, f) can be obtained; where f is the focal length. Here, k, l, cx, cy, and f are all known quantities. In the pinhole model, the lens does not refract light. Therefore, for the infinitely vast world on the right side of the lens, as long as the points on the straight line where P0 is located will project light onto point p, that is, P(((x - c x )×k)*s, ((y - c y )×l)*s, f*s); where s is a real number. If we want to determine its projection point, we must determine s. For example, if we want to project onto the ground, then the term ((y - c y )×l)*s will be a definite number such as ch, then s can be calculated, and thus P can be calculated.

[0039] Using the same principle, if we know a point P(x, y, z) in the world coordinate system, then first we can scale it proportionally to its collinear coordinates. In this way, we can convert it to the image coordinates. This is a definite point. This is the process of 2D-3D conversion, but this is only for the pinhole model.

[0040] For non-pinhole models, essentially, after the light passes through the lens and is refracted, the incident angle is not equal to the exit angle.

[0041] Let the incident angle POQ be a and the exit angle p’Oc’ be b. We only need to establish two functions f1 and f2 such that b = f1(a) and a = f2(b). The mapping of f1 and f2 is non-linear. Generally, a Taylor expansion is used to represent such a non-linear function. We define five parameters, represented by k1, k3, k5, k7, and k9 respectively. We represent f1 as f1(a) = k1×a + 3×k3×a 3 + 5×k5×a 5 + 7×k7×a 7 + 9×k9×a 9 , The most important part of this process is to approximate a non-linear mapping using the 1st, 3rd, 5th, 7th, and 9th terms of the Taylor polynomial. The process of f2 is an iterative process. Given the exit angle b, we estimate the incident angle a. The actual process is as follows: we first give a an initial value a0, and then calculate step 1: b0 = f(a0); step 2: calculate |b - b0|. If |b - b0| < 0.01, then a = a0; step 3: set a0 = a0 - b0, and then go back to step 1; the key here is that this is an iterative optimization process.

[0042] Using the camera model algorithm, we can draw the real pattern in a 2D image, but the prerequisite is that the projection plane must be known. This method is more suitable for drawing reverse parking lines, but the curve must be represented by multiple line segments, and the thickness of the line cannot be achieved with high precision. If it is for an arbitrary plane and an arbitrary angle projection, the difficulty is even greater.

[0043] When using stereo data numbers, if such numbers are used in VR applications. Since this digital image is fixed, it cannot match the actual position and distortion of the camera. Our method can generate different styles of stereo numbers at any position, and the stereo numbers conform to the camera model of the current image and the placement position of the stereo numbers.

[0044] Furthermore, the reference line segment is the line connecting the optical center to any 2D point, and each line segment is a line segment in the pattern.

[0045] Embodiment 2

[0046] Based on the previous embodiment,

[0047] The basis of the present invention is the camera model algorithm, 2D to 3D projection, and the calculation of the minimum distance between skew lines in space.

[0048] Step 1: The width of the image matrix M is w, the height is h, and Mask is a matrix with the same size as M and a channel of 1.

[0049] Step 2: Traverse the entire M. Let p(i, j) be any point on M.

[0050] Step 3: Convert the 2D point p into a 3D point Pij(x, y, z).

[0051] Step 4: Represent the pattern to be drawn with multiple line segments. Each line segment is represented by 2 3D points P, the starting point Pstart and the ending point Pend.

[0052] Step 5: Calculate the shortest distance d between Line 1: Pstart->Pend and Line 2: P0(0, 0, 0)->Pij. If d < thrW, then Mask(i, j) = N (N is defined according to the color represented by the line segment).

[0053] Step 6: When the traversal of M is completed, the image in Mask represents the drawn pattern and can be superimposed on the original image.

[0054] The method for calculating the shortest distance between skew lines in Step 5 is as follows:

[0055] Suppose there are two lines l1 and l2. Let AB be any two points on l1, and CD be any two points on l2. The direction vector of l1, AB = (x1, y1, z1), and the direction vector of l2, CD = (x2, y2, z2). The direction vector N of the common perpendicular is AB x CD (x is the cross product). Calculate the dot product of AB and N.

[0056] The method for calculating the shortest distance between each line segment and the reference line in Step 5 includes: taking any two points on the line segment, represented by point A and point B respectively; taking any two points on the reference line, represented by point C and point D respectively; representing the direction vector of the line segment as AB = (x1, y1, z1), and representing the direction vector of the reference line as CD = (x2, y2, z2); obtaining the direction vector N of the common perpendicular of the two line segments as AB cross CD; and then calculating the dot product of AB and N to obtain the shortest distance.

[0057] It should be noted that for the system provided in the above embodiments, only the division of the above functional units is used for illustration. In practical applications, the above functions can be allocated to different functional units as needed, that is, the units or steps in the embodiments of the present invention can be further decomposed or combined. For example, the units in the above embodiments can be combined into one unit, or further split into multiple sub-units to complete all or part of the functions described above. For the names of the units and steps involved in the embodiments of the present invention, they are only used to distinguish each unit or step, and are not regarded as improper limitations on the present invention.

[0058] Those skilled in the art can clearly understand that for the sake of convenience and brevity of description, the specific working processes and related descriptions of the above-described storage device and processing device can refer to the corresponding processes in the foregoing method embodiments, and will not be repeated here.

[0059] Those skilled in the art should be able to realize that the units and method steps of each example described in combination with the embodiments disclosed herein can be implemented by electronic hardware, computer software, or a combination of the two. The programs corresponding to the software units and method steps can be placed in a random access memory (RAM), internal memory, read-only memory (ROM), electrically programmable ROM, electrically erasable programmable ROM, registers, hard disk, removable disk, CD-ROM, or any other form of storage medium well-known in the art. To clearly illustrate the interchangeability of electronic hardware and software, the components and steps of each example have been generally described according to their functions in the above description. Whether these functions are executed in the form of electronic hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered to exceed the scope of the present invention.

[0060] The terms "first", "another part", etc. are configured to distinguish similar objects, rather than configured to describe or represent a specific order or sequence.

[0061] The term "comprising" or any other similar term is intended to cover non-exclusive inclusion, so that a process, method, article, or unit / device comprising a series of elements not only includes those elements, but also includes other elements not expressly listed, or also includes elements inherent to these processes, methods, articles, or unit / device.

[0062] So far, the technical solution of the present invention has been described in conjunction with the preferred embodiments shown in the accompanying drawings. However, those skilled in the art can easily understand that the protection scope of the present invention is obviously not limited to these specific embodiments. Without departing from the principle of the present invention, those skilled in the art can make equivalent changes or substitutions to the relevant technical marks, and the technical solutions after these changes or substitutions will fall within the protection scope of the present invention.

[0063] As described above, it is only a preferred embodiment of the present invention and is not intended to limit the protection scope of the present invention.

Claims

1. A method for generating an accurate 3D pattern in a 2D image, characterized in that, The method performs the following steps: Step 1: Obtain the image matrix of the original image and generate a projection matrix corresponding to the image matrix; the projection matrix has the same size as the image matrix and is a matrix with 1 channel; Step 2: Traverse the points in the entire image matrix, denoted as p(i, j), which are 2D points. During the traversal, convert the 2D points into 3D points, and the converted 3D points are denoted as p ij (x, y, z), and denote the origin of the 3D points as p0(0, 0, 0); Step 3: Represent the 3D pattern to be generated with polylines, each line segment being represented by two 3D points, one point serving as the starting point of the line segment and the other point serving as the ending point of the line segment; Step 4: Connect the origin p0(0,0,0) of the 3D point with the point p(i,j) in the image matrix as the reference line segment; Step 5: Calculate the shortest distance between each line segment and the reference line, compare the calculated shortest distance with a set threshold value. If the shortest distance is less than the set threshold value, convert the point in the projection matrix to a set fixed value N; this set fixed value N is the color represented by each line segment preset; Step 6: When the traversal of the image matrix is completed, the image in the projection matrix is converted into the pattern to be generated. Superimpose the generated image on the original image to obtain the 3D pattern to be generated; Among them, the method for calculating the shortest distance between each line segment and the reference line in step 5 includes: taking any two points on the line segment, represented by point A and point B respectively; taking any two points on the reference line, represented by point C and point D respectively; representing the direction vector of the line segment as AB = (x1, y1, z1), and representing the direction vector of the reference line as CD = (x2, y2, z2); obtaining the direction vector M of the common perpendicular of the two line segments as the cross product of AB and CD; then calculating the dot product of AB and M to obtain the shortest distance

Images