Intelligent identification system and intelligent identification method using wide-angle camera

Abstract

The invention discloses an intelligent identification system and an intelligent identification method using a wide-angle camera. The system comprises a memory and a processor. The processor is used for loading the instructions stored in the memory so as to carry out the following intelligent identification method. The method comprises the following steps: providing an image correction pattern; shooting the image correction pattern by using a wide-angle camera to obtain a correction pattern image; and carrying out picture rotation correction, lens pitch angle correction, inverse perspective projection and lens deflection angle correction on the wide-angle camera by utilizing the lens rotation correction matrix, the lens pitch correction matrix, the lens deflection correction and inverse perspective projection matrix and the coordinate transformation matrix so as to straighten a plurality of grid lines in the corrected pattern image, and converting a plurality of wide-angle image coordinates in the corrected pattern image into a plurality of world coordinates. Therefore, distorted images shot by the wide-angle camera can be corrected.

Description

TECHNICAL FIELD

[0001] The present application relates to a smart recognition system and method using wide-angle cameras. BACKGROUND

[0002] Driving safety is a very important issue for vehicle drivers and passengers. Many techniques are currently provided to assist driving safety. For example, a plurality of wide-angle cameras are provided on a vehicle to capture images of all directions of the vehicle. For another example, a driving computer is provided to receive the images captured by the wide-angle cameras and to determine whether the current driving condition of the vehicle is safe.

[0003] However, the images captured by the wide-angle cameras are distorted images, that is, the images captured by the wide-angle cameras are distorted, so it is difficult for the driving computer to accurately determine the current road condition. For example, it is difficult for the driving computer to accurately calculate the distance between the vehicle of the driver and the surrounding objects (vehicles), so as to affect the determination of the driving computer on the current road condition.

[0004] Therefore, a smart recognition system and method using wide-angle cameras are needed to correct the distorted images captured by the wide-angle cameras to facilitate image recognition. SUMMARY

[0005] Embodiments of the present application provide a smart recognition system and method using wide-angle cameras, which can correct the distorted images captured by the wide-angle cameras.

[0006] According to some embodiments of the present application, the smart recognition method using wide-angle cameras of the present application comprises: providing an image correction pattern; capturing the image correction pattern by using a wide-angle camera to obtain a correction pattern image; and performing a correction step to use a correction formula to perform picture rotation correction, lens pitch angle correction, inverse perspective projection, and lens yaw angle correction on the wide-angle camera to straighten a plurality of grid lines in the correction pattern image and convert a plurality of wide-angle image coordinates in the correction pattern image into a plurality of world coordinates, wherein the correction formula used in the correction step is:

[0007]

[0008] wherein X w , Y w , and Z w represent three-dimensional coordinates of world coordinate axes; X i , Y i represent two-dimensional coordinates of wide-angle image coordinate axes; matrix A is a lens rotation correction matrix, matrix B is a lens pitch correction and inverse perspective projection matrix, matrix C is a lens yaw correction matrix, and matrix D is a coordinate conversion matrix, which is represented as follows:

[0009] matrix

[0010] matrix

[0011] matrix

[0012] matrix wherein θ yaw is a yaw angle of the lens; θ pitch is a pitch angle of the lens; θ roll is a roll angle of the lens; H is a height of the wide-angle camera; Z c is a depth of the camera; K x is a scale constant of the pixel coordinate of the image X-axis direction and the world coordinate; K y is a scale constant of the pixel coordinate of the image Y-axis direction and the world coordinate; w is a width of the image (e.g., in pixels); and h is a height of the image (e.g., in pixels).

[0013] In some embodiments, the wide-angle camera is a fisheye lens camera.

[0014] In some embodiments, the image correction pattern comprises a black-and-white checkerboard pattern.

[0015] In some embodiments, when the wide-angle camera is used to capture the image correction pattern, the wide-angle camera is disposed on the vehicle, and the image correction pattern is disposed on the ground beside the vehicle.

[0016] In some embodiments, the image correction pattern extends along the ground.

[0017] According to some embodiments of the present disclosure, a smart recognition system using a wide-angle camera comprises a memory and a processor. The memory is used to store a plurality of instructions; and the processor is used to load the instructions to perform the following steps: providing an image correction pattern; using a wide-angle camera to capture the image correction pattern to obtain a correction pattern image; and performing a correction step to use a correction formula to perform a picture rotation correction, a lens pitch angle correction, an inverse perspective projection, and a lens yaw angle correction on the wide-angle camera to straighten a plurality of grid lines in the correction pattern image and convert a plurality of wide-angle image coordinates in the correction pattern image to a plurality of world coordinates, wherein the correction formula used in the correction step is:

[0018]

[0019] wherein X w , Y w , and Z w represent three-dimensional coordinates of the world coordinate axis; X i , Y itwo-dimensional coordinates representing the axes of the wide-angle image; matrix A is a lens rotation correction matrix, matrix B is a lens tilt correction and inverse perspective projection matrix, matrix C is a lens skew correction matrix, and matrix D is a coordinate conversion matrix, which is represented as follows:

[0020] matrix

[0021] matrix

[0022] matrix

[0023] matrix wherein θ yaw is a lens skew angle; θ pitch is a lens tilt angle; θ roll is a lens roll angle; H is the height of the wide-angle camera; Z c is the depth of the camera; K x is the scale constant of the pixel coordinates of the image X-axis direction and the world coordinates; K y is the scale constant of the pixel coordinates of the image Y-axis direction and the world coordinates; w is the image width (e.g., in pixels); and h is the image height (e.g., in pixels).

[0024] In some embodiments, the wide-angle camera is a fisheye lens camera.

[0025] In some embodiments, the image correction pattern comprises a black-and-white checkerboard pattern.

[0026] In some embodiments, when the image correction pattern is photographed by the wide-angle camera, the wide-angle camera is disposed on a vehicle, and the image correction pattern is disposed on a ground beside the vehicle.

[0027] In some embodiments, the image correction pattern extends along the ground. BRIEF DESCRIPTION OF DRAWINGS

[0028] Figure 1 is a schematic diagram illustrating an image photographed by a wide-angle camera.

[0029] Figure 2 is a flowchart illustrating a smart recognition method using a wide-angle camera according to an embodiment of the present application.

[0030] Figure 3 is an image correction pattern according to an embodiment of the present application.

[0031] Figure 4 is a photographing mode of an image correction pattern according to an embodiment of the present application.

[0032] Figure 5is a straightening of a wide-angle image according to an embodiment of the present application.

[0033] Figure 6 is a grid width restoration of a wide-angle image according to an embodiment of the present application.

[0034] Figure 7 is a grid length restoration of a wide-angle image according to an embodiment of the present application.

[0035] Figure 8 is a roll angle rotation of a wide-angle image according to an embodiment of the present application.

[0036] Figure 9 is a smart recognition system using a wide-angle camera according to an embodiment of the present application.

[0037] In the drawings:

[0038] 100: image

[0039] 110: object

[0040] 200: smart recognition method using a wide-angle camera

[0041] 210-230: steps

[0042] 300: image correction pattern

[0043] 312: black grid

[0044] 314: white grid

[0045] 310C: center line

[0046] 410: wide-angle camera

[0047] 420: vehicle

[0048] 420C: center line

[0049] 900: smart recognition system using a wide-angle camera

[0050] 910: memory

[0051] 920: processor

[0052] S1-S4: operations DETAILED DESCRIPTION

[0053] Reference will now be made to Figure 1, which is a schematic diagram illustrating an image 100 captured by a wide-angle camera. The wide-angle camera is disposed on a driver's vehicle to capture a road condition around the driver's vehicle and transmit the captured image 100 to a driving computer for determining whether the driver's vehicle is too close to other vehicles / objects. Specifically, the driving computer detects an object 110 in the image 100 and determines whether the object 110 is too close to the driver's vehicle to further decide whether to issue a warning. However, since the image 100 captured by the wide-angle camera is distorted, the driving computer is difficult to accurately calculate the distance between the driver's vehicle and the surrounding object 110 and accurately issue a warning.

[0054] Please refer to Figure 2 , which is a flowchart illustrating a smart recognition method 200 using a wide-angle camera according to an embodiment of the present application. The smart recognition method 200 using a wide-angle camera is used to correct the image data of the image 100 captured by the wide-angle camera. For example, the coordinate data in the image 100 is corrected to accurately correspond to the camera coordinates, so that the driving computer can accurately calculate the distance between the driver's vehicle and the surrounding object. In the smart recognition method 200 using a wide-angle camera, step 210 is first performed to provide an image correction pattern 300, as shown in Figure 3 In this embodiment, the image correction pattern 300 includes a black-and-white checkerboard pattern. Specifically, black squares 312 and white squares 314 are arranged in a matrix to form the image correction pattern 300, but embodiments of the present application are not limited thereto. In some embodiments, the image correction pattern 300 is formed on a correction plate, but embodiments of the present application are not limited thereto.

[0055] Please refer to Figure 2 After step 210, step 220 is then performed to capture the image correction pattern 300 on the correction plate by using the wide-angle camera 410, as shown in Figure 4 In this embodiment, the wide-angle camera 410 is disposed on the side of the vehicle 420, such as a rearview mirror. The wide-angle camera 410 can be, for example, a fisheye lens, but embodiments of the present application are not limited thereto. The correction plate is disposed on the ground beside the vehicle 420 so that the image correction pattern 300 is aligned with the vehicle 420 and extends along the ground. In this embodiment, the center line 310C of the image correction pattern 300 and the center line 420C of the vehicle 420 are parallel to each other to facilitate the wide-angle camera 410 to accurately capture the image correction pattern 300.

[0056] In some embodiments, the wide-angle camera 410 is disposed behind the vehicle 420, and the calibration target is disposed on the ground behind the vehicle 420 so that the image calibration pattern 300 is aligned with the vehicle 420 and extends along the ground to facilitate the wide-angle camera 410 to accurately capture the image calibration pattern 300.

[0057] Please refer to Figure 2 After step 220, step 230 is performed to calibrate the image calibration pattern captured in step 220. In the calibration step, intrinsic and extrinsic parameters of the wide-angle camera 410 are calculated using a calibration formula to obtain a corresponding coordinate conversion matrix. Specifically, the calibration step of step 230 uses the calibration formula to perform image rotation correction, lens pitch angle correction, inverse perspective projection, and lens yaw angle correction on the wide-angle camera 410 to straighten the grid lines in the image calibration pattern and convert the wide-angle image coordinates in the image calibration pattern to world coordinates. Because the world coordinates are based on the camera as the origin, the distance between the driver's vehicle and the surrounding objects can be calculated. The calibration formula used in the present embodiment is as follows:

[0058]

[0059] wherein X w , Y w , and Z w represent three-dimensional coordinates of the world coordinate axis; X i and Y i represent two-dimensional coordinates of the wide-angle image coordinate axis; matrix A is a lens rotation correction matrix, matrix B is a lens pitch correction and inverse perspective projection matrix, matrix C is a lens yaw correction matrix, and matrix D is a coordinate conversion matrix, which is represented as follows:

[0060] Matrix A

[0061] Matrix B

[0062] Matrix C

[0063] Matrix D

[0064] wherein θ yaw is the lens yaw angle; θ pitch is the lens pitch angle; θ roll is the lens roll angle; H is the height of the wide-angle camera; Z c is the camera depth; K x is the proportionality constant between the image X-axis direction pixel coordinate and the world coordinate; and K yis a proportion constant of the pixel coordinate of the image Y-axis direction and the world coordinate; w is the image width (e.g., in pixels); and h is the image height (e.g., in pixels).

[0065] The design of the above-mentioned matrix A, matrix B, matrix C, and matrix D (hereinafter collectively referred to as the conversion matrix) is to correct the distorted wide-angle image by considering the intrinsic parameters (camera distortion correction) and extrinsic parameters (pitch / yaw / roll angle calculation). For example, as shown in Figure 5 , the distorted wide-angle image can be corrected and straightened after being processed by the step 230 of the present embodiment, i.e. Figure 5 Operation S1. Specifically, Figure 5 The wide-angle image has grid lines, which represent the distortion of the wide-angle image. After being processed by the step 230 of the present embodiment, the grid lines are straightened. For another example, as shown in Figure 6 , the visually deformed grid line width can be restored by the yaw angle correction of the step 230 of the present embodiment, i.e. Figure 6 Operation S2. For another example, as shown in Figure 7 , the visually deformed grid line length can be restored by the pitch angle correction of the step 230 of the present embodiment, i.e. Figure 7 Operation S3. For another example, as shown in Figure 8 , the step 230 of the present embodiment is to perform roll angle rotation on the entire image, i.e. Figure 8 Operation S4.

[0066] The coordinate values (world coordinates / wide-angle image coordinates) of the known positions on the corrected version are substituted into the above-mentioned correction formula, so that the conversion matrix can be obtained, and then the relationship between X w , Y w , Z w (three-dimensional coordinates of the world coordinate axis) and X i , Y i (two-dimensional coordinates of the wide-angle image coordinate axis) can be obtained. When the user uses the wide-angle camera 410 to take a picture, the driving computer can use the above-mentioned conversion matrix to correct the wide-angle image taken by the wide-angle camera 410 and calculate the distance between the vehicle of the driver and the surrounding objects.

[0067] For example, in some embodiments, the wide-angle camera 410 can obtain the conversion matrix of the wide-angle camera 410 after being corrected by the above-mentioned smart recognition method 200 for applying the wide-angle camera. After writing the conversion matrix into the driving computer, the driving computer can use the conversion matrix in the memory to correct the wide-angle image taken by the wide-angle camera 410 and calculate the distance between the vehicle of the driver and the surrounding objects in the subsequent driving stage of the vehicle.

[0068] Please refer to Figure 9, which is a smart recognition system 900 using a wide-angle camera according to an embodiment of the present application. The smart recognition system 900 using a wide-angle camera includes a memory 910 and a processor 920. The memory 910 stores a plurality of instructions, and the processor 920 is electrically connected to the memory 910 to load the above instructions to perform the above smart recognition method 200 using a wide-angle camera. In some embodiments, the smart recognition system 900 using a wide-angle camera can be implemented as a computer device that can receive a wide-angle image captured by the wide-angle camera 410 and perform the smart recognition method 200 using a wide-angle camera to correct the wide-angle image captured by the wide-angle camera 410 to obtain the above conversion matrix.

[0069] In summary, the smart recognition method 200 using a wide-angle camera and the smart recognition system 900 using a wide-angle camera according to the embodiments of the present application use the lens rotation correction matrix (matrix A), the lens tilt correction and inverse perspective projection matrix (matrix B), the lens skew correction matrix (matrix C), and the coordinate conversion matrix (matrix D) to correct the image of the wide-angle camera, eliminate image distortion, and calculate the distance between each object in the image and the coordinate origin (camera). In this way, the accuracy of the driving computer in determining the distance between the driver's vehicle and the surrounding objects can be improved, and the driving computer can easily recognize each object in the image, greatly improving the driving safety of the driver.

[0070] Although the present application has been disclosed with embodiments as above, it is not intended to limit the present application, and anyone with ordinary knowledge in the art can make some changes and modifications without departing from the spirit and scope of the present application. Therefore, the scope of protection of the present application shall be subject to the appended claims.

Claims

1. A smart recognition method using a wide-angle camera, suitable for a wide-angle camera image correction system, characterized in that, The smart recognition method of the wide-angle camera comprises: providing an image correction pattern; using a wide-angle camera to capture the image correction pattern to obtain a correction pattern image; and performing a correction step to use a correction formula to perform picture rotation correction, lens tilt angle correction, inverse perspective projection, and lens skew angle correction on the wide-angle camera to straighten the grid lines in the correction pattern image and convert the wide-angle image coordinates in the correction pattern image into world coordinates, wherein the correction formula used in the correction step is: wherein X w , Y w , Z w represent three-dimensional coordinates of a world coordinate axis; X i , Y i represent two-dimensional coordinates of a wide-angle image coordinate axis; matrix A is a lens rotation correction matrix, matrix B is a lens tilt correction and inverse perspective projection matrix, matrix C is a lens skew correction matrix, and matrix D is a coordinate conversion matrix, which are represented as follows: matrix matrix matrix matrix wherein θ yaw is a yaw angle of the lens; θ pitch is a pitch angle of the lens; θ roll is a roll angle of the lens; H is a height of the wide-angle camera; Z c is a depth of the camera; K x is a scale constant of pixel coordinates in the X-axis direction of the image to world coordinates; K y is a scale constant of pixel coordinates in the Y-axis direction of the image to world coordinates; w is a width of the image; and h is a height of the image. 2.The smart recognition method using a wide-angle camera of claim 1, wherein The wide-angle camera is a fisheye lens camera. 3.The smart recognition method using a wide-angle camera of claim 1, wherein The image correction pattern comprises a black-and-white checkerboard pattern. 4.The smart recognition method using a wide-angle camera of claim 1, wherein When the image correction pattern is captured by the wide-angle camera, the wide-angle camera is arranged on a vehicle, and the image correction pattern is arranged on the ground beside the vehicle. 5.The smart recognition method using a wide-angle camera of claim 4, wherein, The image correction pattern extends along the ground.

6. An intelligent recognition system using a wide-angle camera, characterized by, Comprise: a memory for storing a plurality of instructions; and a processor electrically connected to the memory for loading the instructions to perform the following steps: providing an image correction pattern; using a wide-angle camera to capture the image correction pattern to obtain a correction pattern image; and performing a correction step to use a correction formula to perform picture rotation correction, lens tilt angle correction, inverse perspective projection, and lens skew angle correction on the wide-angle camera to straighten the grid lines in the correction pattern image and convert the wide-angle image coordinates in the correction pattern image into world coordinates, wherein the correction formula used in the correction step is: wherein X w , Y w , Z w represent three-dimensional coordinates of a world coordinate axis; X i , Y i represent two-dimensional coordinates of a wide-angle image coordinate axis; matrix A is a lens rotation correction matrix, matrix B is a lens tilt correction and inverse perspective projection matrix, matrix C is a lens skew correction matrix, and matrix D is a coordinate conversion matrix, which are represented as follows: matrix matrix matrix matrix wherein θ yaw is a yaw angle of the lens; θ pitch is a pitch angle of the lens; θ roll is a roll angle of the lens; H is a height of the wide-angle camera; Z c is a depth of the camera; K x is a scale constant of the pixel coordinate and the world coordinate in the X-axis direction of the image; K y is a scale constant of the pixel coordinate and the world coordinate in the Y-axis direction of the image; w is a width of the image; and h is a height of the image. 7.The smart recognition system using a wide-angle camera of claim 6, wherein The wide-angle camera is a fisheye lens camera. 8.The smart recognition system using a wide-angle camera of claim 6, wherein The image correction pattern comprises a black-and-white checkerboard pattern. 9.The smart recognition system using a wide-angle camera of claim 6, wherein When the image correction pattern is captured by the wide-angle camera, the wide-angle camera is arranged on a vehicle, and the image correction pattern is arranged on the ground beside the vehicle. 10.The smart recognition system using a wide-angle camera of claim 9, wherein The image correction pattern extends along the ground.

Images