Data processing method and computing device

Abstract

The present application is about a data processing method and computing device. The method includes: obtaining pixel coordinate data of multiple calibrated feature points in the current frame image captured by the vehicle-mounted camera, and obtaining the rotation transformation relationship and camera installation height that minimize the overall error through least squares optimization based on the pixel coordinate data of multiple calibrated feature points, camera intrinsic parameter data, pixel coordinates of preset calibrated feature points, vehicle coordinates, the rotation transformation relationship between the camera coordinate system and the vehicle coordinate system, the functional relationship between the camera intrinsic parameters, and the constraint relationship between the preset calibrated feature points.

Description

Technical Field

[0001] The present application relates to computer application technology, and in particular to data processing methods and computing devices. Background Art

[0002] Camera calibration is one of the foundations of computer vision. After camera calibration, the relationship between the 3D geometric position of a point on the surface of a spatial object and its corresponding point in the image can be determined. The parameters that need to be calibrated for a camera usually include intrinsic parameters and extrinsic parameters. Intrinsic parameters are parameters related to the camera's own characteristics, such as the camera's focal length, pixel size, etc. Extrinsic parameters determine the position and orientation of the camera in a certain 3D space.

[0003] In a vehicle-mounted camera system, the vehicle may vibrate due to the road and other reasons during driving, which may cause changes in the extrinsic parameters of the vehicle-mounted camera. Therefore, if the photos taken by the vehicle-mounted camera are used for 3D scene reconstruction, if the camera extrinsic parameters are not calibrated in time, the reliability and accuracy of the final result will be affected. Summary of the invention

[0004] The present application provides a data processing method and a computing device, which can realize timely calibration of the external parameters of a vehicle-mounted camera.

[0005] On one hand, the present application provides a data processing method, comprising:

[0006] Obtain pixel coordinate data of multiple calibrated feature points in the current frame image captured by the vehicle-mounted camera;

[0007] The current camera extrinsic parameters are obtained according to the pixel coordinate data of the multiple calibration feature points.

[0008] In some embodiments, obtaining the current camera extrinsic parameters according to the pixel coordinate data of the plurality of calibrated feature points includes:

[0009] The rotation transformation relationship and the camera installation height are obtained based on the pixel coordinate data and camera intrinsic parameter data of the multiple calibration feature points, as well as the pixel coordinates of the preset calibration feature points, the vehicle coordinates, the rotation transformation relationship between the camera coordinate system and the vehicle coordinate system, and the functional relationship between the camera intrinsic parameters.

[0010] In some embodiments, the rotation transformation relationship is obtained further based on a constraint relationship between preset calibration feature points.

[0011] In some embodiments, obtaining the rotation transformation relationship and the camera installation height includes:

[0012] The rotation transformation relationship and the camera installation height that minimize the preset overall error are obtained through least squares optimization.

[0013] In some embodiments, the functional relationship includes:

[0014] The vehicle coordinates of the calibrated feature points are equal to the inverse matrix of the camera intrinsic parameter matrix, the inverse matrix of the rotation transformation matrix between the camera coordinate system and the vehicle coordinate system multiplied by the pixel coordinates of the calibrated feature points.

[0015] In some embodiments, the plurality of calibrated feature points are located on at least three lane lines, and the constraint relationship includes:

[0016] The z coordinates of the multiple calibration feature points in the vehicle coordinate system are equal;

[0017] The x-coordinates of the calibrated feature points on the same lane line in the vehicle coordinate system are equal;

[0018] The distance between each pair of adjacent lane lines is equal to the preset lane line distance value.

[0019] In some embodiments, the overall error includes the sum of the absolute values ​​of the following error terms:

[0020] The mean square error Di of the x-coordinates of the calibrated feature points on each lane line in the vehicle coordinate system, where i=1, 2, …, K; and

[0021] The maximum error of the distance between each pair of adjacent lane lines is E(j, j+1), where j=1, 2, ..., K-1;

[0022] Among them, K is the number of lane lines in the current frame image.

[0023] In some embodiments, it also includes:

[0024] The vehicle coordinate data of the plurality of calibrated feature points are obtained according to the pixel coordinate data of the plurality of calibrated feature points, the camera internal parameter data, the functional relationship and the camera installation height.

[0025] In some embodiments, the vehicle-mounted camera is a dash cam camera.

[0026] Another aspect of the present application provides a computing device, comprising:

[0027] Processor; and

[0028] The memory stores executable codes thereon, and when the executable codes are executed by the processor, the processor is caused to execute the method as described above.

[0029] A third aspect of the present application provides a non-transitory machine-readable storage medium having executable code stored thereon. When the executable code is executed by a processor of an electronic device, the processor is caused to execute the method as described above.

[0030] In some embodiments of the present application, corresponding camera extrinsic parameters can be obtained based on pixel coordinate data of multiple calibrated feature points in different frame images captured by the vehicle-mounted camera, thereby achieving timely calibration of the vehicle-mounted camera extrinsic parameters.

[0031] In some embodiments of the present application, it is only necessary to obtain the rotation transformation relationship between the camera coordinate system and the vehicle coordinate system, and the pixel coordinates of the calibrated feature points, the vehicle coordinates, the rotation transformation relationship between the camera coordinate system and the vehicle coordinate system, the functional relationship between the camera intrinsic parameters, and the constraint relationship between the calibrated feature points are relatively simple. Therefore, the solution calculation amount is small, which can improve efficiency and save computing resources.

[0032] It should be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present application. BRIEF DESCRIPTION OF THE DRAWINGS

[0033] The above and other objects, features and advantages of the present application will become more apparent through a more detailed description of exemplary embodiments of the present application in conjunction with the accompanying drawings, wherein the same reference numerals generally represent the same components in the exemplary embodiments of the present application.

[0034] Figure 1 is a flowchart of a data processing method according to an embodiment of the present application;

[0035] Figure 2 An example of a target image selected from a video captured by a vehicle-mounted camera is shown;

[0036] Figure 3 is a flowchart of a data processing method according to another embodiment of the present application;

[0037] Figure 4 It is a schematic diagram of the structure of a computing device according to an embodiment of the present application. DETAILED DESCRIPTION

[0038] The preferred embodiments of the present application will be described in more detail below with reference to the accompanying drawings. Although the preferred embodiments of the present application are shown in the accompanying drawings, it should be understood that the present application can be implemented in various forms and should not be limited by the embodiments described herein. On the contrary, these embodiments are provided to make the present application more thorough and complete, and to fully convey the scope of the present application to those skilled in the art.

[0039] The terms used in this application are for the purpose of describing specific embodiments only and are not intended to limit this application. The singular forms of "a", "said" and "the" used in this application and the appended claims are also intended to include plural forms unless the context clearly indicates other meanings. It should also be understood that the term "and / or" used herein refers to and includes any or all possible combinations of one or more associated listed items.

[0040] It should be understood that although the terms "first", "second", "third", etc. may be used in this application to describe various information, this information should not be limited to these terms. These terms are only used to distinguish the same type of information from each other. For example, without departing from the scope of this application, the first information may also be referred to as the second information, and similarly, the second information may also be referred to as the first information. Thus, the features defined as "first" and "second" may explicitly or implicitly include one or more of the features. In the description of this application, the meaning of "multiple" is two or more, unless otherwise clearly and specifically defined.

[0041] The technical solution of the embodiments of the present application is described in detail below with reference to the accompanying drawings.

[0042] Figure 1 is a flow chart of a data processing method according to an embodiment of the present application. Figure 1 As shown, the method includes:

[0043] In S101, pixel coordinate data of a plurality of calibrated feature points in a current frame image captured by a vehicle-mounted camera is obtained.

[0044] In the present application, the vehicle-mounted camera may be, for example, a camera of a driving recorder, but is not limited thereto, and may also be a camera of other equipment installed on the vehicle.

[0045] The on-board camera can collect videos of the road and the environment on both sides while the vehicle is driving. The target image that meets the preset conditions can be selected from the video collected by the on-board camera and processed as the current frame image.

[0046] In this embodiment, if Figure 2 As shown, the selected target image may be, for example, an image with a relatively flat road surface and relatively clear lane lines, with four lane lines in each image, and lanes formed between adjacent lane lines, and four lane lines forming a total of three lanes. It is understandable that the number of lane lines in the target image selected by the present application is not limited to four, and may be, for example, three, or more than four.

[0047] Select the points on the lane line in the target image as the calibration feature points in the current frame image. You can select several points on each lane line. The selected calibration feature points are preferably located at the center of the corresponding lane line width.

[0048] In one implementation, the target image can be opened in an image processing tool, and the operator can specify the calibration feature points on the target image, and the image processing tool outputs the pixel coordinates of the specified calibration feature points. It can be understood that the present application can also obtain the pixel coordinates of the calibration feature points in other ways, not limited to this.

[0049] It is understandable that in other embodiments, calibration feature points on other static objects, such as buildings, etc., may also be selected.

[0050] In S102, the current camera extrinsic parameters are obtained according to the pixel coordinate data of the plurality of calibrated feature points.

[0051] The vehicle coordinate system is a special coordinate system used to describe the movement of the vehicle. It can be measured in meters (m). When the vehicle is stationary on a horizontal road, the X-axis of the vehicle coordinate system points to the right of the driver, the Y-axis is parallel to the ground and points to the front of the vehicle, and the Z-axis points to the top of the vehicle. The coordinates of a point in space in the vehicle coordinate system (referred to as vehicle coordinates in this application) can be recorded as Pcar (x, y, z), which is x meters to the right of the origin of the vehicle coordinate system, y meters in front, and z meters high.

[0052] The pixel coordinate system is based on pixels, with the origin being the upper left corner of the image. The coordinates of a point in the pixel coordinate system (referred to as image coordinates in this application) can be denoted as Puv(u, v).

[0053] In this application, the camera extrinsic parameter refers to the transformation relationship between the camera coordinate system and the vehicle coordinate system, including the rotation transformation relationship and the translation transformation relationship. In this embodiment, it is assumed that the origin of the vehicle coordinate system coincides with the origin of the camera coordinate system. In this way, in the transformation relationship between the camera coordinate system and the vehicle coordinate system, the translation vector is a zero vector, and only the rotation transformation relationship needs to be determined.

[0054] In one implementation, obtaining the current camera extrinsic parameters according to the pixel coordinate data of the plurality of calibrated feature points includes:

[0055] The rotational transformation relationship between the camera coordinate system and the vehicle coordinate system and the camera installation height are obtained based on the pixel coordinate data and camera intrinsic parameter data of multiple calibrated feature points, as well as the pixel coordinates of the preset calibrated feature points, the vehicle coordinates, the rotational transformation relationship between the camera coordinate system and the vehicle coordinate system, and the functional relationship between the camera intrinsic parameters.

[0056] In another implementation, obtaining the current camera extrinsic parameters according to the pixel coordinate data of the plurality of calibration feature points includes:

[0057] According to the pixel coordinate data and camera intrinsic parameter data of the multiple calibration feature points, the pixel coordinates of the preset calibration feature points, the vehicle coordinates, the rotational transformation relationship between the camera coordinate system and the vehicle coordinate system and the functional relationship between the camera intrinsic parameters, and the constraint relationship between the preset calibration feature points, the rotational transformation relationship between the camera coordinate system and the vehicle coordinate system and the camera installation height are obtained.

[0058] In this embodiment, the corresponding camera extrinsic parameters can be obtained according to the pixel coordinate data of multiple calibrated feature points in different frame images collected by the vehicle-mounted camera, thereby realizing timely calibration of the vehicle-mounted camera extrinsic parameters.

[0059] Figure 3 FIG. 1 is a flow chart of a data processing method according to another embodiment of the present invention. Figure 3 As shown, the method of this embodiment includes:

[0060] In S301, pixel coordinate data of a plurality of calibrated feature points in a current frame image captured by a vehicle-mounted camera is obtained.

[0061] It can be understood that S301 can be implemented by referring to the relevant description in S101, and will not be described in detail here.

[0062] In S302, based on the pixel coordinate data of multiple calibration feature points, camera intrinsic parameter data, pixel coordinates of preset calibration feature points, vehicle coordinates, the rotation transformation relationship between the camera coordinate system and the vehicle coordinate system and the functional relationship between the camera intrinsic parameters, and the constraint relationship between the preset calibration feature points, the rotation transformation relationship and camera installation height that minimize the overall error are obtained through least squares optimization.

[0063] In this embodiment, it is assumed that the origin of the vehicle coordinate system coincides with the origin of the camera coordinate system. In this way, in the conversion relationship between the camera coordinate system and the vehicle coordinate system, the translation vector cam_T_car is [0, 0, 0], and only the rotation part cam_R_car needs to be determined.

[0064] At this time, the camera coordinates Pcam of any calibrated feature point in the current frame image is equal to the product of the rotation transformation matrix cam_R_car between the camera coordinate system and the vehicle coordinate system and the vehicle coordinates Pcar of the calibrated feature point, that is:

[0065] Pcam = cam_R_car × Pcar (1)

[0066] According to the camera projection calculation formula and the above formula (1), it can be known that the pixel coordinate Puv of any calibrated feature point in the current frame image is equal to the product of the camera intrinsic parameter matrix A, the rotation transformation matrix cam_R_car between the camera coordinate system and the vehicle coordinate system, and the vehicle coordinate Pcar of the calibrated feature point, that is,

[0067] Puv = A × Pcam = A × cam_R_car × Pcar Formula (2)

[0068] In this application, the camera intrinsic parameter matrix A and the pixel coordinates Puv of the calibrated feature points are known. By transforming the above formula (2), the pixel coordinates of the calibrated feature points, the vehicle coordinates, the rotation transformation relationship between the camera coordinate system and the vehicle coordinate system, and the functional relationship between the camera intrinsic parameters can be obtained: the vehicle coordinates Pcar of any calibrated feature point in the current frame image are equal to the inverse matrix A.inv of the camera intrinsic parameter matrix, the inverse matrix cam_R_car.inv of the rotation transformation matrix between the camera coordinate system and the vehicle coordinate system, and the pixel coordinates of the calibrated feature point multiplied by the inverse matrix A.inv of the camera intrinsic parameter matrix, the inverse matrix cam_R_car.inv of the rotation transformation matrix between the camera coordinate system and the vehicle coordinate system, and the pixel coordinates of the calibrated feature point, that is:

[0069] Pcar = A.inv × cam_R_car.inv × Puv Formula (3)

[0070] In this embodiment, the rotation transformation matrix cam_R_car that minimizes the overall error can be obtained by solving the joint equation group consisting of the functional relationship of the above formula (3) and the constraint relationship between the calibration feature points through least squares optimization.

[0071] It is understandable that in other embodiments, the rotation transformation matrix may be solved by other methods, not limited to the least squares optimization.

[0072] In one implementation, the constraint relationship between the calibration feature points includes the following:

[0073] (a) The z coordinates of each calibrated feature point in the vehicle coordinate system are equal.

[0074] The calibration feature points selected from the current frame image are all points on the ground lane line, so the z coordinates of each calibration feature point in the vehicle coordinate system are equal.

[0075] (b) The x-coordinates of the calibrated feature points on the same lane line in the vehicle coordinate system are equal;

[0076] Each lane line in the current frame image is parallel to the y-axis of the vehicle coordinate system. The calibrated feature points selected from the current frame image are all points on the lane line. Therefore, the x-coordinates of each calibrated feature point on the same lane line in the vehicle coordinate system are equal.

[0077] (c) The distance between each pair of adjacent lane lines is equal to the preset lane line distance d.

[0078] The lane line spacing d may be obtained in advance, for example, may be obtained based on a lane line distance measurement value in a target image.

[0079] In one implementation, the overall error Cost_function may include the sum of the absolute values ​​of the following error terms:

[0080] The mean square error Di of the x-coordinates of the calibrated feature points on each lane line in the vehicle coordinate system, where i=1, 2, …, K; and

[0081] The maximum error of the distance between each pair of adjacent lane lines is E(j, j+1), where j=1, 2, ..., K-1;

[0082] Among them, K is the number of lane lines in the current frame image.

[0083] Right now:

[0084]

[0085] In this embodiment, the maximum error E(j, j+1) of the distance between adjacent lane lines can be obtained by the following formula:

[0086]

[0087] Wherein, Dmax is the maximum distance between the calibrated feature points on the adjacent j-th and j+1-th lane lines in the x-direction of the vehicle coordinate system, Dmin is the minimum distance between the calibrated feature points on the adjacent j-th and j+1-th lane lines in the x-direction of the vehicle coordinate system, and d is the preset lane line distance.

[0088] In this embodiment, the rotation transformation matrix cam_R_car that minimizes the overall error can be obtained through least squares optimization, and the z coordinate of the calibrated feature point in the vehicle coordinate system can also be obtained. Since it is assumed that the origin of the vehicle coordinate system coincides with the origin of the camera coordinate system, the z coordinate of the calibrated feature point in the vehicle coordinate system corresponds to the camera installation height. Therefore, the camera installation height can be obtained through the least squares optimization, and then the vehicle coordinate data of multiple calibrated feature points can be obtained according to the pixel coordinate data of multiple calibrated feature points, the camera intrinsic parameter data, the above functional relationship (1) and the camera installation height.

[0089] In this embodiment, it is only necessary to obtain the rotation transformation relationship between the camera coordinate system and the vehicle coordinate system, and the pixel coordinates of the calibrated feature points, the vehicle coordinates, the rotation transformation relationship between the camera coordinate system and the vehicle coordinate system, the functional relationship between the camera intrinsic parameters, and the constraint relationship between the calibrated feature points are relatively simple. Therefore, the amount of calculation required is small, which can improve efficiency and save computing resources.

[0090] Regarding the device in the above embodiment, the specific manner in which each module performs operations has been described in detail in the embodiment of the method, and will not be elaborated again here.

[0091] Figure 4 It is a schematic diagram of the structure of a computing device according to an embodiment of the present application.

[0092] See also Figure 4 , the computing device 50 includes a memory 501 and a processor 502 .

[0093] The processor 502 may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or any conventional processor, etc.

[0094] The memory 501 may include various types of storage units, such as system memory, read-only memory (ROM), and permanent storage devices. Among them, ROM can store static data or instructions required by the processor 502 or other modules of the computer. The permanent storage device may be a readable and writable storage device. The permanent storage device may be a non-volatile storage device that does not lose the stored instructions and data even after the computer is powered off. In some embodiments, the permanent storage device uses a large-capacity storage device (such as a magnetic or optical disk, flash memory) as a permanent storage device. In some other embodiments, the permanent storage device may be a removable storage device (such as a floppy disk, optical drive). The system memory may be a readable and writable storage device or a volatile readable and writable storage device, such as a dynamic random access memory. The system memory may store some or all instructions and data required by the processor at run time. In addition, the memory 501 may include any combination of computer-readable storage media, including various types of semiconductor memory chips (DRAM, SRAM, SDRAM, flash memory, programmable read-only memory), and disks and / or optical disks may also be used. In some embodiments, the memory 501 may include a readable and / or writable removable storage device, such as a laser disc (CD), a read-only digital versatile disc (such as a DVD-ROM, a double-layer DVD-ROM), a read-only Blu-ray disc, an ultra-density optical disc, a flash memory card (such as an SD card, a mini SD card, a Micro-SD card, etc.), a magnetic floppy disk, etc. The computer-readable storage medium does not include carrier waves and transient electronic signals transmitted wirelessly or wired.

[0095] The memory 501 stores executable codes, and when the executable codes are processed by the processor 502 , the processor 502 can execute part or all of the methods described above.

[0096] The scheme of the present application has been described in detail above with reference to the accompanying drawings. In the above embodiments, the description of each embodiment has its own emphasis. For the part that is not described in detail in a certain embodiment, refer to the relevant description of other embodiments. Those skilled in the art should also know that the actions and modules involved in the description are not necessarily required for the present application. In addition, it can be understood that the steps in the method of the embodiment of the present application can be adjusted in order, merged and deleted according to actual needs, and the modules in the device of the embodiment of the present application can be merged, divided and deleted according to actual needs.

[0097] In addition, the method according to the present application may also be implemented as a computer program or a computer program product, which includes computer program code instructions for executing some or all of the steps in the above method of the present application.

[0098] Alternatively, the present application can also be implemented as a non-temporary machine-readable storage medium (or computer-readable storage medium, or machine-readable storage medium) on which executable code (or computer program, or computer instruction code) is stored. When the executable code (or computer program, or computer instruction code) is executed by a processor of an electronic device (or electronic device, server, etc.), the processor executes part or all of the steps of the above-mentioned method according to the present application.

[0099] Those skilled in the art will further appreciate that the various illustrative logical blocks, modules, circuits, and algorithm steps described in connection with the application herein may be implemented as electronic hardware, computer software, or a combination of both.

[0100] The flow chart and block diagram in the accompanying drawings show the possible architecture, function and operation of the system and method according to multiple embodiments of the present application. In this regard, each square box in the flow chart or block diagram can represent a part of a module, a program segment or a code, and the part of the module, the program segment or the code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the square box can also occur in a sequence different from that marked in the accompanying drawings. For example, two continuous square boxes can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each square box in the block diagram and / or the flow chart, and the combination of the square boxes in the block diagram and / or the flow chart can be implemented with a dedicated hardware-based system that performs the specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0101] The embodiments of the present application have been described above, and the above description is exemplary, not exhaustive, and is not limited to the disclosed embodiments. Many modifications and changes will be apparent to those of ordinary skill in the art without departing from the scope and spirit of the described embodiments. The selection of terms used herein is intended to best explain the principles of the embodiments, practical applications, or improvements to the technology in the market, or to enable other persons of ordinary skill in the art to understand the embodiments disclosed herein.

Claims

1. A data processing method, characterized in that: include: Obtain pixel coordinate data of multiple calibrated feature points in the current frame image captured by the vehicle-mounted camera; Obtaining current camera extrinsics according to the pixel coordinate data of the plurality of calibrated feature points; The obtaining of the current camera extrinsic parameters according to the pixel coordinate data of the plurality of calibrated feature points comprises: According to the pixel coordinate data of the plurality of calibration feature points, the camera intrinsic parameter data, the pixel coordinates of the preset calibration feature points, the vehicle coordinates, the rotation transformation relationship between the camera coordinate system and the vehicle coordinate system and the functional relationship between the camera intrinsic parameters, and the constraint relationship between the preset calibration feature points, the rotation transformation relationship and the camera installation height that minimize the preset overall error are obtained; the overall error includes the sum of the absolute values ​​of the following error terms: The mean square error Di of the x-coordinates of the calibrated feature points on each lane line in the vehicle coordinate system, where i=1, 2, …, K; and The maximum error of the distance between each pair of adjacent lane lines is E(j, j+1), where j=1, 2, ..., K-1; Where K is the number of lane lines in the current frame image; The overall error is obtained by the following formula: 。 2. The method according to claim 1, characterized in that The obtaining of the rotation transformation relationship and the camera installation height that minimize the preset overall error comprises: The rotation transformation relationship and the camera installation height that minimize the preset overall error are obtained through least squares optimization.

3. The method according to any one of claims 1 to 2, characterized in that: The functional relationship includes: The vehicle coordinates of the calibrated feature points are equal to the inverse matrix of the camera intrinsic parameter matrix, the inverse matrix of the rotation transformation matrix between the camera coordinate system and the vehicle coordinate system multiplied by the pixel coordinates of the calibrated feature points.

4. The method according to claim 3, characterized in that The multiple calibrated feature points are located on at least three lane lines; The constraint relationships include: The z coordinates of the multiple calibration feature points in the vehicle coordinate system are equal; The x-coordinates of the calibrated feature points on the same lane line in the vehicle coordinate system are equal; The distance between each pair of adjacent lane lines is equal to the preset lane line distance value.

5. The method according to claim 2, characterized in that Also includes: The vehicle coordinate data of the plurality of calibrated feature points are obtained according to the pixel coordinate data of the plurality of calibrated feature points, the camera internal parameter data, the functional relationship and the camera installation height.

6. The method according to claim 1, characterized in that The vehicle-mounted camera is a camera of a driving recorder.

7. A computing device, characterized in that include: processor; as well as A memory having executable codes stored thereon, which, when executed by the processor, causes the processor to execute the method according to any one of claims 1 to 6.

Images