Method, device, equipment and storage medium for calibrating coordinate system of objects outside camera field of view

Abstract

The present application relates to a method, device, computer equipment and storage medium for calibrating the coordinate system of an object outside the camera's field of view. The method comprises: determining the three-dimensional coordinates of each of a plurality of positioning points on a target object in the object coordinate system of the target object; determining, for each of a plurality of positioning points, a multi-angle extended line image of the positioning point taken under the camera's field of view; determining the two-dimensional coordinates of the intersection of a plurality of straight lines with different extension angles in the multi-angle extended line image of each positioning point in the pixel coordinate system corresponding to the multi-angle extended line image, as the two-dimensional coordinates of the positioning point in the pixel coordinate system; calibrating the conversion relationship parameters between the object coordinate system and the camera coordinate system based on the camera's intrinsic parameters, the three-dimensional coordinates of each positioning point in the object coordinate system, and the two-dimensional coordinates of each positioning point in the pixel coordinate system. The present method can calibrate the conversion relationship between the camera coordinate system and the object coordinate system of an object outside the camera's field of view.

Description

Technical Field

[0001] The present application relates to the technical field of camera calibration, and in particular to a method, device, computer equipment and storage medium for calibrating a coordinate system of an object outside the field of view of a camera. Background Art

[0002] Camera calibration refers to the process of determining the conversion relationship parameters between the pixel coordinate system, camera coordinate system, and world coordinate system through measurement and calculation. By calibrating the camera to determine the camera's intrinsic and extrinsic parameters, the two-dimensional coordinates of the photographed object in the pixel coordinate system and the calibrated intrinsic and extrinsic parameters can be easily used to determine the object's three-dimensional coordinates in the world coordinate system.

[0003] Generally speaking, we can use a camera to photograph the object to be measured, and calibrate the camera's intrinsic and extrinsic parameters based on the two-dimensional coordinates of the object to be measured in the pixel coordinate system and the three-dimensional coordinates of the object to be measured in the world coordinate system, thereby determining the conversion relationship between the camera coordinate system and the world coordinate system where the object is located. However, when the object to be measured is outside the camera's field of view, it becomes very difficult to calibrate the conversion relationship between the camera coordinate system and the object's coordinate system. Summary of the invention

[0004] Based on this, it is necessary to provide a method, device, computer equipment and storage medium for calibrating the coordinate system of objects outside the camera's field of view, which can calibrate the conversion relationship parameters between the camera coordinate system and the object coordinate system of objects outside the camera's field of view.

[0005] On the one hand, a method for calibrating a coordinate system of an object outside a camera's field of view is provided, comprising: determining the three-dimensional coordinates of each of a plurality of positioning points on a target object in the object coordinate system of the target object; determining, for each of the plurality of positioning points, a multi-angle extended line image of the positioning point taken in the camera's field of view; wherein the multi-angle extended line image contains information about a plurality of straight lines with different extension angles extending from the positioning point; determining the two-dimensional coordinates of the intersection of the plurality of straight lines with different extension angles in the multi-angle extended line image of each positioning point in the pixel coordinate system corresponding to the multi-angle extended line image as the two-dimensional coordinates of the positioning point in the pixel coordinate system; acquiring the camera intrinsic parameters of the camera; and calibrating the conversion relationship parameters between the object coordinate system and the camera coordinate system based on the camera intrinsic parameters, the three-dimensional coordinates of each positioning point in the object coordinate system, and the two-dimensional coordinates of each positioning point in the pixel coordinate system.

[0006] In one embodiment, the plurality of positioning points include more than three positioning points, the plurality of positioning points are correspondingly distributed at a plurality of predetermined positions on the target object, and the plurality of predetermined positions are not collinear.

[0007] In one embodiment, determining the three-dimensional coordinates of each of the multiple positioning points on the target object in the object coordinate system of the target object includes: constructing the object coordinate system of the target object based on the multiple positioning points on the target object; and determining the three-dimensional coordinates of each of the multiple positioning points in the object coordinate system.

[0008] In one embodiment, the multi-angle extended line image captured under the camera field of view for determining the positioning point includes: providing a straight line extending from the positioning point as a starting point so that at least a portion of the straight line falls into the camera field of view, photographing the straight line using the camera, repeatedly adjusting the extension angle of the straight line and photographing the straight line in the field of view to obtain multiple initial images; wherein the straight lines in different initial images have different extension angles; and overlapping the multiple initial images to obtain the multi-angle extended line image.

[0009] In one embodiment, the multi-angle extended line image captured under the camera field of view for determining the positioning point includes: providing a plurality of straight lines extending from the positioning point as a starting point so that at least a portion of each of the plurality of straight lines falls within the camera field of view, adjusting the extension angles of the straight lines so that the plurality of straight lines have different extension angles relative to each other, and using the camera to capture the plurality of straight lines in the field of view to obtain the multi-angle extended line image.

[0010] In one embodiment, the straight line comprises a solid line, one end of which is fixed at the positioning point so that the positioning point serves as an extension starting point of the solid line, and the other end of the solid line can be adjusted and stretched to a predetermined position to provide a desired extension angle.

[0011] In one embodiment, the straight line includes a light ray, which is emitted by a light ray emitter. The light ray emitter is fixed at the positioning point to use the positioning point as the extension starting point of the light ray. The light ray emitter can adjust the emission angle of the light ray to provide a desired extension angle.

[0012] In one embodiment, when the number of the multiple straight lines with different extension angles in the multi-angle extended line image of each positioning point is three or more, determining the two-dimensional coordinates of the intersection points of the multiple straight lines with different extension angles in the multi-angle extended line image of each positioning point in the pixel coordinate system corresponding to the multi-angle extended line image as the two-dimensional coordinates of the positioning point in the pixel coordinate system includes: determining multiple initial intersection points of the multiple straight lines with different extension angles in the multi-angle extended line image of each positioning point; determining the center point of the multiple initial intersection points, and determining the two-dimensional coordinates of the center point in the pixel coordinate system as the two-dimensional coordinates of the positioning point in the pixel coordinate system.

[0013] On the other hand, a method for determining object coordinates in an object outside the field of view of a camera is provided, comprising: determining the three-dimensional coordinates of an object to be measured on a target object in an object coordinate system of the target object; obtaining conversion relationship parameters between the object coordinate system and the camera coordinate system; wherein the conversion relationship parameters between the object coordinate system and the camera coordinate system are determined according to the method for calibrating the coordinate system of an object outside the field of view of a camera as described in any of the above embodiments; and according to the conversion relationship parameters between the object coordinate system and the camera coordinate system, converting the three-dimensional coordinates of the object to be measured in the object coordinate system into the three-dimensional coordinates of the object to be measured in the camera coordinate system.

[0014] On the other hand, a device for calibrating a coordinate system of an object outside the camera field of view is provided, comprising: an object coordinate system coordinate determination module, used to determine the three-dimensional coordinates of each of a plurality of positioning points on a target object in the object coordinate system of the target object; a multi-angle extended line image determination module, used to determine, for each of the plurality of positioning points, a multi-angle extended line image of the positioning point taken under the camera field of view; wherein the multi-angle extended line image contains information about a plurality of straight lines extending from the positioning point and having different extension angles;

[0015] A pixel coordinate system coordinate determination module, used to determine the two-dimensional coordinates of the intersection of the multiple straight lines with different extension angles in the multi-angle extended line image of each positioning point in the pixel coordinate system corresponding to the multi-angle extended line image, as the two-dimensional coordinates of the positioning point in the pixel coordinate system; a camera intrinsic parameter acquisition module, used to acquire the camera intrinsic parameters of the camera; and a conversion relationship parameter determination module, used to calibrate the conversion relationship parameters between the object coordinate system and the camera coordinate system based on the camera intrinsic parameters, the three-dimensional coordinates of each positioning point in the object coordinate system, and the two-dimensional coordinates of each positioning point in the pixel coordinate system.

[0016] On the other hand, a computer device is provided, comprising a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, the steps of the method for calibrating the coordinate system of an object outside the field of view of the camera as described in any of the above embodiments are implemented, or the steps of the method for determining the coordinates of an object in the field of view of the camera as described in any of the above embodiments are implemented.

[0017] On the other hand, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the steps of the method for calibrating the coordinate system of an object outside the camera's field of view as described in any of the above embodiments are implemented, or the steps of the method for determining the object coordinates in an object outside the camera's field of view as described in any of the above embodiments are implemented.

[0018] The above-mentioned method, device, computer equipment and storage medium for determining the object coordinates of an object outside the camera's field of view can utilize a multi-angle extended line image captured by the camera in the field of view, and the multi-angle extended line image includes information of multiple straight lines extending from a positioning point outside the camera's field of view. By determining the two-dimensional coordinates of the intersection of the multiple straight lines in the pixel coordinate system, the two-dimensional coordinates of the positioning point outside the camera's field of view in the pixel coordinate system can be determined, and then the conversion relationship between the camera coordinate system of the camera and the object coordinate system of the object outside the camera's field of view can be conveniently calibrated. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] Figure 1 A diagram of an application environment of a method for calibrating a coordinate system of an object outside the camera field of view in one embodiment;

[0020] Figure 2 Schematic diagram of a flow chart of a method for calibrating a coordinate system of an object outside the camera field of view in one embodiment;

[0021] Figure 3 A schematic diagram of calibrating a display screen outside a camera field of view in one embodiment;

[0022] Figure 4 is a schematic diagram of 6 initial images taken by a camera in one embodiment;

[0023] Figure 5 is a schematic diagram of a multi-angle extended line image in one embodiment;

[0024] Figure 6 A schematic diagram of determining the intersection of six straight lines in one embodiment;

[0025] Figure 7 1 is a flow chart of a method for determining object coordinates of an object outside the camera field of view in one embodiment;

[0026] Figure 8 is a structural block diagram of a device for calibrating a coordinate system of an object outside the camera field of view in one embodiment;

[0027] Fig. 9 1 is a structural block diagram of a vehicle in one embodiment. DETAILED DESCRIPTION

[0028] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0029] In some fields, there may be a need to calibrate the conversion relationship between the camera coordinate system of the camera and the object coordinate system of the object outside the camera's field of view. For example, in the field of intelligent driving, the driver monitoring system (DMS) needs to monitor the driver's behavior. When it detects that the car driver is performing dangerous behavior, such as when the driver's eyes are not looking in the direction of the vehicle's travel, the system will give an alarm to reduce the occurrence of dangerous accidents. When the camera used to monitor the driver's behavior is installed on the rearview mirror at the front of the vehicle and facing the cockpit, some vehicle components such as the display screen at the front of the vehicle and the rearview mirrors on both sides of the vehicle will be outside the camera's field of view, making it impossible for the camera to capture these vehicle components, making it difficult to monitor the driver's interaction with these vehicle components outside the camera's field of view through the camera.

[0030] Therefore, the present application provides a method for calibrating the coordinate system of objects outside the camera's field of view, so as to conveniently calibrate the conversion relationship between the camera coordinate system of the camera and the object coordinate system of the object outside the camera's field of view, and further provides a method for determining the coordinates of the object in the object outside the camera's field of view, which can determine the coordinates of the object in the object outside the camera's field of view based on the conversion relationship calibrated by using the provided calibration method.

[0031] The method for calibrating the coordinate system of objects outside the camera field of view provided in this application can be applied to Figure 1 In the application environment shown. Among them, the camera 110 is communicatively connected with the computer device 120, and the camera 110 and the target object 130 are fixed in a first relative position relationship. The camera 110 can be a camera that can collect a two-dimensional image of an object within its field of view. In the calibration stage, when the camera 110 and the target object 130 remain in the first relative position relationship, the computer device 120 executes the calibration method of the coordinate system of the object outside the camera field of view of any embodiment of the present application to calibrate the conversion relationship between the camera coordinate system of the camera 110 and the object coordinate system of the target object 130. After the calibration stage is completed, in the monitoring stage, when the camera 110 and the target object 130 remain unchanged in the first relative position relationship, the computer device 120 can execute the object coordinate determination method in the object outside the camera field of view of any embodiment of the present application to determine the coordinates of the object outside the field of view of the camera 110.

[0032] It can be understood that the computer device 120 may include a single device or include multiple different devices. In the case of including multiple different devices, these different devices may be arranged in the same device or in different devices, and the method for calibrating the coordinate system of the object outside the camera's field of view and the method for determining the object coordinates in the object outside the camera's field of view may be respectively executed by different devices in these devices. For example, in the aforementioned scenario in the field of intelligent driving, the computer device 120 may be built into a vehicle. Before the vehicle is sold, it may be operated by a technician to enable the computer device 120 to execute the method for calibrating the coordinate system of the object outside the camera's field of view to calibrate the conversion relationship between the camera coordinate system and the object coordinate system and store it in the vehicle. After the vehicle is sold, the computer device 120 calls the stored conversion relationship and executes the method for determining the object coordinates in the object outside the camera's field of view to determine the object coordinates. Alternatively, the computer device 120 may also include a first device built into the vehicle and a second device placed outside the vehicle. Before the vehicle is sold, the technician may operate the second device to execute a method for calibrating the coordinate system of objects outside the camera's field of view to calibrate the conversion relationship and store it in the vehicle. After the vehicle is sold, the first device calls the stored conversion relationship and executes a method for determining the object coordinates of objects outside the camera's field of view to determine the object coordinates. In this case, the vehicle and target object 130 used in the calibration phase do not have to be the same vehicle and target object 130 used in the monitoring phase. Instead, it is sufficient to keep the vehicle and target object 130 in the same first relative position relationship between the calibration phase and the monitoring phase.

[0033] In one embodiment, a method for calibrating the coordinate system of an object outside the camera field of view is provided, such as Figure 2 As shown, this method is applied to Figure 1 The application scenario in is taken as an example to illustrate, including steps S210-S250.

[0034] Step S210 , determining the three-dimensional coordinates of each of the plurality of positioning points on the target object in the object coordinate system of the target object.

[0035] The target object may be the object to be observed or an object including the object to be observed. For example, the target object may be a display screen, and the object to be observed may be the display screen itself, or may be a button set on the display screen, a picture displayed on the display screen, or a touch button, etc.

[0036] The number N (N is a positive integer) and the positions of the multiple positioning points on the target object can be determined according to actual needs. In one embodiment, the multiple positioning points include more than three positioning points (i.e., N≥3), and the multiple positioning points are correspondingly distributed at multiple predetermined positions on the target object, and the multiple predetermined positions are not collinear. Therefore, the object coordinate system where the object is located can be determined according to the multiple positioning points.

[0037] Specifically, in one embodiment, the above step S210 includes: constructing an object coordinate system of the target object based on multiple positioning points on the target object; and determining the three-dimensional coordinates of each of the multiple positioning points in the object coordinate system.

[0038] For example, in one embodiment, the multiple positioning points may include four positioning points distributed at four locations in a rectangular distribution in the target object, and one of the four positioning points can be used as the origin to determine the x-axis passing through the positioning point at the origin and a positioning point adjacent to the positioning point at the origin, determine the y-axis passing through the positioning point at the origin and another positioning point adjacent to the positioning point at the origin, and determine the z-axis passing through the positioning point at the origin and perpendicular to the x-axis and the y-axis, thereby conveniently constructing the object coordinate system of the target object.

[0039] Take the target object 130 as the display screen 131 as an example, see Figure 3 As shown, the display screen 131 is located below the camera 110, and the camera 110 cannot directly capture the display screen 131. The multiple positioning points may include four corner points M1, M2, M3, and M4 of the display screen 131. One of the corner points M1 can be used as the origin, the axis passing through M1M2 as the x-axis, the axis passing through M1M4 as the y-axis, and the axis perpendicular to the x-axis and y-axis as the z-axis, thereby determining the object coordinate system of the display screen 131. The three-dimensional coordinates of M1, M2, M3, and M4 in the object coordinate system are respectively determined as M1=(0, 0, 0), M2=(w, 0, 0), M3=(0, h, 0), and M4=(w, h, 0). Any positioning point M i The three-dimensional coordinates in the object coordinate system can be expressed as M i =(x i ,y i , z i ), i∈{1, 2, 3, 4}. Wherein, i represents the i-th positioning point, and w and h are the width and height of the display screen 131.

[0040] Step S220: for each of the plurality of positioning points, determine a multi-angle extended line image of the positioning point captured under the camera field of view, wherein the multi-angle extended line image contains information about a plurality of straight lines extending from the positioning point with different extension angles.

[0041] The multi-angle extended line image may be determined in a variety of ways.

[0042] In one embodiment, determining a multi-angle extended line image captured under the camera field of view of a positioning point includes: providing a straight line extending from the positioning point as a starting point so that at least a portion of the straight line falls into the camera field of view, and photographing the straight line using a camera, repeatedly adjusting the extension angle of the straight line and photographing the straight line in the field of view to obtain multiple initial images; wherein the straight lines in different initial images have different extension angles; and overlapping the multiple initial images to obtain a multi-angle extended line image.

[0043] In one embodiment, determining a multi-angle extended line image captured under the camera field of view of a positioning point includes: providing multiple straight lines extending from the positioning point as a starting point so that at least a portion of each of the multiple straight lines falls into the camera field of view, adjusting the extension angles of the straight lines so that the multiple straight lines have different extension angles to each other, and using a camera to capture the multiple straight lines in the field of view to obtain a multi-angle extended line image.

[0044] In one embodiment, the straight line or each of the multiple straight lines may include a solid rope, one end of which is fixed at a positioning point so that the positioning point serves as the extension starting point of the solid rope, and the other end of the solid rope can be adjusted and stretched to a predetermined position to provide a desired extension angle.

[0045] In one embodiment, the straight line or each of the plurality of straight lines may also include a light ray, which is emitted by a light ray emitter, which is fixed at a positioning point so that the positioning point is used as an extension starting point of the light ray, and the light ray emitter can adjust the emission angle of the light ray to provide a desired extension angle. The light ray may be, for example, a laser, and the corresponding light ray emitter may be, for example, a laser lamp, and the laser emitted by the laser lamp may maintain a highly collimated characteristic, so that the captured light ray remains a straight line.

[0046] In the case where a straight line is provided and the straight line is a rope, for example, a plurality of positioning points are also included Figure 3 As an example, in step S220, when the current positioning point is M1, a rope can be placed in front of the camera 110, and the starting point of the rope is M1, and the end point can be any position, as long as the camera 110 can capture at least a part of the rope. The camera 110 captures the initial image P11, and then the starting point of the rope is kept at M1 and the end point of the rope is moved to a new position, and the camera 110 can still capture at least a part of the rope. At this time, the camera 110 captures the initial image P12, and so on. Ropes placed at different angles can be seen in FIG. Figure 3As shown, a total of 6 initial images P11 to P16 are taken, and the initial images P11 to P16 obtained by taking are shown in FIG. Figure 4 Then, the six initial images can be overlapped to obtain a multi-angle extension line image P2 corresponding to the positioning point M1, see Figure 5 By repeating the above operation, four multi-angle extension line images corresponding to the four positioning points M1, M2, M3, and M4 can be obtained.

[0047] Similarly, in the case where a straight line is provided and the straight line is a light ray, for example, multiple positioning points are also included. Figure 3 For example, when the current positioning point is M1, in step S220, a light ray emitter may be provided at the corner point M1 of the display screen 131, so that the starting point of the light ray emitted by the light ray emitter is M1. The light ray emitter may adjust the emission angle of the light ray, as long as the camera 110 can capture at least a part of the light ray. The camera 110 captures the initial image P 11 Next, the starting point of the light ray is kept at M1 and the emission angle of the light ray is changed to a new angle, which still ensures that the camera 110 can capture at least a part of the light ray. At this time, the camera 110 captures the initial image P 12 , and so on, the light ray rope emitted at different emission angles can be similarly referred to Figure 3 As shown, a total of 6 initial images P are taken 11 ~P 16 , the initial image P captured 11 ~P 16 See also Figure 4 Then, the six initial images can be overlapped to obtain a multi-angle extended line image P2, see Figure 5 By repeating the above operation, four multi-angle extension line images corresponding to the four positioning points M1, M2, M3, and M4 can be obtained.

[0048] In the case where a plurality of straight lines are provided and each straight line is a line, for example, a plurality of positioning points are also included. Figure 3 For example, in step S220, when the current positioning point is M1, six ropes can be placed in front of the camera 110 at the same time. The starting point of the six ropes is M1, and the six end points of the six ropes can be positioned at any six different positions, as long as the camera 110 can capture at least a part of each of the six ropes. The six ropes placed at different angles can be seen in FIG. Figure 3 As shown, the camera 110 captures a multi-angle extended line image P2 corresponding to the positioning point M1, see Figure 5By repeating the above operation, four multi-angle extension line images corresponding to the four positioning points M1, M2, M3, and M4 can be obtained.

[0049] Similarly, in the case where a plurality of straight lines are provided and each straight line is a radial line, for example, a plurality of positioning points are also provided. Figure 3 For example, in step S220, when the current positioning point is M1, a light ray emitter may be provided at the corner point M1 of the display screen 131, so that the light ray emitter simultaneously emits 6 light rays, the starting points of the 6 light rays are all M1, and the 6 light rays have different emission angles. As long as the camera 110 can capture at least a portion of each of the 6 light rays, the 6 light rays emitted at different emission angles may be similarly referred to. Figure 3 As shown, the camera 110 captures a multi-angle extended line image P2, see Figure 5 By repeating the above operation, four multi-angle extension line images corresponding to the four positioning points M1, M2, M3, and M4 can be obtained.

[0050] Step S230, determining the two-dimensional coordinates of the intersection of multiple straight lines with different extension angles in the multi-angle extended line image of each positioning point in the pixel coordinate system corresponding to the multi-angle extended line image as the two-dimensional coordinates of the positioning point in the pixel coordinate system;

[0051] In the aforementioned step S220, N multi-angle extension line images corresponding to N positioning points are obtained, wherein each multi-angle extension line image contains information of P straight lines with different extension angles. P is a positive integer and P≥2. The specific value of P can be set according to actual needs, and P can have the same value or different values ​​for different multi-angle extension line images.

[0052] Generally, only a part of each of the P straight lines is captured in each multi-angle extended line image, that is, only P line segments are captured. Therefore, in this step, the extension lines of the P line segments can be drawn to determine the intersection of the P straight lines. Theoretically, the P straight lines in each multi-angle extended line image should intersect at the same point in the pixel coordinate system, that is, the positioning points corresponding to the multi-angle extended line image are at the corresponding positions in the pixel coordinate system. However, in actual tests, due to various errors, when P ≥ 3, the P straight lines may have more than one intersection in the pixel coordinate system.

[0053] In one embodiment, when the number of the multiple straight lines with different extension angles in the multi-angle extended line image of each positioning point is three or more, the above-mentioned step S230 includes: determining multiple initial intersection points of the multiple straight lines with different extension angles in the multi-angle extended line image of each positioning point; determining the center point of the multiple initial intersection points, and determining the two-dimensional coordinates of the center point in the pixel coordinate system as the two-dimensional coordinates of the positioning point in the pixel coordinate system.

[0054] The center point is a representative point determined based on multiple initial intersection points, and a variety of different methods can be used to determine the center point. For example, the center point located at the center position and the two-dimensional coordinates of the center point can be determined based on the relative position relationship and distance of multiple initial intersection points, or the center point and the two-dimensional coordinates of the center point can be determined by taking the average value of the two-dimensional coordinates of multiple initial intersection points in the pixel coordinate system, etc. When determining the center point, some relatively marginal initial intersection points can be excluded first, and then the center point can be determined based on the remaining initial intersection points.

[0055] For example, see Figure 5 As shown, for the positioning point M1, a corresponding multi-angle extended line image P2 is determined, and the multi-angle extended line image P2 has information of 6 straight lines, that is, the multi-angle extended line image P2 displays 6 line segments. In this step, the 6 line segments displayed in the multi-angle extended line image P2 can be extended to determine the intersection outside the multi-angle extended line image P2. Figure 6 As shown. Figure 6 It can be seen that due to various errors, the six straight lines do not actually intersect at one point, but intersect each other within a region and may generate multiple initial intersection points. In this step, the center point m1 of the multiple initial intersection points can be taken as the intersection point m1 of the six straight lines, and the two-dimensional coordinates of the center point m1 in the pixel coordinate system can be taken as the two-dimensional coordinates of the intersection point m1 of the six straight lines in the pixel coordinate system. Similarly, the two-dimensional coordinates of the four intersection points m1, m2, m3, and m4 corresponding to the four positioning points M1, M2, M3, and M4 can be determined from the four multi-angle extended line images corresponding to the four positioning points M1, M2, M3, and M4, respectively, as the two-dimensional coordinates of the four positioning points M1, M2, M3, and M4 in the pixel coordinate system, where any positioning point M i The two-dimensional coordinate m in the pixel coordinate system i It can be expressed as m i =(u i , v i ), i∈{1, 2, 3, 4}.

[0056] Step S240, obtaining the camera intrinsic parameters of the camera.

[0057] The camera intrinsic parameter A of the camera 110 may be known and may be read from the camera 110 or obtained from the manufacturer of the camera 110. The camera intrinsic parameter A may also be obtained by calibrating the camera using various known or novel calibration methods. For example, the camera 110 may be calibrated using the Zhang Zhengyou calibration method (ZHANG Zhengyou. A flexible new technique for camera calibration [J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2000, 22 (11): 1330-1334).

[0058] The camera intrinsic parameter A of the camera 110 can be expressed as:

[0059]

[0060] In the above formula, f x =αf,f y =βf, f is the focal length of the imaging camera, α is the number of pixels per unit distance of the focal length f on the x-axis of the pixel coordinate system, β is the number of pixels per unit distance of the focal length f on the y-axis of the pixel coordinate system, c x is the coordinate of the optical center on the x-axis of the pixel coordinate system, c y is the coordinate of the optical center on the y-axis of the pixel coordinate system.

[0061] Step S250, calibrating the conversion relationship parameters between the object coordinate system and the camera coordinate system according to the camera intrinsic parameters, the three-dimensional coordinates of each positioning point in the object coordinate system, and the two-dimensional coordinates of each positioning point in the pixel coordinate system.

[0062] For any point i on the target object 130, its three-dimensional coordinate X in the camera coordinate system is i Its three-dimensional coordinates M in the object coordinate system i The following first relationship should be satisfied:

[0063] X i =[R, t]*M i

[0064] Among them, R is a 3×3 rotation matrix, and t is a 3×1 translation vector. The matrix R and the vector t are the transformation relationship parameters that need to be solved in this step.

[0065] For any point i on the target object 130, its two-dimensional coordinate m in the pixel coordinate system of the camera is i Its three-dimensional coordinates M in the object coordinate system i The following second relationship should be satisfied:

[0066] m i ′=HM i '

[0067] Among them, m i ′ is m i The homogeneous coordinates of M i ′ is M i The homogeneous coordinates corresponding to the x-axis and y-axis coordinates of . H is the homography matrix.

[0068] Specifically, m i ′=(u i , v i , 1) T , in M i When both are located on the z=0 plane of the object coordinate system, M i ′=(x i ,y i , 1) T . Then the second relation can be expanded as:

[0069]

[0070] Also with multiple positioning points including Figure 3 Taking the four corner points M1, M2, M3, and M4 of the display screen in FIG. 1 as an example, in the aforementioned step S210, the three-dimensional coordinates of the four positioning points M1, M2, M3, and M4 in the object coordinate system have been determined to be M. i =(x i ,y i , z i ), i∈{1,2,3,4}, that is, M1′=[0,0,1] T M2′=[w,0,1] T , M3′=[0,h,1] T , M4′=[w,h,1] T In the aforementioned step S230, the two-dimensional coordinates of the four positioning points M1, M2, M3, and M4 in the pixel coordinate system are determined to be m. i =(u i , v i ), i∈{1, 2, 3, 4}. Then these known quantities M i =(x i ,y i , z i ) and m i =(u i , v i ) into the second relation, we can solve the homography matrix H.

[0071] Then, the homography matrix H obtained by the solution and the known camera intrinsic parameter A determined in the aforementioned step S240 are substituted into the first relational equation to obtain the values ​​of the matrix R and the vector t, as shown in the following equation:

[0072] R=[r1,r2,r3],t=λA -1 h3.

[0073] Where r1 = λA -1 h1,r2=λA -1 h2, r3=r1×r2, λ=1 / ‖A -1 h1‖, A is the camera intrinsic parameter matrix, and the homography matrix H = [h1, h2, h3].

[0074] Thus, the conversion relationship parameter matrix R and vector t between the object coordinate system and the camera coordinate system are calibrated. Then, the conversion relationship parameters calibrated in this step can be stored for use in subsequent testing stages.

[0075] In the above-mentioned method for determining the object coordinates of an object outside the camera's field of view, a multi-angle extended line image in the camera's field of view can be utilized. The multi-angle extended line image includes information of multiple straight lines extending from a positioning point outside the camera's field of view. By determining the two-dimensional coordinates of the intersection of the multiple straight lines in the pixel coordinate system, the two-dimensional coordinates of the positioning point outside the camera's field of view in the pixel coordinate system can be determined, and then the conversion relationship between the camera coordinate system of the camera and the object coordinate system of the object outside the camera's field of view can be conveniently calibrated.

[0076] In one embodiment, Figure 7 As shown, a method for determining the coordinates of an object outside the camera field of view is provided, comprising:

[0077] Step S710 , determining the three-dimensional coordinates of the object to be measured on the target object in the object coordinate system of the target object.

[0078] The object to be measured may be the target object itself, or may be an object on the target object. Since the object coordinate system of the target object is known, the three-dimensional coordinates of the object in the object coordinate system can be determined based on the positional relationship between the object and the target object.

[0079] Step S720, obtaining the conversion relationship parameters between the object coordinate system and the camera coordinate system; wherein the conversion relationship parameters between the object coordinate system and the camera coordinate system are determined according to the calibration method of the object coordinate system outside the camera field of view in any of the above embodiments.

[0080] The conversion relationship parameters may be pre-calibrated and stored on this end according to the method for calibrating the coordinate system of objects outside the camera's field of view in any of the above embodiments. The conversion relationship parameters may also be pre-calibrated in other external devices and then transmitted and stored on this end according to the method for calibrating the coordinate system of objects outside the camera's field of view in any of the above embodiments. In this step, the stored conversion relationship parameters may be read from this end.

[0081] Step S730 , converting the three-dimensional coordinates of the object to be measured in the object coordinate system into the three-dimensional coordinates of the object to be measured in the camera coordinate system according to the conversion relationship parameters between the object coordinate system and the camera coordinate system.

[0082] By using the above-mentioned method for determining the coordinates of an object outside the camera field of view of the present application, the coordinates of the object of interest outside the field of view of the camera 110 in the camera coordinate system can be determined. Thus, it can be used to determine the interaction behavior between the driver of a vehicle and the object outside the camera field of view, so as to perform corresponding assisted driving operations according to the interaction behavior.

[0083] It should be understood that although Figure 2 and Figure 7 The steps in the flowchart are shown in sequence as indicated by the arrows, but these steps are not necessarily executed in the order indicated by the arrows. Unless otherwise specified in this document, there is no strict order restriction for the execution of these steps, and these steps can be executed in other orders. Moreover, Figure 2 and Figure 7 At least part of the steps may include multiple sub-steps or multiple stages. These sub-steps or stages are not necessarily executed at the same time, but can be executed at different times. The execution order of these sub-steps or stages is not necessarily sequential, but can be executed in turn or alternately with other steps or at least part of the sub-steps or stages of other steps.

[0084] In one embodiment, Figure 8 As shown, a device 810 for calibrating a coordinate system of an object outside the camera field of view is provided, comprising an object coordinate system coordinate determination module 811, a multi-angle extended line image determination module 812, a pixel coordinate system coordinate determination module 813, a camera intrinsic parameter acquisition module 814 and a conversion relationship parameter determination module 815, wherein:

[0085] The object coordinate system coordinate determination module 811 is used to determine the three-dimensional coordinates of each of the multiple positioning points on the target object in the object coordinate system of the target object;

[0086] The multi-angle extension line image determination module 812 is used to determine, for each of the plurality of positioning points, a multi-angle extension line image of the positioning point captured under the camera field of view; wherein the multi-angle extension line image contains information about a plurality of straight lines extending from the positioning point with different extension angles;

[0087] A pixel coordinate system coordinate determination module 813 is used to determine the two-dimensional coordinates of the intersection of multiple straight lines with different extension angles in the multi-angle extended line image of each positioning point in the pixel coordinate system corresponding to the multi-angle extended line image as the two-dimensional coordinates of the positioning point in the pixel coordinate system;

[0088] The camera intrinsic parameter acquisition module 814 is used to acquire the camera intrinsic parameters of the camera;

[0089] The conversion relationship parameter determination module 815 is used to calibrate the conversion relationship parameters between the object coordinate system and the camera coordinate system based on the camera intrinsic parameters, the three-dimensional coordinates of each positioning point in the object coordinate system, and the two-dimensional coordinates of each positioning point in the pixel coordinate system.

[0090] In one embodiment, a device 820 for determining the coordinates of an object outside the camera field of view is further provided, comprising a coordinate determination module 821 for an object coordinate system, a conversion relationship parameter acquisition module 822, and a coordinate determination module 823 for an object camera coordinate system, wherein:

[0091] The object coordinate system coordinate determination module 821 is used to determine the three-dimensional coordinates of the object under test on the target object in the object coordinate system of the target object;

[0092] A conversion relationship parameter acquisition module 822 is used to acquire the conversion relationship parameters between the object coordinate system and the camera coordinate system; wherein the conversion relationship parameters between the object coordinate system and the camera coordinate system are determined according to the camera out-of-field object coordinate system calibration method described in any of the above embodiments; the conversion relationship parameter acquisition module 822 can acquire the conversion relationship parameters from the conversion relationship parameter determination module 815;

[0093] The object camera coordinate system coordinate determination module 823 is used to convert the three-dimensional coordinates of the object under test in the object coordinate system into the three-dimensional coordinates of the object under test in the camera coordinate system according to the conversion relationship parameters between the object coordinate system and the camera coordinate system.

[0094] For the specific definition of the device 810 for calibrating the coordinate system of objects outside the camera's field of view, please refer to the definition of the method for calibrating the coordinate system of objects outside the camera's field of view above. For the specific definition of the device 820 for determining the object coordinates in objects outside the camera's field of view, please refer to the definition of the method for determining the object coordinates in objects outside the camera's field of view above, which will not be repeated here. The various modules in the above-mentioned device 810 for calibrating the coordinate system of objects outside the camera's field of view and the device 820 for determining the object coordinates in objects outside the camera's field of view can be implemented in whole or in part through software, hardware, and a combination thereof. The above-mentioned modules can be embedded in or independent of the processor in the computer device in the form of hardware, or can be stored in the memory in the computer device in the form of software, so that the processor can call and execute the operations corresponding to the above modules.

[0095] In one embodiment, Fig. 9 As shown, the present application also provides a vehicle 900, including a vehicle body 910, and a camera 110, a processor 930 and a memory 940 installed in the vehicle body 910. The processor 930 is electrically connected and / or communicatively connected to the camera 110 and the memory 940 respectively. The camera 110 is installed in the front of the vehicle body 910 and faces the cockpit 911 in the vehicle body 910, so as to collect an image of the driver sitting in the cockpit 911. The memory 940 stores a computer program, and when the processor 930 executes the computer program, the camera out-of-view object coordinate system calibration method of any of the above embodiments can be executed to calibrate the conversion relationship parameters and store them; or the camera out-of-view object coordinate system calibration method of any of the above embodiments can also be executed in an external device, and the processor 930 can receive the calibrated conversion relationship parameters from the external device and store them. In the case where the calibrated conversion relationship parameters have been stored in the vehicle 900, the processor 930 executes the method for determining the coordinates of the object outside the camera's field of view of any of the above embodiments when executing the computer program, so as to determine the three-dimensional coordinates of the object outside the camera's field of view in the camera coordinate system in real time, and at the same time determine the three-dimensional coordinates of the driver's behavior in the camera coordinate system based on the collected image of the driver, and then determine the interaction between the driver and the object outside the camera's field of view. For example, taking the object outside the camera's field of view as a display screen as an example, the driver's gaze direction in the camera coordinate system can be determined in real time based on the collected two-dimensional image of the driver's eyes, and when the gaze direction falls within the three-dimensional coordinate range of the display screen in the camera coordinate system, it can be determined that the driver's behavior is looking at the display screen.

[0096] In one embodiment, when the processor 930 executes the computer program, it also performs the corresponding assisted driving operation when the determined interactive behavior meets the predetermined conditions. For example, the vehicle 900 may also include an audio 950, a display screen 131, etc. installed in the vehicle body 910 and electrically connected and / or communicatively connected to the processor 930. When the interactive behavior meets the predetermined conditions, the processor 930 may control the audio 950, the display screen 131, etc. to issue prompts through voice, prompt sound, visual effects, etc. For example, when it is detected that the driver's gaze direction exceeds the predetermined range, an alarm is issued to remind the driver to drive in a standardized manner and reduce the occurrence of dangerous accidents.

[0097] In one embodiment, the vehicle is a vehicle, and the vehicle body is a vehicle body. In other implementations, the vehicle may also be a ship or an airplane, etc.

[0098] In one embodiment, a computer device 120 is provided, including a memory and a processor, wherein a computer program is stored in the memory, and when the processor executes the computer program, the following steps are implemented:

[0099] Determine the three-dimensional coordinates of each of the plurality of positioning points on the target object in the object coordinate system of the target object;

[0100] For each of the plurality of positioning points, respectively, a multi-angle extended line image of the positioning point captured under the camera field of view is determined; wherein the multi-angle extended line image contains information about a plurality of straight lines extending from the positioning point with different extension angles;

[0101] Determine the two-dimensional coordinates of the intersection of multiple straight lines with different extension angles in the multi-angle extension line image of each positioning point in the pixel coordinate system corresponding to the multi-angle extension line image as the two-dimensional coordinates of the positioning point in the pixel coordinate system;

[0102] Get the camera's intrinsic parameters;

[0103] According to the camera intrinsic parameters, the three-dimensional coordinates of each positioning point in the object coordinate system, and the two-dimensional coordinates of each positioning point in the pixel coordinate system, the conversion relationship parameters between the object coordinate system and the camera coordinate system are calibrated.

[0104] In one embodiment, when the processor executes the computer program, it also implements the steps of the method for calibrating the coordinate system of an object outside the camera's field of view in any of the above embodiments, or implements the steps of the method for determining the object coordinates of an object outside the camera's field of view in any of the above embodiments.

[0105] In one embodiment, a computer readable storage medium is provided, on which a computer program is stored, and when the computer program is executed by a processor, the following steps are implemented:

[0106] Determine the three-dimensional coordinates of each of the plurality of positioning points on the target object in the object coordinate system of the target object;

[0107] For each of the plurality of positioning points, respectively, a multi-angle extended line image of the positioning point captured under the camera field of view is determined; wherein the multi-angle extended line image contains information about a plurality of straight lines extending from the positioning point with different extension angles;

[0108] Determine the two-dimensional coordinates of the intersection of multiple straight lines with different extension angles in the multi-angle extension line image of each positioning point in the pixel coordinate system corresponding to the multi-angle extension line image as the two-dimensional coordinates of the positioning point in the pixel coordinate system;

[0109] Get the camera's intrinsic parameters;

[0110] According to the camera intrinsic parameters, the three-dimensional coordinates of each positioning point in the object coordinate system, and the two-dimensional coordinates of each positioning point in the pixel coordinate system, the conversion relationship parameters between the object coordinate system and the camera coordinate system are calibrated.

[0111] In other embodiments, when the computer program is executed by the processor, it also implements the steps of the method for calibrating the coordinate system of an object outside the camera field of view in any of the above embodiments, or implements the steps of the method for determining the object coordinates of an object outside the camera field of view in any of the above embodiments.

[0112] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, storage, database or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. As an illustration and not limitation, RAM is available in many forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM).

[0113] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0114] The above-mentioned embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the invention patent. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the patent of the present application shall be subject to the attached claims.

Claims

1. A method for calibrating a coordinate system of an object outside a camera's field of view, the method comprising: Determine the three-dimensional coordinates of each of a plurality of positioning points on the target object in an object coordinate system of the target object; For each of the plurality of positioning points, determine a multi-angle extended line image of the positioning point captured under the camera field of view; wherein the multi-angle extended line image contains information about a plurality of straight lines extending from the positioning point with different extension angles; Determine the two-dimensional coordinates of the intersection of the plurality of straight lines with different extension angles in the multi-angle extended line image of each positioning point in the pixel coordinate system corresponding to the multi-angle extended line image as the two-dimensional coordinates of the positioning point in the pixel coordinate system; Obtaining camera intrinsic parameters of the camera; The conversion relationship parameters between the object coordinate system and the camera coordinate system are calibrated according to the camera intrinsic parameters, the three-dimensional coordinates of each positioning point in the object coordinate system, and the two-dimensional coordinates of each positioning point in the pixel coordinate system.

2. The method according to claim 1, characterized in that The multiple positioning points include more than three positioning points, and the multiple positioning points are correspondingly distributed at multiple predetermined positions on the target object, and the multiple predetermined positions are not collinear.

3. The method according to claim 1, characterized in that The step of determining the three-dimensional coordinates of each of the plurality of positioning points on the target object in the object coordinate system of the target object comprises: Constructing an object coordinate system of the target object based on a plurality of positioning points on the target object; Determine the three-dimensional coordinates of each of the plurality of positioning points in the object coordinate system.

4. The method according to claim 1, characterized in that: The step of determining the multi-angle extended line image of the positioning point captured in the camera field of view comprises: Providing a straight line extending from the positioning point as a starting point so that at least a portion of the straight line falls into the field of view of the camera, photographing the straight line with the camera, and repeatedly adjusting the extension angle of the straight line and photographing the straight line in the field of view to obtain a plurality of initial images; wherein the straight lines in different initial images have different extension angles; and The multiple initial images are overlapped to obtain the multi-angle extended line image.

5. The method according to claim 1, characterized in that The step of determining the multi-angle extended line image of the positioning point captured in the camera field of view comprises: Provide a plurality of straight lines extending from the positioning point as a starting point so that at least a portion of each of the plurality of straight lines falls within the field of view of the camera, adjust the extension angles of the straight lines so that the plurality of straight lines have different extension angles relative to each other, and use the camera to photograph the plurality of straight lines in the field of view to obtain the multi-angle extended line image.

6. The method according to claim 4 or 5, characterized in that: The straight line includes a physical line, one end of which is fixed at the positioning point so that the positioning point serves as an extension starting point of the physical line, and the other end of the physical line can be adjusted and stretched to a predetermined position to provide a desired extension angle.

7. The method according to claim 4 or 5, characterized in that: The straight line includes a light ray, which is emitted by a light ray emitter. The light ray emitter is fixed at the positioning point to use the positioning point as an extension starting point of the light ray. The light ray emitter can adjust the emission angle of the light ray to provide a desired extension angle.

8. The method according to claim 1, characterized in that When the number of the plurality of straight lines with different extension angles in the multi-angle extended line image of each positioning point is three or more, determining the two-dimensional coordinates of the intersection of the plurality of straight lines with different extension angles in the multi-angle extended line image of each positioning point in the pixel coordinate system corresponding to the multi-angle extended line image as the two-dimensional coordinates of the positioning point in the pixel coordinate system includes: Determine a plurality of initial intersection points of the plurality of straight lines with different extension angles in the multi-angle extension line image for each positioning point; The center point of the multiple initial intersection points is determined, and the two-dimensional coordinates of the center point in the pixel coordinate system are determined as the two-dimensional coordinates of the positioning point in the pixel coordinate system.

9. A method for determining the coordinates of an object outside the field of view of a camera, the method comprising: Determine the three-dimensional coordinates of the object under test on the target object in the object coordinate system of the target object; Acquire the conversion relationship parameters between the object coordinate system and the camera coordinate system; wherein the conversion relationship parameters between the object coordinate system and the camera coordinate system are determined according to the method described in any one of claims 1 to 8; The three-dimensional coordinates of the measured object in the object coordinate system are converted into the three-dimensional coordinates of the measured object in the camera coordinate system according to the conversion relationship parameters between the object coordinate system and the camera coordinate system.

10. A device for calibrating coordinate system of objects outside camera field of view, characterized in that: The device comprises: An object coordinate system coordinate determination module, used to determine the three-dimensional coordinates of each of a plurality of positioning points on a target object in the object coordinate system of the target object; A multi-angle extended line image determination module, used to determine, for each of the plurality of positioning points, a multi-angle extended line image of the positioning point captured under the camera field of view; wherein the multi-angle extended line image contains information about a plurality of straight lines extending from the positioning point with different extension angles; A pixel coordinate system coordinate determination module, used to determine the two-dimensional coordinates of the intersection of the plurality of straight lines with different extension angles in the multi-angle extended line image of each positioning point in the pixel coordinate system corresponding to the multi-angle extended line image as the two-dimensional coordinates of the positioning point in the pixel coordinate system; A camera intrinsic parameter acquisition module, used to acquire the camera intrinsic parameters of the camera; The conversion relationship parameter determination module is used to calibrate the conversion relationship parameters between the object coordinate system and the camera coordinate system based on the camera intrinsic parameters, the three-dimensional coordinates of each positioning point in the object coordinate system, and the two-dimensional coordinates of each positioning point in the pixel coordinate system.

11. A computer device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 9 are implemented.

12. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 9 are implemented.

Images