A method, apparatus and electronic device for determining a target projection

Abstract

A method, apparatus, and electronic device for determining the projection of a target object are disclosed. The method includes: identifying a first ground contact point of a target object in an image to be processed; transforming the first ground contact point to a reference plane coordinate system based on a perspective transformation matrix to obtain a second ground contact point corresponding to the first ground contact point; marking a missing third ground contact point based on the characteristic that all ground contact points of the target object form a reference polygon; performing an inverse perspective transformation on the second and third ground contact points in the image to be processed; and connecting the second and third ground contact points after the inverse perspective transformation to determine the target projection of the target object in the image to be processed. This method transforms the first ground contact point using a perspective transformation matrix, and can make the scale ratio of the transformed coordinates close to the true scale ratio in both the horizontal and vertical coordinates without requiring accurate true scale information, thus improving the convenience of determining the target projection.

Description

Technical Field

[0001] This application relates to the field of image processing technology, and in particular to a method, apparatus and electronic device for determining target projection. Background Technology

[0002] With the development of vehicle technology, vehicles are becoming increasingly prevalent in people's lives. To detect whether a vehicle has crossed the line on the road or in a parking area, existing intelligent transportation solutions need to extract the vehicle's ground projection area from images or videos. The method for determining the vehicle's ground projection area is as follows:

[0003] Currently, the method used to determine the ground projection of a vehicle involves obtaining video image data from an image acquisition device, detecting specific targets in the video image data, and extracting the feature information of those specific targets. Based on this feature information, at least two pairs of horizontal parallel lines are extracted from the video image data. The vanishing points formed by these parallel lines determine the corresponding vanishing lines, which are straight lines that are parallel in the actual scene and intersect in the image. The points of intersection are called vanishing points. According to preset rules, the corresponding calibration parameters are calculated based on the vanishing points and / or vanishing lines. Finally, the ground projection area of ​​the vehicle is determined based on the calibration parameters.

[0004] In the above method, the vehicle is the target object, and the vehicle's ground projection is the target projection. The method used is to determine the vehicle's ground projection area based on the vanishing point in the video image data. However, in the process of determining the vanishing point, it is necessary to adjust information such as the pitch angle, yaw angle, roll angle, focal length, and height of the image acquisition device in the acquired video image, as well as to provide reference scale information of the reference object. Furthermore, the accuracy of the height information of the image acquisition device directly depends on the accuracy of the reference scale information of the provided reference object. In practical applications, due to the need to adjust a large number of parameters of the image acquisition device and to rely on the reference scale of the reference object, when the reference scale deviates, it will directly affect the height information and cause stretching and deformation of the video image, thus resulting in low convenience in determining the target projection of the target object. Summary of the Invention

[0005] This application provides a method, apparatus, and electronic device for determining target projection. By using the target matrix and the mapping relationship between the pixel coordinates of each pixel in the image plane to be processed and the coordinates of the ground plane, the ground projection of the vehicle can be determined in the image to be processed without determining the reference scale of other reference objects. This avoids the need to measure the accurate size information of the reference target on the ground and improves the convenience of determining the target projection.

[0006] In a first aspect, this application provides a method for determining a target projection, the method comprising:

[0007] The first ground contact point of the target object in the image to be processed is determined, and the first ground contact point is transformed to the reference plane coordinate system based on the perspective transformation matrix to obtain the second ground contact point corresponding to the first ground contact point; the perspective transformation matrix is ​​determined based on multiple reference point groups in the image to be processed, and the multiple reference points contained in the reference point group form a reference polygon on the reference plane coordinate system in physical space.

[0008] The missing third ground contact point is marked by the feature that all ground contact points of the target object form a reference polygon, and the second ground contact point and the third ground contact point are transformed into the image to be processed by inverse perspective transformation.

[0009] By connecting the second ground contact point after the inverse perspective transformation and the third ground contact point, the target projection of the target object in the image to be processed is determined.

[0010] By using the above method, the first ground contact point is transformed by the perspective transformation matrix, thereby obtaining the second ground contact point corresponding to the first ground contact point. This allows the target projection area of ​​the target object to be determined in the image to be processed, improving the convenience of determining the target projection.

[0011] In one possible design, the first ground contact point of the target object in the image to be processed is determined, including:

[0012] In the reference plane coordinate system of the physical space, determine the wheel-ground contact point that satisfies the rectangular relationship;

[0013] Based on the mapping relationship between the reference plane coordinate system and the pixel coordinate system of the image to be processed, multiple coordinate points in the image to be processed that correspond to the ground contact point of the wheel are determined, and each coordinate system is used as the first ground contact point.

[0014] Using the above method, based on the wheel ground contact points that satisfy the rectangular relationship, and the mapping relationship between the reference plane coordinate system and the pixel coordinate system of the image to be processed, the first ground contact point can be determined in the image to be processed, which is beneficial for determining the vehicle ground projection.

[0015] In one possible design, the missing third ground contact point is marked using the characteristics of the reference polygon formed by all ground contact points of the target object, including:

[0016] When the reference polygon is a rectangle, the unlabeled vertices in the reference polygon are determined based on the second ground contact point;

[0017] The unlabeled vertices are considered as missing third ground contact points.

[0018] By using the method described above, the third ground contact point is determined by referencing the missing vertices of the polygon, avoiding the need to rely on the reference scale of other reference objects, thus realizing the determination of the third ground contact point and improving the convenience of determining the vehicle's ground projection.

[0019] Secondly, this application provides a method for determining the perspective transformation matrix of a target projection, the method comprising:

[0020] Obtain multiple reference point groups in the image to be processed, wherein the reference point groups are multiple preset reference points associated with the target object in the image to be processed;

[0021] Each set of reference points is transformed to the reference plane coordinate system using a candidate transformation matrix containing unknown parameters.

[0022] The perspective transformation matrix is ​​determined from the candidate transformation matrices based on the polygons formed by the multiple reference points contained in each of the reference point groups.

[0023] Using the method described above, the perspective transformation matrix is ​​determined from the candidate transformation rectangles by using the polygon formed by multiple reference points in the reference point group, thereby determining the perspective transformation matrix corresponding to the target projection.

[0024] In one possible design, multiple sets of reference points in the image to be processed are obtained, including:

[0025] In the image to be processed, at least one set of first-type positioning points and at least one set of second-type positioning points that form the rectangle on the reference plane in the physical space coordinate system are determined, wherein the sides of the rectangle formed by the first-type positioning points and the sides of the right angle formed by the second-type positioning points are not parallel in the physical space.

[0026] The at least one set of first-type positioning points and the at least one set of second-type positioning points are used as multiple reference point groups.

[0027] Using the above method, at least two types of positioning points that constitute rectangles and right angles are determined from the image to be processed. These two types of positioning points are then used as multiple reference point groups, such that the reference points in the multiple reference point groups satisfy the geometric relationship of rectangles or right triangles, which is helpful in determining the perspective transformation matrix.

[0028] In one possible design, multiple sets of reference points in the image to be processed are obtained, including:

[0029] In the image to be processed, multiple sets of first-type positioning points forming rectangles on the reference plane in the physical space coordinate system are determined, and these multiple sets of first-type positioning points are used as multiple sets of reference points, wherein the sides of the multiple rectangles are not parallel in the physical space; or

[0030] In the image to be processed, multiple sets of second-type positioning points that form a right angle relationship on the reference plane in the physical space coordinate system are identified, and the multiple sets of second-type positioning points are used as multiple reference point groups, wherein the sides of the multiple right triangles that form the right angle relationship are not parallel in the physical space.

[0031] Using the above method, first-type positioning points that satisfy multiple rectangles or second-type positioning points that satisfy multiple right-angle relationships are determined in the image to be processed. Multiple reference point groups are determined by multiple rectangles or multiple right-angle relationships, which is helpful for determining the perspective transformation matrix.

[0032] In one possible design, the perspective transformation matrix is ​​determined from the candidate transformation matrices based on the polygons formed by the multiple reference points contained in each of the reference point groups, including:

[0033] Based on the candidate transformation matrix, each reference point in each of the reference point groups is transformed to the reference plane coordinate system;

[0034] Based on the right-angle relationships in the polygon in the reference plane coordinate system, a system of equations is constructed to calculate each unknown in the candidate transformation matrix;

[0035] Substituting each of the unknowns into the candidate transformation matrix yields the perspective transformation matrix, which describes the mapping relationship between the pixel coordinates of the image plane to be processed and the coordinates of the reference plane.

[0036] By using the above method, the reference point is transformed to the reference plane coordinate system, and a system of equations is constructed based on the rectangular relationships in the reference plane coordinate system. This allows the unknowns in the candidate transformation matrix to be calculated, thereby determining the perspective transformation matrix. This avoids the need to obtain accurate reference scale information, thus improving the convenience of determining the perspective transformation matrix.

[0037] Thirdly, this application provides an apparatus for determining a target projection, the apparatus comprising:

[0038] The determination module is specifically used to determine the first ground contact point of the target object in the image to be processed, and transform the first ground contact point to the reference plane coordinate system based on the perspective transformation matrix to obtain the second ground contact point corresponding to the first ground contact point;

[0039] The annotation module is specifically used to annotate the missing third ground contact point based on the characteristics of the reference polygon formed by all ground contact points of the target object, and to inversely transform the second ground contact point and the third ground contact point to the image to be processed using perspective transformation.

[0040] The connection module is specifically used to connect the second ground contact point and the third ground contact point after the inverse perspective transformation, and to determine the target projection of the target object in the image to be processed.

[0041] In one possible design, the determining module is specifically used to determine the wheel ground contact point that satisfies the rectangular relationship in the reference plane coordinate system of the physical space, and based on the mapping relationship between the reference plane coordinate system and the pixel coordinate system of the image to be processed, determine multiple coordinate points in the image to be processed that correspond to the wheel ground contact point, and use each coordinate system as the first ground contact point.

[0042] In one possible design, the annotation module is specifically used to determine the unannotated vertices in the reference polygon based on the second ground contact point when the reference polygon is a rectangle, and to use the unannotated vertices as the missing third ground contact point.

[0043] Fourthly, this application provides an apparatus for determining the perspective transformation matrix of a target projection, the apparatus comprising:

[0044] The acquisition module is specifically used to acquire multiple reference point groups in the image to be processed, wherein the reference point group is a multiple preset reference points associated with the target object in the image to be processed;

[0045] The transformation module is specifically used to transform each group of reference points to the reference plane coordinate system using a candidate transformation matrix containing unknown parameters.

[0046] The matrix module is specifically used to determine the perspective transformation matrix from the candidate transformation matrices based on the polygons formed by multiple reference points contained in each of the reference point groups.

[0047] In one possible design, the obtaining module is specifically used to determine, in the image to be processed, at least a set of first-type positioning points that form the rectangle on the reference plane in the physical space coordinate system and at least a set of second-type positioning points that form the right angle relationship, and to use the at least a set of first-type positioning points and the at least a set of second-type positioning points as multiple reference point groups.

[0048] In one possible design, the obtaining module is further configured to determine multiple sets of first-type positioning points that form a rectangle on the reference plane in the physical space coordinate system in the image to be processed, and use the multiple sets of first-type positioning points as multiple reference point groups; or to determine multiple sets of second-type positioning points that form a right angle relationship on the reference plane in the physical space coordinate system in the image to be processed, and use the multiple sets of second-type positioning points as multiple reference point groups.

[0049] In one possible design, the matrix module is specifically used to transform each reference point in each of the reference point groups to a reference plane coordinate system based on the candidate transformation matrix, construct a system of equations based on the right angle relationships in the polygons in the reference plane coordinate system, calculate each unknown in the candidate transformation matrix, and substitute each unknown into the candidate transformation matrix to obtain the perspective transformation matrix.

[0050] Fifthly, this application provides an electronic device, comprising:

[0051] Memory, used to store computer programs;

[0052] When a processor executes a computer program stored in the memory, it implements the above-described method steps for determining a target projection and a method steps for determining a perspective transformation matrix for the target projection.

[0053] Sixthly, a computer-readable storage medium storing a computer program that, when executed by a processor, implements the above-described method steps for determining a target projection and a method steps for determining a perspective transformation matrix for the target projection.

[0054] For details on each of the above-mentioned aspects one through six, and the technical effects that each aspect may achieve, please refer to the above description of the technical effects that can be achieved for the first aspect or the various possible solutions in the first aspect; these details will not be repeated here. Attached Figure Description

[0055] Figure 1 A flowchart of the steps of a method for determining a target projection provided in this application;

[0056] Figure 2 A flowchart illustrating the steps of a method for determining the perspective transformation matrix of a target projection provided in this application;

[0057] Figure 3 A schematic diagram illustrating the establishment of a Cartesian coordinate system on the plane of the image to be processed based on pixel coordinates, as provided in this application;

[0058] Figure 4 A schematic diagram of various reference points in the image to be processed provided in this application;

[0059] Figure 5 A schematic diagram of the transformed image of the image to be processed provided in this application;

[0060] Figure 6 A schematic diagram of the structure of a device for determining target projection provided in this application;

[0061] Figure 7A schematic diagram of the structure of a device for determining the perspective transformation matrix of a target projection provided in this application;

[0062] Figure 8 This is a schematic diagram of the structure of an electronic device provided in this application. Detailed Implementation

[0063] To make the objectives, technical solutions, and advantages of this application clearer, the application will be further described in detail below with reference to the accompanying drawings. The specific operational methods in the method embodiments can also be applied to the device embodiments or system embodiments. It should be noted that in the description of this application, "multiple" is understood as "at least two". "And / or" describes the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, and B existing alone. A connected to B can represent: A and B directly connected, and A and B connected through C. Furthermore, in the description of this application, terms such as "first" and "second" are used only for distinguishing the purpose of description and should not be construed as indicating or implying relative importance or order.

[0064] In previous technologies, in order to determine the target projection of a target object, when the target object is a vehicle, the method used is to determine the ground projection area of ​​the vehicle based on the vanishing point in the video image data. This requires adjusting the pitch angle, yaw angle, roll angle, focal length, and height of the image acquisition device, as well as using the reference scale of a reference object. When the reference scale deviates, it will directly affect the height information, thus resulting in low convenience in determining the target projection of the target object.

[0065] To address the aforementioned problems, this application provides a method for determining a target projection, which obtains the target projection of a target object on a reference plane without providing any accurate scale of a reference object, thus improving the convenience of obtaining the target surface projection. The methods and apparatus described in this application are based on the same technical concept. Since the principles by which the methods and apparatus solve the problems are similar, embodiments of the apparatus and methods can be referred to mutually, and repeated details will not be repeated.

[0066] The embodiments of this application will now be described in detail with reference to the accompanying drawings.

[0067] Example 1:

[0068] Reference Figure 1 This application provides a method for determining a target projection, which improves the ease of determining the target projection. The implementation process of this method is as follows:

[0069] Step S1: Determine the first ground contact point of the target object in the image to be processed, and transform the first ground contact point to the reference plane coordinate system based on the perspective transformation matrix to obtain the second ground contact point corresponding to the first ground contact point.

[0070] This application embodiment aims to obtain the target projection of a target object on a reference plane without providing any accurate scale of the reference object. The target object can be a vehicle, and the reference plane can be the ground plane or other planes. This application embodiment will use the projection of a vehicle on the ground plane as an example for illustration. First, it is necessary to determine the first ground contact point in the image to be processed. The specific process for determining the first ground contact point is as follows:

[0071] In the reference plane coordinate system of physical space, the wheel ground contact point that satisfies the preset relationship is a rectangular relationship. Since the image to be processed is an object in physical space, there is a mapping relationship between the reference plane coordinate system in physical space and the pixel coordinate system of the image to be processed. Thus, multiple coordinate points corresponding to the wheel ground contact point can be determined in the image to be processed, and the obtained multiple coordinate points are used as the first ground contact point.

[0072] After determining the first ground contact point, in order to obtain the coordinates of the first ground contact point in the reference plane coordinate system, it is necessary to transform the first ground contact point to the reference plane coordinate system through a perspective transformation matrix, and use the first ground contact point in the reference plane coordinate system as the second ground contact point. This perspective transformation matrix is ​​determined based on multiple reference point groups in the image to be processed, and the reference point groups are contained in multiple reference points on the reference plane in physical space that form a reference polygon. The reference polygon can be a rectangle, triangle, etc., which will not be explained in detail here.

[0073] Step S2: Mark the missing third ground contact point by using the characteristics of the reference polygon formed by all ground contact points of the target object, and transform the second ground contact point and the third ground contact point into the image to be processed through inverse perspective transformation.

[0074] After transforming the first ground contact point to the second ground contact point in the reference plane, since all ground contact points of the target object are vertices of the reference polygon, it is possible to determine the missing vertex of the reference polygon and use that vertex as the third ground contact point.

[0075] After determining the third ground contact point in the reference plane coordinate system, all ground contact points of the vehicle wheels have been determined. Then, the second and third ground contact points are transformed into the image to be processed through inverse perspective transformation.

[0076] Furthermore, if the target object is a vehicle and the reference plane is the ground plane, there are 3 ground contact points corresponding to one vehicle in the image to be processed. The ground contact points are transformed onto the ground plane by the perspective transformation matrix. Thus, the ground contact points of the remaining undetected vehicle wheels in the image to be processed are generated or marked by the feature that the ground contact points of the four wheels form a rectangle. The undetected ground contact points are used as the third ground contact points. These ground contact points can be obtained by key point detection algorithm or manually marked during the annotation stage of the image to be processed.

[0077] Step S3: Connect the second ground contact point and the third ground contact point after the inverse perspective transformation to determine the target projection of the target object in the image to be processed.

[0078] When the second and third ground contact points are inversely transformed into the image to be processed, by connecting the second and third ground contact points in the image to be processed, the target projection of the target object in the image to be processed can be determined. This target projection can be used in the embodiments of this application. Figure 5 The target first graphic or target second graphic, when the target object is a vehicle, the target projection can also be determined by drawing a rotated rectangle with an angled shape based on the outline of the vehicle from a top-down view.

[0079] The above method transforms the image to be processed into a bird's-eye view image, making the vehicle's projection area clearer in the transformed image. Furthermore, it avoids adjusting the parameters of the image acquisition device and relying on other reference objects, thus improving the convenience of obtaining the vehicle's ground projection.

[0080] Example 2:

[0081] Reference Figure 2 This application provides a method for determining the perspective transformation matrix of a target projection. The method can determine the perspective transformation matrix, and the implementation process of the method is as follows:

[0082] Step S21: Obtain multiple reference point groups in the image to be processed.

[0083] In order to determine the perspective transformation matrix, it is necessary to first acquire the image to be processed. The image to be processed can be acquired from an image acquisition device or from other traffic network platforms, which will not be discussed in detail here.

[0084] After obtaining the image to be processed, in order to determine the perspective transformation matrix, multiple sets of reference points need to be determined. These sets of reference points are multiple preset reference points associated with the target object in the image to be processed. These preset reference points can be vertices of rectangles and right triangles. The specific process of determining the set of reference points is as follows:

[0085] Method 1: In the image to be processed, determine at least one set of first-type positioning points that form a rectangle on the reference plane in the physical space coordinate system and at least one set of second-type positioning points that form a right angle relationship. The sides of the rectangle formed by the first-type positioning points and the sides of the right angle formed by the second-type positioning points are not parallel in physical space. The at least one set of first-type positioning points and at least one set of second-type positioning points are used as multiple reference groups.

[0086] Method 2: In the image to be processed, identify multiple sets of first-type positioning points that form rectangles on the reference plane in the physical space coordinate system, and use the first-type positioning points as multiple sets of reference points. Since multiple sets of first-type positioning points can determine multiple rectangles in the physical space coordinate system, the sides of each rectangle determined based on multiple sets of first-type positioning points are not parallel.

[0087] Method 3: In the image to be processed, identify multiple sets of second-type positioning points that form a right angle relationship on the reference plane in the physical space coordinate system, and use the second-type positioning points as multiple sets of reference points. The sides of the right triangles determined based on the multiple sets of second-type positioning points are not parallel.

[0088] The three methods described above can all determine multiple sets of reference points. The method for determining multiple sets of reference points can be selected based on the actual situation, which will not be explained in detail here.

[0089] After identifying multiple reference point groups in the image to be processed, since there is a mapping relationship between the pixel coordinate system of the image to be processed and the physical space coordinate system, it is necessary to establish a Cartesian coordinate system based on the image to be processed. A schematic diagram of establishing a Cartesian coordinate system based on the image to be processed is shown below. Figure 3 As shown, in Figure 3 In this embodiment, a rectangular coordinate system is established with the intersection of the left and bottom edges of the image to be processed as the origin, the line containing the bottom edge of the image to be processed as the horizontal axis, and the line containing the left edge of the image to be processed as the vertical axis. The origin of this rectangular coordinate system can also be located at other positions. This embodiment will use... Figure 3 Let's take an example to illustrate.

[0090] After establishing the Cartesian coordinate system, we can determine each reference point in the multiple reference point groups in the image to be processed. A schematic diagram of each reference point in the image to be processed is shown below. Figure 4 As shown, in Figure 4 In this diagram, a rectangle formed by a set of reference points on the ground plane in physical space is used as the first shape, and a right triangle formed by a set of reference points on the ground plane in physical space is used as the second shape. Due to the varying distances of objects on the ground plane in the image to be processed, both the first and second shapes appear as irregular geometric patterns. A, B, C, D, E, F, and G are reference points, and the coordinates of each reference point are obtained from the x-coordinate and y-coordinate of a rectangular coordinate system. Figure 4 The image also records the ground contact points of the vehicle wheels shown in the image to be processed. In practical applications, the first image can be obtained based on zebra crossings, square manhole covers, etc., on the traffic roads in the image to be processed. The zebra crossings are as described above. Figure 4 The second graphic can be obtained from any ground element that forms a right angle in the shaded area of ​​the image to be processed. Specifically, the second graphic can be obtained based on the ground contact point of the vehicle in the image to be processed, or a corner of the ground traffic sign, etc. The acquisition of the first graphic and the second graphic is the same as the example above, and will not be explained in detail here.

[0091] It should be noted that the first and second images are obtained based on key point detection algorithms and line detection algorithms in the image to be processed, or automatically obtained through image segmentation, or obtained by selection in the image to be processed; no specific limitation is made here.

[0092] Using the above method, a rectangular coordinate system is established on the plane of the image to be processed based on the pixel coordinates, and reference points corresponding to the first and second graphics are determined in the image to be processed, which is beneficial for determining the target image based on each reference point.

[0093] Step S22: Transform each set of reference points to the reference plane coordinate system using candidate transformation matrices containing unknown parameters.

[0094] After identifying the reference points in the image to be processed, in order to determine the perspective transformation matrix, it is necessary to obtain the mapping relationship between the coordinate system of each pixel in the image plane and the coordinate system of the ground plane. The specific process is as follows:

[0095] Since the image to be processed represents an image plane, the ground projection of the vehicle can be determined by transforming the pixel coordinates of the image plane to ground plane coordinates. Therefore, there is a mapping relationship between the coordinates of the image plane and the coordinates of the ground plane, which is expressed by the following formula (1):

[0096]

[0097] In the above formula (1), (x', y') are the coordinates in the image after the image to be processed is transformed, (x, y) are the coordinates in the image to be processed, and H is a candidate transformation matrix with unknowns, which can also be a homography transformation matrix.

[0098] Based on the above formula (1), we can obtain 1 = h 31 x+h 32 y+h 33 , from 1 = h 31 x+h 32 y+h 33 Formulas (2) and (3) can be obtained, as shown below:

[0099]

[0100]

[0101] Multiply both the numerator and denominator of formulas (2) and (3) by the scalar S = 1 / h. 33 , to obtain h' 33 =1, which gives us the formula for solving 8 unknowns. The specific formula (4) is as follows:

[0102]

[0103] After obtaining the above formula, the coordinates of each reference point in the image to be processed are then obtained. The reference points for the first image are A(x1,y1), B(x2,y2), C(x4,y4), and D(x3,y3), and the reference points for the second image are E(x5,y5), F(x6,y6), and G(x7,y7), where point F is a right-angle point. A schematic diagram of the transformed image is shown below. Figure 4 As shown, in Figure 4 In the image to be processed, A(x1,y1) is transformed into A'(x'1,y'1), B(x2,y2) is transformed into B'(x'2,y'2), C(x4,y4) is transformed into C'(x'4,y'4), D(x3,y3) is transformed into D'(x'3,y'3), E(x5,y5) is transformed into E'(x'5,y'5), F(x6,y6) is transformed into F'(x'6,y'6), and G(x7,y7) is transformed into G'(x'7,y'7).

[0104] After transforming each reference point to the reference plane coordinate system, we can obtain a schematic diagram of the transformed image of the image to be processed, as shown below. Figure 5 As shown, in Figure 5 In the diagram, the first target graphic is the first graphic transformed into the reference plane coordinate system, and the second target graphic is the second graphic transformed into the reference plane coordinate system.

[0105] Step S23: Determine the perspective transformation matrix from the candidate transformation matrices based on the polygons formed by the multiple reference points contained in each of the reference point groups.

[0106] The above describes the process of transforming each reference point to the reference plane coordinate system based on a candidate matrix with unknowns. In order to determine the perspective transformation matrix, it is also necessary to design a system of equations: let A(x1,y1) be transformed to A'(x'1,y'1), and B(x2,y2) be transformed to B'(x'2,y'2), and list the following equations:

[0107]

[0108]

[0109]

[0110]

[0111] Since the first figure is a rectangle, line A'C' is perpendicular to line A'B', resulting in equation (5), which is shown below:

[0112]

[0113] Based on the fact that the first figure is a rectangle, we can also deduce that line segment A'C' equals line segment B'D', and obtain the equations for the x-coordinate and y-coordinate respectively, as shown below:

[0114] The equation x'4-x'3=x1'-x'2 is (6)

[0115] Equation (7): y'4-y'3=y1'-y'2

[0116] Based on the above equations (1) to (7) and h 33 =1 Solving simultaneously, we can calculate the homography matrix H and use the homography matrix as the target matrix. In addition, to simplify the solution process, we can make y1' = y'2 and x1' = x'2. When y1' = y'2, x'3 = x'2 and x'4 = x1' can ensure the perpendicular relationship, thereby achieving the purpose of simplifying the calculation.

[0117] Using the method described above, the perspective transformation matrix corresponding to the candidate transformation matrix with unknowns in the image to be processed is calculated. This ensures that the ground projection of the vehicle can be determined based on the perspective transformation matrix and the vehicle's ground contact point. A schematic diagram of the vehicle's ground projection from a bird's-eye view is shown below. Figure 5 As shown, in Figure 5 This avoids the need to obtain accurate reference scale information in real time, and improves the convenience of obtaining vehicle ground projection.

[0118] Based on the above description, multiple reference point groups are marked in the image to be processed, and the ground projection of the vehicle from a bird's-eye view is obtained by mapping the coordinates of each pixel point on the plane of the image to be processed to the coordinates of the reference plane. This allows the target projection of the target object to be determined, avoiding the need to adjust the parameters of the image acquisition device and use other reference objects for positioning, thus improving the convenience of determining the target projection.

[0119] Example 3:

[0120] Based on the same inventive concept, this application also provides a device for determining target projection. This device for determining target projection implements the function of a method for determining target projection. (Refer to...) Figure 6 The device includes:

[0121] The determining module 601 is specifically used to determine the first ground contact point of the target object in the image to be processed, and transform the first ground contact point to the reference plane coordinate system based on the perspective transformation matrix to obtain the second ground contact point corresponding to the first ground contact point.

[0122] The annotation module 602 is specifically used to annotate the missing third ground contact point based on the characteristics of the reference polygon formed by all ground contact points of the target object, and to inversely transform the second ground contact point and the third ground contact point to the image to be processed using perspective transformation.

[0123] The connection module 603 is specifically used to connect the second ground contact point and the third ground contact point after the inverse perspective transformation, and to determine the target projection of the target object in the image to be processed.

[0124] In one possible design, the determining module 601 is specifically used to determine the wheel ground contact point that satisfies the rectangular relationship in the reference plane coordinate system of the physical space, and based on the mapping relationship between the reference plane coordinate system and the pixel coordinate system of the image to be processed, determine multiple coordinate points in the image to be processed that correspond to the wheel ground contact point, and use each coordinate system as the first ground contact point.

[0125] In one possible design, the annotation module 602 is specifically used to determine the unannotated vertices in the reference polygon based on the second ground contact point when the reference polygon is a rectangle, and to use the unannotated vertices as the missing third ground contact point.

[0126] Example 4:

[0127] Based on the same inventive concept, this application also provides an apparatus for determining the perspective transformation matrix of a target projection. This apparatus for determining the perspective transformation matrix of a target projection implements the function of a method for determining the perspective transformation matrix of a target projection. (Refer to...) Figure 7 The device includes:

[0128] The obtaining module 701 is specifically used to obtain multiple reference point groups in the image to be processed, wherein the reference point group is a multiple preset reference points associated with the target object in the image to be processed;

[0129] The transformation module 702 is specifically used to transform each group of reference points to the reference plane coordinate system using a candidate transformation matrix containing unknown parameters.

[0130] Matrix module 703 is specifically used to determine the perspective transformation matrix from the candidate transformation matrices based on the polygons formed by multiple reference points contained in each of the reference point groups.

[0131] In one possible design, the obtaining module 701 is specifically used to determine in the image to be processed at least one set of first-type positioning points that form the rectangle on the reference plane in the physical space coordinate system and at least one set of second-type positioning points that form the right angle relationship, and to use the at least one set of first-type positioning points and the at least one set of second-type positioning points as multiple reference point groups.

[0132] In one possible design, the obtaining module 701 is further configured to determine multiple sets of first-type positioning points that form a rectangle on the reference plane in the physical space coordinate system in the image to be processed, and use the multiple sets of first-type positioning points as multiple reference point groups; or to determine multiple sets of second-type positioning points that form a right angle relationship on the reference plane in the physical space coordinate system in the image to be processed, and use the multiple sets of second-type positioning points as multiple reference point groups.

[0133] In one possible design, the matrix module 703 is specifically used to transform each reference point in each of the reference point groups to a reference plane coordinate system based on the candidate transformation matrix, construct a system of equations based on the right angle relationships in the polygons in the reference plane coordinate system, calculate each unknown in the candidate transformation matrix, and substitute each unknown into the candidate transformation matrix to obtain a perspective transformation matrix.

[0134] Example 5:

[0135] Based on the same inventive concept, this application also provides an electronic device that can realize the functions of the aforementioned device for determining target projection and device for determining the perspective transformation matrix of target projection. (Refer to...) Figure 8 The electronic device includes:

[0136] At least one processor 801 and a memory 802 connected to at least one processor 801. In this embodiment, the specific connection medium between the processor 801 and the memory 802 is not limited. Figure 8 The example shown is the connection between processor 801 and memory 802 via bus 800. Bus 800 is... Figure 8The connections between other components are indicated by thick lines and are for illustrative purposes only, not as limiting information. The 800 bus can be divided into address bus, data bus, control bus, etc., for ease of representation. Figure 8 The term is represented by a single thick line, but this does not imply that there is only one bus or one type of bus. Alternatively, the processor 801 can also be called a controller; there is no restriction on the name.

[0137] In this embodiment, memory 802 stores instructions executable by at least one processor 801. By executing the instructions stored in memory 802, at least one processor 801 can perform the aforementioned method for determining a target projection and a method for determining a perspective transformation matrix for the target projection. Processor 801 can implement... Figure 6 and Figure 7 The functions of each module in the device shown.

[0138] The processor 801 is the control center of the device. It can connect to various parts of the control device through various interfaces and lines. By running or executing instructions stored in memory 802 and calling data stored in memory 802, the processor can perform various functions and process data, thereby monitoring the device as a whole.

[0139] In one possible design, processor 801 may include one or more processing units. Processor 801 may integrate an application processor and a modem processor, wherein the application processor mainly handles the operating system, user interface, and applications, and the modem processor mainly handles wireless communication. It is understood that the modem processor may also not be integrated into processor 801. In some embodiments, processor 801 and memory 802 may be implemented on the same chip; in some embodiments, they may also be implemented on separate chips.

[0140] The processor 801 can be a general-purpose processor, such as a central processing unit (CPU), digital signal processor, application-specific integrated circuit, field-programmable gate array, or other programmable logic device, discrete gate or transistor logic device, or discrete hardware component, capable of implementing or executing the methods, steps, and logic block diagrams disclosed in the embodiments of this application. The general-purpose processor can be a microprocessor or any conventional processor. The steps of the method for determining a target projection and the method for determining a perspective transformation matrix for a target projection disclosed in the embodiments of this application can be directly manifested as being executed by a hardware processor, or executed by a combination of hardware and software modules within the processor.

[0141] Memory 802, as a non-volatile computer-readable storage medium, can be used to store non-volatile software programs, non-volatile computer-executable programs, and modules. Memory 802 may include at least one type of storage medium, such as flash memory, hard disk, multimedia card, card-type memory, random access memory (RAM), static random access memory (SRAM), programmable read-only memory (PROM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), magnetic storage, magnetic disk, optical disk, etc. Memory 802 can be any other medium capable of carrying or storing desired program code in the form of instructions or data structures that can be accessed by a computer, but is not limited thereto. In the embodiments of this application, memory 802 can also be a circuit or any other device capable of implementing storage functions for storing program instructions and / or data.

[0142] By designing and programming the processor 801, the code corresponding to the method for determining a target projection and the method for determining the perspective transformation matrix of the target projection described in the foregoing embodiments can be embedded into the chip, thereby enabling the chip to execute these methods during operation. Figure 1 The illustrated embodiment presents steps for determining a target projection and a perspective transformation matrix for determining the target projection. How to design and program the processor 801 is a technique well-known to those skilled in the art and will not be described further here.

[0143] Based on the same inventive concept, embodiments of this application also provide a storage medium storing computer instructions that, when executed on a computer, cause the computer to perform a method for determining a target projection and a method for determining a perspective transformation matrix for the target projection described above.

[0144] In some possible implementations, various aspects of the method for determining a target projection and the perspective transformation matrix for determining a target projection provided by this application can also be implemented in the form of a program product, which includes program code that, when the program product is run on a device, causes the control device to perform the steps in the method for determining a target projection and the perspective transformation matrix for determining a target projection according to various exemplary embodiments of this application described above.

[0145] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product embodied on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0146] This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to this application. It should be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0147] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0148] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0149] Obviously, those skilled in the art can make various modifications and variations to this application without departing from the spirit and scope of this application. Therefore, if such modifications and variations fall within the scope of the claims of this application and their equivalents, this application also intends to include such modifications and variations.

Claims

1. A method of determining a target projection, characterized by, include: The first ground contact point of the target object in the image to be processed is determined, and the first ground contact point is transformed to the reference plane coordinate system based on the perspective transformation matrix to obtain the second ground contact point corresponding to the first ground contact point; the perspective transformation matrix is ​​determined based on multiple reference point groups in the image to be processed, and the multiple reference points contained in the reference point group form a reference polygon on the reference plane coordinate system in physical space. Based on the characteristic that all ground contact points of the target object form a rectangle, the unlabeled vertices in the rectangle are determined based on the second ground contact point; The unlabeled vertices are taken as the missing third ground contact points, and the second ground contact points and the third ground contact points are transformed into the image to be processed through reverse perspective transformation. By connecting the second ground contact point after the inverse perspective transformation and the third ground contact point, the target projection of the target object in the image to be processed is determined.

2. The method of claim 1, wherein, The first ground contact point of the target object in the image to be processed is determined, including: In the reference plane coordinate system of the physical space, determine the wheel-ground contact point that satisfies the rectangular relationship; Based on the mapping relationship between the reference plane coordinate system and the pixel coordinate system of the image to be processed, multiple coordinate points in the image to be processed that correspond to the wheel ground contact point are determined, and each coordinate point is taken as the first ground contact point.

3. A method for determining the perspective transformation matrix of the target projection of any of claims 1-2, characterized in that, include: Obtain multiple reference point groups in the image to be processed, wherein the reference point groups are multiple preset reference points associated with the target object in the image to be processed; Each set of reference points is transformed to the reference plane coordinate system using a candidate transformation matrix containing unknown parameters. The perspective transformation matrix is ​​determined from the candidate transformation matrices based on the reference polygons formed by the multiple reference points contained in each reference point group on the reference plane coordinate system in physical space.

4. The method of claim 3, wherein, Obtain multiple reference point groups from the image to be processed, including: In the image to be processed, at least one set of first-type positioning points forming a rectangle and at least one set of second-type positioning points forming a right angle relationship are determined on the reference plane in the reference plane coordinate system of the physical space, wherein the side of the rectangle formed by the first-type positioning points and the side of the right angle formed by the second-type positioning points are not parallel in the physical space. The at least one set of first-type positioning points and the at least one set of second-type positioning points are used as multiple reference point groups.

5. The method as described in claim 3, characterized in that, Obtain multiple reference point groups from the image to be processed, including: In the image to be processed, multiple sets of first-type positioning points forming rectangles on the reference plane in the reference plane coordinate system of the physical space are determined, and these multiple sets of first-type positioning points are used as multiple sets of reference points, wherein the sides of the multiple rectangles are not parallel in the physical space; or In the image to be processed, multiple sets of second-type positioning points that form a right angle relationship on the reference plane in the reference plane coordinate system of the physical space are identified, and the multiple sets of second-type positioning points are used as multiple reference point groups, wherein the sides of the multiple right triangles that form the right angle relationship are not parallel in the physical space.

6. The method as described in claim 3, characterized in that, The perspective transformation matrix is ​​determined from the candidate transformation matrices based on the polygons formed by the multiple reference points contained in each of the aforementioned reference point groups, including: Based on the candidate transformation matrix, each reference point in each of the reference point groups is transformed to the reference plane coordinate system; Based on the right-angle relationships in the polygon in the reference plane coordinate system, a system of equations is constructed to calculate each unknown in the candidate transformation matrix; Substituting each of the unknowns into the candidate transformation matrix yields the perspective transformation matrix, which describes the mapping relationship between the pixel coordinates of the image plane to be processed and the coordinates of the reference plane.

7. An apparatus for determining a target projection, characterized by include: The determination module is specifically used to determine the first ground contact point of the target object in the image to be processed, and transform the first ground contact point to the reference plane coordinate system based on the perspective transformation matrix to obtain the second ground contact point corresponding to the first ground contact point; the perspective transformation matrix is ​​determined based on multiple reference point groups in the image to be processed, and the multiple reference points contained in the reference point group form a reference polygon in the reference plane coordinate system of physical space; The annotation module is specifically used to determine the unannotated vertices in the rectangle based on the second ground contact point, taking into account the characteristic that all ground contact points of the target object form a rectangle. The unlabeled vertices are taken as the missing third ground contact points, and the second ground contact points and the third ground contact points are transformed into the image to be processed through reverse perspective transformation. The connection module is specifically used to connect the second ground contact point and the third ground contact point after the inverse perspective transformation, and to determine the target projection of the target object in the image to be processed.

8. An electronic device, comprising: include: Memory, used to store computer programs; A processor, when executing a computer program stored in the memory, implements the steps of the method according to any one of claims 1-6.

9. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, implements the steps of the method described in any one of claims 1-6.

Images