Image processing method and device

Abstract

The invention discloses an image processing method and device, and relates to the technical field of computer vision, and the method comprises the steps: obtaining a first image collected by a fixed monocular camera; determining a target plane existing in the first image; determining an N-polygon on the target plane, and recording position data of N vertexes of the N-polygon, N being an integer not less than 4; the positions of the N vertexes are adjusted, a corresponding transformation matrix is obtained based on the position data, so that a second image obtained after the first image is transformed based on the transformation matrix meets the set requirement, and the set requirement is that the second image is a top view meeting the orthogonal view angle of the target plane; determining the transformation matrix corresponding to the second image meeting the set requirement as a target transformation matrix; a scaling factor is determined based on a physical size of a first object in the first image and a pixel size of the first object in the second image.

Description

Technical Field

[0001] The present application relates to the field of computer vision measurement technology, and more specifically, to an image processing method and device. Background Art

[0002] As a common data acquisition device, cameras are widely used in production and life. In some business areas where cameras are used, it is necessary to perform some measurement operations on the targets in the scene: such as measuring relative distance, measuring speed, etc. However, there are certain problems in applying distance measurement and speed measurement to fixed cameras in actual scenarios: whether it is a newly installed camera or an existing camera, the installation scene is very complex, and the installation height and angle are not fixed. At this time, it is difficult to obtain the parameters required for distance measurement.

[0003] Traditionally, obtaining the parameters required for ranging usually requires implementation based on special hardware acquisition devices, such as binocular cameras, ultrasonic radars, etc. However, these devices are too expensive compared to the most widely used monocular cameras. In addition, obtaining the parameters required for ranging based on binocular cameras, ultrasonic radars and other devices requires complex algorithm processing, and the performance and power consumption requirements of the equipment are also very high. Summary of the invention

[0004] In view of this, this application provides the following technical solutions:

[0005] An image processing method, comprising:

[0006] Obtain a first image captured by a fixed-setting monocular camera;

[0007] determining a target plane present in the first image;

[0008] Determine an N-gon on the target plane, and record position data of N vertices of the N-gon, where N is an integer not less than 4;

[0009] Adjusting the positions of the N vertices, and obtaining a corresponding transformation matrix based on the position data, so that a second image obtained after the first image is transformed based on the transformation matrix meets a set requirement, wherein the set requirement is that the second image is a top view that meets the orthogonal perspective of the target plane;

[0010] Determine the transformation matrix corresponding to the second image that meets the setting requirements as the target transformation matrix;

[0011] A scaling factor is determined based on a physical size of a first object in the first image and a pixel size of the first object in the second image.

[0012] Optionally, the adjusting the positions of the N vertices and obtaining a corresponding transformation matrix based on the position data so that a second image obtained after the first image is transformed based on the transformation matrix meets a set requirement includes:

[0013] Adjusting the positions of the N vertices;

[0014] Determine a transformation matrix based on the position data of the N vertices before and after the positions are adjusted, and convert the first image into a second image based on the transformation matrix;

[0015] The above process of adjusting the position and converting the image is repeated until the converted second image meets the set requirements.

[0016] Optionally, N is 4, and determining the transformation matrix based on the position data of the N vertices before and after the positions are adjusted includes:

[0017] Construct a non-singular homography transformation matrix based on the position data of the four vertices before and after the position adjustment;

[0018] Determine two constraint equations of the homography transformation matrix based on the coordinates of the vertex pair corresponding to each vertex, and obtain eight constraint equations in total, wherein the vertex pair includes the vertex coordinates before the vertex position is adjusted and the vertex coordinates after the position is adjusted;

[0019] Normalizing the homography matrix;

[0020] The homography transformation matrix is ​​solved based on the eight constraint equations to obtain a transformation matrix.

[0021] Optionally, converting the first image into a second image based on the transformation matrix includes:

[0022] Based on the transformation matrix, reverse mapping processing is performed on all pixels in the first image to obtain a second image.

[0023] Optionally, it also includes:

[0024] A distance between a first target object and a second target object in the first acquired image is determined based on the target transformation matrix and the scaling factor.

[0025] Optionally, determining the distance between the first target object and the second target object in the first acquired image based on the target transformation matrix and the scaling factor includes:

[0026] Converting the first acquired image into a second acquired image based on the target transformation matrix;

[0027] determining a pixel distance between a first target object and a second target object in the second acquired image;

[0028] A physical distance between the first target object and the second target object is determined based on the pixel distance and the scaling factor.

[0029] Optionally, it also includes:

[0030] A speed of a first target object in the captured video is determined based on the target transformation matrix and the scaling factor.

[0031] Optionally, determining the speed of the target object in the captured video based on the target transformation matrix and the scaling factor includes:

[0032] Based on the target transformation matrix, converting a reference point of a target object in a first video frame image in the acquired video to a first pixel position, and converting the reference point of the target object in a second video frame image in the acquired video to a second pixel position, wherein the first video frame image and the second video frame image include the same target object;

[0033] determining a pixel distance of the target object based on the first pixel position and the second pixel position;

[0034] Determine the moving distance of the target object based on the pixel distance and the scaling factor;

[0035] Based on the acquisition time difference between the first video frame image and the second video frame image;

[0036] The moving speed of the target object is determined based on the moving distance and the acquisition time difference.

[0037] Optionally, the target plane is a plane whose position does not change.

[0038] The present application also discloses an image processing device, comprising:

[0039] An image acquisition module, used to acquire a first image captured by a fixedly set monocular camera;

[0040] A plane determination module, used to determine a target plane existing in the first image;

[0041] A vertex determination module, used to determine an N-gon on the target plane and record position data of N vertices of the N-gon, where N is an integer not less than 4;

[0042] a vertex adjustment module, used for adjusting the positions of the N vertices, and obtaining a corresponding transformation matrix based on the position data, so that a second image obtained after the first image is transformed based on the transformation matrix meets a set requirement, wherein the set requirement is that the second image is a top view that meets the orthogonal perspective of the target plane;

[0043] A matrix determination module, used to determine the transformation matrix corresponding to the second image that meets the set requirements as a target transformation matrix;

[0044] A scaling factor determination module is configured to determine a scaling factor based on a physical size of a first object in the first image and a pixel size of the first object in the second image.

[0045] Through the above technical solution, it can be known that the embodiment of the present application discloses an image processing method and device, which can obtain the target transformation matrix by analyzing and processing the image captured by the monocular camera, that is, converting the original image captured by the monocular camera into the transformation parameters of the second image under the "top view" perspective orthogonal to the target plane, and can also obtain the scaling factor between the pixel size and the actual physical size in the second image, thereby providing a basis for subsequent measurement tasks. The above-mentioned method of obtaining the target transformation parameters and scaling factors is simple and convenient, and is suitable for various scenarios with the need to obtain the target transformation matrix corresponding to the monocular camera. BRIEF DESCRIPTION OF THE DRAWINGS

[0046] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on the provided drawings without paying any creative work.

[0047] Figure 1 A flowchart of an image processing method disclosed in an embodiment of the present application;

[0048] Figure 2 An example diagram of four vertices in the first image disclosed in the embodiment of the present application;

[0049] Figure 3 The embodiment disclosed in this application Figure 2 An example diagram of a second image converted from a first image;

[0050] Figure 4 A schematic diagram of a process for determining a target transformation matrix disclosed in an embodiment of the present application;

[0051] Figure 5 A schematic diagram of a ranging implementation process disclosed in an embodiment of the present application;

[0052] Figure 6 A schematic diagram of a speed measurement implementation process disclosed in an embodiment of the present application;

[0053] Figure 7 A schematic diagram of the structure of an image processing device disclosed in an embodiment of the present application. DETAILED DESCRIPTION

[0054] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0055] The embodiments of the present application can be applied to electronic devices. The present application does not limit the product form of the electronic device, which can include but is not limited to smart phones, tablet computers, wearable devices, personal computers (PCs), netbooks, etc., and can be selected according to application requirements.

[0056] Figure 1 This is a flowchart of an image processing method disclosed in an embodiment of the present application. Figure 1 As shown, the image processing method may include:

[0057] Step 101: Obtain a first image captured by a fixedly set monocular camera.

[0058] The image processing method disclosed in the present application can be implemented based only on images captured by a monocular camera, and the monocular camera is fixedly set in some places with image or video capture requirements, such as roads, shopping malls, squares, etc.

[0059] In this embodiment, a first image captured by a fixed monocular camera is first obtained, and corresponding parameters can be obtained by corresponding processing of the first image. The parameters can be used for subsequent applications of distance measurement and speed measurement based on the monocular camera.

[0060] Step 102: Determine the target plane existing in the first image.

[0061] The target plane is a plane whose position does not change, that is, a plane that does not physically move, such as the ground or road surface in the first image. Of course, if the image processing method is subsequently applied to a distance measurement or speed measurement scenario, the selected target plane is a plane related to the measurement object. For example, in a highway distance measurement scenario, a vehicle is the measurement object, and the vehicle is traveling on the road surface, then the road surface can be determined as the target plane.

[0062] Step 103: Determine an N-gon on the target plane, and record position data of N vertices of the N-gon, where N is an integer not less than 4.

[0063] The position data of the vertex represents the pixel coordinate position of the vertex in the first image. Since the first image is a two-dimensional image, the position data of the vertex includes horizontal coordinate data and vertical coordinate data. For example, if N is 4, then it is determined to be a quadrilateral with four vertices, such as Figure 2 The coordinates of the four vertices marked in are: pt1 (511, 420), pt2 (1005, 423), pt3 (1096, 566), pt4 (428, 570). Any three of the N vertices are not on the same straight line, so that the four vertices can be connected in sequence to form an N-gon.

[0064] Step 104: Adjust the positions of the N vertices and obtain a corresponding transformation matrix based on the position data, so that a second image obtained after the first image is transformed based on the transformation matrix meets the set requirements, and the set requirements are that the second image is a top view that meets the orthogonal perspective of the target plane.

[0065] After determining and recording the positions of the N vertices, the positions of at least four of the vertices can be adjusted, and the at least four vertices are vertices where any three points are not collinear. It is known to those skilled in the art that the picture / video captured by the camera is a perspective projection effect. If the actual distance of the target in the physical world in the image is to be obtained later, the perspective projection effect must first be transformed into an orthographic projection effect. The parameters required to transform the orthographic projection effect are transformation parameters, which can also be called a priori transformation parameters, that is, the transformation matrix. Since monocular cameras lack depth perception, complex calibration or on-site calibration is often required. Traditionally, for scenes with vehicle-mounted cameras, chessboard calibration is generally used before measurement; chessboard calibration requires the preparation of a chessboard and placing it on-site for image acquisition. It is suitable for close-ups and not for fixed cameras with wide fields of view such as squares.

[0066] In the embodiment of the present application, the transformation parameters do not need to be calibrated on site, and can be achieved by simply adjusting the multiple vertices of the polygon. Specifically, during the process of adjusting the vertex positions, the transformation parameters are calculated synchronously, and the first image is converted in real time to a "top view" perspective orthogonal to the target plane until the image is relatively consistent with the top view. Figure 3 Shown is based on Figure 2The image shown is converted to obtain a second image from a "top view" perspective orthogonal to the target plane. Compared with the perspective projection image, the distance between two pixel points in the orthogonal projection image is the pixel distance in equal linear proportion. After determining the scaling factor between the pixel size and the physical size, the actual distance can be determined based on the pixel distance in the second image.

[0067] Step 105: Determine the transformation matrix corresponding to the second image that meets the setting requirements as the target transformation matrix.

[0068] The first image is converted to a second image from an orthogonal "top-down" perspective, and the determined target transformation matrix can be used to perform measurement tasks later. For existing fixed cameras, this implementation solution can achieve the acquisition of prior parameters (target transformation matrix) in the cloud without the need for users to carry auxiliary devices such as chessboards to the site. After obtaining the prior parameters, the corresponding measurement tasks can be achieved by combining algorithms such as distance measurement and speed measurement.

[0069] Step 106: Determine a scaling factor based on the physical size of the first object in the first image and the pixel size of the first object in the second image.

[0070] If subsequent measurement tasks are to be performed based on the target transformation matrix, it is also necessary to determine the scaling factor between the pixel size of the first object in the second image and its physical size in the real world, so as to subsequently calculate the distance between the two objects in the real environment based on the pixel distance between different objects in the image transformed to the "top view" perspective orthogonal to the target plane.

[0071] Specifically, an object with known actual physical size in the first image or the second image can be selected as a reference to calculate the relationship between the pixel and the actual physical length. Find a reference object of known size in the image, calculate the length and width of the pixel in the transformed image (the second image), and obtain the scaling coefficients rx and ry of the transformed pixel and the actual physical size (representing the actual distance represented by 1 pixel in the x and y directions on the adjusted image, respectively, in units of m / pix).

[0072] like Figure 3 , by consulting the road construction specifications or actual measurements, we can find that the stone brick fence of the tree on the left is 1.2mx1.2m. In the converted image (second image), the vertex coordinates of the fence are A(369,663), B(399,663), and C(399,626). Then AB=30pix, BC=36pix, rx=1.2 / 30=0.04m / pix, ry=1.2 / 36=0.033m / pix. That is, in the converted image coordinate system, the length of each pixel of the horizontal coordinate is about 0.04m, and the length of each pixel of the vertical coordinate is about 0.003m.

[0073] It should be noted that there is no logical sequence relationship between step 105 and step 106. As long as a second image that meets the set requirements is obtained, step 105 can be performed first and then step 106, or step 106 can be performed first and then step 105, or both steps can be performed simultaneously. This application does not impose any fixed restrictions on this.

[0074] The image processing method described in the embodiment of the present application can obtain the target transformation matrix by analyzing and processing the image captured by the monocular camera, that is, converting the original image captured by the monocular camera into the transformation parameters of the second image under the "top view" perspective orthogonal to the target plane, and can also obtain the scaling factor between the pixel size and the actual physical size in the second image, thereby providing a basis for subsequent measurement tasks. The above-mentioned method of obtaining the target transformation parameters and scaling factors is simple and convenient, and is suitable for various scenarios that require obtaining the target transformation matrix corresponding to the monocular camera.

[0075] In the above embodiment, the adjusting the positions of the N vertices and obtaining the corresponding transformation matrix based on the position data so that the second image obtained after the first image is transformed based on the transformation matrix meets the set requirements may include: adjusting the positions of the N vertices; determining the transformation matrix based on the position data before and after the adjustment of the N vertex positions, and converting the first image into the second image based on the transformation matrix; repeating the above process of adjusting the position and converting the image until the converted second image meets the set requirements.

[0076] In one implementation, N is 4, and a non-singular homography transformation matrix can be first constructed based on the position data of the four vertices before and after adjustment, and then the two constraint equations of the homography transformation matrix are determined based on the coordinates of the vertex pair corresponding to each vertex, resulting in a total of eight constraint equations, where the vertex pair includes the vertex coordinates before the vertex is adjusted and the vertex coordinates after the adjustment. Finally, the homography matrix is ​​normalized, and the homography transformation matrix is ​​solved based on the eight constraint equations to obtain a transformation matrix.

[0077] The converting the first image into the second image based on the transformation matrix may include: performing reverse mapping processing on all pixel points in the first image based on the transformation matrix to obtain the second image.

[0078] The transformation matrix that transforms the coordinates of the first image to the second image is referred to as the forward transformation matrix. The forward transformation matrix is ​​full rank, and the inverse matrix is ​​obtained by inverting it. Mapping the first image pixel by pixel according to the forward transformation matrix to the second image can be called forward mapping; mapping the second image pixel by pixel according to the inverse matrix to the first image is called reverse mapping.

[0079] Figure 4 The following is a flow chart of determining a target transformation matrix disclosed in an embodiment of the present application. Figure 4 As shown, in a specific implementation, determining the target transformation matrix may include:

[0080] The first step is to obtain a fixed camera image (first image) for a fixed monocular camera and import it into a ranging auxiliary device, that is, a device for implementing the image processing method described in this application.

[0081] The second step is to select the target plane, such as setting the ground plane as the target plane. The user interactively presets 4 points to form a quadrilateral. It is used as the reference point before transformation. The reference coordinates before transformation are denoted by coordinates x = (x, y, 1) T .

[0082] The third step is to drag the set quadrilateral vertex reference point in the device to adjust the position of the reference point. The adjusted reference point is recorded as x′=(x′,y′,1) T According to the changes before and after the reference point, the device calculates according to the formula x′=Hx to obtain a non-singular homography transformation matrix H 3×3 .

[0083] From the coordinates of one vertex, two constraint equations can be obtained. From four vertex pairs, eight constraint equations can be obtained. Normalize the homography matrix so that h 33 =1, that is, H 3×3 The third row and third column in the matrix H are assigned a value of 1. 3×3 There are 8 parameters to be solved. You can use 4 vertex pairs to solve the homography matrix (where any three points in any group are not collinear). When any three points in the two groups of point pairs are not collinear, the matrix A is a full-rank reversible matrix. Using the knowledge of SVD decomposition and LU decomposition in matrix theory, solve for h and you can get the transformation matrix H 3×3 . This homography transformation matrix is ​​the transformation matrix that needs to be obtained.

[0084] According to the obtained transformation matrix applied to each point on the image, the reverse mapping relationship x = H is applied -1 x′, calculate the position of each point after transformation, transform the image synchronously, and obtain the visualized image; that is, the process of obtaining the orthogonal projection image by multiplying the perspective projection image (i.e. the image captured by the camera) pixel by pixel with the transformation matrix. The user determines the final effect based on the visualized image. Repeat the steps of adjusting the reference point and transforming the image until the transformed image is more consistent with the orthogonal perspective "top view" of the target plane.

[0085] After the adjustment is completed, the transformation matrix can be saved, which is also the target transformation matrix.

[0086] Specifically, the traditional concept in computer graphics is adopted to use a homogeneous coordinate system to represent the coordinates in the image.

[0087] That is, for the coordinates (x, y) on the original image T , which can be expressed as Use x=(x,y,1) T and x′=(x′,y′,1) T Represents the corresponding two point pairs before and after the transformation on the image. Then, a non-singular homography transformation matrix H can be used 3×3 To express the transformation relationship between x and x', x' = Hx, that is:

[0088]

[0089] After finishing, we get:

[0090] h 11 x+h 12 y+h 13 -h 31 xx′-h 32 yx′-h 33 x′=0

[0091] h 21 x+h 22 y+h 23 -h 31 xx′-h 32 yx′-h 33 x′=0

[0092] That is, two constraint equations can be obtained from the coordinates of a pair of points. The homography matrix is ​​usually normalized so that h 33 =1, so the matrix H 3×3 There are 8 parameters to be solved. The homography matrix can be solved using 4 point pairs (where any three points in any group are not collinear), and the equation is as follows:

[0093]

[0094] h=[h 11 h 12 h 13 h 21 h 22 h 23 h 31 h 32 ] Τ

[0095] Y=[x1′y1′x′2y′2x3′y3′x′4y′4] Τ

[0096] Ah=Y

[0097] When any three points of the two sets of point pairs are not collinear, the matrix A is a full-rank reversible matrix. Using the knowledge of SVD decomposition and LU decomposition in matrix theory, we can solve h and get the transformation matrix H. 3×3 .

[0098] After solving the transformation matrix, we can get the forward mapping relationship x′=Hx. Then, the reverse mapping relationship is x=H -1 x′, we can find the corresponding point in the original image from the transformed image coordinates. In this way, we use back projection to traverse the transformed image coordinates and solve the coordinates in the original image. If it is in the original image coordinate range, we calculate the nearest neighbor or linear interpolation point to represent the transformed pixel value. If it is not in the original image range, we use a fixed value such as 0 or 255 to fill it.

[0099] Therefore, the vertices of the quadrilateral on the target plane are selected as reference points from the target plane of the original image, the positions of the reference points are adjusted, the transformation matrix is ​​calculated, and the perspective transformation of the original image is performed at the same time, and the pixel values ​​of the transformed image are calculated using the reverse mapping. The positions of the reference points are adjusted interactively, and the transformed image conforms to the view under the orthogonal top view of the target plane. At this point, with the 4 reference points on the original image (the first image) and the corresponding 4 transformation points under the top view, the corresponding homography transformation matrix can be obtained.

[0100] After the adjustment is completed, the required prior transformation matrix will be saved locally. In the above example, the vertex coordinates of the quadrilateral become pt1(507,451), pt2(749,490), pt3(749,676), pt4(505,659), and the homography matrix from the original image to the target image is:

[0101]

[0102] For other points on the target plane in the image that are not reference points, the transformation formula is used to calculate the transformation point in the top view coordinates corresponding to the point in the original image.

[0103]

[0104] The fourth step is to calculate the relationship between pixels and actual physical length by selecting an object of known size in the image as a reference. The user finds a reference object of known size in the original image, calculates the length and width of the pixels in the transformed image, and obtains the scaling factors rx and ry of the transformed pixels and the actual physical size (representing the actual distance represented by 1 pixel in the x and y directions on the adjusted image, respectively, in m / pix).

[0105] For example, in the imported image above, by querying the road construction specifications or actual measurements, we can find that the stone brick fence of the tree on the left is 1.2mx1.2m. In the converted image, the vertex coordinates of the fence are A(369,663), B(399,663), and C(399,626). Then AB=30pix, BC=36pix, rx=1.2 / 30=0.04m / pix, ry=1.2 / 36=0.033m / pix. That is, in the converted image coordinate system, the length of each pixel in the horizontal coordinate is about 0.04m, and the length of each pixel in the vertical coordinate is about 0.003m.

[0106] The above content introduces in detail the specific implementation of obtaining the prior parameters, that is, the target transformation matrix, including the introduction of manual interactive calibration. By selecting the vertices of the quadrilateral on the target plane as reference points, dragging the reference points to make the scene conform to the top view perspective, and dynamically adjusting the transformation matrix, the target prior transformation parameters are obtained. The above introduction is convenient for technicians in the field to better understand and implement the technical solution of this application.

[0107] In other implementations, after the target transformation matrix and the scaling factor are determined, the method may further include: determining a distance between the first target object and the second target object in the first acquired image based on the target transformation matrix and the scaling factor.

[0108] Specifically, the first acquired image can be converted into a second acquired image based on the target transformation matrix, and then the pixel distance between the first target object and the second target object in the second acquired image can be determined; finally, the physical distance between the first target object and the second target object can be determined based on the pixel distance and the scaling factor.

[0109] Figure 5 This is a schematic diagram of a ranging implementation process disclosed in an embodiment of the present application. Figure 5 As shown, the ranging process may include:

[0110] 1. Select the target sample point for distance calculation. You can use 2D target detection algorithms such as yolo or fcos, and use the ground center point of the predicted target box as the target sample point; or you can use the monocular 3D target detection algorithms commonly used in the industry such as yolo3d or fcos3d, and use the ground projection point corresponding to the predicted 3D box as the target sample point.

[0111] 2. Use the target transformation matrix to calculate the corresponding points under the top view perspective using the forward mapping relationship x′=Hx for each target sample point. For each pair of points, the actual distance value is calculated according to the Euclidean distance formula.

[0112] The above content introduces the implementation of ranging based on the target transformation matrix and the scaling factor. In the implementation, the target transformation matrix can be imported into the edge analysis engine, the target can be detected by the target detection algorithm, the target distance can be measured, and finally the actual distance can be obtained by mapping it with the target transformation matrix.

[0113] In addition to the implementation of the above-mentioned distance measurement task, the image processing method may also include the implementation of a speed measurement task. That is, after obtaining the target transformation matrix and the scaling factor, the speed of the first target object in the captured video may be determined based on the target transformation matrix and the scaling factor.

[0114] Specifically, the process may include: converting the target object in the first video frame image in the acquired video to a first pixel position based on the target transformation matrix, and converting the target object in the second video frame image in the acquired video to a second pixel position, the first video frame image and the second video frame image including the same target object; determining the pixel distance of the target object based on the first pixel position and the second pixel position; determining the moving distance of the target object based on the pixel distance and the scaling factor; determining the moving speed of the target object based on the acquisition time difference between the first video frame image and the second video frame image and the moving distance and the acquisition time difference.

[0115] In the implementation, Kalman filtering, SORT tracking algorithm, bytetrack tracking algorithm and other methods can be used to track the detected target and obtain the trajectory. The distance between the trajectory points of the same target in different frames and the points in the top view can be calculated and divided by time to obtain the speed.

[0116] The frequency or period of the monocular camera to collect video images is fixed, so the acquisition time difference between different frames in the video can be obtained, that is, the product of the image acquisition period and the difference in the number of image frames. The moving distance of the same object can be calculated based on the pixel distance and the scaling factor, so the actual moving speed of the object is the quotient of the moving distance and the acquisition time difference. Figure 6 This is a schematic diagram of a speed measurement implementation process disclosed in an embodiment of the present application, which can be combined with Figure 6 Understand the speed measurement implementation.

[0117] The scheme described in this application can use the vertices of a quadrilateral on a plane as reference points, dynamically adjust the reference points to calculate the homography matrix and adjust the image at the same time to conform to the "overlooking" perspective orthogonal to the target plane, calculate the homography matrix, and calculate the pixel and physical size ratio under the orthogonal "overlooking" perspective. The homography matrix and size ratio are subsequently used for tasks such as target spacing measurement and target speed measurement. Existing pictures, videos, or camera acquisition data can be processed, the method of use is simple, and the actual calculation complexity is low.

[0118] In addition, the technical solution disclosed in this application only requires a single fixed-focus dedistortion camera, without multiple camera devices, and is suitable for cameras in open scenes such as highways, parks, and squares. No additional hardware investment is required. Existing qualified videos and existing cameras can be directly processed and then used to measure target spacing and speed.

[0119] For the aforementioned method embodiments, for the sake of simplicity, they are all described as a series of action combinations, but those skilled in the art should be aware that the present application is not limited by the order of the actions described, because according to the present application, some steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also be aware that the embodiments described in the specification are all preferred embodiments, and the actions and modules involved are not necessarily required by the present application.

[0120] The method is described in detail in the embodiments disclosed in the above-mentioned application. The method of the application can be implemented by various forms of devices. Therefore, the application also discloses a device, and a specific embodiment is given below for detailed description.

[0121] Figure 7 Schematic diagram of the structure of an image processing device disclosed in an embodiment of the present application. Figure 7 As shown, the image processing device 70 may include:

[0122] The image acquisition module 701 is used to obtain a first image captured by a fixed monocular camera.

[0123] The plane determination module 702 is used to determine the target plane existing in the first image.

[0124] The vertex determination module 703 is used to determine an N-gon on the target plane and record the position data of N vertices of the N-gon, where N is an integer not less than 4.

[0125] The vertex adjustment module 704 is used to adjust the positions of the N vertices and obtain the corresponding transformation matrix based on the position data so that the second image obtained after the first image is transformed based on the transformation matrix meets the set requirements, and the set requirements are that the second image is a top view of the orthogonal perspective that meets the target plane.

[0126] A matrix determination module 705, configured to determine the transformation matrix corresponding to the second image that meets the setting requirements as a target transformation matrix;

[0127] The scaling factor determining module 706 is configured to determine a scaling factor based on a physical size of the first object in the first image and a pixel size of the first object in the second image.

[0128] The image processing device described in the embodiment of the present application can obtain the target transformation matrix by analyzing and processing the image captured by the monocular camera, that is, converting the original image captured by the monocular camera into the transformation parameters of the second image under the "top view" perspective orthogonal to the target plane, and can also obtain the scaling factor between the pixel size and the actual physical size in the second image, thereby providing a basis for subsequent measurement tasks. The above-mentioned method of obtaining the target transformation parameters and scaling factors is simple and convenient, and is suitable for various scenarios that require obtaining the target transformation matrix corresponding to the monocular camera.

[0129] The specific implementation of the above-mentioned image processing device and each module included therein can refer to the content introduction of the corresponding part of the method embodiment, which will not be repeated here.

[0130] Any one of the image processing devices described in the above embodiments includes a processor and a memory. The image acquisition module, plane determination module, vertex determination module, vertex adjustment module, matrix determination module, scaling factor determination module, etc. in the above embodiments are all stored in the memory as program modules, and the processor executes the above program modules stored in the memory to implement corresponding functions.

[0131] The processor includes a kernel, which retrieves the corresponding program module from the memory. One or more kernels can be set, and the processing of the access data can be realized by adjusting the kernel parameters.

[0132] The memory may include non-permanent memory in a computer-readable medium, random access memory (RAM) and / or non-volatile memory, such as read-only memory (ROM) or flash RAM, and the memory includes at least one memory chip.

[0133] In an exemplary embodiment, a computer-readable storage medium is also provided, which can be directly loaded into the internal memory of a computer and contains software code. After being loaded and executed by a computer, the computer program can implement the steps shown in any embodiment of the above-mentioned image processing method.

[0134] In an exemplary embodiment, a computer program product is also provided, which can be directly loaded into the internal memory of a computer and contains software codes. After being loaded and executed by a computer, the computer program can implement the steps shown in any embodiment of the above-mentioned image processing method.

[0135] In this specification, each embodiment is described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the embodiments can be referred to each other. For the device disclosed in the embodiment, since it corresponds to the method disclosed in the embodiment, the description is relatively simple, and the relevant parts can be referred to the method part.

[0136] It should also be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the presence of other identical elements in the process, method, article or device including the elements.

[0137] The steps of the method or algorithm described in conjunction with the embodiments disclosed herein may be implemented directly using hardware, a software module executed by a processor, or a combination of the two. The software module may be placed in a random access memory (RAM), a memory, a read-only memory (ROM), an electrically programmable ROM, an electrically erasable programmable ROM, a register, a hard disk, a removable disk, a CD-ROM, or any other form of storage medium known in the art.

[0138] The above description of the disclosed embodiments enables those skilled in the art to implement or use the present application. Various modifications to these embodiments will be apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present application. Therefore, the present application will not be limited to the embodiments shown herein, but will conform to the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. An image processing method, characterized in that: include: Obtain a first image captured by a fixed-setting monocular camera; determining a target plane present in the first image; Determine an N-gon on the target plane, and record position data of N vertices of the N-gon, where N is an integer not less than 4; Adjusting the positions of the N vertices, and obtaining a corresponding transformation matrix based on the position data, so that a second image obtained after the first image is transformed based on the transformation matrix meets a set requirement, wherein the set requirement is that the second image is a top view that meets the orthogonal perspective of the target plane; Determine the transformation matrix corresponding to the second image that meets the setting requirements as the target transformation matrix; A scaling factor is determined based on a physical size of a first object in the first image and a pixel size of the first object in the second image.

2. The image processing method according to claim 1, characterized in that: The adjusting the positions of the N vertices and obtaining a corresponding transformation matrix based on the position data so that a second image obtained after the first image is transformed based on the transformation matrix meets a set requirement includes: Adjusting the positions of the N vertices; Determine a transformation matrix based on the position data of the N vertices before and after the positions are adjusted, and convert the first image into a second image based on the transformation matrix; The above process of adjusting the position and converting the image is repeated until the converted second image meets the set requirements.

3. The image processing method according to claim 2, characterized in that: Where N is 4, the step of determining the transformation matrix based on the position data of the N vertices before and after the position adjustment includes: Construct a non-singular homography transformation matrix based on the position data of the four vertices before and after the position adjustment; Determine two constraint equations of the homography transformation matrix based on the coordinates of the vertex pair corresponding to each vertex, and obtain eight constraint equations in total, wherein the vertex pair includes the vertex coordinates before the vertex position is adjusted and the vertex coordinates after the position is adjusted; Normalizing the homography matrix; The homography transformation matrix is ​​solved based on the eight constraint equations to obtain a transformation matrix.

4. The image processing method according to claim 2, characterized in that: The converting the first image into a second image based on the transformation matrix comprises: Based on the transformation matrix, reverse mapping processing is performed on all pixels in the first image to obtain a second image.

5. The image processing method according to claim 1, characterized in that: Also includes: A distance between a first target object and a second target object in the first acquired image is determined based on the target transformation matrix and the scaling factor.

6. The image processing method according to claim 5, characterized in that: The determining the distance between the first target object and the second target object in the first acquired image based on the target transformation matrix and the scaling factor includes: Converting the first acquired image into a second acquired image based on the target transformation matrix; determining a pixel distance between a first target object and a second target object in the second acquired image; A physical distance between the first target object and the second target object is determined based on the pixel distance and the scaling factor.

7. The image processing method according to claim 1, characterized in that: Also includes: A speed of a first target object in the captured video is determined based on the target transformation matrix and the scaling factor.

8. The image processing method according to claim 7, characterized in that: The determining the speed of the target object in the captured video based on the target transformation matrix and the scaling factor includes: Based on the target transformation matrix, converting a reference point of a target object in a first video frame image in a captured video to a first pixel position, and converting the reference point of the target object in a second video frame image in the captured video to a second pixel position, wherein the first video frame image and the second video frame image include the same target object; determining a pixel distance of the target object based on the first pixel position and the second pixel position; Determine the moving distance of the target object based on the pixel distance and the scaling factor; Based on the acquisition time difference between the first video frame image and the second video frame image; The moving speed of the target object is determined based on the moving distance and the acquisition time difference.

9. The image processing method according to claim 1, characterized in that: The target plane is a plane whose position does not change.

10. An image processing device, characterized in that: include: An image acquisition module, used to acquire a first image captured by a fixedly set monocular camera; A plane determination module, used to determine a target plane existing in the first image; A vertex determination module, used to determine an N-gon on the target plane and record position data of N vertices of the N-gon, where N is an integer not less than 4; a vertex adjustment module, used for adjusting the positions of the N vertices, and obtaining a corresponding transformation matrix based on the position data, so that a second image obtained after the first image is transformed based on the transformation matrix meets a set requirement, wherein the set requirement is that the second image is a top view that meets the orthogonal perspective of the target plane; A matrix determination module, used to determine the transformation matrix corresponding to the second image that meets the set requirements as a target transformation matrix; A scaling factor determination module is configured to determine a scaling factor based on a physical size of a first object in the first image and a pixel size of the first object in the second image.

Images