Image processing method, apparatus and device

Abstract

The application provides an image processing method, device and equipment, and relates to the field of computers. In the method, a second set of coordinate points in a first image is adjusted according to a first set of coordinate points in the first image and a preset first wheelbase and first wheel track; a first inverse perspective transformation matrix is obtained according to the adjusted second set of coordinate points and a third set of coordinate points in a second image; each coordinate point in the second image is converted into a coordinate point in a first coordinate system corresponding to the first image according to the first inverse perspective transformation matrix, and a third image is obtained; a fifth set of coordinate points in the third image is adjusted according to an angle relationship in a fourth set of coordinate points of the third image, a second inverse perspective transformation matrix is obtained according to the adjusted fifth set of coordinate points and the third set of coordinate points, and the accuracy of the inverse perspective transformation matrix is improved.

Description

Technical Field

[0001] This application relates to the field of computer technology, and provides an image processing method, apparatus, and device. Background Technology

[0002] In intelligent transportation scenarios, cameras play a crucial role in road feature detection. However, cameras typically acquire perspective images, which can be understood as images viewed in the camera's coordinate system. This is not conducive to extracting road features. Therefore, it is necessary to convert perspective images to bird's-eye view images, which are digital images that simulate a bird's-eye view. Generally, this is done using an inverse perspective transformation matrix.

[0003] One method for determining the inverse perspective transformation matrix is ​​as follows: determine multiple first coordinate points in the perspective image, and multiple second coordinate points in the bird's-eye view image corresponding to these multiple first coordinate points, and determine the transformation relationship between at least a multiple number of first coordinate points and multiple coordinate points, thus obtaining the inverse perspective transformation matrix. However, due to potential camera shake during image capture, the determined multiple first coordinate points may be inaccurate, leading to inaccurate solutions for the inverse perspective transformation matrix. Summary of the Invention

[0004] This application provides an image processing method, apparatus, and device for improving the accuracy of the inverse perspective transformation matrix.

[0005] In a first aspect, embodiments of this application provide an image processing method, comprising: adjusting a second set of coordinate points in a first image based on a first set of coordinate points in the first image, and a preset first wheelbase and first track width, wherein the first set of coordinate points includes the coordinate points of the wheels of a target vehicle touching the ground in the first image, and the first image is an image of a target scene in a first coordinate system; obtaining a first inverse perspective transformation matrix based on the adjusted second set of coordinate points and a third set of coordinate points, wherein the coordinate points in the third set of coordinate points belong to a second image, and the second image is an image of the target scene in a second coordinate system, the first inverse perspective transformation matrix being used to transform the coordinate points in the second image... The coordinates of the points in the second image are converted to coordinates in the first image. Based on the first inverse perspective transformation matrix, the coordinates of each point in the second image are converted to coordinates in the first coordinate system to obtain the third image. Based on the fourth set of coordinates in the third image, the fifth set of coordinates in the third image is adjusted. The fourth set of coordinates includes the coordinates of the wheels of the target vehicle touching the ground in the third image. The coordinates in the fifth set correspond one-to-one with the coordinates in the second set of coordinates. Based on the adjusted fifth set of coordinates and the third set of coordinates, a second inverse perspective transformation matrix is ​​obtained. This second inverse perspective transformation matrix is ​​used to convert the coordinates of the points in the second image to coordinates in the third image.

[0006] In this embodiment, using the first wheelbase and first track width as references, and the first set of coordinate points in the first image, the second set of coordinate points in the first image is corrected. Based on the corrected second set of coordinate points, a first inverse perspective transformation matrix is ​​obtained. Furthermore, the first image can be corrected according to the first inverse perspective transformation matrix to obtain a third image. Then, based on the relevant features of the wheel contact points in the third image, the fifth set of coordinate points is adjusted to obtain a second inverse perspective transformation matrix. This is equivalent to performing multiple corrections on the image based on fixed reference objects (such as vehicle tires), thus improving the accuracy of the determined inverse perspective transformation matrix.

[0007] Optionally, adjusting the second set of coordinate points in the first image based on the first set of coordinate points in the first image, and the preset first wheelbase and first track width, includes: determining the second wheelbase and second track width of the target vehicle based on the first set of coordinate points; determining a stretching matrix based on the second wheelbase, second track width, first wheelbase, and first track width and first track width, the stretching matrix being used to scale the first image along the first coordinate system; and obtaining the adjusted second set of coordinate points based on the product of the stretching matrix and the second set of coordinate points.

[0008] In the above optional embodiments, a stretching matrix can be determined based on the first set of coordinate points and a preset first wheelbase and first wheelbase. The coordinate points in the first image are then scaled along the first coordinate system based on the stretching matrix. Since the first wheelbase and first wheelbase are relatively fixed, the first image can be adjusted based on the first wheelbase and first wheelbase, thereby making the first inverse perspective transformation matrix more accurate.

[0009] Optionally, based on the fourth set of coordinate points in the third image, the fifth set of coordinate points in the third image is adjusted, including: based on the fourth set of coordinate points, determining the angle between the first direction where the two wheels of the target vehicle are located and the second direction of the third image; based on the angle, adjusting the fifth set of coordinate points in the third image to obtain the adjusted fifth set of coordinate points.

[0010] In the above optional implementation, by adjusting the fifth coordinate point set according to the angle between the first direction where the two wheels of the target vehicle in the third image are located and the second direction of the third image, and the actual angle that the angle may correspond to, the obtained second inverse perspective transformation matrix can be more accurate.

[0011] Optionally, the angle between the third direction where each vehicle in the target vehicle is located and the lane line in the first image is less than a first threshold.

[0012] In the above optional implementation, limiting the angle between the third direction and the lane line to be less than a first threshold can make the target vehicle corresponding to the first set of coordinate points more parallel to the lane line, thereby reducing the error caused by vehicle offset when adjusting the second set of coordinate points. Similarly, it can also make the target vehicle corresponding to the fourth set of coordinate points more parallel to the lane line, thereby reducing the error caused by vehicle offset when adjusting the fifth set of coordinate points.

[0013] Optionally, before adjusting the second set of coordinate points in the first image based on the first set of coordinate points in the first image, and the preset first wheelbase and first wheelbase, the method further includes: determining a third inverse perspective transformation matrix based on the third set of coordinate points and the second set of coordinate points, wherein the coordinate points in the third set of coordinate points correspond one-to-one with the coordinate points in the second set of coordinate points; and determining the first image based on the second image and the third inverse perspective transformation matrix.

[0014] In the above optional implementation, the third inverse perspective transformation matrix is ​​determined by a set of one-to-one corresponding coordinate points, thereby quickly obtaining the first image, making the generation of the first image faster. Determining the first image only requires two sets of corresponding coordinate points, making the determination of the first image simpler.

[0015] Optionally, the method further includes: determining multiple intersection points of two sets of reference lines in the second image, wherein the third set of coordinate points includes the coordinates of the multiple intersection points.

[0016] In the above optional implementation, the third set of coordinate points is determined by the intersection of the reference lines, which makes the determination of the third set of coordinate points more accurate and makes it easier to determine the coordinates by the intersection.

[0017] Secondly, embodiments of this application provide an image processing apparatus, comprising: an adjustment module, configured to adjust a second set of coordinate points in a first image based on a first set of coordinate points in a first image, and a preset first wheelbase and first track width, wherein the first set of coordinate points includes coordinate points of the wheels of a target vehicle touching the ground in the first image, and the first image is an image of a target scene in a first coordinate system; and an acquisition module, configured to obtain a first inverse perspective transformation matrix based on the adjusted second set of coordinate points and a third set of coordinate points, wherein the coordinate points in the third set of coordinate points belong to a second image, and the second image is an image of the target scene in a second coordinate system, and the first inverse perspective transformation matrix is ​​used to convert the coordinate points in the second image into the first image. The acquisition module is further configured to convert each coordinate point in the second image into a coordinate point in a first coordinate system according to the first inverse perspective transformation matrix to obtain the third image; the adjustment module is further configured to adjust the fifth coordinate point set in the third image according to the fourth coordinate point set of the third image, the fourth coordinate point set including the coordinate points of the target vehicle's wheels touching the ground in the third image, and the coordinate points in the fifth coordinate point set correspond one-to-one with the coordinate points in the second coordinate point set; the acquisition module is further configured to obtain the second inverse perspective transformation matrix according to the adjusted fifth coordinate point set and the third coordinate point set, the second inverse perspective transformation matrix being used to convert the coordinate points in the second image into coordinate points in the third image.

[0018] Optionally, the adjustment module is specifically used to determine the second wheelbase and second wheelbase of the target vehicle based on the first set of coordinate points; determine the stretching matrix based on the second wheelbase, the second wheelbase, the first wheelbase, and the first wheelbase, the stretching matrix being used to scale the first image along the first coordinate system; and obtain the adjusted second set of coordinate points based on the product of the stretching matrix and the second set of coordinate points.

[0019] Optionally, the adjustment module is specifically used to determine the angle between the first direction where the two wheels of the target vehicle are located and the second direction of the third image based on the fourth set of coordinate points; and to adjust the fifth set of coordinate points in the third image based on the angle to obtain the adjusted fifth set of coordinate points.

[0020] Optionally, the angle between the third direction where each vehicle in the target vehicle is located and the lane line in the first image is less than a first threshold.

[0021] Optionally, the acquisition module is also used to determine a third inverse perspective transformation matrix based on a third set of coordinate points and a second set of coordinate points, wherein the coordinate points in the third set of coordinate points correspond one-to-one with the coordinate points in the second set of coordinate points; and to determine a first image based on the second image and the third inverse perspective transformation matrix.

[0022] Optionally, the acquisition module is also used to determine multiple intersection points of the two sets of reference lines in the second image, wherein the third set of coordinate points includes the coordinates of the multiple intersection points.

[0023] Thirdly, embodiments of this application provide an image processing apparatus, including: at least one processor and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the at least one processor implements the method as described in any one of the first aspects by executing the instructions stored in the memory.

[0024] Fourthly, embodiments of this application provide a computer program product comprising computer instructions that, when run on a computer, cause the methods described in any one of the first aspects to be implemented.

[0025] Fifthly, embodiments of this application provide a computer-readable storage medium storing computer instructions that, when executed on a computer, cause the computer to perform the method as described in any one of the first aspects.

[0026] The beneficial effects of aspects two through five can be found in the discussion of aspect one, and will not be repeated here. Attached Figure Description

[0027] Figure 1 A schematic diagram of an image processing scenario provided in an embodiment of this application; Figure 2 A flowchart of the image processing method provided in the embodiments of this application; Figure 3 A schematic diagram of the second image and the first image provided for embodiments of this application; Figure 4 A schematic diagram of the second and third images provided in the embodiments of this application; Figure 5 This is a schematic diagram of the structure of the image processing apparatus provided in the embodiments of this application; Figure 6 This is a schematic diagram of the structure of the image processing device provided in the embodiments of this application. Detailed Implementation

[0028] To better understand the technical solutions provided in the embodiments of this application, a detailed description will be given below in conjunction with the accompanying drawings and specific implementation methods.

[0029] This application provides an image processing method applicable to any scenario requiring the calculation of the inverse perspective transformation matrix. The following description, in conjunction with the accompanying drawings, illustrates the applicable scenarios for this application embodiment.

[0030] See Figure 1This diagram illustrates an image processing scenario provided by an embodiment of this application. The scenario includes an image acquisition device, an image processing device, and a terminal device.

[0031] Image processing devices can be devices with processing capabilities, such as servers or terminal equipment. Servers can be virtual or physical servers; virtual servers include cloud servers. Terminal equipment can be in-vehicle terminals, wearable devices, televisions, mobile phones, or computers. Image acquisition devices can be devices with image acquisition functions, such as cameras, webcams, or video cameras, like surveillance cameras installed on roads.

[0032] For example, an image acquisition device acquires an image of the road surface and sends the image (e.g., a perspective image) to an image processing device. The image processing device processes the road surface image and sends the processed image (e.g., a bird's-eye view image) to a terminal device. The terminal device can display the processed image on a screen.

[0033] In one possible implementation, at least two of the image acquisition device, image processing device, and terminal device are the same device, such as a car, or are deployed on the same system, such as an intelligent transportation system.

[0034] The image processing methods involved in the embodiments of this application will be described below with reference to the accompanying drawings.

[0035] Please refer to Figure 2 This is a schematic flowchart of an image processing method provided in an embodiment of this application. The image processing method can be performed by, for example... Figure 1 The image processing device shown is used for execution.

[0036] S201, the image processing device acquires the first image.

[0037] The first image is an image of the target scene in a first coordinate system, the target scene including at least one vehicle. The first coordinate system is the coordinate system of the digitized image of the target scene from a bird's-eye view.

[0038] In one possible implementation, the first image may be an image pre-stored by the image processing device or an image received from another device.

[0039] In another possible implementation, the first image may be obtained by converting the second image using an image processing device. The second image belongs to a second coordinate system. For example, the second image is an image of the target scene captured by an image acquisition device. The second coordinate system may be a coordinate system centered on the image acquisition device, or a coordinate system corresponding to the image captured by the image acquisition device.

[0040] For example, the image processing apparatus may use a third inverse perspective transformation matrix to transform the second image to obtain the first image.

[0041] The method for determining the third inverse perspective transformation matrix will be introduced below.

[0042] For example, the image processing device determines the third inverse perspective transformation matrix based on the third set of coordinate points and the second set of coordinate points in the second image.

[0043] The second set of coordinate points is not a coordinate point in the second image. Optionally, the third set of coordinate points includes multiple intersection points, which can be multiple intersection points of two sets of reference lines in the second image. Optionally, a set of reference lines can be a set of reference parallel lines or a set of horizontal lines, and the multiple intersection points are the intersection points of a set of reference parallel lines and a set of horizontal lines. The coordinate points in the third set of coordinate points correspond one-to-one with the coordinate points in the second set of coordinate points.

[0044] For example, please refer to Figure 3 , Figure 3 This is a schematic diagram of a second image and a first image. Two sets of reference lines are shown... Figure 3 The image shows a set of reference parallel lines and a set of horizontal lines. Figure 3 The image on the left can be considered an example of a second image. Figure 3 The right-hand image in the first image can be considered as an example of the first image. The lines on the left and right sides of the first image can be considered as lines corresponding to the reference parallel lines in the second image. Specifically, line a in the second image corresponds to line a in the first image, and line b in the second image corresponds to line b in the first image. The four hollow vertices of the rectangle in the first image are the second set of coordinate points, and the coordinate points in the second set of coordinate points correspond one-to-one with multiple intersection points.

[0045] In one possible implementation, the two sets of reference lines in the second image are two sets of parallel reference lines. Generally, the two lines included in a lane line can be approximately parallel, the two lines included in a zebra crossing can be approximately parallel, the two lines included in a square manhole cover can be approximately parallel, and the two lines included in a regular curb can be approximately parallel. Therefore, in this embodiment, the image processing device can detect at least one set of parallel reference lines in the second image using an image detection algorithm.

[0046] Optionally, the image detection algorithm can be a lane detection algorithm, such as the LaneNet algorithm. Specifically, the image processing device can input the second image into the binarization segmentation network of the LaneNet algorithm, which outputs lane line pixels from the second image. The lane line pixels are then input into an instance segmentation network to obtain different lane line instances, thereby obtaining at least one set of lane lines, i.e., reference parallel lines.

[0047] If there are two pairs of reference parallel lines in the second image, then multiple intersection points of the two pairs of reference parallel lines can be determined. If there is only one pair of reference parallel lines in the second image, then the image processing device can draw two horizontal lines parallel to the X-axis in the second coordinate system of the second image.

[0048] Optionally, the two horizontal lines can be drawn below the vanishing point, but it is generally recommended to draw them at the bottom of the second image. The vanishing point is the point in the image where parallel lines in space converge towards the distant horizon; it can be obtained using image detection algorithms.

[0049] Optionally, both the second and third sets of coordinate points contain at least four coordinate points to obtain the third inverse perspective transformation matrix.

[0050] A formula for solving the third inverse perspective transformation matrix based on the third set of coordinate points and the second set of coordinate points can be found in the following formula (1): (1) The coordinates of the third set of coordinate points are: The coordinates in the second set of coordinate points are H is the homography matrix, which is used to transform the left side of the third set of coordinate points into the coordinates of the second set of coordinate points.

[0051] From formula (1), we can obtain: (2) From formulas (1) and (2), we can obtain the following formulas (3) and (4).

[0052] (3) (4) and There is no separate numerator or denominator ,but That is, the third inverse perspective transformation matrix, which can be specifically expressed as the content of the following formula (5).

[0053] (5)

[0054] S202, the image processing device adjusts the second set of coordinate points based on the first set of coordinate points in the first image, as well as the preset first wheel track and first wheelbase.

[0055] The first set of coordinate points includes the coordinates of the wheels of the target vehicle touching the ground in the first image. The first track width and the first wheelbase are pre-stored in the image processing device.

[0056] For example, based on the first set of coordinate points, the second track width and second wheelbase corresponding to the target vehicle are determined. Based on the second wheelbase, second track width, first wheelbase, and first track width, the coordinates of the second set of coordinate points are adjusted to obtain the adjusted second set of coordinate points.

[0057] In one possible implementation, S202 can be achieved through S1.1-S1.3, which will be described in detail below.

[0058] S1.1, the image processing device determines the second wheel track and the second axle track of the target vehicle based on the first set of coordinate points.

[0059] In one possible implementation, the image processing device may identify vehicles that meet a first condition from a first image and designate those vehicles as target vehicles. For example, the image processing device may utilize an image recognition algorithm to determine vehicles in the first image that meet the first condition. The target vehicle may include one or more vehicles. The first condition includes a vehicle type that meets a first sub-condition and / or a vehicle location that meets a second sub-condition.

[0060] For example, the first sub-condition could be that the vehicle type is a sedan. The second sub-condition could include, for example, that the angle between the line connecting the two right-side wheel contact points of the vehicle and the lane line in the first image is less than a first threshold. The first threshold can be preset, typically set to 5 degrees.

[0061] Specifically, the vehicle type is determined by image recognition algorithm, and the first set of vehicles with the vehicle type of sedan is selected. The contact point detection model is used to detect the contact points of the two right wheels of each vehicle in the first set. The lane line detection algorithm is used to detect the lane line in the first image. The vehicles that are identified as target vehicles are those whose line connecting the two right wheel contact points of the first vehicle makes an angle of less than a first threshold with the lane line.

[0062] Since there may be more than one target vehicle, the second track width can be the average of the wheelbases of the target vehicles.

[0063] S1.2, the image processing device determines the stretching matrix based on the second wheelbase, the second wheelbase, and the first wheelbase and the first wheelbase.

[0064] The stretching matrix is ​​used to scale the first image along a first coordinate system. The first coordinate system is the coordinate system corresponding to the first image.

[0065] In one possible implementation, the first track width and the first wheelbase can be obtained from a first form pre-stored in the image processing device. The first form stores vehicle types and their corresponding track widths and wheelbases. The vehicle type corresponding to the target vehicle is searched in the first form, and the obtained track width and wheelbase are then used as the first track width and the first wheelbase, respectively.

[0066] One way to determine the stretching matrix is ​​as shown in formulas (6) to (8).

[0067] The specific form of the stretching matrix is ​​as follows (6).

[0068] (6)

[0069] in and The calculation methods are as follows: formula (7) and formula (8).

[0070] (7) (8) in, It's the first wheel track. First wheelbase, It is the second wheel track. It is the second wheelbase.

[0071] S1.3, the image processing device obtains the adjusted second set of coordinate points based on the product of the stretching matrix and the second set of coordinate points.

[0072] One way to calculate the product of the stretching matrix and the second set of coordinate points is as follows (9).

[0073] (9)

[0074] in,( , Let (x, y) be a second coordinate in the set of second coordinate points, and (x, y) be the adjusted second coordinate corresponding to the second coordinate. This is the stretching matrix.

[0075] S203, the image processing device obtains the first inverse perspective transformation matrix based on the adjusted second set of coordinate points and the third set of coordinate points.

[0076] In this system, the coordinate points in the third set of coordinate points belong to the second image. The first inverse perspective transformation matrix is ​​used to convert the coordinate points in the first image into coordinate points in the second image. The structure of the first inverse perspective transformation matrix is ​​the same as that of the third inverse perspective transformation matrix, with a total of 8 unknowns. Since the coordinates in the adjusted second set of coordinate points and the third set of coordinate points are both two-dimensional coordinates, and the coordinates in the adjusted second set of coordinate points correspond one-to-one with the coordinates in the third set of coordinate points, both the adjusted second set of coordinate points and the third set of coordinate points must contain at least four coordinate points to solve for the first inverse perspective transformation matrix.

[0077] The method for solving the first inverse perspective transformation matrix is ​​the same as the method for solving the third inverse perspective transformation matrix described above, and will not be repeated here.

[0078] S204, the image processing device converts each coordinate point in the second image into a coordinate system under the first coordinate system according to the first inverse perspective transformation matrix, and obtains the third image.

[0079] Multiply each coordinate point in the second image by the first inverse perspective transformation matrix to the left, and use the resulting set of coordinate points as the third image. For specific calculation methods, please refer to Formulas 3 and 4 above.

[0080] S205, the image processing device adjusts the fifth set of coordinate points in the third image based on the fourth set of coordinate points in the third image.

[0081] The fourth set of coordinate points includes the coordinate points of the target vehicle's wheels touching the ground in the third image, and the coordinate points in the fifth set of coordinate points correspond one-to-one with the coordinate points in the second set of coordinate points.

[0082] For example, you can refer to Figure 4 This is a schematic diagram of a second and a third image, wherein the set of second coordinate points is... Figure 4 The intersection of the reference lines in the left-hand diagram, the set of the fifth coordinate points is Figure 4 The four apex corners of the dashed rectangle in the right-hand diagram.

[0083] In one possible implementation, the fourth set of coordinate points can be obtained by using the same selection method for the target vehicle in the third image as described in S1.1 above. The fourth set of coordinate points can then be obtained by detecting the contact points of the target vehicle in the third image using an image detection algorithm.

[0084] For specific details, please refer to Figure 4The right side shows an example of a third image. The angle between the third direction where the two right wheels of the target vehicle are located and the lane line in the third image is less than a first threshold. Here, the third direction can be the direction corresponding to the straight line formed by the right front wheel contact point and the right rear wheel contact point, and the lane line in the third image can be considered as... Figure 4 In the diagram, line c, the angle between the third direction and the lane line can be considered as the angle between the third direction and the line parallel to the lane line, which is shown in the figure. horn.

[0085] In one possible implementation, the image processing device may adjust the content of the fifth coordinate point set in the third image according to the fourth coordinate point set in the third image, including the following S2.1 and S2.2.

[0086] S2.1, The image processing device determines the angle between the first direction where the two wheels of the target vehicle are located and the second direction of the third image based on the fourth set of coordinate points.

[0087] The second direction of the third image can be the direction of the lane line in the third image, and the first direction of the two wheels of the target vehicle can be the first direction of the two rear wheels of the target vehicle.

[0088] For example, based on the equations corresponding to the first direction and the second direction, and using the angle formula, calculate the angle between the first direction and the second direction.

[0089] For example, continue to refer to Figure 4 The first direction can be the direction corresponding to the straight line formed by the contact points of the left and right rear wheels of the target vehicle; the second direction can be considered as... Figure 4 The right edge of the rectangle on the right has an angle between the first direction and the second direction. .

[0090] S2.2, the image processing device adjusts the set of fifth coordinate points in the third image according to the included angle to obtain the adjusted set of fifth coordinate points.

[0091] In one possible implementation, a shear matrix is ​​generated by the included angle, and the adjusted set of fifth coordinate points is obtained by multiplying the shear matrix and the set of fifth coordinate points. Here, the shear matrix refers to the transformation of the coordinate system, which is used to non-uniformly stretch the image or coordinate points.

[0092] For example, one shear matrix is ​​as follows:

[0093] in, , It is an included angle The complementary angle.

[0094] S206, the image processing device obtains the second inverse perspective transformation matrix based on the adjusted fifth coordinate point set and the third coordinate point set.

[0095] The second inverse perspective transformation matrix is ​​used to convert the coordinate points in the second image into the coordinate points in the third image. Of course, the second inverse perspective transformation matrix can also be used to convert the coordinate points in the third image into the coordinate points in the second image.

[0096] Specifically, the structure of the second inverse perspective transformation matrix is ​​the same as that of the third inverse perspective transformation matrix, with a total of 8 unknowns. Since the coordinates in the adjusted fifth coordinate point set and the third coordinate point set are both two-dimensional coordinates, and the coordinates in the adjusted fifth coordinate point set correspond one-to-one with the coordinates in the third coordinate point set, the adjusted fifth coordinate point set and the third coordinate point set must each contain at least four coordinate points in order to solve for the first inverse perspective transformation matrix.

[0097] The specific method for solving the second inverse perspective transformation matrix is ​​the same as the method for solving the third inverse perspective transformation matrix mentioned above, and will not be repeated here.

[0098] Based on the same inventive concept, embodiments of this application provide an image processing apparatus. Please refer to... Figure 5 This application provides an image processing apparatus.

[0099] like Figure 5 As shown, the image processing apparatus 500 includes an adjustment module 501 and an acquisition module 502. The image processing apparatus 500 can be applied to applications as described above. Figure 1 In the image processing device shown, or for Figure 1 The image processing device shown may be Figure 1 The software or hardware modules in the image processing device shown.

[0100] For example, the adjustment module 501 is used to adjust the second set of coordinate points in the first image according to the first set of coordinate points in the first image, as well as the preset first track width and first wheelbase. The first set of coordinate points includes the coordinate points of the wheels of the target vehicle touching the ground in the first image. The first image is an image of the target scene in the first coordinate system. The acquisition module 502 is used to obtain a first inverse perspective transformation matrix based on the adjusted second coordinate point set and the third coordinate point set. The coordinate points in the third coordinate point set belong to the second image, which is an image of the target scene in the second coordinate system. The first inverse perspective transformation matrix is ​​used to convert the coordinate points in the second image into coordinate points in the first image. The acquisition module 502 is also used to convert each coordinate point in the second image into coordinate points in the first coordinate system based on the first inverse perspective transformation matrix to obtain the third image. The adjustment module 501 is also used to adjust the fifth coordinate point set in the third image according to the fourth coordinate point set in the third image. The fourth coordinate point set includes the coordinate points of the target vehicle's wheels touching the ground in the third image. The coordinate points in the fifth coordinate point set correspond one-to-one with the coordinate points in the second coordinate point set. The acquisition module 502 is also used to obtain a second inverse perspective transformation matrix based on the adjusted fifth coordinate point set and the third coordinate point set. The second inverse perspective transformation matrix is ​​used to convert the coordinate points in the second image into the coordinate points in the third image.

[0101] Optionally, the adjustment module 501 is specifically used to determine the second wheel track and the second wheelbase of the target vehicle based on the first set of coordinate points; determine the stretching matrix based on the second wheel track, the second wheelbase, the first wheel track, and the first wheelbase, the stretching matrix being used to scale the first image along the first coordinate system; and obtain the adjusted second set of coordinate points based on the product of the stretching matrix and the second set of coordinate points.

[0102] Optionally, the adjustment module 501 is specifically used to determine the angle between the first direction where the two wheels of the target vehicle are located and the second direction of the third image based on the fourth coordinate point set; and to adjust the fifth coordinate point set in the third image based on the angle to obtain the adjusted fifth coordinate point set.

[0103] Optionally, the angle between the third direction where each vehicle in the target vehicle is located and the lane line in the first image is less than a first threshold.

[0104] Optionally, the acquisition module 502 is also used to determine a third inverse perspective transformation matrix based on the third set of coordinate points and the second set of coordinate points, wherein the coordinate points in the third set of coordinate points correspond one-to-one with the coordinate points in the second set of coordinate points; and to determine a first image based on the second image and the third inverse perspective transformation matrix.

[0105] Optionally, the acquisition module 502 is also used to determine multiple intersection points of the two sets of reference lines in the second image, wherein the third set of coordinate points includes the coordinates of the multiple intersection points.

[0106] For the steps regarding the adjustment module 501 and the acquisition module 502, please refer to the previous text. Figure 2 The content of the discussion will not be repeated here.

[0107] Based on the same inventive concept, embodiments of this application provide an image processing device. Please refer to... Figure 6This application provides an image processing apparatus 600, comprising: at least one processor 601, and a memory 602 communicatively connected to the at least one processor 601; wherein the memory 602 stores instructions executable by the at least one processor 601, and the at least one processor 601 implements, for example, […]. Figure 2 The image processing method shown.

[0108] The processor 601 can be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, transistor logic devices, hardware components, or any combination thereof.

[0109] The memory 602 may include volatile memory, such as random access memory (RAM). The memory may also include non-volatile memory, such as read-only memory (ROM), flash memory, hard disk drive (HDD), or solid state drive (SSD).

[0110] Based on the same inventive concept, embodiments of this application provide a computer-readable storage medium storing computer instructions that, when executed on a computer, cause the computer to perform actions such as... Figure 2 The image processing method shown.

[0111] Based on the same inventive concept, embodiments of this application provide a computer program product containing computer instructions that, when run on a computer, cause the computer to perform the aforementioned... Figure 2 The image processing method shown.

[0112] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product embodied on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0113] This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to this application. It should be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0114] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0115] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0116] Obviously, those skilled in the art can make various modifications and variations to this application without departing from the spirit and scope of this application. Therefore, if such modifications and variations fall within the scope of the claims of this application and their equivalents, this application also intends to include such modifications and variations.

Claims

1. An image processing method, characterized in that, include: Based on the first set of coordinate points in the first image, and the preset first track width and first wheelbase, the second set of coordinate points in the first image is adjusted by stretching transformation based on the physical size ratio of the vehicle. The first set of coordinate points includes the coordinate points of the target vehicle's wheels touching the ground in the first image. The first image is an image of the target scene in the first coordinate system. Based on the adjusted second set of coordinate points and the third set of coordinate points, a first inverse perspective transformation matrix is ​​obtained. The coordinate points in the third set of coordinate points belong to the second image, which is an image of the target scene in the second coordinate system. The first inverse perspective transformation matrix is ​​used to convert the coordinate points in the second image into the coordinate points in the first image. Based on the first inverse perspective transformation matrix, each coordinate point in the second image is converted into a coordinate point in the first coordinate system to obtain the third image; Based on the fourth set of coordinate points in the third image, the fifth set of coordinate points in the third image is adjusted by an angle shear transformation based on the consistency of wheel direction. The fourth set of coordinate points includes the coordinate points of the target vehicle's wheels touching the ground in the third image, and the fifth set of coordinate points is the set of reference line intersection points in the third image that correspond one-to-one with the second set of coordinate points before adjustment. Based on the adjusted fifth set of coordinate points and the third set of coordinate points, a second inverse perspective transformation matrix is ​​obtained. The second inverse perspective transformation matrix is ​​used to convert the coordinate points in the second image into the coordinate points in the third image.

2. The method as described in claim 1, characterized in that, Based on the first set of coordinate points in the first image, and the preset first wheelbase and first track width, adjust the second set of coordinate points in the first image, including: Based on the first set of coordinate points, determine the second track width and second wheelbase of the target vehicle; A stretching matrix is ​​determined based on the second wheelbase, the second wheelbase, the first wheelbase, and the first wheelbase. The stretching matrix is ​​used to scale the first image along the first coordinate system. The adjusted second set of coordinate points is obtained by multiplying the stretching matrix with the second set of coordinate points.

3. The method as described in claim 1, characterized in that, Adjusting the fifth set of coordinate points in the third image based on the fourth set of coordinate points in the third image includes: Based on the fourth set of coordinate points, the angle between the first direction where the two wheels of the target vehicle are located and the second direction of the third image is determined; the second direction is the direction of the lane line in the third image. Based on the included angle, the set of fifth coordinate points in the third image is adjusted to obtain the adjusted set of fifth coordinate points.

4. The method as described in claim 1, characterized in that, The angle between the third direction of each of the target vehicles and the lane line in the first image is less than a first threshold.

5. The method according to any one of claims 1-4, characterized in that, Before adjusting the second set of coordinate points in the first image based on the first set of coordinate points in the first image, and the preset first wheelbase and first track width, the method further includes: Based on the third set of coordinate points and the second set of coordinate points, a third inverse perspective transformation matrix is ​​determined. The third set of coordinate points is the set of coordinates of multiple intersection points obtained from two sets of reference lines in the second image, and the second set of coordinate points is the set of coordinates of intersection points in the first image that correspond one-to-one with the third set of coordinate points. The first image is determined based on the second image and the third inverse perspective transformation matrix.

6. The method as described in claim 5, characterized in that, The method further includes: Determine multiple intersection points of the two sets of reference lines in the second image, wherein the third set of coordinate points includes the coordinates of the multiple intersection points.

7. An image processing apparatus, characterized in that, The device includes: The adjustment module is used to adjust the second set of coordinate points in the first image by stretching transformation based on the physical size ratio of the vehicle, according to the first set of coordinate points in the first image and the preset first track width and first wheelbase. The first set of coordinate points includes the coordinate points of the wheels of the target vehicle in the first image that are touching the ground. The first image is an image of the target scene in the first coordinate system. The acquisition module is used to obtain a first inverse perspective transformation matrix based on the adjusted second coordinate point set and the third coordinate point set, wherein the coordinate points in the third coordinate point set belong to the second image, and the second image is an image of the target scene in the second coordinate system. The first inverse perspective transformation matrix is ​​used to convert the coordinate points in the second image into the coordinate points in the first image. The acquisition module is further configured to convert each coordinate point in the second image into a coordinate point in the first coordinate system according to the first inverse perspective transformation matrix, thereby obtaining a third image; The adjustment module is further configured to adjust the fifth coordinate point set in the third image based on the fourth coordinate point set of the third image by means of an angle shear transformation based on the consistency of wheel direction. The fourth coordinate point set includes the coordinate points of the target vehicle's wheels touching the ground in the third image, and the fifth coordinate point set is the set of reference line intersection points in the third image that correspond one-to-one with the second coordinate point set before adjustment. The acquisition module is further configured to obtain a second inverse perspective transformation matrix based on the adjusted fifth coordinate point set and the third coordinate point set, wherein the second inverse perspective transformation matrix is ​​used to convert the coordinate points in the second image into the coordinate points in the third image.

8. An image processing device, characterized in that, include: At least one processor, and A memory that is communicatively connected to the at least one processor; The memory stores instructions that can be executed by the at least one processor, and the at least one processor implements the method as described in any one of claims 1-6 by executing the instructions stored in the memory.

9. A computer program product, characterized in that, It contains computer instructions that, when run on a computer, cause the method described in any one of claims 1-6 to be implemented.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer instructions that, when executed on a computer, cause the computer to perform the method as described in any one of claims 1-6.

Images