Image processing method and device

Abstract

The present disclosure provides an image processing method and device, which relates to the field of artificial intelligence, and in particular to the field of intelligent transportation. The specific implementation scheme is: obtain a top view of the road; identify the position of the lane line from the top view; divide the top view into at least 2 areas, and determine the width of the lane in each area and the average width of the lane in the top view according to the position of the lane line in each area; calculate the first perspective correction matrix by optimizing the first loss function, wherein the first loss function is used to characterize the difference between the width of the lane in each area and the average width of the lane in the top view; perform lateral correction on the top view through the first perspective correction matrix to obtain a first corrected image. This implementation can calibrate the deformation of the top view and greatly improve the accuracy of element positioning in the top view.

Description

Technical Field

[0001] The present disclosure relates to the field of artificial intelligence, in particular to the field of intelligent transportation, and specifically to an image processing method and device. Background Art

[0002] In photography and filmmaking, perspective distortion occurs when an object and its surroundings appear completely different from what they would appear to be through a standard lens, bending or deforming due to changes in the relative proportions of near and far features.

[0003] Map data is a basic and important part of the navigation system, and the bird's-eye view is the key data for map data production. Figure 1 Generally generated by panoramic photo projection, the top view is often greatly deformed by factors such as lens distortion, road slope, equipment movement, etc., that is, perspective deformation. Map elements are estimated by GPS longitude and latitude based on their positions in the top view, so the perspective deformation of the top view seriously affects the positioning accuracy of ground elements. Summary of the invention

[0004] The present disclosure provides an image processing method, apparatus, device, storage medium and computer program product.

[0005] According to a first aspect of the present disclosure, there is provided an image processing method, comprising: acquiring a top view of a road; identifying the position of lane lines from the top view; dividing the top view into at least two areas, and determining the width of the lane in each area and the average width of the lane in the top view according to the position of the lane lines in each area; calculating a first perspective correction matrix by optimizing a first loss function, wherein the first loss function is used to characterize the difference between the width of the lane in each area and the average width of the lane in the top view; and performing a transverse correction on the top view using the first perspective correction matrix to obtain a first corrected image.

[0006] According to a second aspect of the present disclosure, there is provided an image processing device, comprising: an acquisition unit, configured to acquire a top view of a road; an identification unit, configured to identify the position of a lane line from the top view; a determination unit, configured to divide the top view into at least two regions, and determine the width of the lane in each region and the average width of the lane in the top view according to the position of the lane line in each region; a calculation unit, configured to calculate a first perspective correction matrix by optimizing a first loss function, wherein the first loss function is used to characterize the difference between the width of the lane in each region and the average width of the lane in the top view; and a correction unit, configured to perform lateral correction on the top view through the first perspective correction matrix to obtain a first corrected image.

[0007] According to a third aspect of the present disclosure, an electronic device is provided, comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the method described in the first aspect.

[0008] According to a fourth aspect of the present disclosure, a non-transitory computer-readable storage medium storing computer instructions is provided, wherein the computer instructions are used to cause the computer to execute the method described in the first aspect.

[0009] According to a fifth aspect of the present disclosure, a computer program product is provided, comprising a computer program, wherein the computer program implements the method described in the first aspect when executed by a processor.

[0010] The image processing method and device provided by the embodiments of the present disclosure calibrate the deformation of the top view by controlling the width of the same lane to be equal everywhere in the top view based on the lane line recognition result, thereby greatly improving the accuracy of element positioning in the top view.

[0011] It should be understood that the content described in this section is not intended to identify the key or important features of the embodiments of the present disclosure, nor is it intended to limit the scope of the present disclosure. Other features of the present disclosure will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0012] The accompanying drawings are used to better understand the present solution and do not constitute a limitation of the present disclosure.

[0013] Figure 1 is an exemplary system architecture diagram in which an embodiment of the present disclosure may be applied;

[0014] Figure 2 is a flow chart of an embodiment of an image processing method according to the present disclosure;

[0015] Figure 3 is a schematic diagram of lane line extraction and vectorization results according to the image processing method disclosed in the present invention;

[0016] Figure 4 is a schematic diagram of an application scenario of the image processing method according to the present disclosure;

[0017] Figure 5 is a flowchart of another embodiment of the image processing method according to the present disclosure;

[0018] Figure 6 is a schematic structural diagram of an embodiment of an image processing device according to the present disclosure;

[0019] Figure 7 It is a structural diagram of a computer system of an electronic device suitable for implementing an embodiment of the present disclosure. DETAILED DESCRIPTION

[0020] The following is a description of exemplary embodiments of the present disclosure in conjunction with the accompanying drawings, including various details of the embodiments of the present disclosure to facilitate understanding, which should be considered as merely exemplary. Therefore, it should be recognized by those of ordinary skill in the art that various changes and modifications may be made to the embodiments described herein without departing from the scope and spirit of the present disclosure. Similarly, for the sake of clarity and conciseness, descriptions of well-known functions and structures are omitted in the following description.

[0021] Figure 1 An exemplary system architecture 100 is shown to which an embodiment of the image processing method or image processing apparatus of the present application can be applied.

[0022] like Figure 1 As shown, the system architecture 100 may include an unmanned vehicle 101 (referred to as unmanned vehicle).

[0023] The unmanned vehicle 101 is equipped with a driving control device 1011, a network 1012, a sensor 1013, and a server 102. The network 1012 is used to provide a medium for a communication link between the driving control device 1011 and the sensor 1013. The network 1012 may include various connection types, such as wired, wireless communication links, or optical fiber cables.

[0024] The driving control device (also called the vehicle brain) 1011 is responsible for the intelligent control of the unmanned vehicle 101. The driving control device 1011 can be a separate controller, such as a programmable logic controller (PLC), a single-chip microcomputer, an industrial control machine, etc.; it can also be a device composed of other electronic devices with input / output ports and operation control functions; it can also be a computer device installed with a vehicle driving control application.

[0025] It should be noted that in practice, the unmanned vehicle 101 may be equipped with at least one sensor, such as a laser radar, a camera, a gravity sensor, a wheel speed sensor, etc. In some cases, the unmanned vehicle 101 may also be equipped with a GNSS (Global Navigation Satellite S7stem) device and a SINS (Strap-down Inertial Navigation S7stem) and the like.

[0026] The camera of the unmanned vehicle collects road images and then sends them to the server for analysis and processing. The server performs perspective correction on the road recognition results. The imaging ratio of ground objects in the top view is made the same as that in the real world, eliminating the perspective distortion of near objects being larger than far objects. All parallel lines in reality are displayed as parallel in the image. The perspective-corrected image is fed back to the unmanned vehicle. The perspective-corrected image is close to the projection imaging ratio of the real world, and the unmanned vehicle accurately calculates the position of ground features based on the perspective-corrected image.

[0027] Perspective correction, or perspective control, is the process of compositing or editing photos to produce a result that conforms to the public's understanding of perspective distortion. This correction process includes:

[0028] Make all vertical lines in reality appear vertical in the image. These vertical lines include columns, vertical edges of walls, lamp posts, etc. It should be noted that this is the generally accepted understanding of perspective results; the basis of such perspective results is that the farther the object is, the smaller it is in the image. Due to the existence of vertical distance, the top of the building is farther from the ground observer than the base. However, the perspective process is usually established by considering only the horizontal distance, not the vertical distance, that is, the distance between the top of the building and the base is assumed to be the same.

[0029] For example, in a square room, all parallel lines, such as the four horizontal sides, intersect at one point.

[0030] It should be noted that the image processing method provided in the embodiment of the present application is generally executed by the server 102 , and accordingly, the image processing device is generally disposed in the server 102 .

[0031] It should be understood that Figure 1 The number of driving control devices, networks, sensors, and servers in the embodiment is only for illustration. Any number of driving control devices, networks, sensors, and servers may be provided as required.

[0032] Continue to refer Figure 2 , which shows a process 200 of an embodiment of an image processing method according to the present application. The image processing method comprises the following steps:

[0033] Step 201, obtaining a bird's-eye view of the road.

[0034] In this embodiment, the execution subject of the image processing method (for example Figure 1 The server shown in the figure can obtain a bird's-eye view of the road from the camera. Figure 1 It is usually generated by projecting panoramic photos. Affected by factors such as lens distortion, road slope, equipment movement, etc., the overhead view often has a large deformation, that is, perspective deformation.

[0035] Step 202, identifying the position of the lane line from the top view.

[0036] In this embodiment, a deep learning model (including but not limited to fcn, unet, segnet, deeplab series, pspnet, etc.) can be used to perform semantic segmentation on the top view to segment the lane lines. Image processing methods can also be used to segment the lane lines. Not only the position of the lane lines can be identified, but also the type of lane lines can be identified, such as single line, double line, solid line, dashed line, etc. In this article, the double lines can be merged and treated as a single line, and the center line between the double lines can be used as the lane line.

[0037] Step 203: divide the top view into at least two areas, and determine the width of the lane in each area and the average width of the lane in the top view according to the position of the lane line in each area.

[0038] In this embodiment, the top view may be divided into at least two regions by segmenting the lane lines. The segmentation may be average, or the severely deformed regions (e.g., the lane line regions in the distance) may be finely segmented. Optionally, the segmentation may be performed according to the intervals of the lane line dashed lines to ensure that each region has a complete lane line dashed line.

[0039] The lane lines identified in step 202 are searched for their neighbors and paired with each other to obtain a number of matching pair sets P = {<I i ,I j >,<I m ,I j >……<I m ,I n >}, where each matching pair <I i ,I j >For one lane, I i ,I j They are the left and right lane lines of the lane respectively.

[0040] Calculate each lane<I i ,I j The width w of different regions of the image P ={w1,w2,…,w k}.

[0041] For example, the three lane lines I1, I2, and I3 identified from top to bottom in the top view can be combined into three lanes {<I1, I2>, <I1, I3>, <I2, I3>}. The theoretical values ​​of the widths of lanes <I1, I2> and <I2, I3> should be the same and half of the theoretical value of the width of lane <I1, I3>.

[0042] Due to perspective distortion, the lane width is not a constant value in the image. The lane width corresponding to the middle position of each area can be used as the lane width of the area. The average lane width in the area can also be used as the lane width of the area.

[0043] The average width of the lanes in each area is taken as the average width of the lanes in the top view.

[0044] Step 204: Calculate a first perspective correction matrix by optimizing a first loss function.

[0045] In this embodiment, the first loss function is used to characterize the difference between the width of the lane in each area and the average width of the lane in the top view.

[0046] Affected by perspective distortion, the width of the same lane in different areas of the image is different. In order to eliminate perspective distortion, the width of the same lane in different areas of the image needs to be as equal as possible, so the loss function is:

[0047]

[0048] Where M1 is a 3*3 perspective correction matrix, w i is the width of each lane in k different regions of the image, is the average width of the lane in different areas of the image.

[0049] The optimal perspective correction matrix M1 is solved by optimizing the loss function. The objective function is to minimize L(M1).

[0050] If there are multiple lanes, there are multiple loss functions, for example, L1(M1), L2(M1), ... L n (M1). The sum of multiple loss functions is taken as the total loss function. The optimization goal is to minimize the total loss function.

[0051] The first perspective correction matrix may be solved by a mathematical method or by a gradient descent method.

[0052] Step 205 , performing horizontal correction on the top view through a first perspective correction matrix to obtain a first corrected image.

[0053] In this embodiment, the perspective correction result of image t is

[0054] t′=M1*t

[0055] Wherein, M1 is the first perspective correction matrix, t is the top view, and t′ is the first corrected image.

[0056] The method provided by the above-mentioned embodiment of the present disclosure only relies on the perspective correction of the lane lines in the image, does not rely on inertial sensors, and does not rely on camera calibration. If the on-board camera is displaced or rotated due to bumps in the road, it will not affect the effect. It solves the accuracy problem of low-cost cameras and the high calibration cost problem. Compared with the existing solutions, it has higher accuracy. The operator refers to the top view after perspective correction to perform map data production operations, which is more intuitive and accurate. Improve work efficiency and data quality.

[0057] In some optional implementations of this embodiment, the position of the lane line is identified from the top view, including: identifying the lane line pixels and types from the top view through a semantic segmentation model to generate a semantic segmentation map; extracting the position of the lane line from the semantic segmentation map. The semantic segmentation model outputs a mask map of the image segmentation result, and the value of each pixel on the mask map is a category number, 0 represents the background, 1 represents the white solid line, 2 represents the white dashed line, 3 represents the yellow solid line, and so on. Since the segmentation result often produces some noise points, the image expansion and corrosion processing can be used to remove the influence of the noise points, and then the contour tracking algorithm is used to obtain the outer contour of each lane line. The position of the lane line is determined based on the center line of the outer contour. This method can quickly and accurately extract the position of the lane line, reduce the image processing time, and thus ensure the safety of the unmanned vehicle.

[0058] In some optional implementations of the present embodiment, the position of the lane line is extracted from the semantic segmentation map, including: converting the semantic segmentation map into a binary map; performing contour detection on the binary map to obtain a rectangular contour; splitting the rectangular contour into multiple segments along the long side direction of the rectangular contour, and performing contour detection on each segment again to generate multiple sub-contours; extracting the center line of each sub-contour rectangle as the lane line vector line shape; fitting the lane line vector line shape through a quadratic curve to predict and supplement the missing part of the lane line.

[0059] Extracting and vectorizing lane lines from a semantic segmentation map mainly involves four steps:

[0060] 1. Binarization: Extract lane line type pixels from the semantic segmentation map and convert them into a binary map.

[0061] 2. Contour detection: Perform contour detection on the lane line binary image, such as the findContours method of the cv2 library.

[0062] 3. Contour splitting: Figure 3 As shown in the figure, the rectangular contour is broken into m small segments along the long side direction. Contour detection is performed again to generate m sub-contours. The center line of the sub-contour rectangle is extracted as the lane line vector line shape.

[0063] 4. Use quadratic curve fitting to fit the lane line vector shape generated in the previous step to predict and supplement the missing part of the lane line.

[0064] f(y)=Ay 2 +By+C

[0065] Through the above steps, even if the image clarity is not enough and the lane line is incomplete, the complete lane line can still be fitted, and the lane width and the length of the lane line dashed line can be accurately calculated, thereby improving the accuracy of the map and ensuring driving safety.

[0066] In some optional implementations of this embodiment, the width of the lane in each area and the average width of the lane in the overhead view are determined based on the position of the lane lines in each area, including: performing a neighbor search on the identified lane lines, pairing them two by two to obtain a set of matching pairs, wherein each matching pair corresponds to a lane; for each lane, calculating the lane width at the middle position of each area as the width of each area of ​​the lane; and calculating the average width of the lane in the overhead view based on the width of the lane in each area.

[0067] The lane lines identified in step 202 are searched for their neighbors and paired with each other to obtain a number of matching pair sets P = {<I i ,I j >,<I m ,I j >……<I m ,I n >}, where each matching pair <I i ,I j >For one lane, I i ,I j They are the left and right lane lines of the lane respectively.

[0068] Calculate each lane<I i ,I j The width w of different regions of the image P ={w1,w2,…,w k}.

[0069] For example, the three lane lines I1, I2, and I3 identified from top to bottom in the top view can be combined into three lanes {<I1, I2>, <I1, I3>, <I2, I3>}. The theoretical values ​​of the widths of lanes <I1, I2> and <I2, I3> should be the same and half of the theoretical value of the width of lane <I1, I3>.

[0070] Due to perspective distortion, the lane width is not a constant value in the image. The lane width corresponding to the middle position of each area can be used as the lane width of the area. The average lane width in the area can also be used as the lane width of the area.

[0071] Using the lane width in the middle of each area as the width of each area of ​​the lane can reduce the amount of calculation and speed up image processing, thereby reducing latency and ensuring driving safety.

[0072] In some optional implementations of this embodiment, obtaining a top view of the road includes: obtaining a panoramic view of the road; converting the panoramic view into a top view by a perspective projection method. The image collected by the unmanned vehicle is usually a panoramic view, and the panoramic view can be converted into a top view by a perspective projection method. The conversion can be performed using existing conventional software, such as OpenCV. Thus, by correcting the deformation of the top view and approximating the projection imaging ratio of the real world, the position of the ground elements can be accurately calculated based on the top view, and the positioning of the target object in the panorama can be achieved.

[0073] Continue to see Figure 4 , Figure 4 FIG. 1 is a schematic diagram of an application scenario of the image processing method according to this embodiment. Figure 4 In the application scenario, the unmanned vehicle collects panoramic images while driving. The panoramic image is converted into a panoramic bird's-eye view through the image perspective conversion algorithm. The lane lines are identified from the panoramic bird's-eye view through the semantic segmentation model, and the width of the lane is determined. According to the way that the lane width is equal everywhere in the bird's-eye view, the deformation in the bird's-eye view is calibrated to obtain a corrected panoramic bird's-eye view. Then, the pixel points in the panoramic bird's-eye view can be accurately located by GPS.

[0074] Further references Figure 5 , which shows a process 500 of another embodiment of an image processing method. The process 500 of the image processing method includes the following steps:

[0075] Step 501, obtaining a bird's-eye view of the road.

[0076] Step 502, identifying the position of the lane line from the top view.

[0077] Step 503: divide the top view into at least two areas, and determine the width of the lane in each area and the average width of the lane in the top view according to the position of the lane line in each area.

[0078] Step 504: Calculate a first perspective correction matrix by optimizing a first loss function.

[0079] Step 505 , the top view is subjected to horizontal correction through a first perspective correction matrix to obtain a first corrected image.

[0080] Steps 501-505 are substantially the same as steps 201-205, and thus will not be described in detail.

[0081] Step 506, identifying the dashed lane line from the top view.

[0082] In this embodiment, the dashed lane line can be identified in the top view through the semantic segmentation model, that is, there is a type of pixel in the segmentation result, and the dashed line can be identified.

[0083] Optionally, if the dashed lane line is not recognized in the current top view, multiple panoramic images collected by the unmanned vehicle can be continuously acquired and converted into multiple top views, from which the dashed lane line is recognized. Panoramic images collected by the same camera and at the same slope have the same perspective deformation, so the perspective correction matrix calculated from one panoramic image can be shared by these panoramic images.

[0084] Step 507, determining the length of the dashed lane lines in each area and the average length of the dashed lane lines in the top view.

[0085] In this embodiment, if there is no perspective distortion, the length of the dashed lane line is theoretically consistent. The segmented area includes at least one dashed lane line. The length of a dashed lane line other than the head and tail can be selected in each area as the length of the dashed lane line in the area. The average length of the dashed lane line in all areas is taken as the average length of the dashed lane line in the top view. The dashed lane line length can be expressed as l P ={l1,l2,…,l k}.

[0086] Optionally, when dividing the area, it is divided according to the length of the dashed lane line, so that each area has only one dashed lane line.

[0087] Step 508: Calculate a second perspective correction matrix by optimizing a second loss function.

[0088] In this embodiment, the second loss function is used to characterize the difference between the length of the dashed lane line in each area and the average length of the dashed lane line in the top view.

[0089] Similarly, affected by perspective deformation, the length of the same dashed lane line in different areas of the image is different. In order to eliminate perspective deformation, the length of the same lane in different areas of the image needs to be as equal as possible, so there is a loss function:

[0090]

[0091] Where M2 is a 3*3 perspective correction matrix, l i is the length of each lane in k different regions in the top view, and λ is the average length of the lane in different regions in the top view.

[0092] The second perspective correction matrix can be solved by mathematical methods or by gradient descent.

[0093] Step 509: Perform longitudinal correction on the first corrected image using a second perspective correction matrix to obtain a second corrected image.

[0094] In this embodiment, the best second perspective correction matrix M2 is solved by optimizing the loss function, and the final second corrected image t″ of image t is:

[0095] t″=M2*t′

[0096] from Figure 5 It can be seen that Figure 2 Compared with the corresponding embodiment, the process 500 of the image processing method in this embodiment embodies the step of performing longitudinal correction on the top view. Therefore, the solution described in this embodiment can further correct the image and achieve comprehensive perspective correction.

[0097] In some optional implementations of this embodiment, the length of the dashed lane line in each area and the average length of the dashed lane line in the overhead view are determined, including: for each area, the length of the complete dashed lane line in the middle of the area is used as the length of the dashed lane line in the area; based on the length of the dashed lane line in each area, the average length of the dashed lane line is calculated. This method can speed up the calculation speed, reduce the delay, and thus ensure driving safety.

[0098] Further references Figure 6 As an implementation of the methods shown in the above figures, the present disclosure provides an embodiment of an image processing device, and the device embodiment is Figure 2 Corresponding to the method embodiment shown, the device can be specifically applied to various electronic devices.

[0099] like Figure 6 As shown, the image processing device 600 of this embodiment includes: an acquisition unit 601, an identification unit 602, a determination unit 603, a calculation unit 604 and a correction unit 605. The acquisition unit 601 is configured to acquire a top view of the road; the identification unit 602 is configured to identify the position of the lane line from the top view; the determination unit 603 is configured to divide the top view into at least two areas, and determine the width of the lane in each area and the average width of the lane in the top view according to the position of the lane line in each area; the calculation unit 604 is configured to calculate a first perspective correction matrix by optimizing a first loss function, wherein the first loss function is used to characterize the difference between the width of the lane in each area and the average width of the lane in the top view; the correction unit 605 is configured to perform a lateral correction on the top view through the first perspective correction matrix to obtain a first corrected image.

[0100] In this embodiment, the specific processing of the acquisition unit 601, the identification unit 602, the determination unit 603, the calculation unit 604 and the correction unit 605 of the image processing device 600 can refer to Figure 2 Corresponding to step 201, step 202, step 203, step 204 and step 205 in the embodiment.

[0101] In some optional implementations of this embodiment, the recognition unit 602 is further configured to: identify the dashed lane line from the top view; the determination unit 603 is further configured to: determine the length of the dashed lane line in each area and the average length of the dashed lane line in the top view; the calculation unit 604 is further configured to: calculate the second perspective correction matrix by optimizing the second loss function, wherein the second loss function is used to characterize the difference between the length of the dashed lane line in each area and the average length of the dashed lane line in the top view; the correction unit 605 is further configured to: perform longitudinal correction on the first corrected image through the second perspective correction matrix to obtain a second corrected image.

[0102] In some optional implementations of this embodiment, the recognition unit 602 is further configured to: recognize lane line pixels and types from the top view through a semantic segmentation model to generate a semantic segmentation map; and extract the position of the lane line from the semantic segmentation map.

[0103] In some optional implementations of the present embodiment, the recognition unit 602 is further configured to: convert the semantic segmentation map into a binary map; perform contour detection on the binary map to obtain a rectangular contour; split the rectangular contour into multiple segments along the long side direction of the rectangular contour, and perform contour detection on each segment again to generate multiple sub-contours; extract the center line of each sub-contour rectangle as the lane line vector line shape; fit the lane line vector line shape through a quadratic curve to predict and supplement the missing part of the lane line.

[0104] In some optional implementations of this embodiment, the computing unit 604 is further configured to: perform a neighbor search on the identified lane lines, pair them up in pairs to obtain a set of matching pairs, wherein each matching pair corresponds to a lane; for each lane, calculate the lane width at the middle position of each area as the width of each area of ​​the lane; and calculate the average width of the lane in the overhead view based on the width of the lane in each area.

[0105] In some optional implementations of this embodiment, the determination unit 603 is further configured to: for each area, take the length of the complete lane line dashed line in the middle of the area as the length of the lane line dashed line in the area; and calculate the average length of the lane line dashed line based on the length of the lane line dashed line in each area.

[0106] In some optional implementations of this embodiment, the acquisition unit 601 is further configured to: acquire a panoramic image of the road; and convert the panoramic image into a bird's-eye view through a perspective projection device.

[0107] In the technical solution of the present disclosure, the collection, storage, use, processing, transmission, provision and disclosure of user personal information involved are in compliance with the provisions of relevant laws and regulations and do not violate public order and good morals.

[0108] According to an embodiment of the present disclosure, the present disclosure also provides an electronic device, a readable storage medium and a computer program product.

[0109] An electronic device comprises: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the method described in process 200 or 500.

[0110] A non-transitory computer-readable storage medium storing computer instructions, wherein the computer instructions are used to cause the computer to execute the method described in process 200 or 500.

[0111] A computer program product includes a computer program, wherein the computer program implements the method described in flow 200 or 500 when executed by a processor.

[0112] Figure 7 A schematic block diagram of an example electronic device 700 that can be used to implement an embodiment of the present disclosure is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present disclosure described and / or required herein.

[0113] like Figure 7As shown, the device 700 includes a computing unit 701, which can perform various appropriate actions and processes according to a computer program stored in a read-only memory (ROM) 702 or a computer program loaded from a storage unit 708 into a random access memory (RAM) 703. In the RAM 703, various programs and data required for the operation of the device 700 can also be stored. The computing unit 701, the ROM 702, and the RAM 703 are connected to each other via a bus 704. An input / output (I / O) interface 705 is also connected to the bus 704.

[0114] A number of components in the device 700 are connected to the I / O interface 705, including: an input unit 706, such as a keyboard, a mouse, etc.; an output unit 707, such as various types of displays, speakers, etc.; a storage unit 708, such as a disk, an optical disk, etc.; and a communication unit 709, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 709 allows the device 700 to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks.

[0115] The computing unit 701 may be a variety of general and / or special processing components with processing and computing capabilities. Some examples of the computing unit 701 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various dedicated artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, digital signal processors (DSPs), and any appropriate processors, controllers, microcontrollers, etc. The computing unit 701 performs the various methods and processes described above, such as image processing methods. For example, in some embodiments, the image processing method may be implemented as a computer software program, which is tangibly contained in a machine-readable medium, such as a storage unit 708. In some embodiments, part or all of the computer program may be loaded and / or installed on the device 700 via ROM 702 and / or communication unit 709. When the computer program is loaded into RAM 703 and executed by the computing unit 701, one or more steps of the image processing method described above may be performed. Alternatively, in other embodiments, the computing unit 701 may be configured to perform the image processing method in any other appropriate manner (e.g., by means of firmware).

[0116] Various implementations of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), load programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various implementations can include: being implemented in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0117] The program code for implementing the method of the present disclosure may be written in any combination of one or more programming languages. These program codes may be provided to a processor or controller of a general-purpose computer, a special-purpose computer, or other programmable data processing device, so that the program code, when executed by the processor or controller, enables the functions / operations specified in the flow chart and / or block diagram to be implemented. The program code may be executed entirely on the machine, partially on the machine, partially on the machine and partially on a remote machine as a stand-alone software package, or entirely on a remote machine or server.

[0118] In the context of the present disclosure, a machine-readable medium may be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, device, or equipment. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or equipment, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0119] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user can provide input to the computer. Other types of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including acoustic input, voice input, or tactile input).

[0120] The systems and techniques described herein may be implemented in a computing system that includes back-end components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes front-end components (e.g., a user computer with a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such back-end components, middleware components, or front-end components. The components of the system may be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: a local area network (LAN), a wide area network (WAN), and the Internet.

[0121] A computer system may include a client and a server. The client and the server are generally remote from each other and usually interact through a communication network. The relationship of client and server is generated by computer programs running on respective computers and having a client-server relationship with each other. The server may be a cloud server, a server of a distributed system, or a server combined with a blockchain.

[0122] It should be understood that the various forms of processes shown above can be used to reorder, add or delete steps. For example, the steps recorded in this disclosure can be executed in parallel, sequentially or in different orders, as long as the desired results of the technical solutions disclosed in this disclosure can be achieved, and this document does not limit this.

[0123] The above specific implementations do not constitute a limitation on the protection scope of the present disclosure. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions can be made according to design requirements and other factors. Any modification, equivalent substitution and improvement made within the spirit and principle of the present disclosure shall be included in the protection scope of the present disclosure.

Claims

1. An image processing method, comprising: Get a bird's-eye view of the road; Identifying the position of the lane line from the top view; Divide the top view into at least two areas, and determine the width of the lane in each area and the average width of the lane in the top view according to the position of the lane line in each area; Calculating a first perspective correction matrix by optimizing a first loss function, wherein the first loss function is used to characterize the difference between the width of the lane in each area and the average width of the lane in the top view, wherein the first loss function includes a root mean square error loss function; The top view is laterally corrected using the first perspective correction matrix to obtain a first corrected image.

2. The method according to claim 1, wherein: The method further comprises: Identifying a dashed lane line from the top view; Determine the length of the dashed lane line in each area and the average length of the dashed lane line in the top view; Calculating a second perspective correction matrix by optimizing a second loss function, wherein the second loss function is used to characterize a difference between a length of a dashed lane line in each region and an average length of a dashed lane line in the top view, wherein the second loss function includes a root mean square error loss function; The first corrected image is longitudinally corrected using the second perspective correction matrix to obtain a second corrected image.

3. The method according to claim 1, wherein: The identifying the position of the lane line from the top view includes: Identifying lane line pixels and types from the top view using a semantic segmentation model to generate a semantic segmentation map; The position of the lane line is extracted from the semantic segmentation map.

4. The method according to claim 3, wherein: The extracting the position of the lane line from the semantic segmentation map includes: Converting the semantic segmentation map into a binary map; Performing contour detection on the binary image to obtain a rectangular contour; Splitting the rectangular contour into multiple segments along the long side direction of the rectangular contour, performing contour detection on each segment again to generate multiple sub-contours; Extract the center line of each sub-contour rectangle as the lane line vector line shape; The lane line vector line shape is fitted by a quadratic curve to predict and supplement the missing part of the lane line.

5. The method according to claim 1, wherein: Determining the width of the lane in each area and the average width of the lane in the top view according to the position of the lane line in each area includes: Perform a neighbor search on the identified lane lines and pair them up to get a set of matching pairs, where each matching pair corresponds to a lane. For each lane, the lane width at the middle position of each area is calculated as the width of each area of ​​the lane; The average width of the lanes in the top view is calculated based on the width of the lanes in each area.

6. The method according to claim 2, wherein: Determining the length of the dashed lane line in each area and the average length of the dashed lane line in the top view includes: For each area, the length of the complete lane dashed line in the middle of the area is taken as the length of the lane dashed line in the area; The average length of the dashed lane lines is calculated based on the length of the dashed lane lines in each area.

7. The method according to any one of claims 1 to 6, wherein: The obtaining of the overhead view of the road comprises: Get a panoramic view of the road; The panoramic image is converted into a bird's-eye view by a perspective projection method.

8. An image processing device, comprising: an acquisition unit configured to acquire a top view of the road; A recognition unit configured to recognize the position of the lane line from the top view; a determining unit configured to divide the top view into at least two regions, and determine the width of the lane in each region and the average width of the lane in the top view according to the position of the lane line in each region; a computing unit configured to compute a first perspective correction matrix by optimizing a first loss function, wherein the first loss function is used to characterize a difference between a width of a lane in each region and an average width of a lane in the top view, wherein the first loss function comprises a root mean square error loss function; The correction unit is configured to perform a lateral correction on the top view through the first perspective correction matrix to obtain a first corrected image.

9. The device according to claim 8, wherein: The recognition unit is further configured to: recognize a lane dashed line from the top view; The determining unit is further configured to: determine the length of the dashed lane line in each area and the average length of the dashed lane line in the top view; The computing unit is further configured to: calculate a second perspective correction matrix by optimizing a second loss function, wherein the second loss function is used to characterize the difference between the length of the dashed lane line in each region and the average length of the dashed lane line in the top view, wherein the second loss function includes a root mean square error loss function; The correction unit is further configured to: perform longitudinal correction on the first corrected image through the second perspective correction matrix to obtain a second corrected image.

10. The device according to claim 8, wherein: The identification unit is further configured to: Identifying lane line pixels and types from the top view using a semantic segmentation model to generate a semantic segmentation map; The position of the lane line is extracted from the semantic segmentation map.

11. The device according to claim 10, wherein: The identification unit is further configured to: Converting the semantic segmentation map into a binary map; Performing contour detection on the binary image to obtain a rectangular contour; Splitting the rectangular contour into multiple segments along the long side direction of the rectangular contour, performing contour detection on each segment again to generate multiple sub-contours; Extract the center line of each sub-contour rectangle as the lane line vector line shape; The lane line vector line shape is fitted by a quadratic curve to predict and supplement the missing part of the lane line.

12. The device according to claim 8, wherein: The computing unit is further configured to: Perform a neighbor search on the identified lane lines and pair them up to get a set of matching pairs, where each matching pair corresponds to a lane. For each lane, the lane width at the middle position of each area is calculated as the width of each area of ​​the lane; The average width of the lanes in the top view is calculated based on the width of the lanes in each area.

13. The device according to claim 9, wherein: The determining unit is further configured to: For each area, the length of the complete lane dashed line in the middle of the area is taken as the length of the lane dashed line in the area; The average length of the dashed lane lines is calculated based on the length of the dashed lane lines in each area.

14. The device according to any one of claims 8 to 13, wherein: The acquisition unit is further configured to: Get a panoramic view of the road; The panoramic image is converted into a bird's-eye view by a perspective projection device.

15. An electronic device comprising: at least one processor; as well as a memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the method according to any one of claims 1 to 7.

16. A non-transitory computer-readable storage medium storing computer instructions, wherein: The computer instructions are used to cause the computer to execute the method according to any one of claims 1-7.

17. A computer program product, comprising a computer program, which, when executed by a processor, implements the method according to any one of claims 1 to 7.

Images