A representative region extraction method and vehicle driving system
By identifying feature areas and filtering polygon representative boxes in the vehicle camera image, calculating the interchange ratio and area, the problem of low pavement image extraction efficiency in the prior art is solved, and the rapid and accurate identification of pavement state is achieved, and driving is improved Security.
Patent Information
- Application Number
- CN202410715201.3
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2024-06-04
- Publication Date
- 2024-12-10
- Estimated Expiration
- 2044-06-04
AI Technical Summary
The prior art is difficult to quickly extract image blocks from road surfaces in images captured by on-board cameras, and are easily disturbed by other target objects, resulting in the problems of random selection of starting sub-blocks, slow convergence, low efficiency, and slow improvement of accuracy.
The representative area extraction method is adopted to identify feature areas from the image acquisition module, filter feature points, create polygon representative boxes, calculate the interchange ratio and area, filter out the candidate representative area images, and then perform road surface state recognition.
It improves the efficiency and accuracy of image processing, ensures rapid identification of road surface status, and enhances driving safety and economy.
Smart Images
Figure CN118485985B_ABST
Abstract
Description
Technical Field
[0001] The present invention belongs to the field of automatic driving and vehicle networking image data processing, and its background is to use the real-time image of a vehicle-mounted camera to judge the current road condition. Background Art
[0002] This project reuses the real-time image of the vehicle's front-view camera to obtain real-time images of the road surface, uses image classification algorithms to determine the real-time status of the road surface, and the electronic control module or autonomous driving module calls the recognition results.
[0003] In the context of this project, an important problem that needs to be solved is how to quickly extract the image blocks that are most likely to be of the greatest concern to the road surface from the images taken by the on-board camera, rather than extracting other targets in the image, such as roadside buildings, trees, other vehicles, pedestrians, etc.
[0004] Real-time identification of road surface conditions and classification, different road surfaces, or different conditions of the same type of road surface, are of practical significance to vehicle handling and driving safety. For example, asphalt concrete road surface + water accumulation constitutes one road surface condition; asphalt concrete + high temperature constitutes another road surface condition; neat stone road surface + road debris constitutes another road surface condition.
[0005] According to different road conditions, different strategies are adopted for manual driving or automatic driving (such as different safe braking distances, different climbing ATM gears, etc.). Through the visual recognition and classification of this project, the adhesion coefficients of different road surfaces, driving and braking control strategies are identified, which helps to ensure driving safety and economy. Summary of the invention
[0006] The technical problem to be solved by the present invention is: how to quickly extract the image block that is most likely to be of greatest concern to the road surface from the image taken by the on-board camera, rather than extracting other targets in the image, such as roadside buildings, trees, other vehicles, pedestrians, etc., as well as the problems of random selection of starting sub-blocks, slow convergence, low efficiency, and slow accuracy improvement.
[0007] The present invention proposes a representative region extraction method, which is characterized by comprising the following steps:
[0008] A feature area is identified from the shooting interface image acquired by the image acquisition module, feature points are selected from the feature area, a feature starting point set after translation relative to the feature point is acquired in the shooting interface image with the feature point as a starting point, and then a polygonal representative frame is established in the shooting interface image with each feature starting point in the feature starting point set as a vertex of the polygonal representative frame, and an image in the polygonal representative frame is selected as a representative area image;
[0009] The pixels of each representative region image are extracted in turn and compared with the image of the feature region to obtain the intersection-and-union ratio, and the candidate representative region images are screened from all representative region images according to the intersection-and-union ratio and the area or side length of the representative region.
[0010] Furthermore, the feature points include a geometric center or a vertex of the feature region, a midpoint of an edge line of the feature region, and a projection point of a center or a vertex of the feature region on an edge line of the feature region.
[0011] Furthermore, the polygonal representative frame is a rectangle.
[0012] Furthermore, the shape formed by the vertices of the polygonal representative frame is a polygon that is symmetrical in terms of top and bottom and / or left and right, and each vertex is located at the edge of the feature area.
[0013] Further, establishing a polygonal representative frame in the shooting interface image using each feature starting point in the feature starting point set as a vertex of the polygonal representative frame includes:
[0014] From the feature starting point, translate along the first edge line of the feature area perpendicular to the feature starting point in the first direction to the second edge line of the feature area to obtain the second vertex, then translate along the second direction parallel to the first edge line to the third edge line of the feature area to obtain the third vertex, then translate along the first edge line perpendicular to the first edge line in the opposite direction to the first direction to the first edge line of the feature area to obtain the fourth vertex, then return to the feature starting point along the opposite direction of the second direction parallel to the first edge line to obtain the polygonal representative frame;
[0015] The first direction is toward other edge lines of the characteristic region, and the second direction is toward other edge lines of the characteristic region.
[0016] Furthermore, the candidate representative region images are screened from all representative region images according to the intersection-and-union ratio and the area or side length of the representative region, including:
[0017] Calculate the intersection-and-union ratio of the representative region images, discard the representative region images whose intersection-and-union ratio is less than the threshold 1, and then discard the representative region images whose side length of the representative region does not reach the threshold 2;
[0018] Calculate the parameter area of the remaining representative area image, parameter area rounding Representative area image that does not reach the third threshold, area ij is the area of the representative region, β is a hyperparameter, h represents the side length of the representative region, i represents the feature region image of the i-th or i-th frame, and j represents the j-th representative region frame image in the i-th feature region image;
[0019] Select from the remaining representative area images The representative region image corresponding to the maximum value is used as a candidate representative region image.
[0020] Furthermore, a computer includes a processor and a memory, wherein the memory stores instructions, and the instructions can be executed by the processor to perform the above method.
[0021] Furthermore, a vehicle driving system further includes a component actuated based on the representative area image.
[0022] Furthermore, in a vehicle driving system, the memory stores judgment instructions, and the instructions can be executed by the processor to judge the type of road defects and send road defect information.
[0023] Compared with the prior art, the present invention has the following advantages and beneficial effects:
[0024] The prior art provides a loose neighborhood spreading algorithm to calculate the uniformity of the extrapolated segment. In the extrapolated segment, a sub-block (single-point start) or multiple blocks (multi-point start) is randomly selected, and the color similarity and texture similarity of the sub-blocks directly or indirectly adjacent to itself are calculated (that is, spreading outward from the initial block). If both exceed a certain threshold, the adjacent block is considered to be uniform and consistent with itself (just like water droplets gather into puddles). This iterative expansion is carried out until no adjacent sub-blocks can be incorporated. If the total area of the last uniform block exceeds the total area of the entire extrapolated segment (calculated based on the number of pixels) by a certain threshold (for example, 0.8), the current extrapolated segment is considered to be a valid road surface image. The next step is to input it into the subsequent algorithm for road surface state classification. Otherwise, it is considered to be an invalid road surface image, and the subsequent frame is waiting to extract the valid road surface image. Among them, loose means that each sub-block that has been incorporated into uniformity will not only explore its own direct adjacent sub-blocks when exploring the sub-blocks around itself, but also explore the direct adjacent sub-blocks of the directly adjacent sub-blocks, that is, its own indirect adjacent sub-blocks. This loose treatment takes into account that the road surface is often briefly disrupted by lane lines, fallen leaves, waste paper, etc., but the overall continuity remains.
[0025] Compared with the prior art, the present invention draws a series of road surface pixel representative rectangular frames in the image of the drivable area, extracts pixels of the road surface pixel representative rectangular frames and compares them with the original image of the drivable area to obtain the intersection-and-union ratio, and screens candidate road surface pixel representative rectangular frame images according to the intersection-and-union ratio and the area and height of the road surface pixel representative rectangular frame images, and uses the screened road surface pixel representative rectangular frame images to perform road surface state recognition, which solves the objective shortcomings of the prior art of random selection of starting sub-blocks, slow convergence, low efficiency, and slow accuracy improvement. BRIEF DESCRIPTION OF THE DRAWINGS
[0026] Figure 1 It is a system framework diagram of the present invention;
[0027] Figure 2 A schematic diagram of determining a near line according to a drivable area according to the present invention;
[0028] Figure 3 The grayscale image shape, relevant feature points and their area selection of the drivable area;
[0029] Figure 4 is a drivable area diagram of the present invention;
[0030] Figure 5 This is a state recognition diagram using the method of the present invention. DETAILED DESCRIPTION
[0031] In order to make the purpose, technical solutions and advantages of the present invention more clearly explained, the present invention will be further explained in detail below in conjunction with the accompanying drawings.
[0032] The present invention is described in detail below through examples. The examples are only used to further illustrate the present invention and cannot be understood as limiting the scope of protection of the present invention. Some non-essential improvements and adjustments made by technicians in this field based on the contents of the present invention also fall within the scope of protection of the present invention.
[0033] Example 1
[0034] A road surface representative area extraction system framework of the present invention is as follows Figure 1 As shown, based on the image of the drivable area extracted by the drivable area recognition module, a series of road surface pixel representative rectangular frames are drawn in the image according to rules, the pixels of the road surface pixel representative rectangular frames are extracted and compared with the original drivable area image to obtain the intersection-and-union ratio, and the candidate road surface pixel representative rectangular frame images are screened according to the intersection-and-union ratio and the area and height of the road surface pixel representative rectangular frame images, and the screened road surface pixel representative rectangular frame images are used to perform road surface condition recognition.
[0035] It is assumed here that the drivable area obtained by the drivable area recognition module is a closed area full of pixels.
[0036] The intersection-union ratio, a concept used in object detection, is the overlap ratio between the generated candidate box and the original labeled box, that is, the ratio of their intersection to their union. The ideal situation is complete overlap, that is, the ratio is 1.
[0037] A road surface representative area extraction method performed by a road surface representative area extraction system according to the present invention comprises the following steps:
[0038] Step 1. Identify the feature area from the shooting interface image captured by the image acquisition module, the image acquisition module includes a camera or a video camera, the shooting interface image is an image captured by the camera or a frame of an image frame captured by the video camera, and the feature area in one embodiment of the present invention is a drivable area of the road surface. The drivable area recognition module based on the execution of the target detection neural network algorithm can identify the drivable area of the road surface in the shooting interface image.
[0039] Step 2. Select feature points from the feature area. The feature points may be geometric midpoints or vertices of the drivable area, or projection points of the geometric midpoints or vertices of the drivable area on the edge line of the feature area, or other points.
[0040] Vertex includes Figure 2 The left vertex or right vertex or the point with the maximum y in the drivable area;
[0041] If the feature point is other points in the drivable area, the near line needs to be determined based on the drivable area. For example, the near line can be selected as the edge line closer to the front of the vehicle in the drivable area, which is referred to as the starting reference line in this embodiment. Figure 2 The red line in.
[0042] Step 3. Taking the feature point (such as the left point, right point or midpoint of the near line) as the starting point, collect the feature starting point set after translation relative to the feature point in the feasible area, and then use each feature starting point in the feature starting point set as a vertex of the polygon representative box to establish a polygon representative box in the shooting interface image;
[0043] In this embodiment, the polygonal representative frame is a polygon that is symmetrical in both vertical and horizontal directions, and / or symmetrical in both horizontal and vertical directions, and each vertex is located at the edge of the feature area. For example, the polygonal representative frame is a rectangular frame, and the result is as follows: Figure 3 The road surface represents the drawing rectangle in the . The feature starting point set refers to the set of feature points selected in step 2.
[0044] The strategy of drawing a polygonal representative box in the shooting interface image with each feature starting point in the feature starting point set as a vertex of the polygonal representative box can be selected according to the shape of the grayscale image of the drivable area, the relevant feature points and their areas, such as the following method:
[0045] Method 1: 1) Take the pixel point (x0, y0) at the left end of near line 1 as the feature point, move v pixels to the right to (xv, y0), and take (xv, y0) as the feature starting point and move v pixels upward along a straight line to the edge point (xv, yv) of the feasible area, which is the second vertex, and then move v pixels to the right along a straight line to the edge point (xv1, yv) of the feasible area, which is the third vertex. Then move v pixels downward along a straight line to the edge point (xv1, y0) of the feasible area, which is the fourth vertex. A representative rectangular box can be drawn based on the four vertices of the rectangular box: (xv, y0), (xv, yv), (xv1, yv), (xv1, y0).
[0046] 2) Taking (xv1, y0) as the starting point, refer to the steps in 1) to obtain the representative rectangular box ((x2v, y0), (x2v, y2v), (x2v1, y2v), (x2v1, y0)).
[0047] 3) Change the feature starting point and repeat 2) to obtain i representative rectangular boxes.
[0048] Method 2: Starting from the right end point of the near line, move to the left. The steps are similar to method 1, and j representative rectangular boxes can also be obtained.
[0049] Method 3: Take the midpoint of the near line or the projection point of the highest point on the near line as the starting point, and move to the second vertex reached after synchronous or asynchronous translation on both sides, move upward along the straight line to the edge point, take the lower point as the third vertex of the rectangle, and then move downward along the straight line to the near line as the fourth vertex of the rectangle to obtain a representative rectangular frame.
[0050] Referring to this method, continue to move from the starting point to both sides respectively, and then repeat to obtain the third vertex and the fourth vertex, and k representative rectangular boxes can be obtained.
[0051] Method 4: Starting from the geometric center of the drivable area, move synchronously or asynchronously in four directions to find four vertices, and draw representative rectangular boxes based on the four vertices.
[0052] Since the drivable area is random and irregular, you can choose one or more methods to combine to find the representative rectangular frame and get the representative rectangular frame set image of the image. i -presentations, where the subscript i represents the i-th picture or the i-th frame.
[0053] Step 4. In order to obtain a larger road surface representative area, we can avoid information loss caused by upsampling and affect the accuracy of subsequent road classification. At the same time, we can obtain a variety of road surface representative areas and tend to focus on representative areas farther away from the front of the vehicle. In this step, we select the image in the polygonal representative frame as the representative area image, extract each representative area image in turn, and perform the image extraction on the representative area frame set image.i -presentations to analyze and judge, search for the most representative regional subset and incorporate it into the road classification recognition dataset. It includes the following steps:
[0054] 1) Extract the pixels of each representative area image in turn and compare them with the image of the feature area, calculate the intersection-over-union ratio of the representative area image, discard the representative area images whose intersection-over-union ratio IOU is less than threshold 1, and discard the representative area images whose representative rectangular box height h (height h is one side length of the representative area) does not reach threshold 2.
[0055] 2) Calculate the parameter area of the remaining representative area image, parameter area rounding Representative area image that does not reach the third threshold, area ij is the area of the representative region, β is a hyperparameter that can be adjusted according to needs, h represents the side length of the representative region, i represents the i-th feature region image or the i-th frame, and j represents the j-th representative region frame image in the i-th feature region image.
[0056] 3) Select from the remaining representative area images The representative area image corresponding to the maximum value is used as a candidate representative area image, that is, an available representative area image, which is incorporated into the road classification and recognition dataset.
[0057] The height h in step 1) can also be replaced by the width of the representative area.
[0058] When height is selected as the side length to filter representative area images, representative area images that are farther away from the vehicle can be obtained in advance, and road surface information can be obtained in advance.
[0059] When the width is selected as the side length to filter the representative area image, the representative area image far away from both sides of the vehicle can be obtained in advance, which is convenient for obtaining the information on both sides of the road. A computer includes a processor and a memory, the memory stores instructions, and the instructions can be executed by the processor to perform the method according to the above steps 1 to 4.
[0060] A vehicle driving system includes components actuated based on the representative area image selected above.
[0061] The above method can be used to identify the road surface condition classification in real time. Different road surfaces, or different conditions of the same type of road surface, have practical significance for vehicle handling stability and driving safety.
[0062] For example: asphalt concrete pavement + water accumulation constitutes one pavement state; asphalt concrete + high temperature constitutes another pavement state; neat stone pavement + road debris constitutes another pavement state. The state of a road surface is composed of two states: the inherent state and the real-time state. The inherent state refers to the pavement material and grade, which are the inherent properties of the road determined by road construction; the real-time state mainly refers to temporary road coverings or meteorological conditions, water accumulation, ice and snow, frost, road debris, glass nails, fallen leaves, high temperature (asphalt softening), cracks, etc.
[0063] According to different road conditions, different strategies are adopted for manual driving or automatic driving (such as different safe braking distances, different climbing ATM gears, etc.). Through the visual recognition and classification of this project, the adhesion coefficients of different road surfaces, driving and braking control strategies are identified, which helps to ensure driving safety and economy.
[0064] Combining the inherent state and the real-time state, the current real-time state of the road surface is obtained.
[0065] As an optional application, the vehicle driving system is further provided with a memory storing judgment instructions, wherein the instructions can be executed by the processor to judge the type of road defects and send road defect information.
[0066] The above are only preferred embodiments of the present application, and the present application is not limited to the above examples. It is understood that other improvements and changes directly derived or associated by those skilled in the art without departing from the spirit and concept of the present application should be considered to be included in the protection scope of the present application.
Claims
1. A representative region extraction method, characterized in that: The following steps are involved: A feature area is identified from the shooting interface image acquired by the image acquisition module, feature points are selected from the feature area, a feature starting point set after translation relative to the feature point is acquired in the shooting interface image with the feature point as a starting point, and then a polygonal representative frame is established in the shooting interface image with each feature starting point in the feature starting point set as a vertex of the polygonal representative frame, and an image in the polygonal representative frame is selected as a representative area image; Extract pixels of each representative region image in turn and compare them with the image of the feature region, calculate the intersection-and-union ratio of the representative region images, discard representative region images whose intersection-and-union ratio is less than a first threshold, and then discard representative region images whose side lengths of the representative regions do not reach a second threshold; Calculate the parameter area of the remaining representative area image: parameter area ,rounding Representative area images that do not reach the third threshold, is the area of the representative region, β is a hyperparameter, h represents the side length of the representative region, i represents the feature region image of the i-th or i-th frame, and j represents the j-th representative region frame image in the i-th feature region image; Select from the remaining representative area images The representative region image corresponding to the maximum value is used as a candidate representative region image.
2. The method according to claim 1, characterized in that: The feature points include the geometric center or vertex of the feature area, the midpoint of the edge line of the feature area, and the projection point of the center or vertex of the feature area on the edge line of the feature area.
3. The method according to claim 1, characterized in that The polygonal representative frame is a rectangle.
4. The method according to claim 3, characterized in that The shape formed by the vertices of the polygonal representative frame is a polygon that is symmetrical in terms of top and bottom, and / or left and right, and each vertex is located at the edge of the feature area.
5. The method according to claim 4, characterized in that Establishing a polygonal representative frame in the shooting interface image using each feature starting point in the feature starting point set as a vertex of the polygonal representative frame includes: From the feature starting point, translate along the first edge line of the feature area perpendicular to the feature starting point in the first direction to the second edge line of the feature area to obtain the second vertex, then translate along the second direction parallel to the first edge line to the third edge line of the feature area to obtain the third vertex, then translate along the first edge line perpendicular to the first edge line in the opposite direction to the first direction to the first edge line of the feature area to obtain the fourth vertex, then return to the feature starting point along the opposite direction of the second direction parallel to the first edge line to obtain the polygonal representative frame; The first direction is toward other edge lines of the characteristic region, and the second direction is toward other edge lines of the characteristic region.
6. A computer comprising a processor and a memory, the memory storing instructions executable by the processor to perform the method according to one of claims 1 to 5.
7. A vehicle driving system comprising the computer of claim 6, further comprising a component actuated based on the representative area image. 8 . The vehicle driving system according to claim 7 , wherein the memory stores a judgment instruction, and the instruction can be executed by the processor to judge the defect type of the road and send road defect information.