Automatic line patrol method, terminal equipment and computer readable storage medium

Abstract

The invention is suitable for the technical field of intelligent control, and particularly relates to an automatic line patrol method, terminal equipment and a computer readable storage medium. The method includes: acquiring a first shot image of a road in front of a trolley; performing image segmentation processing on a track line in the first shot image to obtain a mask image; detecting first line patrol information of a road in front of the trolley according to the mask image; and controlling the trolley to move according to the first line patrol information. In the embodiment of the invention, the mask image of the track line is extracted from the shot image of the road in front of the trolley, and then the track line is detected according to the mask image of the track line. The mask image is not easily influenced by illumination conditions, and background noise is filtered out of the mask image, so that the detection precision can be effectively improved by detecting the track line through the mask image, and the automatic navigation capability of the line patrol trolley is favorably improved.

Description

Technical Field

[0001] The present application belongs to the field of intelligent control technology, and in particular, relates to an automatic line patrol method, a terminal device, and a computer-readable storage medium. Background Art

[0002] A line patrol car (also called a tracking robot) is an intelligent car that automatically drives along a specific route. Line patrol (or tracking) is a key task in the process of automatic driving of line patrol cars. The principle of line patrol is to detect the track line on the ground through sensors and move along the track line.

[0003] In the current line inspection method, the track line color is usually compared with the background color, or it relies on a fixed template, that is, the track line in the actual scene is matched with the fixed template to detect the track line. In some complex scenes, such as when the light changes greatly or the background scene is complex, the above methods show certain limitations and cannot accurately detect the track line, thus affecting the automatic driving of the line inspection car. Summary of the invention

[0004] The embodiments of the present application provide an automatic line patrol method, a terminal device, and a computer-readable storage medium, which can effectively improve the detection accuracy of road information, thereby helping to improve the automatic line patrol capability of line patrol elimination.

[0005] In a first aspect, an embodiment of the present application provides an automatic line patrol method, comprising:

[0006] Acquire a first captured image of the road ahead of the vehicle;

[0007] Performing image segmentation processing on the track line in the first captured image to obtain a mask image;

[0008] Detecting first patrol line information of the road ahead of the vehicle according to the mask image;

[0009] The movement of the vehicle is controlled according to the first line patrol information.

[0010] In the embodiment of the present application, a mask image of the track line is extracted from the captured image of the road in front of the vehicle, and then the track line is detected based on the mask image of the track line. Since the mask image is not easily affected by lighting conditions and background noise is filtered out in the mask image, detecting the track line through the mask image can effectively improve the detection accuracy, which is conducive to improving the automatic navigation capability of the line patrol vehicle.

[0011] In a possible implementation manner of the first aspect, the first line patrol information includes an intersection type of a road intersection;

[0012] The detecting first patrol line information of the road ahead of the vehicle according to the mask image includes:

[0013] Detecting whether the mask image includes a road intersection;

[0014] If the mask image includes a road intersection, the intersection type of the road intersection is detected according to the mask image.

[0015] In the above method, it is first determined whether a road intersection exists. If a road intersection exists, the subsequent intersection type detection process is performed; if a road intersection does not exist, the subsequent process can be reduced. This helps reduce algorithm redundancy and thus helps improve detection efficiency.

[0016] In a possible implementation manner of the first aspect, detecting the intersection type of the road intersection according to the mask image includes:

[0017] In the first line patrol mode, detecting the number of changes in pixel values ​​within a preset range including the road intersection in the mask image;

[0018] The intersection type of the road intersection is determined according to the number of changes.

[0019] In a possible implementation manner of the first aspect, detecting the number of changes in pixel values ​​within a preset range including the road intersection in the mask image includes:

[0020] Acquire a rectangular frame including the road intersection; wherein the area within the rectangular frame is within the preset range;

[0021] The number of times the pixel value changes at the intersection of the rectangular frame and the track line is counted to obtain the number of changes.

[0022] In the above method, by detecting the number of changes in pixel values ​​within a preset range of the road intersection, it is equivalent to detecting the number of boundaries of the track at the road intersection, thereby realizing the judgment of the intersection type of the road intersection.

[0023] In a possible implementation manner of the first aspect, detecting the intersection type of the road intersection according to the mask image includes:

[0024] In the second line-following mode, the mask image is divided into regions according to the track lines in the mask image to obtain at least one connected region;

[0025] Calculate the minimum circumscribed circle of the trajectory in each of the connected areas;

[0026] The intersection type of the road intersection is detected according to the number of connected areas in the mask image and the radius of each of the minimum circumscribed circles.

[0027] In a possible implementation manner of the first aspect, dividing the mask image into regions according to the track lines in the mask image to obtain at least one connected region includes:

[0028] Dividing the mask image into regions according to the track lines in the mask image to obtain at least one candidate region;

[0029] Calculating the area of ​​each candidate region;

[0030] Candidate regions whose areas are smaller than a preset value are filtered out to obtain at least one of the connected regions.

[0031] Through the above method, combined with the number of connected areas and the radius of the minimum circumscribed circle, the probability of false detection can be effectively reduced, thereby effectively improving the detection accuracy of the intersection type.

[0032] In a possible implementation manner of the first aspect, the first line patrol information includes a line patrol start point and a line patrol end point;

[0033] The detecting first patrol line information of the road ahead of the vehicle according to the mask image includes:

[0034] Searching for a first boundary point of the track line in a first row of pixel points of the mask image; wherein the first row of pixel points is located at the bottom of the mask image;

[0035] Searching for a second boundary point of the track line in a second row of pixel points of the mask image; wherein the second row of pixel points is above the first row of pixel points, and there is a spacing of d rows of pixels between the second row of pixel points and the first row of pixel points;

[0036] Calculate the line patrol starting point according to the first boundary point;

[0037] The line patrol endpoint is calculated according to the second boundary point.

[0038] In a possible implementation manner of the first aspect, controlling the movement of the vehicle according to the first line patrol information includes:

[0039] Acquire second line patrol information; wherein the second line patrol information is line patrol information detected based on at least one frame of captured image before the first captured image;

[0040] If the first line patrol information is inconsistent with the second line patrol information, the movement of the vehicle is controlled according to the second line patrol information.

[0041] Through the above method, the impact of single-frame false detection on line inspection can be reduced, and the detection accuracy is effectively improved.

[0042] In a second aspect, an embodiment of the present application provides an automatic line patrol device, comprising:

[0043] An acquisition unit, used for acquiring a first captured image of the road ahead of the vehicle;

[0044] a segmentation unit, configured to perform image segmentation processing on the track line in the first captured image to obtain a mask image;

[0045] A detection unit, configured to detect first patrol line information of the road ahead of the vehicle according to the mask image;

[0046] A control unit is used to control the movement of the vehicle according to the first line patrol information.

[0047] In a third aspect, an embodiment of the present application provides a terminal device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, an automatic line patrol method as described in any one of the first aspects above is implemented.

[0048] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the automatic line patrol method as described in any one of the above-mentioned first aspects is implemented.

[0049] In a fifth aspect, an embodiment of the present application provides a computer program product. When the computer program product runs on a terminal device, the terminal device executes the automatic line patrol method described in any one of the above-mentioned first aspects.

[0050] It can be understood that the beneficial effects of the second to fifth aspects mentioned above can be found in the relevant description of the first aspect mentioned above, and will not be repeated here. BRIEF DESCRIPTION OF THE DRAWINGS

[0051] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.

[0052] Figure 1 It is a flowchart of the automatic line patrol method provided in the embodiment of the present application;

[0053] Figure 2 It is a node schematic diagram provided in an embodiment of the present application;

[0054] Figure 3 is a schematic diagram of a track line provided in an embodiment of the present application;

[0055] Figure 4 It is a schematic diagram of an intersection in a single track line patrol mode provided in an embodiment of the present application;

[0056] Figure 5 is a schematic diagram of detecting a road intersection provided in an embodiment of the present application;

[0057] Figure 6 is a schematic diagram of a connected area provided in an embodiment of the present application;

[0058] Figure 7 It is a schematic diagram of the line patrol starting point and the line patrol end point provided in an embodiment of the present application;

[0059] Figure 8 is a schematic diagram of a line patrol process provided by an embodiment of the present application;

[0060] Fig. 9 is a structural block diagram of an automatic line patrol device provided in an embodiment of the present application;

[0061] Fig.10 It is a schematic diagram of the structure of the terminal device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0062] In the following description, specific details such as specific system structures, technologies, etc. are provided for the purpose of illustration rather than limitation, so as to provide a thorough understanding of the embodiments of the present application. However, it should be clear to those skilled in the art that the present application may also be implemented in other embodiments without these specific details. In other cases, detailed descriptions of well-known systems, devices, circuits, and methods are omitted to prevent unnecessary details from obstructing the description of the present application.

[0063] It should be understood that when used in the present specification and the appended claims, the term "comprising" indicates the presence of described features, wholes, steps, operations, elements and / or components, but does not exclude the presence or addition of one or more other features, wholes, steps, operations, elements, components and / or combinations thereof.

[0064] It should also be understood that the term “and / or” used in the specification and appended claims refers to any and all possible combinations of one or more of the associated listed items, and includes these combinations.

[0065] As used in the specification and appended claims of this application, the term "if" can be interpreted as "when" or "uponce" or "in response to determining" or "in response to detecting", depending on the context. Similarly, the phrase "if it is determined" or "if [described condition or event] is detected" can be interpreted as meaning "uponce it is determined" or "in response to determining" or "uponce [described condition or event] is detected" or "in response to detecting [described condition or event]", depending on the context.

[0066] In addition, in the description of the present application specification and the appended claims, the terms "first", "second", "third", etc. are only used to distinguish the descriptions and cannot be understood as indicating or implying relative importance.

[0067] References to "one embodiment" or "some embodiments" etc. described in the specification of this application mean that one or more embodiments of the present application include specific features, structures or characteristics described in conjunction with the embodiment. Therefore, the phrases "in one embodiment", "in some embodiments", "in some other embodiments", "in some other embodiments", etc. appearing in different places in this specification do not necessarily refer to the same embodiment, but mean "one or more but not all embodiments", unless otherwise specifically emphasized in other ways.

[0068] A line patrol car (also called a tracking robot) is an intelligent car that automatically drives along a specific route. Line patrol (or tracking) is a key task in the process of automatic driving of line patrol cars. The principle of line patrol is to detect the track line on the ground through sensors and move along the track line.

[0069] In the current line inspection method, the track line color is usually compared with the background color, or it relies on a fixed template, that is, the track line in the actual scene is matched with the fixed template to detect the track line. In some complex scenes, such as when the light changes greatly or the background scene is complex, the above methods show certain limitations and cannot accurately detect the track line, thus affecting the automatic driving of the line inspection car.

[0070] For example, in the template matching method, a predefined fixed template is used to match the intersection in the actual scene. However, this method is sensitive to the shape, size and lighting conditions of the intersection, and cannot cope with non-standard intersections, complex multi-fork intersections, etc., especially in scenes with large changes in light or complex backgrounds, the template matching method often fails.

[0071] For another example, in the edge detection method, track information is extracted by edge detection. This method is difficult to ensure the stability and accuracy of detection when the lighting conditions change (such as shadows, backlighting or different ambient light), when the track edge is blurred or the background noise is complex.

[0072] Based on this, an embodiment of the present application provides an automatic line patrol method. In the embodiment of the present application, a mask image of a track line is extracted from a captured image of the road in front of the vehicle, and then the track line is detected based on the mask image of the track line. Since the mask image is not easily affected by lighting conditions and background noise is filtered out in the mask image, therefore, detecting the track line through the mask image can effectively improve the detection accuracy, which is conducive to improving the automatic navigation capability of the line patrol vehicle.

[0073] See also Figure 1 , is a flow chart of the automatic line patrol method provided in an embodiment of the present application. As an example but not a limitation, the method may include the following steps:

[0074] S101, obtaining a first captured image of the road in front of the vehicle.

[0075] The automatic line patrol method in the embodiment of the present application can be applied to the controller (or processor, etc.) of the vehicle.

[0076] Among them, a camera is installed on the body of the car, and the camera can obtain the shooting image of the road in front of the car. Correspondingly, the controller of the car obtains the shooting image collected by the camera by interacting with the camera.

[0077] The camera can shoot at a preset period. For example, it can shoot once every 1s, once every 5s, etc. In practical applications, the preset period of camera shooting can be determined according to the moving speed of the car. For example, if the moving speed of the car is 1m / s, the preset period of the camera can be set to 1s.

[0078] It is understandable that the preset cycle of the camera can be consistent with the control cycle of the controller. For example, the camera collects a shot image every 1 second, and accordingly, the controller obtains a shot image every 1 second and executes an automatic line patrol method according to the acquired shot image.

[0079] S102, performing image segmentation processing on the track line in the first captured image to obtain a mask image.

[0080] In an embodiment of the present application, an image segmentation algorithm can be used to detect and segment the track lines in the first captured image to obtain a segmented image; then the segmented image is binarized, such as setting the pixels of the track lines to 1 and the pixels of the non-track lines to 0, or setting the pixels of the track lines to 0 and the pixels of the track lines to 1, to obtain a mask image.

[0081] In one embodiment, the first captured image may be input into a trained segmentation model to output a mask image, wherein the segmentation model may perform image segmentation processing on the input image and perform binarization processing on the segmented image.

[0082] Optionally, the segmentation model may adopt a neural network model.

[0083] For example, a method for training a segmentation model may include: collecting a large number of sample images, each sample image corresponds to a reference mask image; inputting the sample image into the segmentation model and outputting a predicted mask image; calculating the difference between the predicted mask image and the reference mask image to obtain a loss value of the segmentation model; if the loss value is less than a preset value, determining the current segmentation model as the trained segmentation model; if the loss value is greater than or equal to the preset value, adjusting the parameters of the segmentation model according to the loss value, and continuing to train the segmentation model until the loss value of the segmentation model is less than the preset value.

[0084] In the above method, since the segmentation model is pre-trained, it can effectively improve the efficiency of image segmentation processing, which is conducive to improving the efficiency of automatic line patrol. In addition, the trained segmentation model has high accuracy, so using the trained segmentation model can obtain more accurate segmentation results, providing a reliable data basis for subsequent automatic line patrol.

[0085] In some implementations, the segmentation model may also be trained to detect the location of the intersection. Accordingly, when the captured image includes a road intersection, the segmentation model may output a mask image and mark the location of the road intersection in the mask image.

[0086] It is understandable that if the segmentation model includes the function of detecting the intersection position, the reference position of the road intersection in the sample image needs to be pre-labeled during the training process. When calculating the loss value of the segmentation model, it is necessary not only to calculate the difference between the predicted mask image and the reference mask image, but also to calculate the difference between the predicted position and the reference position of the road intersection output by the segmentation model.

[0087] For example, see Figure 2 , is a node diagram provided in the embodiment of the present application. As an example and not a limitation, Figure 2 In the mask image shown, the black part represents the track line, and the white part is the area without the track line. Among them, a rectangular box (shaded box) is marked at the road intersection, that is, the node.

[0088] It should be noted that Figure 2 This is only an example of marking a road intersection. In actual applications, the location of the road intersection can also be marked with a dot or a circular frame, etc., and the embodiments of the present application do not specifically limit this.

[0089] S103, detecting first patrol line information of the road ahead of the vehicle according to the mask image.

[0090] In one embodiment, the first line patrol information may include the intersection type of the road intersection. Accordingly, S103 includes:

[0091] Detecting whether the mask image includes a road intersection;

[0092] If the mask image includes a road intersection, the intersection type of the road intersection is detected according to the mask image.

[0093] Optionally, as described in the above embodiment, the segmentation model may output the location information of the road intersection. Accordingly, if the segmentation model outputs the location information of the road intersection, it is determined that the mask image includes the road intersection.

[0094] Optionally, if the segmentation model does not have the function of detecting road intersections, it can be determined whether there is overlap in the black areas in the mask image; if there is overlap, it is determined that the mask image includes a road intersection.

[0095] In the above method, it is first determined whether a road intersection exists. If a road intersection exists, the subsequent intersection type detection process is performed; if a road intersection does not exist, the subsequent process can be reduced. This helps reduce algorithm redundancy and thus helps improve detection efficiency.

[0096] Generally, line inspection methods include single-track line inspection and double-track line inspection. For example, see Figure 3 , is a schematic diagram of the track line provided in the embodiment of the present application. Figure 3 As shown in (a) in the figure, it is a mask image in the single track patrol mode. In this mode, the black area in the mask image represents the track. Figure 3 As shown in (b), it is a mask image obtained in the dual-track patrol mode. In this mode, the black area in the mask image represents the boundary of the track, and the part between the two black lines represents the track.

[0097] The intersection detection methods in the two patrol modes are different, which are introduced below.

[0098] In the single track patrol mode (i.e., the first patrol mode), in one implementation, the intersection type detection method may include:

[0099] Detecting the number of changes in pixel values ​​within a preset range including a road intersection in the mask image;

[0100] The intersection type of the road intersection is determined according to the number of changes.

[0101] Among them, the intersection types may include straight lines, T-intersections, and crossroads.

[0102] For example, see Figure 4, is a schematic diagram of a road junction in a single track patrol mode provided in an embodiment of the present application. As an example and not a limitation, Figure 4 As shown in (a) in the figure, it is a straight line intersection. Figure 4 As shown in (b) in the figure, it is a crossroads. Figure 4 As shown in (c), it is a T-junction.

[0103] from Figure 4 It can be seen that in the single track patrol mode, the black area in the mask image represents the track, and the number of boundaries of the tracks of different types of road intersections is also different, and the pixel value at the track boundary in the mask image jumps. Based on this, in the above method, by detecting the number of changes in pixel values ​​within a preset range of the road intersection, it is equivalent to detecting the number of boundaries of the track at the road intersection, thereby realizing the judgment of the intersection type of the road intersection.

[0104] Optionally, the method of detecting the number of changes may include:

[0105] Acquire a rectangular frame including the road intersection; wherein the area within the rectangular frame is within a preset range;

[0106] The number of pixel value changes at the intersection of the rectangular box and the track line is counted to obtain the number of changes.

[0107] For example, see Figure 5 , is a schematic diagram of the detection of a road intersection provided in an embodiment of the present application. As an example and not a limitation, Figure 5 (a) in FIG. 5 is a schematic diagram of a straight line intersection. A rectangular frame 51 includes a road intersection. There are four intersection points 52 between the rectangular frame 51 and the track line. A pixel change occurs at each intersection point 52, that is, the number of changes is 4.

[0108] like Figure 5 (b) in FIG. 5 is a schematic diagram of a crossroad. A rectangular frame 51 includes a road intersection. There are 8 intersection points 52 between the rectangular frame 51 and the track line. A pixel change occurs at each intersection point 52, that is, the number of changes is 8.

[0109] like Figure 5 (c) in FIG. 5 is a schematic diagram of a T-junction. A rectangular frame 51 includes a road intersection. There are six intersection points 52 between the rectangular frame 51 and the track line. A pixel change occurs at each intersection point 52, that is, the number of changes is 6.

[0110] It should be noted that, in actual applications, a circular frame, an elliptical frame or a diamond frame may also be used to determine the preset range, and the embodiments of the present application do not specifically limit this.

[0111] The pixel value change refers to a change from 1 to 0, or from 0 to 1.

[0112] Optionally, the number of changes in pixel values ​​in the horizontal direction of the image can be counted using the following formula:

[0113]

[0114] The number of changes in pixel values ​​along the vertical axis of the image can be counted using the following formula:

[0115]

[0116] The coordinates of the upper left corner of the rectangle are (rect0x, rect0y), and the coordinates of the lower right corner of the rectangle are (rext1x, rext1y). mask() represents the pixel value of the coordinate point in the brackets. "^" is an exclusive OR operation. 1{} means that the value is 1 when the conditions in the brackets are met.

[0117] In the second line patrol mode, in one implementation, the intersection type detection method may include:

[0118] Dividing the mask image into regions according to the track lines in the mask image to obtain at least one connected region;

[0119] Calculate the minimum circumscribed circle of the orbital line in each connected area;

[0120] The intersection type of the road intersection is detected according to the number of connected areas in the mask image and the radius of each minimum circumscribed circle.

[0121] It can be understood that if a region G on a plane can draw a simple closed curve in it and the interior of the closed curve always belongs to G, then G is called a simply connected region.

[0122] For example, see Figure 6 , is a schematic diagram of a connected area provided in an embodiment of the present application. Figure 6 As shown in (a) in Figure 2, the four dashed curves divide the mask image into four connected regions. Figure 6 As shown in (b), three dotted curves divide the mask image into three connected areas.

[0123] Optionally, the mask image can be input into a trained region detection model to output connected regions.

[0124] Optionally, the connected area detection method may include:

[0125] Dividing the mask image into regions according to the track lines in the mask image to obtain at least one candidate region;

[0126] Calculate the area of ​​each candidate region;

[0127] Candidate regions whose areas are smaller than a preset value are filtered out to obtain at least one connected region.

[0128] In some cases, the connected regions may be incorrectly divided due to false detection. By filtering out candidate regions with smaller areas in the above manner, the probability of false detection can be reduced, thereby helping to improve detection accuracy.

[0129] Optionally, the minimum enclosing rectangle of the track line in each connected area can be calculated using the function in Opencv. The function can output the center and radius of the minimum enclosing rectangle.

[0130] In one implementation, detecting the type of a road intersection according to the number of connected areas in the mask image and the radius of each minimum circumscribed circle may include:

[0131] If the number of connected areas is less than or equal to 2, the intersection type is determined to be a straight line intersection.

[0132] If the number of connected areas is greater than or equal to 4, the intersection type is determined to be a crossroads.

[0133] If the number of connected areas is 3, the intersection type is detected based on the radius of each minimum circumscribed circle. Specifically: if the difference between the radius of one of the minimum circumscribed circles of the three connected areas and the radius of the other two minimum circumscribed circles is greater than a preset value, the intersection type is determined to be a T-junction; if the difference between the radius of the minimum circumscribed circles of the three connected areas is less than a preset value, the intersection type is determined to be a crossroads.

[0134] It is understandable that when three connected areas are detected, it may be a T-shaped intersection, or a boundary of the intersection may not appear in the image due to the shooting angle. Therefore, in this case, further judgment is required based on the radius of the minimum circumscribed circle.

[0135] like Figure 6 As shown in (a) of , it is the case of a crossroads. In this case, the radii of the four smallest circumscribed circles are usually not much different. Figure 6 As shown in (b), it is a T-junction. In this case, the radius of the minimum circumscribed circle of the two connected areas on the right is smaller, while the radius of the minimum circumscribed circle of the connected area on the left is larger.

[0136] Through the above method, combined with the number of connected areas and the radius of the minimum circumscribed circle, the probability of false detection can be effectively reduced, thereby effectively improving the detection accuracy of the intersection type.

[0137] In another embodiment, the first line patrol information includes a line patrol start point and a line patrol end point. Accordingly, S103 may include:

[0138] Searching for a first boundary point of the track line in the first row of pixel points of the mask image; wherein the first row of pixel points is located at the bottom of the mask image;

[0139] Searching for a second boundary point of the track line in the second row of pixel points of the mask image; wherein the second row of pixel points are above the first row of pixel points, and there are d rows of pixels between the second row of pixel points and the first row of pixel points;

[0140] Calculate the line patrol starting point based on the first boundary point;

[0141] Calculate the end point of the line patrol based on the second boundary point.

[0142] Optionally, the first row of pixels may be the pixels in the bottom row of the mask image.

[0143] For example, see Figure 7 , is a schematic diagram of the line patrol start point and line patrol end point provided in the embodiment of the present application. Figure 7 As shown, the line patrol starting point is a point in the bottom row of pixels of the mask image, and the line patrol end point is a point in the dth row of pixels from the bottom to the top of the mask image.

[0144] Optionally, the midpoint of the first boundary point can be determined as the line patrol start point, and the midpoint of the second boundary point can be determined as the line patrol end point. Specifically, the line patrol start point and the line patrol end point can be calculated according to the following formula:

[0145]

[0146]

[0147] Among them, lineStart is the starting point of the line patrol, lineEnd is the end point of the line patrol, leftBottom is the left boundary point in the first row of pixels, rightBottom is the right boundary point in the first row of pixels, leftTop is the left boundary point in the second row of pixels, and rightTop is the right boundary point in the second row of pixels.

[0148] Among them, the boundary points can be detected according to the jump of pixel values.

[0149] S104, controlling the movement of the vehicle according to the first line patrol information.

[0150] In one embodiment, S104 may include:

[0151] Acquire second line patrol information; wherein the second line patrol information is line patrol information detected based on at least one frame of captured image before the first captured image;

[0152] If the first line patrol information is inconsistent with the second line patrol information, the movement of the car is controlled according to the second line patrol information.

[0153] For example, cache the detection results of the last five frames, cache each new result at the end of the queue, and remove the oldest result. If the result of the current frame is consistent with the previous four frames, directly output the detection result of the current frame (line patrol information); if not, output the detection result of the previous frame.

[0154] Through the above method, the impact of single-frame false detection on line inspection can be reduced, and the detection accuracy is effectively improved.

[0155] In the embodiment of the present application, a mask image of the track line is extracted from the captured image of the road in front of the vehicle, and then the track line is detected based on the mask image of the track line. Since the mask image is not easily affected by lighting conditions and background noise is filtered out in the mask image, detecting the track line through the mask image can effectively improve the detection accuracy, which is conducive to improving the automatic navigation capability of the line patrol vehicle.

[0156] See also Figure 8 , is a schematic diagram of the line patrol process provided in the embodiment of the present application. As an example and not a limitation, Figure 8 As shown, the line inspection process may include the following steps:

[0157] S801, when the automatic line patrol task starts, the controller of the car obtains the captured image of the current frame.

[0158] The step of acquiring the captured image can be referred to the description in the S101 embodiment, which will not be repeated here.

[0159] Optionally, before acquiring the captured image, track information and vehicle model information may be acquired, wherein the track information includes a single track patrol mode and a double track patrol mode.

[0160] The vehicle type information may include the vehicle type and its corresponding preset threshold. The preset threshold may be used to fix the line patrol start point and line patrol end point. For example, for two adjacent image frames, if the distances from the line patrol start point and line patrol end point calculated in each image frame to the center point of the road intersection are less than the preset threshold, the line patrol start point and line patrol end point of the previous image frame are used. In other words, when the line patrol start point and line patrol end point are close to the road intersection, there is no need to update the line patrol start point and line patrol end point.

[0161] S802, performing image segmentation processing according to the captured image to obtain a mask image of the track line.

[0162] Step S802 is the same as S102. For details, please refer to the description in the embodiment of S102, which will not be repeated here.

[0163] S803, determining whether the mask in the mask image is empty.

[0164] If the mask is empty, continue to obtain the next frame of captured image. If the mask is not empty, execute S804.

[0165] It can be understood that the mask being empty means that the mask image does not include pixel points used to represent the track line.

[0166] S804, determining whether there is an intersection (road crossing) in the mask image.

[0167] If there is no intersection, execute S807, that is, control the movement of the car according to the line patrol information of the previous frame; if there is an intersection, execute S805.

[0168] S805, detecting line patrol information.

[0169] Step S805 is the same as S103. For details, please refer to the description of the embodiment of S103, which will not be repeated here.

[0170] S806, determining whether the line patrol information is abnormal.

[0171] If the line patrol information is normal, execute S807; if the line patrol information is abnormal, execute S808.

[0172] S807, controlling the movement of the car according to the line patrol information of the current frame.

[0173] S808, controlling the movement of the car according to the line patrol information of the previous frame.

[0174] Steps S806-S808 are the same as S104. For details, please refer to the description of the embodiment of S104, which will not be repeated here.

[0175] In the embodiment of the present application, a mask image of the track line is extracted from the captured image of the road in front of the car, and then the track line is detected based on the mask image of the track line. Since the mask image is not easily affected by lighting conditions and background noise is filtered out in the mask image, the detection accuracy can be effectively improved by detecting the track line through the mask image, which is conducive to improving the automatic navigation capability of the line patrol car. In addition, different detection methods are used to detect the type of intersection in different line patrol modes, which is more targeted and helps to improve the detection accuracy of the intersection, thereby helping to improve the control accuracy of the automatic line patrol.

[0176] It should be understood that the size of the serial numbers of the steps in the above embodiments does not mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present application.

[0177] Corresponding to the automatic line patrol method described in the above embodiment, Fig. 9: is a structural block diagram of the automatic line patrol device provided in an embodiment of the present application. For the sake of convenience of explanation, only the parts related to the embodiment of the present application are shown.

[0178] Reference Fig. 9 , the device 9 comprises:

[0179] The acquisition unit 91 is used to acquire a first captured image of the road in front of the vehicle.

[0180] The segmentation unit 92 is used to perform image segmentation processing on the track line in the first captured image to obtain a mask image.

[0181] The detection unit 93 is used to detect the first patrol line information of the road ahead of the vehicle according to the mask image.

[0182] The control unit 94 is used to control the movement of the vehicle according to the first line patrol information.

[0183] Optionally, the first patrol line information includes the type of the road intersection; the detection unit 93 is further used for:

[0184] Detecting whether the mask image includes a road intersection;

[0185] If the mask image includes a road intersection, the intersection type of the road intersection is detected according to the mask image.

[0186] Optionally, the detection unit 93 is further used for:

[0187] In the first line patrol mode, detecting the number of changes in pixel values ​​within a preset range including the road intersection in the mask image;

[0188] The intersection type of the road intersection is determined according to the number of changes.

[0189] Optionally, the detection unit 93 is further used for:

[0190] Acquire a rectangular frame including the road intersection; wherein the area within the rectangular frame is within the preset range;

[0191] The number of times the pixel value changes at the intersection of the rectangular frame and the track line is counted to obtain the number of changes.

[0192] Optionally, the detection unit 93 is further used for:

[0193] In the second line-following mode, the mask image is divided into regions according to the track lines in the mask image to obtain at least one connected region;

[0194] Calculate the minimum circumscribed circle of the trajectory in each of the connected areas;

[0195] The intersection type of the road intersection is detected according to the number of connected areas in the mask image and the radius of each of the minimum circumscribed circles.

[0196] Optionally, the detection unit 93 is further used for:

[0197] Dividing the mask image into regions according to the track lines in the mask image to obtain at least one candidate region;

[0198] Calculating the area of ​​each candidate region;

[0199] Candidate regions whose areas are smaller than a preset value are filtered out to obtain at least one of the connected regions.

[0200] Optionally, the first line patrol information includes a line patrol start point and a line patrol end point; the detection unit 93 is further used for:

[0201] Searching for a first boundary point of the track line in a first row of pixel points of the mask image; wherein the first row of pixel points is located at the bottom of the mask image;

[0202] Searching for a second boundary point of the track line in a second row of pixel points of the mask image; wherein the second row of pixel points is above the first row of pixel points, and there is a spacing of d rows of pixels between the second row of pixel points and the first row of pixel points;

[0203] Calculate the line patrol starting point according to the first boundary point;

[0204] The line patrol endpoint is calculated according to the second boundary point.

[0205] Optionally, the control unit 94 is further configured to:

[0206] Acquire second line patrol information; wherein the second line patrol information is line patrol information detected based on at least one frame of captured image before the first captured image;

[0207] If the first line patrol information is inconsistent with the second line patrol information, the movement of the vehicle is controlled according to the second line patrol information.

[0208] It should be noted that the information interaction, execution process, etc. between the above-mentioned devices / units are based on the same concept as the method embodiment of the present application. Their specific functions and technical effects can be found in the method embodiment part and will not be repeated here.

[0209] in addition, Fig. 9 The automatic line patrol device shown may be a software unit, a hardware unit, or a combination of software and hardware, which is built into an existing terminal device, or may be integrated into the terminal device as an independent pendant, or may exist as an independent terminal device.

[0210] The technicians in the relevant field can clearly understand that for the convenience and simplicity of description, only the division of the above-mentioned functional units and modules is used as an example for illustration. In practical applications, the above-mentioned function allocation can be completed by different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above. The functional units and modules in the embodiment can be integrated in a processing unit, or each unit can exist physically separately, or two or more units can be integrated in one unit. The above-mentioned integrated unit can be implemented in the form of hardware or in the form of software functional units. In addition, the specific names of the functional units and modules are only for the convenience of distinguishing each other, and are not used to limit the scope of protection of this application. The specific working process of the units and modules in the above-mentioned system can refer to the corresponding process in the aforementioned method embodiment, which will not be repeated here.

[0211] Fig.10 Schematic diagram of the structure of the terminal device provided in the embodiment of the present application. Fig.10 As shown, the terminal device 10 of this embodiment includes: at least one processor 100 ( Fig.10 Only one is shown in the figure) a processor, a memory 101, and a computer program 102 stored in the memory 101 and executable on the at least one processor 100, and when the processor 100 executes the computer program 102, the steps in any of the above-mentioned embodiments of the automatic line patrol method are implemented.

[0212] The terminal device may be a computing device such as a desktop computer, a notebook, a PDA, or a cloud server. The terminal device may include, but is not limited to, a processor and a memory. Those skilled in the art will understand that Fig.10 It is only an example of the terminal device 10 and does not constitute a limitation on the terminal device 10. It may include more or fewer components than shown in the figure, or a combination of certain components, or different components. For example, it may also include input and output devices, network access devices, etc.

[0213] The processor 100 may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or any conventional processor, etc.

[0214] In some embodiments, the memory 101 may be an internal storage unit of the terminal device 10, such as a hard disk or memory of the terminal device 10. In other embodiments, the memory 101 may also be an external storage device of the terminal device 10, such as a plug-in hard disk, a smart memory card (Smart Media Card, SMC), a secure digital (Secure Digital, SD) card, a flash card (Flash Card), etc. equipped on the terminal device 10. Further, the memory 101 may also include both an internal storage unit of the terminal device 10 and an external storage device. The memory 101 is used to store an operating system, an application program, a boot loader (Boot Loader), data and other programs, such as the program code of the computer program, etc. The memory 101 may also be used to temporarily store data that has been output or is to be output.

[0215] An embodiment of the present application further provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps in the above-mentioned method embodiments can be implemented.

[0216] An embodiment of the present application provides a computer program product. When the computer program product is run on a terminal device, the terminal device can implement the steps in the above-mentioned method embodiments when executing the computer program product.

[0217] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the present application implements all or part of the processes in the above-mentioned embodiment method, which can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a computer-readable storage medium, and the computer program can implement the steps of the above-mentioned various method embodiments when executed by the processor. Among them, the computer program includes computer program code, and the computer program code can be in source code form, object code form, executable file or some intermediate form. The computer-readable medium may at least include: any entity or device that can carry the computer program code to the device / terminal device, a recording medium, a computer memory, a read-only memory (ROM, Read-Only Memory), a random access memory (RAM, Random Access Memory), an electric carrier signal, a telecommunication signal, and a software distribution medium. For example, a USB flash drive, a mobile hard disk, a magnetic disk or an optical disk. In some jurisdictions, according to legislation and patent practice, computer-readable media cannot be electric carrier signals and telecommunication signals.

[0218] In the above embodiments, the description of each embodiment has its own emphasis. For parts that are not described or recorded in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0219] Those of ordinary skill in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.

[0220] In the embodiments provided in the present application, it should be understood that the disclosed devices / terminal equipment and methods can be implemented in other ways. For example, the device / terminal equipment embodiments described above are only schematic. For example, the division of the modules or units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0221] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0222] The embodiments described above are only used to illustrate the technical solutions of the present application, rather than to limit them. Although the present application has been described in detail with reference to the aforementioned embodiments, a person skilled in the art should understand that the technical solutions described in the aforementioned embodiments may still be modified, or some of the technical features may be replaced by equivalents. Such modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present application, and should all be included in the protection scope of the present application.

Claims

1. An automatic line patrol method, characterized in that: include: Acquire a first captured image of the road ahead of the vehicle; Performing image segmentation processing on the track line in the first captured image to obtain a mask image; Detecting first patrol line information of the road ahead of the vehicle according to the mask image; The movement of the vehicle is controlled according to the first line patrol information.

2. The automatic line patrol method according to claim 1, characterized in that: The first line patrol information includes the intersection type of the road intersection; The detecting first patrol line information of the road ahead of the vehicle according to the mask image includes: Detecting whether the mask image includes a road intersection; If the mask image includes a road intersection, the intersection type of the road intersection is detected according to the mask image.

3. The automatic line patrol method according to claim 2, characterized in that: The detecting the intersection type of the road intersection according to the mask image comprises: In the first line patrol mode, detecting the number of changes in pixel values ​​within a preset range including the road intersection in the mask image; The intersection type of the road intersection is determined according to the number of changes.

4. The automatic line patrol method according to claim 3, characterized in that: The detecting the number of changes in pixel values ​​within a preset range including the road intersection in the mask image comprises: Acquire a rectangular frame including the road intersection; wherein the area within the rectangular frame is within the preset range; The number of times the pixel value changes at the intersection of the rectangular frame and the track line is counted to obtain the number of changes.

5. The automatic line patrol method according to claim 2, characterized in that: The detecting the intersection type of the road intersection according to the mask image comprises: In the second line-following mode, the mask image is divided into regions according to the track lines in the mask image to obtain at least one connected region; Calculate the minimum circumscribed circle of the trajectory in each of the connected areas; The intersection type of the road intersection is detected according to the number of connected areas in the mask image and the radius of each of the minimum circumscribed circles.

6. The automatic line patrol method according to claim 5, characterized in that: The step of dividing the mask image into regions according to the track lines in the mask image to obtain at least one connected region includes: Dividing the mask image into regions according to the track lines in the mask image to obtain at least one candidate region; Calculating the area of ​​each candidate region; Candidate regions whose areas are smaller than a preset value are filtered out to obtain at least one of the connected regions.

7. The automatic line patrol method according to claim 1, characterized in that: The first line patrol information includes a line patrol start point and a line patrol end point; The detecting first patrol line information of the road ahead of the vehicle according to the mask image includes: Searching for a first boundary point of the track line in a first row of pixel points of the mask image; wherein the first row of pixel points is located at the bottom of the mask image; Searching for a second boundary point of the track line in a second row of pixel points of the mask image; wherein the second row of pixel points is above the first row of pixel points, and there is a spacing of d rows of pixels between the second row of pixel points and the first row of pixel points; Calculate the line patrol starting point according to the first boundary point; The line patrol endpoint is calculated according to the second boundary point.

8. The automatic line patrol method according to any one of claims 1 to 7, characterized in that: The controlling the movement of the vehicle according to the first line patrol information includes: Acquire second line patrol information; wherein the second line patrol information is line patrol information detected based on at least one frame of captured image before the first captured image; If the first line patrol information is inconsistent with the second line patrol information, the movement of the vehicle is controlled according to the second line patrol information.

9. A terminal device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the method according to any one of claims 1 to 8 is implemented.

10. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the method according to any one of claims 1 to 8 is implemented.

Images