Autonomous mobile robot and lane line detection method using the same

Abstract

The present application provides an autonomous mobile robot, a method for lane line detection using the autonomous mobile robot, a method for lateral positioning of the robot within a lane line based on the method, an electronic device, and a non-transitory computer-readable storage medium. The method for lane line detection using the autonomous mobile robot includes: continuously acquiring input images captured by a camera, the input images including a current frame image and a historical frame image; determining the coordinate values of the endpoints of a candidate lane in a robot coordinate system based on the current frame image in the continuously acquired input images; converting the coordinate values of the endpoints of the candidate lane in the robot coordinate system into coordinate values of an odom coordinate system; matching the candidate lane in the current frame image with the candidate lane in the historical frame image based on the coordinate values of the odom coordinate system; and merging the endpoints of the successfully matched candidate lanes, and fitting the merged endpoints into the current lane segment.

Description

Technical Field

[0001] The present application relates to the field of robotics, and in particular to an autonomous mobile robot, a method for lane line detection using the autonomous mobile robot, a method for lateral positioning of the robot within a lane line based on the method, an electronic device, and a non-transitory computer-readable storage medium. Background Art

[0002] Currently, autonomous mobile robots (AMRs) are widely used in scenarios such as automated warehousing to perform tasks such as handling and picking. To complete their tasks, robots can use a variety of positioning and navigation methods, such as QR codes, lasers, and vision. Robots also often use lane markings attached to the ground for auxiliary positioning and navigation.

[0003] If lane lines attached to the ground are used for auxiliary positioning and navigation, the lane lines must first be identified and detected by the camera mounted on the robot, and then the robot is positioned and navigated based on the detected lane lines.

[0004] However, existing technologies do not accurately detect lane markings, which affects positioning and navigation results. For example, autonomous mobile robots used in automated warehousing scenarios are typically short (e.g., less than 20 or 30 cm), so the cameras on the robots are close to the ground. Due to the perspective effect, lane markings on the ground appear larger when closer and smaller when farther away. This leads to large errors in ranging from distant points. However, if only nearby points are used for detection, the number of points is small, resulting in unstable fitting and affecting the lane detection results. Summary of the Invention

[0005] In order to solve the above-mentioned problems in the prior art, the present application provides an autonomous mobile robot, a method for lane line detection using an autonomous mobile robot, a method for lateral positioning of the robot within the lane line based on the method, an electronic device, and a non-transitory computer-readable storage medium.

[0006] According to one aspect of the present application, a method for lane line detection using an autonomous mobile robot is provided, wherein the robot is equipped with a camera, and the method comprises:

[0007] Continuously acquiring input images captured by the camera, the input images including current frame images and historical frame images;

[0008] Determine the coordinate values of the endpoints of the candidate lane in the robot coordinate system according to the current frame image in the continuously acquired input images;

[0009] Convert the coordinate values of the endpoints of the candidate lane in the robot coordinate system to the coordinate values of the odom coordinate system;

[0010] Matching the candidate lanes in the current frame image with the candidate lanes in the historical frame image based on the coordinate values of the odom coordinate system; and

[0011] The endpoints of the successfully matched candidate lanes are merged and the merged endpoints are fitted as the current lane segment.

[0012] According to one embodiment, determining the coordinate values of the endpoints of the candidate lane in the robot coordinate system based on the current frame image in the continuously acquired input images includes:

[0013] Determine the coordinate values of the candidate lane fragments in the camera coordinate system according to the current frame image in the continuously acquired input images;

[0014] Determining the coordinate values of the endpoints of the candidate lane fragments in the robot coordinate system according to the coordinate values of the candidate lane fragments in the camera coordinate system; and

[0015] The candidate lane fragments of the same lane are merged into a candidate lane, and the coordinate values of the endpoints of the candidate lane in the robot coordinate system are determined.

[0016] According to one embodiment, determining the coordinate values of the candidate lane fragment in the camera coordinate system based on the current frame image in the continuously acquired input images includes:

[0017] Obtaining a mask binary image of the lane in the current frame image using a preset neural network model;

[0018] Extracting the edge image of the mask binary image using the Canny algorithm; and

[0019] The extracted edge image is processed by the Hough line detection algorithm to obtain the coordinate values of the candidate lane fragments.

[0020] According to one embodiment, determining the coordinate values of the endpoints of the candidate lane fragments in the robot coordinate system based on the coordinate values of the candidate lane fragments in the camera coordinate system includes:

[0021] Obtaining coordinate values of endpoints of the candidate lane fragment in the camera coordinate system according to the coordinate values of the candidate lane fragment in the camera coordinate system; and

[0022] Based on the transformation relationship between the camera coordinate system and the robot coordinate system, the coordinate values of the endpoints of the candidate lane fragment in the robot coordinate system are determined.

[0023] According to one embodiment, determining the coordinate values of the endpoints of the candidate lane in the robot coordinate system based on the current frame image in the continuously acquired input images further includes:

[0024] Before merging, removing candidate lane fragments whose distance from the origin in the robot coordinate system exceeds a preset first threshold; and / or

[0025] After merging, candidate lanes whose length is less than a preset second threshold are removed.

[0026] According to one embodiment, merging candidate lane fragments of the same lane into a candidate lane includes:

[0027] The direction of each candidate lane fragment and / or the distance between each two candidate lane fragments are used for fitting.

[0028] According to one embodiment, matching the candidate lane in the current frame image with the candidate lane in the historical frame image based on the coordinate values of the odom coordinate system includes:

[0029] Calculating a distance between each candidate lane in the current frame image and a plurality of candidate lanes in the historical frame image;

[0030] For each candidate lane in the current frame image, determining a minimum value among the distances;

[0031] If the minimum value is less than or equal to a predetermined third threshold, the match is successful; and

[0032] If the minimum value is greater than the third threshold, the matching fails.

[0033] According to one embodiment, calculating the distance between each candidate lane in the current frame image and each candidate lane in the historical frame image includes:

[0034] Calculating a slope difference between each candidate lane in the current frame image and each candidate lane in the historical frame image; and / or

[0035] Calculate the distance between each candidate lane in the current frame image and the endpoint of each candidate lane in the historical frame image.

[0036] According to one embodiment, matching the candidate lane in the current frame image with the candidate lane in the historical frame image based on the coordinate values of the odom coordinate system further includes:

[0037] For a candidate lane in the current frame image, if it fails to match all candidate lanes in the historical frame image, it is treated as a new candidate lane and added to the candidate lane set of the historical frame image.

[0038] According to one embodiment, matching the candidate lane in the current frame image with the candidate lane in the historical frame image based on the coordinate values of the odom coordinate system further includes:

[0039] For a candidate lane in the historical frame image, if it fails to match all candidate lanes in the current frame image, it is regarded as a candidate lane that has reached the end and is deleted from the candidate lane set of the historical frame image.

[0040] According to one embodiment, the method further comprises:

[0041] The merged endpoint is added to the candidate lane set of the historical frame image.

[0042] According to one embodiment, merging the endpoints of the successfully matched candidate lanes and fitting the merged endpoints as the current lane segment includes:

[0043] The RANSAC algorithm is used to filter out abnormal points at the endpoints of the candidate lanes.

[0044] According to one embodiment, the method further comprises:

[0045] The current lane segment is transformed from the odom coordinate system to the robot coordinate system.

[0046] According to another aspect of the present application, a method for performing lateral positioning of a robot within a lane line based on the lane line detection method described above is provided, comprising:

[0047] Based on the current lane segment, the lateral position of the robot within the lane is determined.

[0048] According to one embodiment, determining the lateral position of the robot within the lane segment based on the current lane segment includes:

[0049] Matching the current lane segment with a lane segment at a corresponding position in a preset map according to the robot's position information; and

[0050] The lateral position of the robot within the lane line in the map is determined based on the matching result.

[0051] According to one embodiment, determining the lateral position of the robot within the lane line in the map based on the matching result includes:

[0052] Determining a plurality of candidate positions based on a plurality of lane lines matched in the map; and

[0053] A lateral position of the robot is determined based on the plurality of candidate positions.

[0054] According to one embodiment, the lateral position comprises a coordinate position and orientation of the robot.

[0055] According to another aspect of the present application, there is provided an autonomous mobile robot, comprising:

[0056] body;

[0057] processor;

[0058] A driving device, driving the vehicle body to move;

[0059] a camera mounted on the vehicle body and having a lateral field of view relative to the vehicle body; and

[0060] The memory stores a computer program, and when the computer program is executed by the processor, the processor is caused to perform the method described above.

[0061] According to one embodiment, the camera includes a first camera, which is installed at the front of the vehicle body and forms a first angle with the front direction of the vehicle body.

[0062] According to one embodiment, the camera also includes a second camera, which is installed at the front position of the vehicle body and forms a second angle with the front direction of the vehicle body. The first camera and the second camera are separated by a predetermined distance, and the first angle and the second angle are symmetrical relative to the front direction of the vehicle body.

[0063] According to another aspect of the present application, an electronic device is provided, including:

[0064] processor; and

[0065] The memory stores a computer program, and when the computer program is executed by the processor, the processor is caused to perform the method described above.

[0066] According to another aspect of the present application, a non-transitory computer-readable storage medium is provided, on which computer-readable instructions are stored. When the instructions are executed by a processor, the processor is caused to perform the method described above.

[0067] Therefore, due to the setting of the camera of the autonomous mobile robot in this application, it is convenient to detect the lane lines through the monocular camera during the movement of the robot, and the use of the odom coordinate system and the historical frame matching method are more helpful to improve the detection accuracy and precision. BRIEF DESCRIPTION OF THE DRAWINGS

[0068] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0069] Figure 1 A side schematic diagram of an autonomous mobile robot according to one embodiment of the present application is shown.

[0070] Figure 2 A schematic diagram of the field of view of a camera according to an embodiment of the present application is shown.

[0071] Figure 3 A flow chart of a method for lane line detection using an autonomous mobile robot according to one embodiment of the present application is shown.

[0072] Figure 4 A flowchart is shown for determining the coordinate values of the endpoints of a candidate lane in the robot coordinate system based on the current frame image according to one embodiment of the present application.

[0073] Figure 5 A schematic diagram illustrating merging candidate lane fragments of the same lane into a candidate lane according to an embodiment of the present application is shown.

[0074] Figure 6 A flowchart is shown for determining the coordinate values of candidate lane fragments in a camera coordinate system based on a current frame image according to one embodiment of the present application.

[0075] Figure 7 A flowchart is shown for determining the coordinate values of the endpoints of the candidate lane fragments in the robot coordinate system based on the coordinate values of the candidate lane fragments in the camera coordinate system according to one embodiment of the present application.

[0076] Figure 8 A flowchart is shown for determining the coordinate values of the endpoints of a candidate lane in the robot coordinate system based on the current frame image according to another embodiment of the present application.

[0077] Figure 9 A flowchart is shown for matching candidate lanes in a current frame image with candidate lanes in a historical frame image based on coordinate values in an odom coordinate system according to one embodiment of the present application.

[0078] Figure 10 A flowchart of a method for lane line detection using an autonomous mobile robot according to another embodiment of the present application is shown.

[0079] Figure 11A flowchart of a method for lane line detection using an autonomous mobile robot according to another embodiment of the present application is shown.

[0080] Figure 12 A flowchart of determining the lateral position of a robot within a lane line based on a current lane line segment according to one embodiment of the present application is shown.

[0081] Figure 13 A flowchart is shown for determining the lateral position of a robot within a lane line in a map based on matching results according to one embodiment of the present application. DETAILED DESCRIPTION

[0082] In order to better understand the technical solutions and advantages of the present application, the contents of the present application are further described in detail below with reference to the accompanying drawings and specific embodiments. However, the specific embodiments described herein are only used to explain the present application and are not intended to limit the present application. In addition, the technical features involved in the various embodiments of the present application described below can be used in combination, except where they conflict with each other, to form other embodiments within the scope of the present application.

[0083] The content described below provides many different embodiments or examples for realizing different structures of the present application. In order to simplify the disclosure of the present application, the components and settings of specific examples are described below. Of course, they are merely examples and are not intended to limit the present application. In addition, the present application may repeat reference numbers and / or reference letters in different examples. Such repetition is for the purpose of simplicity and clarity and does not in itself indicate the relationship between the various embodiments and / or settings discussed.

[0084] Figure 1 FIG1 shows a side view of an autonomous mobile robot according to an embodiment of the present application. Figure 1 As shown, the autonomous mobile robot 100 may include a body 110, a drive unit 120, and a camera 130. The drive unit 120 may be mounted on the lower portion of the body 110 to drive the body 110 to move. The camera 130 may be mounted on the body 110 and have a lateral field of view relative to the body 110. For example, the camera 130 may be mounted facing the side of the body 110, and the angle between the camera 130 and the front of the body 110 may be set to yaw. Figure 2 FIG. 1 shows a schematic diagram of the field of view of a camera according to an embodiment of the present application. Figure 2 As shown, take yaw set to 30 degrees as an example, Figure 2 The origin O in the coordinates is the location of the camera, the X-axis is in the front of the vehicle body, and the dotted line OA is the direction of the camera. Therefore, the camera is set up to detect the lane line on the side of the vehicle body while taking into account the front of the vehicle body. For example, if the camera's field of view angle is 120 degrees, then C 11OC 12 This is the camera’s field of view FOV. The camera can be set up in this way so that the closest visible point P of the lane line is closer to the camera. Figure 2 As shown in the figure, lDist is the actual distance from the camera to the lane line, and the visible lane line of the camera is within the FOV range. Then the distance between the nearest visible point P of the lane line and the camera OP = lDist / acos(π / 2-FOV / 2-yaw), where the larger the yaw, the smaller the OP, and it satisfies 0° <FOV / 2+yaw<90°。

[0085] For another example, cameras may be provided on both sides of the front of the vehicle body 110. Figure 2 The center point O is the right camera, Cm is the midpoint between the two cameras, and OCm is 6 cm. Therefore, the blind spot directly in front of the vehicle body 110 is OCm*tan(π / 2-(FOV / 2-yaw))<12 cm, so this blind spot can be ignored. Although the autonomous mobile robot is equipped with two cameras in this example, the present application is not limited to this. Since each camera can independently complete lane line detection, the present application can also be implemented using a monocular camera.

[0086] Therefore, according to the setting of the camera on the autonomous mobile robot of this embodiment, the detection and recognition capability of the lane lines on the side of the vehicle body can be improved while taking into account the front of the vehicle body.

[0087] In addition, the autonomous mobile robot 100 may further include a processor and a memory (not shown in the figure), wherein the memory stores a computer program. When the computer program is executed by the processor, the processor executes the method described below.

[0088] Figure 3 The flowchart of the method for lane line detection using an autonomous mobile robot according to one embodiment of the present application is shown. Figure 3 As shown, the method 200 may include steps S210, S220, S230, S240, and S250. In step S210, input images captured by a camera are continuously acquired. During the movement of the autonomous mobile robot, the camera mounted thereon continuously captures images, thereby obtaining multiple frames of input images captured continuously. The input images may include a current frame image and historical frame images.

[0089] In step S220, the robot coordinate system coordinates of the endpoints of the candidate lanes are determined based on the current frame of the continuously acquired input image. The robot coordinate system coordinates of the endpoints of the candidate lanes can be determined in the current frame of the input image. Candidate lanes are portions of the image identified as possible lane markings. The specific method for determining these values is described in detail below.

[0090] In step S230, the coordinates of the endpoints of the candidate lane in the robot coordinate system are converted to the coordinates of the odom coordinate system. The odom coordinate system, also known as the odometry coordinate system, is a world-fixed coordinate system. When the autonomous mobile robot initially begins moving, the odom coordinate system coincides with the map coordinate system. However, over time, deviations develop, representing the accumulated odometry error. If the odometry is highly accurate and error-free, the map coordinate system and the odom coordinate system will always coincide. However, in reality, odometry has deviations. The pose of the mobile platform in the odom coordinate system can move arbitrarily without any bounds. This movement makes the odom coordinate system unsuitable for use as a long-term global reference. However, the robot's pose in the odom coordinate system is guaranteed to be continuous, meaning that the pose of the mobile platform in the odom coordinate system always changes smoothly without abrupt changes. In a typical setup, the odom coordinate system is calculated based on an odometry source, such as a wheel odometry, visual odometry, or an inertial measurement unit. The odom coordinate system is useful as a precise, short-term local reference, but its drift makes it unsuitable for use as a long-term reference. In the application of the present application, several adjacent frames of the continuously acquired input image are considered, so the use of the odom coordinate system for calculation can ensure that the result is accurate.

[0091] In step S240, the candidate lanes in the current frame are matched with those in the historical frames based on the coordinate values in the odom coordinate system. When the autonomous mobile robot begins operating, the camera also begins acquiring input images. When processing the first frame, no historical frames are available. However, by the time the second, third, and subsequent frames are processed, candidate lanes are already in the historical frames. At this point, the candidate lanes in the current frame can be matched with those in the historical frames in the odom coordinate system.

[0092] In step S250, the endpoints of the successfully matched candidate lanes are merged and fitted to form the current lane segment. If the match is successful, the endpoints of the candidate lanes are merged and fitted to form the current lane segment, thus completing lane detection. According to one embodiment, before merging and fitting, the RANSAC algorithm can be used to filter out outliers in the candidate lane endpoints.

[0093] Therefore, due to the setting of the camera of the autonomous mobile robot in this application, it is convenient to detect the lane lines through the monocular camera during the movement of the robot, and the use of the odom coordinate system and the historical frame matching method are more helpful to improve the detection accuracy and precision.

[0094] Figure 4FIG1 shows a flow chart of determining the coordinate values of the endpoints of the candidate lane in the robot coordinate system according to the current frame image according to one embodiment of the present application. Figure 4 As shown, step S220 may include sub-steps S221, S222, and S223. In sub-step S221, the coordinate values of the candidate lane fragment in the camera coordinate system are determined based on the current frame image in the continuously acquired input image. When each frame image is acquired, the coordinate values first obtained are those in the camera coordinate system. The specific method for determining the coordinate values of the candidate lane fragment in the camera coordinate system will be described in detail below.

[0095] In sub-step S222, the coordinates of the endpoints of the candidate lane fragments in the robot coordinate system are determined based on the coordinates of the candidate lane fragments in the camera coordinate system. After determining the coordinates of the candidate lane fragments in the camera coordinate system in sub-step S221, these coordinates are used in sub-step S222 to determine the coordinates of the endpoints of the candidate lane fragments in the robot coordinate system. For example, this can be defined as laneSeg3dList. The specific implementation of the transformation from the camera coordinate system to the robot coordinate system will be further described below.

[0096] In sub-step S223, the candidate lane fragments of the same lane are merged into a candidate lane, and the coordinate values of the endpoints of the candidate lanes in the robot coordinate system are determined. For the multiple candidate lane fragments whose endpoint coordinate values are determined in sub-step S222, the coordinate values can be used to determine which candidate lane fragments belong to the same lane for merging, and a lane3dList is obtained, such as Figure 5 shown. Figure 5 This diagram illustrates how lane candidate fragments from the same lane are merged into a candidate lane according to one embodiment of the present application. After merging, the coordinates of the endpoints of the candidate lane in the robot coordinate system are determined. Subsequently, in step S230, lane3dList is transformed into the odom coordinate system to obtain lane3dListInOdom.

[0097] According to an embodiment of the present application, in sub-step S223, when merging candidate lane fragments of the same lane into candidate lanes, fitting can be performed using the direction of each candidate lane fragment and / or the distance between every two candidate lane fragments.

[0098] Therefore, the coordinate values of the endpoints of the candidate lanes in the robot coordinate system can be determined in the current frame image through coordinate transformation and merging of lane fragments.

[0099] Figure 6 FIG1 shows a flow chart of determining the coordinate values of candidate lane fragments in the camera coordinate system according to the current frame image according to one embodiment of the present application. Figure 6As shown, the above sub-step S221 may include sub-steps S221A, S221B, and S221C. In sub-step S221A, a preset neural network model is used to obtain a binary mask image of the lane in the current frame image. The neural network model can be a pre-trained neural network model, so that it can be directly used to obtain the binary mask image.

[0100] In sub-step S221B, the edge image of the mask binary image is extracted using the Canny algorithm. Subsequently, in sub-step S221C, the extracted edge image is processed using the Hough line detection algorithm to obtain the coordinate values of the candidate lane fragments in the camera coordinate system.

[0101] Figure 7 The flowchart of determining the coordinate values of the endpoints of the candidate lane fragments in the robot coordinate system according to the coordinate values of the candidate lane fragments in the camera coordinate system according to one embodiment of the present application is shown. Figure 7 As shown, the above-mentioned sub-step S222 may include sub-steps S222A and S222B. In sub-step S222A, based on the coordinate values of the candidate lane fragments in the camera coordinate system, the coordinate values of the endpoints of the candidate lane fragments in the camera coordinate system are obtained. Subsequently, in sub-step S222B, based on the transformation relationship between the camera coordinate system and the robot coordinate system, the coordinate values laneSeg3dList of the endpoints of the candidate lane fragments in the robot coordinate system are determined. According to the present application, since the camera is pre-installed on the body of the autonomous mobile robot and its parameters are known, the transformation relationship between the camera coordinate system and the robot coordinate system can be preset or known. Therefore, the coordinate values of the endpoints of the lane fragments in the camera coordinate system obtained in sub-step S222A can be transformed into coordinate values of the robot coordinate system.

[0102] Figure 8 FIG1 shows a flow chart of determining the coordinate values of the endpoints of the candidate lane in the robot coordinate system according to the current frame image according to another embodiment of the present application. Figure 8 As shown, in addition to sub-steps S221, S222, and S223, the above step S220 may further include sub-steps S224 and / or S225.

[0103] In sub-step S224, before merging in sub-step S223, candidate lane fragments whose distance from the origin in the robot coordinate system exceeds a preset first threshold are removed. According to this sub-step, if a detected candidate lane fragment is too far away (i.e., exceeds the preset first threshold), to avoid large ranging errors at distant points, the candidate lane fragment can be deleted to achieve noise reduction. In other words, the points in laneSeg3dList that are too far away are removed, resulting in laneSeg3dListFiltered.

[0104] In sub-step S225, after the merging in sub-step S223, candidate lanes whose lengths are less than a preset second threshold are removed. According to this sub-step, if the length of a candidate lane after merging is too short (i.e., less than the preset second threshold), it can also be considered as noise and deleted.

[0105] Through the above sub-steps S224 and S225, noises with too long distances and too short lengths can be removed, thereby improving detection accuracy and precision.

[0106] Figure 9 The flowchart of matching the candidate lanes in the current frame image with the candidate lanes in the historical frame image based on the coordinate values of the odom coordinate system according to one embodiment of the present application is shown. Figure 9 As shown, the above step S240 may include sub-steps S241, S242, S243, and S244. In sub-step S241, the distance between each candidate lane in the current frame image and multiple candidate lanes in the historical frame images is calculated.

[0107] When calculating the distance between the candidate lanes in the current frame and the candidate lanes in the historical frame, the slope difference between each candidate lane in the current frame image and each candidate lane in the historical frame image can be calculated as the distance between the two. The distance between the endpoints of each candidate lane in the current frame image and each candidate lane in the historical frame image can also be calculated as the distance between the two.

[0108] In sub-step S242, for each candidate lane in the current frame image, determine the minimum value matchDistmin of the distance between it and all candidate lanes in the historical frame image. That is, traverse the candidate lanes in the historical frame image and determine the one with the closest distance. In addition, a third threshold is preset in advance. If the minimum value matchDistmin is less than or equal to the third threshold, the match is successful (sub-step S243); if the minimum value matchDistmin is greater than the third threshold, the match fails (sub-step S244). Taking the j-th candidate lane in the current frame lane3dListInOdom and the i-th candidate lane in the historical frame histList as an example, match the distance between histList[i] and lane3dListInOdom[j]. If it is less than or equal to the preset third threshold, it is considered a match and recorded as histList[i].matchDist. If multiple j match i, the lane3dListInOdom[j] with the smallest matchDist can be taken, and histList[i] is spliced to obtain the updated histList[i].

[0109] According to one embodiment, if a candidate lane in the current frame image fails to match all candidate lanes in the historical frame image, it indicates that the candidate lane is newly added to the current frame image. In this case, it can be added to the candidate lane set of the historical frame image (because when the next frame is processed, the current frame will become the historical frame). For example, the histList can be traversed to obtain an updated histList. Thus, during the movement of the autonomous mobile robot, newly appearing lanes captured by the camera can be added to the candidate lane set of the historical frame image for calculation and processing of subsequent frames.

[0110] According to another embodiment, for a candidate lane in a historical frame image, if it fails to match all candidate lanes in the current frame image, it means that the candidate lane has disappeared in the current frame (for example, the lane has reached the end in the previous frame), and it can be deleted from the candidate lane set of the historical frame image. For example, for a candidate lane in a historical frame, if no matching candidate lane can be found by traversing lane3dListInOdom, it can be considered that the lane has reached the end and can be deleted from histList. Therefore, during the movement of the autonomous mobile robot, when a lane disappears from the camera's field of view, the lane can be deleted from the candidate lane set of the historical frame to avoid affecting subsequent calculations and processing.

[0111] Figure 10 FIG1 shows a flow chart of a method for lane line detection using an autonomous mobile robot according to another embodiment of the present application. Figure 10 As shown, in addition to steps S210 to S250, method 200 may further include step S260. In step S260, the merged endpoint is added to the candidate lane set of the historical frame image. Thus, as the autonomous mobile robot continues to move and the camera continues to capture images, the current frame becomes a historical frame during the processing of the next image. Therefore, the merged endpoint in the current frame image can be added to the candidate lane set of the historical frame image for calculation and processing of subsequent frames.

[0112] Figure 11 FIG. 1 is a flow chart showing a method for lane line detection using an autonomous mobile robot according to another embodiment of the present application. Figure 11 As shown, in addition to steps S210 to S250, the method 200 may further include step S270. In step S270, the current lane segment is transformed from the odom coordinate system to the robot coordinate system. This facilitates the subsequent positioning process to perform lateral positioning of the autonomous mobile robot.

[0113] As described above, for histList[i], the Ransac algorithm is used to filter out outliers at the endpoints of the candidate lanes, and then fit to obtain histList[i].fitline. Then, in step S270, histList[i].fitline in the odom coordinate system is transformed into histList[i].fitlineInBase in the robot coordinate system. Finally, the fitlineInBase of all candidate lanes is obtained and used in the subsequent positioning process to perform lateral positioning of the autonomous mobile robot.

[0114] According to the present application, based on the lane line detection results of the above-mentioned method 200, the robot can be laterally positioned within the lane line. The method for laterally positioning the robot within the lane line may include step S310. In step S310, the robot's lateral position within the lane line can be determined using the current lane line segment obtained in method 200. For example, a number of lane line images taken at different angles and / or different distances from the lane line can be preset in the robot system. Subsequently, the current lane line segment obtained in method 200 is matched with the preset lane line image, and the robot's lateral coordinates and orientation are obtained based on the matching results.

[0115] If you consider different lane marking types, you can also preset different images based on the different lane marking types. For example, lane markings may include single solid lines, single dashed lines, double solid lines, and other different types. Then, you can preset different images based on these different lane marking types to facilitate matching.

[0116] Figure 12 FIG1 shows a flow chart of determining the lateral position of a robot within a lane line based on the current lane line segment according to one embodiment of the present application. Figure 12 As shown, step S310 may include sub-steps S311 and S312. In sub-step S311, based on the robot's position information, the current lane segment is matched with the lane segment at the corresponding location in the preset map. The specific matching method can be referred to the method of sub-steps S241 to S244 above and will not be repeated here.

[0117] Then, in sub-step S312 , the lateral position of the robot within the lane line in the map is determined based on the matching result.

[0118] Therefore, the lane line detection result of the above method 200 can be used to perform lateral positioning of the robot within the lane line.

[0119] Figure 13 FIG1 shows a flow chart of determining the lateral position of a robot within a lane line in a map based on the matching results according to one embodiment of the present application. Figure 13As shown, sub-step S312 may include sub-steps S312A and S312B. In sub-step S312A, multiple candidate positions are determined based on multiple lane lines matched in the map. Subsequently, in sub-step S312B, the robot's lateral position is determined based on the multiple candidate positions. As described above, the autonomous mobile robot may be equipped with two or more cameras, each of which can independently capture input images and use the captured input images to detect lane lines. In other words, using two or more cameras, multiple current lane lines can be detected (for example, two cameras, one on the left and one on the right, can detect lane lines on the left and right sides of the vehicle body, respectively), and these lane lines can be used to laterally locate the robot. Therefore, there may be multiple candidate positions. That is, the lateral position of the robot obtained based on different lane lines detected using the input images captured by each camera may deviate. The robot's lateral position can be determined based on the multiple candidate positions using any known technique, for example, by averaging the multiple candidate positions.

[0120] According to one embodiment, the lateral position may include the coordinate position (x, y) and orientation θ of the robot.

[0121] According to another aspect of the present application, an electronic device is provided, which includes a processor and a memory, wherein the memory stores a computer program, and when the computer program is executed by the processor, the processor executes the method described in any one of the above embodiments.

[0122] According to another aspect of the present application, a non-transitory computer-readable storage medium is provided, on which computer-readable instructions are stored. When the instructions are executed by a processor, the processor can execute the method described in any of the above embodiments.

[0123] In particular, according to an embodiment of the present disclosure, the process described above with reference to the flowchart can be implemented as a computer software program. For example, an embodiment of the present disclosure includes a computer program product, which includes a computer program carried on a computer-readable medium, and the computer program includes a program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from a network through its communication components, and / or installed from a removable medium. When the computer program is executed by a central processing unit (CPU), the above-mentioned functions defined in the method of the present application are executed. It should be noted that the computer-readable medium of the present application can be a computer-readable signal medium or a computer-readable storage medium or any combination of the above two. The computer-readable storage medium can be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to, an electrical connection having one or more conductors, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination thereof. In this application, a computer-readable storage medium may be any tangible medium that contains or stores a program that can be used by or in conjunction with an instruction execution system, apparatus, or device. In this application, a computer-readable signal medium may include a data signal propagated in baseband or as part of a carrier wave, which carries computer-readable program code. Such a propagated data signal may take a variety of forms, including, but not limited to, electromagnetic signals, optical signals, or any suitable combination thereof. A computer-readable signal medium may also be any computer-readable medium other than a computer-readable storage medium that can transmit, propagate, or transfer a program for use by or in conjunction with an instruction execution system, apparatus, or device. The program code embodied on a computer-readable medium may be transmitted using any suitable medium, including, but not limited to, wireless, wire, optical cable, RF, etc., or any suitable combination thereof.

[0124] Computer program code for performing the operations of the present application may be written in one or more programming languages, or a combination thereof, including object-oriented programming languages such as Java, Smalltalk, C++, and conventional procedural programming languages such as "C" or similar programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, as a stand-alone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving a remote computer, the remote computer may be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., through the Internet using an Internet service provider).

[0125] The flow charts and block diagrams in the accompanying drawings illustrate the possible architecture, functions and operations of the systems, methods and computer program products according to various embodiments of the present application. In this regard, each box in the flow chart or block diagram can represent a module, program segment or a part of code, and the module, program segment or a part of code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a different order than that marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram and / or flow chart, and the combination of the boxes in the block diagram and / or flow chart can be implemented by a dedicated hardware-based system that performs the specified function or operation, or can be implemented by a combination of dedicated hardware and computer instructions.

[0126] The units involved in the embodiments described in this application may be implemented in software or hardware, and may also be provided in a processor.

[0127] As another aspect, the present application further provides a computer-readable medium, which may be included in the apparatus described in the above embodiments, or may exist independently and not be incorporated into the apparatus. The computer-readable medium carries one or more programs, and when the one or more programs are executed by the apparatus, the apparatus performs the method described above.

[0128] In the above embodiments, the description of each embodiment has its own emphasis. For parts not described in detail in a particular embodiment, please refer to the relevant description of other embodiments. The technical features of the above embodiments can be combined in any way. To keep the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0129] The embodiments of the present application are described in detail above. Specific examples are used herein to illustrate the principles and implementation methods of the present application. The description of the above embodiments is only intended to help understand the method and core ideas of the present application. At the same time, changes or modifications made by those skilled in the art based on the ideas of the present application, the specific implementation methods, and the scope of application of the present application, all fall within the scope of protection of the present application. In summary, the contents of this specification should not be construed as limiting the present application.

Claims

1. A method for lane line detection using an autonomous mobile robot, wherein the robot is equipped with a camera, the method comprising: Continuously acquiring input images captured by the camera, the input images including current frame images and historical frame images; Determine the coordinate values of the endpoints of the candidate lane in the robot coordinate system according to the current frame image in the continuously acquired input images; Convert the coordinate values of the endpoints of the candidate lane in the robot coordinate system to the coordinate values of the odom coordinate system; Matching the candidate lanes in the current frame image with the candidate lanes in the historical frame image based on the coordinate values of the odom coordinate system; as well as The endpoints of the successfully matched candidate lanes are merged and the merged endpoints are fitted as the current lane segment.

2. The method according to claim 1, wherein According to the current frame image in the continuously acquired input images, the coordinate values of the endpoints of the candidate lane in the robot coordinate system are determined including: Determine the coordinate values of the candidate lane fragments in the camera coordinate system according to the current frame image in the continuously acquired input images; Determining the coordinate values of the endpoints of the candidate lane fragments in the robot coordinate system according to the coordinate values of the candidate lane fragments in the camera coordinate system; and The candidate lane fragments of the same lane are merged into a candidate lane, and the coordinate values of the endpoints of the candidate lane in the robot coordinate system are determined.

3. The method of claim 2, wherein determining the coordinate values of the candidate lane fragment in the camera coordinate system based on the current frame image in the continuously acquired input images comprises: Obtaining a mask binary image of the lane in the current frame image using a preset neural network model; Extracting the edge image of the mask binary image using the Canny algorithm; as well as The extracted edge image is processed by the Hough line detection algorithm to obtain the coordinate values of the candidate lane fragments.

4. The method of claim 2 , wherein determining the coordinate values of the endpoints of the candidate lane fragment in the robot coordinate system based on the coordinate values of the candidate lane fragment in the camera coordinate system comprises: Obtaining coordinate values of endpoints of the candidate lane fragment in the camera coordinate system according to the coordinate values of the candidate lane fragment in the camera coordinate system; as well as Based on the transformation relationship between the camera coordinate system and the robot coordinate system, the coordinate values of the endpoints of the candidate lane fragment in the robot coordinate system are determined.

5. The method according to claim 2, wherein: Determining the coordinate values of the endpoints of the candidate lane in the robot coordinate system based on the current frame image in the continuously acquired input images also includes: Before merging, removing candidate lane fragments whose distance from the origin in the robot coordinate system exceeds a preset first threshold; and / or After merging, candidate lanes whose length is less than a preset second threshold are removed.

6. The method of claim 2, wherein: Merging lane candidate fragments of the same lane into a candidate lane includes: The direction of each candidate lane fragment and / or the distance between each two candidate lane fragments are used for fitting.

7. The method of claim 1, wherein: Matching the candidate lane in the current frame image with the candidate lane in the historical frame image based on the coordinate values of the odom coordinate system includes: Calculating a distance between each candidate lane in the current frame image and a plurality of candidate lanes in the historical frame image; For each candidate lane in the current frame image, determining a minimum value among the distances; If the minimum value is less than or equal to a predetermined third threshold, the match is successful; and If the minimum value is greater than the third threshold, the matching fails.

8. The method of claim 7, wherein: Calculating the distance between each candidate lane in the current frame image and each candidate lane in the historical frame image includes: Calculating a slope difference between each candidate lane in the current frame image and each candidate lane in the historical frame image; and / or Calculate the distance between each candidate lane in the current frame image and the endpoint of each candidate lane in the historical frame image.

9. The method of claim 7, wherein matching the candidate lanes in the current frame image with the candidate lanes in the historical frame image based on the coordinate values of the ODOM coordinate system further comprises: For a candidate lane in the current frame image, if it fails to match all candidate lanes in the historical frame image, it is treated as a new candidate lane and added to the candidate lane set of the historical frame image.

10. The method of claim 7, wherein matching the candidate lanes in the current frame image with the candidate lanes in the historical frame image based on the coordinate values of the ODOM coordinate system further comprises: For a candidate lane in the historical frame image, if it fails to match all candidate lanes in the current frame image, it is regarded as a candidate lane that has reached the end and is deleted from the candidate lane set of the historical frame image.

11. The method of claim 1 , further comprising: The merged endpoint is added to the candidate lane set of the historical frame image.

12. The method of claim 1, wherein: Merging the endpoints of the successfully matched candidate lanes and fitting the merged endpoints into the current lane segment includes: The RANSAC algorithm is used to filter out abnormal points at the endpoints of the candidate lanes.

13. The method of claim 1 , further comprising: The current lane segment is transformed from the odom coordinate system to the robot coordinate system.

14. A method for lateral positioning of a robot within a lane based on the lane detection method according to any one of claims 1 to 13, comprising: Based on the current lane segment, the lateral position of the robot within the lane is determined.

15. The method for lateral positioning according to claim 14, wherein determining the lateral position of the robot within the lane line based on the current lane line segment comprises: Matching the current lane segment with a lane segment at a corresponding position in a preset map according to the position information of the robot; as well as The lateral position of the robot within the lane line in the map is determined based on the matching result.

16. The method for lateral positioning according to claim 15, wherein: Determining the lateral position of the robot within the lane line in the map according to the matching result includes: Determining a plurality of candidate positions based on a plurality of lane lines matched in the map; and A lateral position of the robot is determined based on the plurality of candidate positions.

17. The method for lateral positioning according to claim 15 or 16, wherein: The lateral position includes the coordinate position and orientation of the robot.

18. An autonomous mobile robot comprising: body; processor; A driving device, driving the vehicle body to move; a camera mounted on the vehicle body and having a lateral field of view relative to the vehicle body; as well as A memory storing a computer program, which, when executed by the processor, causes the processor to perform the method according to any one of claims 1 to 17.

19. The autonomous mobile robot according to claim 18, wherein the camera comprises a first camera installed at the front of the vehicle body and forming a first angle with the front direction of the vehicle body.

20. The autonomous mobile robot as claimed in claim 19, wherein the camera further comprises a second camera, which is installed at the front position of the vehicle body and forms a second angle with the front direction of the vehicle body, the first camera and the second camera are separated by a predetermined distance, and the first angle and the second angle are symmetrical relative to the front direction of the vehicle body.

21. An electronic device comprising: processor; as well as A memory storing a computer program, which, when executed by the processor, causes the processor to perform the method according to any one of claims 1 to 17.

22. A non-transitory computer-readable storage medium having computer-readable instructions stored thereon, which, when executed by a processor, causes the processor to perform the method according to any one of claims 1 to 17.

Images