Door frame positioning method based on image recognition and robot

Abstract

The application discloses a door frame positioning method and robot based on image recognition, and the door frame positioning method comprises the following steps: the robot walks in an indoor working area according to a preset planning path, collects environment images in the walking process, connects a reference door beam line from the collected environment images, determines direction information of a position of a to-be-measured door frame where the reference door beam line is located relative to the robot, and records the direction information; after the robot walks through the indoor working area according to the preset planning path, the robot selects a to-be-scored position as a matching center to perform template matching, and when the matching result of the direction information of the position of the to-be-measured door frame where the reference door beam line is located relative to the position walked through by the robot and the matching result of angle information of the corresponding reference door beam line both satisfy a preset angle matching condition, the robot determines that the to-be-scored position is a center point of a door beam of the door frame, so that the door frame positioning is realized.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of machine vision, and in particular to a door frame positioning method based on image recognition and a robot. BACKGROUND

[0002] For a sweeping robot, zoned cleaning is a very important function. Although a home environment is a highly structured environment, it is often difficult to meet the demand for dividing the area in terms of actual rooms by relying only on the environment contour to divide the cleaning area.

[0003] The position of a door in a home environment is relatively constant. The door is a passageway connecting one room to another room, and the width and shape of the door have a unified standard. The prior art determines the key boundary connecting two room areas in a home environment by identifying the position of the door, but it collects and combines multiple dimensions of door line features under the condition that the robot remains static (if the sample selection is not rich enough, it will not be able to adapt to various door recognitions), and it also collects and establishes multiple pairs of matching quantities for recognition processing by means of binocular vision, which is easily affected by the opening and closing state of the door leaf. Thus, the design cost of the sensor and the computational complexity are high, and the real-time performance of the robot in the home environment for cross-room navigation work is not satisfactory. SUMMARY

[0004] To solve the above technical problems, the present application realizes the recognition and positioning of a door only by the angle orientation and positional relationship of a straight line segment identified by image recognition, including the recognition and positioning of a door jamb line, a door beam line, and a door frame. The specific technical solutions are as follows:

[0005] A door frame positioning method based on image recognition, the execution subject of the door frame positioning method being a robot, the robot being equipped with a camera with a lens set upward, the center point of an environment image collected by the camera representing the position directly above the robot; the door frame positioning method comprising: step S1, the robot walking in an indoor working area according to a preset planning path and collecting environment images in the process of walking; step S2, the robot connecting a reference door beam line from the collected environment images, then determining the direction information of the position of a to-be-measured door frame where the reference door beam line is located relative to the robot, and recording the angle information of the reference door beam line and the direction information of the position of the to-be-measured door frame where the reference door beam line is located relative to the robot; step S3, after the robot walks through the indoor working area according to the preset planning path, the robot selects a to-be-scored position as a matching center for template matching, and when the matching result of the direction information of the position of the to-be-measured door frame where the reference door beam line is located relative to the position walked through by the robot and the matching result of the angle information of the corresponding reference door beam line both satisfy a preset angle matching condition, the robot determines that the to-be-scored position is the center point of the door beam of the door frame, so as to realize the positioning of the door frame.

[0006] Further, in the step S2, during the robot walking, the robot extracts two reference door column lines from the current collected environment image, connects a reference door beam line based on the two reference door column lines, and determines the direction information of the to-be-measured door frame relative to the current position of the robot according to the positional relationship between the center point of the reference door beam line and the center point of the current collected environment image, and records the direction information of the to-be-measured door frame relative to the current position of the robot and the angle information of the reference door beam line in a grid corresponding to the current position of the robot in the grid map constructed by the robot in the process of walking according to the preset planned path.

[0007] Further, the step S3 further includes: after the robot walks through the indoor working area according to the preset planned path, recording the direction information of the to-be-measured door frame relative to the position of the robot corresponding to the grid in the grid map constructed by the robot, and the angle information of the reference door beam line in the corresponding grid; the direction information of the to-be-measured door frame relative to the position walked through by the robot includes the direction information detected by the robot at different positions on different sides of the same reference door beam line, so that the robot detects the to-be-measured door frame of the same reference door beam line from multiple different perspectives.

[0008] Further, in the step S3, the robot traverses each grid in the grid map using a door orientation template to obtain a pair of angle matching results of the neighborhood of the grid, wherein the center of the door orientation template is the matching center; in the process of traversal, the center of the door orientation template is configured to cover each grid in the grid map; wherein the pair of angle matching results of the neighborhood of the grid covered by the center of the door orientation template includes the matching result of the direction information of the to-be-measured door frame relative to the position walked through by the robot and the matching result of the angle information of the same reference door beam line; the position corresponding to the grid covered by the center of the door orientation template is the to-be-scored position; then a door positioning evaluation score is given to each grid in the grid map by counting the pair of angle matching results; if it is detected that the door positioning evaluation score given to a grid is greater than a preset score threshold, it is determined that the grid is the center of the grid area occupied by the door frame.

[0009] Further, the step S3 specifically includes: step S31, controlling the pre-set door orientation template to start traversing the grids in the grid map;

[0010] Step S32, whenever the center of the door orientation template covers one grid of the grid map, the coverage area of the door orientation template in the grid map is determined, and the area in the coverage area except the grid covered by the center is marked as a matching area; Step S33, in the matching area, whenever it is judged that the angle information of the reference door beam line recorded in one grid is in the reference door beam angle range, and it is judged that the orientation angle formed by the door frame where the same reference door beam line is located relative to the position corresponding to the grid is in the corresponding reference deviation angle range, the door positioning evaluation score of the grid covered by the center is counted by one, and the robot is determined to assign a door positioning evaluation score to each grid in the grid map by counting each angle matching result pair until each grid in the matching area is traversed; wherein, the direction information of the position corresponding to the grid relative to the door frame where the same reference door beam line is located includes the orientation angle formed by the door frame where the same reference door beam line is located relative to the position corresponding to the grid; Step S34, whether the door positioning evaluation score of the grid covered by the center in step S33 is greater than the preset score threshold, yes, it is determined that the position corresponding to the grid covered by the center in step S33 is the center point of the door beam of the door frame, and it is also determined that the matching result of the direction information of the position relative to the position walked by the robot of the door frame where the same reference door beam line is located and the matching result of the angle information of the corresponding reference door beam line both satisfy the preset angle matching condition; otherwise, the door orientation template is controlled to translate in the grid map, so that the center of the door orientation template traverses the next grid, and then step S32 is executed until the center of the door orientation template traverses each grid of the grid map.

[0011] Further, the step S3 further comprises: after the robot determines that the door frame covers the position corresponding to the grid covered by the center in step S33, the opposite two extension directions formed between the two endpoints of the same reference door beam line are determined, the position corresponding to the grid covered by the center in step S33 is taken as the starting point, and the reference straight line segment is obtained by extending a preset extension length in the two extension directions respectively, and it is determined that the door frame exists at the reference straight line segment, wherein, the position corresponding to the grid covered by the center in step S33 is the center point of the door beam of the door frame; the grid covered by the center in step S33 is the grid occupied by the center point of the reference door beam line existing at the grid; there are at least one room area on each side of the door frame which is passable in the indoor working area, and the reference straight line segment becomes a boundary line between different room areas.

[0012] Further, each grid in the to-be-matched region is configured with a corresponding reference deflection angle range, and the angle matching result of the neighborhood of the grid covered by the center of the door orientation template includes whether the angle of the same reference door beam line recorded in each grid in the to-be-matched region is within the reference door beam angle range, and whether the azimuth angle formed by the to-be-detected door frame in which the same reference door beam line is located relative to each grid in the to-be-matched region is within the corresponding reference deflection angle range; wherein one reference door beam line corresponds to an angle information in the indoor working area, and the angle information is recorded in the grid corresponding to the position where the robot walks; one reference door beam line is matched using one door orientation template, and one door orientation template corresponds to one reference door beam angle range.

[0013] Further, among the direction information of the to-be-detected door frame in which the same reference door beam line is located relative to the position where the robot walks, at least the direction information of the same reference door beam line located in front of the robot relative to the position of the robot, the direction information of the same reference door beam line located behind the robot relative to the position of the robot, the direction information of the same reference door beam line located directly above the robot relative to the position of the robot, the direction information of the same reference door beam line located on the left side of the robot relative to the position of the robot, and / or the direction information of the same reference door beam line located on the right side of the robot relative to the position of the robot exist.

[0014] Further, in the step S1, in the environment image collected by the robot, the robot first extracts a straight line segment through a pre-set image recognition algorithm; then the robot marks each straight line segment intersecting the center point of the environment image as a reference door column line; wherein the reference door column line is used to represent a line segment perpendicular to the ground surface of the indoor working area; the robot selects two reference door column lines, and then sets an end point close to the center point of the environment image as a corner point in each selected reference door column line, and then connects the two corner points to form a reference door beam line, so that the two selected reference door column lines and the connected reference door beam line form a to-be-detected door frame.

[0015] Further, after the robot sets the corner point in each selected reference door column line, the included angle between the direction from one corner point to the other corner point and the coordinate axis of the image coordinate system, or the angle converted to the world coordinate system, is set as the angle information of the reference door beam line, so as to convert the extension direction of the reference door beam line in the indoor working area.

[0016] Further, the method for determining the direction information of the to-be-detected door frame relative to the current position of the robot according to the positional relationship between the center point of the reference door beam line and the center point of the currently collected environment image comprises: the robot uses the center point of the environment image collected at the current position to represent the current position of the robot; then the robot sets the direction of the center point of the currently collected environment image to the center point of the reference door beam line as the distribution direction of the to-be-detected door frame relative to the current position of the robot; then sets the included angle between the distribution direction and a coordinate axis of the image coordinate system as the azimuth angle of the to-be-detected door frame relative to the current position of the robot, or sets the included angle between the distribution direction and the reference door beam line as the azimuth angle of the to-be-detected door frame relative to the current position of the robot; and finally sets the distribution direction of the to-be-detected door frame relative to the current position of the robot and / or the azimuth angle of the to-be-detected door frame relative to the current position of the robot as the direction information of the to-be-detected door frame relative to the current position of the robot, thereby forming the direction information of the to-be-detected door frame currently detected by the robot.

[0017] A robot is provided with a camera with a lens pointing upwards, and the center point of an environment image collected by the camera is used to represent the center of the body of the robot, and the robot is configured to perform the door frame positioning method.

[0018] The technical effect of the present application is that the present application determines the reference door column line and the reference door beam line as candidate line segments by extracting straight line segments and constructing geometric connection relationships between different straight line segments, and then obtains the reference door beam line collected by the robot at different positions, the corresponding geometric angle features and the direction relationship relative to the robot based on the passability of the door. After the robot walks through the indoor working area, the geometric angle features (corresponding to the angle information of the reference door beam line) and the direction relationship relative to the robot (corresponding to the direction information of the to-be-detected door frame relative to the position of the robot) of the reference door beam line recorded at each position are matched one by one using the template information, wherein the reference door beam line and the two reference door column lines connected thereto constitute a to-be-detected door frame as a candidate door frame. Then, the matching result information at each position can be compared, and the position with the highest similarity corresponding to the matching result information is regarded as the position where the real door frame exists. Therefore, the door positioning in a home environment is effectively realized by a relatively low-power algorithm, and no additional markers need to be added to the door. As long as the two side frame lines and the door beam line at the top of the door appear in the field of view of the camera of the present application, the positioning of the door can be realized.

[0019] The door frame recognition accuracy is improved, and square columns, door leaves, and rotatable door leaves (components provided in the door frame and capable of being opened and closed left and right) in the door frame to be detected are avoided from being mistaken as real door frames. The door frame can be effectively recognized in various door recognition scenes such as closed, half-open, and fully open states, and whether the door frame exists at the corresponding position in the same grid map is recognized. Of course, if the reference door beam line to be matched changes (corresponding to another door frame in the same indoor working area to be matched and positioned), another template information is replaced to perform corresponding matching, so as to adapt to the change of the geometric angle characteristics of each reference door beam line participating in matching and the direction relationship relative to the robot, and then the door frame of various door beam extension directions is recognized at the corresponding position. On this basis, the present application uses the position information of the existing door frame to determine the corresponding boundary line or multiple discrete boundary positions, so as to divide the room area close to the actual environment in the indoor working area, improve the rationality of the robot partition, and realize the real sense of room cleaning. BRIEF DESCRIPTION OF DRAWINGS

[0020] Figure 1 is a flowchart of a door frame positioning method based on image recognition disclosed by an embodiment of the present application.

[0021] Figure 2 is a schematic diagram of an image collected by a robot in front of a door frame and marked with a reference door column line A1B1 and a reference door beam line B1C1, wherein a camera of the robot is set to face the ceiling. DETAILED DESCRIPTION

[0022] The technical solutions in the embodiments of the present application will be described in detail below with reference to the accompanying drawings. In order to further illustrate the embodiments, the present application provides drawings which are part of the disclosure of the present application, mainly used to illustrate the embodiments, and can be used to explain the operation principle of the embodiments in conjunction with the related description of the specification.

[0023] The embodiment of the application discloses a door frame positioning method based on image recognition, and an execution subject of the door frame positioning method is a robot. The robot can move autonomously in an indoor working area. When the robot is a sweeping robot with a circular chassis, the robot can be planned to perform coverage cleaning in at least two room areas in a home environment. The robot is provided with a camera with an upwardly arranged lens. The lens of the camera can be arranged obliquely upward on the top surface of the robot. A center point of an environment image collected by the camera represents a position directly above the robot. In some embodiments, a projection position of the center point of the environment image collected by the camera in a walking plane of the robot is used to represent the center of the body of the robot. Therefore, the center point of the environment image collected by the camera can be directly used to represent the current position of the robot in an image coordinate system. Of course, the specific coordinate position of the robot in the image coordinate system can be converted into a world coordinate system. Referring to Figure 1 It can be known that the door frame positioning method comprises the following steps.

[0024] In step S1, the robot walks in the indoor working area according to a preset planning path and collects environment images in the walking process. Then, step S2 is performed. In step S1, the robot walks on the ground in the indoor working area. The camera of the robot can be installed towards the ceiling and can be configured to be consistent with the orientation of the camera in the vertical ground column line and the side wall line. Therefore, the camera of the robot always collects images of the ceiling in the field of view without being blocked by obstacles. When the robot walks in the indoor working area according to the preset detour path, the robot can pass through the door and be inserted between at least two room areas and traverse in the two room areas, respectively. Preferably, the preset detour path is an arch-shaped path. The robot can be a cleaning robot. According to the cleaning task instruction, the cleaning robot performs arch-shaped cleaning in the corresponding room area. The cleaning robot takes pictures while cleaning, especially at different positions in different room areas to collect images of the ceiling and the door frame at different angles. The viewing angle of the camera in the embodiment can cover at least the area between the door frame or the door beam and the ceiling.

[0025] Step S2, the robot connects the reference door beam line from the collected environment image, then determines the direction information of the position of the reference door beam line relative to the robot, and records the angle information of the reference door beam line and the direction information of the position of the reference door beam line relative to the robot. Then step S3 is executed. Wherein, based on the existence of multiple room areas in the indoor working area, multiple doors are set as the entrances and exits between the corresponding two room areas, and then one or more reference door beam lines and the door frames where they are located are determined from the same frame of environment image collected at one position or multiple frames of environment image accumulated at different positions, that is, multiple door frames are determined simultaneously. In the indoor working area in this embodiment, the reference door beam line in the same direction can have one or more, and the position of the reference door beam line can be determined by the connecting node of the two reference door column lines connected with the reference door beam line. Wherein, the reference door beam line and the door frame where it is located can be in the same frame of environment image, or the door frame can be set in the grid map or the actual environment, the reference door column line is the candidate door column line in the environment image, and the reference door beam line is the candidate door beam line in the environment image. This embodiment also needs to record the direction information of the position of the reference door beam line relative to the robot (the different positions actually walked by the robot, and the reference door beam line and the door frame where it is located can be collected and extracted at the corresponding positions). Then, combined with the angle information of the same reference door beam line and the direction information of the position of the reference door beam line relative to the robot, the actually existing door beam line and the door frame where it is located can be determined in step S3 to exclude the misjudgment interference of the door leaf and the side wall line perpendicular to the ground. Wherein, the direction information of the position of the reference door beam line relative to the robot is actually the direction information of the reference door beam line relative to the current walking position of the robot (the position of collecting the environment image), which indicates that the robot collects the reference door beam line from a specific view angle at the current walking position. The direction of the center point of the reference door beam line (i.e. the midpoint) pointed to by the center point of the currently collected environment image (used to represent the current position of the robot in the environment image) represents the direction information of the position of the door frame where the reference door beam line is located relative to the robot. Preferably, the angle of the door beam line can be regarded as the angle formed with the image coordinate axis, the direction of the image coordinate axis can be regarded as the forward direction of the robot, the center of the image can be regarded as the center of the robot body, and the angle included in the direction information of the position of the door frame where the reference door beam line is located relative to the robot can be regarded as the included angle of the center point of the currently collected environment image and the center point of the reference door beam line relative to the reference door beam line.

[0026] It should be noted that there is generally a part of the wall on the door, which needs to be supported like a bridge to prevent a wooden or reinforced concrete prefabricated beam, called a door beam, which can reduce the impact of wall subsidence on the door frame. The door beam can be processed into a door beam line or a reference door beam line in the environment image of the embodiment by image recognition algorithm; the two sides below the door beam are two door columns perpendicular to the ground, which are also processed into a door column line or a reference door column line in the environment image of the embodiment by image recognition algorithm, and then two adjacent reference door column lines and a reference door beam line can form a door frame to be measured to form a simplified model of a door. Of course, objects that meet the characteristics of this simplified model of the door in the home environment also include square columns, side door leaves, door leaves, walls between the ceiling and the door frame, door sills, side walls and front walls. Then, the characteristic lines representing the shape of the door, such as the reference door beam line, are matched in the direction angle and the similarity count score processing is performed in step S3.

[0027] Step S3, after the robot walks through the indoor working area according to the preset planning path, records the direction information of the door frame to be measured where the reference door beam line is located relative to each position walked by the robot, and the angle information of the reference door beam line; then the robot selects a position to be scored as a matching center for template matching, and detects the matching result of the direction information of the door frame to be measured where the reference door beam line is located relative to the position walked by the robot and the matching result of the angle information of the corresponding reference door beam line in the process of template matching. Generally, a rectangular area (structural element) is used as the template area for template matching and the corresponding matching results of each position are counted for scoring. The position to be scored is a position walked by the robot, which is traversed one by one according to the predetermined translation direction in the process of template matching. Each position walked by the robot is marked as a position to be scored, and the direction information and the angle information recorded in the neighborhood (including eight neighborhoods and larger neighborhoods, according to the actual coverage area of the currently used template) of the position to be scored are matched. When the matching result of the direction information of the door frame to be measured where the reference door beam line is located relative to the position walked by the robot and the matching result of the angle information of the corresponding reference door beam line both meet the preset angle matching condition, the robot determines that the position to be scored is covered by the door frame, i.e. there is a door frame at the position to be scored; and the robot determines the center point of the door beam of the door frame to realize the positioning of the door frame.

[0028] Specifically, the to-be-scored position is a coordinate position of a center point of a reference door beam line identified / extracted by the robot in advance at the to-be-scored position in a world coordinate system; and the center point of the door beam of the door frame is the to-be-scored position, that is, the matching center currently used, and when the actual width of the door frame or the door beam thereof is determined, the specific distribution position of the door frame can be determined according to the geometric characteristics of the line segment.

[0029] In some embodiments, when the matching result of the direction information of the door frame to be detected where the reference door beam line is located relative to the positions walked through by the robot satisfies a first preset angle matching condition, and the matching result of the angle information of the corresponding reference door beam line satisfies a second preset angle matching condition, the robot determines that the door frame covers the to-be-scored position, and the robot determines the center point of the door beam of the door frame to realize positioning of the door frame; preferably, the first preset angle matching condition is equivalent to the second preset angle matching condition, and specifically, the statistical value of the matching result is equal, and threshold judgment can be performed using the statistical value of the matching result.

[0030] It should be noted that for the same reference door beam line located at the same position and having the same orientation, that is, the reference door beam line at one angle information in the same image region, a set of template information is used to match the direction information of the door frame to be detected where the reference door beam line is located relative to each position walked through by the robot and the angle information of the reference door beam line one by one, to determine that at least one position or a corresponding marked grid exists in the same door frame corresponding to the same reference door beam line (the position of the reference door beam line is the same and the orientation satisfies the corresponding matching condition); on the other hand, for the same or different reference door beam lines located at new positions and having the same or different orientations, that is, the reference door beam lines at different image regions and having the same or different angle information, a new set of template information (the set of template information (direction information and angle information) is adaptively adjusted according to the change of the extension direction of the reference door beam line) is used to match the direction information of the door frame to be detected where the reference door beam line is located relative to each position walked through by the robot and the angle information of the reference door beam line one by one, to determine that at least one position or a corresponding marked grid exists in the same door frame corresponding to the same reference door beam line (the position of the reference door beam line is the same and the orientation satisfies the corresponding matching condition). Therefore, if the reference door beam line to be matched changes (corresponding to another door frame in the same indoor working area), another template information is replaced to perform corresponding matching, so as to adapt to the change of the geometric angle characteristics of each reference door beam line participating in matching and the direction relationship thereof relative to the robot, and thus the door frame where the door beam of each extension direction is located is identified at the corresponding position.

[0031] Preferably, in the home environment, one reference door beam line is perpendicular to one wall surface, and another reference door beam line is parallel to the same wall surface, and one matching template is set for each reference door beam line to perform different round template matching in the grid map. If the angle information of all reference door beam lines needs to be adapted, it is determined that the reference door beam lines extracted by the pre-set image recognition algorithm are mainly concentrated in which angles, and then the templates corresponding to the angles are set to match, so that the subsequent matching time will not be doubled with the increase of the angle information of the reference door beam lines.

[0032] As an embodiment, in the step S2, the robot extracts two reference door column lines from the currently collected environment image in the process of walking, and connects the reference door column lines to obtain a reference door beam line; in the step S2, the robot can extract any two reference door column lines from the same frame of environment image, and then connect multiple reference door beam lines, and further obtain the extension direction of each reference door beam line, i.e. the angle direction of each reference door beam line in the image coordinate system (which can be the angle relative to the Y axis or the X axis, generally 90 degrees or 0 degrees); then the robot determines the direction information of the door frame to be measured relative to the current position of the robot according to the position relationship between the center point of each reference door beam line and the center point of the currently collected environment image, which is equivalent to the direction information of the reference door beam line relative to the current position of the robot in the environment image coordinates, which can be the direction information of the line connecting the two points; the direction information of the door frame to be measured relative to the current position of the robot, and the angle information of the reference door beam line are recorded in a grid corresponding to the current position of the robot, wherein the grid is a cell in the grid map constructed by the robot in the process of walking according to the pre-set planning path, and can be used as a storage address in the memory space of the robot; the robot constructs the grid map in the process of walking according to the pre-set planning path, which meets the requirements of the instant positioning and synchronous map construction algorithm to obtain the door frame orientation information detected by the robot at the corresponding grid; in this embodiment, each two reference door column lines and the reference door beam line connected thereby constitute a door frame to be measured, so as to be identified as an actually existing door frame by subsequent matching.

[0033] As an embodiment, in the step S1, the robot extracts straight line segments from the environment image collected by the robot by using a pre-set image recognition algorithm. Generally, the robot can obtain straight line segments from the environment image (which can be pre-processed by binarization) by using Hough transform, and calculate the angle of each straight line segment (which can be obtained based on the slope of the equation of the straight line segment). Then, the robot selects straight line segments that can represent door column lines, door beam lines, or side wall lines based on the length and angle of the straight line segments. Preferably, the robot can also recognize straight line segments in the image by using a canny edge detection algorithm, a sobel edge detection algorithm, etc. Then, the robot marks each straight line segment that intersects with the extension line of the center point of the environment image as a reference door column line; wherein the reference door column line is used to represent a line segment that is perpendicular to the ground of the indoor working area; since the lens of the camera of the robot is installed towards the ceiling, the view angle of the camera can cover the ceiling above the robot and the area below the ceiling, and if the door column that is perpendicular to the ground is consistent with the orientation of the camera, based on the perspective principle of the lens, the extension line of the reference door column line formed in the environment image will eventually intersect with the center of the imaging plane of the camera; as for the embodiment, each straight line segment that intersects with the extension line of the center point of the environment image can be obtained by solving the equations of the straight line segments.

[0034] Then, the robot selects two reference door column lines, and sets the end point close to the center point of the environment image as a corner point in each selected reference door column line, and then connects the corner points to form a reference door beam line, so that the two selected reference door column lines and the connected reference door beam line form a to-be-measured door frame. Referring to Figure 2 It can be seen that, Figure 2 is a schematic diagram of the environment image collected by the robot in front of the door frame, Figure 2 In the figure, the center point of the environment image currently collected by the camera of the robot is point O, which can represent the current position of the robot; wherein the straight line segment A1B1 is a reference door column line located on the left side of the robot extracted by the robot, and the straight line segment D1C1 is a reference door column line located on the right side of the robot extracted by the robot. Then, the end point B1 close to the center point O of the environment image is selected as the left corner point from the two end points of the reference door column line A1B1, and the end point C1 close to the center point O of the environment image is selected as the right corner point from the two end points of the reference door column line D1C1. Then, the line segment B1C1 is formed by connecting the corner point B1 and the corner point C1, and the line segment B1C1 forms a reference door beam line, and then Figure 2The extracted reference doorpost lines A1B1, C1D1, and B1C1 are connected to form a door frame to be tested, which may be a door frame existing in the actual environment.

[0035] Preferably, the robot sets a corner point on each selected reference gatepost line. When the distance between two corner points is within a preset threshold length range, the line segment connecting the two corner points is set as the reference gatepost line. Wherein, if the two corner points are respectively... Figure 2 If corner points B1 and C1 are given, then the preset threshold length range can be the door width within the indoor working area used in this embodiment, converted to... Figure 2 The error range of the width value within the image coordinate system is determined by the distortion of the image caused by lens distortion of the camera. The robot then sets the angle between the direction from one corner point to another and the coordinate axes of the image coordinate system, or the angle transformed into the world coordinate system, as the angle information of the reference door beam line. This is converted into fixed orientation information of the reference door beam line within the indoor working area and corresponds to the two selected reference doorpost lines. The direction from one corner point to another is an extension direction of the reference door beam line within the image coordinate system, corresponding to... Figure 2 Corner point B1 points in the direction of corner point C1, or corner point C1 points in the direction of corner point B1.

[0036] It should be noted that the robot detects the straight line segment in the environment image by the Hough algorithm. Each pixel coordinate point is transformed into a unified dimension that contributes to the straight line feature, for example: a straight line segment in the environment image is a collection of a series of discrete points, and the discrete polar equation of a straight line can express the geometric equation of the discrete points of the straight line as: x*cos(theta)+y*sin(theta)= r, wherein the angle theta refers to the included angle between r and the X axis, which can be converted into or directly used as the angle information of the reference door beam line required by the embodiment; r is the geometric perpendicular distance to the straight line. Any point on the straight line can be expressed as x, y, wherein r and theta are constants. In the field of image processing, the pixel coordinates P(x, y) of the image are known, and r and theta are variables to be found. If we can draw each (r, theta) value according to the pixel coordinate P(x, y) value, then the transformation from the image Cartesian coordinate system to the polar Hough space system is called the Hough transformation of the straight line. The transformation quantizes the Hough parameter space into a finite number of interval divisions or accumulative grids. When the Hough transformation algorithm starts, each pixel coordinate point P(x, y) is converted to a curve point on (r, theta), and is accumulated to the corresponding grid data point. When a wave crest appears, it means that there is a straight line. Then the pixel coordinates of the straight line are converted to the coordinates of the robot to obtain the angle of the straight line segment in the grid map, and the angle of the straight line segment is preferably the included angle between the straight line segment and the wheel axis or walking direction of the robot, and the wheel axis or walking direction of the robot is preferably a coordinate axis of the coordinate system of the grid map.

[0037] As an embodiment, the method for determining the direction information of the door frame to be measured relative to the current position of the robot according to the position relationship between the center point of the reference door beam line and the center point of the currently collected environment image comprises: the robot uses the center point of the environment image collected at the current position to represent the current position of the robot, which corresponds to point O; then the robot sets the direction of the center point of the currently collected environment image to the center point of the reference door beam line as the distribution direction of the door frame to be measured relative to the current position of the robot, which corresponds to point O. Figure 2 Figure 2 ​In the specific embodiment, the center point of the reference door beam line is the midpoint H1 of the straight segment B1C1, and the direction of point O pointing to point H1 is the distribution direction of the to-be-measured door frame in which the reference door beam line B1C1 is located relative to the current position of the robot (specifically, the pixel coordinate position of the robot in the image), which can also be understood as the direction of the robot being located in the H1O direction of the to-be-measured door frame in which the reference door beam line B1C1 is located. Then, the robot sets the included angle between the distribution direction and a coordinate axis of the image coordinate system as the azimuth angle formed by the to-be-measured door frame relative to the current position of the robot, or sets the included angle between the distribution direction and the reference door beam line as the azimuth angle formed by the to-be-measured door frame relative to the current position of the robot; and then the distribution direction of the to-be-measured door frame relative to the current position of the robot and / or the azimuth angle formed by the to-be-measured door frame relative to the current position of the robot form the direction information of the to-be-measured door frame relative to the current position of the robot. It should be noted that the direction information of the to-be-measured door frame relative to the current position of the robot in which the reference door beam line is located is equivalent to the direction information of the reference door beam line relative to the center point of the current collected environment image in the same image coordinate system; wherein the orientation of the camera of the robot is set to be tilted upward, so that the field of view of the camera of the robot covers at least the area between the reference door beam line and the ceiling, including the reference door beam line. Generally, the present embodiment converts the aforementioned azimuth angle into the world coordinate system, and specifically, the center point of the reference door beam line and the center point of the environment image are both converted into the world coordinate system to realize the conversion of the aforementioned azimuth angle and distribution direction into the world coordinate system, that is, into the grid map and positioned by means of the world coordinate system, which is suitable for use by the robot during navigation and movement, and forms the direction information of the to-be-measured door frame currently detected by the robot.

[0038] As an embodiment, in the step S3, after the robot walks through the indoor working area according to the preset planning path, the robot records the direction information of the to-be-measured door frame where the reference door beam line is located relative to the position of the robot corresponding to the grid in the corresponding grid of the grid map constructed by the robot, and the angle information of the reference door beam line, which can be unified into the world coordinate system where the grid map is located or remain in the image coordinate system; the direction information of the to-be-measured door frame where the reference door beam line is located relative to the position of the robot corresponding to the grid can be extended to the direction information of the to-be-measured door frame where the same reference door beam line is located relative to the position walked through by the robot, specifically including the direction information detected by the robot at positions on different sides of the same reference door beam line, so that the robot detects the to-be-measured door frame where the same reference door beam line is located from multiple different perspectives. Thus, in the process of walking through the indoor working area according to the preset planning path, the robot extracts straight line segments located at different orientations of the door frame from the environmental image in sequence, enriches the types of samples for subsequent matching, and avoids misjudgment caused by the fact that only the robot can detect the feature lines constituting the door frame shape from a single direction in the indoor working area, such as the feature lines of a square column and the feature lines of a door leaf.

[0039] In the present embodiment, in the process of walking through the indoor working area according to the preset planning path, the direction and angle information of the door frame detected by the robot are recorded in the corresponding grid of the grid map. Due to the passability of the door, the same door frame can be detected in front of, behind, and below the door frame. However, the square column and the door leaf in the home environment can only be detected by the robot in a single direction. The passability of the door allows the upwardly disposed camera of the robot to detect the same door frame in the three perspective ranges of the front, the back, and the directly below of the door frame. This further excludes the influence of the multiple vertical line segments similar to the door frame shape formed on the boundary between the front wall and the side wall of the indoor working area, and improves the accuracy of door recognition.

[0040] Preferably, the direction information of the door frame to be detected where the same reference door beam line is located relative to the position where the robot moves, at least includes the direction information of the same reference door beam line located in front of the robot relative to the position of the robot, the direction information of the same reference door beam line located behind the robot relative to the position of the robot, the direction information of the same reference door beam line located directly above the robot relative to the position of the robot, the direction information of the same reference door beam line located on the left side of the robot relative to the position of the robot, and / or the direction information of the same reference door beam line located on the right side of the robot relative to the position of the robot, so that the robot detects the door frame to be detected where the same reference door beam line is located from at least multiple different perspectives. Since the aforementioned direction information is obtained from the environment image collected by the robot at different positions, it needs to be converted into a unified world coordinate system and updated into the corresponding grid to form the coordinate information of each grid in the same grid map.

[0041] It should be noted that the position of the robot below the door frame to be detected where the same reference door beam line is located is between the position of the robot on the left side of the same reference door beam line and the position of the robot on the right side of the same reference door beam line. Therefore, if the robot starts to move from the left side of the same reference door beam line to the right side of the same reference door beam line, the center point of the same reference door beam line is located in front of the current position of the robot, which means that the robot is located on the left side of the same reference door beam line. If the robot moves from the left side of the same reference door beam line to the right side of the same reference door beam line, the center point of the same reference door beam line is located behind the current position of the robot, which means that the robot is located on the right side of the same reference door beam line. The grid corresponding to the position of the robot below the door frame to be detected where the same reference door beam line is located, the grid corresponding to the position of the robot on the left side of the door frame to be detected where the same reference door beam line is located, the grid corresponding to the position of the robot on the right side of the door frame to be detected where the same reference door beam line is located, the grid corresponding to the position of the robot in front of the door frame to be detected where the same reference door beam line is located, and the grid corresponding to the position of the robot behind the door frame to be detected where the same reference door beam line is located all record the angle information of the same reference door beam line.

[0042] It should be noted that the image coordinate system and the world coordinate system are both mapped with the tilt angle of the lens of the camera, and the center point of the environment image collected by the camera can represent the current position of the robot. Preferably, Figure 2The diagonal segment A1B1 and the diagonal segment C1D1 in the image are converted to the world coordinate system, and correspond to two vertical lines perpendicular to the walking ground of the robot. When the inclination angle of the camera relative to the ground plane is larger, the inclination of the reference door pillar line extracted from the environment image relative to the vertical direction of the drawing is larger, and the conversion effect of the transformation relationship between the coordinate systems is more significant.

[0043] As an embodiment, in the step S3, the robot traverses each grid in the grid map using the door orientation template to obtain a pair of angle matching results of the neighborhood of the grid, wherein the center of the door orientation template is the matching center, and the grid neighborhood involves a grid region including but not limited to an eight-neighborhood, a twenty-four-neighborhood, but all in the actual coverage plane of the grid map constructed by the robot walking in the indoor working area. In the process of the robot traversing the grid map using the door orientation template, the grid at the top left of the grid map can be selected, and the grid is traversed from left to right in the same row and from top to bottom in the same column until the grid at the bottom right of the grid map is traversed, wherein in the process of traversal, the center of the door orientation template is configured to cover each grid in the grid map; the pair of angle matching results of the neighborhood of the grid covered by the center of the door orientation template includes the matching result of the direction information of the to-be-tested door frame where the reference door beam line is located relative to the position walked by the robot and the matching result of the angle information of the same reference door beam line, and the pair of angle matching results can also be understood as the matching result information included in the matching angle pair; wherein each position walked by the robot can be understood as corresponding to one grid covered by the center of the door orientation template, and the position corresponding to each grid covered by the center of the door orientation template is updated as the to-be-scored position in turn. Then the robot gives a door positioning evaluation score to each grid in the grid map by counting the pair of angle matching results, that is, gives a door positioning evaluation score to the to-be-scored position, to represent the possibility that a door frame actually exists at the to-be-scored position or the grid to which the door positioning evaluation score is given; it should be noted that the door positioning evaluation score represents the number of all angle matching result pairs that meet the angle threshold, and is used to describe the similarity of the angle information recorded in the grid covered by the door orientation template and the corresponding angle information configured in the door orientation template, and the similarity of the direction information recorded in the grid covered by the door orientation template and the corresponding direction information configured in the door orientation template. Similarity is a measure of similarity, preferably, when the angle difference value is used to represent the similarity, the smaller the corresponding angle difference value, the higher the similarity, and the higher the door positioning evaluation score configured.

[0044] In addition, the embodiment configures the same door positioning evaluation score for the corresponding direction information and the angle information, and only when both are matched successfully (the matching result meets the error requirement), the door positioning evaluation score is added by one. On this basis, if it is detected that a grid is assigned with a door positioning evaluation score greater than a preset score threshold, it is determined that the grid is the center of the grid area occupied by the door frame, and the door frame exists at the to-be-scored position, which is the center of the door jamb of the door frame, and in the image coordinate system, it corresponds to the center point of the reference door jamb line to which the to-be-scored position belongs; then, in the embodiment, the robot takes the position corresponding to the grid as the center point of the door jamb of the door frame, extends a reference straight line segment with a preset standard length in the opposite two extension directions indicated by the angle information of the same reference door jamb line, and determines that the door frame exists at the reference straight line segment, wherein the door frame exists on both sides of at least one room area in the passable indoor working area, so as to simulate a reference straight line segment with a preset standard length as a door jamb line and become a boundary line between two room areas, so that the robot enters and exits the corresponding room area through the reference straight line segment.

[0045] In summary of steps S1 to S3, the application not only simplifies the recognition step and complexity of the door, but also assists in improving the accuracy of door frame recognition, avoids misjudging the square column, door leaf, and rotatable door leaf (a component provided in the door frame and capable of being opened and closed left and right) as a real door frame, effectively identifies the door frame in various recognition scenes of the door in the closed, half-open, fully open, and other states, and further identifies whether the door frame exists at the corresponding position in the same grid map. Of course, if the reference door jamb line to be matched changes (corresponding to another door frame in the same indoor working area for matching and positioning), another template information is replaced for corresponding matching, so as to adapt to the change of the geometric angle features of each reference door jamb line and the change of the direction relationship thereof relative to the robot, and further identify the door frame in which the door jamb in various extension directions exists at the corresponding position.

[0046] As a more specific embodiment, step S3 specifically includes:

[0047] Step S31: the robot controls the pre-set door orientation template to start traversing the grid in the grid map, so that the door orientation template partially or completely covers the grid map; when the door orientation template starts to traverse the grid map, the local area of the door orientation template first covers the grid map; wherein when the door orientation template completely covers the grid map, it is configured as a rectangular grid area with a center grid, and the center grid corresponds to the center of the door orientation template; then step S32 is executed.

[0048] Step S32, whenever the center (center grid) of the door orientation template covers a grid of the grid map, the coverage area of the door orientation template in the grid map is determined, and the area in the coverage area except the grid covered by the center is marked as a to-be-matched area, a neighborhood of the grid covered by the center is formed, and then step S33 is performed. Each position in the door orientation template corresponds to a position in the coverage area of the grid map through which the robot has walked; and each position in the door orientation template is marked with reference direction information of a to-be-detected door frame in which the reference door beam line is located relative to the position (a fixed angle value for the same reference door beam line) and a reference extension angle of the same reference door beam line; and the direction information of the to-be-detected door frame in which the reference door beam line is located relative to the corresponding position of each grid in the to-be-matched area and the angle information of the same reference door beam line are involved in matching of the related reference information at the corresponding position in the door orientation template, and the matching results of the direction information and the angle information are used together to evaluate the probability of existence of a door frame at the grid covered by the center of the door orientation template.

[0049] Step S33, in the to-be-matched area, whenever it is judged that the angle information of the reference door beam line recorded in a grid is in the reference door beam angle range and that the azimuth angle of the to-be-detected door frame in which the same reference door beam line is located relative to the corresponding position of the grid is in the corresponding reference deflection angle range, the door positioning evaluation score of the grid covered by the center of the door orientation template is counted by one, and the robot is determined to assign a door positioning evaluation score to each grid in the grid map by counting each angle matching result pair, until each grid in the to-be-matched area is traversed, and then step S34 is performed; wherein the direction information of the to-be-detected door frame in which the reference door beam line is located relative to the corresponding position of the grid includes an azimuth angle of the to-be-detected door frame in which the same reference door beam line is located relative to the corresponding position of the grid; based on the foregoing embodiments, the azimuth angle of the to-be-detected door frame in which the same reference door beam line is located relative to the corresponding position of the grid can be specifically an angle of a included angle between a center point of the same reference door beam line and a center point of an environmental image collected by the robot at the corresponding position of the grid in the same image coordinate system, or an angle of a included angle between the center point of the same reference door beam line and one coordinate axis of the image coordinate system, preferably, the foregoing related included angles can be converted from the image coordinate system to the world coordinate system and then recorded in the grid corresponding to the position through which the robot has walked, to adapt to representation of related line segments and angle directions in the grid map.

[0050] It should be noted that each grid in the to-be-matched region is configured with a reference deflection angle range, and the angle matching result of the neighborhood of the grid covered by the center of the door orientation template includes whether the angle of the same reference door beam line recorded in each grid in the to-be-matched region is in the reference door beam angle range (corresponding to one angle matching result), and whether the azimuth angle formed by the to-be-measured door frame in which the same reference door beam line is located relative to each grid in the to-be-matched region is in the corresponding reference deflection angle range (another angle matching result); wherein one reference door beam line corresponds to an angle information in the indoor working area, and the angle information is recorded in the grid corresponding to the position where the robot walks; one reference door beam line is matched using one door orientation template, and one door orientation template corresponds to one reference door beam angle range; preferably, the same door orientation template only translates in the grid map, and does not rotate in the grid map.

[0051] Specifically, the angle information of the recorded reference door beam line in a grid is within a reference door beam angle range, specifically represented as: the absolute value of the difference between the angle value of the recorded reference door beam line in the grid and the reference extension angle is within a reference door beam angle error range, preferably, the sum of the reference extension angle and the upper limit value of the reference door beam angle error range is set as a first reference door beam angle, and the sum of the reference extension angle and the lower limit value of the reference door beam angle error range is set as a second reference door beam angle, the reference door beam angle range is greater than or equal to the second reference door beam angle but less than or equal to the first reference door beam angle, wherein when the center (center grid) of the door orientation template covers a new grid, the reference extension angle and the reference door beam angle error range do not change, but the reference extension angle and the reference door beam angle error range will adaptively change with the change of the reference direction information (fixed angle value for the same reference door beam line) contained in the door orientation template itself and the reference extension angle of the reference door beam line. The azimuth angle formed by the to-be-measured door frame corresponding to the same reference door beam line relative to the grid is within a corresponding reference deviation angle range, specifically represented as: the absolute value of the difference between the azimuth angle formed by the to-be-measured door frame corresponding to the same reference door beam line relative to the grid and the reference detection angle corresponding to the grid is within a reference deviation angle error range, preferably, the sum of the reference detection angle and the upper limit value of the reference deviation angle error range is set as a first reference deviation angle, and the sum of the reference detection angle and the lower limit value of the reference deviation angle error range is set as a second reference deviation angle, the reference deviation angle range is greater than or equal to the second reference deviation angle but less than or equal to the first reference deviation angle; because the azimuth angle formed by each grid relative to the to-be-measured door frame corresponding to the same reference door beam line is different, when the center (center grid) of the door orientation template covers a new grid, a new reference detection angle will be used, and the reference deviation angle range will change with the change of the grid covered by the center (center grid) of the door orientation template, but the reference deviation angle error range does not change. Thus, the door positioning evaluation score is used to describe the similarity between the angle information recorded in the grid covered by the door orientation template in the grid map and the corresponding angle information configured in the door orientation template, and the similarity between the direction information recorded in the grid covered by the door orientation template in the grid map and the corresponding direction information configured in the door orientation template.

[0052] Step S34, judging whether the door positioning evaluation score of the grid covered by the center (the center grid of the door orientation template) in step S33 is greater than the preset score threshold, if yes, determining that the position corresponding to the grid covered by the center (the center grid of the door orientation template) in step S33 is the center point of the door beam of the door frame, that is, the grid covered by the center (the center grid of the door orientation template) in step S33 exists a door frame, and the door beam of the door frame is consistent with the extension direction of the reference door beam line recorded in the grid, and the matching result of the direction information of the door frame relative to the position walked by the robot and the matching result of the angle information of the corresponding reference door beam line also satisfy the preset angle matching condition, which also indicates that the angle information recorded in the grid covered by the door orientation template in the grid map and the corresponding angle information configured in the door orientation template reach the preset similarity, and the direction information recorded in the grid covered by the door orientation template in the grid map and the corresponding direction information configured in the door orientation template also reach the preset similarity. Otherwise, controlling the door orientation template to translate in the grid map, making the center (the center grid of the door orientation template) of the door orientation template traverse the next grid, and then executing step S32, and iteratively traversing the grid map until the center (the center grid of the door orientation template) of the door orientation template traverses each grid of the grid map; preferably, the next grid is adjacent to the grid covered by the center grid in step S32, including the adjacent in the up, down, left or right direction.

[0053] When the robot determines the position of the center covered grid in step S33, the robot extends in the opposite directions between the two end points of the same reference door beam line from the position of the center covered grid in step S33 by a preset extension length, respectively, to obtain the reference straight line segment, which is equivalent to: in the coordinate system corresponding to the grid map, extending in the direction and the opposite direction of the extension direction of one end of the same reference door beam line from the center covered grid in step S33 by a preset extension length, respectively, to obtain the reference straight line segment in the grid map; or in the image coordinate system, extending in the opposite directions between the two end points of the same reference door beam line from the pixel position of the center covered grid in step S33 by a preset extension length, respectively, to obtain the reference straight line segment, and determining that the door frame exists at the reference straight line segment. After determining the position information of the center point of the door beam of the door frame, the robot extends a reference straight line segment with a preset standard length in the opposite directions indicated by the angle information of the same reference door beam line, and determines that the door frame exists at the reference straight line segment. The preset standard length is twice the preset extension length. Thus, the relative accurate position of the door frame is determined. Therefore, the embodiment of the present application does not limit the judgment order of the size conditions such as the height and width of the door frame, but determines the grid corresponding to the center point of the door beam of the door frame from the angle of the actual door beam, the direction relationship of the center point of the door beam relative to the robot, and the passability of the door between rooms, in combination with the pre-collected image feature line angle information and the template matching grid result, to further determine the boundary position of the door frame, improve the accuracy of identifying the door frame, and improve the rationality of dividing the room area. More comprehensively, the reference straight line segment is avoided to be set under the furniture such as a table, a chair, a bed or a sofa, and a boundary line that truly meets the entrance area feature of the room area is divided, that is, the door or the door frame is identified at the reference straight line segment.

[0054] It should be noted that the position of the center covered grid in step S33 is the center point of the door beam of the door frame; the center covered grid in step S33 is the grid occupied by the center point of the reference door beam line existing at the grid; and the passable two sides of the door frame in the indoor working area each exist at least one room area, and the reference straight line segment becomes a boundary line between different room areas. In the embodiment, the same reference door beam line is also the reference door beam line corresponding to the angle information recorded in the center covered grid in step S33.

[0055] In summary, the foregoing embodiments determine the reference door pillar line and the reference door beam line as the candidate line segment by extracting the straight line segment and constructing the geometric connection relationship between different straight line segments, and then obtain the reference door beam line, the corresponding geometric angle feature and the direction relationship relative to the robot based on the passability of the door, which are collected by the robot when walking to different positions. After the robot walks through the indoor working area, the geometric angle feature (corresponding to the angle information of the reference door beam line) and the direction relationship relative to the robot (corresponding to the direction information of the position of the to-be-measured door frame relative to the robot) of the reference door beam line recorded at each position are matched one by one using the template information, wherein the reference door beam line and the two reference door pillar lines connected thereto constitute a to-be-measured door frame as a candidate door frame. Then, the matching result information at each position can be compared, and the position with the highest similarity of the matching result information is regarded as the position where the real door frame exists. Therefore, the door positioning in the home environment is effectively realized by a relatively low-power algorithm, and it is not necessary to increase redundant markers on the door. The positioning of the door can be realized as long as the two side frame lines and the door beam line of the top of the door are simultaneously present in the field of view of the camera of the present application.

[0056] In the foregoing embodiments, the conversion between the image coordinate system and the world coordinate system (robot coordinate system) involves the camera imaging geometric model (involving the focal length of the lens and the triangular geometric model of the pinhole imaging), and the coordinate conversion relationship is established in combination with the lens orientation of the camera and the rigid connection relationship on the robot body. Then, the relative angle in the image coordinate system is converted into the absolute angle of the actual environment by substituting the current position of the robot, so as to facilitate the matching of the door orientation template. In addition, the orientation of the camera of the robot is set to be inclined upward, and the viewing angle of the camera of the robot covers the area of the ceiling and below, so that the center point of the collected environment image represents the current position of the robot, so as to determine the pixel coordinates of the current position of the robot in the image coordinate system, and to determine the direction position relationship of the feature line in the environment image relative to the current position of the robot.

[0057] Based on the foregoing embodiments, the present application further discloses a robot which is equipped with a camera with the lens upwardly arranged, and the center point of the environment image collected by the camera is used to represent the center of the robot body. The robot is configured to perform the door frame positioning method disclosed in the foregoing embodiments. The program code corresponding to the door frame positioning method disclosed in the foregoing embodiments is stored in the controller or the memory built in the robot. When the robot calls the program code, the robot is configured to perform the door frame positioning method disclosed in the foregoing embodiments. Compared with the prior art, the laser sensor is not equipped, the sensor cost is saved, and the real-time navigation of the robot between different room areas is ensured.

[0058] In the embodiment, the camera is arranged at the front end of the robot body, the orientation of the camera of the robot is arranged horizontally and obliquely upward, the optical axis of the camera can be close to 90 degrees with the horizontal plane, and the environment image collected by the camera can cover at least the area between the door frame or the wall surface where the door beam of the door frame is located and the ceiling. In the embodiment, a single camera can be arranged, and the camera can be kept during the execution of the cleaning task. Thus, the robot can determine the position of the door frame of the room by using the angle direction information of the door column line and the door beam line, specifically, determine the corresponding boundary line or multiple discrete boundary positions, divide the room area close to the actual environment in the indoor working area, improve the rationality of the partition of the robot, and realize the cleaning in the room in a true sense.

[0059] The present application is described with reference to flowcharts and / or block diagrams of the method, device (system), and computer program product according to the embodiments of the present application. It should be understood that each flow and / or block in the flowcharts and / or block diagrams, and the combination of the flows and / or blocks in the flowcharts and / or block diagrams can be implemented by computer program instructions. These computer program instructions can be provided to a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing devices to produce a machine, so that the instructions executed by the computer or other programmable data processing devices generate a device that realizes the functions specified in the flowcharts and / or block diagrams. Figure 1 The functions specified in one or more flows and / or blocks Figure 1 The functions specified in one or more flows and / or blocks Figure 1 The functions specified in one or more flows and / or blocks Figure 1 The functions specified in one or more flows and / or blocks Figure 1 The functions specified in one or more flows and / or blocks Figure 1 The functions specified in one or more flows and / or blocks

[0060] Obviously, those skilled in the art can make various modifications and variations to the present application without departing from the spirit and scope of the present application. Thus, if these modifications and variations of the present application belong to the scope of the claims of the present application and their equivalent technologies, the present application also intends to include these modifications and variations.

Claims

1. An image recognition-based door frame positioning method, characterized by, The door frame positioning method is executed by a robot, the robot is equipped with a camera with a lens pointing upwards, and a center point of an environment image captured by the camera represents a position directly above the robot; The door frame positioning method comprises: Step S1, the robot walks in an indoor working area according to a preset planning path and collects environment images in the process of walking; Step S2, the robot connects a reference door beam line from the collected environment images, determines direction information of a position of the reference door beam line relative to the robot, and records angle information of the reference door beam line and the direction information of the position of the reference door beam line relative to the robot; Step S3, after the robot walks through the indoor working area according to the preset planning path, the robot selects a to-be-scored position as a matching center for template matching, and when the matching result of the direction information of the position of the reference door beam line relative to the robot walked through and the matching result of the corresponding angle information of the reference door beam line both satisfy a preset angle matching condition, the robot determines that the to-be-scored position is a center point of a door beam of the door frame, so as to realize positioning of the door frame; The angle of the door beam line is an angle formed with an image coordinate axis, and the direction of the image coordinate axis is a forward direction of the robot.

2. The door frame positioning method of claim 1, wherein, In the step S2, in the process of walking, the robot extracts two reference door column lines from the currently collected environment images, connects a reference door beam line based on the two reference door column lines, and then determines direction information of a position of the reference door beam line relative to the current position of the robot according to a position relationship between a center point of the reference door beam line and a center point of the currently collected environment images, and records the direction information of the position of the reference door beam line relative to the current position of the robot and the angle information of the reference door beam line in a corresponding grid of the robot, wherein the grid is a unit cell in a grid map constructed by the robot in the process of walking according to the preset planning path.

3. The door frame positioning method of claim 2, wherein, The step S3 further comprises: after the robot walks through the indoor working area according to the preset planning path, the corresponding grid of the grid map constructed by the robot records the direction information of the position of the reference door beam line relative to the position of the robot corresponding to the grid and the angle information of the reference door beam line; The direction information of the position of the reference door beam line relative to the position of the robot walked through by the robot includes: the direction information detected by the robot at positions on different sides of the same reference door beam line, so that the robot detects the position of the reference door beam line from multiple different perspectives.

4. The door frame positioning method of claim 3, wherein, In the step S3, the robot uses a door orientation template to traverse each grid in the grid map to obtain angle matching result pairs of a neighborhood of the grid, and a center of the door orientation template is the matching center. In the process of traversing, the center of the door orientation template is configured to cover each grid in the grid map; wherein the angle matching result pair of the neighborhood of the grid covered by the center of the door orientation template matches the matching result of the direction information of the to-be-evaluated door frame in which the reference door beam line is located relative to the position walked by the robot, and the matching result of the angle information of the same reference door beam line; the position corresponding to the grid covered by the center of the door orientation template is the to-be-evaluated position; Then, the door positioning evaluation score of each grid in the grid map is given by counting the angle matching result pair. If it is detected that the door positioning evaluation score given to a grid is greater than a preset score threshold, it is determined that the grid is the center of the grid area occupied by the door frame.

5. The door frame positioning method of claim 4, wherein, The step S3 specifically comprises: Step S31, control the pre-set door orientation template to start traversing the grids in the grid map; Step S32, whenever the center of the door orientation template covers a grid in the grid map, determine the coverage area of the door orientation template in the grid map, and mark the area in the coverage area except the grid covered by the center as a to-be-matched area; Step S33, in the to-be-matched area, whenever it is judged that the angle information of the reference door beam line recorded in a grid is within the reference door beam angle range, and it is judged that the orientation angle formed by the to-be-evaluated door frame in which the same reference door beam line is located relative to the position corresponding to the grid is within the corresponding reference deviation angle range, the door positioning evaluation score of the grid covered by the center is counted by one, and it is determined that the door positioning evaluation score of each grid in the grid map is given by the robot by counting the angle matching result pair, until each grid in the to-be-matched area is traversed; wherein the direction information of the to-be-evaluated door frame in which the reference door beam line is located relative to the position corresponding to the grid includes the orientation angle formed by the to-be-evaluated door frame in which the same reference door beam line is located relative to the position corresponding to the grid; Step S34, judge whether the door positioning evaluation score of the grid covered by the center in step S33 is greater than the preset score threshold, if yes, it is determined that the position corresponding to the grid covered by the center in step S33 is the center point of the door beam of the door frame, and it is also determined that the matching result of the direction information of the to-be-evaluated door frame in which the reference door beam line is located relative to the position walked by the robot and the matching result of the angle information of the corresponding reference door beam line all satisfy the preset angle matching condition; otherwise, control the door orientation template to translate in the grid map, so that the center of the door orientation template traverses the next grid, and then execute step S32, until the center of the door orientation template traverses each grid of the grid map.

6. The door frame positioning method of claim 5, wherein, The step S3 further comprises: When the robot determines the position corresponding to the grid where the center is covered in step S33, the robot extends in opposite directions between the two end points of the same reference door beam line by a preset extension length from the position corresponding to the grid where the center is covered in step S33 to obtain a reference straight line segment, and determines that the reference straight line segment is the door frame, wherein the position corresponding to the grid where the center is covered in step S33 is the center point of the door beam of the door frame; the grid where the center is covered in step S33 is the grid occupied by the center point of the reference door beam line; and the door frame has at least one room area on each side of the passable area in the indoor working area, and the reference straight line segment becomes a boundary line between different room areas.

7. The door frame positioning method of claim 5, wherein, Each grid in the to-be-matched area is configured with a corresponding reference bias angle range, and the angle matching result of the neighborhood of the grid where the center of the door orientation template is covered includes whether the angle of the same reference door beam line recorded in each grid in the to-be-matched area is within the reference door beam angle range, and whether the orientation angle of the to-be-detected door frame where the same reference door beam line is located relative to each grid in the to-be-matched area is within the corresponding reference bias angle range; wherein one reference door beam line corresponds to one angle information in the indoor working area, and the angle information is recorded in the grid corresponding to the position where the robot walks; one reference door beam line is matched using one door orientation template, and one door orientation template corresponds to one reference door beam angle range.

8. The door frame positioning method of claim 3, wherein, Among the direction information of the to-be-detected door frame where the same reference door beam line is located relative to the position where the robot walks, there are at least: the direction information of the same reference door beam line located in front of the robot relative to the position of the robot, the direction information of the same reference door beam line located behind the robot relative to the position of the robot, the direction information of the same reference door beam line located directly above the robot relative to the position of the robot, the direction information of the same reference door beam line located on the left side of the robot relative to the position of the robot, and / or the direction information of the same reference door beam line located on the right side of the robot relative to the position of the robot.

9. The door frame positioning method of claim 2, wherein, In the step S1, the robot extracts a straight line segment from the environment image collected by the robot through a pre-set image recognition algorithm; Then the robot marks each straight line segment intersecting the center point of the environment image as a reference door column line; wherein the reference door column line is used to represent a line segment perpendicular to the ground surface of the indoor working area; The robot selects two reference door column lines, sets an end point close to the center point of the environment image as a corner point in each selected reference door column line, and then connects the two corner points to form a reference door beam line, so that the two selected reference door column lines and the connected reference door beam line form a to-be-detected door frame.

10. The door frame positioning method of claim 9, wherein, After the robot sets an angle point on each selected reference door column line, the robot sets an angle between a direction from one angle point to another angle point on the reference door column line and a coordinate axis of an image coordinate system, or an angle converted from the angle to a world coordinate system, as angle information of the reference door column line, to convert an extension direction of the reference door column line in an indoor working area.

11. The door frame positioning method of claim 2, wherein, The method for determining the direction information of the door frame relative to the current position of the robot according to the positional relationship between the center point of the reference door column line and the center point of the currently collected environment image comprises: The robot uses the center point of the environment image collected at the current position of the robot to represent the current position of the robot; Then the robot sets a direction from the center point of the currently collected environment image to the center point of the reference door column line as a distribution direction of the door frame relative to the current position of the robot; Then an angle between the distribution direction of the door frame relative to the current position of the robot and a coordinate axis of an image coordinate system is set as a bearing angle of the door frame relative to the current position of the robot, or an angle between the distribution direction of the door frame relative to the current position of the robot and the reference door column line is set as the bearing angle of the door frame relative to the current position of the robot; The distribution direction of the door frame relative to the current position of the robot and / or the bearing angle of the door frame relative to the current position of the robot are used to form the direction information of the door frame relative to the current position of the robot, to form the direction information of the door frame currently detected by the robot.

12. A robot, characterized in that The robot is equipped with a camera with a lens pointing upwards, and a center point of an environment image collected by the camera is used to represent a center of a body of the robot. The robot is configured to perform the door frame positioning method according to any one of claims 1 to 11.

Images