Ground paving material paving direction identification method, control device and sweeping robot
By using the camera on the sweeping robot to obtain the floor laying image, perform material identification and edge linear detection, and determine the laying direction, the problem that the sweeping robot cannot accurately detect the ground conditions, realize accurate navigation and obstacle avoidance, and improve cleaning efficiency and quality.
Patent Information
- Application Number
- CN202410995254.5
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-07-24
- Publication Date
- 2024-11-12
AI Technical Summary
Existing sweeping robots cannot accurately detect the conditions on the ground, resulting in the inability to achieve accurate navigation and obstacle avoidance.
By setting up a camera on the sweeping robot, obtaining the floor laying image, performing identification processing to determine the floor laying material, performing edge linear detection and offset angle recognition, and determining the laying direction of the laying material, thereby achieving accurate identification of the floor laying material.
It realizes accurate identification of floor paving materials, improves the navigation and obstacle avoidance capabilities of the sweeping robot, and enhances the cleaning efficiency and quality in complex environments.
Smart Images

Figure CN118941934A_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the technical field of sweeping robots, and in particular to a method for identifying the laying direction of floor paving materials, a control device, a sweeping robot and a computer-readable storage medium. Background Art
[0002] With the continuous development of social economy and the continuous improvement of people's living standards, artificial intelligence has been increasingly applied to people's daily lives. Among them, sweeping robots, as an important tool in the field of household cleaning, have received widespread attention and application in recent years. However, current sweeping robots cannot accurately detect the condition of the ground, and thus cannot achieve accurate navigation and obstacle avoidance. Summary of the invention
[0003] The embodiments of the present application provide a method for identifying the laying direction of floor paving materials, a control device, a sweeping robot and a computer-readable storage medium, which can accurately detect the condition of the ground, thereby achieving precise navigation and obstacle avoidance.
[0004] An embodiment of the first aspect of the present application provides a method for identifying the laying direction of floor paving materials, which is applied to a mobile robot, wherein the mobile robot is provided with a camera, and the method comprises:
[0005] Acquiring a ground paving material image through the camera;
[0006] Performing recognition processing on the floor paving material image to determine the material of the floor paving material;
[0007] In the case where the floor paving material is a preset paving material type, edge straight line detection is performed on the floor paving material image to obtain floor paving material edge straight line feature information;
[0008] Performing offset angle recognition processing on the straight line feature information of the ground paving material edge to obtain the offset angle of the paving material laying direction;
[0009] The paving material laying direction is determined according to the paving material laying direction offset angle.
[0010] The method for identifying the laying direction of floor paving materials according to the first aspect of the embodiment of the present application has at least the following beneficial effects: in the process of identifying the laying direction of floor paving materials, firstly, a camera arranged on a mobile robot is used to obtain a floor paving material image, and then the floor paving material image is identified and processed to determine the material of the floor paving material, and then, when the material of the floor paving material is a preset paving material type, edge straight line detection is performed on the floor paving material image to obtain the floor paving material edge straight line feature information; then, the floor paving material edge straight line feature information is identified and processed to obtain the paving material laying direction offset angle. , and finally the laying direction of the paving materials can be determined according to the deviation angle of the laying direction of the paving materials; through the above technical scheme, the material of the ground paving materials is identified and processed. When the material of the ground paving materials is of the preset paving material type, the edge straight line of the ground paving material image will be detected and processed to obtain the edge straight line feature information of the ground paving materials, and then the edge straight line feature information of the ground paving materials will be processed for deviation angle identification to determine the deviation angle of the laying direction of the paving materials, and finally the laying direction of the paving materials will be determined according to the deviation angle of the laying direction, so that the laying direction identification of the ground paving materials can be more accurate, thereby realizing the precise navigation and obstacle avoidance of the sweeping robot.
[0011] In some embodiments, the step of performing edge line detection on the floor paving material image to obtain floor paving material edge line feature information includes:
[0012] Preprocessing the floor paving material image to obtain a floor paving material grayscale image;
[0013] Performing edge recognition on the floor paving material grayscale image to obtain an edge information image;
[0014] Performing Hough line detection on the edge information map to determine line parameters in the edge information map;
[0015] Clustering is performed on the straight line parameters in the edge information map to obtain the straight line feature information of the floor paving material edge.
[0016] In some embodiments, the step of performing offset angle recognition processing on the straight line feature information of the floor paving material edge to obtain the offset angle of the paving material laying direction includes:
[0017] Determine a first intersection point set of all straight lines from the straight line feature information of the floor paving edge;
[0018] Filtering the first intersection point set of all straight lines according to a preset coordinate range to obtain a second intersection point set;
[0019] The centroid of the second intersection point set is used as the vanishing point of the edge line of the floor paving material;
[0020] The paving material laying direction offset angle is determined according to the ground paving material edge line vanishing point, the width of the ground paving material image and preset image acquisition device internal parameters.
[0021] In some embodiments, determining the paving material laying direction according to the paving material laying direction offset angle includes:
[0022] Adding the paving material laying direction offset angle and the current robot posture deflection to obtain the paving material laying direction offset;
[0023] The paving material laying direction is obtained by performing angle conversion processing on the paving material laying direction offset.
[0024] In some embodiments, after determining the paving material laying direction according to the paving material laying direction offset angle, the method further includes:
[0025] According to a preset robot field of view, a ground paving material laying direction recognition process is performed at different positions in a preset area to obtain a plurality of ground paving material laying directions;
[0026] Accumulating the probability of coordinate points of a plurality of paving material laying directions to obtain a deflection angle probability map;
[0027] Determining the paving direction of the area according to the deflection angle probability map;
[0028] The cleaning direction of the mobile robot is controlled according to the paving direction of the area.
[0029] In some embodiments, the preprocessing of the floor paving material image to obtain a floor paving material grayscale image includes:
[0030] De-distorting the floor paving material image according to preset image acquisition device internal parameters and distortion parameters to obtain a first processed image;
[0031] Performing cropping processing on the first processed image according to a preset ranging range to obtain a second processed image;
[0032] Performing bilateral filtering on the second processed image to obtain a third processed image;
[0033] The third processed image is converted to obtain the ground paving material grayscale image.
[0034] In some embodiments, performing edge recognition on the floor paving material grayscale image to obtain an edge information image includes:
[0035] Determine the image binarization segmentation threshold according to the ground paving material grayscale image;
[0036] Determine a first edge detection threshold and a second edge detection threshold according to the image binarization segmentation threshold;
[0037] The edge information map is obtained by performing edge detection processing on the floor paving material grayscale map according to the first edge detection threshold and the second edge detection threshold.
[0038] In some embodiments, the step of accumulating the coordinate point probabilities of the plurality of paving material laying directions to obtain a deflection angle probability map includes:
[0039] Performing probability statistical processing on the plurality of paving material laying directions to obtain a plurality of statistical probabilities;
[0040] The statistical probability greater than a preset first set threshold is used as the paving material laying direction of the corresponding coordinate point to obtain the deflection angle probability map.
[0041] In some embodiments, determining the paving direction of the area according to the deflection angle probability map includes:
[0042] Determining a proportion of direction recognition results according to the number of paving material laying directions of all coordinate points in the deflection angle probability map and the number of statistical points in the deflection angle probability map;
[0043] The direction in which the proportion of the direction recognition results is greater than a preset second set threshold is used as the paving direction of the area.
[0044] In some embodiments, the controlling and processing of the cleaning direction of the mobile robot according to the paving direction of the regional paving materials includes:
[0045] Based on the paving direction of the regional paving materials, the cleaning direction of the mobile robot is controlled to be parallel or perpendicular to the paving direction of the regional paving materials.
[0046] In some embodiments, the camera includes one of the following: a monocular camera, a binocular camera, and a depth camera.
[0047] In some embodiments, the controlling and processing of the cleaning direction of the mobile robot according to the paving direction of the regional paving materials includes:
[0048] In the case where the floor covering material is carpet, the cleaning direction of the mobile robot is controlled to be along the laying direction of the floor covering in the area.
[0049] The second aspect of the present application provides a control device, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, the method for identifying the laying direction of floor paving materials as described in the first aspect of the present application is implemented.
[0050] A third aspect of the present application provides a sweeping robot, comprising the control device as described in the second aspect of the present application.
[0051] The fourth aspect of the present application provides a computer-readable storage medium storing computer-executable instructions, wherein the computer-executable instructions are used to execute the method for identifying the laying direction of floor paving materials as described in the first aspect above.
[0052] Other features and advantages of the present application will be described in the following description, and partly become apparent from the description, or understood by practicing the present application. The purpose and other advantages of the present application can be realized and obtained by the structures specifically pointed out in the description, claims and drawings. BRIEF DESCRIPTION OF THE DRAWINGS
[0053] Figure 1 is a flow chart of a method for identifying the laying direction of floor paving materials provided in an embodiment of the present application;
[0054] Figure 2 This is a flow chart of edge straight line detection for a floor paving material image provided by an embodiment of the present application;
[0055] Figure 3 This is a flow chart of performing offset angle recognition processing on straight line feature information of floor paving edge provided by an embodiment of the present application;
[0056] Figure 4 is a schematic diagram of a camera coordinate system provided in an embodiment of the present application;
[0057] Figure 5 is a flow chart for determining the laying direction of paving materials provided in an embodiment of the present application;
[0058] Figure 6 It is a schematic diagram of angle division provided in an embodiment of the present application;
[0059] Figure 7 is a flow chart of a method for identifying the laying direction of floor paving materials provided by another embodiment of the present application;
[0060] Figure 8 is a flow chart of preprocessing a ground paving material image provided by an embodiment of the present application;
[0061] Fig. 9 This is a flow chart of edge recognition processing of a grayscale image of floor paving materials provided in an embodiment of the present application;
[0062] Fig.10 is a flow chart of determining a deflection angle probability map provided by an embodiment of the present application;
[0063] Fig.11is a flow chart for determining the paving direction of a region provided by an embodiment of the present application;
[0064] Fig.12 is a flow chart of controlling the cleaning direction of a mobile robot provided in an embodiment of the present application;
[0065] Fig.13 is a flow chart of controlling the cleaning direction of a mobile robot provided by another embodiment of the present application;
[0066] Fig.14 is a schematic diagram of the edge line identified according to an embodiment of the present application;
[0067] Fig.15 It is a schematic diagram after straight line clustering provided in an embodiment of the present application;
[0068] Fig.16 It is a schematic diagram of a probability map provided in an embodiment of the present application;
[0069] Fig.17 It is a schematic diagram of the structure of the control device provided in an embodiment of the present application. DETAILED DESCRIPTION
[0070] In order to make the purpose, technical solutions and advantages of the present application clearer, the present application is further described in detail below in conjunction with the accompanying drawings and examples. It should be understood that the specific embodiments described herein are only used to explain the present application and are not intended to limit the present application. In addition, the characteristics, operations or features described in the specification can be combined in any appropriate manner to form various implementation methods. At the same time, the steps or actions in the method description can also be replaced or adjusted in order in a manner that is obvious to those skilled in the art. Therefore, the various sequences in the specification and the accompanying drawings are only for the purpose of clearly describing a certain embodiment and are not meant to be a necessary sequence, unless otherwise specified that a certain sequence must be followed.
[0071] In the description of this application, "several" means one or more, "more" means more than two, "greater than", "less than", "exceed", etc. are understood to exclude the number itself, and "above", "below", "within", etc. are understood to include the number itself. If there is a description of "first" or "second", it is only used for the purpose of distinguishing technical features, and cannot be understood as indicating or implying relative importance or implicitly indicating the number of the indicated technical features or implicitly indicating the order of the indicated technical features.
[0072] The serial numbers of the components in this document, such as "first", "second", etc., are only used to distinguish the objects described and do not have any order or technical meaning. The "connection" and "coupling" mentioned in this application, unless otherwise specified, include direct and indirect connections (couplings).
[0073] With the continuous development of social economy and the continuous improvement of people's living standards, artificial intelligence has been increasingly applied to people's daily lives. Among them, sweeping robots, as an important tool in the field of household cleaning, have received widespread attention and application in recent years. However, current sweeping robots cannot accurately detect the condition of the ground, and thus cannot achieve accurate navigation and obstacle avoidance.
[0074] Based on this, the embodiments of the present application provide a method for identifying the laying direction of floor paving materials, a control device, a sweeping robot and a computer-readable storage medium, which can accurately detect the condition of the ground, thereby achieving precise navigation and obstacle avoidance.
[0075] The following is a description with reference to the accompanying drawings:
[0076] Reference Figure 1 A method for identifying the laying direction of ground paving materials provided in an embodiment of the present application is applied to a mobile robot, and the mobile robot carries a camera. The method for identifying the laying direction of ground paving materials may include but is not limited to step S100, step S200, step S300, step S400 and step S500.
[0077] Step S100, obtaining a ground paving material image through a camera;
[0078] Step S200, identifying and processing the floor paving material image to determine the floor paving material material;
[0079] Step S300, when the floor paving material is of a preset floor paving material type, edge straight line detection is performed on the floor paving material image to obtain floor paving material edge straight line feature information;
[0080] Step S400, performing offset angle recognition processing on the straight line feature information of the ground paving material edge to obtain the offset angle of the paving material laying direction;
[0081] Step S500, determining the paving material laying direction according to the paving material laying direction offset angle.
[0082] According to the method for identifying the laying direction of ground paving materials provided in the embodiment of the present application, the ground paving material image can be acquired by a camera set on a mobile robot; then the ground paving material image can be identified by using a pre-trained neural network to determine the type of paving material, thereby determining the material of the ground paving material; and when the material of the ground paving material is a pre-recorded paving material type, the ground paving material image can continue to be detected by edge straight line detection, thereby determining the ground paving material edge straight line feature information; subsequently, the obtained ground paving material edge straight line feature information can continue to be identified by offset angle, so as to determine the offset angle of the laying direction of the paving material; finally, the laying direction of the paving material can be obtained based on the offset angle of the laying direction of the paving material. Through the above technical solution, the ground condition can be accurately detected, thereby achieving accurate navigation and obstacle avoidance.
[0083] It is worth noting that the mobile robot in the embodiment of the present application may be a sweeping robot, which is also called an automatic sweeper, smart vacuum cleaner, robot vacuum cleaner, etc. It is a kind of smart home appliance that can automatically complete floor cleaning in the room with artificial intelligence. Generally, brushing and vacuuming are used to absorb the debris on the ground into its own garbage storage box, thereby completing the function of floor cleaning. Generally speaking, robots that complete cleaning, vacuuming, and mopping are also collectively classified as sweeping robots. The camera carried by the sweeping robot in the embodiment of the present application may be a monocular camera, a binocular camera, or a depth camera; among them, the monocular camera is increasingly promoted and applied because of its lower cost and simpler implementation method, while the binocular camera and the depth camera can well improve the ranging accuracy. In the embodiment of the present application, the camera carried by the sweeping robot is not limited, and the production personnel can set it according to actual needs. In addition, cameras can be divided into two categories: digital cameras and analog cameras; digital cameras can convert the analog video signals generated by video acquisition devices into digital signals, and then store them in the computer; the video signals captured by analog cameras must pass through a specific video capture card to convert the analog signals into digital mode, and then compress them before they can be converted to the computer for use.
[0084] It is worth noting that the floor paving materials may include wooden boards, tiles, carpets, etc. Among them, since wooden boards and tiles are generally obtained by splicing multiple pieces, wooden boards and tiles can be used as pre-set paving material types; when it is determined that the floor paving material is wooden boards or tiles, the subsequent paving material laying direction identification process can be continued; when it is determined that the floor paving material is carpet, there is no need to continue the subsequent paving material laying direction identification process. Among them, the floor paving material image can be classified and processed based on texture features and / or color features. If it is known that the floor paving material in a certain area is plastic board, etc., the laying direction can also be identified based on the carpet recognition result to improve the robot's edge cleaning performance.
[0085] It is worth noting that the sweeping robot of the present application example may include a floor material recognition module, an edge feature detection and fusion module, a laying direction deviation angle recognition module, and a probability map module combined with the robot's posture. The present application embodiment can automatically identify the floor material type in the room, such as ceramic tiles, wooden floors, etc., through the floor material recognition module; this allows the sweeping robot to more accurately adjust the cleaning strategy according to different floor materials when cleaning in a complex environment, thereby improving cleaning efficiency and quality; at the same time, the floor material recognition module also provides basic data for subsequent identification of the laying direction of floor paving materials, thereby improving the accuracy of the identification of the laying direction of floor paving materials. The edge feature detection and fusion module uses edge detection algorithms and straight line detection methods to effectively extract the edge features of floor paving materials; clustering algorithms are used to cluster and optimize straight line parameters, further improving the accuracy of straight line detection results; through these functional modules, the sweeping robot can accurately obtain floor laying direction information, thereby performing more intelligent navigation and obstacle avoidance in actual work. The laying direction offset angle recognition module uses the recognition results of the boundary line or texture line of the floor paving material, combines the conversion from the camera coordinate system to the pixel coordinate system to calculate the offset angle relative to the camera shooting direction, and combines it with the robot posture at this time; this enables the sweeping robot to update and optimize its recognition results of the laying direction of the floor paving materials in the room in real time during the exploration process, further improving the performance of navigation and obstacle avoidance. And the probability map module combined with the robot posture analyzes the laying direction of the floor paving materials at different positions in the room, and combines the robot field of view to accumulate the laying direction probability within the field of view of each position, so that the sweeping robot can output a single or multiple laying direction results after completing the exploration task, and improve the comprehensive judgment ability of the laying direction of the floor paving materials in the current room, which will help the sweeping robot provide better cleaning effects in actual work. This application scheme uses existing computer vision and image processing technologies, combined with machine learning or deep learning methods, to realize the function of floor paving material laying direction recognition; the application of these technologies in the field of sweeping robots is innovative and practical, and can promote the development and innovation of technologies in this field. The solution of this application is easy to implement and can be easily combined with existing sweeping robot technology, so it has high commercial value; applying this application to actual products can significantly improve the working efficiency and cleaning quality of sweeping robots in complex environments and meet potential market demand. In summary, the embodiments of this application can significantly optimize the cleaning method of sweeping robots and improve their working efficiency and cleaning quality in complex environments.
[0086] In some embodiments of the present application, in the process of using a pre-trained neural network to identify and classify the paving material of a ground paving material image, the neural network can be a convolutional neural network and a residual neural network, etc., which are not limited here, and only need to be able to identify the paving material of the ground paving material image. Among them, a neural network is a computing model composed of a large number of nodes (or neurons) connected to each other, each node represents a specific output function, called an activation function; the connection between each two nodes represents a weighted value for the signal passing through the connection, called a weight, which is equivalent to the memory of an artificial neural network; the output of the network varies depending on the connection mode of the network, the weight value and the activation function; and the network itself is usually an approximation of a certain algorithm or function in nature, or it may be an expression of a logical strategy.
[0087] Reference Figure 2 , the above-mentioned step S300 may include but is not limited to step S310, step S320, step S330 and step S340.
[0088] Step S310, preprocessing the ground paving material image to obtain a ground paving material grayscale image;
[0089] Step S320, performing edge recognition on the ground paving material grayscale image to obtain an edge information image;
[0090] Step S330, performing Hough line detection on the edge information graph to determine the line parameters in the edge information graph;
[0091] Step S340: clustering the straight line parameters in the edge information map to obtain straight line feature information of the floor paving material edge.
[0092] In the edge line detection process provided by the embodiment of the present application, the ground paving material image is first preprocessed to determine the ground paving material grayscale image; then edge recognition is performed on the ground paving material grayscale image to determine the edge information image; then Hough line detection is performed on the edge information image to obtain the line parameters in the above-mentioned edge information image; finally, the obtained line parameters can be clustered to determine the edge line feature information of the ground paving material, so as to provide data support for the subsequent identification of the laying direction of the paving material.
[0093] It is worth noting that the preprocessing in the embodiment of the present application can convert the ground paving material image into a ground paving material grayscale image, which is a prerequisite for subsequent data processing. Specifically, the edge feature detection and fusion module in the embodiment of the present application mainly focuses on detecting edge features in the image and fusing similar or overlapping edge features; the detection and fusion of edge features are of great significance for the identification of the deviation angle of the laying direction. In order to improve the accuracy and effect of the subsequent analysis, the input image needs to be preprocessed; the image is first dedistorted to facilitate the subsequent intersection calculation; the calibrated monocular camera is used to obtain the measurable range, the image is cropped, and only the measurable part of the ground in the image is retained to eliminate the influence of objects on the ground of the home scene on the image processing; the edge information of the cropped image is then detected by the edge detection algorithm, and the edge detection algorithm is not restricted here; next, the straight line in the edge image is identified using the straight line detection method, and the straight line detection is not restricted here to obtain the boundary line or texture line of the ground paving material; according to the detected straight line information, a clustering algorithm can be used to realize the merging of similar straight lines, the filtering of horizontal lines by angle restrictions, and the clustering algorithm is not restricted here; in addition, in order to ensure the accuracy and stability of the edge feature detection and fusion module, the algorithm can also be optimized, such as adjusting parameter settings, removing nearly horizontal straight lines, etc.
[0094] Reference Figure 3 , the above-mentioned step S400 may include but is not limited to step S410, step S420, step S430 and step S440.
[0095] Step S410, determining a first intersection point set of all straight lines from the straight line feature information of the floor paving edge;
[0096] Step S420, filtering the first intersection point set of all straight lines according to a preset coordinate range to obtain a second intersection point set;
[0097] Step S430, taking the centroid of the second intersection point set as the ground paving material edge line vanishing point;
[0098] Step S440, determining the paving material laying direction offset angle according to the ground paving material edge line vanishing point, the width of the ground paving material image and the preset image acquisition device internal parameters.
[0099] In the deviation angle recognition process provided by the embodiment of the present application, first, a first set of intersection points corresponding to all straight lines is determined in the straight line feature information of the ground paving edge; then, the first set of intersection points obtained is filtered using a preset coordinate range, and then a second set of intersection points can be determined; then, the centroid of the remaining second set of intersection points is determined as the ground paving edge line vanishing point; finally, based on the ground paving edge line vanishing point, the calculation formula, the width of the ground paving image and the internal parameters of the image acquisition device, the deviation angle of the paving laying direction can be calculated to prepare for the subsequent determination of the paving laying direction.
[0100] Specifically, in the laying direction deviation angle recognition module, the floor laying direction will be identified by calculating the straight line intersection and the deviation angle. To achieve this goal, it is necessary to first determine the boundary line or texture line of the floor paving material, which can be achieved through the edge feature detection and fusion module. The straight lines that intersect at a distance on the image can be considered as parallel ground paving material boundary lines or texture lines. The intersection point is the vanishing point of the parallel lines. Therefore, the polar coordinate results of the Hough line detection are converted into homogeneous coordinates, and the intersection matrix is obtained by solving, and the vanishing points of all parallel lines can be obtained; the intersection points are filtered according to the pre-specified range, so that only the situation where the laying direction is along the camera shooting direction can be retained; after obtaining all the intersection points that fall within the specified range, the outliers are filtered out using the quarter distance or other statistical indicators; the centroid of all the intersection points is calculated as the vanishing point of the ground paving material edge line or texture line, and the pixel coordinates are (u, v); finally, through the conversion relationship from the camera coordinate system to the pixel coordinate system, the offset angle of the ground paving material edge line or texture line in the camera coordinate system relative to the camera shooting direction (z axis of the camera coordinate system) can be obtained through the distance from the horizontal coordinate of the vanishing point to the horizontal coordinate of the midpoint of the image. The calculation process is as follows:
[0101] In the camera coordinate system, with the camera as the origin, the polar coordinate expressions of all straight lines on the ground can be obtained:
[0102] x·sinθ+z·cosθ=r
[0103] Among them, r is the vertical distance from the straight line to the camera, and θ is the angle between the straight line and the z-axis of the camera coordinate system, such as Figure 4 shown.
[0104] The camera is a calibrated monocular camera, and the conversion equation from the camera coordinate system to the pixel coordinate system is known:
[0105]
[0106] The image has been distorted, that is, x = x′, y = y′, z = z′; from the above formula, we can get:
[0107]
[0108] For the vanishing point of the straight line in the image, it is the infinite point of the ground straight line in the camera coordinate system. Therefore, in the camera coordinate system, z→∞, at this time, r→0, thus we can get:
[0109] u=-tanθ·f x +C x
[0110] And because θ=0, the above formula can be obtained:
[0111] u0=C x
[0112] That is, the vanishing point of the straight line z=0 and other parallel lines in the camera coordinate system. It is not difficult to deduce that the horizontal coordinate of the vanishing point is the center of the image, that is, the image width is W, and the horizontal coordinate of the vanishing point is
[0113] Therefore, the coordinates of any vanishing point (u, v) can be expressed as the distance from the horizontal coordinate to the horizontal coordinate of the midpoint of the image:
[0114]
[0115] Where W is the image width, from which the offset angle of the ground paving material laying direction relative to the camera z-axis can be obtained:
[0116]
[0117] During the calculation process, in order to ensure the accuracy of angle judgment, mathematical methods (such as trigonometric functions, vector operations, etc.) will be used to calculate the deviation angle, and possible sources of error will be considered, such as local viewing angle deviation, camera internal and external parameter errors, etc. In addition, in order to improve the robustness of the deviation angle recognition in the laying direction, optimization measures will be taken for specific problems, such as introducing fuzzy logic and improving the straight line merging method.
[0118] Reference Figure 5 , the above step S500 may include but is not limited to step S510 and step S520.
[0119] Step S510, adding the paving material laying direction offset angle and the current robot posture deflection to obtain the paving material laying direction offset;
[0120] Step S520: performing angle conversion processing on the paving material laying direction offset to obtain the paving material laying direction.
[0121] According to the embodiment of the present application, in the process of determining the paving material laying direction, the paving material laying direction offset angle and the current robot posture deflection can be added to determine the paving material laying direction offset; then the paving material laying direction offset is converted to obtain the paving material laying direction, and the paving material laying direction corresponds to the ground paving material image currently collected, so as to prepare for the subsequent navigation of the sweeping robot.
[0122] Specifically, calculate the deviation angle of the laying direction recognized by the robot at different positions; record the posture information of the robot at different positions (such as position coordinates, orientation, etc.), which can be achieved through sensors such as encoders and inertial measurement units; add the robot deflection yaw to the recognized deviation angle and convert it to the interval (0, 2π) to obtain the deflection angle α of the laying direction relative to the world coordinate system; pre-set (0, 2π) as Figure 6 As shown in the example, (0, 2π) is pre-divided into multiple intervals to represent different laying directions. The number of partitions is not limited here. The laying direction of the paving material at the current robot position is determined according to the interval where α is located; combined with the field of view of the robot at different positions, the probability of different directions of the coordinate points within the field of view is accumulated according to the deflection angle calculated at the current position. After all positions are identified, a deflection angle probability map combined with the robot posture is obtained; probability statistics are performed on each room containing recognition results to obtain the laying direction results of the floor paving materials; the task execution of the sweeping robot is optimized in combination with the robot posture; each room can have a single or multiple laying direction results, for example, for square tiles, two horizontal and vertical recognition results are output; for herringbone wooden floors, two results of 45° upward and 45° downward are output; for non-square rectangular paving materials, a single horizontal or vertical recognition result is output, and the recognition direction is along the long side of the non-square rectangle. It is worth noting that (0, 2π) is subdivided into more intervals to accommodate more laying directions of floor paving materials.
[0123] Reference Figure 7 After executing the above step S500, the method for identifying the laying direction of floor paving materials may also include but is not limited to step S610, step S620, step S630 and step S640.
[0124] Step S610, performing ground paving material laying direction recognition processing on different positions in a preset area according to a preset robot field of view to obtain a plurality of paving material laying directions;
[0125] Step S620, accumulating the probability of coordinate points for multiple paving material laying directions to obtain a deflection angle probability map;
[0126] Step S630, determining the paving direction of the area according to the deflection angle probability map;
[0127] Step S640: Control the cleaning direction of the mobile robot according to the paving direction of the area.
[0128] According to the floor paving direction identification method provided in the embodiment of the present application, the floor paving direction identification operation performed in the above embodiment can be performed on different positions in the area based on the field of view of the sweeping robot, and multiple paving directions can be obtained; then, the coordinate point probability accumulation processing can be performed on the multiple paving directions obtained to determine the deflection angle probability map; then, the regional paving direction can be obtained through the deflection angle probability map; finally, the regional paving direction can be used to control and adjust the cleaning direction of the sweeping robot, so as to greatly improve the navigation accuracy of the sweeping robot.
[0129] Reference Figure 8 , the above step S310 may include but is not limited to step S311, step S312, step S313 and step S314.
[0130] Step S311, performing dedistortion processing on the ground paving material image according to preset image acquisition device internal parameters and distortion parameters to obtain a first processed image;
[0131] Step S312, cropping the first processed image according to a preset distance measurement range to obtain a second processed image;
[0132] Step S313, performing bilateral filtering on the second processed image to obtain a third processed image;
[0133] Step S314: convert the third processed image to obtain a grayscale image of the ground paving materials.
[0134] In the preprocessing process provided by the embodiment of the present application, the first processed image can be obtained by dedistorting the obtained ground paving material image according to the device internal parameters and distortion parameters of the camera; then the first processed image is cropped based on a preset ranging range to obtain a second processed image; then the second processed image is subjected to bilateral filtering to obtain a third processed image; finally, the third processed image is converted to obtain a grayscale image of the ground paving material, in preparation for subsequent image conversion and recognition processing.
[0135] Reference Fig. 9 , the above-mentioned step S320 may include but is not limited to step S321, step S322 and step S323.
[0136] Step S321, determining the image binarization segmentation threshold according to the ground paving material grayscale image;
[0137] Step S322, determining a first edge detection threshold and a second edge detection threshold according to the image binarization segmentation threshold;
[0138] Step S323, performing edge detection processing on the floor paving material grayscale image according to the first edge detection threshold and the second edge detection threshold to obtain an edge information image.
[0139] In the process of edge recognition processing provided by the embodiment of the present application, first, the image binarization segmentation threshold of the ground paving material grayscale image is determined based on the ground paving material grayscale image; then, the first edge detection threshold and the second edge detection threshold can be obtained based on the image binarization segmentation threshold; finally, the ground paving material grayscale image can be subjected to edge detection based on the first edge detection threshold and the second edge detection threshold to obtain the corresponding edge information map, in preparation for the subsequent edge line recognition.
[0140] Reference Fig.10 , the above step S620 may include but is not limited to step S621 and step S622.
[0141] Step S621, performing probability statistical processing on multiple paving material laying directions to obtain multiple statistical probabilities;
[0142] Step S622: taking the paving material laying direction of the corresponding coordinate point with a statistical probability greater than a preset first set threshold value to obtain a deflection angle probability map.
[0143] In the coordinate point probability accumulation process provided in the embodiment of the present application, firstly, probability statistics processing is performed on multiple paving material laying directions to obtain multiple statistical probabilities; then, the statistical probability among the multiple statistical probabilities that is larger than the first set threshold value can be used as the paving material laying direction of the corresponding coordinate point, and after multiple comparisons and screenings, a deflection angle probability map can be obtained, thereby providing a data basis for subsequent regional paving material laying direction identification.
[0144] Reference Fig.11 , the above step S630 may include but is not limited to step S631 and step S632.
[0145] Step S631, determining the proportion of direction recognition results according to the number of paving material laying directions of all coordinate points in the deflection angle probability map and the number of statistical points in the deflection angle probability map;
[0146] Step S632: The direction in which the proportion of direction recognition results is greater than a preset second set threshold is used as the paving direction of the regional paving material.
[0147] In some embodiments of the present application, the proportion of direction recognition results can be obtained by dividing the number of paving directions of all coordinate points in the deflection angle probability map by the number of statistical points in the deflection angle probability map; then the direction whose proportion of direction recognition results is greater than a preset second set threshold is used as the regional paving direction, and the subsequent sweeping robot can control and adjust its cleaning direction according to the regional paving direction, so as to greatly improve the navigation accuracy of the sweeping robot.
[0148] Reference Fig.12 , the above step S640 may include but is not limited to step S641.
[0149] Step S641, based on the laying direction of the regional paving materials, control the cleaning direction of the mobile robot to be parallel or perpendicular to the laying direction of the regional paving materials.
[0150] In some embodiments of the present application, after determining the laying direction of the regional paving materials, the cleaning direction of the mobile robot can be controlled to be parallel or perpendicular to the laying direction of the regional paving materials, thereby achieving precise navigation and cleaning processing.
[0151] Reference Fig.13 , the above step S640 may include but is not limited to step S642.
[0152] Step S642, when the floor covering material is carpet, control the cleaning direction of the mobile robot to be along the laying direction of the regional floor covering material.
[0153] In some embodiments of the present application, when the floor paving material is carpet, the cleaning direction of the mobile robot can be controlled along the laying direction of the regional paving materials, so that the mobile robot moves along the laying direction of the regional paving materials, thereby performing edge cleaning on the carpet and improving the robot's edge cleaning performance.
[0154] In order to more clearly illustrate the method for identifying the laying direction of floor paving materials according to an embodiment of the present application, a specific embodiment is provided below.
[0155] Step 1: Image acquisition;
[0156] During the exploration process, the robot vacuum cleaner takes images with a calibrated monocular camera. The image width is W and the height is H. The image can be measured at a height of (h, H) obtained through calibration. The camera internal parameter f x ,f y ,C x ,C y , distortion parameter dist coe ff.
[0157] Step 2: Ground material identification;
[0158] The mobilenet v2 image classification network is used to identify floor coverings. If the identification result is wooden floor or tile, the following steps are performed. If it is carpet or other floor coverings without edges, the following steps are not calculated.
[0159] Step 3: Image preprocessing;
[0160] Use the intrinsic parameters and distortion parameters obtained in step 1 to dedistort the captured image, use the measurable range obtained in step 1 to crop the image, and obtain a cropped image with a width W and a height Hh as the input for subsequent steps. Use bilateral filtering to filter out noise in the cropped image, retain line information as much as possible, and then convert the image from RGB space to grayscale space;
[0161] Step 4: Edge identification;
[0162] Calculate the maximum inter-class variance method threshold thresh of the grayscale image obtained in step 3, use 0.5*thresh as the canny detection low threshold, 0.8*thresh as the high threshold, use canny edge detection to obtain the edge information map, thereby obtaining the edge features in the cropped image, that is, the boundary or texture features of the floor paving materials, such as Fig.14 shown.
[0163] Step 5: straight line detection;
[0164] The straight line in the edge image obtained in step 4 of the Hough line detection method is used to obtain the boundary line or texture line of the floor paving material.
[0165] Step 6: Line parameter clustering;
[0166] Use the dbscan clustering algorithm to cluster the parameters of the straight lines obtained in step 5, so as to merge similar straight lines and filter out horizontal lines by angle restriction, such as Fig.15 shown.
[0167] Step 7: Calculate the intersection points and determine the vanishing points;
[0168] First, calculate all the intersection points of the filtered straight lines obtained in step 6, then retain the intersection points with abscissas in the range of (W / 2-200, W / 2+200) and ordinates in the range of (hH, 0), and use the quarter-distance algorithm to filter out outliers. Finally, take the centroid of the remaining intersection points as the intersection point of all straight lines, with pixel coordinates of (u, v), which is the vanishing point of the parallel lines in the camera coordinate system in the image.
[0169] Step 8: Calculate the deviation angle and identify the laying direction of the floor paving materials in the room;
[0170] Calculate the distance d from the intersection point's horizontal coordinate u to the image center's horizontal coordinate u0, according to the formula and the camera's intrinsic parameter f obtained in step 1x The offset angle θ is obtained, and the offset angle is added to the yaw of the robot's posture at this time and converted to the interval (0, 2π) to obtain the laying angle α of the ground paving material in this image; Figure 3 As shown, (0, 2π) is divided into the following 4 intervals:
[0171] (1) Horizontal laying interval:
[0172]
[0173] (2) 45° upward paving section:
[0174]
[0175] (3) Vertical laying section:
[0176]
[0177] (4) 45° downward paving section:
[0178]
[0179] By determining in which interval the laying angle α falls, the laying direction of the ground paving material captured in this frame of image can be obtained.
[0180] Step 9: Probability map calculation;
[0181] By identifying the laying direction of floor paving materials at different locations in the same room, and combining the robot's field of view, such as Fig.16 As shown on the left, the probabilities of the four laying directions are accumulated in the field of view of each position. If the current position is identified as the horizontal direction, the probability of the horizontal direction of all points in the field of view is increased accordingly, and the probabilities of the other three directions are reduced. After completing the shooting exploration of all points, the probabilities of the four directions of all points are counted, and the probability value of each point exceeding the predetermined threshold is taken as the laying direction of the current point, and the obtained Fig.16 An example of a probability map is shown on the right. There can be multiple direction results at the same time. The laying directions of all points are counted and divided by the number of statistical points to get the proportion of recognition results in the four directions. The direction exceeding the specified threshold is used as the laying result of the current room. Each room can have a single or multiple laying direction results.
[0182] It is worth noting that, as an important tool in the field of household cleaning, intelligent sweeping robots have received extensive attention and application in recent years. With the continuous development of science and technology, especially the progress of computer vision technology and artificial intelligence algorithms, the performance and functional requirements of intelligent sweeping robots are getting higher and higher. Computer vision technology is of great significance in this field. By processing and analyzing the images collected by the camera, the sweeping robot can obtain a lot of information about the surrounding environment. For example, ground material recognition and laying direction recognition are crucial for the robot to achieve precise cleaning tasks. At the same time, in the context of current industry development, monocular cameras have become the main choice of many sweeping robot detection equipment. Compared with binocular cameras, monocular cameras have lower costs and simpler implementation methods, but further research and development of related algorithms are needed to improve their ranging capabilities. Image processing methods and algorithms play a core role in the present invention. By preprocessing the input image (such as operations such as distortion removal, noise reduction and filtering), the accuracy and effect of subsequent analysis can be greatly improved. At the same time, by combining Hough transform, clustering algorithm and other methods, the present invention realizes the effective detection and analysis of floor texture features; finally, by converting the camera coordinate system to the pixel coordinate system, the offset angle of the laying direction of the floor paving material in the camera coordinate system is directly calculated through the pixel coordinates. Robot navigation and positioning is another important subfield. The present invention combines the laying direction of the floor paving materials at different positions in the room with the robot posture to achieve multi-position probability accumulation. Such a design enables the robot to determine the laying direction of the floor paving materials in the entire room after completing the exploration, thereby achieving more accurate and effective cleaning tasks. The solution mainly uses computer vision, image processing, robot navigation and positioning and other technologies to innovatively solve the problems encountered by sweeping robots in practical applications. In this technical field, the status of the proposal of the present invention has a leading advantage, and can effectively realize real-time recognition of floor materials and laying directions, thereby improving the performance and functions of intelligent sweeping robots.
[0183] In addition, refer to Fig.17 The embodiment of the present application also provides a control device 1000, including a memory 1200, a processor 1100, and a computer program stored in the memory 1200 and executable on the processor 1100. When the processor executes the computer program, the method for identifying the laying direction of floor paving materials as described in the above embodiment is implemented.
[0184] In addition, an embodiment of the present application further provides a computer-readable storage medium storing computer-executable instructions, and the computer-executable instructions are used to execute the method for identifying the laying direction of floor paving materials as described in the above embodiment.
[0185] The device embodiments described above are merely illustrative, and the units described as separate components may or may not be physically separated, that is, they may be located in one place or distributed on multiple network nodes. Some or all of the modules may be selected according to actual needs to achieve the purpose of the solution of this embodiment.
[0186] It will be appreciated by those skilled in the art that all or some of the steps and systems in the disclosed method above may be implemented as software, firmware, hardware and appropriate combinations thereof. Some physical components or all physical components may be implemented as software executed by a processor, such as a central processing unit, a digital signal processor or a microprocessor, or may be implemented as hardware, or may be implemented as an integrated circuit, such as an application specific integrated circuit. Such software may be distributed on a computer-readable medium, which may include a computer-readable storage medium (or a non-transitory medium) and a communication medium (or a temporary medium). As known to those skilled in the art, the term computer-readable storage medium includes volatile and non-volatile, removable and non-removable media implemented in any method or technology for storing information (such as computer-readable instructions, data structures, program modules or other data). Computer-readable storage media include, but are not limited to, RAM, ROM, EEPROM, flash memory or other memory technologies, CD-ROM, digital versatile disks (DVD) or other optical disk storage, magnetic cassettes, magnetic tapes, disk storage or other magnetic storage devices, or any other medium that may be used to store desired information and may be accessed by a computer. Furthermore, it is well known to those skilled in the art that communication media typically embodies computer readable instructions, data structures, program modules, or other data in a modulated data signal such as a carrier wave or other transport mechanism, and may include any information delivery media.
[0187] The above is a specific description of the preferred implementation of the present application, but the present application is not limited to the above-mentioned implementation mode. Technical personnel familiar with the field can also make various equivalent deformations or substitutions without violating the spirit of the present application. These equivalent deformations or substitutions are all included in the scope defined by the claims of the present application.
Claims
1. A method for identifying the laying direction of ground paving materials, characterized in that: Applied to a mobile robot, the mobile robot is provided with a camera, and the method comprises: Acquiring a ground paving material image through the camera; Performing recognition processing on the floor paving material image to determine the material of the floor paving material; In the case where the floor paving material is a preset paving material type, edge straight line detection is performed on the floor paving material image to obtain floor paving material edge straight line feature information; Performing offset angle recognition processing on the straight line feature information of the ground paving material edge to obtain the offset angle of the paving material laying direction; The paving material laying direction is determined according to the paving material laying direction offset angle.
2. The method according to claim 1, characterized in that: The performing edge straight line detection on the floor paving material image to obtain the floor paving material edge straight line feature information includes: Preprocessing the floor paving material image to obtain a floor paving material grayscale image; Performing edge recognition on the floor paving material grayscale image to obtain an edge information image; Performing Hough line detection on the edge information map to determine line parameters in the edge information map; Clustering is performed on the straight line parameters in the edge information map to obtain the straight line feature information of the floor paving edge.
3. The method according to claim 1, characterized in that The step of performing offset angle recognition processing on the straight line feature information of the ground paving material edge to obtain the offset angle of the paving material laying direction includes: Determine a first intersection point set of all straight lines from the straight line feature information of the floor paving edge; Filtering the first intersection point set of all straight lines according to a preset coordinate range to obtain a second intersection point set; The centroid of the second intersection point set is used as the vanishing point of the floor paving material edge line; The paving material laying direction offset angle is determined according to the ground paving material edge line vanishing point, the width of the ground paving material image and preset image acquisition device internal parameters.
4. The method according to claim 1, characterized in that: The step of determining the paving material laying direction according to the paving material laying direction offset angle comprises: Adding the paving material laying direction offset angle and the current robot posture deflection to obtain the paving material laying direction offset; The paving material laying direction is obtained by performing angle conversion processing on the paving material laying direction offset.
5. The method according to claim 1, characterized in that After determining the paving material laying direction according to the paving material laying direction offset angle, the method further comprises: According to a preset robot field of view, a ground paving material laying direction recognition process is performed at different positions in a preset area to obtain a plurality of ground paving material laying directions; Accumulating the probability of coordinate points of a plurality of paving material laying directions to obtain a deflection angle probability map; Determining the paving direction of the area according to the deflection angle probability map; The cleaning direction of the mobile robot is controlled according to the paving direction of the regional paving materials.
6. The method according to claim 2, characterized in that The preprocessing of the ground paving material image to obtain a ground paving material grayscale image includes: De-distorting the floor paving material image according to preset image acquisition device internal parameters and distortion parameters to obtain a first processed image; Performing cropping processing on the first processed image according to a preset ranging range to obtain a second processed image; Performing bilateral filtering on the second processed image to obtain a third processed image; The third processed image is converted to obtain the ground paving material grayscale image.
7. The method according to claim 2, characterized in that The step of performing edge recognition on the floor paving material grayscale image to obtain an edge information image includes: Determine the image binarization segmentation threshold according to the ground paving material grayscale image; Determine a first edge detection threshold and a second edge detection threshold according to the image binarization segmentation threshold; The edge information map is obtained by performing edge detection processing on the floor paving material grayscale map according to the first edge detection threshold and the second edge detection threshold.
8. The method according to claim 5, characterized in that The step of accumulating the probability of coordinate points of the plurality of paving material laying directions to obtain a deflection angle probability map includes: Performing probability statistical processing on the plurality of paving material laying directions to obtain a plurality of statistical probabilities; The statistical probability greater than a preset first set threshold is used as the paving material laying direction of the corresponding coordinate point to obtain the deflection angle probability map.
9. The method according to claim 8, characterized in that The determining of the regional paving material laying direction according to the deflection angle probability map comprises: Determining a proportion of direction recognition results according to the number of paving material laying directions of all coordinate points in the deflection angle probability map and the number of statistical points in the deflection angle probability map; The direction in which the proportion of the direction recognition results is greater than a preset second set threshold is used as the paving direction of the area.
10. The method according to claim 5, characterized in that The controlling and processing of the cleaning direction of the mobile robot according to the paving direction of the regional paving materials includes: Based on the paving direction of the regional paving materials, the cleaning direction of the mobile robot is controlled to be parallel or perpendicular to the paving direction of the regional paving materials.
11. The method according to claim 1, characterized in that: The camera includes one of the following: a monocular camera, a binocular camera, and a depth camera.
12. The method according to claim 5, characterized in that The controlling and processing of the cleaning direction of the mobile robot according to the paving direction of the regional paving materials includes: In the case where the floor covering material is carpet, the cleaning direction of the mobile robot is controlled to be along the laying direction of the floor covering in the area.
13. A control device, characterized in that: The method comprises a memory, a processor and a computer program stored in the memory and executable on the processor, wherein the processor implements the method for identifying the laying direction of floor paving materials as claimed in any one of claims 1 to 12 when executing the computer program.
14. A sweeping robot, characterized in that: Comprising the control device as claimed in claim 13.
15. A computer-readable storage medium, characterized in that: Computer executable instructions are stored, and the computer executable instructions are used to execute the method for identifying the laying direction of floor paving materials according to any one of claims 1 to 12.
