A method and storage medium for detecting laser doors at any angle based on convolutional neural network

Abstract

The present invention discloses a method for detecting a laser door at any angle based on a convolutional neural network and a storage medium, comprising the following steps: S1, acquiring original laser data; S2, converting the original laser data into a laser image; S3, using a model to detect the laser image and extract a characteristic curve of a door in the laser image; S4, using a rectangular frame or / and an inclined frame to mark the characteristic curve of the door, and acquiring position information of the rectangular frame or / and the inclined frame, wherein the position information includes a center point, length, width and angle information; the present invention can accurately provide the inclined frame position information of the door in the image, and can provide rich high-level information for other tasks during actual use.

Description

Technical Field

[0001] The present invention relates to the field of image recognition, and in particular to an arbitrary-angle laser door detection method based on a convolutional neural network and a storage medium. Background Art

[0002] The general laser door detection method is directly based on laser data and combined with some traditional machine learning algorithms to detect the location of the door; or, it uses some existing convolutional neural network-based detectors to directly detect the location of the door.

[0003] In the former case, the extracted features are generally superficial and sensitive to scene changes, resulting in their robustness failing to meet actual usage requirements;

[0004] As for the latter, since most of the existing detection algorithms can only detect rectangular frames without angles, the results obtained by this type of method cannot provide posture information such as the door line and direction, and the available information is limited. Summary of the invention

[0005] The technical solution of the present invention to solve the above technical problems is as follows: a method for detecting an arbitrary angle laser door based on a convolutional neural network, characterized in that it comprises the following steps:

[0006] S1, obtaining raw laser data;

[0007] S2, converting the original laser data into a laser image;

[0008] S3, using a model to detect the laser image and extract a characteristic curve of a door in the laser image;

[0009] S4. Mark the characteristic curve of the door with a rectangular frame or / and an inclined frame, and obtain position information of the rectangular frame or / and the inclined frame, wherein the position information includes center point, length, width and angle information.

[0010] Furthermore, the step S2 specifically includes:

[0011] S201, laser coordinate conversion;

[0012] S202: Grayscale image filling.

[0013] Furthermore, the step S201 includes converting the distance information into coordinate information, and the formula for converting the polar coordinate laser data into Cartesian coordinate data is as follows:

[0014] x = r × cosθ;

[0015] y=r×sinθ

[0016] Where r is the distance from the center of the rectangular frame to the origin of the coordinate system, x, y represents the position of the center of the rectangular frame in Cartesian coordinates, and θ is the angle of laser emission.

[0017] Furthermore, the coordinate values ​​need to be further filled into the grayscale image according to the actual usage ratio t. When filling, the coordinate positions scaled according to the ratio t are filled with 255, and the remaining positions are 0, so that an actually usable laser image can be obtained.

[0018] Furthermore, the step S3 specifically includes:

[0019] Step S301, image preprocessing;

[0020] Step S302: model processing;

[0021] Step S303: Post-processing of results.

[0022] Furthermore, the step S301 specifically includes: the laser image needs to undergo certain normalization and data augmentation (training phase) image preprocessing operations before being input into the model, wherein the normalization needs to normalize 0 and 255 on the laser image to -0.5 and 0.5 respectively, and the data augmentation operation includes random flipping and rotation operations.

[0023] Furthermore, the step S302 specifically includes: the detection model uses YOLO-v3 as a benchmark model, and the output layer of the YOLO-v3 is modified accordingly;

[0024] The detection of the tilted box (w, h, θ) is added to the last regression layer, where the regression target of the tilted box is directly calculated based on the original anchor box, and the center point (x, y) is the same as the rectangular box.

[0025] Furthermore, the step S302 specifically includes: in the model training stage, a loss function is used to characterize the difference between the predicted value and the true value. The length or width of the tilted box uses the same L2 loss as the rectangular box as the loss function, while the angle θ uses the sine function sin as the loss function. The specific function is as follows:

[0026] L2=(x gt -x pred ) 2 ;

[0027] L θ = sin(|θ gt -θ pred |)

[0028] Among them, x gtis the length or width of the tilted box, x pred is the length or width of the rectangular box, θ gt is the angle of the tilted frame, θ pred is the angle of the rectangular box.

[0029] Furthermore, the step S303 specifically includes: the post-processing operation of the detection model is non-maximum suppression (NMS)

[0030] Get the reserved index value after the rectangular box NMS;

[0031] Based on the index value, a corresponding tilted frame is obtained and a post-processed rectangular frame and a tilted frame are obtained;

[0032] The position information (x, y, w, h, θ) of the inclined frame is acquired based on the inclined frame.

[0033] A third aspect of an embodiment of the present invention provides a robot, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps of the above-mentioned robot's arbitrary-angle laser door detection method based on a convolutional neural network when executing the computer program.

[0034] A fourth aspect of an embodiment of the present invention provides a computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, the steps of the above-mentioned arbitrary-angle laser door detection method based on a convolutional neural network are implemented.

[0035] Compared with the prior art, the present invention has the following beneficial effects:

[0036] This method is based on the existing algorithm and is improved for specific scenarios, so that it has the ability to detect doors at any angle. Specifically, in this method, the input raw laser data is first converted into a laser image as the input of the subsequent detection model, and then processed using the improved version of the detection model. While outputting the rectangular frame position of the door in the image, the position information of its tilted frame is given, which includes information such as the center point, length, width, and angle. If you want to use it in actual scenarios, such as robot riding an elevator, robot movement, and other behavioral control, you can also convert the image detection results into the coordinates of the robot's perspective for use in downstream tasks. The present invention can accurately give the tilted frame position information of the door in the image, and can provide rich high-level information for other tasks during actual use. BRIEF DESCRIPTION OF THE DRAWINGS

[0037] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings required for use in the embodiments are briefly introduced below. It should be understood that the following drawings only show certain embodiments of the present invention and therefore should not be regarded as limiting the scope. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without creative work.

[0038] Figure 1 It is a flow chart of the arbitrary angle laser door detection method based on convolutional neural network provided in Example 1;

[0039] Figure 2 is a flow chart of laser data conversion provided in Example 2;

[0040] Figure 3 is a flow chart of model detection provided in Example 3;

[0041] Figure 4 The present invention is an effect diagram of arbitrary angle laser door detection provided by an embodiment of the present invention;

[0042] Figure 5 This is a result comparison effect diagram provided in Example 3;

[0043] Figure 6 This is a comparison diagram of the results provided in Example 3. DETAILED DESCRIPTION

[0044] In order to make the purpose, technical scheme and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present invention.

[0045] In some embodiments, Figure 1 A flowchart of an arbitrary angle laser door detection method based on a convolutional neural network is provided, comprising the following steps:

[0046] S1. Obtaining original laser data. There are many methods for obtaining laser data. The following provides three situations for using 3D laser radar. The specific situations are as follows: If the robot scans while walking, it takes time to process the image. In order to reduce the time for the robot to process the image, it is necessary to manually collect and process the image before the robot moves. The user walks in the target area with the 3D laser radar in hand. The user drives the 3D laser radar to move to the target position. The 3D laser radar will detect and map the ground in real time. There are multiple users, and multiple users scan in different areas until all target areas are covered; or, the 3D laser radar is installed on a mobile robot, and the robot is allowed to move and scan autonomously, which can save the user's working time;

[0047] Or, if the robot walks while scanning, it takes time to process the image. In order to reduce the time for the robot to process the image, it is necessary to manually collect and process the image before the robot moves, which consumes manpower. If multiple laser scanning users are set in the target area, a schematic diagram of a target area setting device is provided. Schematically, the above-mentioned device sets three stations in the target area. The allocation scheme is: the three equipments are triangles, and the preferred triangle angle is 600. The device establishes contact with the robot through a wireless communication device, and three-dimensional laser raw data;

[0048] S2, converting the original laser data into a laser image, where the original laser data is just a collection of some point cloud data;

[0049] S3, using a model to detect the laser image and extract a characteristic curve of the door in the laser image, wherein the algorithm for extracting the characteristic curve is a convolutional neural network, the characteristic curve includes a door horizontal line characteristic curve and a door contour characteristic curve, and both the door horizontal line characteristic curve and the door contour characteristic curve are composed of point cloud data, that is, multiple points constitute the curve, and when extracting the characteristic curve using the convolutional neural network, it is necessary to collect positive samples of the door and continuously improve the convolutional neural network extraction model;

[0050] S4. A rectangular frame or / and an inclined frame are used to mark the characteristic curve of the door, and the position information of the rectangular frame or / and the inclined frame is obtained, wherein the position information includes the center point, length, width and angle information of the rectangular frame or / and the inclined frame. The difference between the rectangular frame and the inclined frame is relative to the reference point. The laser image and the visual image have the same reference point, and the two-dimensional coordinates of XOY are established with the reference point as the origin. Therefore, the position information of the rectangular frame or / and the inclined frame relative to the reference point can be obtained. The rectangular frame is in a vertical state in the coordinate system, and the inclined frame has a certain inclination angle according to the extracted characteristic curve of the door. The inclination is mainly related to the angle of collecting laser data. When using laser collection equipment, in order to save time, it is impossible to calculate the angle very perfectly. Therefore, a certain angle deviation will definitely occur, and in this case, a tilted frame will occur.

[0051] This method is based on the existing algorithm and is improved for specific scenarios, so that it has the ability to detect doors at any angle.

[0052] In this embodiment, Figure 4 The present invention is an effect diagram of arbitrary angle laser door detection provided by an embodiment of the present invention, (a), (b) and (c) are respectively the detection results of doors at different positions. Comparison between (a) and (b) shows that the rectangular boxes have different angles, and (c) shows that multiple targets can be detected and marked.

[0053] Specifically, in this method, the input raw laser data is first converted into a laser image as the input of the subsequent detection model, and then processed using the improved detection model. While outputting the position of the rectangular frame of the door in the image, the position information of its inclined frame is given, which includes information such as the center point, length, width, and angle.

[0054] In addition, if you want to use it in actual scenarios, such as controlling the robot's elevator ride, robot movement, etc., you can also convert the image detection results into the coordinates of the robot's perspective for use in downstream tasks. The present invention can accurately give the tilt frame position information of the door in the image, and can provide rich high-level information for other tasks during actual use.

[0055] It needs to be explained that the convolutional neural network (CNN) is a feedforward neural network whose artificial neurons can respond to surrounding units within a part of the coverage range, and has excellent performance in large-scale image processing. It includes a convolutional layer and a pooling layer. Generally, the basic structure of CNN includes two layers, one of which is a feature extraction layer. The input of each neuron is connected to the local receptive field of the previous layer, and the local features are extracted. Once the local feature is extracted, its positional relationship with other features is also determined; the second is a feature mapping layer. Each computing layer of the network consists of multiple feature maps, each feature map is a plane, and the weights of all neurons on the plane are equal. The feature mapping structure uses a sigmoid function with a small influence function kernel as the activation function of the convolutional network, so that the feature map has displacement invariance. In addition, since neurons on a mapping surface share weights, the number of free parameters of the network is reduced. Each convolutional layer in the convolutional neural network is followed by a computing layer for local averaging and secondary extraction. This unique two-time feature extraction structure reduces the feature resolution.

[0056] In this embodiment, a scheme is schematically provided for how to use a convolutional neural model to identify a door and mark it with a rectangular frame. In this application document, this part is not the focus. The focus is on how to detect the direction and width of the door according to the position information of the rectangular frame. Schematically, based on a pre-built and trained deep convolutional neural network including a region proposal network, a fully connected layer, a classifier, a first regressor, and a second regressor, the method includes the following steps: scanning the position of the door as a first laser image by a laser sensor installed on the robot; compressing the first image into a second image; the preset size of the second image is M×N; extracting a feature map G of the second image based on the region proposal network, wherein the feature map includes door features and horizontal baseline features, and extracting candidate rectangular frames for the door features to construct a first candidate rectangular frame set; and using region of interest pooling to collect the first candidate rectangular frames. The features corresponding to each candidate rectangular box in the set on the feature map G are processed to obtain a first feature vector; the first feature vector is processed by the fully connected layer to obtain a second feature vector; the second feature vectors corresponding to each candidate rectangular box in the first candidate rectangular box set are respectively sent to the classifier for category determination, and a second candidate rectangular box set is constructed; the first regressor is used to adjust each candidate rectangular box in the second candidate rectangular box set, respectively, to obtain a positioning rectangular box corresponding to each candidate rectangular box, and a positioning rectangular box set is constructed; for each positioning rectangular box in the positioning rectangular box set, the second regressor is used to calculate the grasping position of the object corresponding to each positioning rectangular box, wherein the first feature vector and the second feature vector are related to the position information including the center point, length, width and angle information, and can be converted into each other through mathematical changes. The following content mainly involves further improvements to the rectangular box generation method.

[0057] In this embodiment, the step S2 specifically further includes:

[0058] S201, laser coordinate conversion;

[0059] S202: Grayscale image filling.

[0060] In this embodiment, step S201 includes converting the distance information into coordinate information. The original laser data only contains the distance information, that is, the distance reached by each laser beam. The data in polar coordinate format cannot be used directly. Therefore, the distance information needs to be converted into coordinate information. Taking the robot coordinate system as the left-hand coordinate system as an example (the front is the x-axis), the formula for converting the polar coordinate laser data into the Cartesian coordinate data is as follows:

[0061] x = r × cos θ;

[0062] y=r×sinθ

[0063] Where r is the distance from the center of the rectangular frame to the origin of the coordinate system, x, y represents the position of the center of the rectangular frame in Cartesian coordinates, and θ is the angle of laser emission.

[0064] In this embodiment, the coordinate values ​​need to be further filled into the grayscale image according to the actual usage ratio t. When filling, the coordinate positions scaled according to the ratio t are filled with 255, and the remaining positions are 0, so that an actually usable laser image can be obtained.

[0065] In this embodiment, step S3 specifically includes:

[0066] Step S301, image preprocessing;

[0067] Step S302: model processing;

[0068] Step S303: Post-processing of results.

[0069] In this embodiment, the step S301 specifically includes: the laser image needs to undergo certain normalization and data augmentation (training phase) image preprocessing operations before inputting into the model, wherein the normalization needs to normalize 0 and 255 on the laser image to -0.5 and 0.5 respectively, and the data augmentation operation includes random flipping and rotation operations, which can make the detection model more robust.

[0070] In this embodiment, the step S302 specifically includes: the detection model uses YOLO-v3 as a benchmark model, and the output layer of the YOLO-v3 is modified accordingly;

[0071] The detection of the tilted box (w, h, θ) is added to the last regression layer, where the regression target of the tilted box is directly calculated based on the original anchor box, and the center point (x, y) is the same as the rectangular box.

[0072] In this embodiment, the step S302 specifically includes: in the model training stage, a loss function is used to characterize the difference between the predicted value and the true value. The length or width of the inclined box uses the same L2 loss as the rectangular box as the loss function, and the angle θ uses the sine function sin as the loss function. The specific function is as follows:

[0073] L2=(x gt -x pred ) 2 ;

[0074] L θ = sin(|θ gt -θ pred |)

[0075] Among them, x gt is the length or width of the tilted box, x pred is the length or width of the rectangular box, θ gt is the angle of the tilted frame, θ pred is the angle of the rectangular box.

[0076] In this embodiment, the step S303 specifically includes: the post-processing operation of the detection model is non-maximum suppression (NMS);

[0077] Get the reserved index value after the rectangular box NMS;

[0078] Based on the index value, a corresponding tilted frame is obtained and a post-processed rectangular frame and a tilted frame are obtained;

[0079] Based on the tilted frame, the position information (x, y, w, h, θ) of the tilted frame is obtained. After the model is processed, it is necessary to further perform post-processing operations to finally output it. Generally speaking, the post-processing operation of the detection model is non-maximum suppression (NMS). Since the model in the present invention can output both rectangular frames and tilted frames at the same time, the original NMS algorithm cannot be directly used. In the modified NMS algorithm of the present invention, the rectangular frame is used as the main input. After obtaining the reserved index value after the rectangular frame NMS, the corresponding tilted frame will be obtained based on this index value, so that the post-processed rectangular frame and the tilted frame can be obtained at the same time by simple modification. Compared with directly using the tilted frame as input to perform NMS based on the tilted frame, the modification in the present invention has the advantages of fast and high efficiency, and due to the sparsity of the detection target, the accuracy of the result can also be guaranteed. After post-processing, the final tilted frame output, that is, the position information (x, y, w, h, θ) of the door will be obtained.

[0080] This technical solution can be applied to elevator door detection during the robot's elevator riding process. Generally speaking, when a robot needs to take an elevator, it will first determine whether it is at the elevator door based on its global positioning, and then make a decision to take the elevator. However, due to errors caused by many factors such as sensors and data conversion in actual situations, there may be a large deviation between the global positioning and the actual position, which will have a great impact on the robot's elevator entry action. Therefore, the elevator door position information obtained by this technical solution can be used to assist the robot's elevator entry process. Figure 5This is an example diagram of the detection results of elevator doors at any angle in the present technical solution. The green box in the figure is the detection result. It can be seen from the figure that no matter whether the robot is facing the elevator door in a frontal posture or tilted at a certain angle, the present technical solution can obtain very good detection effects, and its angle is consistent with the actual angle of the elevator door. This result can effectively overcome the situation where the robot fails to enter the elevator due to global positioning deviation, and further reduce the losses caused by accidents such as the robot hitting the wall and taking the wrong elevator due to global positioning deviation.

[0081] In this embodiment, the position of the laser gate can be accurately detected, and the detection result contains detailed length, width and angle information, which greatly increases the usability of the laser gate information. At the same time, due to the richness of the state information, it has a good promoting effect on downstream tasks. Figure 5-Figure 6 As shown, Figure 5-Figure 6 The left frame (a) in the figure is other methods, and the right frame (b) is the result of this technical solution. As can be seen from the figure, this technical solution can accurately detect the location of the door and obtain information such as its direction angle.

[0082] The data in the comparison chart are for laser elevator doors, but this technical solution is not limited to elevator doors. Other doors such as hotel room doors, conference room doors, etc. are also within the scope of application of this technical solution.

[0083] Figure 3 A circuit structure diagram of a controller is provided. The controller includes one or more processors and a memory. One processor is taken as an example.

[0084] The processor and the memory may be connected via a bus or other means, with connection via a bus being taken as an example.

[0085] As a non-volatile computer-readable storage medium, the memory can be used to store non-volatile software programs, non-volatile computer executable programs and modules, such as the program instructions / modules corresponding to the robot's arbitrary-angle laser door detection method based on a convolutional neural network in the embodiment of the present invention. The processor executes various functional applications and data processing of the robot's arbitrary-angle laser door detection device based on a convolutional neural network by running the non-volatile software programs, instructions and modules stored in the memory, that is, realizes the robot's arbitrary-angle laser door detection method based on a convolutional neural network provided by the above method embodiment and the functions of each module or unit of the above device embodiment.

[0086] The memory may include a high-speed random access memory and may also include a non-volatile memory, such as at least one disk storage device, a flash memory device, or other non-volatile solid-state storage device. In some embodiments, the memory may optionally include a memory remotely arranged relative to the processor, and these remote memories may be connected to the processor via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0087] The program instructions / modules are stored in the memory, and when executed by the one or more processors, the arbitrary-angle laser door detection method based on a convolutional neural network for a robot in any of the above method embodiments is executed.

[0088] An embodiment of the present invention also provides a non-volatile computer storage medium, which stores computer executable instructions. The computer executable instructions are executed by one or more processors, so that the one or more processors can execute the robot's arbitrary-angle laser door detection method based on a convolutional neural network in any of the above method embodiments.

[0089] An embodiment of the present invention also provides a computer program product, which includes a computer program stored on a non-volatile computer-readable storage medium, and the computer program includes program instructions. When the program instructions are executed by an electronic device, the electronic device executes any one of the robot's arbitrary-angle laser door detection methods based on a convolutional neural network.

[0090] The above described device or equipment embodiments are merely illustrative, wherein the unit modules described as separate components may or may not be physically separated, and the components displayed as module units may or may not be physical units, that is, they may be located in one place, or may be distributed on multiple network module units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the present embodiment.

[0091] Through the description of the above implementation methods, those skilled in the art can clearly understand that each implementation method can be implemented by means of software plus a general hardware platform, and of course, by hardware. Based on this understanding, the above technical solution is essentially or the part that contributes to the relevant technology can be embodied in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a disk, an optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or some parts of the embodiments.

[0092] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit them. Under the concept of the present invention, the technical features in the above embodiments or different embodiments may also be combined, the steps may be implemented in any order, and there are many other changes in different aspects of the present invention as described above, which are not provided in detail for the sake of simplicity. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they may still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for detecting an arbitrary angle laser door based on a convolutional neural network, characterized in that: The steps include: S1, obtaining raw laser data; S2, converting the original laser data into a laser image; S3, using a detection model to detect the laser image and extract a characteristic curve of a door in the laser image; S4, marking the characteristic curve of the door with a rectangular frame or / and an inclined frame, and obtaining position information of the rectangular frame or / and the inclined frame, wherein the position information includes center point, length, width and angle information; The step S2 specifically includes: S201, laser coordinate conversion; S202, grayscale image filling; The step S201 includes converting the distance information into coordinate information. The formula for converting the polar coordinate laser data into Cartesian coordinate data is as follows: x = r × cosθ; y = r × sinθ; Where r is the distance from the center of the rectangular frame to the origin of the coordinate system, x,y represents the position of the center of the rectangular frame in Cartesian coordinates, and θ is the angle at which the laser is emitted; It is necessary to further fill the coordinate values ​​into the grayscale image according to the actual usage ratio t. When filling, the coordinate positions scaled according to the ratio t are filled with 255, and the remaining positions are 0, so that an actually usable laser image can be obtained; The step S3 specifically includes: Step S301, image preprocessing; Step S302: model processing; Step S303, result post-processing; The step S301 specifically includes: the laser image needs to be subjected to certain normalization and data augmentation image preprocessing operations before being input into the model, wherein the normalization needs to normalize 0 and 255 on the laser image to -0.5 and 0.5 respectively, and the data augmentation operation includes random flipping and rotation operations; The step S302 specifically includes: the detection model uses YOLO-v3 as a benchmark model, and the output layer of the YOLO-v3 is modified accordingly; The detection of the tilted box (w, h, θ) is added to the last regression layer. The regression target of the tilted box is directly calculated based on the original anchor box, and the center point (x, y) is the same as the rectangular box. The step S302 specifically includes: in the model training stage, the difference between the predicted value and the true value is characterized by a loss function. The length or width of the tilted box uses the same L2 loss as the rectangular box as the loss function, while the angle θ uses the sine function sin as the loss function. The specific function is as follows: L2=(x gt -x pred ) 2 ; L θ =sin(|θ gt -θ pred |); Among them, x gt is the length or width of the tilted box, x pred is the length or width of the rectangular box, θ gt is the angle of the tilted frame, θ pred is the angle of the rectangular box; The step S303 specifically includes: the post-processing operation of the detection model is non-maximum suppression; Get the retained index value after non-maximum suppression of the rectangular frame; Based on the index value, a corresponding tilted frame is obtained and a post-processed rectangular frame and a tilted frame are obtained; The position information (x, y, w, h, θ) of the inclined frame is acquired based on the inclined frame.

2. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the arbitrary-angle laser door detection method based on a convolutional neural network as described in claim 1 is implemented.

Images