Vision-based escalator recognition and processing method and device

Abstract

The present invention discloses a vision-based escalator recognition and processing method and device. In this method, a vision sensor is used to obtain vision sensor data; when it is determined according to the vision sensor data that the category information of the current target object is an escalator, the area information of the escalator is obtained; from the distance information obtained by the ranging device, the first position information corresponding to the position points collected by the vision sensor is extracted; the area information and each of the first position information are unified into a coordinate system, and the second position information corresponding to each position point in the area information is extracted; the second position information is analyzed according to the characteristic information of the escalator to obtain the escalator distance information, the position information of the escalator is determined according to the escalator distance information, and corresponding decision-making processing is performed according to the position information of the escalator. By adopting this solution, it is possible to quickly respond to emergencies in dangerous scenarios in a timely manner, and greatly improve the safety during the operation of the robot.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence, and in particular, to a vision-based escalator recognition and processing method and device. Background Art

[0002] With the increasing maturity of artificial intelligence technology, intelligent devices have gradually entered people's lives. The artificial intelligence technology has begun to exert continuous efforts to improve work efficiency through an automated mode, thereby liberating the labor force.

[0003] During the operation of an automated device (for example, a robot), it is necessary to monitor the surrounding environmental state in real time. In particular, for a target object, such as an escalator, it is necessary to be able to identify and adopt corresponding decision-making and processing solutions. If the robot fails to effectively identify the escalator, the robot may step onto the escalator steps and is easily carried away by the escalator, resulting in dangerous events such as falling and tipping over, thereby causing safety problems.

[0004] In related technologies, for dangerous scenarios caused by escalators (such as a falling scenario, a tipping-over scenario, etc.), mainly including: Solution 1, using a drop sensor to identify an area with drop characteristics; Solution 2, using a depth sensor to obtain the three-dimensional coordinates of an object in a scene, and then analyzing to obtain an area with drop characteristics. The areas detected by the above two solutions are used as dangerous scenario areas and provided to a control module for decision-making and processing.

[0005] However, the drop sensor can only identify at a short distance, which brings troubles to the movement and control of the robot. The distance is too short to make a decision in time; the depth sensor can predict and identify a cliff-like falling scenario, but it cannot handle dangerous scenarios such as escalators because there is an active area at the front end of the escalator that is level with the ground, and the above two solutions cannot detect it; therefore, whether for an upward escalator scenario or a downward escalator scenario, using a drop sensor and a depth sensor cannot effectively avoid the occurrence of safety problems. Summary of the Invention

[0006] The main object of the present invention is to disclose a vision-based escalator recognition and processing method and device, so as to at least solve the problems in related technologies that for dangerous scenarios caused by escalators, solutions such as using a drop sensor and a depth sensor have problems such as too short a distance to make a decision in time or inability to detect and identify.

[0007] According to one aspect of the present invention, a vision-based escalator recognition and processing method is provided.

[0008] The vision-based escalator recognition and processing method according to the present invention includes: acquiring vision sensor data by using a vision sensor; when determining that the category information of the current target object is an escalator according to the vision sensor data, acquiring the area information of the escalator; extracting, from the distance information acquired by the ranging device, the first position information corresponding to the position points collected by the vision sensor; unifying the area information and each of the first position information into a coordinate system, and extracting the second position information corresponding to each position point in the area information; analyzing the second position information according to the feature information of the escalator, acquiring the escalator distance information, determining the position information of the escalator according to the escalator distance information, and performing corresponding decision-making processing according to the position information of the escalator.

[0009] On the other hand, according to the present invention, a vision-based escalator recognition and processing device is provided.

[0010] The vision-based escalator recognition and processing device according to the present invention includes: a first acquisition module for acquiring vision sensor data by using a vision sensor; a second acquisition module for acquiring the area information of the escalator when determining that the category information of the current target object is an escalator according to the vision sensor data; a first extraction module for extracting, from the distance information acquired by the ranging device, the first position information corresponding to the position points collected by the vision sensor; a second extraction module for unifying the area information and each of the first position information into a coordinate system, and extracting the second position information corresponding to each position point in the area information; a processing module for analyzing the second position information according to the feature information of the escalator, acquiring the escalator distance information, determining the position information of the escalator according to the escalator distance information, and performing corresponding decision-making processing according to the position information of the escalator.

[0011] According to the present invention, vision sensor data is acquired by using a vision sensor; when determining that the category information of the current target object is an escalator according to the vision sensor data, the area information of the escalator is acquired; the first position information corresponding to the position points collected by the vision sensor is extracted from the distance information acquired by the ranging device; the area information and each of the first position information are unified into a coordinate system, and the second position information corresponding to each position point in the area information is extracted; the second position information is analyzed according to the feature information of the escalator, the escalator distance information is acquired, the position information of the escalator is determined according to the escalator distance information, and corresponding decision-making processing is performed according to the position information of the escalator. The problem that only short-distance recognition can be performed by using a drop sensor and it may be too late to make a decision when the distance is too short is solved, and the problem that a depth sensor cannot handle dangerous scenarios such as escalators is also solved. By combining the vision sensor and the ranging device, the position information of the escalator can be accurately acquired, and the sudden situation in a dangerous scenario caused by the escalator can be timely and quickly responded to, which greatly improves the safety during the operation of the robot. Description of the Drawings

[0012] Figure 1 is a flowchart of a vision-based escalator recognition method according to an embodiment of the present invention;

[0013] Figure 2 is a schematic diagram of recognizing area information of an escalator according to a preferred embodiment of the present invention;

[0014] Figure 3 is a schematic diagram of determining the position information of an escalator according to a preferred embodiment of the present invention;

[0015] Figure 4 is a schematic diagram of determining the position information of an escalator according to an example of the present invention;

[0016] Figure 5 is a schematic diagram of map management according to a preferred embodiment of the present invention;

[0017] Figure 6 is a schematic diagram of map update according to a preferred embodiment of the present invention;

[0018] Figure 7 is a schematic diagram of an escalator recognition and decision-making method according to a preferred embodiment of the present invention;

[0019] Figure 8 is a structural block diagram of a vision-based escalator recognition device according to an embodiment of the present invention;

[0020] Figure 9 is a structural block diagram of a vision-based escalator recognition device according to a preferred embodiment of the present invention. Detailed implementation manners

[0021] The following provides a detailed description of the specific implementation manners of the present invention with reference to the accompanying drawings of the specification.

[0022] According to an embodiment of the present invention, a vision-based escalator recognition method is provided.

[0023] Figure 1 is a flowchart of a vision-based escalator recognition method according to an embodiment of the present invention. As Figure 1 shown, the escalator recognition method includes:

[0024] Step S101: Obtain vision sensor data by using a vision sensor;

[0025] Step S102: When determining that the category information of the current target object is an escalator according to the vision sensor data, obtain the area information of the escalator;

[0026] Step S103: Extract first position information corresponding to the position points collected by the vision sensor from the distance information obtained by the ranging device;

[0027] Step S104: Unify the region information and each of the first position information into a coordinate system, and extract the second position information corresponding to each position point in the region information;

[0028] Step S105: Analyze the second position information according to the feature information of the escalator to obtain the escalator distance information, determine the position information of the escalator according to the escalator distance information, and perform corresponding decision-making processing according to the position information of the escalator.

[0029] According to Figure 1 the method shown above, use a vision sensor to obtain vision sensor data; when determining that the category information of the current target object is an escalator according to the vision sensor data, obtain the region information of the escalator; use a ranging device to obtain the first position information corresponding to each position point in the above region information; unify the above region information and each of the above first position information into a coordinate system, and extract the second position information corresponding to each position point in the region information; analyze the second position information according to the feature information of the escalator to obtain the escalator distance information, determine the position information of the escalator according to the escalator distance information, and perform corresponding decision-making processing according to the position information of the escalator. This solves the problem that the fall sensor can only identify at a short distance, and the distance is too short to make a decision in time, and also solves the problem that the depth sensor cannot handle dangerous scenarios such as escalators. Therefore, by combining the vision sensor and the ranging device, the position information of target objects such as escalators can be accurately obtained, and corresponding decision-making processing (such as detouring, turning back, etc.) can be performed. Timely and quickly respond to emergencies in dangerous scenarios, and greatly improve safety during the operation of the robot.

[0030] Among them, the above vision sensor can be a camera, etc., and the above vision sensor data can be a picture, or it can also be a point cloud, etc. The category information of the above target object refers to the type of the target object. For example, people, vehicles, chairs, escalators, animals, etc. The present invention relates to an identification and processing solution for escalators.

[0031] As Figure 2 shown, turn on the camera and read the camera data (for example, a picture); use a neural network to Figure 2 detect from the picture shown above that the target object is an escalator, and the region information of the escalator; for example, send the picture (for example, the picture size can be 416*416) into a neural network (which can be a target detection neural network or an instance segmentation network), and the region information and category information of the escalator are extracted through the network.

[0032] Preferably, the ranging device includes at least one of the following: a two-dimensional ranging device, a three-dimensional ranging device.

[0033] That is, a two-dimensional ranging device solution, a three-dimensional ranging device solution, or a combination of a two-dimensional ranging device and a three-dimensional ranging device can be adopted.

[0034] Among them, the above ranging device can be a two-dimensional ranging device (for example, a lidar sensor). The laser emitted by the two-dimensional ranging device is no longer a pulsed signal but a continuous laser, and the result read by the receiver also becomes a set of continuous values. In this way, without moving the rangefinder, the continuous distance numbers of objects in a plane (i.e., the plane scanned by the lateral movement of the rangefinder) can be continuously scanned, and the returned values can be used to construct a planar graph.

[0035] Preferably, in the above step S103, obtaining the first position information corresponding to each position point in the above area information by using the ranging device may further include: turning on the above two-dimensional ranging device, and extracting, from the distance information obtained by the ranging device, the first position information corresponding to the position points collected by the visual sensor.

[0036] In the preferred implementation process, as Figure 3 shown, a two-dimensional ranging device (for example, a lidar for single-line ranging) is used for data calculation. Turn on the sensor, and according to the first position information corresponding to the position points collected within the field of view of the visual sensor:

[0037] D = Distance (x，y)

[0038] Unify each of the above first position information and the area information of the escalator (for example, the position point information within a rectangular frame area) into the same coordinate system (for example, under the world coordinate system), and extract the second position information corresponding to each position point in the area information; among them, the second position information includes multiple two-dimensional data.

[0039] It should be noted that there may be noise data among the above multiple two-dimensional data. Therefore, it is necessary to analyze the above second position information according to the characteristic information of the escalator to remove the noise data. For example, analyze the edge area information of the escalator according to the area characteristics (such as handrail characteristics), and remove the noise data from the second position information to obtain the distance analysis result.

[0040] After that, obtain the escalator distance information according to the distance analysis result, and determine the position information of the escalator according to the escalator distance information.

[0041] Among them, the above distance measurement device can also be a three-dimensional distance measurement device (for example, a depth camera, a multi-view stereo matching sensor, a TOF sensor, etc.). The three-dimensional distance measurement device emits continuous pulses to the target scene, and then uses the sensor to receive the light pulses reflected back from the object surface. By comparing the phase difference between the emitted light pulse and the light pulse reflected by the object, the transmission delay between the light pulses can be deduced, and then the distance of the object relative to the transmitter can be obtained.

[0042] In the preferred implementation process, as Figure 3 shown, data calculation is performed using a three-dimensional distance measurement device data calculation (for example, a depth camera, a multi-view stereo matching sensor, a TOF sensor, etc.). The sensor is turned on, and according to the first position information corresponding to the position points collected within the field of view of the visual sensor:

[0043] D = Distance (x，y，z)

[0044] Unify each of the above first position information and the area information of the escalator (for example, the position point information within the rectangular frame area) into the same coordinate system (for example, under the world coordinate system); extract the second position information corresponding to each position point in the area information; where the second position information is composed of a series of three-dimensional data points Point(x, y, z).

[0045] It should be noted that there may be noise data in the above-mentioned multiple three-dimensional data. Therefore, it is necessary to analyze the above second position information according to the characteristic information of the escalator to remove the noise data. For example, analyze the escalator edge area information according to the area characteristics (such as handrail characteristics), and remove the noise data from the second position information (the above three-dimensional data points) to obtain the distance analysis result.

[0046] After that, the escalator distance information is obtained according to the distance analysis result, and the position information of the escalator is determined according to the escalator distance information.

[0047] Preferably, when the above distance measurement device includes both a two-dimensional distance measurement device and a three-dimensional distance measurement device at the same time, determining the position information of the escalator according to the escalator distance information can include the following two processing methods:

[0048] Method 1: Perform position distance calculation on the position information of the escalator obtained by using the two-dimensional distance measurement device solution and the position information of the escalator obtained by using the three-dimensional distance measurement device solution; compare the position distance calculation result with a first preset threshold, and determine the position information of the escalator obtained by using the two-dimensional distance measurement device solution or the three-dimensional distance measurement device according to the comparison result as the final position information.

[0049] In a preferred implementation process, when the calculated position distance is less than or equal to the first preset threshold, when the three-dimensional ranging device is a depth camera or a TOF sensor or other devices, the position information of the escalator obtained by the three-dimensional ranging device solution can be used as the final position information; when the three-dimensional ranging device is a multi-view stereo matching or other devices, the position information of the escalator obtained by the two-dimensional ranging device solution can be used as the final position information; when the position distance is greater than the first preset threshold, the position information of the escalator obtained by the two-dimensional ranging device solution (for example, a lidar sensor) is used as the final position information.

[0050] Method 2: Calculate the distance difference between the escalator distance information obtained by the two-dimensional ranging device solution and the escalator distance information obtained by the three-dimensional ranging device solution; compare the calculated distance difference result with the second preset threshold, and determine the escalator distance information obtained by the two-dimensional ranging device solution or the three-dimensional ranging device according to the comparison result as the final escalator distance information.

[0051] In a preferred implementation process, when the distance difference is less than or equal to the second preset threshold, when the three-dimensional ranging device is a depth camera or a TOF sensor or other devices, the escalator distance information obtained by the three-dimensional ranging device solution is used as the final escalator distance information; when the three-dimensional ranging device is a multi-view stereo matching or other devices, the escalator distance information obtained by the two-dimensional ranging device solution can be used as the final escalator distance information; when the distance difference is greater than the second preset threshold, the escalator distance information obtained by the two-dimensional ranging device solution is used as the final escalator distance information.

[0052] The two-dimensional ranging device (for example, a lidar sensor) usually has the advantages of long-distance measurement and high accuracy; the three-dimensional ranging device has an advantage in anti-occlusion because three-dimensional information is collected; therefore, by combining the two-dimensional ranging device and the three-dimensional ranging device, high-precision distance recognition with long-distance anti-interference can be achieved. The two-dimensional ranging device solution and the three-dimensional ranging device solution obtain two position results or distance results; calculate the distance between the two positions or the difference between the two distances. When the calculated result is greater than the preset value, it can be considered that serious occlusion has occurred, so the result of the three-dimensional ranging device solution is adopted; when the calculated result is less than the threshold, it can be considered that the two ranging results are basically the same, and the result of the two-dimensional ranging device solution is more accurate.

[0053] Preferably, analyzing the second position information according to the characteristic information of the escalator includes: determining whether the second position information meets the characteristic conditions of the escalator.

[0054] In a preferred implementation process, determining whether the second position information meets the characteristic conditions of the escalator may include, but is not limited to, at least one of the following:

[0055] Determine whether the obtained edge region width information is within a first preset range of the escalator handrail width;

[0056] Determine whether the obtained edge region spacing information is within a second preset range of the escalator handrail spacing;

[0057] Determine whether the obtained edge region curvature information is within a third preset range of the escalator handrail curvature;

[0058] Determine whether the obtained edge region gradient information is within a fourth preset range of the escalator gradient.

[0059] It should be noted that the above first preset range, second preset range, third preset range, and fourth preset range are all ranges determined in advance according to the actual escalator standards.

[0060] Preferably, obtaining the escalator distance information may further include: analyzing the second position information according to the characteristic information of the escalator to obtain a distance analysis result, and using the mean value information of the distance analysis result as the escalator distance information.

[0061] Preferably, obtaining the escalator distance information may further include the following processing:

[0062] S1: Analyze the second position information according to the characteristic information of the escalator to obtain a distance analysis result, and determine the maximum distance value and the minimum distance value from the distance analysis result;

[0063] S2: Set the above minimum distance value as the initial threshold;

[0064] S3: For the first region where the distance value in the distance analysis result is greater than the initial threshold, calculate the variance and area of the first region. For the second region where the distance value in the distance analysis result is less than the initial threshold, calculate the variance and area of the second region respectively;

[0065] S4: Perform an increasing operation on the initial threshold successively, loop and execute S3. According to the variances and areas corresponding to multiple regions obtained, calculate and obtain the effective region according to the variances and areas corresponding to multiple regions, and use the distance information corresponding to the effective region as the escalator distance information.

[0066] As Figure 3 and Figure 4 shown, it is necessary to perform numerical fusion calculation on the cropped target object region to obtain the escalator distance information, and determine the position information of the escalator according to the escalator distance information.

[0067] Taking the numerical fusion calculation of a two-dimensional position area as an example for illustration:

[0068] This two-dimensional position area is composed of a series of two-dimensional points D(x, y), such as a two-dimensional array, where x and y respectively represent the two dimensions of the coordinate system, and D represents the distance information of this point. Points without distance information are invalid points and are ignored here; for the situation of local occlusion of the object and some noise, the fusion algorithm of the present invention can better solve this problem:

[0069] 1. Analyze the second position information according to the characteristic information of the escalator to obtain a distance analysis result, and determine the maximum value and the minimum value from the distance analysis result;

[0070] 2. Set the initial threshold as the above minimum value;

[0071] 3. Divide the above area into two parts, namely the part where the distance value is less than the threshold and the part where the value is greater than or equal to the threshold; they are respectively denoted as R s and R l ;

[0072] 4. Calculate the mean value, variance and area for the two parts of the area respectively;

[0073] Among them, for the area of R s , that is, the area (set of points) where the distance value is less than the threshold, the mean value Ms is as follows: where Ns is the number of two-dimensional points, Di is the distance information corresponding to each two-dimensional point, and i = 1, 2, 3,... Ns;

[0074] The variance Var s is as follows:

[0075] where Ms is the mean value, Ns is the number of two-dimensional points, Di is the distance information corresponding to each two-dimensional point, and i = 1, 2, 3,... Ns;

[0076] The area is N S , that is, one point is regarded as a unit area, and the number of points is regarded as the area value.

[0077] Similarly, for the area of R l , that is, the area (set of points) where the distance value is greater than or equal to the threshold, the mean value M l is as follows: where N l is the number of two-dimensional points, Di is the distance information corresponding to each two-dimensional point, and i = 1, 2, 3,... N l ;

[0078] The variance Var l is as follows:

[0079] Among them, M l is the mean value, N l is the number of two-dimensional points, Di is the distance information corresponding to each two-dimensional point, and i = 1, 2, 3,..., N l ;

[0080] The area is N l , that is, one point is regarded as a unit area, and the number of points is regarded as the area value.

[0081] It should be noted that the three-dimensional position area numerical fusion calculation method is similar to the above method, and will not be elaborated here.

[0082] In the preferred implementation process, any one of the above two schemes can be used to determine the second position information corresponding to the second escalator area. For the first scheme, the calculation amount is small, it is simple and easy to implement, and since the escalator features have been used to analyze the first escalator area and the noise data has been removed, using the first scheme can basically meet the accuracy requirements. For the second scheme, variance and area are used to obtain the effective area, and the distance information corresponding to the effective area is used as the above-mentioned second position information, further filtering out the noise data and making the calculation result more accurate.

[0083] Preferably, after determining the position information of the escalator according to the escalator distance information, at least one of the following may further be included:

[0084] If there is an escalator set in the current map, determine whether the weight value of the escalator set is greater than a preset weight threshold. If the weight value of the escalator set is less than or equal to the preset weight threshold, delete the set;

[0085] Under the condition of meeting the predetermined map update condition, perform a map update operation on the current map according to the position information of the escalator;

[0086] Under the condition of meeting the predetermined correction condition, for example, when the machine working environment changes, etc., trigger a map position correction operation, where the map position correction operation includes: correcting the coordinates and distance information of each set and the elements inside the set in the current map.

[0087] Preferably, under the condition of meeting the predetermined map update condition, performing a map update operation on the current map according to the position information of the escalator may further include: reading the information of the current map. If there is no associated escalator set in the current map (for example, there is no escalator set in the current map, or each escalator set in the current map is not associated with the position information of the escalator), add a new set in the current map according to the position information of the escalator; when there is an associated escalator set in the current map, perform a set update operation on the associated escalator set according to the position information of the escalator.

[0088] Among them, the above escalator set can be a set of points, a set of line segments, or a set of continuous regions.

[0089] Send the position information of the escalator to the map management module for map building; where the position information can be either a set of points, or a region or a line; the following takes the region as an example for illustration;

[0090] Such as Figure 5 shown, the map management steps include:

[0091] 1. At startup, perform map initialization, including reading the existing offline map from the computer;

[0092] 2. Adopt the above-mentioned escalator recognition and processing method based on vision to obtain a new escalator region;

[0093] 3. Send the escalator region to the update module to perform map update;

[0094] 4. Execute the map addition and deletion strategy;

[0095] (a) When there is no associated escalator region in the map, add the newly obtained escalator region to the map;

[0096] (b) When the weight of the escalator region is less than or equal to the first preset threshold, delete the escalator region; otherwise, perform map update;

[0097] (c) Judge whether the map update is successful. If it fails (for example, each escalator set in the current map is not associated with the position information of the escalator), add the new region to the map.

[0098] 5. When the working environment of the machine changes, trigger the execution of the map position correction operation; the correction operation includes: correcting each escalator set in the map and the coordinate information related to the escalator set;

[0099] 6. After the robot finishes executing the task, save the current escalator map.

[0100] Such as Figure 6 shown, taking the position information of the escalator as an example of a region and the escalator set as an example of a continuous region, the map update process is as follows:

[0101] 1. Read the region information in the current map and the newly obtained escalator region information;

[0102] 2. Take a region from the map and calculate the correlation value between the current region and the new region. The calculation of the correlation value can use the intersection-over-union method (i.e., intersection divided by union); if the correlation value is greater than the second preset threshold, increase the weight value of the current region and update the region, that is, obtain the union of the above new region and the above current region; otherwise, decrease the weight value of the current region.

[0103] 3. The process ends, or repeat the above step 2 until all regions in the map have been updated.

[0104] The following is combined with Figure 7 to further describe the above preferred embodiment.

[0105] Figure 7 is a schematic diagram of the escalator recognition and decision-making method according to the preferred embodiment of the present invention. As Figure 7 shown, the escalator recognition and decision-making method includes:

[0106] Step 1: Obtain the information of the escalator in the image;

[0107] For example, turn on the camera and read the camera data (i.e., the picture); send the picture (e.g., with a size of 416*416) into the neural network (which can be an object detection neural network or an instance segmentation network). When the category information of the target object is determined to be an escalator through the neural network, obtain the region information of the escalator.

[0108] Step 2: Obtain the position information of the escalator.

[0109] (1) Solution 1: Use a two-dimensional ranging device (such as a single-line lidar)

[0110] Use a two-dimensional ranging device to obtain the first position information corresponding to each position point in the region information of the escalator; unify the above region information and each of the above first position information into a coordinate system, extract the second position information corresponding to each position point in the region information, which is composed of a series of two-dimensional points Point(x, y); analyze the second position information according to the characteristic information of the escalator to obtain the escalator distance information, and determine the position information of the escalator according to the escalator distance information.

[0111] Among them, the above-mentioned obtaining of the escalator distance information can use two methods.

[0112] The first: Analyze the second position information according to the characteristic information of the escalator to obtain a distance analysis result, and use the mean information of the distance analysis result as the escalator distance information.

[0113] The second: mainly includes the following processing:

[0114] S1: Analyze the second position information based on the characteristic information of the escalator to obtain a distance analysis result, and determine the maximum distance value and the minimum distance value from the distance analysis result;

[0115] S2: Set the above minimum distance value as the initial threshold;

[0116] S3: For the first region where the distance value in the distance analysis result is greater than the initial threshold, calculate the variance and area of the first region. For the second region where the distance value in the distance analysis result is less than the initial threshold, calculate the variance and area of the second region respectively;

[0117] S4: Perform an increasing operation on the initial threshold successively, loop and execute S3. According to the variances and areas corresponding to multiple regions obtained, calculate and obtain an effective region based on the variances and areas corresponding to multiple regions, and use the distance information corresponding to the effective region as the escalator distance information.

[0118] (2) Solution 2: Use a three-dimensional ranging device (such as a depth camera, a multi-view stereo matching sensor, a 3D TOF sensor)

[0119] Use a three-dimensional ranging device to obtain the first position information corresponding to each position point in the regional information of the escalator; unify the above regional information and each of the above first position information into a coordinate system, and extract the second position information corresponding to each position point in the regional information, which is composed of a series of three-dimensional points Point(x, y, z); analyze the second position information based on the characteristic information of the escalator to obtain the escalator distance information, and determine the position information of the escalator according to the escalator distance information.

[0120] Among them, the principle of obtaining the second position information corresponding to the second escalator region is the same as above and will not be elaborated here.

[0121] (3) Solution 3: Combine the second position information obtained from Solution 1 and Solution 2 to achieve more accurate distance calculation.

[0122] Specifically, obtain the position information or the results of two distance information of the two escalators from Solution 1 and Solution 2; calculate the position distance or the difference between the two distances of the two escalators, compare the calculated result of the position distance with the first preset threshold, and determine the position information of the escalator obtained by using the two-dimensional ranging device scheme or the three-dimensional ranging device as the final position information of the escalator according to the comparison result. Compare the calculated result of the distance difference with the second preset threshold, and determine the distance information of the escalator obtained by using the two-dimensional ranging device scheme or the three-dimensional ranging device as the final distance information of the escalator according to the comparison result. For example, when the position distance is less than or equal to the first preset threshold, it is considered that there is serious occlusion, so the result of Solution 2 is taken; when the distance difference is greater than the second preset threshold, it is considered that the ranging results of the two are basically the same, and the result of Solution 1 is more accurate.

[0123] Step 3: Map management (including map update)

[0124] For example, read the information of the current map. If there is no associated escalator set in the current map, add a new set in the current map according to the position information of the escalator; if there is an escalator set in the current map, determine whether the weight value of the escalator set is greater than the preset weight threshold (thresh1). If the weight value of the escalator set is less than or equal to the preset weight threshold (thresh1), delete the set. If the weight value of the escalator set is greater than the preset weight threshold (thresh1), perform an update operation on the current map according to the position information of the escalator. If the predetermined correction condition is satisfied, trigger a map position correction operation, where the map position correction operation includes: correcting each escalator set in the current map and the coordinate information related to the escalator set.

[0125] Among them, performing a set (taking a continuous area as an example) update operation on the current map according to the position information of the escalator can further include (taking the position information of the escalator as an area and the escalator set as a set of continuous areas for illustration): obtain an escalator area from the current map, obtain the association degree value between the current area and the new area (such as the intersection over union ratio. For example, if the intersection is 5 grids and the union is 10 grids, then the intersection over union ratio is 0.5). If the association degree value is greater than the second preset threshold (thresh2), increase the weight value of the current area, update the current area according to the new area, obtain the union of the new area and the current area, and then the process ends. Or, loop through the above steps until all escalator areas in the current map are traversed.

[0126] Step 4: Feed the semantic map obtained in Step 3 back to the control system for decision-making and planning.

[0127] For example, when the category of the target object is identified as an escalator and the specific position information of the escalator is also obtained, corresponding operations such as detouring or turning back can be determined based on these two pieces of information to avoid dangerous situations, thereby greatly improving safety during the operation of the robot.

[0128] According to an embodiment of the present invention, there is also provided a vision-based escalator recognition and processing device.

[0129] Figure 8 It is a structural block diagram of a vision-based escalator recognition and processing device according to an embodiment of the present invention. As Figure 8 shown, the vision-based escalator recognition and processing device includes: a first acquisition module 80 for acquiring vision sensor data using a vision sensor; a second acquisition module 82 for acquiring the area information of the escalator when the category information of the current target object is determined to be an escalator according to the vision sensor data; a first extraction module 84 for extracting the first position information corresponding to the position points collected by the vision sensor from the distance information obtained by the ranging device; a second extraction module 86 for unifying the area information and each of the first position information into a coordinate system and extracting the second position information corresponding to each position point in the area information; a processing module 88 for analyzing the second position information according to the feature information of the escalator, obtaining the escalator distance information, determining the position information of the escalator according to the escalator distance information, and performing corresponding decision-making processing according to the position information of the escalator.

[0130] Using Figure 8 the device shown, the first acquisition module 80 acquires vision sensor data using a vision sensor; the second acquisition module 82 acquires the area information of the escalator when the category information of the current target object is determined to be an escalator according to the vision sensor data; the first extraction module 84 extracts the first position information corresponding to the position points collected by the vision sensor from the distance information obtained by the ranging device; the second extraction module 86 unifies the area information and each of the first position information into a coordinate system and extracts the second position information corresponding to each position point in the area information; the processing module 88 analyzes the second position information according to the feature information of the escalator, obtains the escalator distance information, determines the position information of the escalator according to the escalator distance information, and performs corresponding decision-making processing according to the position information of the escalator. It solves the problem that only short-distance recognition can be performed using a drop sensor, and the distance is too short to make a decision in time, and also solves the problem that a depth sensor cannot handle dangerous scenarios such as escalators. Therefore, by combining a vision sensor and a ranging device, the position information of target objects such as escalators can be accurately obtained, and corresponding decision-making processing (such as detouring, turning back, etc.) can be performed. Timely and quickly respond to emergencies in dangerous scenarios, and greatly improve safety during the operation of the robot.

[0131] Preferably, asFigure 9 As shown, the above device may further include, but is not limited to, a map management module 90, where the map management module 90 includes at least one of the following:

[0132] A deletion unit ( Figure 9 not shown in the figure) for determining whether the weight value of the escalator set is greater than a preset weight threshold when there is an escalator set in the current map. If the weight value of the escalator set is less than or equal to the preset weight threshold, the escalator set is deleted;

[0133] An update unit ( Figure 9 not shown in the figure) for performing a map update operation on the current map according to the position information of the escalator under the condition of meeting a predetermined map update condition; in a preferred implementation process, the update unit is used to read the information of the current map. If there is no associated escalator set in the current map, a new set is added to the current map according to the position information of the escalator; when there is an associated escalator set in the current map, a set update operation is performed on the associated escalator set according to the position information of the escalator.

[0134] A correction unit ( Figure 9 not shown in the figure) for triggering a map position correction operation under the condition of meeting a predetermined correction condition, where the map position correction operation includes: correcting each escalator set in the current map and the coordinate information related to the escalator set.

[0135] It should be noted that for the preferred implementation manners of the combination of the various modules in the escalator identification device, specific reference may be made to Figures 1 to 7 the corresponding relevant descriptions and effects in the embodiments shown, which will not be elaborated here.

[0136] In summary, by means of the implementation manner provided by the present invention, the combination of the visual sensor and the ranging device can accurately obtain the category and position of the escalator, so as to perform corresponding decision-making processes (such as avoidance, detour, etc.), avoid the robot from stepping onto the escalator steps during operation, thereby preventing dangerous events such as falling and tipping, and can timely and quickly respond to emergencies caused by escalators, greatly improving the safety during the operation of the robot.

[0137] The above discloses only several specific embodiments of the present invention. However, the present invention is not limited thereto, and any changes that can be thought of by those skilled in the art should fall within the protection scope of the present invention.

Claims

1. A vision-based escalator recognition and processing method, characterized in that, it includes: Using a vision sensor to obtain vision sensor data; When determining that the category information of the current target object is an escalator according to the vision sensor data, obtaining the area information of the escalator; Extracting the first position information corresponding to the position points collected by the vision sensor from the distance information obtained by the ranging device; Unifying the area information and each of the first position information into a coordinate system, and extracting the second position information corresponding to each position point in the area information; Analyzing the second position information according to the characteristic information of the escalator to obtain the escalator distance information, determining the position information of the escalator according to the escalator distance information, and performing corresponding decision-making processing according to the position information of the escalator, wherein, the obtaining of the escalator distance information includes: S1: Analyzing the second position information according to the characteristic information of the escalator to obtain a distance analysis result, and determining the maximum distance value and the minimum distance value from the distance analysis result; S2: Setting the minimum distance value as the initial threshold; S3: For the first region in the distance analysis result where the distance value is greater than the initial threshold, calculate the variance and area of the first region. For the second region in the distance analysis result where the distance value is less than the initial threshold, calculate the variance and area of the second region respectively. Among them, the variance of the first region M l is the mean value, N l is the number of data points, Di is the distance information corresponding to each data point, i = 1, 2, 3,..., N l , the area of the first region is N l ; the variance of the second region Ms is the mean value, Ns is the number of data points, Di is the distance information corresponding to each data point, i = 1, 2, 3,..., Ns, and the area of the second region is Ns; S4: Performing an increasing operation on the initial threshold successively, looping through S3, calculating the variance and area corresponding to multiple regions according to the obtained variance and area corresponding to multiple regions, calculating and obtaining an effective region according to the variance and area corresponding to multiple regions, and using the distance information corresponding to the effective region as the escalator distance information.

2. The method according to claim 1, characterized in that, the ranging device includes at least one of the following: a two-dimensional ranging device, a three-dimensional ranging device.

3. The method according to claim 2, characterized in that, when the ranging device includes: a two-dimensional ranging device and a three-dimensional ranging device, determining the position information of the escalator according to the escalator distance information includes: Calculating the position distance between the position information of the escalator obtained by using the two-dimensional ranging device scheme and the position information of the escalator obtained by using the three-dimensional ranging device scheme; Comparing the position distance calculation result with a first preset threshold, and determining the position information of the escalator obtained by using the two-dimensional ranging device scheme or the three-dimensional ranging device according to the comparison result as the final position information of the escalator.

4. The method according to claim 2, characterized in that, when the ranging device includes: a two-dimensional ranging device and a three-dimensional ranging device, the obtaining of the escalator distance information includes: Calculating the distance difference between the escalator distance information obtained by using the two-dimensional ranging device scheme and the escalator distance information obtained by using the three-dimensional ranging device scheme; Comparing the distance difference calculation result with a second preset threshold, and determining the escalator distance information obtained by using the two-dimensional ranging device scheme or the three-dimensional ranging device according to the comparison result as the final escalator distance information.

5. The method according to claim 1, characterized in that, analyzing the second position information according to the characteristic information of the escalator includes: judging whether the second position information meets the characteristic conditions of the escalator.

6. The method according to claim 5, characterized in that, judging whether the second position information meets the characteristic conditions of the escalator includes at least one of the following: Judging whether the obtained edge region width information is within a first preset range of the escalator handrail width; Determine whether the obtained edge region spacing information is within the second preset range of the escalator handrail spacing; Determine whether the obtained edge region curvature information is within the third preset range of the escalator handrail curvature; Determine whether the obtained edge region gradient information is within the fourth preset range of the escalator gradient.

7. The method according to claim 1, wherein, the obtaining of the escalator distance information includes: analyzing the second position information according to the characteristic information of the escalator to obtain a distance analysis result, and taking the mean information of the distance analysis result as the escalator distance information.

8. The method according to claim 1, wherein, after determining the position information of the escalator according to the escalator distance information, at least one of the following is further included: If there is an escalator set in the current map, determine whether the weight value of the escalator set is greater than a preset weight threshold. If the weight value of the escalator set is less than or equal to the preset weight threshold, delete the set; Under the condition of meeting the predetermined map update condition, perform a map update operation on the current map according to the position information of the escalator; If the predetermined correction condition is met, trigger a map position correction operation, wherein the map position correction operation includes: correcting each escalator set in the current map and the coordinate information related to the escalator set.

9. The method according to claim 8, wherein, Under the condition of meeting the predetermined map update condition, performing a map update operation on the current map according to the position information of the escalator includes: Read the information of the current map. If there is no associated escalator set in the current map, add a new set to the current map according to the position information of the escalator; When there is an associated escalator set in the current map, perform a set update operation on the associated escalator set according to the position information of the escalator.

10. A vision-based escalator recognition and processing device, wherein, comprises: A first acquisition module for acquiring visual sensor data by using a visual sensor; A second acquisition module for acquiring the region information of the escalator when determining that the category information of the current target object is an escalator according to the visual sensor data; A first extraction module for extracting the first position information corresponding to the position points collected by the visual sensor from the distance information obtained by the ranging device; A second extraction module for unifying the region information and each of the first position information into a coordinate system, and extracting the second position information corresponding to each position point in the region information; A processing module for analyzing the second position information according to the characteristic information of the escalator, obtaining escalator distance information, determining the position information of the escalator according to the escalator distance information, and performing corresponding decision processing according to the position information of the escalator, wherein the processing module is further used for: S1: Analyze the second position information according to the characteristic information of the escalator to obtain a distance analysis result, and determine the maximum distance value and the minimum distance value from the distance analysis result; S2: Set the minimum distance value as the initial threshold; S3: For the first region in the distance analysis result where the distance value is greater than the initial threshold, calculate the variance and area of the first region. For the second region in the distance analysis result where the distance value is less than the initial threshold, calculate the variance and area of the second region respectively. Among them, the variance of the first region M l is the mean value, N l is the number of data points, Di is the distance information corresponding to each data point, i = 1, 2, 3,... N l , the area of the first region is N l ; the variance of the second region Ms is the mean value, Ns is the number of data points, Di is the distance information corresponding to each data point, i = 1, 2, 3,... Ns, and the area of the second region is Ns; S4: Perform an increasing operation on the initial threshold successively, loop and execute S3, obtain the variances and areas corresponding to multiple regions, calculate and obtain the effective region according to the variances and areas corresponding to the multiple regions, and use the distance information corresponding to the effective region as the escalator distance information.

11. The device according to claim 10, wherein, it further comprises: a map management module, and the map management module includes at least one of the following: a deletion unit, configured to, when there is an escalator set in the current map, determine whether the weight value of the escalator set is greater than a preset weight threshold, and delete the escalator set if the weight value of the escalator set is less than or equal to the preset weight threshold; an update unit, configured to perform a map update operation on the current map according to the position information of the escalator under the condition of meeting a predetermined map update condition; a correction unit, configured to trigger a map position correction operation under the condition of meeting a predetermined correction condition, wherein the map position correction operation includes: correcting each escalator set in the current map and the coordinate information related to the escalator set.

Images