A control method and device, intelligent robot and storage medium

Abstract

The application discloses a control method and device, an intelligent robot and a storage medium. The method comprises the following steps: acquiring an infrared image and a depth image in a working area through a depth camera arranged on the intelligent robot; identifying an identification block in the infrared image, and clustering the identification block to obtain an identification block distribution pattern; determining three-dimensional coordinates of the identification block distribution pattern according to the depth image; according to a distribution type of the identification block distribution pattern, adopting a corresponding matching algorithm to match the three-dimensional coordinates of the identification block distribution pattern with a standard identification pattern, and obtaining a dangerous area identification pattern in the working area; determining a dangerous area in the working area according to the dangerous area identification pattern, and controlling the intelligent robot to avoid the dangerous area. The application solves the problem that the intelligent robot cannot accurately identify the dangerous area, and avoids the intelligent robot from entering the dangerous area, thereby causing personnel injury or property loss.

Description

Technical Field

[0001] This invention relates to the field of robotics, and more particularly to a control method, device, intelligent robot, and storage medium. Background Technology

[0002] Intelligent mobile robots are highly intelligent devices that integrate multiple functions such as environmental perception, dynamic decision-making and planning, behavior control and execution. They are widely used in public places such as shopping malls, supermarkets, and stadiums. Due to hardware or design limitations, robots cannot use escalators or walk down stairs like humans. These areas are considered dangerous for robots, and accidental entry could cause danger and injure both pedestrians and the robots themselves.

[0003] Normally, robots have accurate localization and know the location of dangerous areas, so they will avoid these areas when planning their paths. However, when localization is lost, robots currently generally use cameras to capture images of the surrounding environment and perform identification. However, in scenarios such as stairs and escalators, which cannot be accurately identified as dangerous areas, the robot may enter these areas, causing personal injury or property damage. Summary of the Invention

[0004] This invention provides a control method, device, intelligent robot, and storage medium to solve the problem that intelligent robots cannot accurately identify dangerous areas, thereby preventing intelligent robots from entering dangerous areas and causing personal injury or property damage.

[0005] According to one aspect of the present invention, a control method is provided for an intelligent robot, the intelligent robot comprising: a depth camera; and dangerous area marker graphics affixed around a dangerous area within the working area of ​​the intelligent robot; the method comprising:

[0006] The robot acquires infrared and depth images of the working area using a depth camera mounted on it.

[0007] Identify the marked regions in the infrared image and cluster the marked regions to obtain a marked region distribution pattern;

[0008] The three-dimensional coordinates of the marked block distribution pattern are determined based on the depth image;

[0009] Based on the distribution type of the marked block distribution graphic, a corresponding matching algorithm is used to match the three-dimensional coordinates of the marked block distribution graphic and the standard marked graphic to obtain the dangerous area marked graphic within the working area;

[0010] The dangerous areas within the work area are identified based on the dangerous area identification graphic, and the intelligent robot is controlled to avoid the dangerous areas.

[0011] According to another aspect of the present invention, a control device is provided for use in an intelligent robot, the intelligent robot including: a depth camera; and a danger zone marking graphic affixed around a danger zone within the working area of ​​the intelligent robot; the device includes:

[0012] The image acquisition module is used to acquire infrared and depth images of the working area through the depth camera installed on the intelligent robot;

[0013] The identifier recognition module is used to identify identifier blocks in the infrared image and cluster the identifier blocks to obtain an identifier block distribution pattern;

[0014] A coordinate determination module is used to determine the three-dimensional coordinates of the identification block distribution pattern based on the depth image;

[0015] The matching module is used to match the three-dimensional coordinates of the marked block distribution graphic with the standard marked graphic using a corresponding matching algorithm based on the distribution type of the marked block distribution graphic, so as to obtain the dangerous area marked graphic within the working area;

[0016] The control module is used to determine the dangerous areas within the work area based on the dangerous area identification graphic, and to control the intelligent robot to avoid the dangerous areas.

[0017] According to another aspect of the present invention, an intelligent robot is provided, the intelligent robot comprising:

[0018] At least one processor; and

[0019] A memory communicatively connected to the at least one processor; wherein,

[0020] The memory stores a computer program that can be executed by the at least one processor, the computer program being executed by the at least one processor to enable the at least one processor to perform the control method described in any embodiment of the present invention.

[0021] According to another aspect of the present invention, a computer-readable storage medium is provided, the computer-readable storage medium storing computer instructions for causing a processor to execute and implement the control method described in any embodiment of the present invention.

[0022] The technical solution of this invention uses a depth camera mounted on an intelligent robot to acquire infrared and depth images of the working area. It identifies marker blocks in the infrared images and clusters these blocks to obtain a marker block distribution pattern. The three-dimensional coordinates of the marker block distribution pattern are determined based on the depth image. According to the distribution type of the marker block distribution pattern, a corresponding matching algorithm is used to match the three-dimensional coordinates of the marker block distribution pattern with standard marker patterns to obtain a danger zone marker pattern within the working area. The danger zone within the working area is then identified based on the danger zone marker pattern, and the intelligent robot is controlled to avoid the danger zone. By matching the marker block distribution pattern with the standard marker pattern using a corresponding matching algorithm based on the distribution type of the identified marker blocks, the accuracy of danger zone representation image recognition is improved, solving the problem of the intelligent robot's inability to accurately identify danger zones and preventing the intelligent robot from entering danger zones, thus avoiding personal injury or property damage.

[0023] It should be understood that the description in this section is not intended to identify key or essential features of the embodiments of the present invention, nor is it intended to limit the scope of the invention. Other features of the invention will become readily apparent from the following description. Attached Figure Description

[0024] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0025] Figure 1 This is a flowchart of a control method provided in Embodiment 1 of the present invention;

[0026] Figure 2 This is a flowchart of a control method provided in Embodiment 2 of the present invention;

[0027] Figure 3 This is a schematic diagram of the linearly distributed identifier block distribution pattern in a control method provided in Embodiment 2 of the present invention;

[0028] Figure 4 This is a schematic diagram of the polygonal distribution of the identifier block pattern in a control method provided in Embodiment 2 of the present invention;

[0029] Figure 5 This is a schematic diagram of the structure of a control device provided in Embodiment 3 of the present invention;

[0030] Figure 6 This is a schematic diagram of the structure of an intelligent robot that implements the control method of the embodiments of the present invention; Detailed Implementation

[0031] To enable those skilled in the art to better understand the present invention, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of the present invention.

[0032] It should be noted that the terms "comprising" and "having" and any variations thereof in the specification, claims and accompanying drawings of this invention are intended to cover non-exclusive inclusion. For example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units that are explicitly listed, but may include other steps or units that are not explicitly listed or that are inherent to such processes, methods, products or devices.

[0033] Example 1

[0034] Figure 1 The flowchart below provides a control method according to Embodiment 1 of the present invention. This embodiment is applicable to situations where a robot is controlled to avoid dangerous areas. The method can be executed by a control device, which can be implemented in hardware and / or software and can be configured in an intelligent robot. The intelligent robot includes: a depth camera; and dangerous area marker graphics are affixed around the dangerous areas within the working area of ​​the intelligent robot. The dangerous area marker graphics are used to identify dangerous areas and can be recognized by the intelligent robot.

[0035] like Figure 1 As shown, the method includes:

[0036] S110: The robot acquires infrared and depth images of the work area using a depth camera mounted on it.

[0037] Intelligent robots can be various types of intelligent devices, such as cleaning robots, home robots, transportation robots, and robots that provide various services and perform various tasks. Intelligent robots are equipped with depth cameras, which actively emit infrared light and receive information from the returned infrared light to calculate depth information. The working area of ​​an intelligent robot includes hazardous areas, such as escalators and stairs, where there may be dangers for the robot.

[0038] Specifically, by utilizing the ability of depth cameras to generate infrared images from the acquired infrared light and calculate depth information based on the information in the infrared light, danger zone markers can be posted at multiple locations within the work area, such as the entrance to dangerous areas.

[0039] When the intelligent robot loses its location or travels into a certain work area, it uses its onboard depth camera to collect infrared and depth images of the surrounding environment within the work area during operation. This allows it to identify danger zone markers based on the infrared and depth images, thus avoiding dangerous areas.

[0040] S120. Identify the marked blocks in the infrared image and cluster the marked blocks to obtain the marked block distribution pattern.

[0041] In this context, the identified marker blocks are the pixel blocks that serve an identifying function. The marker block distribution pattern is a graphic composed of the distribution of multiple marker blocks. In this embodiment, the hazard area marker graphic can be a graphic composed of marker elements of preset shapes arranged according to preset rules. Therefore, a marker block can be understood as a pixel block composed of infrared rays returned by marker elements; the marker block distribution pattern can be understood as a distribution graphic composed of pixel blocks composed of infrared rays returned by marker elements of preset shapes in the hazard area marker graphic.

[0042] To accurately identify hazard area sign graphics, the sign elements can be made of a material with high infrared reflectivity, forming special shapes such as circles, squares, triangles, or other irregular shapes. High reflectivity refers to a material with a reflectivity exceeding a certain threshold; the material of the sticker in this embodiment is not limited. The hazard area sign graphics are formed by arranging the sign elements according to a preset arrangement rule, such as vertical or horizontal lines, or triangles, squares, polygons, or other irregular shapes. This embodiment does not limit the shape of the hazard area sign graphics and sign elements; they can be set based on actual needs.

[0043] Specifically, pixels in the infrared image are identified to obtain the marked blocks composed of the marked pixels; since the dangerous area marking graphics are composed of marked elements of preset shapes arranged according to preset arrangement rules, the marking blocks are clustered to obtain the marking block distribution graphics composed of the marking blocks.

[0044] S130. Determine the three-dimensional coordinates of the distribution pattern of the marked blocks based on the depth image.

[0045] Since infrared images are two-dimensional, it is difficult to determine the position coordinates of the marked area distribution pattern in the work scene using two-dimensional images. Therefore, the three-dimensional coordinates of the marked area distribution pattern in the work scene can be calculated by combining the depth map, which helps intelligent robots to identify specific dangers and avoid dangerous areas.

[0046] Specifically, after identifying the pattern of the marked area distribution, the pixel position of the marked area distribution pattern in the infrared image corresponding to the working area can be determined, and then the pixel position of the marked area distribution pattern in the depth image can be determined. Then, the three-dimensional coordinates of the marked area distribution pattern in the working scene can be calculated based on the depth value and pixel position in the depth image.

[0047] S140. Based on the distribution type of the marked block distribution graphic, the corresponding matching algorithm is used to match the three-dimensional coordinates of the marked block distribution graphic and the standard marked graphic to obtain the dangerous area marked graphic within the work area.

[0048] The distribution type of the marking blocks can include: linear distribution, polygonal distribution, or other irregular distribution. Standard marking graphics are standard hazard area marking graphics, and hazard area marking graphics must be affixed around hazard areas according to the distribution rules of the standard marking images. Specific distribution rules can include: the shape and size of the stickers, the distance between stickers, and their distribution position.

[0049] Specifically, since different distribution types have different characteristics, different preset matching algorithms are used to match the three-dimensional coordinates of the identification block distribution graphics and the standard identification graphics according to the distribution type of the identification block distribution graphics. If an identification block distribution graphics that successfully matches the three-dimensional coordinates of the standard identification graphics are found in the working area, then the identification block distribution graphics are identified as the danger zone identification graphics.

[0050] In this embodiment, based on the distribution type of the marked block distribution graphic, a corresponding matching algorithm is used to match the three-dimensional coordinates of the marked block distribution graphic and the standard marked graphic to obtain the dangerous area marked graphic within the working area, which can further improve the accuracy of the identified marked block distribution graphic.

[0051] S150. Determine the hazardous areas within the work area based on the hazardous area identification graphics, and control the intelligent robot to avoid the hazardous areas.

[0052] Specifically, the area corresponding to the three-dimensional coordinates of the danger zone marker graphic is identified as the danger zone. The robot is then assisted in avoiding the danger zone based on the three-dimensional coordinates of the danger zone marker graphic and replans its travel route, thereby preventing the robot from entering the danger zone and ensuring the safety of the robot during operation.

[0053] The technical solution of this invention uses a depth camera mounted on an intelligent robot to acquire infrared and depth images of the working area. It identifies marker blocks in the infrared images and clusters these blocks to obtain a marker block distribution pattern. The three-dimensional coordinates of the marker block distribution pattern are determined based on the depth image. According to the distribution type of the marker block distribution pattern, a corresponding matching algorithm is used to match the three-dimensional coordinates of the marker block distribution pattern with standard marker patterns to obtain a danger zone marker pattern within the working area. The danger zone within the working area is then identified based on the danger zone marker pattern, and the intelligent robot is controlled to avoid the danger zone. By matching the marker block distribution pattern with the standard marker pattern using a corresponding matching algorithm based on the distribution type of the identified marker blocks, the accuracy of danger zone representation image recognition is improved, solving the problem of the intelligent robot's inability to accurately identify danger zones and preventing the intelligent robot from entering danger zones, thus avoiding personal injury or property damage.

[0054] Example 2

[0055] Figure 2 This is a flowchart of a control method provided in Embodiment 2 of the present invention. The matching algorithm used in this embodiment is further defined compared to the embodiments described above. For example... Figure 2 As shown, the method includes:

[0056] S210: The robot acquires infrared and depth images of the work area using a depth camera mounted on it.

[0057] S220. Identify the marked blocks in the infrared image and cluster the marked blocks to obtain the marked block distribution pattern.

[0058] Optionally, in step S220, identifying the identifier blocks in the infrared image includes:

[0059] S221. Obtain the binarized image of the infrared image.

[0060] Specifically, the acquired infrared images are binarized to obtain binarized images.

[0061] S222. Filter the pixel values ​​of the binarized image to obtain multiple pixel blocks.

[0062] Specifically, because the hazard area marker graphics are made of infrared-reflective material, the marker blocks reflect a significant amount of infrared light, resulting in higher brightness in the infrared image. Consequently, the pixel values ​​of the marker blocks in the infrared image will differ considerably from those of other areas. Therefore, filtering the binarized image using a set pixel threshold can yield pixel blocks composed of areas with higher pixel values.

[0063] S223. Calculate the block graphic feature value of each pixel block.

[0064] In this context, the block graphic feature value can be understood as the feature value that reflects the graphic elements of a pixel block. For example, if the sign element constituting the danger zone sign graphic is a circle, then the block graphic feature value can be roundness and area; if the sign element constituting the danger zone sign graphic is a triangle or polygon, then the block graphic feature value can be the number of sides constituting the graphic; if the sign element constituting the danger zone sign graphic is an irregular shape, the block graphic feature value can be the feature points or feature edges of the sign element.

[0065] Specifically, based on the shape of the marking elements that constitute the dangerous area marking graphic, the corresponding block graphic features are obtained, and the block graphic feature value of the pixel block on the block graphic features is calculated.

[0066] S224. Pixel blocks whose feature values ​​meet preset conditions are identified as identifier blocks in the infrared image.

[0067] The preset conditions for the block graphic feature values ​​are used to determine whether a pixel block meets the requirements of the identifier block corresponding to the identifier element. Specifically, the preset conditions may include: For an identifier element that is circular, the block graphic feature values ​​of the pixel block include roundness and area, and the preset conditions include: the difference between the roundness of the pixel block and the roundness of the identifier element is less than a roundness threshold, and the difference between the area of ​​the pixel block and the area of ​​the identifier element is less than an area threshold. For an identifier element that is triangular or polygonal, the block graphic feature values ​​of the pixel block include the number of sides constituting the shape, and the preset condition includes: the identifier element and the pixel block have the same number of sides.

[0068] Specifically, pixel blocks are filtered based on their graphic feature values. Pixel blocks whose graphic feature values ​​meet preset conditions are filtered out, while those whose graphic feature values ​​meet preset conditions are retained and identified as marker blocks in the infrared image. This process filters out the impact of background noise on the accuracy of marker block recognition, making the recognition of marker blocks more accurate and thus improving the accuracy of dangerous area marker recognition.

[0069] Optionally, in step S220, clustering the identified blocks to obtain an identified block distribution pattern includes:

[0070] S225. Calculate the distance between adjacent identifier blocks.

[0071] Specifically, after identifying the marked blocks, in order to more accurately identify the hazard area marking graphics, it is necessary to cluster the marked blocks. Generally, the distance between adjacent predetermined blocks in the hazard area marking graphics is small, and adjacent marked blocks are not too far apart. Therefore, the distance between adjacent marked blocks can be calculated. Specifically, the distance between the centers of two adjacent marked blocks can be calculated.

[0072] S226. Cluster the identified blocks in the infrared image according to the distance to obtain at least one identified block distribution domain.

[0073] Specifically, if the distance between adjacent marker blocks is less than or equal to a predetermined distance, these two marker blocks can be considered to belong to the same category and grouped into the same group. If the distance between adjacent marker blocks is greater than or equal to the predetermined distance, these two marker blocks can be considered not to belong to the same category. Thus, at least one marker block distribution region can be obtained from the clustering of marker blocks in the infrared image. The predetermined distance can be adjusted according to the user's actual settings in the working scenario.

[0074] S227. Filter the distribution domain of the identifier block based on preset conditions to obtain the target identifier block distribution domain.

[0075] Specifically, in order to improve the recognition accuracy of the identification block distribution pattern, the clustered identification block distribution domain needs to be further filtered. The preset conditions that the identification block distribution domain needs to meet may include: the area difference of each identification block in a clustered identification block distribution domain is less than a preset value, such as a certain percentage of the area of ​​the smallest identification block, such as 20%; and the number of identification blocks contained in an identification block distribution domain is within a preset number threshold, such as 5-10 identification blocks.

[0076] S228. Determine the identifier block distribution pattern corresponding to the identifier block distribution domain based on the position of the centroid of each identifier block in the target identifier block distribution domain.

[0077] Specifically, based on the layout formed by the positions of the centroids of each identifier block in the target identifier block distribution domain, the identifier block distribution pattern corresponding to the identifier block distribution domain is determined to be either a linear distribution or a polygonal distribution.

[0078] S230. Determine the three-dimensional coordinates of the distribution pattern of the marked blocks based on the depth image.

[0079] Optionally, in step S230, determining the three-dimensional coordinates of the identification block distribution pattern based on the depth image includes:

[0080] S231. Determine the matching pixel points in the depth image that match the distribution pattern of the identified blocks.

[0081] Among them, matching pixels refer to the depth pixels in the depth image that match the distribution pattern of the identified blocks.

[0082] Specifically, since the depth image and the infrared image are in one-to-one correspondence, the pixels in the infrared image and the pixels in the depth image are also in one-to-one correspondence. Therefore, it is possible to identify the depth image corresponding to the identified block distribution pattern, find the corresponding position in the depth image that corresponds to the infrared image, and thus determine the matching pixel in the depth image that matches the identified block distribution pattern.

[0083] S232. Obtain the two-dimensional coordinates of the matching pixel in the depth image.

[0084] Specifically, the position of the pixels in the identification block distribution pattern in the infrared image can be determined based on their position in the identification block distribution pattern. Then, the matching pixels in the depth image that correspond to the identification block distribution pattern can be determined, thereby obtaining the depth value corresponding to the matching pixels in the depth image.

[0085] S233. Calculate the three-dimensional coordinates of the identification block distribution pattern based on the two-dimensional coordinates and depth values ​​of each matched pixel, as well as the intrinsic and extrinsic parameters of the depth camera.

[0086] Specifically, based on the two-dimensional coordinates and depth values ​​of the matched pixels, the three-dimensional coordinates of the matching points (i.e., the distribution pattern of the identification blocks) corresponding to the matched pixels in the work scene can be calculated, and then the spatial location of the danger zone can be obtained based on the three-dimensional coordinates of the matching points.

[0087] S240. If the distribution type of the marked block distribution graphic is a linear distribution, then the linear fitting matching algorithm is used to match the three-dimensional coordinates of the marked block distribution graphic and the standard marked graphic to obtain the dangerous area marked graphic within the working area.

[0088] The line fitting matching algorithm matches the lines obtained from the three-dimensional coordinates of the marked block distribution graphic with the corresponding lines of the standard marked graphic. Specific matching parameters may include the slope, intercept, and center point of the line.

[0089] Specifically, if the distribution type of the marker block distribution graphic is determined to be a linear distribution based on the distribution position of each pixel block in the marker block distribution graphic, then a linear fitting matching algorithm is used to match the three-dimensional coordinates of the marker block distribution graphic with the standard marker graphic. If the match is successful, the marker block distribution graphic is identified as a danger zone marker graphic.

[0090] For example, Figure 3It is a schematic diagram of the linearly distributed identifier blocks. Figure 3 It is a linear distribution pattern of marker blocks consisting of 4 marker blocks (10). It can be understood that the linear distribution pattern can be vertical, horizontal, or diagonal lines, etc. There is no limit to the number of marker blocks or lines contained in the linear distribution pattern; for example, a straight line formed by two columns of marker blocks can be defined as a marker block distribution pattern.

[0091] Optionally, the line fitting matching algorithm includes:

[0092] Obtain the three-dimensional coordinates of the centroid of each marker block in the marker block distribution graphic;

[0093] The target fitted line is obtained by fitting the three-dimensional coordinates of the centroid of each of the identified blocks with a straight line;

[0094] Calculate the slope of the target fitted line and the slope of the standard identification graphic;

[0095] If the target fitted line satisfies the linear distribution condition, then the slope of the target fitted line is matched with the slope of the standard label graphic based on the linear fitting matching algorithm;

[0096] If a match is successful, the distribution graphic of the marked blocks is determined to be the dangerous area marked graphic within the work area.

[0097] Among them, the straight line distribution condition is the condition that the target fitted straight line must satisfy to participate in the straight line matching algorithm. Specifically, it may include: the distance between the centroid of each identified block and the target fitted straight line is less than a first preset threshold; the spacing between the centroids of each identified block is less than a second preset threshold and is uniformly distributed; the distance between the center point of the fitted straight line and the target fitted straight line is less than a third preset threshold; and the height information of the center point of the fitted straight line meets the preset height range.

[0098] Specifically, for each marker block in the marker block distribution graphic, the three-dimensional coordinates of the marker block's centroid are determined based on the three-dimensional coordinates of each pixel within the marker block. Then, a target fitted line is obtained by fitting the three-dimensional coordinates of the centroids of each marker block in the marker block distribution graphic, and the slope of the target fitted line is calculated. Additionally, for standard marker graphics, the slope of the line formed by the centroids of each marker element in the standard marker graphic can also be calculated, i.e., the slope of the standard marker graphic.

[0099] Further filtering is needed for the target fitted line. First, it is determined whether the target fitted line meets the line distribution conditions. If not, the target fitted line is filtered out; if it does, the target fitted line is retained. Based on the line fitting matching algorithm, the slope of the target fitted line is matched with the slope of the standard marking graphic. If the slope match is successful, for example, if the slope difference between the slope of the target fitted line and the slope of the standard marking graphic is less than the preset slope, then the marking block distribution graphic is determined as the danger zone marking graphic within the work area.

[0100] S250. If the distribution type of the marked block distribution graphic is a polygonal distribution, the iterative nearest point matching algorithm is used to match the three-dimensional coordinates of the marked block distribution graphic and the standard marked graphic to obtain the dangerous area marked graphic within the working area.

[0101] The basic idea of ​​the iterative nearest point matching algorithm is to match data based on certain geometric characteristics, and set these matching points as hypothetical corresponding points. Then, the motion parameters are solved based on this correspondence, and then these motion parameters are used to transform the data.

[0102] Specifically, if the distribution type of the marked block distribution graphic is a polygonal distribution, in order to more accurately match and identify the marked block distribution graphic, it is necessary to use the iterative nearest point matching algorithm to match the three-dimensional coordinates of the marked block distribution graphic with the standard marked graphic. If the match is successful, the marked block distribution graphic is identified as a dangerous area marked graphic.

[0103] For example, Figure 4 This is a schematic diagram of the polygonal distribution of the identifier blocks. Figure 4 It is a hexagonal pattern consisting of 7 marker blocks 41. It can be understood that the marker block distribution pattern can also include triangles, rectangles, pentagons, etc.

[0104] Optionally, the iterative nearest point matching algorithm includes:

[0105] Iteratively calculate the three-dimensional coordinates of the nearest point in the standard identification graphic for each pixel in the identification block distribution graphic;

[0106] Based on the rigid body transformation that minimizes the average distance corresponding to the nearest point, determine the rotation and translation parameters between the identification block distribution graphic and the standard identification graphic;

[0107] The pixel in the identifier block distribution pattern is transformed according to the rotation parameters and the translation parameters to obtain the transformed identifier block distribution pattern;

[0108] The iterative calculation stops when the average distance between the transformed marker block distribution graphic and the standard marker graphic satisfies the objective function, and the marker block distribution graphic is then designated as the hazardous area marker graphic within the working area.

[0109] Specifically, for the pixel set X1 in the standard marker graphic and the pixel set X2 in the marker block distribution graphic, calculate the nearest point in the pixel set X1 of the standard marker graphic for each pixel in the pixel set X2 of the marker block distribution graphic; find the rigid body transformation that minimizes the average distance between the corresponding points, and obtain the translation and rotation parameters. Using the obtained translation and rotation parameters, transform the pixel set X2 of the marker block distribution graphic to obtain the pixel set X3 corresponding to the transformed marker block distribution graphic; if the average distance between the pixel set X3 and the pixel set X1 is less than a given threshold, stop the iterative calculation; otherwise, the pixel set X3 corresponding to the transformed marker block distribution graphic is used as the new pixel set X2 of the marker block distribution graphic to continue the iteration until the requirements of the objective function are met, and the marker block distribution graphic is designated as the danger zone marker graphic within the working area.

[0110] S260. Determine the hazardous areas within the work area based on the hazardous area identification graphics, and control the intelligent robot to avoid the hazardous areas.

[0111] The technical solution of this invention involves acquiring infrared and depth images of the working area using a depth camera mounted on an intelligent robot; identifying marker blocks in the infrared images and clustering these marker blocks to obtain a marker block distribution pattern; determining the three-dimensional coordinates of the marker block distribution pattern based on the depth images; if the marker block distribution pattern is linear, a linear fitting matching algorithm is used to match the three-dimensional coordinates of the marker block distribution pattern with standard marker patterns; if the marker block distribution pattern is polygonal, an iterative nearest point matching algorithm is used to match the three-dimensional coordinates of the marker block distribution pattern with standard marker patterns to obtain a danger zone marker pattern within the working area; determining the danger zone within the working area based on the danger zone marker pattern, and controlling the intelligent robot to avoid the danger zone. By employing different matching strategies for linear or polygonal marker block distribution patterns to determine whether the marker block distribution pattern is a danger zone marker pattern within the working area, the accuracy of danger zone representation image recognition is further improved, solving the problem of the intelligent robot's inability to accurately identify danger zones and preventing the intelligent robot from entering danger zones, thus avoiding personal injury or property damage.

[0112] Example 3

[0113] Figure 5This is a schematic diagram of a control device provided in Embodiment 3 of the present invention. Figure 5 As shown, the device includes:

[0114] The image acquisition module 310 is used to acquire infrared images and depth images within the working area through the depth camera installed on the intelligent robot;

[0115] The identifier recognition module 320 is used to identify identifier blocks in the infrared image and cluster the identifier blocks to obtain an identifier block distribution pattern.

[0116] The coordinate determination module 330 is used to determine the three-dimensional coordinates of the identification block distribution pattern based on the depth image;

[0117] The matching module 340 is used to match the three-dimensional coordinates of the marked block distribution graphic with the standard marked graphic using a corresponding matching algorithm according to the distribution type of the marked block distribution graphic, so as to obtain the dangerous area marked graphic within the working area.

[0118] The control module 350 is used to determine the dangerous areas within the work area based on the dangerous area identification graphic, and to control the intelligent robot to avoid the dangerous areas.

[0119] Optionally, the matching module 340 includes:

[0120] If the distribution type of the marked block distribution graphic is a linear distribution, then a linear fitting matching algorithm is used to match the three-dimensional coordinates of the marked block distribution graphic and the standard marked graphic to obtain the dangerous area marked graphic within the working area;

[0121] If the distribution type of the marked block distribution graphic is a polygonal distribution, then the iterative nearest point matching algorithm is used to match the three-dimensional coordinates of the marked block distribution graphic and the standard marked graphic to obtain the dangerous area marked graphic within the working area.

[0122] Optionally, the line fitting matching algorithm includes:

[0123] Obtain the three-dimensional coordinates of the centroid of each marker block in the marker block distribution graphic;

[0124] The target fitted line is obtained by fitting the three-dimensional coordinates of the centroid of each of the identified blocks with a straight line;

[0125] Calculate the slope of the target fitted line and the slope of the standard identification graphic;

[0126] If the target fitted line satisfies the linear distribution condition, then the slope of the target fitted line is matched with the slope of the standard label graphic based on the linear fitting matching algorithm;

[0127] If a match is successful, the distribution graphic of the marked blocks is determined to be the dangerous area marked graphic within the work area.

[0128] Optionally, the iterative nearest point matching algorithm includes:

[0129] Iteratively calculate the three-dimensional coordinates of the nearest point in the standard identification graphic for each pixel in the identification block distribution graphic;

[0130] Based on the rigid body transformation that minimizes the average distance corresponding to the nearest point, determine the rotation and translation parameters between the identification block distribution graphic and the standard identification graphic;

[0131] The pixel in the identifier block distribution pattern is transformed according to the rotation parameters and the translation parameters to obtain the transformed identifier block distribution pattern;

[0132] The iterative calculation stops when the average distance between the transformed marker block distribution graphic and the standard marker graphic satisfies the objective function, and the marker block distribution graphic is then designated as the hazardous area marker graphic within the working area.

[0133] Optionally, the identifier recognition module 320 is specifically used for:

[0134] Obtain the binarized image of the infrared image;

[0135] The binarized image is filtered by pixel values ​​to obtain multiple pixel blocks;

[0136] Calculate the block graphic feature value of each pixel block;

[0137] Pixel blocks whose graphic feature values ​​meet preset conditions are identified as the identifier blocks in the infrared image.

[0138] The identification module 320 is also used for:

[0139] Calculate the distance between adjacent identifier blocks;

[0140] Cluster the identified blocks in the infrared image based on the distance to obtain at least one identified block distribution domain;

[0141] The target identifier block distribution domain is obtained by filtering the identifier block distribution domain based on preset conditions;

[0142] The identifier block distribution pattern corresponding to the identifier block distribution domain is determined based on the position of the centroid of each identifier block in the target identifier block distribution domain.

[0143] Optionally, the coordinate determination module 330 includes:

[0144] Identify matching pixels in the depth image that match the pattern of the identified blocks;

[0145] Obtain the two-dimensional coordinates of the matching pixel in the depth image;

[0146] The three-dimensional coordinates of the identification block distribution pattern are calculated based on the two-dimensional coordinates and depth values ​​of each matching pixel, as well as the intrinsic and extrinsic parameters of the depth camera.

[0147] The control device provided in the embodiments of the present invention can execute the control method provided in any embodiment of the present invention, and has the corresponding functional modules and beneficial effects of the execution method.

[0148] Example 4

[0149] Figure 6 A schematic diagram of the structure of an intelligent robot 10, which can be used to implement embodiments of the present invention, is shown. The intelligent robot is intended to represent various forms of intelligent devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the invention described and / or claimed herein.

[0150] like Figure 6 As shown, the intelligent robot 10 includes at least one processor 11 and a memory, such as a read-only memory (ROM) 12 or a random access memory (RAM) 13, communicatively connected to the at least one processor 11. The memory stores computer programs executable by the at least one processor. The processor 11 can perform various appropriate actions and processes based on the computer programs stored in the ROM 12 or loaded from storage unit 18 into the RAM 13. The RAM 13 can also store various programs and data required for the operation of the intelligent robot 10. The processor 11, ROM 12, and RAM 13 are interconnected via a bus 14. An input / output (I / O) interface 15 is also connected to the bus 14.

[0151] Multiple components in the intelligent robot 10 are connected to the I / O interface 15, including: an input unit 16, such as a depth camera; an output unit 17, such as various types of displays, speakers, etc.; a storage unit 18, such as a disk, optical disk, etc.; and a communication unit 19, such as a network card, modem, wireless transceiver, etc. The communication unit 19 allows the intelligent robot 10 to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunications networks.

[0152] Processor 11 can be a variety of general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of processor 11 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special-purpose artificial intelligence (AI) computing chips, various processors running machine learning model algorithms, a digital signal processor (DSP), and any suitable processor, controller, microcontroller, etc. Processor 11 performs the various methods and processes described above, such as control methods.

[0153] In some embodiments, the control method may be implemented as a computer program tangibly contained in a computer-readable storage medium, such as storage unit 18. In some embodiments, part or all of the computer program may be loaded and / or installed on the intelligent robot 10 via ROM 12 and / or communication unit 19. When the computer program is loaded into RAM 13 and executed by processor 11, one or more steps of the control method described above may be performed. Alternatively, in other embodiments, processor 11 may be configured to execute the control method by any other suitable means (e.g., by means of firmware).

[0154] Various embodiments of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), systems-on-a-chip (SoCs), payload-programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments may include implementations in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which may be a dedicated or general-purpose programmable processor, capable of receiving data and instructions from a storage system, at least one input device, and at least one output device, and transmitting data and instructions to the storage system, the at least one input device, and the at least one output device.

[0155] Computer programs used to implement the methods of the present invention may be written in any combination of one or more programming languages. These computer programs may be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device, such that when executed by the processor, the computer programs cause the functions / operations specified in the flowcharts and / or block diagrams to be performed. The computer programs may be executed entirely on a machine, partially on a machine, or as a standalone software package, partially on a machine and partially on a remote machine, or entirely on a remote machine or server.

[0156] In the context of this invention, a computer-readable storage medium can be a tangible medium that may contain or store a computer program for use by or in conjunction with an instruction execution system, apparatus, or device. A computer-readable storage medium may include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination thereof. Alternatively, a computer-readable storage medium may be a machine-readable signal medium. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fibers, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof.

[0157] To provide interaction with a user, the systems and techniques described herein can be implemented on an intelligent robot having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user provides input to the intelligent robot. Other types of devices can also be used to provide interaction with the user; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including sound input, voice input, or tactile input).

[0158] The systems and technologies described herein can be implemented in computing systems that include backend components (e.g., as data servers), or computing systems that include middleware components (e.g., application servers), or computing systems that include frontend components (e.g., user computers with graphical user interfaces or web browsers through which users can interact with implementations of the systems and technologies described herein), or any combination of such backend, middleware, or frontend components. The components of the system can be interconnected via digital data communication of any form or medium (e.g., communication networks). Examples of communication networks include local area networks (LANs), wide area networks (WANs), blockchain networks, and the Internet.

[0159] A computing system can include clients and servers. Clients and servers are generally located far apart and typically interact through communication networks. The client-server relationship is created by computer programs running on the respective computers and having a client-server relationship with each other. The server can be a cloud server, also known as a cloud computing server or cloud host, which is a hosting product within the cloud computing service system to address the shortcomings of traditional physical hosts and VPS services, such as high management difficulty and weak business scalability.

[0160] It should be understood that the various forms of processes shown above can be used, with steps reordered, added, or deleted. For example, the steps described in this invention can be executed in parallel, sequentially, or in different orders, as long as the desired result of the technical solution of this invention can be achieved, and this is not limited herein.

[0161] The specific embodiments described above do not constitute a limitation on the scope of protection of this invention. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this invention should be included within the scope of protection of this invention.

Claims

1. A control method, characterized in that, The method is applied to an intelligent robot, which includes: a depth camera; and dangerous area marker graphics are affixed around dangerous areas within the working area of ​​the intelligent robot; the method includes: The robot acquires infrared and depth images of the working area using a depth camera mounted on it. Identify the marked regions in the infrared image and cluster the marked regions to obtain a marked region distribution pattern; The three-dimensional coordinates of the marked block distribution pattern are determined based on the depth image; Based on the distribution type of the marked block distribution graphic, a corresponding matching algorithm is used to match the three-dimensional coordinates of the marked block distribution graphic and the standard marked graphic to obtain the dangerous area marked graphic within the work area; the standard marked graphic is a standard dangerous area marked graphic. Based on the danger zone identification graphic, the danger zone within the work area is determined, and the intelligent robot is controlled to avoid the danger zone; The step of matching the three-dimensional coordinates of the marked block distribution pattern with the standard marked pattern using a corresponding matching algorithm based on the distribution type of the marked block distribution pattern to obtain the danger zone marked pattern within the working area includes: If the distribution type of the marked block distribution graphic is a linear distribution, then a linear fitting matching algorithm is used to match the three-dimensional coordinates of the marked block distribution graphic and the standard marked graphic to obtain the dangerous area marked graphic within the working area; If the distribution type of the marked block distribution graphic is a polygonal distribution, then the iterative nearest point matching algorithm is used to match the three-dimensional coordinates of the marked block distribution graphic and the standard marked graphic to obtain the dangerous area marked graphic within the working area.

2. The method according to claim 1, characterized in that, The line fitting matching algorithm includes: Obtain the three-dimensional coordinates of the centroid of each marker block in the marker block distribution graphic; The target fitted line is obtained by fitting the three-dimensional coordinates of the centroid of each of the identified blocks with a straight line; Calculate the slope of the target fitted line and the slope of the standard identification graphic; If the target fitted line satisfies the linear distribution condition, then the slope of the target fitted line is matched with the slope of the standard label graphic based on the linear fitting matching algorithm; If a match is successful, the distribution graphic of the marked blocks is determined to be the dangerous area marked graphic within the work area.

3. The method according to claim 1, characterized in that, The iterative nearest point matching algorithm includes: Iteratively calculate the three-dimensional coordinates of the nearest point in the standard identification graphic for each pixel in the identification block distribution graphic; Based on the rigid body transformation that minimizes the average distance corresponding to the nearest point, determine the rotation and translation parameters between the identification block distribution graphic and the standard identification graphic; The pixel in the identifier block distribution pattern is transformed according to the rotation parameters and the translation parameters to obtain the transformed identifier block distribution pattern; The iterative calculation stops when the average distance between the transformed marker block distribution graphic and the standard marker graphic satisfies the objective function, and the marker block distribution graphic is then designated as the hazardous area marker graphic within the working area.

4. The method according to any one of claims 1-3, characterized in that, The identification of the marked regions in the infrared image includes: Obtain the binarized image of the infrared image; The binarized image is filtered by pixel values ​​to obtain multiple pixel blocks; Calculate the block graphic feature value of each pixel block; Pixel blocks whose graphic feature values ​​meet preset conditions are identified as the identifier blocks in the infrared image.

5. The method according to claim 4, characterized in that, The step of clustering the identified blocks to obtain the identified block distribution pattern includes: Calculate the distance between adjacent identifier blocks; Cluster the identified blocks in the infrared image based on the distance to obtain at least one identified block distribution domain; The target identifier block distribution domain is obtained by filtering the identifier block distribution domain based on preset conditions; The identifier block distribution pattern corresponding to the identifier block distribution domain is determined based on the position of the centroid of each identifier block in the target identifier block distribution domain.

6. The method according to claim 1, characterized in that, Determining the three-dimensional coordinates of the identifier block distribution pattern based on the depth image includes: Identify matching pixels in the depth image that match the pattern of the identified blocks; Obtain the two-dimensional coordinates of the matching pixel in the depth image; The three-dimensional coordinates of the identification block distribution pattern are calculated based on the two-dimensional coordinates and depth values ​​of each matching pixel, as well as the intrinsic and extrinsic parameters of the depth camera.

7. A control device, characterized in that, include: The device is applied to an intelligent robot, which includes: a depth camera; and dangerous area marker graphics are affixed around dangerous areas within the working area of ​​the intelligent robot; the device includes: The image acquisition module is used to acquire infrared and depth images of the working area through the depth camera installed on the intelligent robot; The identifier recognition module is used to identify identifier blocks in the infrared image and cluster the identifier blocks to obtain an identifier block distribution pattern; A coordinate determination module is used to determine the three-dimensional coordinates of the identification block distribution pattern based on the depth image; The matching module is used to match the three-dimensional coordinates of the marked block distribution graphic with the standard marked graphic using a corresponding matching algorithm based on the distribution type of the marked block distribution graphic, to obtain the danger zone marked graphic within the working area; the standard marked graphic is a standard danger zone marked graphic. The control module is used to determine the dangerous areas within the work area based on the dangerous area identification graphic, and to control the intelligent robot to avoid the dangerous areas; The matching module includes: If the distribution type of the marked block distribution graphic is a linear distribution, then a linear fitting matching algorithm is used to match the three-dimensional coordinates of the marked block distribution graphic and the standard marked graphic to obtain the dangerous area marked graphic within the working area; If the distribution type of the marked block distribution graphic is a polygonal distribution, then the iterative nearest point matching algorithm is used to match the three-dimensional coordinates of the marked block distribution graphic and the standard marked graphic to obtain the dangerous area marked graphic within the working area.

8. An intelligent robot, characterized in that, The intelligent robot includes: at least one processor; and A memory communicatively connected to the at least one processor; wherein, The memory stores a computer program that can be executed by the at least one processor, the computer program being executed by the at least one processor to enable the at least one processor to perform the control method according to any one of claims 1-6.

9. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer instructions that cause a processor to execute the control method according to any one of claims 1-6.

Images