Method, device and robot for recognizing a target object

Abstract

The application discloses a target object recognition method, device and robot. In the method, visual sensor data is acquired by using a visual sensor; target object related information is acquired according to the visual sensor data, wherein the related information includes region information of the target object; first position information corresponding to position points collected by the visual sensor is extracted from distance information acquired by a distance measuring device; the region information and each first position information are unified into a coordinate system, and second position information corresponding to each position point in the region information is extracted; and target object position information in the region information is acquired according to the second position information. By using the technical scheme, the position information of the target object can be accurately acquired, and a sudden situation in a dangerous scene can be responded to in time and quickly, thereby greatly improving safety in a robot operation process.

Description

Technical Field

[0001] This invention relates to the field of artificial intelligence, and more specifically, to a method, apparatus, and robot for identifying target objects. Background Technology

[0002] As artificial intelligence (AI) technology matures, intelligent devices are gradually entering people's lives. In the cleaning market, AI technology is playing a significant role in improving work efficiency through automation, thereby freeing up labor. During the operation of automated equipment (e.g., robots), real-time monitoring of the surrounding environment is crucial, especially for predicting, identifying, and making decisions regarding dangerous scenarios (e.g., falls, collisions, overturning) to prevent safety issues.

[0003] In related technologies, for the prediction and identification of dangerous scenarios, taking a fall scenario as an example, the main solutions include: Solution 1, using a fall sensor to identify the fall scenario; Solution 2, using a depth sensor to obtain the three-dimensional coordinates of objects in the scene, and then analyzing them to determine the areas with fall characteristics. The areas detected by the above two solutions are designated as dangerous scenario areas and provided to the control module for decision-making.

[0004] However, drop sensors can only detect falls at close range, which poses challenges to the robot's movement and control, as the short distance may prevent timely decision-making. Depth sensors can predict and identify cliff-like falls, but they cannot handle dangerous scenarios like escalators, because there is a section of the escalator's movement that is level with the ground at the front, which neither of the above two methods can detect. When a robot walks on an escalator step, it can easily be carried away by the escalator, leading to falls, overturns, and other dangerous incidents. Therefore, neither drop sensors nor depth sensors can predict and identify falls on either upward or downward escalators. Summary of the Invention

[0005] The main objective of this invention is to disclose a method, device, and robot for identifying target objects, so as to at least solve the problems in related technologies where, in dangerous scenarios, solutions such as drop sensors and depth sensors are used, the distance is too short to make a decision or cannot detect and identify the target object.

[0006] According to one aspect of the present invention, a method for identifying a target object is provided.

[0007] The target object identification method according to the present invention includes: acquiring visual sensor data using a visual sensor; acquiring target object related information based on the visual sensor data, wherein the related information includes: region information of the target object; extracting first position information corresponding to the position points collected by the visual sensor from distance information acquired by a ranging device; unifying the region information and each of the first position information into a coordinate system, and extracting second position information corresponding to each position point of the region information; and acquiring the position information corresponding to the target object in the region information based on the second position information.

[0008] According to another aspect of the present invention, a target object identification device is provided.

[0009] The target object identification device according to the present invention includes: a first acquisition module, configured to acquire visual sensor data using a visual sensor; a second acquisition module, configured to acquire target object related information based on the visual sensor data, wherein the related information includes: region information of the target object; a first extraction module, configured to extract first position information corresponding to the position points collected by the visual sensor from distance information acquired by a ranging device; a second extraction module, configured to unify the region information and each of the first position information into a coordinate system, and extract second position information corresponding to each position point of the region information; and a third acquisition module, configured to acquire the position information corresponding to the target object in the region information based on the second position information.

[0010] According to another aspect of the present invention, a robot is provided.

[0011] The robot according to the present invention includes: a memory and a processor, wherein the memory is used to store computer-executable instructions; and the processor is used to execute the computer-executable instructions stored in the memory, causing the robot to perform any of the methods described above.

[0012] According to the present invention, a visual sensor is used to acquire visual sensor data; regional information of the target object is acquired based on the visual sensor data; first position information corresponding to the position points collected by the visual sensor is extracted from the distance information acquired by the ranging device; the regional information and each of the first position information are unified into a coordinate system, and second position information corresponding to each position point in the regional information is extracted; the second position information is acquired, and the position information corresponding to the target object in the regional information is acquired. This solves the problem that drop sensors can only identify objects at close range, and may not be able to make decisions in time if the distance is too short, and also solves the problem that depth sensors cannot handle dangerous scenarios such as escalators. By combining a visual sensor and a ranging device, the position information of the target object can be accurately acquired, enabling timely and rapid response to emergencies in dangerous scenarios, greatly improving safety during robot operation. Attached Figure Description

[0013] Figure 1 This is a flowchart of a target object identification method according to an embodiment of the present invention;

[0014] Figure 2 This is a schematic diagram illustrating the region information for identifying a target object according to a preferred embodiment of the present invention;

[0015] Figure 3 This is a schematic diagram illustrating the determination of the position information of a target object according to a preferred embodiment of the present invention;

[0016] Figure 4 This is a schematic diagram illustrating the determination of the position information of a target object according to an embodiment of the present invention;

[0017] Figure 5 This is a schematic diagram of map management according to a preferred embodiment of the present invention;

[0018] Figure 6 This is a schematic diagram of map updating according to a preferred embodiment of the present invention;

[0019] Figure 7 This is a schematic diagram of area updating in a map according to a preferred embodiment of the present invention;

[0020] Figure 8 This is a schematic diagram of a target object identification and decision-making method according to a preferred embodiment of the present invention;

[0021] Figure 9 This is a structural block diagram of a target object identification device according to an embodiment of the present invention;

[0022] Figure 10 This is a structural block diagram of a target object identification device according to a preferred embodiment of the present invention;

[0023] Figure 11This is a structural block diagram of a robot according to an embodiment of the present invention. Detailed Implementation

[0024] The specific implementation of the present invention will now be described in detail with reference to the accompanying drawings.

[0025] According to an embodiment of the present invention, a flowchart of a method for identifying a target object is provided.

[0026] Figure 1 This is a flowchart of a target object identification method according to an embodiment of the present invention. Figure 1 As shown, the method for identifying the target object includes:

[0027] Step S101: Acquire visual sensor data using a visual sensor;

[0028] Step S102: Obtain relevant information about the target object based on the aforementioned visual sensor data, wherein the aforementioned relevant information includes: the region information of the target object;

[0029] Step S103: Extract the first position information corresponding to the position point collected by the visual sensor from the distance information obtained by the ranging device;

[0030] Step S104: Unify the above-mentioned regional information and each of the above-mentioned first position information into a coordinate system, and extract the second position information corresponding to each position point of the above-mentioned regional information;

[0031] Step S105: Based on the second location information mentioned above, obtain the target object location information in the area information mentioned above.

[0032] according to Figure 1 The method described employs a visual sensor to acquire visual sensor data; obtains the region information of the target object based on the visual sensor data; extracts first position information corresponding to the position points collected by the visual sensor from the distance information acquired by the ranging device; unifies the region information and each of the first position information into a coordinate system, and extracts second position information corresponding to each position point in the region information; obtains the second position information, and thus obtains the target object position information from the region information. This method solves the problem that drop sensors can only identify objects at close range, and may not be able to make decisions in time at short distances, and also solves the problem that depth sensors cannot handle dangerous scenarios such as escalators. Combining a visual sensor and a ranging device can accurately obtain the position information of the target object, enabling timely and rapid response to emergencies in dangerous scenarios, greatly improving safety during robot operation.

[0033] The aforementioned relevant information may further include: the category information of the target object. Therefore, after step S105, the second location information, after obtaining the location information corresponding to the target object in the aforementioned area information, may further include: performing corresponding decision-making based on the location information and category information of the target object (e.g., person, vehicle, chair, escalator, animal, etc.). In dangerous scenarios such as escalators during robot operation, when a robot walks on an escalator step, it can easily be carried away by the escalator, leading to falls, overturns, and other dangerous events. Therefore, combining visual sensors and ranging devices can accurately obtain the location information of target objects such as escalators and perform corresponding decision-making (e.g., detour, reversal, etc.). Timely and rapid response to emergencies in dangerous scenarios greatly improves safety during robot operation.

[0034] Preferably, the aforementioned visual sensor can be a camera or the like, and the aforementioned visual sensor data can be an image or a point cloud or the like.

[0035] like Figure 2 As shown, the camera is opened and camera data (e.g., images) is read; a neural network is used to extract data from the camera. Figure 2 The image shown detects the region information of the target object; the image (for example, the image size can be 416*416) is fed into a neural network (which can be an object detection neural network or an instance segmentation network), and the network extracts the region information and category information of the object.

[0036] Preferably, in step S105 above, obtaining the location information corresponding to the target object in the above-mentioned area information may further include: obtaining the distance information corresponding to the target object area in the above-mentioned area information, and determining the target object location information based on the distance information and the target object area.

[0037] Preferably, the ranging device includes, but is not limited to, at least one of the following: a two-dimensional ranging device and a three-dimensional ranging device.

[0038] That is, a two-dimensional ranging device can be used, a three-dimensional ranging device can be used, or a combination of two-dimensional and three-dimensional ranging devices can be used.

[0039] The aforementioned ranging device can be a two-dimensional ranging device (e.g., a lidar sensor). The laser emitted by the two-dimensional ranging device is no longer a pulse signal, but a continuous laser. The result read by the receiver also becomes a set of continuous values. In this way, without moving the rangefinder, it is possible to continuously scan the continuous distance values ​​of objects in a plane (i.e., the plane scanned by the lateral movement of the rangefinder). The returned values ​​can then be used to construct a planar map.

[0040] In the process of optimal implementation, such as Figure 3 As shown, a two-dimensional ranging device (e.g., a lidar with single-line ranging) is used for data calculation. The sensor is turned on, and the two-dimensional ranging device acquires distance information. From this distance information, the first position information corresponding to the position point acquired by the aforementioned visual sensor is extracted.

[0041] D = Distance (x，y)

[0042] The first location information and the region information of the target object (e.g., the location point information within the rectangular frame) are unified into the same coordinate system (e.g., the world coordinate system). The region of the target object is cropped from the current data using the region identified by the visual sensor. Then, the distance information corresponding to each location point of the region information (i.e., the second location information) is obtained. Based on the second location information, the distance information corresponding to the target object region in the region information is obtained. The location information of the target object is determined based on the distance information and the target object region.

[0043] The aforementioned ranging device can be a three-dimensional ranging device (e.g., a depth camera, a multi-view stereo matching sensor, a TOF sensor, etc.). The three-dimensional ranging device emits continuous pulses at the target scene and then uses a sensor to receive the light pulses reflected back from the object's surface. By comparing the phase difference between the emitted light pulses and the light pulses reflected by the object, the propagation delay between the light pulses can be calculated, thus obtaining the distance between the object and the transmitter.

[0044] In the process of optimal implementation, such as Figure 3 As shown, a three-dimensional ranging device (e.g., depth camera, multi-view stereo matching sensor, TOF sensor, etc.) is used for data processing. The sensor is turned on to obtain distance information. From this distance information, the first position information corresponding to the position point acquired by the aforementioned visual sensor is extracted:

[0045] D = Distance (x，y,z)

[0046] The aforementioned first position information and the target object's region information (e.g., position point information within a rectangular frame) are unified into the same coordinate system (e.g., the world coordinate system). The region identified by the visual sensor is used to crop out the target object's region from the current data, which consists of a series of three-dimensional points Point(x, y, z). Then, the distance information corresponding to each position point of the aforementioned region information (i.e., the aforementioned second position information) is obtained. Based on the aforementioned second position information, the distance information corresponding to the target object's region in the aforementioned region information is obtained. Based on the distance information and the aforementioned target object's region, the position information of the aforementioned target object is determined.

[0047] Preferably, when the above-mentioned ranging device includes both a two-dimensional ranging device and a three-dimensional ranging device, it may also include the following two processing methods:

[0048] Method 1: The position information of the target object obtained by the two-dimensional ranging device scheme and the position information of the target object obtained by the three-dimensional ranging device scheme are used to calculate the position distance; the position distance calculation result is compared with a first preset threshold, and the position information of the target object obtained by the two-dimensional ranging device scheme or the three-dimensional ranging device scheme is determined according to the comparison result as the final position information.

[0049] In the preferred implementation, when the positional distance is less than or equal to a first preset threshold, if the three-dimensional ranging device is a depth camera or a TOF sensor, the positional information of the target object obtained by the three-dimensional ranging device can be used as the final positional information; if the three-dimensional ranging device is a multi-view stereo matching device, the positional information of the target object obtained by the two-dimensional ranging device can be used as the final positional information; if the positional distance is greater than the first preset threshold, the positional information of the target object obtained by the two-dimensional ranging device (e.g., a lidar sensor) will be used as the final positional information.

[0050] Method 2: Calculate the distance difference between the distance information obtained using a two-dimensional ranging device and the distance information obtained using a three-dimensional ranging device; compare the calculated distance difference with a second preset threshold, and determine the distance information corresponding to the target object region obtained using either a two-dimensional or three-dimensional ranging device based on the comparison result, as the final distance information.

[0051] In the preferred implementation, when the distance difference is less than or equal to the second preset threshold, if the three-dimensional ranging device is a depth camera or a TOF sensor, the distance information corresponding to the target object region obtained by the three-dimensional ranging device is used as the final distance information; if the three-dimensional ranging device is a multi-view stereo matching device, the distance information corresponding to the target object region obtained by the two-dimensional ranging device can be used as the final distance information; if the distance difference is greater than the second preset threshold, the distance information corresponding to the target object region obtained by the two-dimensional ranging device is used as the final distance information.

[0052] Two-dimensional ranging devices (e.g., lidar sensors) typically offer advantages in long-distance measurement and high accuracy; three-dimensional ranging devices, due to their acquisition of three-dimensional information, are more resistant to occlusion. Therefore, a combined approach using two-dimensional and three-dimensional ranging devices can achieve high-precision distance identification with strong anti-interference capabilities over long distances. Two location or distance results are obtained using both two-dimensional and three-dimensional ranging methods. The distance between the two locations or the difference between the two distances is calculated. If the calculated result exceeds a preset value, severe occlusion is considered, and the result from the three-dimensional ranging method is used. If the calculated result is less than a threshold, the two ranging results are considered essentially identical, and the result from the two-dimensional ranging method is more accurate.

[0053] Preferably, in step S105 above, obtaining the distance information corresponding to the target object region in the above-mentioned region information may further include:

[0054] S1: Determine the maximum and minimum distance values ​​from the second location information mentioned above;

[0055] S2: Set the minimum distance value mentioned above as the initial threshold;

[0056] S3: For the first region whose distance value is greater than the initial threshold in the second location information, calculate the variance and area of ​​the first region. For the second region whose distance value is less than the initial threshold in the second location information, calculate the variance and area of ​​the second region respectively.

[0057] S4: The initial threshold is incremented sequentially. S3 is executed repeatedly. Based on the variance and area of ​​multiple regions, the effective region is calculated and obtained. The distance information of the effective region is used as the distance information of the target object region.

[0058] like Figure 3 and Figure 4 As shown, it is necessary to perform numerical fusion calculations on the cropped target object region to obtain the distance information corresponding to the target object region, and then determine the position information of the target object based on the distance information and the target object region.

[0059] The following explanation uses two-dimensional location region numerical fusion calculation as an example:

[0060] The two-dimensional location region consists of a series of two-dimensional points D(x, y), such as a two-dimensional array, where x and y represent the two dimensions of the coordinate system, and D represents the distance information of the point. Points without distance information are invalid and are ignored here. In existing technologies, the conventional fusion strategy is to take the average value, but this cannot handle the situation of partial occlusion of objects and some noise. The fusion algorithm of this invention can solve this problem well:

[0061] 1. Take the maximum and minimum values ​​from the second position information;

[0062] 2. Set the initial threshold to the minimum value mentioned above;

[0063] 3. Divide the above region into two parts: those with distance values ​​less than the threshold and those with values ​​greater than or equal to the threshold; denoted as R respectively. s and R l ;

[0064] 4. Calculate the mean, variance, and area for each of the two regions;

[0065] Among them, for R s The region, i.e., the region (set of points) whose distance value is less than the threshold, has the following mean Ms: Where Ns is the number of two-dimensional points, and Di is the distance information corresponding to each two-dimensional point, i = 1, 2, 3, ... Ns;

[0066] Variance Var s as follows:

[0067] Where Ms is the mean, Ns is the number of two-dimensional points, and Di is the distance information corresponding to each two-dimensional point, i = 1, 2, 3, ... Ns;

[0068] Area N S That is, a point is considered as a unit area, and the number of points is considered as the area value.

[0069] Similarly, for R l The region, i.e., the set of points whose distance value is greater than or equal to the threshold, has a mean M. l as follows: Where, N l Let Di be the number of two-dimensional points, and Di be the distance information corresponding to each two-dimensional point, i = 1, 2, 3, ... N. l ;

[0070] Variance Var l as follows:

[0071] Among them, M l Let N be the mean. lLet Di be the number of two-dimensional points, and Di be the distance information corresponding to each two-dimensional point, i = 1, 2, 3, ... N. l ;

[0072] Area N l That is, a point is considered as a unit area, and the number of points is considered as the area value.

[0073] It should be noted that the numerical fusion calculation method for three-dimensional location regions is similar to the method described above, and will not be repeated here.

[0074] Preferably, after obtaining the location information corresponding to the target object in the above-mentioned area information based on the second location information, the method may further include at least one of the following: reading the information of the current map; if there is a set in the current map, determining whether the features of the elements in the set meet the preset feature conditions; if not, deleting the set; performing a map update operation on the current map based on the location information of the target object when a predetermined map update condition is met; and triggering a map position correction operation when a predetermined correction condition is met, for example, when the machine's working environment changes, wherein the map position correction operation includes: correcting the coordinates and distance information of each set in the current map and the elements within that set.

[0075] The aforementioned set can be a set of points, a set of line segments, or a set of continuous regions. The aforementioned elements are the components of the set, and can be points, line segments, or continuous regions.

[0076] The location information of the target object is sent to the map management module for map creation; this location information can be a set of points, a region, or a line segment; the following explanation uses a region as an example.

[0077] like Figure 5 As shown, the map management steps include:

[0078] 1. Upon startup, map initialization is performed, including reading existing offline maps from the computer;

[0079] 2. Enter the location area of ​​the new target object;

[0080] 3. Create and delete maps;

[0081] (1) If there is a region on the map, determine whether the number of grid cells in that region is greater than a first preset threshold. Figure 5 If thresh1 is not specified, then delete the region.

[0082] (2) If the associated area does not exist in the map, the new area will be added to the map; if it exists, the map will be updated.

[0083] (3) Determine whether the map update was successful. If it fails (for example, each set in the current map is not associated with the location information of the target object), add the new area to the map.

[0084] 4. When the machine's working environment changes, perform map position correction;

[0085] 5. This correction includes revising the coordinates and distance information of each region and the grid within each region on the map;

[0086] 6. After cleaning is complete, save the current map to your computer.

[0087] Preferably, under the condition of satisfying the predetermined map update, performing a map update operation on the current map based on the location information of the target object may further include: when there is no associated set in the current map (for example, there is no set in the current map, or each set in the current map is not associated with the location information of the target object), adding a new set in the current map based on the location information of the target object; when there is an associated set in the current map, performing a set update operation on the associated set based on the location information of the target object.

[0088] like Figure 6 As shown, the map update process is as follows (using a set of contiguous areas as an example):

[0089] 1. Sequentially acquire and merge the regions from the original map and the new region;

[0090] 2. After obtaining information about two regions, determine the correlation between the two regions. The correlation can be calculated using the intersection-union ratio method (i.e., intersection divided by union).

[0091] 3. If the correlation value is greater than the second preset threshold ( Figure 6 If the thresh2 in the map is the same area, the update is performed and the process ends after the update is successful; or, steps 1 and 2 are executed repeatedly until all areas in the map have been traversed.

[0092] Preferably, updating the current set according to the new set may further include: comparing the elements of the current set and the new set; for a first element that overlaps between the current set and the new set, increasing the weight value of the first element; for a second element newly added to the new set relative to the current set and the aforementioned first element, performing a weight value comparison operation, for elements whose weight value is greater than or equal to a first preset weight value, increasing the weight value of that element, and taking the current element as the new map set; for elements whose weight value is less than the first preset weight value, decreasing the weight value of that element, comparing the decreased weight value with a second preset weight value, and deleting elements that are less than the second preset weight value.

[0093] like Figure 7 As shown, taking a set of continuous regions as an example, the method for updating the current region based on the new region is as follows:

[0094] 1. Divide the semantic map into a grid;

[0095] 2. Examine the grids in each region in turn, and match the grids of the new region with the grids of the original region; when the same grid is matched, the weight of that grid is increased; when no match is matched, the grid is added to the original region.

[0096] 3. After the mesh update is completed, a portion of the meshes are filtered out based on the weight using the first preset threshold; the convex hull of these meshes is then taken to obtain the new region.

[0097] 4. Use the above-mentioned area as the new map area;

[0098] 5. The weight of the grid used in step 3 is increased, and the weight of the unused grid is decreased;

[0099] 6. When the grid weight is lower than the second preset weight, the grid is deleted from the map area.

[0100] The following combination Figure 8 The preferred embodiments described above are further described below.

[0101] Figure 8 This is a schematic diagram of a target object identification and decision-making method according to a preferred embodiment of the present invention.

[0102] like Figure 8 As shown, the target object ( Figure 8 The identification and decision-making methods for escalators (taking escalators as an example) include:

[0103] Step 1: Obtain information about the target object in the image;

[0104] For example, open the camera and read the camera data (i.e., the image); use a neural network to detect the region information of the target object from the image; then feed the image (e.g., 416*416 size) into the neural network (which can be an object detection neural network or an instance segmentation network), and extract the category information (e.g., escalator) and region information through the neural network;

[0105] Step 2: Obtain the position information of the target object.

[0106] Option 1: Use a two-dimensional ranging device (e.g., a single-line lidar).

[0107] A two-dimensional ranging device is used to acquire distance information. From the acquired distance information, the first position information corresponding to the position point collected by the aforementioned visual sensor is extracted. The aforementioned region information and each of the aforementioned first position information are unified into a coordinate system, and the second position information corresponding to each position point of the aforementioned region information is extracted, which consists of a series of two-dimensional points Point(x, y). Based on the aforementioned second position information, the distance information corresponding to the target object region in the aforementioned region information is acquired, and the position information of the aforementioned target object is determined based on the distance information and the aforementioned target object region.

[0108] Among them, obtaining the distance information corresponding to the target object region in the above-mentioned region information includes:

[0109] S1: Determine the maximum and minimum distance values ​​from the second location information mentioned above;

[0110] S2: Set the minimum distance value mentioned above as the initial threshold;

[0111] S3: For the first region whose distance value is greater than the initial threshold in the second location information, calculate the variance and area of ​​the first region. For the second region whose distance value is less than the initial threshold in the second location information, calculate the variance and area of ​​the second region respectively.

[0112] S4: Perform the increment operation on the initial threshold one by one, and repeat the above S3. Based on the variance and area corresponding to multiple regions, calculate the effective region and use the distance information corresponding to the effective region as the distance information.

[0113] Option 2: Use a 3D ranging device (e.g., depth camera, multi-view stereo matching sensor, 3D TOF sensor).

[0114] Distance information is acquired using a three-dimensional ranging device. First position information corresponding to the position points collected by the aforementioned visual sensor is extracted from the acquired distance information. The aforementioned region information and each of the aforementioned first position information are unified into a coordinate system. Second position information corresponding to each position point of the aforementioned region information is extracted, consisting of a series of three-dimensional points Point(x, y, z). Based on the aforementioned second position information, distance information corresponding to the target object region in the aforementioned region information is acquired. The position information of the aforementioned target object is determined based on the distance information and the aforementioned target object region.

[0115] The principle for obtaining the distance information corresponding to the aforementioned target object region is the same as above, and will not be repeated here.

[0116] Option 3: Combine the location or distance information obtained from Option 1 and Option 2 to achieve more accurate distance calculation.

[0117] Specifically, two location information or two distance information results are obtained from Scheme 1 and Scheme 2; the location distance or distance difference is calculated, and the calculated location distance is compared with a first preset threshold. Based on the comparison result, the location information of the target object obtained by the two-dimensional ranging device or the three-dimensional ranging device is determined as the final target object location information. The distance difference calculation result is compared with a second preset threshold, and the distance information corresponding to the target object area obtained by the two-dimensional ranging device or the three-dimensional ranging device is determined as the final distance information. For example, when the location distance is less than or equal to the first preset threshold, it is considered that there is severe occlusion, so the result of Scheme 2 is taken; when the distance difference is greater than the second preset threshold, it is considered that the two ranging results are basically consistent, and the result of Scheme 1 is more accurate.

[0118] Step 3: Map Management (including map updates)

[0119] Read the information of the current map. If there is a region in the current map (taking a set of contiguous regions as an example), determine whether the number of grid cells in the region is greater than a first preset threshold. Figure 5 If the target object is not associated with a region, the region is deleted. If there is no associated region in the current map, a new region is added to the current map based on the location information of the target object. If there is an associated region in the current map, a region update operation is performed on the current map based on the location information of the target object.

[0120] The process of updating the current map based on the location information of the target object (e.g., escalator) may further include: obtaining a region from the current map, obtaining the correlation value between the current region and the new region (e.g., intersection-union ratio; for example, if the intersection is 5 grids and the union is 10 grids, then the intersection-union ratio is 0.5); if the correlation value is greater than a second preset threshold, then the current region is updated based on the new region, and the process ends; or, the above steps are repeated until all regions in the current map have been traversed.

[0121] The process of updating the current region based on the new region can further include: comparing the grids of the current region and the new region; increasing the weight value of the first grid that overlaps with the current region and the new region; performing a weight value comparison operation on the second grid newly added to the new region relative to the current region, increasing the weight value of the grid for grids in the second region that have a weight value greater than or equal to a first preset weight value, and taking the convex hull of the current grid as the new map region; decreasing the weight value of the grid for grids that have a weight value less than the first preset weight value, comparing the decreased weight value with a second preset weight value, and deleting grids that have a weight value less than the second preset weight value.

[0122] Step 4: Feed the semantic map obtained in Step 3 back to the control system for decision-making and planning.

[0123] For example, when the target object is identified as an escalator and its specific location information is obtained, the appropriate operation can be determined based on this information, such as detouring or turning back. This avoids dangerous situations and greatly improves safety during robot operation.

[0124] According to an embodiment of the present invention, a target object identification device is also provided.

[0125] Figure 9 This is a structural block diagram of a target object recognition device according to an embodiment of the present invention. Figure 9 As shown, the target object recognition device includes: a first acquisition module 90, used to acquire visual sensor data using a visual sensor; a second acquisition module 91, used to acquire relevant information about the target object based on the visual sensor data, wherein the relevant information includes: the region information of the target object; a first extraction module 92, used to extract first position information corresponding to the position points collected by the visual sensor from the distance information acquired by the ranging device; a second extraction module 93, used to unify the region information and each of the first position information into a coordinate system, and extract second position information corresponding to each position point of the region information; and a third acquisition module 94, used to acquire the position information corresponding to the target object in the region information based on the second position information.

[0126] use Figure 9The device shown comprises a first acquisition module 90 that acquires visual sensor data; a second acquisition module 91 that acquires the region information of the target object based on the visual sensor data; a first extraction module 92 that extracts first position information corresponding to the position points collected by the visual sensor from the distance information acquired by the ranging device; a second extraction module 93 that unifies the region information and each of the first position information into a coordinate system and extracts second position information corresponding to each position point of the region information; and a third acquisition module 94 that acquires the position information corresponding to the target object in the region information based on the second position information. This solves the problem that drop sensors can only identify objects at close range, and may not be able to make decisions in time at short distances, and also solves the problem that depth sensors cannot handle dangerous scenarios such as escalators. Combining a visual sensor and a ranging device can accurately acquire the position information of the target object, enabling timely and rapid response to emergencies in dangerous scenarios, greatly improving safety during robot operation.

[0127] Preferably, such as Figure 10 As shown, the above-mentioned device may also include, but is not limited to, a map management module 95, wherein the map management module 95 includes at least one of the following:

[0128] The deletion unit is used to read the information of the current map. If there is a set in the current map, it determines whether the characteristics of the elements in the set meet the preset characteristic conditions. If not, the set is deleted.

[0129] The update unit is configured to perform a map update operation on the current map based on the location information of the target object, under predetermined map update conditions. In a preferred embodiment, when there is no associated set in the current map (e.g., there is no set in the current map, or each set in the current map is not associated with the location information of the target object), a new set is added to the current map based on the location information of the target object; when there is an associated set in the current map, a set update operation is performed on the associated set based on the location information of the target object.

[0130] The correction unit is used to trigger a map position correction operation when a predetermined correction condition is met. The map position correction operation includes correcting the coordinates and distance information of each set in the current map and the elements within that set.

[0131] It should be noted that the preferred embodiment of the combination of the modules in the above-mentioned target object recognition device can be found in [reference needed]. Figures 1 to 8 The relevant descriptions and effects in the illustrated embodiments are for understanding purposes only and will not be repeated here.

[0132] According to an embodiment of the present invention, a robot is provided.

[0133] Figure 11 This is a structural block diagram of a robot according to an embodiment of the present invention. Figure 11 As shown, the robot according to the present invention includes: a memory 100 and a processor 102. The memory 100 is used to store computer execution instructions; the processor 102 is used to execute the computer execution instructions stored in the memory, causing the robot to perform the area state determination method provided in the above embodiments.

[0134] Processor 102 may be a central processing unit (CPU). Processor 102 may also be other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, or combinations thereof.

[0135] The memory 100, as a non-transitory computer-readable storage medium, can be used to store non-transitory software programs, non-transitory computer-executable programs, and modules, such as the program instructions / modules corresponding to the target object recognition method in this embodiment of the invention. The processor executes various functional applications and data processing by running the non-transitory software programs, instructions, and modules stored in the memory.

[0136] The memory 100 may include a program storage area and a data storage area. The program storage area may store the operating system and applications required for at least one function; the data storage area may store data created by the processor, etc. Furthermore, the memory may include high-speed random access memory and non-transitory memory, such as at least one disk storage device, flash memory device, or other non-transitory solid-state storage device. In some embodiments, the memory 100 may optionally include memory remotely located relative to the processor, which can be connected to the processor via a network. Examples of such networks include, but are not limited to, the Internet, corporate intranets, local area networks, mobile communication networks, and combinations thereof.

[0137] One or more of the above modules are stored in the memory 100, and when executed by the processor 102, they perform the following: Figure 1 and Figure 8 The target object identification method in the illustrated embodiment.

[0138] For specific details about the aforementioned robots, please refer to the relevant documentation. Figure 1 and Figure 2 The relevant descriptions and effects in the illustrated embodiments are for understanding purposes only and will not be repeated here.

[0139] In summary, using the embodiments provided by this invention, a visual sensor is used to acquire visual sensor data; based on the visual sensor data, the region information and category information of the target object are acquired; from the distance information acquired by the ranging device, the first position information corresponding to the position point collected by the visual sensor is extracted; the region information and each of the first position information are unified into a coordinate system, and the second position information corresponding to each position point in the region information is extracted; the second position information is acquired, and the position information corresponding to the target object in the region information is acquired. Then, based on the position information and category information of the target object, corresponding decision processing is performed. By combining a visual sensor and a ranging device, the technical solution provided by this invention can accurately acquire the position information of the target object, enabling timely and rapid response to emergencies in dangerous scenarios (e.g., collision scenarios, fall scenarios, overturning scenarios, etc.), greatly improving safety during robot operation.

[0140] The above-disclosed embodiments are merely a few specific examples of the present invention. However, the present invention is not limited thereto, and any variations that can be conceived by those skilled in the art should fall within the protection scope of the present invention.

Claims

1. A method for identifying a target object, characterized in that, include: Visual sensor data is acquired using a visual sensor. The relevant information of the target object is obtained based on the visual sensor data, wherein the relevant information includes: the region information of the target object; Extract the first position information corresponding to the position point collected by the visual sensor from the distance information obtained by the ranging device; Unify the region information and each of the first location information into a coordinate system, and extract the second location information corresponding to each location point of the region information; Based on the second location information, distance information corresponding to the target object region in the region information is obtained, and the target object location information is determined based on the distance information and the target object region; The acquisition of distance information corresponding to the target object region in the region information includes: S1: Determine the maximum distance value and the minimum distance value from the second location information; S2: Set the minimum distance value as the initial threshold; S3: For the first region in the second location information whose distance value is greater than the initial threshold, calculate the variance and area of ​​the first region. For the second region in the second location information whose distance value is less than the initial threshold, calculate the variance and area of ​​the second region respectively. The variance of the first region... , The mean, The number of data points. The distance information for each data point, i=1,2,3,... The area of ​​the first region is The variance of the second region , The mean, The number of data points. The distance information for each data point, i=1,2,3,... The area of ​​the second region is ; S4: The initial threshold is incremented sequentially, and S3 is executed repeatedly. Based on the variance and area of ​​multiple regions, an effective region is calculated and obtained. The distance information corresponding to the effective region is used as the distance information corresponding to the target object region.

2. The method according to claim 1, characterized in that, The relevant information also includes: the category information of the target object; then, after obtaining the location information corresponding to the target object in the region information based on the second location information, it further includes: Based on the location information and category information of the target object, perform corresponding decision processing.

3. The method according to claim 1, characterized in that, The ranging device includes at least one of the following: a two-dimensional ranging device and a three-dimensional ranging device.

4. The method according to claim 3, characterized in that, When the ranging device includes a two-dimensional ranging device and a three-dimensional ranging device, acquiring the target object position information in the area information includes: The positional distance is calculated by comparing the positional information of the target object obtained using a two-dimensional ranging device with the positional information of the target object obtained using a three-dimensional ranging device. The location distance calculation result is compared with a first preset threshold, and the location information of the target object obtained by using a two-dimensional ranging device or a three-dimensional ranging device is determined based on the comparison result, and is used as the final target object location information.

5. The method according to claim 1, characterized in that, When the ranging device includes a two-dimensional ranging device and a three-dimensional ranging device, acquiring the distance information corresponding to the target object region in the region information includes: The distance difference between the distance information obtained using a two-dimensional ranging device and the distance information obtained using a three-dimensional ranging device will be calculated. The distance difference calculation result is compared with the second preset threshold, and the distance information corresponding to the target object area obtained by using a two-dimensional ranging device or a three-dimensional ranging device is determined based on the comparison result, and is used as the final distance information.

6. The method according to claim 1, characterized in that, After obtaining the location information corresponding to the target object in the region information based on the second location information, the method further includes at least one of the following: Read the information of the current map. If there is a set in the current map, determine whether the characteristics of the elements in the set meet the preset characteristic conditions. If not, delete the set. Under predetermined map update conditions, a map update operation is performed on the current map based on the location information of the target object; Under predetermined correction conditions, a map position correction operation is triggered, wherein the map position correction operation includes: correcting the coordinates and distance information of each set in the current map and the elements within that set.

7. The method according to claim 6, characterized in that, Under predetermined map update conditions, performing a map update operation on the current map based on the location information of the target object includes: If there is no associated set in the current map, a new set is added to the current map based on the location information of the target object; When there is an associated set in the current map, a set update operation is performed on the associated set based on the location information of the target object.

8. The method according to claim 7, characterized in that, Performing a set update operation on the associated set based on the location information of the target object includes: Compare the elements of the current set with the position information of the target object; For the first element whose position information overlaps with that of the current set and the target object, increase the weight value of the first element; For the position information of the target object relative to the newly added second element and the first element of the current set, a weight value comparison operation is performed; For elements whose weight value is greater than or equal to the first preset weight value, increase the weight value of the element and take the current element as the new map set; For elements whose weight value is less than the first preset weight value, reduce the weight value of the element, compare the reduced weight value with the second preset weight value, and delete the elements that are less than the second preset weight value.

9. A target object identification device, characterized in that, include: The first acquisition module is used to acquire visual sensor data using a visual sensor. The second acquisition module is used to acquire relevant information about the target object based on the visual sensor data, wherein the relevant information includes: region information of the target object; The first extraction module is used to extract first position information corresponding to the position point collected by the visual sensor from the distance information obtained by the ranging device. The second extraction module is used to unify the region information and each of the first position information into a coordinate system, and extract the second position information corresponding to each position point of the region information; The third acquisition module is used to acquire distance information corresponding to the target object region in the region information based on the second location information, and to determine the target object location information based on the distance information and the target object region. The third acquisition module is further configured to acquire distance information corresponding to the target object region in the region information through the following method: S1: Determine the maximum distance value and the minimum distance value from the second location information; S2: Set the minimum distance value as the initial threshold; S3: For the first region in the second location information whose distance value is greater than the initial threshold, calculate the variance and area of ​​the first region. For the second region in the second location information whose distance value is less than the initial threshold, calculate the variance and area of ​​the first region respectively. Calculate the variance and area of ​​the second region, where the variance of the first region is... , The mean, The number of data points. The distance information for each data point, i=1,2,3,... The area of ​​the first region is The variance of the second region , The mean, The number of data points. The distance information for each data point, i=1,2,3,... The area of ​​the second region is ; S4: The initial threshold is incremented sequentially, and S3 is executed repeatedly. Based on the variance and area of ​​multiple regions, an effective region is calculated and obtained. The distance information corresponding to the effective region is used as the distance information corresponding to the target object region.

10. The apparatus according to claim 9, characterized in that, The device further includes: a map management module, wherein the map management module includes at least one of the following: The deletion unit is used to read the information of the current map. If there is a set in the current map, it determines whether the characteristics of the elements in the set meet the preset characteristic conditions. If not, the set is deleted. The update unit is used to perform a map update operation on the current map based on the location information of the target object, provided that a predetermined map update condition is met. The correction unit is used to trigger a map position correction operation when a predetermined correction condition is met. The map position correction operation includes correcting the coordinates and distance information of each set in the current map and the elements within that set.

11. A robot, comprising: Memory and processor, characterized in that, Memory is used to store instructions executed by the computer; A processor for executing computer execution instructions stored in memory, causing a robot to perform the method as claimed in any one of claims 1 to 8.

Images