A tracking object detection method, device, equipment and computer storage medium

Abstract

The application is suitable for the field of robots, and provides a tracking object detection method, device and equipment and computer storage medium, which are applied to robots. The method comprises the following steps: continuously acquiring multiple laser mask images by a laser radar, updating an initial weight image based on the multiple laser mask images to obtain a first weight image; when a to-be-tracked object is recognized in the laser mask image, a camera acquires a first image of an object angle range corresponding to the to-be-tracked object; if the first image comprises the to-be-tracked object, a second weight image is obtained by updating the first weight image based on a first detection angle range of the laser radar, a second detection angle range of the camera and the object angle range; and a target tracking object is determined based on the second weight image. In the method, the camera and the laser radar are used together to detect the tracking object, which effectively reduces the false detection rate of the tracking object detected by using only the laser radar, and improves the accuracy of the tracking object detection.

Description

Technical Field

[0001] This application belongs to the field of robotics, and in particular relates to a method, apparatus, device, and computer storage medium for tracking object detection. Background Technology

[0002] Due to its high resolution and strong anti-interference capabilities, lidar is widely used in the detection of tracked objects. A common method for lidar object detection is to determine a weighted image from the lidar-acquired laser mask image, and then directly analyze this weighted image to identify the target. However, this method suffers from inaccurate detection results due to the potential presence of significant noise in the point cloud data, thus reducing detection efficiency. Summary of the Invention

[0003] The purpose of this application is to provide a tracking object detection method, apparatus, device, and computer storage medium, aiming to improve the accuracy of tracking object detection.

[0004] The embodiments of this application are implemented as follows: Firstly, this application provides a tracking object detection method applied to a robot, the robot including a lidar and a camera. The method includes: continuously acquiring multiple frames of lidar mask images using the lidar; updating an initial weight image based on the multiple frames of lidar mask images to obtain a first weight image; when a target object is identified in the lidar mask image, acquiring a first image of the object angle range corresponding to the target object using the camera; if the first image includes the target object, updating the first weight image using the first detection angle range of the lidar, the second detection angle range of the camera, and the object angle range to obtain a second weight image; and determining the target object to be tracked based on the second weight image.

[0005] The tracking object detection method provided in this application first updates the initial weight image using the laser mask image acquired by the LiDAR to obtain a first weight image. Then, it acquires the first image using the camera's visibility and performs secondary identification on the object to be tracked in the laser mask image. If the object to be tracked is present in the first image, the first weight image is updated to obtain a second weight image. Finally, the target tracking object is determined based on the second weight image. By using both the LiDAR and the camera to detect the object to be tracked, the false detection rate of tracking object detection using only the LiDAR is effectively reduced, and the accuracy of tracking object detection is improved.

[0006] In one embodiment, updating the initial weight image based on multiple frames of laser mask images to obtain a first weight image includes:

[0007] An initial weighted image is determined based on the laser mask image, with each pixel being a first grid value;

[0008] Based on the raster value corresponding to each pixel in the multi-frame laser mask image, the raster value of the pixel in the initial weight image corresponding to each pixel position in the laser mask image is updated sequentially according to the first preset rule to obtain the first weight image.

[0009] In one embodiment, the step of updating the grid values ​​of pixels in the initial weighted image corresponding to each pixel position in the laser mask image according to the grid values ​​corresponding to each pixel in the multi-frame laser mask image in sequence according to a first preset rule includes:

[0010] When the grid value of a pixel in the laser mask image is the first threshold, the grid value of the pixel in the initial weight image corresponding to the pixel position in the laser mask image is increased by the first parameter value.

[0011] When the grid value of a pixel in the laser mask image is the second threshold, the grid value of the pixel in the initial weight image corresponding to the pixel position in the laser mask image will be reduced by the second parameter value.

[0012] In one embodiment, updating the first weighted image using the first detection angle range of the lidar, the second detection angle range of the camera, and the object angle range to obtain the second weighted image includes:

[0013] The area within the second detection angle range of the camera where neither the lidar nor the camera detects the object to be tracked is defined as the first area; the area where both the lidar and the camera detect the object to be tracked is defined as the second area.

[0014] The raster values ​​of pixels in the first weighted image corresponding to the first region that are greater than the second threshold are updated to the first threshold.

[0015] The third parameter is used to reduce the raster values ​​of pixels in the first weighted image corresponding to the second region that are greater than the second threshold.

[0016] In one embodiment, determining the target tracking object based on the second weighted image includes:

[0017] Obtain all human leg regions in the laser mask image;

[0018] Match the two nearest leg regions in all leg regions;

[0019] If a match is successful, the center of the two people's leg regions will be used as the location of the object to be tracked.

[0020] If a match fails, the center of the nearest human leg will be used as the location of the object to be tracked.

[0021] The target location is determined in the second weighted image based on the location of the object to be tracked.

[0022] If the sum of the raster values ​​of the pixels within a circle with the target location as the center and a preset length as the radius is less than a second preset threshold, then the object to be tracked is taken as the target tracking object.

[0023] In one embodiment, after determining the target object to be tracked based on the second weighted image, the method further includes:

[0024] The Kalman filter algorithm is used to track the target object.

[0025] Secondly, embodiments of this application provide a tracking object detection device applied to a robot, the robot including a lidar and a camera, the device comprising:

[0026] The acquisition unit is used to continuously acquire multiple frames of laser mask images through the lidar, and update the initial weight image based on the multiple frames of laser mask images to obtain the first weight image;

[0027] The first judgment unit is used to acquire a first image of the object angle range corresponding to the object to be tracked through a camera when the object to be tracked is identified in the laser mask image.

[0028] The second judgment unit is used to update the first weighted image to obtain the second weighted image if the first image includes the object to be tracked, by using the first detection angle range of the lidar, the second detection angle range of the camera, and the object angle range.

[0029] The determination unit is used to determine the target tracking object based on the second weighted image.

[0030] In one embodiment, updating the initial weight image based on multiple frames of laser mask images to obtain a first weight image includes:

[0031] An initialization unit is used to determine an initial weight image corresponding to the laser mask image, with each pixel being a first grid value, based on the laser mask image;

[0032] The update unit is used to update the grid values ​​of pixels in the initial weight image corresponding to each pixel position in the laser mask image according to the grid values ​​corresponding to each pixel in the multi-frame laser mask image in sequence according to the first preset rule, so as to obtain the first weight image.

[0033] Thirdly, embodiments of this application provide a terminal device, the device comprising: a processor and a memory, the memory for storing a computer program, and the processor for calling and running the computer program from the memory, causing the device to perform the method described in any one of the first and second aspects.

[0034] Fourthly, embodiments of this application provide a computer-readable storage medium storing a computer program that, when executed by a processor, causes the processor to perform the method described in any one of the first and second aspects.

[0035] Fifthly, embodiments of this application provide a computer program product comprising: computer program code, which, when executed by a computer, causes the computer to perform the method described in either the first or second aspect.

[0036] It is understood that the beneficial effects of the second to fifth aspects mentioned above can be found in the relevant descriptions in the first aspect mentioned above, and will not be repeated here. Attached Figure Description

[0037] Figure 1 This is a flowchart of a tracking object detection method provided in an embodiment of this application;

[0038] Figure 2 This is a schematic diagram of the first detection angle range of the lidar provided in the embodiments of this application;

[0039] Figure 3 This is a schematic diagram of the positional information between the robot and the tracked object provided in an embodiment of this application;

[0040] Figure 4 This is a schematic diagram of the structure of a tracking object detection device provided in an embodiment of this application;

[0041] Figure 5 This is a schematic diagram of the structure of a terminal device provided in an embodiment of this application. Detailed Implementation

[0042] To make the objectives, technical solutions, and advantages of this application clearer, the embodiments of this application will be described in further detail below with reference to the accompanying drawings.

[0043] Due to its high resolution and strong anti-interference capabilities, lidar is widely used in the detection of tracked objects. A common method for lidar object detection is to determine a weighted image from the lidar-acquired laser mask image, and then directly analyze this weighted image to identify the target. However, this method suffers from inaccurate detection results due to the potential presence of significant noise in the point cloud data, thus reducing detection efficiency.

[0044] To improve the accuracy of object detection, this application provides an object detection method. First, an initial weight image is updated using a laser mask image acquired by a LiDAR to obtain a first weight image. Then, the first image is acquired using the camera's visibility. The object to be tracked in the laser mask image is then re-identified. If the object to be tracked is present in the first image, the first weight image is updated to obtain a second weight image. Finally, the target object is determined based on the second weight image. By using both LiDAR and a camera to detect the object, the false detection rate of object detection using only LiDAR is effectively reduced, thus improving the accuracy of object detection.

[0045] The execution subject of the tracking object detection method provided in this application embodiment can be a robot equipped with a lidar and a camera, or other terminal devices capable of processing the data collected by the lidar and camera. The terminal device can be a laptop, handheld computer, personal computer (PC), tablet computer, mobile phone, or other devices.

[0046] Based on actual testing, the object detection method provided in this application can be applied to the object detection process of a robot combining a single-line LiDAR and a camera. The single-line LiDAR has only one laser emitter and one laser receiver. Using the object detection method provided in this application improves the utilization rate of the single-line LiDAR. It is easy to understand that the LiDAR can also be a multi-line LiDAR, which is a laser rotating rangefinder that simultaneously emits and receives multiple laser beams. This application does not limit the type of LiDAR installed on the robot. Tracked objects include, but are not limited to, people or other objects.

[0047] The following describes the technical solution of this application in detail, taking pedestrian detection as an example. The embodiments described below with reference to the accompanying drawings are exemplary and intended to explain this application, and should not be construed as limiting this application.

[0048] In one possible implementation, such as Figure 1 The diagram shown is a flowchart of a tracking object detection method provided in an embodiment of this application. See also... Figure 1 This method is applied to robots equipped with lidar and cameras, enabling the robot to detect pedestrians.

[0049] The object detection method includes the following steps S101-S104.

[0050] S101: Continuously acquire multiple frames of laser mask images using a lidar, and update the initial weight image based on the multiple frames of laser mask images to obtain the first weight image.

[0051] In this application, the laser mask image is an image formed by two grid values: 0 and 255. One grid is equivalent to one pixel, and each pixel corresponds to one grid value. The meaning of these two grid values ​​can be defined according to the actual application. For example, if there is no object at the pixel, the grid value corresponding to that pixel in the mask image can be defined as 0; conversely, if there is an object at the pixel, the grid value corresponding to that pixel in the mask image can be defined as 255.

[0052] It should be understood that different types of LiDAR collect different types of data, and corresponding LiDAR mask images are created based on these different data types. For example, when the LiDAR is a single-line LiDAR, the point cloud data collected by the LiDAR is 2D point cloud data, and a corresponding LiDAR mask image is created based on the 2D point cloud data.

[0053] It's worth noting that because the point cloud data acquired by LiDAR and the images acquired by the camera belong to different coordinate systems, the point cloud data acquired by LiDAR cannot be directly used for object detection. Therefore, a coordinate system transformation is required for the acquired point cloud data. Thus, in practical applications, each frame of the multi-frame LiDAR mask image continuously acquired by LiDAR involves determining the LiDAR mask image corresponding to the point cloud data based on the positional relationship between the LiDAR and the robot, and the preset image resolution.

[0054] In other words, firstly, based on the installation position of the LiDAR on the robot, the point cloud data collected by the LiDAR is transformed into the coordinate system of the robot, resulting in the corresponding transformed point cloud data in the robot coordinate system; then, based on the preset image resolution, the corresponding transformed point cloud data in the robot coordinate system is transformed into the image coordinate system. In this way, a LiDAR mask image corresponding to the point cloud data is established through the transformation of the coordinate system.

[0055] The preset image resolution refers to the smallest unit of distance that the image can distinguish. For example, if one pixel in an image can clearly represent a distance of 0.05 meters in reality, then the resolution of that image is 0.05 meters. It is easy to understand that in this embodiment, the preset image resolution is determined by the detection accuracy of the LiDAR, and can be flexibly adjusted according to different practical applications.

[0056] Taking a square area centered on the robot with sides of 4 meters as an example, when the preset image resolution is 0.05 meters, a laser mask image of size 80*80 corresponding to the point cloud data can be constructed. It should be understood that, based on practical applications, LiDAR can measure relatively long distances, but the farther the measured distance, the sparser the acquired point cloud data becomes, which is detrimental to the detection of the tracked object. Therefore, this application can construct an 80*80 laser mask image based on the above example.

[0057] In one embodiment, updating an initial weight image based on multiple frames of laser mask images to obtain a first weight image includes: determining an initial weight image corresponding to the laser mask image and where each pixel is a first grid value based on the laser mask image; and updating the grid values ​​of pixels in the initial weight image corresponding to each pixel position in the laser mask image according to a first preset rule based on the grid values ​​corresponding to each pixel in the multiple frames of laser mask images to obtain the first weight image.

[0058] In a specific embodiment, the initial weight image can be the same size as the laser mask image or they can be different sizes.

[0059] In an optional embodiment, the initial weight image can be a global image, while the laser mask image can be a local image. For each pixel of each frame of the laser mask image, the corresponding pixel can be found in the initial weight image, so that the initial weight image and the laser mask image can correspond to each other.

[0060] In other embodiments, the initial weight image may also be a local image, just like the laser mask image, and its size may be the same as the laser mask image; this is not a limitation.

[0061] The first weighted image is a corresponding laser mask image, determined based on multiple laser mask images. This first weighted image is used to describe the probability of tracking objects in the predicted point cloud data.

[0062] In some embodiments, the initial weight image refers to a weight image after uniformly initializing the raster values ​​corresponding to all pixels in the image. For example, the raster values ​​corresponding to all pixels in the initial weight image are all set to 0. Of course, depending on different actual situations, the raster values ​​corresponding to all pixels in the weight image can also be uniformly set to other values.

[0063] Optionally, the step of updating the grid values ​​of pixels in the initial weight image corresponding to each pixel position in the laser mask image according to the grid values ​​corresponding to each pixel in the multi-frame laser mask image in sequence according to the first preset rule includes: when the grid value of a pixel in the laser mask image is a first threshold, then the grid value of the pixel in the initial weight image corresponding to the pixel position in the laser mask image is increased by a first parameter value; when the grid value of a pixel in the laser mask image is a second threshold, then the grid value of the pixel in the initial weight image corresponding to the pixel position in the laser mask image is decreased by a second parameter value.

[0064] It should be understood that the first threshold and the second threshold can be determined according to the actual application. For example, if there is no object on the pixel, the grid value corresponding to the pixel in the mask image can be defined as 0. Conversely, if there is an object on the pixel, the grid value corresponding to the pixel in the mask image can be defined as 255.

[0065] It is easy to understand that the first and second parameter values ​​are used to further increase the difference between the object to be tracked and the non-tracked object (e.g., the background in the image) in the initial weight image, thereby improving the detection accuracy of the object to be tracked in the first weight image. Therefore, in the embodiments of this application, the specific values ​​of the first and second parameter values ​​can be determined based on the detection results of this method in practical applications, and this application does not impose any limitations on the specific values ​​of the first and second parameter values.

[0066] In an optional embodiment, the first parameter value can be 10, 20, or 30. The second parameter value can also be 10, 20, or 30; there is no limitation on this.

[0067] To reduce the impact of noise data in point cloud data on the detection results of tracked objects, optionally, the laser mask image is dilated; the raster values ​​of each pixel in the initial weight image are updated based on the dilated laser mask image to obtain the first weight image.

[0068] After obtaining the laser mask image, the laser mask image is dilated to obtain the dilated laser mask image; then, based on the grid values ​​in the dilated laser mask image, the grid values ​​corresponding to each pixel in the weight image are adjusted accordingly.

[0069] For example, suppose that in the dilated laser mask image, the grid value corresponding to a pixel with no object is 0, and the grid value corresponding to a pixel with an object is 255. Then, if the grid value corresponding to a pixel in the dilated laser mask image is 255, then the grid value of the pixel in the initial weight image corresponding to that pixel in the dilated laser mask image is increased by 20; if the grid value corresponding to a pixel in the dilated laser mask image is 0, then the grid value of the pixel in the initial weight image corresponding to that pixel in the dilated laser mask image is decreased by 10.

[0070] Based on the above example, in order to ensure the validity of the data and facilitate calculation, the maximum value of the raster value of each pixel in the initial weight image determined by the dilated laser mask image is 255, and the minimum value is 0.

[0071] It's easy to understand that by sequentially updating the grid values ​​of pixels in the initial weight image corresponding to each pixel position in the laser mask image according to the first preset rule, based on the grid values ​​corresponding to each pixel in the multi-frame laser mask image, a first weight image can be obtained. The probabilities of tracking objects and obstacles in the point cloud data can be determined based on the grid values ​​corresponding to each pixel in the first weight image. Based on the above example, when the grid values ​​in the weight image are smaller, the probability of tracking objects is higher, and the probability of obstacles is lower; conversely, when the grid values ​​in the weight image are larger, the probability of tracking objects is lower, and the probability of obstacles is higher.

[0072] In specific scenarios, considering that the tracked object (pedestrian) is moving, its grid value on the first weighted image will not be too large (it may increase or decrease). However, for an obstacle, since the obstacle is stationary and will not move, the grid value corresponding to the obstacle will keep increasing. Therefore, it can be inferred that if the grid value is very large, it is very likely to be an obstacle.

[0073] In practical applications, in order to improve the accuracy of the weighted image and thus enhance the accuracy of object detection, this application can use multi-frame dilated laser mask images to determine the first weighted image. That is, multiple laser mask images can be dilated to obtain a dilated laser mask image corresponding to the multiple laser mask images. The grid values ​​of each pixel in the preset initial weighted image are iteratively updated based on the multiple dilated laser mask images to obtain the first weighted image.

[0074] For example, the initial weight image after the first update is determined based on the first frame of the dilated laser mask image, and then the values ​​of the raster values ​​corresponding to each pixel in the first updated initial weight image are continuously changed based on the 19 consecutive frames of dilated laser mask images received, so as to obtain a first weight image that can be used to detect the tracked object.

[0075] S102, when the object to be tracked is identified in the laser mask image, the first image of the object angle range corresponding to the object to be tracked is acquired by the camera.

[0076] Optionally, when an object to be tracked is identified in the laser mask image, a first image of the object's angular range corresponding to the object to be tracked can be obtained by using a camera.

[0077] Optionally, the detection range of the lidar is a first detection angle range, and the detection range of the camera is a second detection angle range.

[0078] In an optional embodiment, the second detection angle range is smaller than the first detection angle range. Therefore, when the object to be tracked is identified in the laser mask image, there may be two situations: the first situation is that the object to be tracked is also within the second detection angle range of the camera, and the second situation is that the object to be tracked is not within the second detection angle range of the camera. If it is the second situation, the camera or robot can be controlled to rotate to adjust the shooting angle of the camera so that the camera can acquire the first image of the object angle range corresponding to the object to be tracked.

[0079] The detection method can be any image-based deep neural network algorithm, such as the YOLO (You Only Look Once) object detection algorithm, or other algorithms that can detect the object to be tracked. This application does not limit this method.

[0080] After acquiring a laser mask image using point cloud data obtained from a LiDAR, the camera is controlled to acquire a first image. This first image can refer to several image data points acquired by the camera within the LiDAR's scanning range, corresponding to multiple angle ranges within which the tracked object exists.

[0081] It should be understood that when multiple tracking objects exist in the acquired laser mask image, and these multiple tracking objects are at different angles in the laser mask image, the camera can be controlled to acquire multiple first images corresponding to the multiple tracking objects in the laser mask image. The information of the tracking object determined by different first image data may be different.

[0082] In practical applications of pedestrian detection, in order to improve the speed of pedestrian detection and ensure the detection rate, the first image is the image data acquired by the camera that is relatively close to the camera position within the scanning range of the LiDAR.

[0083] S103, if the first image includes the object to be tracked, the first weighted image is updated using the first detection angle range of the lidar, the second detection angle range of the camera, and the object angle range to obtain the second weighted image.

[0084] In an optional embodiment, if the first image also includes the object to be tracked, the first weighted image is updated using the first detection angle range of the lidar, the second detection angle range of the camera, and the object angle range to obtain the second weighted image.

[0085] In an optional embodiment, since the first weighted image is continuously updated over time, when the object to be tracked is not within the second detection angle range of the camera, the robot or camera will be controlled to take a picture of the object to be tracked, which will consume some time. Therefore, to avoid the above situation, this embodiment updates the first weighted image with the first detection angle range of the lidar, the second detection angle range of the camera, and the object angle range to obtain the second weighted image. The first weighted image can be the first weighted image at the time frame corresponding to the first image. Here, "corresponding" specifically refers to the first weighted image with the shortest time interval from the first image.

[0086] Selectively, the first weighted image is the weighted image corresponding to the time frame of the first image.

[0087] In other embodiments, the first weight image corresponding to the time frame and the laser mask image when the object to be tracked is determined can also be updated. That is, when the laser mask image determines that the object to be tracked is determined at a certain moment, the first weight image corresponding to that moment is determined and the first weight image is updated in the subsequent calculation process.

[0088] In one embodiment, updating the first weighted image corresponding to the first image time frame using the first detection angle range of the lidar, the second detection angle range of the camera, and the object angle range to obtain the second weighted image includes:

[0089] In an optional embodiment, both the lidar and the camera can use preset extrinsic parameters to convert the information into the same coordinate system, specifically, into the same robot's coordinate system.

[0090] The first step is to define the area within the second detection angle range of the camera where neither the lidar nor the camera detects the object to be tracked as the first area; and the area where both the lidar and the camera detect the object to be tracked as the second area.

[0091] As an example, not a limitation, such as Figure 2 The diagram shows the first detection angle range of the lidar, and the second detection angle range of the camera is... Figure 2 In the A1 and A2 regions, the first region where neither the LiDAR nor the camera detects the object to be tracked is marked as A1, and the second region where both the LiDAR and the camera detect the object to be tracked is marked as A2. In practical applications, the position information of the object to be tracked, i.e. the angular range of the A2 region, can be obtained in the following way.

[0092] like Figure 3 The diagram shown is a schematic representation of the positional information between the robot and the tracked object provided in an embodiment of this application. (See also...) Figure 3 Detect the first region where the object to be tracked is located in the first image, define the distance between the first region and the center of the first image as DX, and set the focal length of the camera as F. Then, the orientation θ of the object to be tracked relative to the robot can be determined according to formula (1):

[0093]

[0094] In the above formula (1), the focal length of the camera refers to the distance between the camera's focal point and the imaging plane. In practical applications, the focal length of the camera can be obtained through camera calibration. Based on the above formula (1), the position information of the object to be tracked, i.e., the angular range of region A2, can be determined.

[0095] It should be understood that in practical applications, the division of the A2 region can be based on the calculated orientation of the tracked object relative to the robot, with the addition of a preset range. For example, if the value of θ is in the range of 30°, then the range of 25°-35° can be marked as A2 according to the actual application, that is, the region of θ±5° can be marked as the A2 region.

[0096] The second step involves updating the raster values ​​of pixels in the first weighted image corresponding to the first region that are greater than the second threshold to the first threshold. The first weighted image is the weighted image corresponding to the time frame of the first image. The third parameter is then reduced for pixels in the first weighted image corresponding to the second region whose raster values ​​are greater than the second threshold. For example, based on the embodiments of this application, the third parameter can be the sum of the first and second parameters. This application does not impose any limitations on the value of the third parameter.

[0097] It is worth noting that the third parameter can be set according to different actual application processes. That is, in the embodiments of this application, the third parameter is a value, but for different actual application situations, the third parameter can also be other parameters, and this application does not limit it.

[0098] Based on the above example, since area A1 is an area where neither the LiDAR nor the camera detected the tracked object, the raster value corresponding to the pixel in the weighted image of this area can be configured to 255, indicating that the probability of this area being the tracked object in the point cloud data acquired by the LiDAR is relatively small, and the probability of it being an obstacle is relatively large.

[0099] In region A2, both the LiDAR and the camera detected the tracked object, so it is necessary to reduce the raster value corresponding to the pixel in the weight image of this region. In this embodiment, during the update of the initial weight image, if the raster value in the dilated LiDAR mask image is 255, a first parameter value is added to the raster value of the pixel in the weight image corresponding to the pixel in the dilated LiDAR mask image. This increases the probability of an obstacle. Therefore, for region A2, the raster value corresponding to the pixel in the weight image of this region can be reduced by the sum of the first and second parameters mentioned above. The minimum value of each raster value after updating the weight image of this region is 0.

[0100] The first weight image can be updated using the method described above to obtain the second weight image, and then the target object can be determined based on the second weight image.

[0101] S104, determine the target object to be tracked based on the second weighted image.

[0102] Alternatively, the target object to be tracked can be determined based on the object to be tracked and the second weight image together.

[0103] The step of identifying the object to be tracked in the laser mask image is as follows:

[0104] Obtain all human leg regions in the laser mask image;

[0105] Match the two nearest leg regions in the human leg region;

[0106] If a match is successful, the two leg regions will be taken as the objects to be tracked, and the center of the two leg regions will be taken as the position of the objects to be tracked.

[0107] In a specific scenario, the legs of a pedestrian will not be too far apart. Therefore, if the distance between the two legs is less than the distance threshold, it can be considered a successful match.

[0108] If a match fails, the nearest leg region will be selected as the target to be tracked, and the center of that leg region will be used as the location of the target to be tracked.

[0109] Further determination of the target tracking object based on the second weighted image includes:

[0110] The target position is determined in the second weighted image based on the position of the object to be tracked. That is, the target position is determined in the second weighted image based on the position of the object to be tracked. If the sum of the grid values ​​of the pixels in the circle with the target position as the center and a preset length as the radius is less than the second preset threshold, then the object to be tracked is taken as the target tracking object.

[0111] Alternatively, based on similar considerations, if it is a pedestrian, the second weighted image will be reduced as the pedestrian is constantly moving. Furthermore, due to the addition of the camera, the corresponding raster values ​​in the second weighted image are further reduced when the object is identified as the target. Therefore, if the sum of the raster values ​​of the pixels within a circle with the target location as the center and a preset length as the radius is less than the second preset threshold, the target object can be considered as the target object.

[0112] Methods for detecting the human leg region in a laser mask image include, but are not limited to, using clustering methods to detect the geometric size (including the width and length of each cluster of point cloud data) in the point cloud data collected by the lidar. If the geometric size of each cluster of point cloud data is within a preset geometric threshold range, then the human leg region of the pedestrian corresponding to that cluster of point cloud data is determined.

[0113] The second preset threshold can be determined based on multiple experiments with the human leg in the second weighted image during the actual detection process. This application does not impose any restrictions on the value of the second preset threshold.

[0114] It is easy to understand that the aforementioned preset geometric threshold, first preset threshold, and second preset threshold can all be determined according to the actual application situation, and this application does not impose any restrictions on them.

[0115] In one embodiment, after determining the target object to be tracked, the method further includes: tracking the target object using a Kalman filter algorithm.

[0116] It is easy to understand that by using the object detection method provided in this application to acquire multiple frames of point cloud data about a target pedestrian, and based on the changes between the acquired multiple frames of point cloud data about the target pedestrian, the movement state information of the target pedestrian can be determined. This movement state information includes the target pedestrian's walking speed, direction, and position. The acquired movement state information of the target pedestrian can be applied to different practical application scenarios; for example, it can be used in the process of robot route planning for a target pedestrian.

[0117] In some embodiments, the Kalman filter algorithm is used to predict the movement state of the target pedestrian. The predicted target pedestrian is associated with the pedestrian detected at the current time that has the smallest distance to it. If the pedestrian detected at the current time is not associated with the predicted target pedestrian, the pedestrian detected at the current time can be considered a newly appeared pedestrian, and an identifier number and a score (e.g., a score of 1) are assigned to it. The identifier number is used to identify the target pedestrian. If the pedestrian detected at the current time is associated with the predicted target pedestrian, the target pedestrian continues to use its corresponding identifier number and is assigned a score. If the pedestrian was associated with the predicted target pedestrian in the previous frame, but the pedestrian detected at the current time is not associated with the predicted target pedestrian, the target pedestrian is considered to have lost tracking, and the target pedestrian's score is reduced by one, and the number of tracking loss is recorded.

[0118] Optionally, the target pedestrian can be deleted after the number of tracking errors reaches a preset tracking threshold.

[0119] In one possible implementation, based on the tracking results of the target pedestrians, the target pedestrians whose scores are greater than a preset score threshold within a certain period of time are identified as pedestrians with stable tracking.

[0120] This application provides a tracking object detection method. First, a laser mask image is acquired using point cloud data collected by a LiDAR. Based on the acquired multi-frame laser mask image, an initial weight image is updated to obtain a first weight image. Then, after the target object is identified in the laser mask image, a camera is controlled to acquire a first image corresponding to the angular range of the target object identified in the laser mask image. If the target object is detected in the first image, the first weight image is updated using the detection angular range of the LiDAR, the detection angular range of the camera, and the angular range of the target object to obtain a second weight image. The first weight image is a weight image corresponding to a time frame of the first image. Finally, the target object is determined based on the second weight image. This method uses the laser mask image acquired by the LiDAR to update the initial weight image to obtain a first weight image. Then, it uses the camera's visibility to acquire the first image, performs secondary identification of the target object in the laser mask image, and updates the first weight image to obtain a second weight image when the target object is present in the first image. Finally, the target object is determined based on the second weight image. By using both lidar and camera to detect the object being tracked, the false detection rate of tracking objects using lidar alone is effectively reduced, and the accuracy of tracking object detection is improved.

[0121] like Figure 4As shown, this application embodiment also provides a tracking object detection device applied to a robot, wherein the robot includes a LiDAR and a camera. The tracking object detection device 200 includes:

[0122] The acquisition unit 201 is used to continuously acquire multiple frames of laser mask images through the lidar, and update the initial weight image based on the multiple frames of laser mask images to obtain the first weight image;

[0123] The first judgment unit 202 is used to acquire a first image of the object angle range corresponding to the object to be tracked through a camera when the object to be tracked is identified in the laser mask image;

[0124] The second judgment unit 203 is used to update the first weighted image to obtain the second weighted image if the first image includes the object to be tracked, by using the first detection angle range of the lidar, the second detection angle range of the camera and the object angle range.

[0125] The determination unit 204 is used to determine the target tracking object based on the second weight image.

[0126] In one embodiment, the acquisition unit 201 updates the initial weight image based on multiple frames of laser mask images to obtain a first weight image, including:

[0127] An initialization unit is used to determine an initial weight image corresponding to the laser mask image, with each pixel being a first grid value, based on the laser mask image;

[0128] The update unit is used to update the grid values ​​of pixels in the initial weight image corresponding to each pixel position in the laser mask image according to the grid values ​​corresponding to each pixel in the multi-frame laser mask image in sequence according to the first preset rule, so as to obtain the first weight image.

[0129] In one embodiment, the step of updating the grid values ​​of pixels in the initial weighted image corresponding to each pixel position in the laser mask image according to the grid values ​​corresponding to each pixel in the multi-frame laser mask image in sequence according to a first preset rule includes:

[0130] When the grid value of a pixel in the laser mask image is the first threshold, the grid value of the pixel in the initial weight image corresponding to the pixel position in the laser mask image is increased by the first parameter value.

[0131] When the grid value of a pixel in the laser mask image is the second threshold, the grid value of the pixel in the initial weight image corresponding to the pixel position in the laser mask image will be reduced by the first parameter value.

[0132] In one embodiment, the second determination unit 203 updates the first weighted image to obtain a second weighted image using the first detection angle range of the lidar, the second detection angle range of the camera, and the object angle range, including:

[0133] The area within the second detection angle range of the camera where neither the lidar nor the camera detects the object to be tracked is defined as the first area; the area where both the lidar and the camera detect the object to be tracked is defined as the second area.

[0134] The raster values ​​of pixels in the first weighted image corresponding to the first region that are greater than the second threshold are updated to the first threshold.

[0135] The third parameter is used to reduce the raster values ​​of pixels in the first weighted image corresponding to the second region that are greater than the second threshold.

[0136] In one embodiment, the determining unit 204 is further configured to:

[0137] Obtain all human leg regions in the laser mask image;

[0138] Match the two nearest leg regions in all leg regions;

[0139] If a match is successful, the center of the two people's leg regions will be used as the location of the object to be tracked.

[0140] If a match fails, the center of the nearest human leg will be used as the location of the object to be tracked.

[0141] The target location is determined in the second weighted image based on the location of the object to be tracked.

[0142] If the sum of the raster values ​​of the pixels within a circle with the target location as the center and a preset length as the radius is less than a second preset threshold, then the object to be tracked is taken as the target tracking object.

[0143] In one embodiment, after the determining unit 204 determines the target tracking object based on the second weighted image, the device further includes a tracking unit for tracking the target tracking object using a Kalman filter algorithm.

[0144] Based on the same inventive concept, embodiments of this application also provide a terminal device, the terminal device 300 including as follows: Figure 5 As shown.

[0145] like Figure 5As shown, the terminal device 300 of this embodiment includes: a processor 301, a memory 302, and a computer program 303 stored in the memory 302 and executable on the processor 301. The computer program 303 can be executed by the processor 301 to generate instructions, and the processor 301 can implement the steps in the above-described authorization authentication method embodiments according to the instructions. Alternatively, when the processor 301 executes the computer program 303, it implements the functions of each module / unit in the above-described device embodiments.

[0146] For example, computer program 303 may be divided into one or more modules / units, one or more of which are stored in memory 302 and executed by processor 301 to complete this application. One or more modules / units may be a series of computer program instruction segments capable of performing a specific function, which describe the execution process of computer program 303 in terminal device 300.

[0147] Those skilled in the art will understand that Figure 5 This is merely an example of terminal device 300 and does not constitute a limitation on terminal device 300. It may include more or fewer components than shown, or combine certain components, or different components. For example, terminal device 300 may also include input / output devices, network access devices, buses, etc.

[0148] Processor 301 can be a Central Processing Unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor can be a microprocessor or any conventional processor.

[0149] The memory 302 can be an internal storage unit of the terminal device 300, such as a hard disk or RAM of the terminal device 300. The memory 302 can also be an external storage device of the terminal device 300, such as a plug-in hard disk, Smart Media Card (SMC), Secure Digital (SD) card, or FlashCard equipped on the terminal device 300. Furthermore, the memory 302 can include both internal and external storage units of the terminal device 300. The memory 302 is used to store computer programs and other programs and data required by the terminal device 300. The memory 302 can also be used to temporarily store data that has been output or will be output.

[0150] The terminal device provided in this embodiment can execute the above method embodiment, and its implementation principle and technical effect are similar, so they will not be described again here.

[0151] This application also provides a computer-readable storage medium storing a computer program thereon, which, when executed by a processor, implements the method described in the above method embodiments.

[0152] This application also provides a computer program product that, when run on a terminal device, enables the terminal device to implement the method described in the above method embodiments.

[0153] If the integrated units described above are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a computer-readable storage medium, and when executed by a processor, it can implement the steps of the various method embodiments described above. The computer program includes computer program code, which can be in the form of source code, object code, executable files, or certain intermediate forms. The computer-readable storage medium can include at least: any entity or device capable of carrying the computer program code to a photographing device / terminal device, a recording medium, a computer memory, a read-only memory (ROM), a random access memory (RAM), an electrical carrier signal, a telecommunication signal, and a software distribution medium. Examples include USB flash drives, portable hard drives, magnetic disks, or optical disks.

[0154] References to "one embodiment" or "some embodiments" as described in this application mean that one or more embodiments of this application include a specific feature, structure, or characteristic described in connection with that embodiment. Therefore, the phrases "in one embodiment," "in some embodiments," "in other embodiments," "in still other embodiments," etc., appearing in different parts of this specification do not necessarily refer to the same embodiment, but rather mean "one or more, but not all, embodiments," unless otherwise specifically emphasized. The terms "comprising," "including," "having," and variations thereof mean "including but not limited to," unless otherwise specifically emphasized.

[0155] In the description of this application, it should be understood that the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features indicated. Therefore, a feature defined as "first" or "second" may explicitly or implicitly include at least one of that feature.

[0156] Furthermore, in this application, unless otherwise expressly specified and limited, the terms "connection" and "linkage" should be interpreted broadly. For example, they can refer to mechanical connections or electrical connections; they can refer to direct connections or indirect connections through an intermediate medium; they can refer to the internal connection of two components or the interaction between two components. Unless otherwise expressly limited, those skilled in the art can understand the specific meaning of the above terms in this application according to the specific circumstances.

[0157] The above embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some or all of the technical features therein. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of this application.

Claims

1. A tracking object detection method applied to a robot, the robot comprising a lidar and a camera, characterized in that, The method comprises: continuously acquiring multiple frames of laser mask images by the laser radar, updating the initial weight image based on the multiple frames of laser mask images to obtain a first weight image; when a to-be-tracked object is identified in the laser mask image, acquiring a first image of an object angle range corresponding to the to-be-tracked object by the camera; if the first image includes the to-be-tracked object, updating the first weight image by using a first detection angle range of the laser radar, a second detection angle range of the camera and the object angle range to obtain a second weight image; acquiring all human leg regions in the laser mask image; matching the two closest human leg regions in the all human leg regions; if the matching is successful, taking the center of the two human leg regions as the position of the to-be-tracked object; if the matching fails, taking the center of the closest human leg as the position of the to-be-tracked object; determining a corresponding target position in the second weight image based on the position of the to-be-tracked object; if the sum of the grid values of the pixels in a circle with the target position as the center and a preset length as the radius is less than a second preset threshold, taking the to-be-tracked object as a target tracked object.

2. The method of claim 1, wherein, updating the initial weight image based on the multiple frames of laser mask images to obtain a first weight image comprises: determining an initial weight image corresponding to the laser mask image and having each pixel being a first grid value based on the laser mask image; updating the grid value of the pixel in the initial weight image corresponding to each pixel position in the laser mask image according to the grid value corresponding to each pixel in the multiple frames of laser mask images in turn according to a first preset rule to obtain the first weight image.

3. The method of claim 2, wherein, The step of updating the grid value of the pixel in the initial weight image corresponding to each pixel position in the laser mask image according to the grid value corresponding to each pixel in the multiple frames of laser mask images in turn according to a first preset rule comprises: when the grid value of the pixel in the laser mask image is a first threshold value, the grid value of the pixel in the initial weight image corresponding to the pixel position in the laser mask image is increased by a first parameter value; when the grid value of the pixel in the laser mask image is a second threshold value, the grid value of the pixel in the initial weight image corresponding to the pixel position in the laser mask image is reduced by a second parameter value.

4. The method of claim 3, wherein, The step of updating the first weight image by using the first detection angle range of the laser radar, the second detection angle range of the camera and the object angle range to obtain a second weight image comprises: determining a first region in which neither the laser radar nor the camera detects the to-be-tracked object in the second detection angle range of the camera; determining a second region in which both the laser radar and the camera detect the to-be-tracked object; updating the grid value of the pixel in the first weight image corresponding to the first region which is greater than the second threshold value to the first threshold value; reducing the grid value of the pixel in the first weight image corresponding to the second region which is greater than the second threshold value by a third parameter.

5. The method of claim 1, wherein, After determining the target tracking object based on the second weight image, the method further includes: tracking the target tracking object by using a Kalman filtering algorithm. 6.A tracking object detection apparatus applied to a robot, the robot comprising a laser radar and a camera, characterized in that, The device includes: an acquisition unit configured to continuously acquire a plurality of laser mask images by using the laser radar, and update an initial weight image based on the plurality of laser mask images to obtain a first weight image; a first judgment unit configured to, when a to-be-tracked object is identified in the laser mask image, acquire a first image of an object angle range corresponding to the to-be-tracked object by using the camera; a second judgment unit configured to, if the first image includes the to-be-tracked object, update the first weight image by using a first detection angle range of the laser radar, a second detection angle range of the camera, and the object angle range to obtain a second weight image; a determination unit configured to acquire all human leg regions in the laser mask image, match the two closest human leg regions in the all human leg regions, if the matching is successful, take the center of the two human leg regions as the position of the to-be-tracked object, if the matching fails, take the center of the closest human leg as the position of the to-be-tracked object, determine a corresponding target position based on the position of the to-be-tracked object in the second weight image, and if the sum of the grid values of the pixels in a circle with the target position as the center and a preset length as the radius is less than a second preset threshold, take the to-be-tracked object as a target tracking object.

7. The apparatus of claim 6, wherein, updating the initial weight image based on the plurality of laser mask images to obtain the first weight image includes: an initialization unit configured to determine an initial weight image corresponding to the laser mask image and having each pixel being a first grid value based on the laser mask image; an updating unit configured to update the grid value of the pixel in the initial weight image corresponding to each pixel position in the laser mask image according to the grid value corresponding to each pixel in the plurality of laser mask images in sequence according to a first preset rule to obtain the first weight image.

8. A terminal device, comprising: The device includes a processor and a memory, the memory is configured to store a computer program, and the processor is configured to call and run the computer program from the memory, so that the device executes the method in any one of claims 1 to 5.

9. A computer-readable storage medium, characterized in that, The computer readable storage medium stores a computer program, and when the computer program is executed by the processor, the processor executes the method in any one of claims 1 to 5.

Images